Gsm8k Eval

---
name: gsm8k-eval
description: Evaluate an LLM on GSM8K — 1K grade-school math word problems requiring 2-8 step arithmetic reasoning. Use when the user wants to measure math reasoning, mentions GSM8K, or asks "how good is my model at multi-step word problems?". Reports exact-match accuracy on the final numeric answer (parsed from "#### N" suffix).
metadata:
  source_paper: arXiv:2110.14168
  source_repo: https://github.com/openai/grade-school-math
  task_type: math-word-problems
  metrics: [exact_match_accuracy, "test@N (pass-rate over N samples)"]
  splits: [train (7473), test (1319)]
---

# gsm8k-eval — Grade-school math word problems

Source: Cobbe et al., "Training Verifiers to Solve Math Word Problems" (arXiv:2110.14168).

## What this benchmark tests

Multi-step arithmetic reasoning over 2–8-step word problems. Solutions are natural language with a final numeric answer. Designed to be solvable by a "bright middle school student" but to expose LMs' shaky chain-of-thought arithmetic.

## When to invoke this skill

- User says "test on GSM8K" / "evaluate math reasoning" / "math word problems"
- User asks for a single-number sanity check on basic LM math
- User mentions verifier-style approaches (test@N), self-consistency, MathPrompter, etc. — they all use this dataset

## Dataset structure

Each example has:
```json
{
  "question": "Natalia sold clips to 48 of her friends in April...",
  "answer": "Natalia sold 48/2 = <<48/2=24>>24 clips in May.\nNatalia sold 48+24 = <<48+24=72>>72 clips in April and May.\n#### 72"
}
```
- The chain-of-thought is in plain text, with `<<expr=val>>` calculator annotations.
- The **final numeric answer** is everything after `####`.

Splits (HF `gsm8k`, config `main` or `socratic`):
- `train`: 7,473 problems (use for finetuning, NOT eval)
- `test`: 1,319 problems (the only thing you should report on)

## Evaluation protocol

1. **Input prompt**: the `question` text. The paper's recipe is direct: ask the model for a step-by-step solution ending with `#### <number>`. Modern recipes use 8-shot CoT prompts (Wei et al. 2022) for reproducibility.
2. **Sampling**:
   - `test@1`: temperature 0, single sample → check final answer.
   - `test@N`: temperature 0.7, N samples → check whether **any** is correct (`pass@N`).
   - `maj@N`: temperature 0.7, N samples → take majority vote on extracted answer.
3. **Answer extraction**: regex the model's final line for `####\s*(-?[\d,]+(?:\.\d+)?)`. Strip commas. Compare numerically (not as string) to the gold final answer (also extracted via the same regex from the gold `answer`).
4. **Metric**: `exact_match_accuracy = #correct / #total`. Report 2–4 decimal places.

## Reference scoring

```python
import re
def extract_answer(text: str) -> float | None:
    m = re.search(r"####\s*(-?[\d,]+(?:\.\d+)?)", text)
    return float(m.group(1).replace(",", "")) if m else None

def gsm8k_em(pred: str, gold: str) -> bool:
    p, g = extract_answer(pred), extract_answer(gold)
    return p is not None and g is not None and abs(p - g) < 1e-6

# usage
from datasets import load_dataset
ds = load_dataset("gsm8k", "main", split="test")
correct = sum(gsm8k_em(your_model_output(ex["question"]), ex["answer"]) for ex in ds)
print(f"acc = {correct/len(ds):.4f}")
```

## Reporting format

Always state exact-match accuracy + which protocol:
- `gsm8k acc (test@1, 0-shot CoT) = 0.812`
- `gsm8k acc (test@1, 8-shot CoT) = 0.873`
- `gsm8k maj@64 (T=0.7) = 0.926`

## Don'ts

- Don't compare different sampling configs as if they were the same metric — `maj@64` ≠ `test@1`.
- Don't report on `train`. The 1319 in `test` is the only legitimate number.
- Don't string-compare the final answer; "1,000" vs "1000" vs "1000.0" must all match.
- Don't forget that some gold answers are negative; the regex must capture leading `-`.

## Citation

```bibtex
@article{cobbe2021gsm8k,
  title={Training Verifiers to Solve Math Word Problems},
  author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and others},
  journal={arXiv:2110.14168}, year={2021}
}
```