Gsm8k Eval

---
name: gsm8k-eval
description: Evaluates a model's ability to perform multi-step arithmetic reasoning by generating natural language solutions to grade school math word problems and verifying their correctness. Use when the user wants to benchmark on GSM8K, or asks about evaluating this task. Reports solve rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.14168
  bibtex_key: cobbe2021trainingverifiers
  confidence: high
---

# gsm8k-eval

> Training Verifiers to Solve Math Word Problems — Cobbe et al. (2021) (arXiv:2110.14168, 2021)

## What this evaluates

Evaluates a model's ability to perform multi-step arithmetic reasoning by generating natural language solutions to grade school math word problems and verifying their correctness.

## Datasets

- **GSM8K** — total ?; splits: train (-1), test (-1)

## Metrics

- `solve rate` **(primary)** — range: percent
  - The percentage of test problems for which the model's generated final answer exactly matches the ground truth answer. Evaluated as test@1 (single low-temperature sample) or test@N (best-of-N samples).

## Input / output format

**Input**: A natural language grade school math word problem.

**Output**: A step-by-step natural language solution followed by a final numerical answer.

## Scoring recipe

```python
def compute_solve_rate(predictions, gold_answers, n_samples=1):
    correct = 0
    for i in range(0, len(predictions), n_samples):
        batch = predictions[i:i+n_samples]
        gold = gold_answers[i // n_samples]
        final_answers = [extract_final_answer(completion) for completion in batch]
        if gold in final_answers:
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Models may reach the correct final answer using flawed reasoning, leading to false positives in verification.
- Overfitting to the training set causes the model to become overconfident and reduces solution space coverage at test time.
- Temperature selection critically impacts the trade-off between solution diversity (coverage) and calibration.

## Evidence (verbatim from paper)

> At test time, we judge performance by autoregressively sampling a single low temperature solution and checking whether the final answer is correct. ... We use test@N to denote the percentage of problems solved correctly at least once when allowing the model to make N separate guesses for each problem. ... estimate that a model with 10^16 parameters would be required to reach an 80% solve rate, when using the full GSM8K training set.

## Citation

```bibtex
@misc{cobbe2021trainingverifiers,
  title={Training Verifiers to Solve Math Word Problems},
  author={Cobbe et al. (2021)},
  year={2021},
  note={arXiv:2110.14168}
}
```

- arXiv: 2110.14168