Naturalreasoning Eval

---
name: naturalreasoning-eval
description: Evaluates the zero-shot reasoning capabilities of models trained via knowledge distillation or self-training on the NaturalReasoning dataset. It measures performance across diverse mathematics and science benchmarks to assess scaling efficiency and generalization. Use when the user wants to benchmark on MATH, GPQA, GPQA-Diamond, MMLU-Pro, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.13124
  bibtex_key: yuan2025naturalreasoning
  confidence: high
---

# naturalreasoning-eval

> NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions — Yuan et al. (2025) (arXiv:2502.13124, 2025)

## What this evaluates

Evaluates the zero-shot reasoning capabilities of models trained via knowledge distillation or self-training on the NaturalReasoning dataset. It measures performance across diverse mathematics and science benchmarks to assess scaling efficiency and generalization.

## Datasets

- **MATH** — total ?; splits: test (-1)
- **GPQA** — total ?; splits: test (-1)
- **GPQA-Diamond** — total ?; splits: test (-1)
- **MMLU-Pro** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match accuracy: the proportion of instances where the model's predicted answer exactly matches the ground truth label.

## Input / output format

**Input**: Zero-shot prompts for mathematics and science reasoning questions.

**Output**: Model-generated answers or reasoning traces.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
    total += 1
return correct / total
```

## Common pitfalls

- Averaging accuracy over the last three saved checkpoints during training rather than the final model.
- Using greedy decoding for inference, which may penalize models that benefit from temperature sampling.
- Evaluating strictly in a zero-shot setting without few-shot examples or chain-of-thought prompting.

## Evidence (verbatim from paper)

> We evaluate our models on a diverse set of benchmarks that encompass both math and science reasoning: MATH, GPQA, GPQA-Diamond*(Rein et al., [2024])* and MMLU-Pro. To ensure a fair and consistent comparison, we adopt a zero-shot evaluation setting across all trained models. For inference we use vllm *(Kwon et al., [2023])* and employ greedy decoding to maintain determinism and eliminate variability introduced by stochastic generation. Unless mentioned otherwise, we report accuracy averaged over the last three saved model checkpoints during training.

## Citation

```bibtex
@misc{yuan2025naturalreasoning,
  title={NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questions},
  author={Yuan et al. (2025)},
  year={2025},
  note={arXiv:2502.13124}
}
```

- arXiv: 2502.13124