Math General Reasoning Eval

---
name: math-general-reasoning-eval
description: Evaluates large language models on mathematical and general reasoning capabilities using a standardized suite of benchmarks. It measures the model's problem-solving accuracy under self-play training conditions, tracking sustained performance gains across multiple evolution iterations. Use when the user wants to benchmark on AMC, Minerva, MATH, GSM8K, Olympiad, AIME25, AIME24, SuperGPQA, MMLU-Pro, BBEH, or asks about evaluating this task. Reports pass@1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.13103
  bibtex_key: li2026rdiverse
  confidence: high
---

# math-general-reasoning-eval

> R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training — Li et al. (2026) (arXiv:2602.13103, 2026)

## What this evaluates

Evaluates large language models on mathematical and general reasoning capabilities using a standardized suite of benchmarks. It measures the model's problem-solving accuracy under self-play training conditions, tracking sustained performance gains across multiple evolution iterations.

## Datasets

- **AMC** — total ?; splits: test (-1)
- **Minerva** — total ?; splits: test (-1)
- **MATH** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **Olympiad** — total ?; splits: test (-1)
- **AIME25** — total ?; splits: test (-1)
- **AIME24** — total ?; splits: test (-1)
- **SuperGPQA** — total ?; splits: test (-1)
- **MMLU-Pro** — total ?; splits: test (-1)
- **BBEH** — total ?; splits: test (-1)

## Metrics

- `pass@1 accuracy` **(primary)** — range: [0, 1]
  - 1 if the single greedy-decoded model response exactly matches the ground truth answer, 0 otherwise. Averaged across instances.
- `mean@32` — range: [0, 1]
  - Average accuracy computed over 32 independently sampled model responses per question. Used specifically for AMC and AIME benchmarks.
- `Math AVG` — range: [0, 1]
  - Arithmetic mean of pass@1 accuracy (or mean@32 for AIME/AMC) across the seven mathematical reasoning benchmarks.
- `Overall AVG` — range: [0, 1]
  - Arithmetic mean of pass@1 accuracy (or mean@32 for AIME/AMC) across all ten mathematical and general reasoning benchmarks.

## Input / output format

**Input**: Natural language problem statements from mathematical and general reasoning benchmarks.

**Output**: Model-generated text response containing the final answer or solution steps.

## Scoring recipe

```python
def score_pass1(pred, gold):
    return 1.0 if normalize(pred) == normalize(gold) else 0.0

def score_mean32(preds, gold):
    correct = sum(1 for p in preds if normalize(p) == normalize(gold))
    return correct / 32.0

def compute_avg(accuracy_dict):
    return sum(accuracy_dict.values()) / len(accuracy_dict)
```

## Common pitfalls

- Using greedy decoding for AMC and AIME instead of mean@32, which violates the specified evaluation protocol.
- Reporting unweighted averages across benchmarks with vastly different scales or question counts.
- Confusing pass@1 (single greedy sample) with pass@k or mean@k sampling strategies.

## Evidence (verbatim from paper)

> We evaluate on seven mathematical and three general reasoning benchmarks (App.[E]). We report pass@1 accuracy with greedy decoding for all benchmarks except AMC and AIME, where we use mean@32 following prior work*(Huang et al., [2025])*.

## Citation

```bibtex
@misc{li2026rdiverse,
  title={R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2602.13103}
}
```

- arXiv: 2602.13103