Math Reasoning Diversity Eval

---
name: math-reasoning-diversity-eval
description: This evaluation protocol probes the mathematical reasoning capabilities and solution diversity of large language models. It measures how well models solve grade-school and college-level math problems, and whether reinforcement learning fine-tuning preserves or degrades the variety of generated solution paths. Use when the user wants to benchmark on GSM8K, MATH500, Olympiad Bench, College Math, or asks about evaluating this task. Reports Pass@1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.23433
  bibtex_key: [REDACTED]
  confidence: high
---

# math-reasoning-diversity-eval

> Diversity-Aware Policy Optimization for Large Language Model Reasoning — Jian Yao et al. (2025) (arXiv:2505.23433, 2025)

## What this evaluates

This evaluation protocol probes the mathematical reasoning capabilities and solution diversity of large language models. It measures how well models solve grade-school and college-level math problems, and whether reinforcement learning fine-tuning preserves or degrades the variety of generated solution paths.

## Datasets

- **GSM8K** — total ?; splits: train (-1), test (1319); HF `gsm8k`
- **MATH500** — total ?; splits: test (500); HF `hendrycks2021measuring`
- **Olympiad Bench** — total ?; splits: test (-1); HF `he2024olympiadbench`
- **College Math** — total ?; splits: test (-1); HF `tang2024mathscale`

## Metrics

- `Pass@1 accuracy` **(primary)** — range: percent
  - Percentage of questions where the single generated response exactly matches the ground truth answer.
- `Avg@8 accuracy` — range: percent
  - Average accuracy computed by generating 8 responses per question and checking if any matches the ground truth, averaged across all questions.
- `Div-Equ` — range: percent
  - Proportion of distinct equations or solution paths across generated responses.
- `Div-N-gram` — range: percent
  - Proportion of distinct n-grams per response, capturing intra-response diversity.
- `Div-Self-BLEU` — range: percent
  - 100 minus the Self-BLEU score across generated responses, capturing inter-response diversity.

## Input / output format

**Input**: Mathematical word problems or equations presented as text prompts.

**Output**: Natural language reasoning steps followed by a final numerical or symbolic answer.

## Scoring recipe

```python
def compute_pass_at_k(predictions, gold_answers, k=1):
    correct = 0
    for i in range(0, len(predictions), k):
        batch = predictions[i:i+k]
        gold = gold_answers[i // k]
        if any(extract_final_answer(p) == gold for p in batch):
            correct += 1
    return (correct / len(gold_answers)) * 100

# For Avg@8, set k=8. For Pass@1, set k=1.
# Diversity metrics are computed over the set of generated responses per question.
```

## Common pitfalls

- Using benchmarks with very few test items (e.g., AIME24 with 30 questions) leads to unstable and biased evaluation outcomes.
- Direct performance comparisons across baselines are confounded by differing computational resources and training datasets used during RL fine-tuning.
- Diversity metrics like Self-BLEU and N-gram diversity can be sensitive to tokenization choices and response formatting conventions.

## Evidence (verbatim from paper)

> We evaluate the reasoning performance using Pass@1 accuracy, as shown in Table[1]. We also report the performance against training steps in Figure [4] against the training steps ‣ Appendix E More experiment results ‣ Diversity-Aware Policy Optimization for Large Language Model Reasoning") in Appendix. In our experiment, R1-zero-Div demonstrates superior performance compared to R1-zero, achieving an average improvement of $3.5\%$.

## Citation

```bibtex
@misc{yao2025diversityaware,
  title={Diversity-Aware Policy Optimization for Large Language Model Reasoning},
  author={Jian Yao et al. (2025)},
  year={2025},
  note={arXiv:2505.23433}
}
```

- arXiv: 2505.23433