Llm Reasoning Rl Eval

---
name: llm-reasoning-rl-eval
description: This evaluation protocol assesses the mathematical reasoning and multi-step problem-solving capabilities of LLMs after reinforcement learning fine-tuning. It measures how well models generalize from a specialized arithmetic training task to standard academic and competitive math benchmarks. Use when the user wants to benchmark on GSM8K, BBH, MATH, MMLU-Pro, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.07611
  bibtex_key: lian2025comparative
  confidence: high
---

# llm-reasoning-rl-eval

> Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement — Lian (2025) (arXiv:2512.07611, 2025)

## What this evaluates

This evaluation protocol assesses the mathematical reasoning and multi-step problem-solving capabilities of LLMs after reinforcement learning fine-tuning. It measures how well models generalize from a specialized arithmetic training task to standard academic and competitive math benchmarks.

## Datasets

- **GSM8K** — total 8500; splits: test (-1)
- **BBH** — total ?; splits: test (-1)
- **MATH** — total 12500; splits: test (-1)
- **MMLU-Pro** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered problems out of the total number of problems in the benchmark split. Computed via exact match or normalized string comparison for open-ended math tasks, and option selection for multiple-choice tasks.

## Input / output format

**Input**: Natural language math word problems, multi-step reasoning prompts, or multiple-choice questions presented as text.

**Output**: Text generation up to a maximum of 2048 tokens, containing the model's reasoning steps and final answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == normalize_answer(g))
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Failing to cap the maximum output token length, which artificially inflates performance on math benchmarks (Table 2 shows gains up to 8192 tokens).
- Using inconsistent prompt formatting or stopping criteria, as LLM evaluation results are highly sensitive to these protocol choices.

## Evidence (verbatim from paper)

> Among the methods tested, DAPO without dynamic sampling emerged as the most effective algorithm, achieving the highest accuracy on all four benchmarks, including a peak of $53.3\%$ on GSM8K and $30.0\%$ on MMLU-Pro.

## Citation

```bibtex
@misc{lian2025comparative,
  title={Comparative Analysis and Parametric Tuning of PPO, GRPO, and DAPO for LLM Reasoning Enhancement},
  author={Lian (2025)},
  year={2025},
  note={arXiv:2512.07611}
}
```

- arXiv: 2512.07611