Deepmath 103k Eval

---
name: deepmath-103k-eval
description: Evaluates mathematical reasoning capabilities on a curated, decontaminated dataset of challenging problems, measuring performance across standardized math competitions and academic benchmarks. Use when the user wants to benchmark on DeepMath-103K, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.11456
  bibtex_key: he2025deepmath103k
  confidence: high
---

# deepmath-103k-eval

> DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning — He et al. (2025) (arXiv:2504.11456, 2025)

## What this evaluates

Evaluates mathematical reasoning capabilities on a curated, decontaminated dataset of challenging problems, measuring performance across standardized math competitions and academic benchmarks.

## Datasets

- **DeepMath-103K** — total 103000; splits: train (-1); repo https://github.com/zwhe99/DeepMath

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of problems where the model's final answer exactly matches the verified ground truth.

## Input / output format

**Input**: Mathematical problem statements from DeepMath-103K, typically requiring step-by-step reasoning or direct answers.

**Output**: Model-generated solutions or final answers.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
return (correct / len(predictions)) * 100
```

## Common pitfalls

- Benchmarks like AIME and AMC use exact-match or verified answer formats, not partial credit.
- Decontamination is critical; standard benchmarks often contain training data leakage, which this dataset explicitly addresses.

## Evidence (verbatim from paper)

> Table 5: Math reasoning performance after fine-tuning Qwen-2.5-7B via SFT. We also add DeepMath-Zero-7B as an RL counterpart for reference.

## Citation

```bibtex
@misc{he2025deepmath103k,
  title={DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning},
  author={He et al. (2025)},
  year={2025},
  note={arXiv:2504.11456}
}
```

- arXiv: 2504.11456