Usamo Proof Eval

---
name: usamo-proof-eval
description: This benchmark probes the ability of large language models to generate rigorous, step-by-step mathematical proofs for high-school olympiad-level problems. It evaluates logical coherence, justification of assumptions, and adherence to formal proof standards rather than just numerical correctness. Use when the user wants to benchmark on 2025 USA Math Olympiad, or asks about evaluating this task. Reports proof_points.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.21934
  bibtex_key: petrov2025prooforbluff
  confidence: high
---

# usamo-proof-eval

> Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad — Ivo Petrov et al. (2025) (arXiv:2503.21934, 2025)

## What this evaluates

This benchmark probes the ability of large language models to generate rigorous, step-by-step mathematical proofs for high-school olympiad-level problems. It evaluates logical coherence, justification of assumptions, and adherence to formal proof standards rather than just numerical correctness.

## Datasets

- **2025 USA Math Olympiad** — total 6; splits: test (6); repo https://github.com/eth-sri/matharena

## Metrics

- `proof_points` **(primary)** — range: [0, 42]
  - Sum of points awarded across all grading categories in the step-wise rubric. Maximum possible score is 42 points across the 6 problems.

## Input / output format

**Input**: Problem statement embedded in a user prompt requesting a thorough, LaTeX-formatted answer with all proof steps included.

**Output**: LaTeX-formatted mathematical proof containing all logical steps, bounds, and justifications.

## Scoring recipe

```python
def compute_proof_points(solution, grading_scheme):
    total = 0
    for category in grading_scheme:
        awarded = judge_grade(solution, category['desc'])
        total += awarded
    return total
```

## Common pitfalls

- Models often skip crucial logical steps or make unjustified assumptions, which heavily penalizes the score.
- Failing to rigorously prove bounds or choose sufficiently large N leads to lost points even if the final result is correct.
- LLM judges may struggle with subjective creativity requirements in proof construction, leading to inconsistent scoring.

## Evidence (verbatim from paper)

> The awarded number of points for each category should follow after the 'points' key - The reasoning for each decision should be explained after the 'desc' key.

## Citation

```bibtex
@misc{petrov2025prooforbluff,
  title={Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad},
  author={Ivo Petrov et al. (2025)},
  year={2025},
  note={arXiv:2503.21934}
}
```

- arXiv: 2503.21934