Olympiad Math Eval

---
name: olympiad-math-eval
description: This evaluation probes an agent's ability to solve complex, long-horizon mathematical problems requiring multi-step deduction, proof construction, and lemma-based reasoning. It tests performance across standard competition benchmarks (AIME, HMMT) and rigorous Olympiad-level contests (IMO, CNMO, CMO), focusing on non-geometry problems where novel proof paths are required. Use when the user wants to benchmark on AIME2025, HMMT2025 Feb, IMO2025, CNMO2025, CMO2025, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.10739
  bibtex_key: gao2025longhorizon
  confidence: high
---

# olympiad-math-eval

> Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving — Songyang Gao et al. (arXiv:2512.10739, 2025)

## What this evaluates

This evaluation probes an agent's ability to solve complex, long-horizon mathematical problems requiring multi-step deduction, proof construction, and lemma-based reasoning. It tests performance across standard competition benchmarks (AIME, HMMT) and rigorous Olympiad-level contests (IMO, CNMO, CMO), focusing on non-geometry problems where novel proof paths are required.

## Datasets

- **AIME2025** — total ?; splits: test (-1)
- **HMMT2025 Feb** — total ?; splits: test (-1)
- **IMO2025** — total ?; splits: test (-1)
- **CNMO2025** — total ?; splits: test (-1)
- **CMO2025** — total 6; splits: test (6)

## Metrics

- `pass@1` **(primary)** — range: [0, 1]
  - Unbiased pass@k estimator: E[1/k * sum_{i=1}^k I(solution_i is correct)]. Correctness is determined by fine-grained grading points via CompassVerifier (solution-based) or OPV (proof-based). For IMO2025, pass@4 is used instead.

## Input / output format

**Input**: Mathematical competition problems (solution-based and proof-based), excluding geometry for IMO2025 and CNMO2025.

**Output**: Final mathematical solution or proof, generated via multi-round hierarchical decomposition and lemma-based memory management.

## Scoring recipe

```python
def compute_passk(predictions, k, verifier_type):
    correct = []
    for pred in predictions[:k]:
        if verifier_type == 'solution':
            correct.append(1 if CompassVerifier(pred) else 0)
        else:
            correct.append(1 if OPV(pred) else 0)
    return sum(correct) / k
# Usage: k=1 for AIME/HMMT/CNMO/CMO; k=4 for IMO2025.
# For CMO2025, human experts grade per question (max 21 pts each, total 126).
```

## Common pitfalls

- Geometry problems are explicitly excluded from IMO2025 and CNMO2025 evaluations.
- The pass@k value varies by dataset: pass@1 is used for AIME2025, HMMT2025, CNMO2025, and CMO2025, while pass@4 is strictly used for IMO2025.
- Two different verifiers are employed based on problem type: CompassVerifier for solution-based questions and OPV for proof-based questions.

## Evidence (verbatim from paper)

> For each sample, we perform 16 independent rollouts and use the unbiased pass@1 [chen2021evaluating] as the metric, except for IMO2025, which we use pass@4.

## Citation

```bibtex
@misc{gao2025longhorizon,
  title={Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving},
  author={Songyang Gao et al.},
  year={2025},
  note={arXiv:2512.10739}
}
```

- arXiv: 2512.10739