Imo2025 Eval

---
name: imo2025-eval
description: Evaluates a model's ability to solve Olympiad-level mathematical proof problems by generating rigorous solutions and iteratively refining them through self-verification. Use when the user wants to benchmark on IMO 2025, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.15855
  bibtex_key: [REDACTED]
  confidence: high
---

# imo2025-eval

> Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipeline — Huang et al. (2025) (arXiv:2507.15855, 2025)

## What this evaluates

Evaluates a model's ability to solve Olympiad-level mathematical proof problems by generating rigorous solutions and iteratively refining them through self-verification.

## Datasets

- **IMO 2025** — total 6; splits: test (6); repo https://github.com/lyang36/IMO25

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of problems for which the pipeline accepts a solution. A solution is accepted only if it passes the verifier five consecutive times without critical errors or major justification gaps.

## Input / output format

**Input**: Natural language statement of a mathematical proof problem from the IMO 2025 competition.

**Output**: A step-by-step mathematical proof or solution. The pipeline accepts the output if it robustly passes verification; otherwise it is rejected.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for problem, solution in zip(predictions, gold):
        passes = True
        for _ in range(5):
            if verifier(solution) != 'PASS':
                passes = False
                break
        if passes:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- The verifier is model-agnostic and runs iteratively; accuracy depends heavily on the number of verification passes (fixed at 5 in this work).
- Solutions are evaluated on mathematical rigor rather than just the final answer, so minor justification gaps may cause rejection even if the final result is correct.

## Evidence (verbatim from paper)

> We accept a solution if it robustly passes the verification process and decline a solution if there are always critical errors or major justification gaps during the iterations. ... we run the verifier five times and accept a solution only if it passes every time.

## Citation

```bibtex
@misc{huang2025winninggoldimo2025,
  title={Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipeline},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2507.15855}
}
```

- arXiv: 2507.15855