Imo Shortlist Eval

---
name: imo-shortlist-eval
description: This benchmark probes an LLM's ability to produce logically sound, step-by-step mathematical reasoning for Olympiad-level problems. It specifically measures the gap between achieving the correct final answer and maintaining rigorous, fallacy-free solution processes. Use when the user wants to benchmark on IMO shortlist problems (2009-2023), or asks about evaluating this task. Reports Final Answer Accuracy (%), Correct|Correct Final Answer (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.01995
  bibtex_key: mahdavi2025brainsvsbytes
  confidence: high
---

# imo-shortlist-eval

> Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics — Mahdavi et al. (2025) (arXiv:2504.01995, 2025)

## What this evaluates

This benchmark probes an LLM's ability to produce logically sound, step-by-step mathematical reasoning for Olympiad-level problems. It specifically measures the gap between achieving the correct final answer and maintaining rigorous, fallacy-free solution processes.

## Datasets

- **IMO shortlist problems (2009-2023)** — total 455; splits: test (455)

## Metrics

- `Final Answer Accuracy (%)` **(primary)** — range: percent
  - Percentage of problems where the model's final numerical or symbolic answer matches the ground truth, regardless of the reasoning steps.
- `Correct|Correct Final Answer (%)` **(primary)** — range: percent
  - Conditional probability of a fully correct solution (free of logical fallacies) given that the final answer is correct.
- `Correct (%)` — range: percent
  - Human-annotated percentage of solutions deemed fully correct based on logical soundness and completeness.
- `Partially Correct (%)` — range: percent
  - Human-annotated percentage of solutions with some valid steps but containing logical gaps or minor errors.
- `Incorrect (%)` — range: percent
  - Human-annotated percentage of solutions with fundamental errors or completely wrong approaches.

## Input / output format

**Input**: Olympiad-level mathematics problems from the IMO shortlists (2009-2023), spanning algebra, combinatorics, geometry, and number theory.

**Output**: Step-by-step mathematical solutions concluding with a final answer.

## Scoring recipe

```python
def evaluate(predictions, gold_answers, human_labels):
    # 1. Final Answer Accuracy
    correct_answers = sum(1 for p, g in zip(predictions, gold_answers) if p.final_answer == g)
    final_acc = correct_answers / len(predictions)

    # 2. Correct|Correct Final Answer
    correct_answer_indices = [i for i, p in enumerate(predictions) if p.final_answer == gold_answers[i]]
    if not correct_answer_indices:
        correct_given_acc = 0.0
    else:
        correct_solutions = sum(1 for i in correct_answer_indices if human_labels[i].solution_correct)
        correct_given_acc = correct_solutions / len(correct_answer_indices)

    # 3. Fallacy Frequencies
    fallacy_counts = {f: 0 for f in human_labels.fallacy_types}
    for label in human_labels:
        for f in label.fallacies:
            fallacy_counts[f] += 1
    return final_acc, correct_given_acc, fallacy_counts
```

## Common pitfalls

- Relying solely on final answer accuracy grossly overestimates reasoning abilities, as models often use heuristics or trial-and-error to guess answers while containing logical fallacies.
- Human evaluation is strictly required to assess solution soundness and identify specific fallacies (e.g., Proof by Example, Inventing Wrong Facts), which automated exact-match checks cannot capture.
- Problems without explicit final answers require logical proofs, which significantly increases the frequency of fallacies like Proposal Without Verification and Begging the Question.

## Evidence (verbatim from paper)

> Final Answer Accuracy denotes the percentage of correct final answers, whereas Correct|Correct Final Answer represents the percentage of fully correct solutions among instances where the final answer is correct.

## Citation

```bibtex
@misc{mahdavi2025brainsvsbytes,
  title={Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics},
  author={Mahdavi et al. (2025)},
  year={2025},
  note={arXiv:2504.01995}
}
```

- arXiv: 2504.01995