Liveaopsbench Eval

---
name: liveaopsbench-eval
description: Evaluates large language models' mathematical reasoning capabilities on Olympiad-level competition problems. It specifically probes whether models possess genuine problem-solving skills or merely rely on memorized pre-training data by using a continuously updated, timestamped benchmark to measure contamination-resistant accuracy. Use when the user wants to benchmark on AoPS24, Math, OlympiadBench, OmniMath, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.14275
  bibtex_key: mahdavi2025aops
  confidence: high
---

# liveaopsbench-eval

> Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation — Sadegh Mahdavi et al. (2025) (arXiv:2501.14275, 2025)

## What this evaluates

Evaluates large language models' mathematical reasoning capabilities on Olympiad-level competition problems. It specifically probes whether models possess genuine problem-solving skills or merely rely on memorized pre-training data by using a continuously updated, timestamped benchmark to measure contamination-resistant accuracy.

## Datasets

- **AoPS24** — total ?; splits: test (-1)
- **Math** — total ?; splits: test (-1)
- **OlympiadBench** — total 675; splits: test (675)
- **OmniMath** — total 4428; splits: test (4428)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of test instances where the model's final parsed answer exactly matches the ground truth answer. Calculated as (correct predictions / total predictions) * 100.

## Input / output format

**Input**: Math problem text (question), formatted within the model's specific chat template (e.g., <s>[INST] question [/INST]).

**Output**: Step-by-step solution and final answer, generated autoregressively within the model's chat template.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        pred_ans = extract_final_answer(pred)
        gold_ans = extract_final_answer(gold)
        if normalize(pred_ans) == normalize(gold_ans):
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Static benchmarks often suffer from pre-training contamination, inflating accuracy scores; this protocol mitigates it via timestamped data.
- Some Olympiad problems lack a single concrete final answer (e.g., proof-based questions), requiring manual 'no-answer' classification instead of strict exact-match.
- Benchmarks like OlympiadBench must be filtered to exclude questions containing images or figures to ensure text-only evaluation compatibility.

## Evidence (verbatim from paper)

> We also ablate the effect of solution rewriting, which is an important part of our pipeline. As shown by Figure [5(b)], rewriting solutions into a step-by-step format substantially improves the test accuracy across all benchmarks. ... verify the correctness of the final parsed answers by referring to the original post. As a result, we found that 92% of the annotations were marked as correct, while 5% were incorrect and 3% fell under the no-answer category.

## Citation

```bibtex
@misc{mahdavi2025aops,
  title={Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation},
  author={Sadegh Mahdavi et al. (2025)},
  year={2025},
  note={arXiv:2501.14275}
}
```

- arXiv: 2501.14275