Olymmath Eval

---
name: olymmath-eval
description: Evaluates advanced mathematical reasoning capabilities on Olympiad-level problems. It probes a model's ability to perform rigorous, step-by-step logical deduction and numerical verification across algebra, geometry, number theory, and combinatorics. The benchmark also assesses cross-lingual reasoning performance between English and Chinese. Use when the user wants to benchmark on OlymMATH, or asks about evaluating this task. Reports Pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.21380
  bibtex_key: sun2025olymmath
  confidence: high
---

# olymmath-eval

> Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models — Sun et al. (2025) (arXiv:2503.21380, 2025)

## What this evaluates

Evaluates advanced mathematical reasoning capabilities on Olympiad-level problems. It probes a model's ability to perform rigorous, step-by-step logical deduction and numerical verification across algebra, geometry, number theory, and combinatorics. The benchmark also assesses cross-lingual reasoning performance between English and Chinese.

## Datasets

- **OlymMATH** — total 400; splits: OlymMATH-EN-HARD (-1), OlymMATH-EN-EASY (-1), OlymMATH-ZH-HARD (-1), OlymMATH-ZH-EASY (-1); repo https://github.com/RUCAIBox/OlymMATH

## Metrics

- `Pass@1` **(primary)** — range: [0, 1]
  - Mean accuracy across all sampled responses for a given problem. Computed as the fraction of problems where the single sampled answer matches the gold answer.
- `Cons@k` — range: [0, 1]
  - Accuracy after applying majority voting across k sampled responses per problem. The consensus answer is the most frequent prediction, and accuracy is the fraction of problems where this consensus matches the gold answer.

## Input / output format

**Input**: A mathematical problem statement presented in either English or Chinese.

**Output**: A final numerical answer, optionally preceded by a chain-of-thought reasoning trace.

## Scoring recipe

```python
def pass_at_1(predictions, gold):
    return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)

def cons_at_k(predictions_list, gold, k):
    correct = 0
    for preds, g in zip(predictions_list, gold):
        consensus = max(set(preds), key=preds.count)
        if consensus == g: correct += 1
    return correct / len(gold)
```

## Common pitfalls

- Models may use empirical guesses or heuristics that yield correct answers but lack rigorous reasoning, misleading rule-based or LLM-as-judge evaluators.
- Performance stability varies significantly with sample size; models evaluated with only 8 samples show higher variance than those with 64 samples.
- English-language benchmarks consistently yield higher scores due to pre-training data bias, which can skew cross-lingual comparisons if not accounted for.

## Evidence (verbatim from paper)

> For the Pass@1 metric, we compute the mean accuracy across all sampled responses to derive the final accuracy score. For the Cons@64 and Cons@8 metric, we implement majority voting to determine a consensus answer for each problem, subsequently calculating the average accuracy across the entire dataset.

## Citation

```bibtex
@misc{sun2025olymmath,
  title={Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models},
  author={Sun et al. (2025)},
  year={2025},
  note={arXiv:2503.21380}
}
```

- arXiv: 2503.21380