Omni Math Eval

---
name: omni-math-eval
description: Evaluates large language models on rigorous, Olympiad-level mathematical reasoning across diverse domains and difficulty levels. It probes the model's ability to perform complex logical deduction, multi-step problem solving, and handle non-standard answer formats without relying on trivial or non-mathematical content. Use when the user wants to benchmark on Omni-MATH, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.07985
  bibtex_key: gao2024omnimath
  confidence: high
---

# omni-math-eval

> Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models — Gao et al. (2024) (arXiv:2410.07985, 2024)

## What this evaluates

Evaluates large language models on rigorous, Olympiad-level mathematical reasoning across diverse domains and difficulty levels. It probes the model's ability to perform complex logical deduction, multi-step problem solving, and handle non-standard answer formats without relying on trivial or non-mathematical content.

## Datasets

- **Omni-MATH** — total 4428; splits: test (4428); repo https://github.com/KbsdJames/Omni-MATH

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - The proportion of problems where the model's generated solution is judged consistent with the reference answer. Calculated as (number of correctly solved problems) / (total number of problems).

## Input / output format

**Input**: Mathematical problem statement (typically in LaTeX or text) and the corresponding reference answer.

**Output**: Model-generated solution or final answer.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        # LLM judge compares pred.solution with gold.answer
        is_consistent = llm_judge_consistency_check(pred.solution, gold.answer)
        if is_consistent:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Diverse final answer formats make fixed rule-based evaluation unreliable, necessitating LLM-based consistency checks.
- Model outputs may contain correct reasoning but fail due to formatting issues or require additional steps to extract the final answer for evaluation.

## Evidence (verbatim from paper)

> We employ cross-validation to enhance the robustness of our findings, yielding an accuracy rate of 92.7%.

## Citation

```bibtex
@misc{gao2024omnimath,
  title={Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models},
  author={Gao et al. (2024)},
  year={2024},
  note={arXiv:2410.07985}
}
```

- arXiv: 2410.07985