Mathodyssey Eval

---
name: mathodyssey-eval
description: Evaluates large language models' mathematical reasoning capabilities across Olympiad, high school, and university-level problems. It probes multi-step logic, chain-of-thought reasoning, and advanced derivations in algebra, calculus, and number theory. Use when the user wants to benchmark on MathOdyssey, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.18321
  bibtex_key: fang2024mathodyssey
  confidence: high
---

# mathodyssey-eval

> MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data — Fang et al. (2024) (arXiv:2406.18321, 2024)

## What this evaluates

Evaluates large language models' mathematical reasoning capabilities across Olympiad, high school, and university-level problems. It probes multi-step logic, chain-of-thought reasoning, and advanced derivations in algebra, calculus, and number theory.

## Datasets

- **MathOdyssey** — total 387; splits: test (387)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Accuracy is the fraction of correctly answered problems out of the total. Correctness is determined by verifying mathematical equivalence between the predicted and ground-truth answers using symbolic computation, numerical tolerance (correct to at least two decimal places), and multiple-choice matching rules.

## Input / output format

**Input**: Mathematical problem statement (text), evaluated using chain-of-thought prompting.

**Output**: Step-by-step reasoning followed by a final answer (exact value, choice label, or numerical approximation).

## Scoring recipe

```python
def is_correct(pred, gold):
    if pred == gold: return True
    if is_numerically_equivalent(pred, gold, tol=0.01): return True
    if is_symbolically_equivalent(pred, gold): return True
    if is_multiple_choice_match(pred, gold): return True
    return False

accuracy = sum(1 for p, g in zip(predictions, golds) if is_correct(p, g)) / len(golds)
```

## Common pitfalls

- Assuming exact string matching is sufficient; the protocol explicitly requires symbolic and numerical equivalence checking.
- Overlooking the chain-of-thought requirement; models must be prompted with CoT to generate step-by-step reasoning before the final answer.
- Misinterpreting numerical tolerance; answers must be correct to at least two decimal places, not just rounded integers.

## Evidence (verbatim from paper)

> We use GPT-4 to assist in evaluating model accuracy, particularly for open-answer questions. The metric measures the similarity between the predicted and ground truth answers. In the MathOdyssey dataset, various types of questions and answers are included. We employ a prompt-based method to provide scores for evaluation, considering the following criteria: Mathematical Equivalence: Verify answers based on mathematical equivalence using advanced tools like symbolic computation software to confirm the equivalence of different algebraic or symbolic expressions. Scoring: Assign a score of ‘1’ for answers that match or are equivalent to the provided solution (exact value, choice label, or correctly rounded numerical approximation). Assign a score of ‘0’ for incorrect answers without providing explanatory feedback.

## Citation

```bibtex
@misc{fang2024mathodyssey,
  title={MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data},
  author={Fang et al. (2024)},
  year={2024},
  note={arXiv:2406.18321}
}
```

- arXiv: 2406.18321