U Math Eval

---
name: u-math-eval
description: Evaluates LLMs' ability to solve university-level mathematical problems, both text-based and multimodal. It also includes a meta-evaluation component to assess how well models can judge the correctness of free-form mathematical solutions. Use when the user wants to benchmark on U-MATH, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.03205
  bibtex_key: [REDACTED]
  confidence: high
---

# u-math-eval

> U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs — Chernyshev et al. (2024) (arXiv:2412.03205, 2024)

## What this evaluates

Evaluates LLMs' ability to solve university-level mathematical problems, both text-based and multimodal. It also includes a meta-evaluation component to assess how well models can judge the correctness of free-form mathematical solutions.

## Datasets

- **U-MATH** — total 1100; splits: test (1100); repo https://github.com/toloka/u-math

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of problems correctly solved, determined by GPT-4o as-a-judge comparing the model's free-form output to the ground truth.
- `macro F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall for predicting solution correctness (Yes/No). 'Inconclusive' judgments are treated as incorrect (No).

## Input / output format

**Input**: Mathematical problem statement (text or text + image). For meta-evaluation, the problem statement plus a candidate free-form solution.

**Output**: A free-form mathematical solution. For meta-evaluation, a single label: 'Yes', 'No', or 'Inconclusive'.

## Scoring recipe

```python
def score_umath(predictions, golds, judge):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if judge(pred) == gold: correct += 1
    return correct / len(golds) * 100

def score_mu_math(predictions, golds):
    tp = fp = fn = 0
    for pred, gold in zip(predictions, golds):
        if pred == 'Inconclusive': pred = 'No'
        pred_correct = (pred == 'Yes')
        if pred_correct and gold: tp += 1
        elif pred_correct and not gold: fp += 1
        elif not pred_correct and gold: fn += 1
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Text-only models evaluated on visual problems often guess or misinterpret missing images, yielding noisy accuracy.
- 'Inconclusive' judge outputs are strictly treated as incorrect, penalizing conservative models and lowering F1.
- Using LLM-as-a-judge introduces systematic bias and false positive/negative rates that limit evaluation precision.

## Evidence (verbatim from paper)

> We report accuracy based on GPT-4o-2024-08-06 as-a-judge for our final results, despite it not being the best performing judge — due to the model still residing among the top-ranked judges, being the more conservative one in terms of false positive rate, as well as widely available, leading to easier reproducibility. The judge’s output is also further processed by an extractor model (Qwen2.5 72B is fixed for consistency), prompted to produce a single label — either ‘Yes’, ‘No’ or ‘Inconclusive’. We include ‘Inconclusive’ for cases when judge refuses to evaluate or generation fails; such judgments are treated as incorrect.

## Citation

```bibtex
@misc{chernyshev2024umath,
  title={U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs},
  author={Chernyshev et al. (2024)},
  year={2024},
  note={arXiv:2412.03205}
}
```

- arXiv: 2412.03205