Mathnet Solve Eval

---
name: mathnet-solve-eval
description: Evaluates a model's ability to solve Olympiad-level mathematical problems across multiple domains (algebra, geometry, combinatorics, number theory) and modalities (text and images). It measures whether models can produce consistent, correct reasoning rather than just guessing the final answer. Use when the user wants to benchmark on MathNet-Solve, or asks about evaluating this task. Reports Problem Solving Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18584
  bibtex_key: [REDACTED]
  confidence: high
---

# mathnet-solve-eval

> MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval — Shaden Alshammari et al. (arXiv:2604.18584, 2026)

## What this evaluates

Evaluates a model's ability to solve Olympiad-level mathematical problems across multiple domains (algebra, geometry, combinatorics, number theory) and modalities (text and images). It measures whether models can produce consistent, correct reasoning rather than just guessing the final answer.

## Datasets

- **MathNet-Solve** — total 6400; splits: test (6400); repo https://github.com/ShadeAlsha/MathNet

## Metrics

- `Problem Solving Accuracy` **(primary)** — range: percent
  - Binarized score from a GPT-5 judge. The judge scores the model's solution against the reference solution on a 0–7 scale. Scores ≥ 6 are marked correct (1), others incorrect (0). Accuracy is the percentage of correct solutions.

## Input / output format

**Input**: Problem statement (text and/or image). For multimodal models, both text and image are provided; otherwise, a text-only description of the image is supplied.

**Output**: A generated solution to the mathematical problem.

## Scoring recipe

```python
score = gpt5_judge(problem_statement, reference_solution, model_output) # 0-7
is_correct = 1 if score >= 6 else 0
accuracy = sum(is_correct) / total_problems
```

## Common pitfalls

- Relies on an LLM judge (GPT-5) for grading rather than exact string matching, which introduces potential judge bias or inconsistency.
- Binarizes a 0-7 scale at ≥ 6, so minor errors are penalized heavily compared to a lenient threshold.
- Evaluates both text-only and multimodal inputs; models must handle image descriptions if they lack native vision capabilities.

## Evidence (verbatim from paper)

> On MathNet-Solve, we evaluate two types of models: (i) LLMs and LMMs... For each problem, the judge model is provided with the problem statement, the reference solution, and the model-generated solution, and is asked to assess whether the output is consistent with the correct answer using a numeric score from 0–7. We binarize the score by marking outputs with score ≥ 6 as correct (fully correct or containing only minor errors) and scores <6 as incorrect. This allows us to distinguish between models that arrive at the correct final answer by coincidence versus those that demonstrate consistent reasoning ability. Table 3: Problem Solving Accuracy (%, ↑) on MathNet-Solve-Test (6,400 problems).

## Citation

```bibtex
@misc{alshammari2026mathnet,
  title={MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval},
  author={Shaden Alshammari et al.},
  year={2026},
  note={arXiv:2604.18584}
}
```

- arXiv: 2604.18584