Mathnet Rag Eval

---
name: mathnet-rag-eval
description: Evaluates how retrieval quality impacts downstream mathematical problem solving. It compares zero-shot performance against retrieval-augmented settings using either embedding-retrieved or expert-paired problems with their solutions. Use when the user wants to benchmark on MathNet-RAG, or asks about evaluating this task. Reports Retrieval-Augmented Problem Solving Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18584
  bibtex_key: [REDACTED]
  confidence: high
---

# mathnet-rag-eval

> MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval — Shaden Alshammari et al. (arXiv:2604.18584, 2026)

## What this evaluates

Evaluates how retrieval quality impacts downstream mathematical problem solving. It compares zero-shot performance against retrieval-augmented settings using either embedding-retrieved or expert-paired problems with their solutions.

## Datasets

- **MathNet-RAG** — total 35; splits: test (35); repo https://github.com/ShadeAlsha/MathNet

## Metrics

- `Retrieval-Augmented Problem Solving Accuracy` **(primary)** — range: percent
  - Percentage of correctly solved problems under three settings: Zero Shot, Embed-RAG (retrieves 1 problem via gemini-embedding-001), and Expert-RAG (provides expert-paired problem). Graded by humans or averaged LLM judge.

## Input / output format

**Input**: Target problem statement. In RAG settings, the retrieved problem and its official solution are appended as context.

**Output**: A generated solution to the target problem.

## Scoring recipe

```python
context = get_context(setting) # retrieved problem + solution or None
model_output = generate(target_problem, context)
is_correct = human_or_llm_judge(target_problem, reference_solution, model_output)
accuracy = sum(is_correct) / total_problems
```

## Common pitfalls

- Small test set (35 problems) leads to high standard errors in reported accuracy.
- Grading is done by humans or averaged LLM judges, introducing potential subjectivity or judge-model bias.
- Embed-RAG can sometimes hurt performance if the retriever returns near-miss problems that add noise.

## Evidence (verbatim from paper)

> To assess the impact of retrieval on downstream problem solving, we evaluate three settings. In Zero Shot, the model receives only the target problem. In Embed-RAG, we retrieve one related problem using gemini-embedding-001, then provide the retrieved problem and its official solution as additional context. In Expert-RAG, we replace the retrieved example with the expert-paired related problem from MathNet-RAG, again together with its official solution. Table 5: Retrieval-Augmented Problem Solving Accuracy (%, ↑) on MathNet-RAG (35 problems).

## Citation

```bibtex
@misc{alshammari2026mathnet,
  title={MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval},
  author={Shaden Alshammari et al.},
  year={2026},
  note={arXiv:2604.18584}
}
```

- arXiv: 2604.18584