Mathnet Retrieve Eval

---
name: mathnet-retrieve-eval
description: Probes a model's ability to retrieve mathematically equivalent problems from a large corpus using embeddings. It measures whether retrieval systems can recognize structural and symbolic invariance rather than relying on superficial lexical overlap. Use when the user wants to benchmark on MathNet-Retrieve, or asks about evaluating this task. Reports Recall@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.18584
  bibtex_key: alshammari2026mathnet
  confidence: high
---

# mathnet-retrieve-eval

> MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval — Shaden Alshammari et al. (arXiv:2604.18584, 2026)

## What this evaluates

Probes a model's ability to retrieve mathematically equivalent problems from a large corpus using embeddings. It measures whether retrieval systems can recognize structural and symbolic invariance rather than relying on superficial lexical overlap.

## Datasets

- **MathNet-Retrieve** — total 10000; splits: test (10000); repo https://github.com/ShadeAlsha/MathNet

## Metrics

- `Recall@k` **(primary)** — range: percent
  - Measures whether any of the top-k retrieved problems correspond to a “correct” match from the equivalent versions of each anchor problem. Reported for k=1 and k=5.

## Input / output format

**Input**: Embedding of the anchor problem statement.

**Output**: Ranked list of retrieved problem statements.

## Scoring recipe

```python
retrieved = retrieve_top_k(anchor_embedding, corpus, k=k)
is_hit = any(retrieved_problem in correct_equivalent_set for retrieved_problem in retrieved)
recall_at_k = sum(is_hit) / total_anchors
```

## Common pitfalls

- Embeddings often match on superficial keywords (e.g., 'triangle') rather than mathematical structure, leading to low Recall@1 despite high Recall@10.
- Uses cosine similarity over embeddings, which may not capture symbolic invariance.
- Hard negatives (near misses) are explicitly constructed to test fine-grained distinctions, but models struggle to separate them.

## Evidence (verbatim from paper)

> The primary evaluation metric for our retrieval task is Recall@k, which measures whether any of the top-k retrieved problems correspond to a “correct” match from our equivalent versions of each problem. We report Recall@1 and Recall@5.

## Citation

```bibtex
@misc{alshammari2026mathnet,
  title={MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval},
  author={Shaden Alshammari et al.},
  year={2026},
  note={arXiv:2604.18584}
}
```

- arXiv: 2604.18584