Mamut Mir Eval

---
name: mamut-mir-eval
description: Evaluates mathematical information retrieval capabilities by testing whether models can match natural language names or LaTeX formulas to their corresponding mathematical identities from a candidate pool. It probes the model's ability to learn structural and notational variations in mathematical expressions through pretraining and fine-tuning. Use when the user wants to benchmark on MAMUT-generated datasets (MF, MT, NMF, MFR), or asks about evaluating this task. Reports nDCG.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.20855
  bibtex_key: drechsel2025mamut
  confidence: high
---

# mamut-mir-eval

> MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training — Drechsel et al. (2025) (arXiv:2502.20855, 2025)

## What this evaluates

Evaluates mathematical information retrieval capabilities by testing whether models can match natural language names or LaTeX formulas to their corresponding mathematical identities from a candidate pool. It probes the model's ability to learn structural and notational variations in mathematical expressions through pretraining and fine-tuning.

## Datasets

- **MAMUT-generated datasets (MF, MT, NMF, MFR)** — total ?; splits: train (-1), test (-1); repo https://github.com/aieng-lab/math-mutator

## Metrics

- `nDCG` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank k, measuring ranking quality by assigning higher scores to relevant items appearing higher in the list, normalized by the ideal DCG.
- `p@k` — range: [0, 1]
  - Precision at rank k, measuring the fraction of relevant items in the top-k retrieved results.
- `AP` — range: [0, 1]
  - Average Precision, the area under the precision-recall curve, averaging precision at each rank where a relevant item is retrieved.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall for binary classification of query-relevance.

## Input / output format

**Input**: A query consisting of either a natural language name or a LaTeX formula, paired with a candidate set of formulas to retrieve from.

**Output**: A ranked list of candidate formulas or binary relevance scores for each candidate.

## Scoring recipe

```python
import math
def compute_metrics(predictions, gold):
    k = 10
    p_at_k = len(gold & set(predictions[:k])) / k
    dcg = sum(1 / math.log2(i+2) for i, c in enumerate(predictions) if c in gold)
    idcg = sum(1 / math.log2(i+2) for i in range(min(len(gold), k)))
    ndcg = dcg / idcg if idcg > 0 else 0
    ap = sum(len(gold & set(predictions[:i+1])) / (i+1) for i, c in enumerate(predictions) if c in gold) / len(gold)
    tp = len(gold & set(predictions))
    p = tp / len(predictions) if predictions else 0
    r = tp / len(gold) if gold else 0
    f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
    return {'p@k': p_at_k, 'ndcg': ndcg, 'ap': ap, 'f1': f1}
```

## Common pitfalls

- Models pretrained on NMF/MFR during pretraining are excluded from the main comparison to avoid data leakage; only MLM-pretrained models (MF/MT) are fine-tuned on NMF/MFR for fair evaluation.
- Metrics are averaged over five independent fine-tuning runs to ensure robustness, not just a single split or run.

## Evidence (verbatim from paper)

> We evaluate using both binary classification metrics (precision (P), recall (R), F1) and standard IR ranking metrics: Precisión at  $k$  ( $p@k$ ), Average Precision (AP), and nDCG, that are averaged over all test queries, with higher values indicating better performance (Manning, 2009; Radlinski & Craswell, 2010).

## Citation

```bibtex
@misc{drechsel2025mamut,
  title={MAMUT: A Novel Framework for Modifying Mathematical Formulas for the Generation of Specialized Datasets for Language Model Training},
  author={Drechsel et al. (2025)},
  year={2025},
  note={arXiv:2502.20855}
}
```

- arXiv: 2502.20855