Nemotron Math Eval

---
name: nemotron-math-eval
description: Evaluates long-context mathematical reasoning and tool-integrated reasoning capabilities of language models on competition-style and open-domain advanced math problems. It probes symbolic precision, multi-step deduction, and the ability to leverage Python code execution for verification. Use when the user wants to benchmark on Comp-Math-24-25, HLE-Math, or asks about evaluating this task. Reports maj@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.15489
  bibtex_key: du2025nemotronmath
  confidence: high
---

# nemotron-math-eval

> Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision — Du et al. (2025) (arXiv:2512.15489, 2025)

## What this evaluates

Evaluates long-context mathematical reasoning and tool-integrated reasoning capabilities of language models on competition-style and open-domain advanced math problems. It probes symbolic precision, multi-step deduction, and the ability to leverage Python code execution for verification.

## Datasets

- **Comp-Math-24-25** — total ?; splits: test (-1)
- **HLE-Math** — total 976; splits: test (976)

## Metrics

- `pass@1` — range: [0, 1]
  - Measures the average accuracy across independent single-run generations per problem.
- `maj@k` **(primary)** — range: [0, 1]
  - Computes accuracy under majority voting over k generated solutions per problem. k=16 for AIME24/AIME25/HMMT-24-25 and k=4 for HLE-Math.

## Input / output format

**Input**: Text-based mathematical problem statements (competition-style or open-domain), optionally with Python tool-integrated reasoning (TIR) enabled.

**Output**: Step-by-step reasoning trace and final answer. In TIR settings, includes executable Python code blocks.

## Scoring recipe

```python
def compute_metrics(predictions, gold_answers, k):
    maj_correct = []
    for preds, gold in zip(predictions, gold_answers):
        from collections import Counter
        vote = Counter(preds).most_common(1)[0][0]
        maj_correct.append(1 if vote == gold else 0)
    maj_at_k = sum(maj_correct) / len(maj_correct)
    pass_at_1 = sum(1 if p == gold else 0 for p, g in zip(predictions, gold_answers)) / len(predictions)
    return {'pass@1': pass_at_1, 'maj@k': maj_at_k}
```

## Common pitfalls

- Using math-verify for HLE-Math instead of the specified LLM-as-a-judge protocol (Qwen2.5-32B-Instruct).
- Applying different decoding hyperparameters (temperature, top-p, max tokens) than the specified 1.0/1.0/120K, which invalidates maj@k comparisons.
- Confusing pass@1 (single-sample accuracy) with maj@k (majority voting over k samples), leading to incorrect reporting of model robustness.

## Evidence (verbatim from paper)

> We report both pass@1 and maj@k, where pass@1 measures the average accuracy across different independent runs and maj@k computes accuracy under majority voting over $k$ generated solutions (with $k\=16$ for AIME24, AIME25, HMMT-24-25 and $k\=4$ for HLE-Math).

## Citation

```bibtex
@misc{du2025nemotronmath,
  title={Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervision},
  author={Du et al. (2025)},
  year={2025},
  note={arXiv:2512.15489}
}
```

- arXiv: 2512.15489