Reasoning Benchmarks Eval

---
name: reasoning-benchmarks-eval
description: Evaluates language model reasoning and general knowledge across multiple standard benchmarks. It measures the impact of data mixture optimization on model performance using accuracy scores on commonsense, scientific, and factual QA tasks. Use when the user wants to benchmark on PIQA, ARC_C, ARC_E, HellaSwag, WinoGrande, SIQA, MMLU, or asks about evaluating this task. Reports test accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.13161
  bibtex_key: diao2025nemotronclimb
  confidence: high
---

# reasoning-benchmarks-eval

> Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training — Diao et al. (2025) (arXiv:2504.13161, 2025)

## What this evaluates

Evaluates language model reasoning and general knowledge across multiple standard benchmarks. It measures the impact of data mixture optimization on model performance using accuracy scores on commonsense, scientific, and factual QA tasks.

## Datasets

- **PIQA** — total ?; splits: test (-1)
- **ARC_C** — total ?; splits: test (-1)
- **ARC_E** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **WinoGrande** — total ?; splits: test (-1)
- **SIQA** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)

## Metrics

- `test accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions on the test set. Calculated as (number of correct predictions / total number of examples) * 100.

## Input / output format

**Input**: Multiple-choice questions or open-ended prompts formatted for the LM-Evaluation Harness, provided in a 0-shot setting (5-shot for MMLU).

**Output**: Model-generated text or selected answer option.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Using validation splits for final reported results instead of test sets.
- Inconsistent shot settings across benchmarks (0-shot default vs 5-shot for MMLU) can skew comparisons if not explicitly noted.
- Averaging accuracy across benchmarks with different scales or question types without weighting.

## Evidence (verbatim from paper)

> For evaluation, we test on reasoning benchmarks: PIQA, ARC_C, ARC_E, HellaSwag, WinoGrande, and SIQA. We optimize using PIQA, ARC_E, and HellaSwag validation data, then evaluate on test sets. LM-Evaluation harness is used, with all datasets in a 0-shot setting except MMLU (5-shot).

## Citation

```bibtex
@misc{diao2025nemotronclimb,
  title={Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training},
  author={Diao et al. (2025)},
  year={2025},
  note={arXiv:2504.13161}
}
```

- arXiv: 2504.13161