Ultra Fineweb Eval

---
name: ultra-fineweb-eval
description: Evaluates the quality of pre-training datasets by measuring the downstream performance of models trained on them. It probes general language understanding, commonsense reasoning, and multilingual capabilities through standard zero-shot benchmarks. Use when the user wants to benchmark on MMLU, ARC-C, ARC-E, CommonSenseQA, HellaSwag, OpenbookQA, PIQA, SIQA, Winogrande, C-Eval, CMMLU, or asks about evaluating this task. Reports Average.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.05427
  bibtex_key: wang2025ultrafineweb
  confidence: high
---

# ultra-fineweb-eval

> Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data — Wang et al. (2025) (arXiv:2505.05427, 2025)

## What this evaluates

Evaluates the quality of pre-training datasets by measuring the downstream performance of models trained on them. It probes general language understanding, commonsense reasoning, and multilingual capabilities through standard zero-shot benchmarks.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **ARC-C** — total ?; splits: test (-1)
- **ARC-E** — total ?; splits: test (-1)
- **CommonSenseQA** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **OpenbookQA** — total ?; splits: test (-1)
- **PIQA** — total ?; splits: test (-1)
- **SIQA** — total ?; splits: test (-1)
- **Winogrande** — total ?; splits: test (-1)
- **C-Eval** — total ?; splits: test (-1)
- **CMMLU** — total ?; splits: test (-1)

## Metrics

- `AverageEnglish` — range: percent
  - Arithmetic mean of zero-shot accuracy scores across the nine English benchmarks (MMLU, ARC-C, ARC-E, CommonSenseQA, HellaSwag, OpenbookQA, PIQA, SIQA, Winogrande).
- `AverageChinese` — range: percent
  - Arithmetic mean of zero-shot accuracy scores across the two Chinese benchmarks (C-Eval, CMMLU).
- `Average` **(primary)** — range: percent
  - Arithmetic mean of zero-shot accuracy scores across all listed English and Chinese benchmarks.

## Input / output format

**Input**: Zero-shot multiple-choice prompts from standard benchmarks fed to a trained LLM.

**Output**: Model-generated token predictions used to compute exact-match accuracy against provided answer choices.

## Scoring recipe

```python
all_scores = []
for benchmark in [MMLU, ARC-C, ARC-E, CommonSenseQA, HellaSwag, OpenbookQA, PIQA, SIQA, Winogrande, C-Eval, CMMLU]:
    acc = compute_zero_shot_accuracy(model, benchmark.test_set)
    all_scores.append(acc)
average = sum(all_scores) / len(all_scores)
return average
```

## Common pitfalls

- The evaluation strictly uses a zero-shot setting via the Lighteval library; applying few-shot prompting or instruction tuning during evaluation will invalidate the reported scores.
- Scores are reported as percentage points (pp) averaged across tasks; misinterpreting them as raw accuracy, median scores, or task-specific scores will lead to incorrect comparisons.
- The 'Average' metric combines English and Chinese benchmarks; separating them requires using AverageEnglish and AverageChinese instead.

## Evidence (verbatim from paper)

> The evaluation metrics include:

* •

    AverageEnglish: Average score across standard English metrics including MMLU*(Hendrycks et al., [2020])*, ARC-C*(Clark et al., [2018])*, ARC-E*(Clark et al., [2018])*, CommonSenseQA*(Talmor et al., [2018])*, HellaSwag*(Zellers et al., [2019])*, OpenbookQA*(Mihaylov et al., [2018])*, PIQA*(Bisk et al., [2020])*, SIQA*(Sap et al., [2019])*, and Winogrande*(Sakaguchi et al., [2021])*.

* •

    AverageChinese: Average score of Chinese metrics, including C-Eval*(Huang et al., [2023])* and CMMLU*(Li et al., [2023])*.

* •

    Average: The combined average score of all the above evaluation metrics.

## Citation

```bibtex
@misc{wang2025ultrafineweb,
  title={Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.05427}
}
```

- arXiv: 2505.05427