Meta Rater Eval

---
name: meta-rater-eval
description: Evaluates the downstream performance of language models pre-trained on data selected by various quality-based methods compared to random sampling. It probes how different data curation strategies impact general knowledge, commonsense reasoning, and reading comprehension capabilities. Use when the user wants to benchmark on ARC-Challenge, ARC-Easy, SciQ, HellaSwag, SIQA, WinoGrande, RACE, OpenbookQA, or asks about evaluating this task. Reports average accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.14194
  bibtex_key: zhuang2025metarater
  confidence: high
---

# meta-rater-eval

> Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models — Zhuang et al. (2025) (arXiv:2504.14194, 2025)

## What this evaluates

Evaluates the downstream performance of language models pre-trained on data selected by various quality-based methods compared to random sampling. It probes how different data curation strategies impact general knowledge, commonsense reasoning, and reading comprehension capabilities.

## Datasets

- **ARC-Challenge, ARC-Easy, SciQ, HellaSwag, SIQA, WinoGrande, RACE, OpenbookQA** — total ?; splits: test (-1)

## Metrics

- `average accuracy` **(primary)** — range: percent
  - Arithmetic mean of per-task accuracy scores across eight downstream benchmarks. Computed as the sum of individual task accuracies divided by the number of tasks.

## Input / output format

**Input**: Downstream task prompts formatted for in-context learning via lm-evaluation-harness.

**Output**: Model-generated predictions (multiple-choice answers or text completions) for each prompt.

## Scoring recipe

```python
accuracies = []
for task in [ARC_Challenge, ARC_Easy, SciQ, HellaSwag, SIQA, WinoGrande, RACE, OpenbookQA]:
    preds = model.generate(task.prompts, in_context=True)
    acc = compute_accuracy(preds, task.golds)
    accuracies.append(acc)
return sum(accuracies) / len(accuracies)
```

## Common pitfalls

- The evaluation uses in-context learning via lm-evaluation-harness rather than zero-shot or fine-tuning, which may inflate or deflate scores differently than other protocols.
- The 'Average' metric is a simple arithmetic mean across tasks of varying difficulty and scale, which can mask significant performance drops in specific categories like Reading Comprehension.
- Data selection methods sample fixed token counts (e.g., 30B) while maintaining domain proportions, meaning the actual number of documents and domain mix varies across methods.

## Evidence (verbatim from paper)

> To comprehensively assess the capabilities of pre-trained models, we conduct holistic evaluations on various downstream tasks covering three significant categories: General Knowledge (including ARC-Challenge *Clark et al. ([2018])*, ARC-Easy, and SciQ *Welbl et al. ([2017])*), Commonsense Reasoning (including HellaSwag *Zellers et al. ([2019])*, SIQA *Sap et al. ([2019])*, and WinoGrande *Sakaguchi et al. ([2020])*), and Reading Comprehension (including RACE *Lai et al. ([2017])* and OpenbookQA *Mihaylov et al. ([2018])*). Evaluations are conducted using the lm-evaluation-harness *Gao et al. ([2023])*) framework with in-context learning setting, and average accuracy is reported for convenient comparison.

## Citation

```bibtex
@misc{zhuang2025metarater,
  title={Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models},
  author={Zhuang et al. (2025)},
  year={2025},
  note={arXiv:2504.14194}
}
```

- arXiv: 2504.14194