Downstream Eval

---
name: downstream-eval
description: Evaluates the downstream language and symbolic capabilities of LLMs pretrained on filtered web corpora. It probes general knowledge, reasoning, comprehension, and code/math problem-solving to assess how different data filtering strategies impact model performance. Use when the user wants to benchmark on Dolma (v1.6), Pile-github, or asks about evaluating this task. Reports average normalized accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.18577
  bibtex_key: [REDACTED]
  confidence: high
---

# downstream-eval

> Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity — Yeongbin Seo et al. (2025) (arXiv:2509.18577, 2025)

## What this evaluates

Evaluates the downstream language and symbolic capabilities of LLMs pretrained on filtered web corpora. It probes general knowledge, reasoning, comprehension, and code/math problem-solving to assess how different data filtering strategies impact model performance.

## Datasets

- **Dolma (v1.6)** — total 6300000000; splits: train (3000000000), test (-1)
- **Pile-github** — total 6000000000; splits: train (3000000000), test (-1); repo https://www.kaggle.com/datasets/dschettler8845/the-pile-github-files-part-01

## Metrics

- `average normalized accuracy` **(primary)** — range: [0, 1] or percent
  - Normalized per-task accuracy is computed as $a_{n}=(a_{m}-a_{r})/(1-a_{r})$, where $a_{m}$ is model accuracy and $a_{r}$ is random guessing accuracy. The final score is the arithmetic mean of $a_{n}$ across all evaluated benchmarks.

## Input / output format

**Input**: Filtered pretraining corpus blocks (512 tokens) for model training; downstream benchmark prompts for evaluation.

**Output**: Model-generated predictions or answers for each downstream benchmark task.

## Scoring recipe

```python
def normalized_accuracy(model_acc, random_acc):
    return (model_acc - random_acc) / (1 - random_acc)

def compute_metric(predictions, gold, random_acc):
    task_acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    return normalized_accuracy(task_acc, random_acc)

# Final score: mean of task normalized accuracies across all benchmarks
```

## Common pitfalls

- Excludes downstream benchmarks where baseline normalized accuracy < 0.001 for 1.5B models, reducing the total count from the full gauntlet.
- DSIR baseline is excluded for Pile-github experiments due to the difficulty of selecting an appropriate reference dataset.
- Normalization requires knowing the exact random guessing accuracy ($a_r$) per task, which varies by benchmark format.

## Evidence (verbatim from paper)

> We normalize the accuracy of the individual task as $a_{n}=rac{a_{m}-a_{r}}{1-a_{r}}$, where $a_{m}$ is the accuracy of the model and $a_{r}$ is the expected accuracy of random guessing. We report the average normalized accuracy for each task, task category, and the average across all categories.

## Citation

```bibtex
@misc{seo2025priorbasednoisy,
  title={Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity},
  author={Yeongbin Seo et al. (2025)},
  year={2025},
  note={arXiv:2509.18577}
}
```

- arXiv: 2509.18577