Downstream Scaling Eval

---
name: downstream-scaling-eval
description: This evaluation probes how reliably language model scaling laws predict performance in over-trained regimes, where models are trained with significantly more tokens than parameters. It measures both next-token prediction accuracy on a held-out corpus and generalization across a broad suite of downstream zero-shot and few-shot tasks. Use when the user wants to benchmark on C4 eval, LLM-foundry, or asks about evaluating this task. Reports Validation loss.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.08540
  bibtex_key: gadre2024scaling
  confidence: high
---

# downstream-scaling-eval

> Language models scale reliably with over-training and on downstream tasks — Gadre et al. (2024) (arXiv:2403.08540, 2024)

## What this evaluates

This evaluation probes how reliably language model scaling laws predict performance in over-trained regimes, where models are trained with significantly more tokens than parameters. It measures both next-token prediction accuracy on a held-out corpus and generalization across a broad suite of downstream zero-shot and few-shot tasks.

## Datasets

- **C4 eval** — total ?; splits: eval (-1)
- **LLM-foundry** — total ?; splits: eval (-1)

## Metrics

- `Validation loss` **(primary)** — range: other
  - Cross-entropy between the model's output distribution and the one-hot ground truth, averaged over all tokens in a sequence and across all sequences in the dataset.
- `Average top-1 error` — range: percent
  - Uniform average of the top-1 error (1 - accuracy) across 46 downstream tasks from LLM-foundry.
- `Relative prediction error` — range: other
  - |ζ(C,M) - ζ_GT| / ζ_GT, where ζ represents either the validation loss L or the average top-1 error Err.

## Input / output format

**Input**: Text sequences for language modeling; prompts containing task instructions and optional few-shot examples for downstream evaluations.

**Output**: Next-token probability distributions for language modeling; discrete class labels or text completions for downstream tasks.

## Scoring recipe

```python
def compute_metrics(predictions, golds, metric_type):
    if metric_type == "validation_loss":
        total = sum(cross_entropy(p, g) for p, g in zip(predictions, golds))
        return total / len(golds)
    elif metric_type == "average_top_1_error":
        errors = []
        for task_preds, task_golds in zip(predictions, golds):
            acc = sum(1 for p, g in zip(task_preds, task_golds) if p == g) / len(task_golds)
            errors.append(1.0 - acc)
        return mean(errors)
    elif metric_type == "relative_prediction_error":
        pred_val, gt_val = predictions, golds
        return abs(pred_val - gt_val) / gt_val
```

## Common pitfalls

- Models trained between 5.2×10^16 and 5.2×10^17 FLOPs are explicitly excluded from scaling law fitting because they over-perform due to receiving more optimization steps than neighboring configurations.
- Hyperparameters are tuned exclusively on the OpenLM eval validation set; no tuning is performed on downstream task validation sets or for token multipliers other than M=20.
- Runs that would require more tokens than available in a dataset (e.g., 0.411B model at M=640 on C4) are omitted, creating uneven compute coverage across datasets.

## Evidence (verbatim from paper)

> We consider three main metrics: (i) Validation loss, which is the cross entropy between a model’s output and the one-hot ground truth, averaged over all tokens in a sequence and over all sequences in a dataset. (ii) Average top-1 error, which is a uniform average over 46 downstream evaluations sourced from LLM-foundry. We also look at the mean top-1 error for the subset of 17 evaluations identified in the paragraph above. For a complete list of downstream evaluation datasets, see Appendix[D]. To measure how good a prediction ζ(C,M) is, we measure (iii) Relative prediction error: |ζ(C,M)-ζ_GT|/ζ_GT, where ζ is the loss L or the average top-1 error Err.

## Citation

```bibtex
@misc{gadre2024scaling,
  title={Language models scale reliably with over-training and on downstream tasks},
  author={Gadre et al. (2024)},
  year={2024},
  note={arXiv:2403.08540}
}
```

- arXiv: 2403.08540