Held Out Test Loss

---
name: held-out-test-loss
description: Evaluates language model generalization and overfitting by measuring cross-entropy loss on a held-out test set. It probes how well the model retains predictive performance when trained on repeated or constrained data subsets. Use when the user has predictions and gold and needs to compute held-out test loss.
metadata:
  skill_kind: metric
  source_arxiv: 2305.16264
  bibtex_key: muennighoff2023scalingdata
  confidence: high
---

# held-out-test-loss

> Scaling Data-Constrained Language Models — Muennighoff et al. (2023) (arXiv:2305.16264, 2023)

## What this evaluates

Evaluates language model generalization and overfitting by measuring cross-entropy loss on a held-out test set. It probes how well the model retains predictive performance when trained on repeated or constrained data subsets.

## Datasets

- **C4** — total ?; splits: test (-1)

## Metrics

- `held-out test loss` **(primary)** — range: other
  - Average cross-entropy loss computed over a held-out test set drawn from the same underlying dataset as the training data. Lower values indicate better generalization.

## Input / output format

**Input**: Tokenized text sequences from the C4 dataset.

**Output**: Predicted probability distribution over the next token.

## Scoring recipe

```python
loss = 0.0
for tokens in test_set:
    logits = model(tokens)
    loss += cross_entropy(logits, tokens)
return loss / len(test_set)
```

## Common pitfalls

- Using training loss instead of held-out test loss can mask severe overfitting when data is repeated across epochs.
- Failing to shuffle data between epochs or ensuring maximal overlap between training subsets can bias the evaluation of data-constrained scaling.

## Evidence (verbatim from paper)

> we report loss on a held-out test set unless otherwise specified (see Appendix K). This contrasts training loss used in [42], but should not alter our findings as the held-out data stems from the same underlying dataset.

## Citation

```bibtex
@misc{muennighoff2023scalingdata,
  title={Scaling Data-Constrained Language Models},
  author={Muennighoff et al. (2023)},
  year={2023},
  note={arXiv:2305.16264}
}
```

- arXiv: 2305.16264