Bbh Mmlu Predictability Eval

---
name: bbh-mmlu-predictability-eval
description: This protocol evaluates how well aggregate and per-task benchmark performance can be predicted from scaled compute using scaling law fits. It probes the monotonicity and predictability of LLM capabilities across compute scaling, distinguishing between stable scaling trends and emergent or non-monotonic behaviors. Use when the user wants to benchmark on BIG-Bench Hard (BBH), MMLU, or asks about evaluating this task. Reports mean absolute error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.04757
  bibtex_key: owen2024predictable
  confidence: high
---

# bbh-mmlu-predictability-eval

> How predictable is language model benchmark performance? — Owen et al. (2024) (arXiv:2401.04757, 2024)

## What this evaluates

This protocol evaluates how well aggregate and per-task benchmark performance can be predicted from scaled compute using scaling law fits. It probes the monotonicity and predictability of LLM capabilities across compute scaling, distinguishing between stable scaling trends and emergent or non-monotonic behaviors.

## Datasets

- **BIG-Bench Hard (BBH)** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)

## Metrics

- `mean absolute error` **(primary)** — range: percent
  - The absolute difference between the predicted benchmark accuracy (from a fitted sigmoid curve) and the actual reported accuracy, measured in percentage points. Averaged over held-out data points.

## Input / output format

**Input**: Historical pairs of scaled compute (FLOP) and benchmark accuracy scores for a set of models.

**Output**: Predicted benchmark accuracy (%) for a target compute value, along with the absolute error relative to the true accuracy.

## Scoring recipe

```python
def compute_mean_absolute_error(compute_history, accuracy_history, held_out_compute, held_out_accuracy):
    model = SigmoidFit()
    model.fit(compute_history, accuracy_history)
    predictions = model.predict(held_out_compute)
    errors = [abs(pred - true_acc) for pred, true_acc in zip(predictions, held_out_accuracy)]
    return sum(errors) / len(errors)
```

## Common pitfalls

- Confusing 'scaled compute' (FLOP × dataset size) with raw training FLOP.
- Assuming all tasks follow monotonic scaling; emergent capabilities cause sharp non-monotonic jumps that break sigmoid fits.
- Treating percentage point (pp) error as a probability or ratio rather than an absolute difference in accuracy.

## Evidence (verbatim from paper)

> predictions have average error of 3.9pp across a doubling of compute (0.33 orders of magnitude), rising to 12.5pp at two orders of magnitude scaling. ... Mean absolute error is 17pp, and the median absolute error is 8.4pp.

## Citation

```bibtex
@misc{owen2024predictable,
  title={How predictable is language model benchmark performance?},
  author={Owen et al. (2024)},
  year={2024},
  note={arXiv:2401.04757}
}
```

- arXiv: 2401.04757