Big Bench Predictability Eval

---
name: big-bench-predictability-eval
description: Evaluates the predictability of LLM performance across diverse BIG-bench tasks using an MLP-based predictor. It probes how well model scale, task type, and in-context examples correlate with actual benchmark scores, and tests robustness under different holdout strategies. Use when the user wants to benchmark on BIG-bench, or asks about evaluating this task. Reports R².
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.14947
  bibtex_key: [REDACTED]
  confidence: medium
---

# big-bench-predictability-eval

> How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench — Ye et al. (2023) (arXiv:2305.14947, 2023)

## What this evaluates

Evaluates the predictability of LLM performance across diverse BIG-bench tasks using an MLP-based predictor. It probes how well model scale, task type, and in-context examples correlate with actual benchmark scores, and tests robustness under different holdout strategies.

## Datasets

- **BIG-bench** — total ?; splits: test (-1); repo https://github.com/google/BIG-bench

## Metrics

- `R²` **(primary)** — range: [0, 1]
  - Coefficient of determination measuring the proportion of variance in actual BIG-bench scores explained by the MLP predictor. Calculated as 1 - (SS_res / SS_tot).
- `RMSE` — range: [0, 1]
  - Root Mean Squared Error between predicted and actual task performance scores.

## Input / output format

**Input**: Task identifier, model scale/architecture parameters, and in-context example configurations.

**Output**: Predicted performance score (e.g., accuracy or normalized score) for each BIG-bench task.

## Scoring recipe

```python
def compute_metrics(predictions, actuals):
    mean_actual = sum(actuals) / len(actuals)
    ss_res = sum((p - a)**2 for p, a in zip(predictions, actuals))
    ss_tot = sum((a - mean_actual)**2 for a in actuals)
    r2 = 1 - (ss_res / ss_tot)
    rmse = (ss_res / len(actuals))**0.5
    return r2, rmse
```

## Common pitfalls

- Predictability drops significantly under Cartesian product holdout, indicating high sensitivity to distribution assumptions.
- Emergent tasks are harder to predict but remain partially predictable when related tasks are included in the training set.

## Evidence (verbatim from paper)

> Large language model (LLM) performance on BIG-bench is highly predictable, with an MLP-based predictor achieving >95% R² and <0.05 RMSE, revealing learnable patterns across model scale, task, and in-context examples.

## Citation

```bibtex
@misc{ye2023predictable,
  title={How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench},
  author={Ye et al. (2023)},
  year={2023},
  note={arXiv:2305.14947}
}
```

- arXiv: 2305.14947