Tinybenchmarks Sampling Eval

---
name: tinybenchmarks-sampling-eval
description: Evaluates the efficiency and accuracy of LLM benchmarking by testing how well a small, strategically selected subset of examples predicts overall model performance on standard evaluation scenarios. Use when the user wants to benchmark on HELM, MMLU, AlpacaEval 2.0, Open LLM Leaderboard, or asks about evaluating this task. Reports estimation error.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.14992
  bibtex_key: polo2024tinybenchmarks
  confidence: medium
---

# tinybenchmarks-sampling-eval

> tinyBenchmarks: evaluating LLMs with fewer examples — Felipe Maia Polo et al. (arXiv:2402.14992, 2024)

## What this evaluates

Evaluates the efficiency and accuracy of LLM benchmarking by testing how well a small, strategically selected subset of examples predicts overall model performance on standard evaluation scenarios.

## Datasets

- **HELM** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **AlpacaEval 2.0** — total ?; splits: test (-1)
- **Open LLM Leaderboard** — total ?; splits: test (-1)

## Metrics

- `estimation error` **(primary)** — range: percent
  - The absolute difference between the performance estimated from a curated subset of examples and the true performance on the full benchmark scenario. Calculated as |estimated_perf - true_perf|, typically reported as a percentage.

## Input / output format

**Input**: A curated subset of evaluation examples $\widehat{\mathcal{I}}_{j}$ from a benchmark scenario $\mathcal{I}_{j}$, selected via stratified random sampling, K-Means clustering on correctness vectors, or IRT-based clustering.

**Output**: Estimated performance scores for evaluated LLMs on the full benchmark, derived using weighted aggregation of subset correctness or IRT latent ability models.

## Scoring recipe

```python
def scoring_recipe(predictions, gold):
    # predictions: estimated performance from subset
    # gold: true performance on full benchmark
    return abs(predictions - gold)

def estimate_performance(subset_correctness, weights):
    return sum(w * y for w, y in zip(weights, subset_correctness))
```

## Common pitfalls

- Assuming uniform weights ($w_i = 1/|\widehat{\mathcal{I}}_j|$) are sufficient; cluster/IRT weights based on representative fractions yield better estimates.
- Ignoring distribution shifts in model correctness patterns over time, which can degrade clustering-based anchor point selection.
- Failing to account for the curse of dimensionality when using correctness vectors from a large training set for K-Means clustering.

## Evidence (verbatim from paper)

> showing that 100 carefully curated examples suffice to estimate performance on major benchmarks like MMLU, HELM, and Open LLM Leaderboard with <2% error.

## Citation

```bibtex
@misc{polo2024tinybenchmarks,
  title={tinyBenchmarks: evaluating LLMs with fewer examples},
  author={Felipe Maia Polo et al.},
  year={2024},
  note={arXiv:2402.14992}
}
```

- arXiv: 2402.14992