Mmlu Bbh Ifeval Eval

---
name: mmlu-bbh-ifeval-eval
description: Evaluates instruction-tuned language models on factual knowledge, complex reasoning, and instruction-following capabilities. The protocol measures how different data selection methods and model sizes impact performance under strict compute budgets. Use when the user wants to benchmark on MMLU, BBH, IFEval, or asks about evaluating this task. Reports 5-shot accuracy, 3-shot exact match score, 0-shot accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.16208
  bibtex_key: yin2024computeconstrained
  confidence: high
---

# mmlu-bbh-ifeval-eval

> Compute-Constrained Data Selection — Yin et al. (2024) (arXiv:2410.16208, 2024)

## What this evaluates

Evaluates instruction-tuned language models on factual knowledge, complex reasoning, and instruction-following capabilities. The protocol measures how different data selection methods and model sizes impact performance under strict compute budgets.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **BBH** — total ?; splits: test (-1)
- **IFEval** — total ?; splits: test (-1)

## Metrics

- `5-shot accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered multiple-choice questions across 57 subjects using 5 in-context examples.
- `3-shot exact match score` **(primary)** — range: [0, 1]
  - Percentage of tasks where the model's generated answer exactly matches the ground truth using 3 in-context examples.
- `0-shot accuracy` **(primary)** — range: [0, 1]
  - Percentage of instruction-following prompts where the model correctly follows the specified constraints without in-context examples.

## Input / output format

**Input**: Instruction-tuned finetuning data (curated from COT, Flan V2, Dolly, Open Assistant 1). Evaluation inputs are benchmark prompts formatted with 5-shot, 3-shot, or 0-shot examples depending on the task.

**Output**: Model-generated text responses to benchmark prompts, evaluated for exact match or multiple-choice correctness.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return correct / len(golds)
```

## Common pitfalls

- Compute budget trade-offs between model size and data selection method are often misinterpreted as pure data quality effects.
- Pareto frontiers are fitted using power laws in linear-log space, which can obscure non-monotonic scaling behaviors.
- Few-shot evaluation settings (5-shot, 3-shot, 0-shot) must be strictly adhered to, as performance varies significantly with prompt length and in-context examples.

## Evidence (verbatim from paper)

> For MMLU, we report 5-shot accuracy; for BBH, we report 3-shot exact match score; and for IFEval, we report 0-shot accuracy.

## Citation

```bibtex
@misc{yin2024computeconstrained,
  title={Compute-Constrained Data Selection},
  author={Yin et al. (2024)},
  year={2024},
  note={arXiv:2410.16208}
}
```

- arXiv: 2410.16208