Flan 2022 Eval

---
name: flan-2022-eval
description: Evaluates the instruction-tuning effectiveness of models trained on the Flan 2022 collection across held-in, chain-of-thought, and held-out benchmarks. It probes zero-shot and few-shot generalization capabilities on reasoning, knowledge, and natural language understanding tasks. Use when the user wants to benchmark on MMLU, BBH, or asks about evaluating this task. Reports zero-shot/few-shot accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.13688
  bibtex_key: longpre2023flancollection
  confidence: high
---

# flan-2022-eval

> The Flan Collection: Designing Data and Methods for Effective Instruction Tuning — Longpre et al. (2023) (arXiv:2301.13688, 2023)

## What this evaluates

Evaluates the instruction-tuning effectiveness of models trained on the Flan 2022 collection across held-in, chain-of-thought, and held-out benchmarks. It probes zero-shot and few-shot generalization capabilities on reasoning, knowledge, and natural language understanding tasks.

## Datasets

- **MMLU** — total 57; splits: test (57)
- **BBH** — total 23; splits: test (23)

## Metrics

- `zero-shot/few-shot accuracy` **(primary)** — range: percent
  - Standard exact-match accuracy computed over the validation/test sets for each benchmark. Reported separately for zero-shot and few-shot prompting settings.

## Input / output format

**Input**: Instruction prompts, optionally augmented with few-shot examples or Chain-of-Thought reasoning traces. For held-out tasks, inputs are provided in zero-shot or few-shot formats.

**Output**: Model-generated text containing the predicted answer or reasoning trace.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Evaluating on held-in tasks without distinguishing between zero-shot and few-shot settings, as performance varies significantly between the two.
- Assuming task scaling beyond 1800 tasks yields linear improvements without considering task diversity and mixture weighting, which can cause performance saturation.

## Evidence (verbatim from paper)

> We evaluate on (a) a suite of 8 "Held-In" tasks represented within the 1800+ training task collection (4 question answering and 4 natural language inference validation sets), (b) Chain-of-Thought (CoT) tasks (5 validation sets), and (c) the MMLU (Hendrycks et al., 2020) and BBH (Suzgun et al., 2022) benchmarks as our set of "Held-Out" tasks, as they are not included as part of Flan 2022 finetuning. Metrics are reported in both zero-shot / few-shot settings across Held-In, Chain-of-Thought, and Held-Out (MMLU, BBH) tasks.

## Citation

```bibtex
@misc{longpre2023flancollection,
  title={The Flan Collection: Designing Data and Methods for Effective Instruction Tuning},
  author={Longpre et al. (2023)},
  year={2023},
  note={arXiv:2301.13688}
}
```

- arXiv: 2301.13688