Opt Iml Bench Eval

---
name: opt-iml-bench-eval
description: Evaluates instruction-tuned language models on generalization across 1,991 NLP tasks spanning 100+ categories. It probes zero-shot and few-shot (5-shot) performance on held-out categories, unseen tasks within seen categories, and fully supervised tasks, measuring both generation quality and classification accuracy. Use when the user wants to benchmark on OPT-IML Bench, or asks about evaluating this task. Reports Rouge-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.12017
  bibtex_key: iyer2022optiml
  confidence: high
---

# opt-iml-bench-eval

> OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization — Iyer et al. (2022) (arXiv:2212.12017, 2022)

## What this evaluates

Evaluates instruction-tuned language models on generalization across 1,991 NLP tasks spanning 100+ categories. It probes zero-shot and few-shot (5-shot) performance on held-out categories, unseen tasks within seen categories, and fully supervised tasks, measuring both generation quality and classification accuracy.

## Datasets

- **OPT-IML Bench** — total 1991; splits: test (-1)

## Metrics

- `Rouge-L` **(primary)** — range: [0, 100] | percent
  - Longest common subsequence overlap between the model's generated text and the reference answer, reported as a percentage.
- `Accuracy` — range: [0, 100] | percent
  - Percentage of predictions that exactly match the gold label or answer.

## Input / output format

**Input**: Instruction prompt with 0 or 5 demonstration examples (task-specific format).

**Output**: Free-form text generation or classification label.

## Scoring recipe

```python
def compute_metrics(predictions, golds, metric_type):
    if metric_type == 'accuracy':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds) * 100
    elif metric_type == 'rouge-l':
        scores = [rouge_l_score(p, g) * 100 for p, g in zip(predictions, golds)]
        return sum(scores) / len(scores)
    return None
```

## Common pitfalls

- Results are reported in 0-shot/5-shot format; summarization tasks only report 0-shot performance.
- Averages across task categories mix Rouge-L and accuracy scores, which are on different scales and should not be directly compared without normalization.
- Some evaluation tables use 32-shot instead of 5-shot (e.g., Table 16), so shot counts must be verified per table.

## Evidence (verbatim from paper)

> Most tasks are generation tasks, for which we report Rouge-L. We report accuracy for MMLU. Some tasks in the Cause Effect Cluster also use accuracy, which is averaged with Rouge-L for presentation purposes.

## Citation

```bibtex
@misc{iyer2022optiml,
  title={OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization},
  author={Iyer et al. (2022)},
  year={2022},
  note={arXiv:2212.12017}
}
```

- arXiv: 2212.12017