Zero Shot Generalization Eval

---
name: zero-shot-generalization-eval
description: Evaluates a model's ability to generalize to unseen natural language tasks without task-specific fine-tuning or prompt tuning. It probes zero-shot performance across traditional NLP benchmarks and novel BIG-bench tasks using accuracy. Use when the user wants to benchmark on BIG-bench & Held-out NLP Tasks, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.08207
  bibtex_key: sanh2021tzero
  confidence: high
---

# zero-shot-generalization-eval

> Multitask Prompted Training Enables Zero-Shot Task Generalization — Victor Sanh et al. (2021) (arXiv:2110.08207, 2021)

## What this evaluates

Evaluates a model's ability to generalize to unseen natural language tasks without task-specific fine-tuning or prompt tuning. It probes zero-shot performance across traditional NLP benchmarks and novel BIG-bench tasks using accuracy.

## Datasets

- **BIG-bench & Held-out NLP Tasks** — total ?; splits: val (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted options among all test instances. For multiple-choice tasks, predictions are made via rank classification by selecting the option with the highest unnormalized log-likelihood.

## Input / output format

**Input**: Input text (prompt + question/options) fed to an encoder, truncated to 1024 tokens.

**Output**: Target text (correct answer or option) generated autoregressively by the decoder, truncated to 256 tokens.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
# For multiple-choice: prediction = argmax(log_prob(option) for option in options)
# Final report: median accuracy across all prompt templates per dataset.
```

## Common pitfalls

- Selecting the best prompt based on validation performance leaks evaluation data and violates the true zero-shot setting.
- Log-likelihoods for multiple-choice options are not length-normalized, which can bias predictions toward shorter answers.

## Evidence (verbatim from paper)

> We evaluate zero-shot generalization on 11 datasets in 4 held-out traditional NLP tasks: natural language inference, coreference, word sense disambiguation, and sentence completion, as well as 14 novel tasks from BIG-bench (§3). Unless specified otherwise, we report performance on the validation splits. All reported datasets use accuracy as their metric.

## Citation

```bibtex
@misc{sanh2021tzero,
  title={Multitask Prompted Training Enables Zero-Shot Task Generalization},
  author={Victor Sanh et al. (2021)},
  year={2021},
  note={arXiv:2110.08207}
}
```

- arXiv: 2110.08207