Gpt3 Few Shot Eval

---
name: gpt3-few-shot-eval
description: Evaluates the few-shot, one-shot, and zero-shot learning capabilities of large autoregressive language models across diverse NLP tasks including language modeling, cloze completion, question answering, translation, and commonsense reasoning. Use when the user wants to benchmark on Penn Tree Bank (PTB), LAMBADA, HellaSwag, StoryCloze 2016, Natural Questions, WebQuestions, TriviaQA, WMT14/WMT16 Translation, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.14165
  bibtex_key: brown2020gpt3
  confidence: high
---

# gpt3-few-shot-eval

> Language Models are Few-Shot Learners — Tom B. Brown et al. (2020) (arXiv:2005.14165, 2020)

## What this evaluates

Evaluates the few-shot, one-shot, and zero-shot learning capabilities of large autoregressive language models across diverse NLP tasks including language modeling, cloze completion, question answering, translation, and commonsense reasoning.

## Datasets

- **Penn Tree Bank (PTB)** — total ?; splits: test (-1)
- **LAMBADA** — total ?; splits: test (-1)
- **HellaSwag** — total ?; splits: test (-1)
- **StoryCloze 2016** — total ?; splits: test (-1)
- **Natural Questions** — total ?; splits: test (-1)
- **WebQuestions** — total ?; splits: test (-1)
- **TriviaQA** — total ?; splits: test (-1)
- **WMT14/WMT16 Translation** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of predictions that exactly match the gold answer string.
- `perplexity` — range: [0, ∞)
  - Exponential of the average cross-entropy loss over the test set.
- `BLEU` — range: [0, 100]
  - Multi-bleu score using XLM tokenization, as measured by multi-bleu.perl.

## Input / output format

**Input**: Text prompts containing task instructions and K demonstrations (examples), followed by the query to be completed. For zero-shot, only the instruction or raw text is provided.

**Output**: Autoregressive text completion. For cloze/QA tasks, a single word or short answer. For translation, the translated sentence.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    ppl = math.exp(sum(-math.log(p) for p in model_probs) / len(golds))
    bleu = sacrebleu.corpus_bleu(predictions, [golds])
    return {'accuracy': acc, 'perplexity': ppl, 'bleu': bleu}
```

## Common pitfalls

- Data contamination in training set affects some benchmarks (e.g., LAMBADA, TriviaQA).
- Zero-shot vs few-shot formatting differences significantly impact performance (e.g., LAMBADA requires fill-in-the-blank framing for few-shot).
- PTB only evaluated zero-shot due to lack of clear few-shot split.

## Evidence (verbatim from paper)

> We evaluate all tasks in the few-shot, one-shot, and zero-shot settings. ... We calculate zero-shot perplexity on the Penn Tree Bank (PTB) dataset ... GPT-3 achieves 86.4% accuracy in the few-shot setting ... We report BLEU scores on the WMT'14 Fr↔En WMT'16 De↔En, and WMT'16 Ro↔En datasets as measured by multi-bleu.perl with XLM's tokenization in order to compare most closely with prior unsupervised NMT work.

## Citation

```bibtex
@misc{brown2020gpt3,
  title={Language Models are Few-Shot Learners},
  author={Tom B. Brown et al. (2020)},
  year={2020},
  note={arXiv:2005.14165}
}
```

- arXiv: 2005.14165