Few Shot Nlp Eval

---
name: few-shot-nlp-eval
description: Evaluates the zero-shot and few-shot capabilities of large language models across a diverse suite of NLP, reasoning, and commonsense benchmarks. It measures how efficiently a model scales with compute and whether additional training objectives unlock emergent reasoning abilities. Use when the user wants to benchmark on GPT-3 suite, BigBench Emergent Suite, Commonsense QA benchmarks, Closed-book QA benchmarks, or asks about evaluating this task. Reports average score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.11399
  bibtex_key: tay2022transcending
  confidence: high
---

# few-shot-nlp-eval

> Transcending Scaling Laws with 0.1% Extra Compute — Yi Tay et al. (2022) (arXiv:2210.11399, 2022)

## What this evaluates

Evaluates the zero-shot and few-shot capabilities of large language models across a diverse suite of NLP, reasoning, and commonsense benchmarks. It measures how efficiently a model scales with compute and whether additional training objectives unlock emergent reasoning abilities.

## Datasets

- **GPT-3 suite** — total ?; splits: test (-1)
- **BigBench Emergent Suite** — total ?; splits: test (-1)
- **Commonsense QA benchmarks** — total ?; splits: test (-1)
- **Closed-book QA benchmarks** — total ?; splits: test (-1)

## Metrics

- `average score` **(primary)** — range: percent
  - Un-normalized arithmetic mean of per-task accuracy scores across all evaluated tasks.
- `exact-match accuracy` — range: percent
  - Fraction of predictions that exactly match the ground-truth answer or label.

## Input / output format

**Input**: Zero-shot or few-shot natural language prompts (e.g., 5-shot for BigBench tasks, 0-shot or few-shot for NLP benchmarks), often with task-specific templates or chain-of-thought instructions.

**Output**: Natural language text or discrete answer tokens corresponding to the task's expected output format.

## Scoring recipe

```python
def compute_average_score(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        scores.append(1.0 if pred.strip() == gold.strip() else 0.0)
    return sum(scores) / len(scores) * 100
```

## Common pitfalls

- Averages are computed over un-normalized task scores, which can be skewed by tasks with different score ranges or difficulty levels.
- Chain-of-thought prompting is used for reasoning tasks but explicitly avoided for the BigBench Emergent Suite, making direct comparisons with CoT-tuned models invalid.
- Savings rates and scaling curves are reported in FLOPs rather than wall-clock time or tokens processed, complicating real-world efficiency comparisons.

## Evidence (verbatim from paper)

> For evaluation, we use the average score of NLU and NLG tasks from the GPT-3 suite*(Brown et al., [2020](#bib.bib5 ""))*. In total we use 26 tasks (e.g., TriviaQA, NaturalQuestions, SuperGLUE, PIQA, OpenbookQA, ANLI etc). Details and exact scores for Figure [2](#S4.F2 "Figure 2 ‣ 4.1 Improved Scaling Properties on Few-shot Learning ‣ 4 Experiments ‣ Transcending Scaling Laws with 0.1% Extra Compute") can be found in the Appendix.

## Citation

```bibtex
@misc{tay2022transcending,
  title={Transcending Scaling Laws with 0.1% Extra Compute},
  author={Yi Tay et al. (2022)},
  year={2022},
  note={arXiv:2210.11399}
}
```

- arXiv: 2210.11399