Palm Fewshot Nlp Eval

---
name: palm-fewshot-nlp-eval
description: Evaluates the few-shot and fine-tuned capabilities of large autoregressive language models across a wide range of English NLP benchmarks, including question answering, reading comprehension, common sense reasoning, and natural language inference. It also assesses performance on a large collection of collaborative reasoning and language tasks to probe multi-step reasoning and general language understanding. Use when the user wants to benchmark on English NLP Benchmarks (29 tasks), MMLU, BIG-bench (textual), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2204.02311
  bibtex_key: [REDACTED]
  confidence: high
---

# palm-fewshot-nlp-eval

> PaLM: Scaling Language Modeling with Pathways — Chowdhery et al. (2022) (arXiv:2204.02311, 2022)

## What this evaluates

Evaluates the few-shot and fine-tuned capabilities of large autoregressive language models across a wide range of English NLP benchmarks, including question answering, reading comprehension, common sense reasoning, and natural language inference. It also assesses performance on a large collection of collaborative reasoning and language tasks to probe multi-step reasoning and general language understanding.

## Datasets

- **English NLP Benchmarks (29 tasks)** — total ?; splits: test (-1)
- **MMLU** — total ?; splits: test (-1)
- **BIG-bench (textual)** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly predicted options or labels for multiple-choice and classification tasks.
- `exact-match accuracy` — range: [0, 1]
  - Percentage of generated answers that exactly match the gold label string.
- `F1 score` — range: [0, 1]
  - Harmonic mean of token-level precision and recall between the generated answer and the gold answer.
- `preferred metric` — range: [0, 100]
  - Task-specific metric normalized to [0, 100] by setting the maximum possible score to 100 and random chance to 0.

## Input / output format

**Input**: Autoregressive text prompts containing task instructions and k-shot exemplars (0, 1, or few-shot with k ranging from 2 to 100 depending on the task). For fine-tuning, a task-proportionate mixture of SuperGLUE tasks is used.

**Output**: Autoregressive text completion predicting the next token(s) until a stop condition or maximum length. For multiple-choice tasks, the model generates the answer option text.

## Scoring recipe

```python
def score(predictions, golds, metric_type):
    if metric_type == 'exact-match':
        return sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    elif metric_type == 'f1':
        return compute_f1(predictions, golds)
    elif metric_type == 'accuracy':
        return sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds)
    elif metric_type == 'normalized':
        raw = compute_task_metric(predictions, golds)
        return normalize_to_0_100(raw)
```

## Common pitfalls

- Using different few-shot shot counts (k) across tasks without reporting the exact k used, making cross-task comparison difficult.
- Comparing results from models that use fine-tuning or multi-task adaptation against purely few-shot pretrained models, as the paper explicitly excludes them from primary comparisons.
- Ignoring the normalization convention for BIG-bench tasks, where scores are scaled to [0, 100] with random chance at 0, which can produce negative values if performance is below random.

## Evidence (verbatim from paper)

> For each task, the results of its preferred metric are used. The results are normalized by setting the maximum score to 100 and the random chance score to 0 for multiple-choice tasks, so that they are negative valued if the model performs worse than random chance.

## Citation

```bibtex
@misc{chowdhery2022palm,
  title={PaLM: Scaling Language Modeling with Pathways},
  author={Chowdhery et al. (2022)},
  year={2022},
  note={arXiv:2204.02311}
}
```

- arXiv: 2204.02311