Superni Eval

---
name: superni-eval
description: Evaluates a language model's ability to follow diverse natural language instructions across various NLP tasks in a zero-shot setting. It measures how well the model generalizes to unseen tasks without in-context examples. Use when the user wants to benchmark on SUPER-NATURALINSTRUCTIONS, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.10560
  bibtex_key: wang2022selfinstruct
  confidence: high
---

# superni-eval

> Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. (2022) (arXiv:2212.10560, 2022)

## What this evaluates

Evaluates a language model's ability to follow diverse natural language instructions across various NLP tasks in a zero-shot setting. It measures how well the model generalizes to unseen tasks without in-context examples.

## Datasets

- **SUPER-NATURALINSTRUCTIONS** — total 11900; splits: test (11900)

## Metrics

- `ROUGE-L` **(primary)** — range: percent
  - Measures the longest common subsequence of words between the predicted and reference outputs, normalized by the reference length. Reported as a percentage score.

## Input / output format

**Input**: Task definition (instruction) provided as a prompt, without in-context demonstration examples.

**Output**: Model-generated text response to the instruction.

## Scoring recipe

```python
scores = []
for instance in dataset:
    pred = model.generate(instance.prompt, temperature=0)
    scores.append(rouge_l_score(pred, instance.reference))
return mean(scores)
```

## Common pitfalls

- Using in-context examples despite the zero-shot protocol specified.
- Using non-deterministic generation (temperature > 0) which can cause high variance in ROUGE-L scores.
- Not filtering out prompt tokens from the generated output before scoring.

## Evidence (verbatim from paper)

> We use the evaluation set of SUPERNI (Wang et al., 2022), which consists of 119 tasks with 100 instances in each task. In this work, we mainly focus on the zero-shot setup, i.e., the model is prompted with the definition of the tasks only, without in-context demonstration examples. Table 3: Evaluation results on unseen tasks from SUPERNI (§4.3). From the results, we see that ① SELFINSTRUCT can boost GPT3 performance by a large margin (+33.1%) and ② nearly matches the performance of InstructGPT001. The table reports ROUGE-L scores.

## Citation

```bibtex
@misc{wang2022selfinstruct,
  title={Self-Instruct: Aligning Language Models with Self-Generated Instructions},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2212.10560}
}
```

- arXiv: 2212.10560