Instruction Robustness Eval

---
name: instruction-robustness-eval
description: Evaluates the zero-shot robustness of instruction-tuned language models to variations in instruction phrasing, even when instructions are semantically equivalent. It measures how well models maintain performance on unobserved instruction variants compared to observed ones. Use when the user wants to benchmark on MMLU, BBL, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.11270
  bibtex_key: sun2023evaluating
  confidence: high
---

# instruction-robustness-eval

> Evaluating the Zero-shot Robustness of Instruction-tuned Language Models — Sun et al. (2023) (arXiv:2306.11270, 2023)

## What this evaluates

Evaluates the zero-shot robustness of instruction-tuned language models to variations in instruction phrasing, even when instructions are semantically equivalent. It measures how well models maintain performance on unobserved instruction variants compared to observed ones.

## Datasets

- **MMLU** — total ?; splits: OBS. (-1), UNOBS. (-1); repo https://github.com/jiudingsun01/InstructionEval
- **BBL** — total ?; splits: QA (-1), BC (-1), MC (-1); repo https://github.com/jiudingsun01/InstructionEval

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard classification accuracy: the proportion of correctly predicted answers out of the total number of instances, reported as a percentage.

## Input / output format

**Input**: Instruction prompt combined with a question or task description (multiple-choice format).

**Output**: Selected answer option or generated text response.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Models show significant performance drops on unobserved instruction phrasings even when semantically equivalent to training instructions.
- Evaluating only on observed instructions masks robustness issues; unobserved splits must be reported separately.
- Accuracy gains from alignment methods may be small in absolute terms but critical for variance reduction.

## Evidence (verbatim from paper)

> Table 6 shows that our method brings observed and unobserved instruction representations closer together. The similarity is most increased in the case of the biggest accuracy gain, further suggesting the mechanism of improvement provided by soft prompt alignment.

## Citation

```bibtex
@misc{sun2023evaluating,
  title={Evaluating the Zero-shot Robustness of Instruction-tuned Language Models},
  author={Sun et al. (2023)},
  year={2023},
  note={arXiv:2306.11270}
}
```

- arXiv: 2306.11270