Instruction Tuning Eval

---
name: instruction-tuning-eval
description: Evaluates the instruction-following capability and alignment (helpfulness, honesty, harmlessness) of instruction-tuned LLMs on unseen tasks across English and Chinese. Use when the user wants to benchmark on User-Oriented-Instructions-252, Vicuna-Instructions-80, Unnatural Instructions, or asks about evaluating this task. Reports Relative Score (GPT-4).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.03277
  bibtex_key: [REDACTED]
  confidence: high
---

# instruction-tuning-eval

> Instruction Tuning with GPT-4 — Peng et al. (2023) (arXiv:2304.03277, 2023)

## What this evaluates

Evaluates the instruction-following capability and alignment (helpfulness, honesty, harmlessness) of instruction-tuned LLMs on unseen tasks across English and Chinese.

## Datasets

- **User-Oriented-Instructions-252** — total 252; splits: test (252)
- **Vicuna-Instructions-80** — total 80; splits: test (80)
- **Unnatural Instructions** — total 68478; splits: test (68478)

## Metrics

- `Relative Score (GPT-4)` **(primary)** — range: percent
  - Pairwise comparison rated 1-10 by GPT-4. Relative score = (sum of scores for model A) / (sum of scores for model A + sum of scores for model B) * 100%.
- `HHH Human Vote %` — range: percent
  - Human annotators on MTurk vote on three criteria: Helpfulness, Honesty, Harmlessness. Reported as percentage of votes for win, tie, or lose against a baseline.
- `ROUGE-L` — range: [0, 1]
  - Standard ROUGE-L score averaged over 9,000 samples from Unnatural Instructions, grouped by ground-truth response length.

## Input / output format

**Input**: Instruction prompt (text) provided to the model.

**Output**: Text response generated by the model.

## Scoring recipe

```python
def compute_relative_score(preds_A, preds_B, prompts):
    scores_A, scores_B = [], []
    for p, a, b in zip(prompts, preds_A, preds_B):
        scores_A.append(gpt4_pairwise_score(a, b))
        scores_B.append(gpt4_pairwise_score(b, a))
    return sum(scores_A) / (sum(scores_A) + sum(scores_B)) * 100
```

## Common pitfalls

- The GPT-4 automatic evaluation uses pairwise comparisons against a specific opponent (ChatGPT or GPT-4), so scores are relative, not absolute.
- Human evaluation only covers the 252-user-oriented instructions, not the full Vicuna or Unnatural Instructions sets.
- ROUGE-L is computed on a 9K subset of Unnatural Instructions, not the full 68K.

## Evidence (verbatim from paper)

> We compare LLaMA-GPT4 with GPT-4 and Alpaca unnatural instructions in Figure 6. In terms of the average ROUGE-L scores, Alpaca outperforms the other two models.

## Citation

```bibtex
@misc{peng2023instructiontuning,
  title={Instruction Tuning with GPT-4},
  author={Peng et al. (2023)},
  year={2023},
  note={arXiv:2304.03277}
}
```

- arXiv: 2304.03277