Instruction Pretraining Eval

---
name: instruction-pretraining-eval
description: Evaluates the generalization and domain-adaptive capabilities of language models pre-trained with instruction-augmented corpora. Probes zero/few-shot instruction following, general knowledge, and specialized performance in biomedicine and finance. Use when the user wants to benchmark on MMLU, PubMedQA, ChemProt, RCT, MQP, UMSLE, ConvFinQA, Headline, FiQA SA, FPB, NER, or asks about evaluating this task. Reports average task score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.14491
  bibtex_key: cheng2024instruction
  confidence: medium
---

# instruction-pretraining-eval

> Instruction Pre-Training: Language Models are Supervised Multitask Learners — Cheng et al. (2024) (arXiv:2406.14491, 2024)

## What this evaluates

Evaluates the generalization and domain-adaptive capabilities of language models pre-trained with instruction-augmented corpora. Probes zero/few-shot instruction following, general knowledge, and specialized performance in biomedicine and finance.

## Datasets

- **MMLU** — total ?; splits: test (-1)
- **PubMedQA** — total ?; splits: test (-1)
- **ChemProt** — total ?; splits: test (-1)
- **RCT** — total ?; splits: test (-1)
- **MQP** — total ?; splits: test (-1)
- **UMSLE** — total ?; splits: test (-1)
- **ConvFinQA** — total ?; splits: test (-1)
- **Headline** — total ?; splits: test (-1)
- **FiQA SA** — total ?; splits: test (-1)
- **FPB** — total ?; splits: test (-1)
- **NER** — total ?; splits: test (-1)

## Metrics

- `average task score` **(primary)** — range: percent
  - Standard dataset-specific accuracy or F1 scores, averaged across all datasets within a domain (e.g., BioMed or Finance).
- `MMLU` — range: percent
  - Zero/few-shot accuracy on the MMLU benchmark, tracking performance trends during instruction tuning.

## Input / output format

**Input**: Instruction-response pairs for domain tasks; multiple-choice questions for MMLU.

**Output**: Predicted answer or generated response per instruction/question.

## Scoring recipe

```python
domain_scores = []
for dataset in domain_datasets:
    preds = model.generate(inputs)
    scores.append(compute_metric(preds, golds))
domain_scores.extend(scores)
return sum(domain_scores) / len(domain_scores)
```

## Common pitfalls

- Evaluated datasets were explicitly excluded from the instruction synthesizer's fine-tuning data to ensure generalization testing.
- Finance NER benchmark exhibits considerable variance and may be unreliable for strict performance comparisons.

## Evidence (verbatim from paper)

> Figure[4] shows the zero/few-shot performance on MMLU during instruction tuning from the pre-trained models. The model pre-trained viaInstruction Pre-Trainingquickly outperforms the model pre-trained viaVanilla Pre-Training, and we observe a stable increasing trend of our model throughout the instruction tuning process. As shown in Table[3], Instruction Pre-Trainingconsistently outperforms Vanilla Pre-Trainingon almost all domain-specific tasks. We report the average task scores within each domain.

## Citation

```bibtex
@misc{cheng2024instruction,
  title={Instruction Pre-Training: Language Models are Supervised Multitask Learners},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2406.14491}
}
```

- arXiv: 2406.14491