Instruction Nmt Eval

---
name: instruction-nmt-eval
description: Evaluates whether neural machine translation models can follow diverse natural language instructions (e.g., formality, voice, casing, simplification) without task-specific retraining, while maintaining general translation quality. Use when the user wants to benchmark on WMT'20 News Translation (EN-DE), Multi-30K, Custom Instruction Dataset, or asks about evaluating this task. Reports RR (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.05553
  bibtex_key: [REDACTED]
  confidence: high
---

# instruction-nmt-eval

> On Instruction-Finetuning Neural Machine Translation Models — Raunak et al. (2024) (arXiv:2410.05553, 2024)

## What this evaluates

Evaluates whether neural machine translation models can follow diverse natural language instructions (e.g., formality, voice, casing, simplification) without task-specific retraining, while maintaining general translation quality.

## Datasets

- **WMT'20 News Translation (EN-DE)** — total ?; splits: test (-1)
- **Multi-30K** — total 29000; splits: train (29000)
- **Custom Instruction Dataset** — total 30000; splits: finetune (27000), eval (3000)

## Metrics

- `RR (%)` **(primary)** — range: percent
  - Resolution Rate (RR) measures the percentage of test instances where the model's output successfully satisfies the given natural language instruction constraint.
- `ChrF_instruction` — range: [0, 1]
  - Character n-gram F-score computed on the instruction-modified translation output to evaluate stylistic and constraint adherence.
- `BLEU` — range: [0, 1]
  - Standard n-gram precision-based metric for general translation quality, computed via SacreBLEU.
- `ChrF2` — range: [0, 1]
  - Character n-gram F-score (order 2) for general translation quality.
- `TER` — range: [0, 1]
  - Translation Edit Rate measuring the number of edits required to transform the hypothesis into the reference.
- `COMET` — range: [0, 1]
  - Reference-based neural metric that scores translation quality using a pretrained model.
- `Formal Accuracy` — range: percent
  - Percentage of translations correctly classified as formal by an external evaluator or classifier.
- `Informal Accuracy` — range: percent
  - Percentage of translations correctly classified as informal by an external evaluator or classifier.

## Input / output format

**Input**: Source sentence concatenated with a natural language instruction (or image tokens for multimodal tasks) prepended to the source text.

**Output**: Translated target sentence that adheres to the specified instruction constraints.

## Scoring recipe

```python
def compute_metrics(predictions, instructions, references):
    rr_correct = sum(1 for p, i in zip(predictions, instructions) if satisfies_constraint(p, i))
    rr = (rr_correct / len(predictions)) * 100
    chrf_inst = sacrechrf(predictions, references).score
    return {'RR (%)': rr, 'ChrF_instruction': chrf_inst}
```

## Common pitfalls

- The instruction evaluation uses a separate 10% held-out split of the custom 30-task dataset, not the WMT'20 test set.
- Zero-shot composition is evaluated on instruction combinations never seen during the 90% finetuning phase.
- General translation metrics (BLEU/ChrF2) are measured on WMT'20, while instruction-specific metrics (RR/ChrF_instruction) are measured on the task-specific 1K-sample test sets.

## Evidence (verbatim from paper)

> The metrics BLEU, ChrF2, TER Papineni et al. ([2002]); Popović ([2015]); Snover et al. ([2006]) for the trained model on the WMT’20 validation and test sets (under beam size of $1$) as measured using SacreBLEU Post ([2018]) are presented in Appendix B, alongside reference-based COMET Rei et al. ([2020]) scores.

## Citation

```bibtex
@misc{raunak2024instructionfinetuning,
  title={On Instruction-Finetuning Neural Machine Translation Models},
  author={Raunak et al. (2024)},
  year={2024},
  note={arXiv:2410.05553}
}
```

- arXiv: 2410.05553