Nmt Kd Eval

---
name: nmt-kd-eval
description: Evaluates neural machine translation quality under knowledge distillation settings. It measures how well student models can replicate teacher performance on standard cross-lingual translation benchmarks. Use when the user wants to benchmark on WMT'14 En-De, WMT'14 En-Fr, WMT'16 En-Ro, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.08096
  bibtex_key: zhang2023towards
  confidence: high
---

# nmt-kd-eval

> Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation — Zhang et al. (2023) (arXiv:2305.08096, 2023)

## What this evaluates

Evaluates neural machine translation quality under knowledge distillation settings. It measures how well student models can replicate teacher performance on standard cross-lingual translation benchmarks.

## Datasets

- **WMT'14 En-De** — total ?; splits: test (-1)
- **WMT'14 En-Fr** — total ?; splits: test (-1)
- **WMT'16 En-Ro** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram overlap metric between system output and reference translations, typically computed with tokenized text.
- `COMET` — range: [0, 1]
  - A reference-based neural metric that uses a pre-trained model to score translation quality based on semantic similarity and fluency.

## Input / output format

**Input**: English source sentence tokenized with BPE (32k merges).

**Output**: Target language sentence tokenized with the shared BPE vocabulary.

## Scoring recipe

```python
bleu = sacrebleu.corpus_bleu(predictions, [references])
comet_model = load_comet_model('unbabel/wmt22-comet-da')
scores = comet_model.predict({'src': sources, 'mt': predictions, 'ref': references})
comet_score = scores['scores'].mean()
```

## Common pitfalls

- BLEU scores are highly sensitive to tokenization; the paper uses BPE with 32k merges, so standard sacrebleu tokenization may differ.
- COMET scores depend on the specific model version; the paper cites Rei et al. (2020) but does not specify the exact checkpoint used.
- Knowledge distillation experiments often report only test BLEU; the paper also reports COMET, which requires a separate inference step and model loading.

## Evidence (verbatim from paper)

> To make the results more convincing, we report both BLEU and COMET Rei et al. ([2020]) scores in Tab.[6] Using Transformerbig as the teacher, our method can boost the Transformerbase students by +1.04/+0.60/+1.11 BLEU scores and +4.52/+2.57/+4.80 COMET scores on three tasks, respectively.

## Citation

```bibtex
@misc{zhang2023towards,
  title={Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2305.08096}
}
```

- arXiv: 2305.08096