Wmt2016 Mt Eval

---
name: wmt2016-mt-eval
description: Evaluates neural machine translation systems across multiple language pairs (EN-DE, EN-CS, CS-EN, EN-RO, RO-EN, EN-RU, RU-EN) on news text. It measures translation quality using BLEU scores on held-out test sets to assess the impact of techniques like back-translation, ensembling, and subword segmentation. Use when the user wants to benchmark on WMT 2016 News Translation, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1606.02891
  bibtex_key: sennrich2016edinburgh
  confidence: high
---

# wmt2016-mt-eval

> Edinburgh Neural Machine Translation Systems for WMT 16 — Sennrich et al. (2016) (arXiv:1606.02891, 2016)

## What this evaluates

Evaluates neural machine translation systems across multiple language pairs (EN-DE, EN-CS, CS-EN, EN-RO, RO-EN, EN-RU, RU-EN) on news text. It measures translation quality using BLEU scores on held-out test sets to assess the impact of techniques like back-translation, ensembling, and subword segmentation.

## Datasets

- **WMT 2016 News Translation** — total ?; splits: test (-1), dev (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard corpus-level BLEU score measuring 4-gram precision with a brevity penalty. Reported as a percentage (0-100) in the tables.

## Input / output format

**Input**: Source language sentence, typically tokenized and encoded with BPE subword operations (or raw text for baselines).

**Output**: Target language sentence, typically tokenized and encoded with BPE subword operations (or raw text for baselines).

## Scoring recipe

```python
def compute_bleu(hypotheses, references):
    # hypotheses: list of str (model outputs)
    # references: list of list of str (gold translations)
    # Uses standard corpus-level BLEU with 4-gram precision and brevity penalty
    bleu_score = corpus_bleu(references, hypotheses)
    return round(bleu_score * 100, 1)
```

## Common pitfalls

- BLEU scores are highly sensitive to tokenization and BPE segmentation; results are not directly comparable across different preprocessing pipelines or character-level baselines.
- The paper reports both dev and test scores, but test scores are on held-out data (newstest2016) and must not be used for hyperparameter tuning or model selection.
- Ensemble and reranking results aggregate multiple model checkpoints, which can mask per-checkpoint variance and inflate reported gains compared to single-model baselines.

## Evidence (verbatim from paper)

> Table 2: English↔German translation results (Bleu) on dev (newstest2015) and test (newstest2016). Submitted system in bold.

## Citation

```bibtex
@misc{sennrich2016edinburgh,
  title={Edinburgh Neural Machine Translation Systems for WMT 16},
  author={Sennrich et al. (2016)},
  year={2016},
  note={arXiv:1606.02891}
}
```

- arXiv: 1606.02891