Wmt17 Mt Eval

---
name: wmt17-mt-eval
description: Evaluates neural machine translation systems across multiple language pairs in news and biomedical domains. Probes translation quality, domain adaptation, and system combination techniques like ensembling and reranking on held-out parallel test sets. Use when the user wants to benchmark on WMT17 News Task, HimL Biomedical Task, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1708.00726
  bibtex_key: sennrich2017wmt17
  confidence: high
---

# wmt17-mt-eval

> The University of Edinburgh's Neural MT Systems for WMT17 — Rico Sennrich et al. (WMT17 / arXiv:1708.00726, 2017)

## What this evaluates

Evaluates neural machine translation systems across multiple language pairs in news and biomedical domains. Probes translation quality, domain adaptation, and system combination techniques like ensembling and reranking on held-out parallel test sets.

## Datasets

- **WMT17 News Task** — total ?; splits: dev (-1), test (-1)
- **HimL Biomedical Task** — total ?; splits: tuning (-1), test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Sentence-level BLEU with 4-gram precision, geometric mean, and brevity penalty. Scores are reported as raw values (e.g., 30.9) representing the percentage scale.

## Input / output format

**Input**: Source language sentence, preprocessed with language-specific tokenization (e.g., Jieba for Chinese, custom tokenizer for Latvian) and BPE/subword segmentation.

**Output**: Target language sentence, postprocessed to remove extra spaces and convert ASCII punctuation to appropriate CJK Unicode equivalents where applicable.

## Scoring recipe

```python
def compute_bleu(predictions, references):
    precisions = []
    for n in range(1, 5):
        matches = sum(min(count_ngram(pred, n), count_ngram(ref, n)) for pred, ref in zip(predictions, references))
        total = sum(count_ngram(pred, n) for pred in predictions)
        precisions.append(matches / total if total > 0 else 0)
    bp = math.exp(1 - len(references)/len(predictions)) if len(predictions) < len(references) else 1
    return bp * math.exp(sum(math.log(p) for p in precisions if p > 0) / 4) * 100
```

## Common pitfalls

- Final scores rely on complex system combination: ensembling left-to-right and right-to-left models, followed by reranking, not single-model inference.
- Length penalty (alpha) is tuned per language pair on the dev set (e.g., 0.6 for EN→LV, 0.7 for LV→EN) and significantly impacts BLEU.
- Biomedical domain adaptation requires careful synthetic data generation and diacritic normalization, which can introduce errors if not handled correctly.
- Back-translation data quality and filtering heavily influence performance, especially for low-resource or morphologically complex pairs.

## Evidence (verbatim from paper)

> The final system is an ensemble of the best validation BLEU model from each of the three target left-right runs, rescored with the three target right-left runs, and reranked.

## Citation

```bibtex
@misc{sennrich2017wmt17,
  title={The University of Edinburgh's Neural MT Systems for WMT17},
  author={Rico Sennrich et al.},
  year={2017},
  note={WMT17 / arXiv:1708.00726}
}
```

- arXiv: 1708.00726