Wmt21 News Translation Eval

---
name: wmt21-news-translation-eval
description: Evaluates the translation quality and inference speed of non-autoregressive versus autoregressive machine translation models on English-German news text. It probes the practical trade-offs between decoding latency and translation accuracy under realistic deployment conditions. Use when the user wants to benchmark on WMT21 News Translation, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.01966
  bibtex_key: helcl2022nonautoregressive
  confidence: medium
---

# wmt21-news-translation-eval

> Non-Autoregressive Machine Translation: It's Not as Fast as it Seems — Helcl et al. (2022) (arXiv:2205.01966, 2022)

## What this evaluates

Evaluates the translation quality and inference speed of non-autoregressive versus autoregressive machine translation models on English-German news text. It probes the practical trade-offs between decoding latency and translation accuracy under realistic deployment conditions.

## Datasets

- **WMT21 News Translation** — total ?; splits: train (-1), test (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Standard n-gram overlap metric between predicted and reference translations, typically computed with sentence-level averaging and smoothing.
- `Latency` — range: ms or words/sec
  - Inference time measured in milliseconds per sentence or words per second, evaluated under varying batch sizes and hardware conditions (CPU vs GPU).

## Input / output format

**Input**: English source sentence (string)

**Output**: German target sentence (string)

## Scoring recipe

```python
def compute_bleu(predictions, references):
    return nltk.translate.bleu_score.corpus_bleu(references, predictions) * 100

def compute_latency(model, inputs, batch_size=1, device='cpu'):
    times = []
    for batch in get_batches(inputs, batch_size):
        start = time.time()
        _ = model.generate(batch, device=device)
        times.append((time.time() - start) * 1000)
    return sum(times) / len(times)  # ms per batch
```

## Common pitfalls

- Measuring latency on GPU with batch size 1, which artificially favors NAR models and ignores real-world batching overhead.
- Focusing only on peak throughput without accounting for per-sentence decoding latency or quality degradation under realistic CPU inference conditions.
- Using outdated or small test sets that do not reflect modern domain shifts or sentence length distributions.

## Evidence (verbatim from paper)

> A CTC-based NAR model achieves state-of-the-art BLEU on WMT14 but underperforms on recent test sets, highlighting the quality-speed trade-off and calling for more rigorous, standardized evaluation that includes both translation quality and realistic inference conditions.

## Citation

```bibtex
@misc{helcl2022nonautoregressive,
  title={Non-Autoregressive Machine Translation: It's Not as Fast as it Seems},
  author={Helcl et al. (2022)},
  year={2022},
  note={arXiv:2205.01966}
}
```

- arXiv: 2205.01966