Wmt17 Nmt Benchmark Eval

---
name: wmt17-nmt-benchmark-eval
description: Evaluates the training efficiency, inference speed, and translation quality of neural machine translation systems on standard WMT17 benchmarks. It probes the trade-offs between model architecture, hardware acceleration (FP16/INT8), and batching strategies. Use when the user wants to benchmark on WMT17 English-German, WMT17 Russian-English, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.05851
  bibtex_key: hieber2022sockeye3
  confidence: high
---

# wmt17-nmt-benchmark-eval

> Sockeye 3: Fast Neural Machine Translation with PyTorch — Hieber et al. (2022) (arXiv:2207.05851, 2022)

## What this evaluates

Evaluates the training efficiency, inference speed, and translation quality of neural machine translation systems on standard WMT17 benchmarks. It probes the trade-offs between model architecture, hardware acceleration (FP16/INT8), and batching strategies.

## Datasets

- **WMT17 English-German** — total 5900000; splits: train (-1)
- **WMT17 Russian-English** — total 25000000; splits: train (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Standard n-gram overlap metric for machine translation quality. Computed on the full test set after averaging the 8 best model checkpoints.
- `Training Time (Hours)` — range: hours
  - Wall-clock time required to complete the specified number of training updates (25K for En-De, 70K for Ru-En) on 8 GPUs.
- `Translation Speed (Sent/Sec)` — range: sent/sec
  - Number of source sentences processed per second during inference, averaged over 3 runs. Measured on GPU (g4dn.xlarge) and CPU (c5.2xlarge) with batch sizes 1 and 64.

## Input / output format

**Input**: Source language sentences pre-processed with byte-pair encoding (32K operations) and filtered to a maximum length of 95 tokens.

**Output**: Target language sentences generated via greedy search or beam search.

## Scoring recipe

```python
train_time = measure_wall_clock_time(model, dataset, updates=25000, devices=8)
speed = measure_throughput(model, dataset, batch_size=64, device='gpu')
predictions = []
for ckpt in best_8_checkpoints:
    predictions.append(translate(dataset, ckpt))
final_predictions = average_predictions(predictions)
bleu = compute_bleu(gold, final_predictions)
```

## Common pitfalls

- BLEU scores are computed after averaging the 8 best model checkpoints, not from a single checkpoint.
- Inference speed varies drastically with batch size (1 vs 64) and hardware (GPU vs CPU); results are not directly comparable across these settings.
- The exact BLEU implementation (e.g., Moses vs SacreBLEU) is not specified, which can cause minor score discrepancies.

## Evidence (verbatim from paper)

> We select two translation tasks for which pre-processed data sets are available: WMT17 English-German (5.9M sentences) and Russian-English (25M sentences). We further process the data by applying byte-pair encoding with 32K operations and filtering out sentences longer than 95 tokens. Models are trained for either 25K updates (En-De) or 70K updates (Ru-En) with checkpoints every 500 updates. The 8 best checkpoints are averaged to produce the final model weights. Shown in Table 4, Sockeye and Fairseq are fastest, training models with comparable BLEU scores in comparable time.

## Citation

```bibtex
@misc{hieber2022sockeye3,
  title={Sockeye 3: Fast Neural Machine Translation with PyTorch},
  author={Hieber et al. (2022)},
  year={2022},
  note={arXiv:2207.05851}
}
```

- arXiv: 2207.05851