Wmt21 Nmt Eval

---
name: wmt21-nmt-eval
description: Evaluates neural machine translation performance across news and biomedical domains for English-German and English-Russian language pairs. It probes the model's ability to handle domain-specific vocabulary, cross-lingual alignment, and translation quality under constrained data conditions typical of shared task tracks. Use when the user wants to benchmark on WMT21 News & Biomedical Shared Tasks, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2111.08634
  bibtex_key: subramanian2021nemo
  confidence: high
---

# wmt21-nmt-eval

> NVIDIA NeMo Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21 — Subramanian et al. (2021) (arXiv:2111.08634, 2021)

## What this evaluates

Evaluates neural machine translation performance across news and biomedical domains for English-German and English-Russian language pairs. It probes the model's ability to handle domain-specific vocabulary, cross-lingual alignment, and translation quality under constrained data conditions typical of shared task tracks.

## Datasets

- **WMT21 News & Biomedical Shared Tasks** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard corpus-level BLEU score computed over 1-4 gram precisions with a brevity penalty. Calculated using the official WMT sacreBLEU script with Moses tokenization and lowercasing conventions.

## Input / output format

**Input**: Source sentence in English, German, or Russian.

**Output**: Translated sentence in the target language.

## Scoring recipe

```python
def compute_bleu(predictions, references):
    # Normalize text (lowercase, Moses tokenize)
    # Compute 1-4 gram precisions for each prediction-reference pair
    # Apply brevity penalty based on length ratio
    # Return geometric mean of precisions * brevity_penalty
    return sacrebleu.corpus_bleu(predictions, [references])
```

## Common pitfalls

- Domain-specific vocabulary filtering thresholds (e.g., 0.90 cutoff on classifier scores) drastically change training data composition; failing to replicate these steps alters model capacity and final scores.
- Post-processing steps like quote replacement must be applied to model outputs before scoring to match official WMT evaluation scripts.
- BPE tokenizers are trained separately per language pair and domain; using a shared or mismatched tokenizer will skew n-gram alignment and BLEU computation.

## Evidence (verbatim from paper)

> resulting in a 43.8 BLEU score on Ru→En and 40.3 on En→Ru, surpassing prior year results.

## Citation

```bibtex
@misc{subramanian2021nemo,
  title={NVIDIA NeMo Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21},
  author={Subramanian et al. (2021)},
  year={2021},
  note={arXiv:2111.08634}
}
```

- arXiv: 2111.08634