Opus 100 Nmt Eval

---
name: opus-100-nmt-eval
description: Evaluates massively multilingual neural machine translation models on translation quality and language accuracy across 100 languages, including zero-shot translation between unseen language pairs. Use when the user wants to benchmark on OPUS-100, or asks about evaluating this task. Reports BLEU_94.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.11867
  bibtex_key: zhang2020improving
  confidence: high
---

# opus-100-nmt-eval

> Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation — Zhang et al. (2020) (arXiv:2004.11867, 2020)

## What this evaluates

Evaluates massively multilingual neural machine translation models on translation quality and language accuracy across 100 languages, including zero-shot translation between unseen language pairs.

## Datasets

- **OPUS-100** — total ?; splits: train (-1), test (-1); repo https://github.com/EdinburghNLP/opus-100-corpus

## Metrics

- `BLEU_94` **(primary)** — range: percent
  - Average SacreBLEU score computed over all 94 language pairs that have dedicated test sets in the one-to-many or many-to-many setting.
- `BLEU_zero` — range: percent
  - Average SacreBLEU score computed over all zero-shot translation directions (unseen X→Y pairs) in the test set.
- `ACC_zero` — range: percent
  - Translation-language accuracy measuring the proportion of generated outputs that are correctly identified as the target language using the langdetect library.
- `WR` — range: percent
  - Win ratio counting the proportion of tasks or language pairs where the proposed approach outperforms the baseline model.

## Input / output format

**Input**: Source sentence in language X, prefixed with a language token specifying the target language (e.g., <en> for English).

**Output**: Translated sentence in the specified target language.

## Scoring recipe

```python
bleu_scores = [sacrebleu.corpus_bleu(pred, [ref]).score for pred, ref in zip(predictions, references)]
metric_value = sum(bleu_scores) / len(bleu_scores)

correct = 0
for pred, target_lang in zip(predictions, target_languages):
    detected_lang = langdetect.detect(pred)
    if detected_lang == target_lang:
        correct += 1
acc_zero = correct / len(predictions)
```

## Common pitfalls

- Off-target translation severely degrades zero-shot BLEU scores, making language accuracy a critical diagnostic metric.
- Training data is highly imbalanced across the 100 languages, causing English→X performance to lag behind X→English.
- ROBT improves zero-shot translation but slightly sacrifices in-target translation BLEU scores.

## Evidence (verbatim from paper)

> We adopt BLEU (Papineni et al., 2002) for translation evaluation with the toolkit SacreBLEU (Post, 2018). We employ the langdetect library to detect the language of translations, and measure the translation-language accuracy for zero-shot cases. Rather than providing numbers for each language pair, we report average BLEU over all 94 language pairs with test sets (BLEU_94).

## Citation

```bibtex
@misc{zhang2020improving,
  title={Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation},
  author={Zhang et al. (2020)},
  year={2020},
  note={arXiv:2004.11867}
}
```

- arXiv: 2004.11867