Mnmt Eval

---
name: mnmt-eval
description: Evaluates multilingual neural machine translation performance across many-to-one, one-to-many, and many-to-many translation scenarios, testing how dynamic parameter differentiation impacts translation quality across diverse language pairs and resource levels. Use when the user wants to benchmark on OPUS, WMT, IWSLT'17, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.13619
  bibtex_key: wang2021parameter
  confidence: high
---

# mnmt-eval

> Parameter Differentiation based Multilingual Neural Machine Translation — Qian Wang et al. (2021) (arXiv:2112.13619, 2021)

## What this evaluates

Evaluates multilingual neural machine translation performance across many-to-one, one-to-many, and many-to-many translation scenarios, testing how dynamic parameter differentiation impacts translation quality across diverse language pairs and resource levels.

## Datasets

- **OPUS** — total ?; splits: train (-1), val (-1)
- **WMT** — total ?; splits: train (-1)
- **IWSLT'17** — total ?; splits: train (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Standard n-gram precision score computed using SacreBLEU, which handles tokenization and reference formatting automatically.

## Input / output format

**Input**: Source sentence in a source language (e.g., English, German, etc.)

**Output**: Target sentence in the target language

## Scoring recipe

```python
import sacrebleu
def compute_bleu(predictions, references):
    return sacrebleu.corpus_bleu(predictions, [references]).score
```

## Common pitfalls

- Using a different BLEU implementation (e.g., Moses or custom tokenization) instead of SacreBLEU will yield non-comparable scores.
- Forgetting to apply the temperature-based sampling (τ=5) during WMT dataset training, which is explicitly required for fair comparison.
- Comparing model sizes without normalizing for the parameter differentiation upper bound (O), as the method's size scales with the differentiation limit rather than language count.

## Evidence (verbatim from paper)

> We measure the translation quality by BLEU score (Papineni et al. 2002) with SacreBLEU. We use the public OPUS and WMT multilingual datasets to evaluate our method on many-to-one (M2O) and one-to-many (O2M) translation scenarios, and the IWSLT datasets for the many-to-many (M2M) translation scenario.

## Citation

```bibtex
@misc{wang2021parameter,
  title={Parameter Differentiation based Multilingual Neural Machine Translation},
  author={Qian Wang et al. (2021)},
  year={2021},
  note={arXiv:2112.13619}
}
```

- arXiv: 2112.13619