Wmt22 Ted Translation Eval

---
name: wmt22-ted-translation-eval
description: Evaluates machine translation models on English-to-multiple-target-language pairs using standard development sets. It measures translation quality via BLEU scores to compare bilingual versus multilingual decoder representations and capacity. Use when the user wants to benchmark on WMT22 General Machine Translation, Multitarget TED talks, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.14230
  bibtex_key: verma2023representational
  confidence: high
---

# wmt22-ted-translation-eval

> Exploring Representational Disparities Between Multilingual and Bilingual Translation Models — Verma et al. (2023) (arXiv:2305.14230, 2023)

## What this evaluates

Evaluates machine translation models on English-to-multiple-target-language pairs using standard development sets. It measures translation quality via BLEU scores to compare bilingual versus multilingual decoder representations and capacity.

## Datasets

- **WMT22 General Machine Translation** — total ?; splits: train (-1), dev (-1)
- **Multitarget TED talks** — total ?; splits: train (-1), dev (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram overlap metric for machine translation, computed using sacrebleu. It measures the geometric mean of modified n-gram precisions (typically up to 4-grams) with a brevity penalty.

## Input / output format

**Input**: Source sentence in English, optionally prepended with a target language ID token for multilingual models.

**Output**: Generated target language sentence.

## Scoring recipe

```python
import sacrebleu
# predictions: list of generated target sentences
# references: list of gold target sentences
bleu_score = sacrebleu.corpus_bleu(predictions, [references]).score
```

## Common pitfalls

- Evaluation is performed on development sets rather than held-out test sets, which may not reflect final generalization performance.
- Checkpoint selection criteria differ across scales: small/TED models use best validation BLEU after 80 epochs, while large models use average validation loss after 240k updates, potentially confounding metric comparisons.
- Vocabulary size differs between bilingual (16K) and multilingual (32K) setups, meaning BLEU differences may reflect tokenization capacity rather than representational isotropy.

## Evidence (verbatim from paper)

> We report BLEU scores on our dev sets computed with sacrebleu *Papineni et al. ([2002](#bib.bib25 "")); Post ([2018](#bib.bib26 ""))*.

## Citation

```bibtex
@misc{verma2023representational,
  title={Exploring Representational Disparities Between Multilingual and Bilingual Translation Models},
  author={Verma et al. (2023)},
  year={2023},
  note={arXiv:2305.14230}
}
```

- arXiv: 2305.14230