Wikimatrix Eval

---
name: wikimatrix-eval
description: Assesses the quality of automatically mined parallel sentence pairs by training neural machine translation models and measuring their downstream translation accuracy. Use when the user wants to benchmark on WikiMatrix, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1907.05791
  bibtex_key: schwenk2019wikimatrix
  confidence: high
---

# wikimatrix-eval

> WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia — Schwenk et al. (2019) (arXiv:1907.05791, 2019)

## What this evaluates

Assesses the quality of automatically mined parallel sentence pairs by training neural machine translation models and measuring their downstream translation accuracy.

## Datasets

- **WikiMatrix** — total 135000000; splits: train (-1); repo https://github.com/facebookresearch/LASER

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard BLEU score (Papineni et al., 2002) computed on the TED test set. It measures the geometric mean of 1- to 4-gram precisions with a brevity penalty to penalize overly short translations.

## Input / output format

**Input**: Source sentences from the TED test set.

**Output**: Translated target sentences generated by the trained NMT model.

## Scoring recipe

```python
def compute_bleu(predictions, references):
    # Standard BLEU-4 calculation with brevity penalty
    return nltk.translate.bleu_score.corpus_bleu(
        [[ref] for ref in references],
        predictions,
        weights=(0.25, 0.25, 0.25, 0.25)
    ) * 100
```

## Common pitfalls

- Evaluation is only reported for language pairs with >25k mined sentences, not the full 1620 pairs.
- The metric measures downstream NMT translation quality rather than direct bitext alignment accuracy.

## Evidence (verbatim from paper)

> We then turn to a qualitative assessment by training NMT systems for all language pairs with more than twenty-five thousand mined sentences (Section 5.2). Evaluated via NMT on TED, the mined data achieves strong BLEU scores, demonstrating effectiveness for training machine translation systems in low-resource and non-English language settings.

## Citation

```bibtex
@misc{schwenk2019wikimatrix,
  title={WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia},
  author={Schwenk et al. (2019)},
  year={2019},
  note={arXiv:1907.05791}
}
```

- arXiv: 1907.05791