Wmt Bleu Eval

---
name: wmt-bleu-eval
description: Evaluates the quality of an unsupervised web-mined parallel corpus by training neural machine translation models on it and measuring translation performance on standard WMT test sets. It probes whether mined pseudo-parallel data can effectively substitute for human-labeled data in both supervised and unsupervised MT training pipelines. Use when the user wants to benchmark on WMT2014 test set, WMT2016 test set, or asks about evaluating this task. Reports BELU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2009.08595
  bibtex_key: lai2020unsupervised
  confidence: high
---

# wmt-bleu-eval

> Unsupervised Parallel Corpus Mining on Web Data — Lai et al. (2020) (arXiv:2009.08595, 2020)

## What this evaluates

Evaluates the quality of an unsupervised web-mined parallel corpus by training neural machine translation models on it and measuring translation performance on standard WMT test sets. It probes whether mined pseudo-parallel data can effectively substitute for human-labeled data in both supervised and unsupervised MT training pipelines.

## Datasets

- **WMT2014 test set** — total ?; splits: test (-1)
- **WMT2016 test set** — total ?; splits: test (-1)

## Metrics

- `BELU` **(primary)** — range: [0, 100]
  - Standard corpus-level BLEU score with 4-gram precision and brevity penalty. The paper consistently misspells the metric as 'BELU' in the text and tables.

## Input / output format

**Input**: Source sentences from WMT2014/WMT2016 test sets (English, French, German, Romanian).

**Output**: Target language translations generated by the NMT model.

## Scoring recipe

```python
def compute_bleu(predictions, references):
    # predictions and references are lists of tokenized sentences
    # Uses standard corpus-level BLEU with 4-grams and brevity penalty
    bleu_score = corpus_bleu(references, predictions, weights=(0.25, 0.25, 0.25, 0.25))
    return bleu_score * 100  # Scale to percentage
```

## Common pitfalls

- The paper consistently misspells 'BLEU' as 'BELU' in the text and tables, which may cause confusion when searching for the metric.
- The evaluation mixes supervised (Scaling NMT) and unsupervised (XLM/MASS) training protocols but reports the same metric, making it unclear whether back-translation augmentation was applied for each reported score.
- The ablation study shows raw (unfiltered) data performs similarly to filtered data, which may mislead readers about the filtration step's necessity despite the classifier successfully removing low-quality pairs.

## Evidence (verbatim from paper)

> We follow the experiment setting in the Scaling NMT paper (Ott et al., 2018), including model architecture and choice of the hyper-parameters, and report the BELU score on the En-Fr and En-De directions on the WMT2014 test sets.

## Citation

```bibtex
@misc{lai2020unsupervised,
  title={Unsupervised Parallel Corpus Mining on Web Data},
  author={Lai et al. (2020)},
  year={2020},
  note={arXiv:2009.08595}
}
```

- arXiv: 2009.08595