Back Translation Wake Sleep Eval

---
name: back-translation-wake-sleep-eval
description: Evaluates neural machine translation models on English-German, German-English, English-Latvian, and Latvian-English translation tasks. It probes the effectiveness of iterative back-translation (wake-sleep extension) compared to standard back-translation and baseline MLE training across supervised and semi-supervised domain adaptation scenarios. Use when the user wants to benchmark on WMT 2017, TED (IWSLT 2014), or asks about evaluating this task. Reports BLEU (SACREBLEU v1.2.3).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1806.04402
  bibtex_key: cotterell2018explaining
  confidence: high
---

# back-translation-wake-sleep-eval

> Explaining and Generalizing Back-Translation through Wake-Sleep — Cotterell et al. (2018) (arXiv:1806.04402, 2018)

## What this evaluates

Evaluates neural machine translation models on English-German, German-English, English-Latvian, and Latvian-English translation tasks. It probes the effectiveness of iterative back-translation (wake-sleep extension) compared to standard back-translation and baseline MLE training across supervised and semi-supervised domain adaptation scenarios.

## Datasets

- **WMT 2017** — total ?; splits: train (-1), dev (-1), test (-1)
- **TED (IWSLT 2014)** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `BLEU (SACREBLEU v1.2.3)` **(primary)** — range: percent
  - Standard n-gram overlap metric. Evaluated using SACREBLEU v1.2.3 with specific tokenization and smoothing: TED uses `case.lc+numrefs.1+smooth.exp+tok.13a.version.1.2.3`, WMT uses `case.mixed+numrefs.1+smooth.exp+tok.13a.version.1.2.3`.

## Input / output format

**Input**: Source sentence in English, German, or Latvian.

**Output**: Target sentence in the corresponding target language.

## Scoring recipe

```python
import sacrebleu

def compute_bleu(hypotheses, references, dataset_type):
    if dataset_type == 'TED':
        score = sacrebleu.corpus_bleu(
            hypotheses, references,
            lowercase=True,
            smooth_method='exp',
            tokenize='13a'
        )
    elif dataset_type == 'WMT':
        score = sacrebleu.corpus_bleu(
            hypotheses, references,
            lowercase=False,
            smooth_method='exp',
            tokenize='13a',
            case_mixed=True
        )
    return score.score
```

## Common pitfalls

- Using different tokenization or smoothing settings than SACREBLEU v1.2.3, which drastically changes BLEU scores.
- Confusing the baseline (Iteration 0, MLE-only) with the back-translation iterations when reporting improvements.
- Failing to use the correct test set sizes or mixing up dev/test splits for early stopping.

## Evidence (verbatim from paper)

> The models are evaluated with respect to BLEU (Papineni et al., 2002) using the SACREBLEU tool (v.1.2.3) (Post, 2018) on tokenized (WMT: recased) system outputs.

## Citation

```bibtex
@misc{cotterell2018explaining,
  title={Explaining and Generalizing Back-Translation through Wake-Sleep},
  author={Cotterell et al. (2018)},
  year={2018},
  note={arXiv:1806.04402}
}
```

- arXiv: 1806.04402