Wmt Ape Eval

---
name: wmt-ape-eval
description: Evaluates a model's ability to perform Automatic Post-Editing (APE) by correcting machine-translated German sentences using the original source text. It probes error detection, grammatical correction, and word-copying capabilities in a multi-source sequence-to-sequence setting. Use when the user wants to benchmark on WMT APE, or asks about evaluating this task. Reports case-sensitive BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.03627
  bibtex_key: huang2019learning
  confidence: high
---

# wmt-ape-eval

> Learning to Copy for Automatic Post-Editing — Huang et al. (2019) (arXiv:1911.03627, 2019)

## What this evaluates

Evaluates a model's ability to perform Automatic Post-Editing (APE) by correcting machine-translated German sentences using the original source text. It probes error detection, grammatical correction, and word-copying capabilities in a multi-source sequence-to-sequence setting.

## Datasets

- **WMT APE** — total ?; splits: train (23000), dev2016 (1000), test2016 (2000), test2017 (2000), dev2018 (1000)

## Metrics

- `case-sensitive BLEU` **(primary)** — range: [0, 100]
  - Standard n-gram precision with brevity penalty, computed using multi-bleu.perl. Evaluated case-sensitively as per WMT APE official guidelines.
- `TER` — range: percent
  - Translation Edit Rate: minimum number of edits (insertions, deletions, substitutions, shifts) required to change the hypothesis into the reference, divided by reference length. Calculated using TERcom.

## Input / output format

**Input**: Pair of sequences: original source sentence (src) and machine-translated sentence (mt).

**Output**: Corrected German sentence (post-edit, pe).

## Scoring recipe

```python
bleu = multi_bleu.perl(predictions, references)  # case-sensitive
ter = TERcom(predictions, references)            # lower score is better
return {'bleu': bleu, 'ter': ter}
```

## Common pitfalls

- BLEU is computed case-sensitively, unlike standard lowercased BLEU, which significantly impacts scores.
- The dataset distinguishes between PBSMT and NMT variants with different training sizes; results must be compared within the same task variant.
- Ensemble models report higher scores than single models; comparisons must specify which variant is used.

## Evidence (verbatim from paper)

> We used the same evaluation metrics as the official WMT APE task (Chatterjee et al., 2018): casesensitive BLEU and TER. BLEU is computed by multi-bleu.perl (Koehn et al., 2007). TER is calculated using TERcom.

## Citation

```bibtex
@misc{huang2019learning,
  title={Learning to Copy for Automatic Post-Editing},
  author={Huang et al. (2019)},
  year={2019},
  note={arXiv:1911.03627}
}
```

- arXiv: 1911.03627