Wmt2016 Ape Eval

---
name: wmt2016-ape-eval
description: Evaluates Automatic Post-Editing (APE) systems by measuring how well they correct machine-translated German sentences using monolingual and bilingual neural translation models combined via log-linear weighting. Use when the user wants to benchmark on WMT 2016 APE Shared Task Development Set, or asks about evaluating this task. Reports TER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1605.04800
  bibtex_key: junczysdowmunt2016loglinear
  confidence: high
---

# wmt2016-ape-eval

> Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing — Junczys-Dowmunt et al. (2016) (arXiv:1605.04800, 2016)

## What this evaluates

Evaluates Automatic Post-Editing (APE) systems by measuring how well they correct machine-translated German sentences using monolingual and bilingual neural translation models combined via log-linear weighting.

## Datasets

- **WMT 2016 APE Shared Task Development Set** — total ?; splits: dev (-1)

## Metrics

- `TER` **(primary)** — range: percent
  - Translation Edit Rate: minimum number of edits (insertions, deletions, substitutions, shifts) to transform the hypothesis into the reference, normalized by reference length.
- `BLEU` — range: percent
  - Bilingual Evaluation Understudy: geometric mean of n-gram precisions (1-4) with brevity penalty to penalize short translations.

## Input / output format

**Input**: Machine-translated German sentence (for monolingual model) or English source sentence concatenated with MT output (for bilingual model).

**Output**: Post-edited German sentence.

## Scoring recipe

```python
def compute_ter(hypothesis, reference):
    # Align hypothesis to reference using edit operations
    edits = edit_distance_align(hypothesis, reference)
    return (len(edits) / len(reference)) * 100

def compute_bleu(hypothesis, references):
    # Compute n-gram precisions for n=1..4
    precisions = [ngram_precision(hypothesis, ref, n) for n in range(1, 5)]
    # Geometric mean with brevity penalty
    bp = brevity_penalty(len(hypothesis), len(reference))
    return bp * exp(mean(log(precisions))) * 100
```

## Common pitfalls

- Model weights and PEP penalty are tuned on the development set, potentially inflating dev performance.
- Original post-editing data is oversampled 20x during training, which may bias the model toward the original MT errors rather than natural corrections.
- Faithfulness is enforced via a string-matching penalty during decoding, not as a post-hoc filter, which changes the generation dynamics.

## Evidence (verbatim from paper)

> We tune the weights on the development set towards lower TER scores; two iterations seem to be enough.

## Citation

```bibtex
@misc{junczysdowmunt2016loglinear,
  title={Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing},
  author={Junczys-Dowmunt et al. (2016)},
  year={2016},
  note={arXiv:1605.04800}
}
```

- arXiv: 1605.04800