Iwslt2017 Nmt Eval

---
name: iwslt2017-nmt-eval
description: Evaluates neural machine translation quality of character-level versus subword models across multiple language pairs. It probes morphological generalization, noise robustness, and the impact of sequence length expansion on training and inference efficiency. Use when the user wants to benchmark on IWSLT 2017, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.08191
  bibtex_key: libovicky2021charnmt
  confidence: high
---

# iwslt2017-nmt-eval

> Why don't people use character-level machine translation? — Libovický et al. (2021) (arXiv:2110.08191, 2021)

## What this evaluates

Evaluates neural machine translation quality of character-level versus subword models across multiple language pairs. It probes morphological generalization, noise robustness, and the impact of sequence length expansion on training and inference efficiency.

## Datasets

- **IWSLT 2017** — total ?; splits: train (200000), test (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Standard n-gram precision with brevity penalty, typically computed with SacreBLEU.
- `chrF` — range: [0, 100]
  - Character n-gram F-score measuring overlap between reference and hypothesis at the character level.
- `COMET` — range: [-1, 1]
  - Neural metric scoring translation quality based on contextual embeddings of source, hypothesis, and reference.

## Input / output format

**Input**: Source sentence in English paired with target language (German, French, or Arabic). Subword models use Moses tokenizer + BPE (16k merges); character models use raw UTF-8 characters (vocabulary limited to 300).

**Output**: Translated sentence in the target language.

## Scoring recipe

```python
def compute_metrics(predictions, references, sources=None):
    bleu = sacrebleu.corpus_bleu(predictions, [references]).score
    chrf = sacrebleu.corpus_chrf(predictions, [references]).score
    comet = mean(comet_model.predict({'source': s, 'target': p, 'reference': r}) 
                 for s, p, r in zip(sources, predictions, references)) if sources else None
    return {'BLEU': bleu, 'chrF': chrf, 'COMET': comet}
```

## Common pitfalls

- Beam search on subword models requires length normalization to prevent quality degradation, whereas character models are more resilient to beam size changes.
- Minimum Bayes Risk (MBR) decoding performs poorly on character-level models due to exposure bias and lower perplexity distributions, making beam search superior.
- Increasing the downsampling rate from 3 to 5 consistently degrades translation quality across all character processing architectures.

## Evidence (verbatim from paper)

> We evaluate the translation quality using the BLEU score (Papineni et al., 2002), the chrF score (Popovic, 2015) (as implemented in SacreBLEU; Post, 2018), and the COMET score (Rei et al., 2020). We run each experiment 4 times and report the mean value and standard deviation.

## Citation

```bibtex
@misc{libovicky2021charnmt,
  title={Why don't people use character-level machine translation?},
  author={Libovický et al. (2021)},
  year={2021},
  note={arXiv:2110.08191}
}
```

- arXiv: 2110.08191