Mt Orthographic Robustness Eval

---
name: mt-orthographic-robustness-eval
description: Evaluates the robustness of neural machine translation systems to orthographic and interpunctual noise by measuring translation quality and output consistency on perturbed inputs. Use when the user wants to benchmark on Baltic MT test sets (ET-EN, LV-EN, LT-EN), or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2009.05460
  bibtex_key: bergmanis2020robust
  confidence: high
---

# mt-orthographic-robustness-eval

> Robust Neural Machine Translation: Modeling Orthographic and Interpunctual Variation — Bergmanis et al. (2020) (arXiv:2009.05460, 2020)

## What this evaluates

Evaluates the robustness of neural machine translation systems to orthographic and interpunctual noise by measuring translation quality and output consistency on perturbed inputs.

## Datasets

- **Baltic MT test sets (ET-EN, LV-EN, LT-EN)** — total ?; splits: dev (-1), test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram precision metric for machine translation, reported in percentage points. Higher values indicate better translation quality.
- `10NT-TER` — range: [0, 1]
  - Translation Edit Rate variant without normalization, using a 10-gram window to measure output consistency under input noise. Lower values indicate higher robustness and noise invariance.

## Input / output format

**Input**: Source sentence in Latvian, Estonian, or Lithuanian, optionally perturbed with orthographic/interpunctual noise (e.g., letter insertion, deletion, punctuation changes).

**Output**: Target English sentence.

## Scoring recipe

```python
bleu = sacrebleu.corpus_bleu(predictions, references)
ter = compute_10nt_ter(predictions, references) # Lower is better
# Statistical significance tested via bootstrap resampling (p < 0.05)
```

## Common pitfalls

- BLEU scores naturally drop on noisy data; the paper emphasizes 10NT-TER to specifically measure noise invariance/consistency.
- Adversarial training uses a strict 1:1 augmentation ratio, which may not reflect standard data scaling practices.
- Performance gains on clean data are small (+0.5 BLEU) and require bootstrap resampling to confirm statistical significance.

## Evidence (verbatim from paper)

> Results (see Table 6) of our experiments show that using adversarial examples in training improves the robustness and noise invariance of the MT systems measured in 10NT-TER (see Section 2) on average by 0.1 10NT-TER points or in relative terms an average consistency improvement of about 50%.

## Citation

```bibtex
@misc{bergmanis2020robust,
  title={Robust Neural Machine Translation: Modeling Orthographic and Interpunctual Variation},
  author={Bergmanis et al. (2020)},
  year={2020},
  note={arXiv:2009.05460}
}
```

- arXiv: 2009.05460