Wmt18 News Eval

---
name: wmt18-news-eval
description: Evaluates the effectiveness of automatic web data selection for domain-specific machine translation in the news domain. It probes whether a document-level topic classifier can filter noisy parallel data to improve MT system performance compared to state-of-the-art baselines on the WMT-18 benchmark. Use when the user wants to benchmark on WMT-18 News Shared Task, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.10243
  bibtex_key: vu2021mtcustomization
  confidence: high
---

# wmt18-news-eval

> Machine Translation Customization via Automatic Training Data Selection from the Web — Vu et al. (2021) (arXiv:2102.10243, 2021)

## What this evaluates

Evaluates the effectiveness of automatic web data selection for domain-specific machine translation in the news domain. It probes whether a document-level topic classifier can filter noisy parallel data to improve MT system performance compared to state-of-the-art baselines on the WMT-18 benchmark.

## Datasets

- **WMT-18 News Shared Task** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram overlap metric between system translations and reference translations, computed using the sacrebleu toolkit with default WMT settings (case-insensitive, tokenized with 13a).
- `Accuracy` — range: percent
  - Proportion of correctly classified document batches (news vs. non-news) by the topic classifier.
- `Average z-score` — range: other
  - Human evaluation metric where individual assessor scores are standardized by their mean and standard deviation, then averaged across assessors for each system.

## Input / output format

**Input**: Source language sentences or document batches (English or German) provided as parallel text pairs for translation.

**Output**: Target language translations (German or English) generated by the MT model.

## Scoring recipe

```python
# Compute BLEU using sacrebleu as specified in the paper
import sacrebleu

def compute_bleu(predictions, references):
    # predictions: list of system translations (str)
    # references: list of reference translations (list of str or list of list of str)
    score = sacrebleu.corpus_bleu(predictions, [references])
    return score.score

# Compute Accuracy for classifier
# predictions: list of predicted labels (0/1)
# gold: list of true labels (0/1)
accuracy = sum(p == g for p, g in zip(predictions, gold)) / len(gold)

# Compute Average z-score for human eval
# scores: list of raw assessor scores per system
# mean, std: assessor-level mean and std
z_scores = [(s - mean) / std for s in scores]
avg_z = sum(z_scores) / len(z_scores)
```

## Common pitfalls

- Using sentence-level classification instead of document/batch-level classification significantly degrades classifier accuracy and data selection quality.
- Computing BLEU without sacrebleu or with different tokenization standards leads to non-comparable scores against the reported WMT-18 baselines.
- Comparing model performance without accounting for differences in training data volume, model architecture (e.g., Transformer-Base vs Big), or use of back-translation.

## Evidence (verbatim from paper)

> We evaluated such models against the standard WMT-2017 and WMT-2018 test sets, using BLEU as our evaluation metric. The results are reported in Table 3, under the column Buckets, using the evaluation tool, sacrebleu [17].

## Citation

```bibtex
@misc{vu2021mtcustomization,
  title={Machine Translation Customization via Automatic Training Data Selection from the Web},
  author={Vu et al. (2021)},
  year={2021},
  note={arXiv:2102.10243}
}
```

- arXiv: 2102.10243