Wmt20 En De Eval

---
name: wmt20-en-de-eval
description: Evaluates machine translation quality by scoring system-generated German translations against human reference sentences and expert MQM scores. It measures how well automatic metrics correlate with human judgments across different translation systems. Use when the user wants to benchmark on WMT20 EN-DE, or asks about evaluating this task. Reports COMET.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.04139
  bibtex_key: first2021billboard
  confidence: high
---

# wmt20-en-de-eval

> Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand — First Author et al. (2021) (arXiv:2112.04139, 2021)

## What this evaluates

Evaluates machine translation quality by scoring system-generated German translations against human reference sentences and expert MQM scores. It measures how well automatic metrics correlate with human judgments across different translation systems.

## Datasets

- **WMT20 EN-DE** — total ?; splits: test (-1)

## Metrics

- `COMET` **(primary)** — range: [0, 1]
  - A neural reference-based metric trained to predict human judgments. Scores are typically normalized to [0, 1].
- `COMET-QE` — range: [0, 1]
  - A referenceless variant of COMET that uses only the source sentence and hypothesis to predict quality.
- `Prism-ref` — range: [0, 1]
  - A probabilistic reference-based metric that scores translations using a language model over the reference.

## Input / output format

**Input**: Source sentence in English and reference sentence(s) (Human-A, Human-B, and Human-P).

**Output**: Generated German translation sentence.

## Scoring recipe

```python
scores = []
for src, ref, hyp in dataset:
    metric_score = compute_metric(hyp, ref)
    human_score = average(expert_MQM_scores[hyp])
    scores.append((metric_score, human_score))
return pearson_correlation([s[0] for s in scores], [s[1] for s in scores])
```

## Common pitfalls

- Ignoring the Human-P paraphrased reference, which mitigates translationese effects and diversifies the reference set.
- Not accounting for the arbitrary distinction between Human-A and Human-B when swapping roles for mixed-effects analysis.

## Evidence (verbatim from paper)

> Table 1: Summary of Billboards as of Jan. 10, 2022. ... Top Metric: COMET ... Ensemble of Metrics: 1.19·COMET+0.36·COMET-QE+0.02·Prism-ref ... In addition to Human-A and Human-B, WMT20 EN-DE provides a translation that is created by linguists who are asked to paraphrase Human-A and Human-B as much as possible (Human-P)... These paraphrased translations are shown to increase correlations with human judgments by mitigating the translationese effect and diversifying the reference when the generation quality is measured by reference-based metrics

## Citation

```bibtex
@misc{first2021billboard,
  title={Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand},
  author={First Author et al. (2021)},
  year={2021},
  note={arXiv:2112.04139}
}
```

- arXiv: 2112.04139