Wmt20 Zh En Eval

---
name: wmt20-zh-en-eval
description: Evaluates machine translation quality by scoring system-generated English translations against human reference sentences and expert MQM scores. It measures how well automatic metrics correlate with human judgments across different translation systems. Use when the user wants to benchmark on WMT20 ZH-EN, or asks about evaluating this task. Reports COMET.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.04139
  bibtex_key: first2021billboard
  confidence: high
---

# wmt20-zh-en-eval

> Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand — First Author et al. (2021) (arXiv:2112.04139, 2021)

## What this evaluates

Evaluates machine translation quality by scoring system-generated English translations against human reference sentences and expert MQM scores. It measures how well automatic metrics correlate with human judgments across different translation systems.

## Datasets

- **WMT20 ZH-EN** — total ?; splits: test (-1)

## Metrics

- `COMET` **(primary)** — range: [0, 1]
  - A neural reference-based metric trained to predict human judgments. Scores are typically normalized to [0, 1].
- `COMET-QE` — range: [0, 1]
  - A referenceless variant of COMET that uses only the source sentence and hypothesis to predict quality.
- `BLEURT` — range: [0, 1]
  - A BERT-based metric fine-tuned on human judgments specifically for in-English translation evaluation.
- `Prism-ref` — range: [0, 1]
  - A probabilistic reference-based metric that scores translations using a language model over the reference.

## Input / output format

**Input**: Source sentence in Chinese and reference sentence(s) (Human-A and Human-B).

**Output**: Generated English translation sentence.

## Scoring recipe

```python
scores = []
for src, ref, hyp in dataset:
    metric_score = compute_metric(hyp, ref)
    human_score = average(expert_MQM_scores[hyp])
    scores.append((metric_score, human_score))
return pearson_correlation([s[0] for s in scores], [s[1] for s in scores])
```

## Common pitfalls

- Using only a single human reference instead of all available references (Human-A and Human-B) reduces correlation with human judgments.
- Failing to detokenize model outputs before scoring can artificially lower reference-based metric scores.

## Evidence (verbatim from paper)

> Table 1: Summary of Billboards as of Jan. 10, 2022. ... Top Metric: COMET ... Ensemble of Metrics: 1.72·COMET-QE+1.48·COMET+1.21·BLEURT ... We use all human translations available as a reference set for reference-based metrics. Concretely, every test instance in WMT20 ZH-EN has two translations provided by different human translation services: Human-A and Human-B... Each output sentence is evaluated by three professional translators. Following Freitag et al. (2021), the three scores are averaged to get an instance-level score.

## Citation

```bibtex
@misc{first2021billboard,
  title={Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand},
  author={First Author et al. (2021)},
  year={2021},
  note={arXiv:2112.04139}
}
```

- arXiv: 2112.04139