Wmt21 Qe Eval

---
name: wmt21-qe-eval
description: Evaluates machine translation quality estimation systems by predicting human judgments on translation adequacy and fluency (Direct Assessment) and classifying translation errors (CED). It probes the model's ability to correlate predicted scores with human ratings and accurately detect translation quality issues across multiple language pairs. Use when the user wants to benchmark on WMT 2021 Quality Estimation Shared Task datasets, or asks about evaluating this task. Reports Pearson's correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.14890
  bibtex_key: wang2021qemind
  confidence: high
---

# wmt21-qe-eval

> QEMind: Alibaba's Submission to the WMT21 Quality Estimation Shared Task — Wang et al. (2021) (arXiv:2112.14890, 2021)

## What this evaluates

Evaluates machine translation quality estimation systems by predicting human judgments on translation adequacy and fluency (Direct Assessment) and classifying translation errors (CED). It probes the model's ability to correlate predicted scores with human ratings and accurately detect translation quality issues across multiple language pairs.

## Datasets

- **WMT 2021 Quality Estimation Shared Task datasets** — total ?; splits: test (-1), dev (-1)

## Metrics

- `Pearson's correlation` **(primary)** — range: [-1, 1]
  - Measures the linear correlation between predicted quality scores and z-standardized human Direct Assessment (DA) scores.
- `Matthews correlation coefficient (MCC)` — range: [-1, 1]
  - Measures the correlation between predicted and actual error classification labels, accounting for true/false positives and negatives.

## Input / output format

**Input**: Source sentence and target machine-translated sentence pair.

**Output**: Predicted quality score (for DA) or error classification label (for CED).

## Scoring recipe

```python
def pearson(preds, gold):
    n = len(preds)
    mp, mg = sum(preds)/n, sum(gold)/n
    cov = sum((p-mp)*(g-mg) for p,g in zip(preds, gold))
    sp = (sum((p-mp)**2 for p in preds)/n)**0.5
    sg = (sum((g-mg)**2 for g in gold)/n)**0.5
    return cov/(sp*sg) if sp*sg > 0 else 0.0

def mcc(preds, gold):
    tp = sum(1 for p,g in zip(preds,gold) if p==1 and g==1)
    fp = sum(1 for p,g in zip(preds,gold) if p==1 and g==0)
    fn = sum(1 for p,g in zip(preds,gold) if p==0 and g==1)
    tn = sum(1 for p,g in zip(preds,gold) if p==0 and g==0)
    num = tp*tn - fp*fn
    den = ((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))**0.5
    return num/den if den > 0 else 0.0
```

## Common pitfalls

- Forgetting to z-standardize human DA scores before computing Pearson correlation, as specified in the protocol.
- Mixing up the regression task (DA) with the classification task (CED), which require different metrics and evaluation protocols.
- Using WMT 2020 test sets for final model selection instead of the blind WMT 2021 test sets specified for the shared task submission.

## Evidence (verbatim from paper)

> The Pearson's correlations between our model's predictions and the human DA judges (z-standardized mean DA score) are shown in Table 1. Brief results of Matthews correlations (MCC) on development sets are shown in Table 4.

## Citation

```bibtex
@misc{wang2021qemind,
  title={QEMind: Alibaba's Submission to the WMT21 Quality Estimation Shared Task},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2112.14890}
}
```

- arXiv: 2112.14890