Wmt2023 Qe Eval

---
name: wmt2023-qe-eval
description: Evaluates machine translation quality estimation across sentence-level scoring, word-level error detection, and fine-grained error span classification. It probes a model's ability to predict translation quality and localize specific errors without relying on human references during inference. Use when the user wants to benchmark on WMT2022 QE EN-DE dataset, WMT2022 Metric EN-DE dataset, WMT17/19/20 Post-editing EN-DE datasets, or asks about evaluating this task. Reports MCC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.13230
  bibtex_key: geng2023unify
  confidence: high
---

# wmt2023-qe-eval

> Unify word-level and span-level tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task — Geng et al. (2023) (arXiv:2309.13230, 2023)

## What this evaluates

Evaluates machine translation quality estimation across sentence-level scoring, word-level error detection, and fine-grained error span classification. It probes a model's ability to predict translation quality and localize specific errors without relying on human references during inference.

## Datasets

- **WMT2022 QE EN-DE dataset** — total ?; splits: train (-1), val (-1)
- **WMT2022 Metric EN-DE dataset** — total ?; splits: train (-1), val (-1)
- **WMT17/19/20 Post-editing EN-DE datasets** — total ?; splits: train (-1)

## Metrics

- `MCC` **(primary)** — range: [-1, 1]
  - Matthews Correlation Coefficient measures the quality of binary classifications. Formula: (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall for error detection. Formula: 2 * (precision * recall) / (precision + recall).
- `Spearman score` — range: [-1, 1]
  - Spearman's rank correlation coefficient measures the monotonic relationship between predicted quality scores and human judgments.

## Input / output format

**Input**: Source sentence and machine-translated target sentence (hypothesis).

**Output**: Sentence-level quality score, word-level error tags (e.g., GOOD/BAD), and fine-grained error spans with severity classification.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 0)
    mcc = (tp*tn - fp*fn) / ((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))**0.5
    f1 = 2 * tp / (2*tp + fp + fn) if (2*tp + fp + fn) > 0 else 0
    spearman = scipy.stats.spearmanr(predictions, gold).correlation
    return {'MCC': mcc, 'F1': f1, 'Spearman': spearman}
```

## Common pitfalls

- MCC and F1 are reported for both word-level and span-level tasks, but the paper notes they are computed after a post-processing pipeline converts word-level 'BAD' tags into spans.
- Spearman correlation is used for sentence-level ranking, which is sensitive to score calibration and may not reflect absolute quality differences.
- The evaluation relies on the WMT2023 QE Shared Task test set, which is not publicly released for independent benchmarking.

## Evidence (verbatim from paper)

> We achieve the best results on EN-DE for both word-level and fine-grained error span detection sub-tasks with an MCC of 29.7 (+4.1 than the second best system) and F1 score of 28.4 (+1.1) respectively. We rank 2nd place on sentence-level sub-tasks with a Spearman score of 47.9 (-0.4 than the best system).

## Citation

```bibtex
@misc{geng2023unify,
  title={Unify word-level and span-level tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task},
  author={Geng et al. (2023)},
  year={2023},
  note={arXiv:2309.13230}
}
```

- arXiv: 2309.13230