Wqe Metric Eval

---
name: wqe-metric-eval
description: Evaluates the ability of unsupervised and supervised metrics to identify word-level translation errors by comparing their continuous scores against human-annotated error spans and multi-annotator agreement rates. Use when the user has predictions and gold and needs to compute Average Precision (AP).
metadata:
  skill_kind: metric
  source_arxiv: 2505.23183
  bibtex_key: sarti2025unsupervised
  confidence: high
---

# wqe-metric-eval

> Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement — Sarti et al. (2025) (arXiv:2505.23183, 2025)

## What this evaluates

Evaluates the ability of unsupervised and supervised metrics to identify word-level translation errors by comparing their continuous scores against human-annotated error spans and multi-annotator agreement rates.

## Datasets

- **QE4PE** — total ?; splits: test (-1)
- **DivEMT** — total ?; splits: test (-1)

## Metrics

- `Average Precision (AP)` **(primary)** — range: [0, 1]
  - Area under the precision-recall curve computed over continuous metric scores against binary human error labels. Measures overall ranking quality across all thresholds.
- `F1*` — range: [0, 1]
  - The maximum F1 score obtained by iterating over all possible thresholds to binarize continuous metric scores into positive/negative labels matching human annotations.
- `Spearman correlation` — range: [-1, 1]
  - Rank correlation between continuous WQE metric scores and human edit counts (number of annotators marking a token as an error) across multiple annotation sets.

## Input / output format

**Input**: Continuous scores from WQE metrics for each token in a machine-translated sentence, paired with human-annotated binary error labels or multi-annotator edit counts.

**Output**: Binary classification (error/non-error) or continuous score, depending on the metric evaluation phase.

## Scoring recipe

```python
def compute_metrics(scores, gold_binary, edit_counts=None):
    ap = average_precision_score(gold_binary, scores)
    f1_star = max(f1_score(gold_binary, scores > t) for t in np.linspace(0, 1, 1000))
    if edit_counts is not None:
        spearman_rho = spearmanr(scores, edit_counts).correlation
    return ap, f1_star, spearman_rho
```

## Common pitfalls

- Random baseline AP values match the proportion of tokens marked as errors, which can vary greatly across datasets and languages.
- Default xcomet metrics show high precision but very low recall (32-26%), which is problematic for WQE applications where omitting errors causes oversights.
- Metric rankings are highly unstable when evaluated against a single annotator's labels due to subjective variability.

## Evidence (verbatim from paper)

> We report Average Precision (AP) as a general measure of metric quality across the full score range, and we estimate calibrated metric performance as the best F1 score (F1∗) across all thresholds for binarizing continuous metric scores into pos./neg. labels matching human annotation.

## Citation

```bibtex
@misc{sarti2025unsupervised,
  title={Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement},
  author={Sarti et al. (2025)},
  year={2025},
  note={arXiv:2505.23183}
}
```

- arXiv: 2505.23183