Wmt Mt Meta Eval

---
name: wmt-mt-meta-eval
description: Evaluates machine translation metrics by measuring their alignment with human judgments at segment and system levels. It probes whether metrics can correctly rank translations and systems based on quality, and tests the robustness of meta-evaluation statistics like correlation and pairwise ranking accuracy. Use when the user wants to benchmark on WMT Test Sets, or asks about evaluating this task. Reports Pearson correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.13831
  bibtex_key: perrella2024guardians
  confidence: high
---

# wmt-mt-meta-eval

> Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In! — Perrella et al. (2024) (arXiv:2408.13831, 2024)

## What this evaluates

Evaluates machine translation metrics by measuring their alignment with human judgments at segment and system levels. It probes whether metrics can correctly rank translations and systems based on quality, and tests the robustness of meta-evaluation statistics like correlation and pairwise ranking accuracy.

## Datasets

- **WMT Test Sets** — total ?; splits: test (-1)

## Metrics

- `System-level pairwise ranking accuracy` — range: [0, 1]
  - Measures the percentage of system pairs where the metric's ranking matches the human judgment ranking.
- `Pearson correlation` **(primary)** — range: [-1, 1]
  - Measures the linear correlation between metric scores and human judgment scores. Calculated as the covariance of the scores divided by the product of their standard deviations.
- `Segment-level pairwise ranking accuracy with tie calibration` — range: [0, 1]
  - Evaluates the metric's ability to rank segments in the same order as human judgments, with adjustments for correctly predicting ties.

## Input / output format

**Input**: Source segment, machine-translated candidate, and human judgment scores (scalar) for the candidate.

**Output**: A scalar quality score assigned to each translation (segment-level) or system (system-level) by the evaluated metric.

## Scoring recipe

```python
def compute_pairwise_accuracy(metric_scores, human_scores):
    correct = 0
    total = 0
    for i in range(len(metric_scores)):
        for j in range(i + 1, len(metric_scores)):
            if human_scores[i] != human_scores[j]:
                total += 1
                if (metric_scores[i] > metric_scores[j]) == (human_scores[i] > human_scores[j]):
                    correct += 1
    return correct / total if total > 0 else 0.0

def compute_pearson(metric_scores, human_scores):
    n = len(metric_scores)
    mean_m = sum(metric_scores) / n
    mean_h = sum(human_scores) / n
    cov = sum((m - mean_m) * (h - mean_h) for m, h in zip(metric_scores, human_scores))
    std_m = (sum((m - mean_m)**2 for m in metric_scores))**0.5
    std_h = (sum((h - mean_h)**2 for h in human_scores))**0.5
    return cov / (std_m * std_h) if std_m * std_h != 0 else 0.0
```

## Common pitfalls

- Segment-level correlations must be calculated after grouping translations by source segment; otherwise, scores are artificially inflated.
- Tie calibration should not be conducted on the test set, as it introduces evaluation bias.
- Metrics optimized to mimic human judgments may exploit spurious correlations rather than learning robust translation quality.

## Evidence (verbatim from paper)

> Specifically, metrics are evaluated at two granularity levels: at the segment level, metrics assign a score to every translation, and they are ranked according to their ability to discern between higher- and lower-quality translations; at the system level, metrics assign a score to each MT system, and they are ranked according to their ability to discern between superior and inferior systems. At both granularity levels, metrics can be evaluated using several statistical methods, such as the Kendall τ and Pearson ρ correlation coefficients, which have traditionally been applied at the segment and system levels, respectively. A final metrics ranking is derived by aggregating results from all the chosen statistics. For example, at WMT23, the final ranking was computed from the following three statistics: 1. System-level pairwise ranking accuracy Kocmi et al. ([2021]), which evaluates metrics based on their ability to rank systems in the same order as human judgments. 2. System- and segment-level Pearson correlation, which measures the degree to which metric scores and human scores are correlated linearly. 3. Segment-level pairwise ranking accuracy with tie calibration Deutsch et al. (

## Citation

```bibtex
@misc{perrella2024guardians,
  title={Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!},
  author={Perrella et al. (2024)},
  year={2024},
  note={arXiv:2408.13831}
}
```

- arXiv: 2408.13831