Soft Pairwise Accuracy

---
name: soft-pairwise-accuracy
description: Evaluates the reliability and discriminative power of automatic machine translation metrics by comparing their statistical significance against human MQM judgments. It measures how well a metric's pairwise system rankings align with human preferences using permutation-based p-values rather than hard binary decisions. Use when the user has predictions and gold and needs to compute Soft Pairwise Accuracy (SPA).
metadata:
  skill_kind: metric
  source_arxiv: 2409.09598
  bibtex_key: thompson2024softpairwiseaccuracy
  confidence: high
---

# soft-pairwise-accuracy

> Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy — Thompson et al. (2024) (arXiv:2409.09598, 2024)

## What this evaluates

Evaluates the reliability and discriminative power of automatic machine translation metrics by comparing their statistical significance against human MQM judgments. It measures how well a metric's pairwise system rankings align with human preferences using permutation-based p-values rather than hard binary decisions.

## Datasets

- **WMT Metrics Shared Task 2022/2023** — total ?; splits: test (-1); repo https://github.com/google-research/mt-metrics-eval

## Metrics

- `Soft Pairwise Accuracy (SPA)` **(primary)** — range: [0, 1]
  - Aggregates continuous confidence levels (p-values) from paired permutation tests between system-level metric scores and human MQM judgments, rather than binarizing pairwise comparisons.
- `Pairwise Accuracy (PA)` — range: [0, 1]
  - Binarizes pairwise comparisons between system-level metric scores and human judgments, discarding uncertainty information.
- `Pearson r` — range: [-1, 1]
  - Measures ranking stability by computing the correlation between a metric's ranking on a subset of systems and its ranking on all systems.

## Input / output format

**Input**: System-level scores for multiple MT systems on a test set, alongside human MQM quality judgments for the same systems.

**Output**: p-values for pairwise comparisons between systems, aggregated into SPA or PA scores, and ranking stability measured as change in Pearson r.

## Scoring recipe

```python
# 1. Compute system-level scores by averaging segment-level scores for each MT system.
# 2. For each pair of systems (A, B):
#    a. Run paired permutation test (1000 random permutations) on scores vs human judgments.
#    b. Extract p-value.
#    c. For PA: binarize p-value (e.g., p < 0.05 → 1, else 0).
#    d. For SPA: retain continuous p-value as confidence weight.
# 3. Aggregate pairwise results to compute final SPA/PA score.
# 4. Compute ranking stability: 1 - |Pearson_r(subset_ranking) - Pearson_r(full_ranking)|.
```

## Common pitfalls

- Averaging segment-level scores to compute system-level metrics assumes linear behavior, which is false for metrics like BLEU and chrF.
- Caching permutations across system pairs breaks independence between p-value computations, though the authors argue it is inconsequential for SPA's confidence estimation.
- Using only 1000 permutations introduces slight randomness in SPA values across different runs.

## Evidence (verbatim from paper)

> Values are averaged over 1000 random trials. We find SPA to be more stable than PA in all cases.

## Citation

```bibtex
@misc{thompson2024softpairwiseaccuracy,
  title={Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy},
  author={Thompson et al. (2024)},
  year={2024},
  note={arXiv:2409.09598}
}
```

- arXiv: 2409.09598