Bradley Terry

---
name: bradley-terry
description: Evaluates the stability and reliability of global pointwise scores (accuracy, AUC, F1) versus pairwise Bradley-Terry rankings for ordering NLP models across classification and text generation tasks. Use when the user has predictions and gold and needs to compute Bradley-Terry.
metadata:
  skill_kind: metric
  source_arxiv: 2507.01633
  bibtex_key: levtsov2025confidence
  confidence: high
---

# bradley-terry

> Confidence and Stability of Global and Pairwise Scores in NLP Evaluation — Levtsov et al. (2025) (arXiv:2507.01633, 2025)

## What this evaluates

Evaluates the stability and reliability of global pointwise scores (accuracy, AUC, F1) versus pairwise Bradley-Terry rankings for ordering NLP models across classification and text generation tasks.

## Datasets

- **Jigsaw** — total ?; splits: test (-1)
- **SST-5** — total ?; splits: test (-1)
- **CEval** — total ?; splits: test (-1); repo https://github.com/aix-group/CEval-Counterfactual-Generation-Benchmark

## Metrics

- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified instances out of the total test set.
- `AUC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between classes.
- `F1` — range: [0, 1]
  - Harmonic mean of precision and recall, balancing false positives and false negatives.
- `Bradley-Terry` **(primary)** — range: other
  - A pairwise comparison model that estimates latent skill parameters for each model by maximizing the likelihood of observed pairwise preferences. Rankings are derived from these estimated parameters.
- `Bradley-Terry binary` — range: other
  - Binary variant of the Bradley-Terry model applied to pairwise comparisons.
- `Spearman correlation` — range: [-1, 1]
  - Rank-based correlation coefficient used to measure the monotonic relationship between different scoring methods' model rankings.

## Input / output format

**Input**: Model predictions (decision-function values or probabilities) for classification tasks; generated text outputs for CEval. Ground-truth labels or majority-vote reconstructed labels are used for global scoring.

**Output**: Global score values per model (Accuracy, AUC, F1) and a ranked list of models derived from the Bradley-Terry pairwise comparison procedure.

## Scoring recipe

```python
def compute_global_scores(predictions, gold):
    acc = mean(predictions == gold)
    auc = compute_auc(predictions, gold)
    f1 = compute_f1(predictions, gold)
    return acc, auc, f1

def compute_bt_ranking(models, test_instances, m):
    n_samples = min(len(test_instances), 12 * m * log(m))
    sampled = sample_with_replacement(test_instances, n_samples)
    bt_scores = fit_bradley_terry(sampled, models)
    return sort_models_by(bt_scores, descending=True)
```

## Common pitfalls

- Using all available test instances for pairwise comparisons instead of the recommended 12m log(m) sampling, which increases computational cost and may reduce ranking stability due to frequent ties.
- Reconstructing ground-truth labels via majority vote when official test labels are unavailable (as in Jigsaw), which can introduce bias and mask true model performance.
- Directly comparing global score magnitudes with pairwise Bradley-Terry rankings without accounting for their different scales, convergence rates, and sensitivity to rare high-impact errors.

## Evidence (verbatim from paper)

> For each test instance, we compared the outputs of $m$ different models in a pairwise fashion, yielding $\binom{m}{2}$ model pairs. For each pair, we then drew $12m\log(m)$ comparisons at random with replacement, or else used all available test instances if their count was smaller. Finally, we applied these sampled comparisons to build a Bradley–Terry ranking of the models.

## Citation

```bibtex
@misc{levtsov2025confidence,
  title={Confidence and Stability of Global and Pairwise Scores in NLP Evaluation},
  author={Levtsov et al. (2025)},
  year={2025},
  note={arXiv:2507.01633}
}
```

- arXiv: 2507.01633