Chatbot Arena Eval

---
name: chatbot-arena-eval
description: Evaluates large language models by collecting human preference votes on pairwise responses to real-world prompts, then ranks them using Bradley-Terry models to measure alignment and real-world utility. Use when the user wants to benchmark on Chatbot Arena, or asks about evaluating this task. Reports BT coefficients.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.04132
  bibtex_key: chiang2024chatbotarena
  confidence: high
---

# chatbot-arena-eval

> Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference — Chiang et al. (2024) (arXiv:2403.04132, 2024)

## What this evaluates

Evaluates large language models by collecting human preference votes on pairwise responses to real-world prompts, then ranks them using Bradley-Terry models to measure alignment and real-world utility.

## Datasets

- **Chatbot Arena** — total 213576; splits: all (213576)

## Metrics

- `BT coefficients` **(primary)** — range: other
  - Models pairwise win probability as $P(A \succ B) = \frac{e^{\theta_A}}{e^{\theta_A} + e^{\theta_B}}$. Scores $\theta$ are estimated via maximum likelihood on collected votes, with confidence intervals computed using CLT or multiplicity correction.

## Input / output format

**Input**: Pairwise model responses to a single user prompt.

**Output**: Human preference vote indicating which response is better (or a tie).

## Scoring recipe

```python
# Estimate BT scores via MLE on pairwise votes
scores = maximize_likelihood(votes, model=BradleyTerry)
# Compute confidence intervals for ranking stability
ci_lower, ci_upper = compute_clt_interval(scores, alpha=0.05)
# Rank models by estimated score
ranking = sort_models_by(scores, descending=True)
# Evaluate active sampling efficiency
sample_efficiency = compare_interval_widths(random_vs_adaptive_samples)
```

## Common pitfalls

- Multiplicity correction for confidence intervals is technically required for formal ranking but makes intervals wider and more conservative, potentially altering rankings.
- Active sampling sample-efficiency gains are relative to a random baseline and depend heavily on the target precision threshold (e.g., 0.2 vs 0.3).
- Anomalous user detection thresholds (alpha) involve a direct trade-off between true positive and true negative rates.

## Evidence (verbatim from paper)

> For this experiment, we ran a replay of $T\=213,576$ historical votes from our online platform and calculate the BT coefficients using our earlier-described estimation algorithm with confidence intervals; see Figure 5 for these intervals (with and without multiplicity correction; the formal notion of approximate ranking technically requires multiplicity correction, but it makes the intervals looser).

## Citation

```bibtex
@misc{chiang2024chatbotarena,
  title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference},
  author={Chiang et al. (2024)},
  year={2024},
  note={arXiv:2403.04132}
}
```

- arXiv: 2403.04132