Comparative Evaluation

---
name: comparative-evaluation
description: A/B testing, side-by-side comparison, and preference ranking for AI outputs.
---
# Comparative Evaluation
Absolute quality scores are useful but limited. Comparative evaluation — putting outputs side by side and asking which is better — often reveals quality differences that rubrics miss.
## Comparison Methods
- **A/B testing**: Show different users different versions and compare outcomes
- **Side-by-side evaluation**: Show evaluators two outputs for the same input and ask which is better
- **Preference ranking**: Show evaluators multiple outputs and rank them from best to worst
- **Paired comparison**: Compare every pair of options to build a complete ranking
- **Elo rating**: Use tournament-style comparisons to develop continuous quality scores
## Designing A/B Tests for AI
A/B testing AI is different from A/B testing UI:
- **Variance is high**: The same prompt can produce different outputs, so you need more samples
- **Context matters**: The same change might help for one task and hurt for another
- **Metrics lag**: AI quality changes may take time to show up in user behavior
- **Interaction effects**: A change to one part of the conversation affects all subsequent parts
Design A/B tests with:
- Sufficient sample sizes to account for output variance
- Segmentation by task type and user experience level
- Multiple metrics (don't optimise for one at the expense of others)
- Guardrails to catch severe quality regressions quickly
## Side-by-Side Evaluation Design
For human evaluation of AI outputs:
- **Blind evaluation**: Evaluators shouldn't know which version is which
- **Consistent inputs**: Compare outputs generated from the same input
- **Structured criteria**: Give evaluators specific dimensions to compare on, not just "which is better"
- **Multiple evaluators**: Use at least 3 evaluators per comparison for reliability
- **Diverse inputs**: Test across a representative sample of real user inputs
## When to Use Comparative vs. Absolute Evaluation
- **Comparative**: Best for choosing between alternatives, detecting subtle quality differences, and model selection
- **Absolute**: Best for measuring against a standard, tracking progress over time, and certification
## Design Artefacts
- A/B test design templates
- Side-by-side evaluation protocols
- Evaluator instructions and rubrics
- Sample size calculators for AI experiments
- Comparison result analysis frameworks