Pairjudge Rm Eval

---
name: pairjudge-rm-eval
description: Evaluates whether a reward model can correctly select the right solution from a set of N generated candidates for mathematical reasoning problems. It probes the model's ability to perform pairwise correctness judgments and rank solutions without relying on arbitrary scalar scores. Use when the user wants to benchmark on MATH-500, Olympiad Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.13007
  bibtex_key: liu2025pairjudge
  confidence: high
---

# pairjudge-rm-eval

> PairJudge RM: Perform Best-of-N Sampling with Knockout Tournament — Liu et al. (2025) (arXiv:2501.13007, 2025)

## What this evaluates

Evaluates whether a reward model can correctly select the right solution from a set of N generated candidates for mathematical reasoning problems. It probes the model's ability to perform pairwise correctness judgments and rank solutions without relying on arbitrary scalar scores.

## Datasets

- **MATH-500** — total ?; splits: test (-1)
- **Olympiad Bench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Pass@N accuracy is calculated as the fraction of problems where the solution selected by the Best-of-N sampling process matches the ground-truth answer. Accuracy = (number of correctly solved problems) / (total number of problems).

## Input / output format

**Input**: A mathematical problem statement and a set of N candidate solutions (or pairs of solutions for pairwise judgment) generated by a base LLM.

**Output**: A binary correctness judgment (True/False) for each candidate solution, derived via chain-of-thought reasoning.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for prob in predictions:
        candidates = prob['candidates']
        # Select best candidate via reward model scores or knockout tournament
        best = select_best(candidates)
        if best['answer'] == gold['answer']:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Confusing pass@N accuracy with standard pass@1; the metric measures selection success across N candidates, not single-generation accuracy.
- Assuming the reward model outputs a continuous scalar score; PairJudge RM uses a pairwise knockout tournament with binary correctness judgments via chain-of-thought.
- Failing to replicate the exact N values (16, 32, 64) used in the table, which significantly impact the reported accuracy.

## Evidence (verbatim from paper)

> The results are reported in terms of accuracy. The pass@1 accuracy of these three LLMs are 42.0, 73.6, and 59.2 on MATH-500, and 12.3, 35.7, and 25.9 on Olympiad Bench, respectively. @16, @32, and @64 denote the accuracy with Best-of-16, Best-of-32, and Best-of-64 sampling, respectively.

## Citation

```bibtex
@misc{liu2025pairjudge,
  title={PairJudge RM: Perform Best-of-N Sampling with Knockout Tournament},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2501.13007}
}
```

- arXiv: 2501.13007