Reward Model Benchmarking Eval

---
name: reward-model-benchmarking-eval
description: Evaluates reward models on their ability to correctly rank or score LLM-generated responses across diverse domains like safety, mathematics, coding, and instruction following. It probes pairwise preference accuracy, robustness to superficial biases, cross-sample score calibration, and consistency in assigning absolute quality scores. Use when the user wants to benchmark on RewardBench, RM-Bench, PPE, JudgeBench, or asks about evaluating this task. Reports binary choice accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.08829
  bibtex_key: peng2026wildreward
  confidence: high
---

# reward-model-benchmarking-eval

> WildReward: Learning Reward Models from In-the-Wild Human Interactions — Hao Peng et al. (2026) (arXiv:2602.08829, 2026)

## What this evaluates

Evaluates reward models on their ability to correctly rank or score LLM-generated responses across diverse domains like safety, mathematics, coding, and instruction following. It probes pairwise preference accuracy, robustness to superficial biases, cross-sample score calibration, and consistency in assigning absolute quality scores.

## Datasets

- **RewardBench** — total ?; splits: test (-1)
- **RM-Bench** — total ?; splits: easy (-1), normal (-1), hard (-1)
- **PPE** — total ?; splits: Human (-1), Correctness (-1)
- **JudgeBench** — total ?; splits: test (-1)

## Metrics

- `binary choice accuracy` **(primary)** — range: [0, 1]
  - Proportion of prompts where the model correctly selects the ground-truth chosen response over the rejected response.
- `Best-of-N accuracy` — range: [0, 1]
  - Proportion of prompts where the model correctly identifies the single best response among N candidates.
- `Expected Calibration Error (ECE)` — range: [0, 1]
  - Weighted mean of the absolute difference between predicted confidence and actual accuracy across binned predictions. Lower values indicate better calibration.
- `ROC-AUC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the probability that a positive instance is ranked higher than a negative one across the global distribution.

## Input / output format

**Input**: For pairwise benchmarks: a prompt paired with two candidate responses (chosen and rejected). For pointwise calibration/consistency: conversation history, user query, model response, and a binary feedback label (positive/negative).

**Output**: For pairwise: a binary selection indicating the preferred response. For pointwise/calibration: a continuous score or margin between chosen and rejected responses.

## Scoring recipe

```python
# Pairwise Accuracy
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = correct / len(gold_labels)

# Best-of-N Accuracy
bon_correct = sum(1 for batch in batches if max_score(batch) == gold_best_idx)
bon_accuracy = bon_correct / len(batches)

# ECE (binned)
ece = sum(bin_size * abs(bin_acc - bin_conf) for bin_size, bin_acc, bin_conf in bins)
```

## Common pitfalls

- PPE Correctness uses a Best-of-N setting rather than binary choice, so standard pairwise accuracy does not apply.
- Reward models trained with Bradley-Terry objectives often lack global score calibration; raw margins must be calibrated (e.g., via Platt scaling) before computing ECE or using scores for cross-sample filtering.
- RM-Bench splits (easy/normal/hard) test different robustness properties; reporting only the average masks performance on length/style bias or factual correctness.

## Evidence (verbatim from paper)

> We adopt the widely used Expected Calibration Error (ECE) as the metric, where a lower value indicates better calibration. WILDREWARD achieves a remarkably low ECE of 2.76% . This implies that, on average, the discrepancy between the predicted confidence and its actual accuracy is less than 3% .

## Citation

```bibtex
@misc{peng2026wildreward,
  title={WildReward: Learning Reward Models from In-the-Wild Human Interactions},
  author={Hao Peng et al. (2026)},
  year={2026},
  note={arXiv:2602.08829}
}
```

- arXiv: 2602.08829