Judgebench Eval

---
name: judgebench-eval
description: Probes the factual and logical reliability of LLM-based judges and reward models by testing their ability to distinguish between objectively correct responses and subtly flawed ones across knowledge, reasoning, math, and coding domains. Use when the user wants to benchmark on JudgeBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.12784
  bibtex_key: tan2024judgebench
  confidence: high
---

# judgebench-eval

> JudgeBench: A Benchmark for Evaluating LLM-based Judges — Tan et al. (2024) (arXiv:2410.12784, 2024)

## What this evaluates

Probes the factual and logical reliability of LLM-based judges and reward models by testing their ability to distinguish between objectively correct responses and subtly flawed ones across knowledge, reasoning, math, and coding domains.

## Datasets

- **JudgeBench** — total ?; splits: Knowledge (-1), Reasoning (-1), Math (-1), Coding (-1); repo https://github.com/ScalerLab/JudgeBench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of response pairs where the aggregated two-trial verdict matches the objectively correct response. Ties are aggregated with the majority vote; inconsistent verdicts or double-ties are counted as incorrect.

## Input / output format

**Input**: A prompt template (e.g., Vanilla or Arena-Hard) containing a question and two candidate responses (Response A and Response B).

**Output**: A categorical verdict: 'A > B', 'A < B', or 'A = B' (tie).

## Scoring recipe

```python
correct = 0
for pair in dataset:
    v1 = judge(pair.A, pair.B)
    v2 = judge(pair.B, pair.A)
    if (v1 in ['A>B', 'A=B'] and v2 in ['A>B', 'A=B']):
        agg = 'A>B'
    elif (v1 in ['A<B', 'A=B'] and v2 in ['A<B', 'A=B']):
        agg = 'A<B'
    else:
        agg = 'incorrect'
    if (agg == 'A>B' and pair.gold == 'A') or (agg == 'A<B' and pair.gold == 'B'):
        correct += 1
return (correct / len(dataset)) * 100
```

## Common pitfalls

- Evaluating judges on a single response order, which introduces positional bias and distorts accuracy.
- Counting ties as correct or discarding them, whereas the protocol explicitly aggregates ties with the majority vote or marks inconsistent/double-tie outcomes as incorrect.
- Assuming high accuracy implies robust reasoning, as the benchmark is specifically constructed to be near-random for current models, exposing gaps in factual/logical verification.

## Evidence (verbatim from paper)

> To mitigate this, we evaluate the LLM-based judge twice, swapping the order of the response pairs in the second trial. ... if both trials yield  $A > B$  or one trial gives  $A > B$  and the other  $A = B$ , we consider the aggregate decision to be  $A > B$ . Inconsistent decisions (e.g., $A > B$ in one trial, $A < B$ in the other) or ties in both trials are deemed incorrect, as they indicate the judge is either guessing or unable to reliably distinguish between responses. ... Even a strong model like GPT-4o struggles, achieving accuracy no better than random guessing when using the vanilla prompt.

## Citation

```bibtex
@misc{tan2024judgebench,
  title={JudgeBench: A Benchmark for Evaluating LLM-based Judges},
  author={Tan et al. (2024)},
  year={2024},
  note={arXiv:2410.12784}
}
```

- arXiv: 2410.12784