Relevance Judgment Eval

---
name: relevance-judgment-eval
description: Evaluates whether pointwise re-rankers can function as binary relevance judges by predicting whether a document is relevant to a query. It probes the capability of adapted ranking models to perform direct relevance classification and compares their performance against LLM-based judges. Use when the user wants to benchmark on TREC-DL, or asks about evaluating this task. Reports binary accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.04455
  bibtex_key: meng2026rerankers
  confidence: medium
---

# relevance-judgment-eval

> Re-Rankers as Relevance Judges — Meng et al. (2026) (arXiv:2601.04455, 2026)

## What this evaluates

Evaluates whether pointwise re-rankers can function as binary relevance judges by predicting whether a document is relevant to a query. It probes the capability of adapted ranking models to perform direct relevance classification and compares their performance against LLM-based judges.

## Datasets

- **TREC-DL** — total ?; splits: test (-1)

## Metrics

- `binary accuracy` **(primary)** — range: [0, 1]
  - The proportion of correctly predicted binary relevance labels (0 or 1) out of the total number of query-document pairs evaluated.
- `Precision, Recall, MRR, MAP` — range: [0, 1]
  - Standard information retrieval metrics computed under binary relevance judgments. Precision measures the fraction of retrieved relevant documents, Recall measures the fraction of relevant documents retrieved, MRR is the reciprocal rank of the first relevant document, and MAP averages precision across all relevant documents.

## Input / output format

**Input**: Query $q$ and document $d$.

**Output**: Discrete relevance label $l \in \{0, 1\}$, where $l=1$ denotes relevant and $l=0$ denotes not relevant.

## Scoring recipe

```python
def compute_binary_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)
```

## Common pitfalls

- Self-preference and cross-family bias can artificially inflate scores when re-rankers judge outputs from similar or identical models.
- Score thresholding introduces arbitrary cutoffs that may not align with ground truth relevance boundaries, affecting binary label generation.
- Circularity risks arise when the same model family is used for both retrieval and judging, violating independent evaluation assumptions.

## Evidence (verbatim from paper)

> many widely used IR evaluation metrics (e.g., Precision, Recall, MRR, and MAP) are commonly applied under binary relevance judgments

## Citation

```bibtex
@misc{meng2026rerankers,
  title={Re-Rankers as Relevance Judges},
  author={Meng et al. (2026)},
  year={2026},
  note={arXiv:2601.04455}
}
```

- arXiv: 2601.04455