Umbrela Relevance Assessment Eval

---
name: umbrela-relevance-assessment-eval
description: This protocol evaluates the reliability of automatically generated relevance judgments (via the UMBRELA tool) compared to human assessments across different workflow conditions. It measures how well LLM-generated qrels align with human qrels in ranking retrieval systems using standard IR metrics and rank correlation. Use when the user wants to benchmark on TREC 2024 RAG Track, or asks about evaluating this task. Reports Kendall's τ.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.08275
  bibtex_key: upadhyay2024large
  confidence: high
---

# umbrela-relevance-assessment-eval

> A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look — Upadhyay et al. (2024) (arXiv:2411.08275, 2024)

## What this evaluates

This protocol evaluates the reliability of automatically generated relevance judgments (via the UMBRELA tool) compared to human assessments across different workflow conditions. It measures how well LLM-generated qrels align with human qrels in ranking retrieval systems using standard IR metrics and rank correlation.

## Datasets

- **TREC 2024 RAG Track** — total 301; splits: test (301)

## Metrics

- `Kendall's τ` **(primary)** — range: [-1, 1]
  - Measures the rank correlation between two sets of evaluation scores (e.g., run-level nDCG@20 scores computed from UMBRELA qrels vs. manual qrels). Computed across run-level scores, averaged per-topic correlations, or all topic/run combinations.
- `nDCG@20` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff 20, evaluating early precision in ranked retrieval lists.
- `nDCG@100` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff 100, capturing effectiveness over deeper ranked lists.
- `Recall@100` — range: [0, 1]
  - Recall at cutoff 100, quantifying the proportion of relevant passages retrieved within the top 100 results.

## Input / output format

**Input**: Query, retrieved ranked list of passages, and (for human/LLM assessment) the task of assigning a relevance grade to each passage. For the correlation analysis, the input is two sets of qrels (one from UMBRELA, one from human assessment) for the same set of topics/runs.

**Output**: Relevance grade per passage (not relevant, related, highly relevant, perfectly relevant). For the final evaluation step, a rank correlation coefficient (Kendall's τ) between two score distributions.

## Scoring recipe

```python
# Compute run-level scores for each metric using qrels
scores_umbrela = [compute_metric(run, qrels_umbrela) for run in runs]
scores_manual = [compute_metric(run, qrels_manual) for run in runs]
# Compute Kendall's tau between the two score vectors
tau = kendalltau(scores_umbrela, scores_manual)
# For per-topic avg: compute tau per topic, then average
# For all topic/runs: flatten all topic/run score pairs and compute tau
```

## Common pitfalls

- Confusing 'run-level correlation' (correlating aggregate scores across runs) with 'per-topic average correlation' (averaging correlations computed per topic).
- Assuming LLM-assisted human workflows yield higher correlation with gold standards than fully manual assessments (the study finds no tangible benefit).
- Mixing disjoint and overlapping topic sets when computing correlations, which changes the statistical properties and comparability of the results.

## Evidence (verbatim from paper)

> At a high-level, our analyses compute rank correlations between evaluation scores induced by UMBRELA judgments (qrels) vs. judgments (qrels) derived by the other assessment processes. Following common practice in IR meta-evaluations, rank correlation is captured using Kendall’s τ. However, there are multiple ways to design a rank correlation analysis. ... We focus on three: nDCG@20, nDCG@100, and Recall@100.

## Citation

```bibtex
@misc{upadhyay2024large,
  title={A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look},
  author={Upadhyay et al. (2024)},
  year={2024},
  note={arXiv:2411.08275}
}
```

- arXiv: 2411.08275