Legal Rag Bench Eval

---
name: legal-rag-bench-eval
description: Evaluates end-to-end performance of legal retrieval-augmented generation systems by measuring retrieval accuracy, answer correctness, and answer groundedness. It uses a full factorial design across embedding and generative models, and introduces a hierarchical error decomposition to isolate hallucinations, retrieval failures, and reasoning errors. Use when the user wants to benchmark on Legal RAG Bench, or asks about evaluating this task. Reports correctness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.01710
  bibtex_key: butler2026legalragbench
  confidence: high
---

# legal-rag-bench-eval

> Legal RAG Bench: an end-to-end benchmark for legal RAG — Butler et al. (2026) (arXiv:2603.01710, 2026)

## What this evaluates

Evaluates end-to-end performance of legal retrieval-augmented generation systems by measuring retrieval accuracy, answer correctness, and answer groundedness. It uses a full factorial design across embedding and generative models, and introduces a hierarchical error decomposition to isolate hallucinations, retrieval failures, and reasoning errors.

## Datasets

- **Legal RAG Bench** — total 100; splits: test (100); repo https://github.com/isaacus-dev/legal-rag-bench

## Metrics

- `correctness` **(primary)** — range: [0, 1]
  - Binary metric: 1 if the generated answer entails the reference answer, 0 otherwise. Evaluated using an LLM-as-a-judge (GPT-5.2 high reasoning mode) against a clear rubric.
- `groundedness` — range: [0, 1]
  - Binary metric: 1 if the generated answer is supported by the retrieved passages provided to the model, 0 otherwise. Evaluated using an LLM-as-a-judge.
- `retrieval_accuracy` — range: [0, 1]
  - Binary metric: 1 if the annotated supporting passage is successfully retrieved by the embedding model, 0 otherwise.

## Input / output format

**Input**: A legal question and a set of retrieved text passages generated by an embedding model.

**Output**: A long-form natural language answer to the legal question.

## Scoring recipe

```python
def score_instance(question, retrieved_passages, generated_answer, reference_answer, annotated_passage):
    retrieval_acc = 1 if annotated_passage in retrieved_passages else 0
    groundedness = 1 if llm_judge(generated_answer, retrieved_passages) == 'supported' else 0
    correctness = 1 if llm_judge(generated_answer, reference_answer) == 'entails' else 0
    if not groundedness:
        error_type = 'Hallucination'
    elif not correctness and not retrieval_acc:
        error_type = 'Retrieval error'
    elif not correctness and retrieval_acc:
        error_type = 'Reasoning error'
    else:
        error_type = 'None'
    return correctness, groundedness, retrieval_acc, error_type
```

## Common pitfalls

- Overall RAG accuracy can mask hallucinations that coincidentally produce correct answers, making component-level metrics essential.
- Groundedness is evaluated strictly against the retrieved passages, not necessarily the ground-truth relevant passages, meaning a model can be marked grounded even if it retrieves irrelevant text.
- Retrieval accuracy only checks for the annotated supporting passage, ignoring the possibility that other retrieved passages might also be relevant.

## Evidence (verbatim from paper)

> In particular, for each question $i$, embedding model $e$, and LLM $l$, we assessed the following evaluation dimensions: 1. Correctness $(c_{eli})$: 1 if the model's answer entails the reference answer; 0 otherwise. 2. Groundedness $(g_{eli})$: 1 if the answer is supported by the retrieved passages provided to the model (irrespective of whether those passages are actually relevant); 0 otherwise. 3. Retrieval accuracy $(r_{ei})$: 1 if the annotated supporting passage is retrieved by the embedding model; 0 otherwise.

## Citation

```bibtex
@misc{butler2026legalragbench,
  title={Legal RAG Bench: an end-to-end benchmark for legal RAG},
  author={Butler et al. (2026)},
  year={2026},
  note={arXiv:2603.01710}
}
```

- arXiv: 2603.01710