Legalbench Rag Eval

---
name: legalbench-rag-eval
description: Evaluates the retrieval fidelity of RAG systems in the legal domain by measuring how precisely and completely a model retrieves minimal, highly relevant text snippets from legal documents to answer specific queries. Use when the user wants to benchmark on LegalBench-RAG, or asks about evaluating this task. Reports Precision.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.10343
  bibtex_key: pipitone2024legalbenchrag
  confidence: high
---

# legalbench-rag-eval

> LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain — Pipitone et al. (2024) (arXiv:2408.10343, 2024)

## What this evaluates

Evaluates the retrieval fidelity of RAG systems in the legal domain by measuring how precisely and completely a model retrieves minimal, highly relevant text snippets from legal documents to answer specific queries.

## Datasets

- **LegalBench-RAG** — total 6858; splits: test (-1); repo https://github.com/zeroentropy-cc/legalbenchrag

## Metrics

- `Precision` **(primary)** — range: percent
  - Fraction of retrieved chunks in the top-k that are relevant to the query. Calculated as |relevant ∩ top_k| / k.
- `Recall` — range: percent
  - Fraction of all relevant chunks in the document that are successfully retrieved within the top-k results. Calculated as |relevant ∩ top_k| / |relevant|.

## Input / output format

**Input**: A legal query paired with a legal document (or corpus) from which the system must retrieve relevant text chunks.

**Output**: A ranked list of top-k retrieved text chunks/snippets.

## Scoring recipe

```python
def score(retrieved, gold, k):
    top_k = retrieved[:k]
    hits = len(set(top_k) & set(gold))
    prec = (hits / k) * 100
    rec = (hits / len(gold)) * 100
    return prec, rec
# Average per dataset, then equally weight datasets
```

## Common pitfalls

- Averaging scores globally across datasets instead of weighting each dataset equally regardless of size, as explicitly required.
- Failing to account for chunking strategy (fixed-size vs. recursive splitter) and reranker presence, which drastically shift Precision/Recall.
- Choosing k without balancing context richness against noise/hallucination risks, as noted in post-processing design decisions.

## Evidence (verbatim from paper)

> Performance comparison on different datasets for Precision and Recall at various k values for the Naive Method.

## Citation

```bibtex
@misc{pipitone2024legalbenchrag,
  title={LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain},
  author={Pipitone et al. (2024)},
  year={2024},
  note={arXiv:2408.10343}
}
```

- arXiv: 2408.10343