Public Defense Retrieval Eval

---
name: public-defense-retrieval-eval
description: Evaluates the ability of retrieval and reranking models to surface relevant appellate brief paragraphs for public defender search queries. It probes domain-specific adaptation, query expansion strategies, and the impact of synthetic data generation on legal information retrieval. Use when the user wants to benchmark on PD Dataset, NJ OPD Dataset, BarExam-QA, LePaRD, or asks about evaluating this task. Reports recall@5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.14348
  bibtex_key: stammbach2026legalretrieval
  confidence: high
---

# public-defense-retrieval-eval

> Legal Retrieval for Public Defenders — Stammbach et al. (2026) (arXiv:2601.14348, 2026)

## What this evaluates

Evaluates the ability of retrieval and reranking models to surface relevant appellate brief paragraphs for public defender search queries. It probes domain-specific adaptation, query expansion strategies, and the impact of synthetic data generation on legal information retrieval.

## Datasets

- **PD Dataset** — total 170; splits: test (170); repo https://github.com/dominiksinsaarland/PublicDefenderRetrieval
- **NJ OPD Dataset** — total ?; splits: test (-1)
- **BarExam-QA** — total ?; splits: train (-1)
- **LePaRD** — total ?; splits: train (-1)

## Metrics

- `recall@5` **(primary)** — range: percent
  - Fraction of queries where at least one relevant passage appears in the top-5 retrieved results. Official metric for the PD dataset.
- `recall@1` — range: percent
  - Fraction of queries where at least one relevant passage appears in the top-1 retrieved results.
- `F1 score` — range: percent
  - Harmonic mean of precision and recall for binary relevance judgment in reranking experiments.

## Input / output format

**Input**: A natural language search query from a public defender, paired against a corpus of independent paragraphs from appellate briefs.

**Output**: For retrieval: a ranked list of top-k (k=1,5) candidate paragraphs. For reranking: a binary relevance label (relevant/irrelevant) for each candidate paragraph.

## Scoring recipe

```python
def recall_at_k(predictions, gold, k):
    hits = 0
    for q_preds, q_gold in zip(predictions, gold):
        if any(g in q_preds[:k] for g in q_gold):
            hits += 1
    return hits / len(predictions)

def f1_score(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
```

## Common pitfalls

- Data leakage in synthetic data generation: failing to remove paragraphs already annotated as retrieval targets inflates performance.
- Domain shift: fine-tuning on general legal benchmarks (BarExam-QA, LePaRD) decreases performance on public defense queries.
- Reranker baseline: most off-the-shelf rerankers underperform a simple majority baseline because the dataset is heavily skewed toward relevant paragraphs.

## Evidence (verbatim from paper)

> Given the practitioner-facing nature of the PD dataset, we report recall@5 as the official metric for the PD dataset (the BriefBank by default also returns five search results).

## Citation

```bibtex
@misc{stammbach2026legalretrieval,
  title={Legal Retrieval for Public Defenders},
  author={Stammbach et al. (2026)},
  year={2026},
  note={arXiv:2601.14348}
}
```

- arXiv: 2601.14348