Legal Information Retrieval Eval

---
name: legal-information-retrieval-eval
description: This benchmark evaluates information retrieval systems on Swiss legal rulings and legislation. It tests the ability to rank relevant legal documents against long, multilingual queries and corpora. Use when the user wants to benchmark on Legal Information Retrieval, or asks about evaluating this task. Reports NDCG.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.09237
  bibtex_key: stern2023legal
  confidence: high
---

# legal-information-retrieval-eval

> One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support — Stern et al. (2023) (arXiv:2306.09237, 2023)

## What this evaluates

This benchmark evaluates information retrieval systems on Swiss legal rulings and legislation. It tests the ability to rank relevant legal documents against long, multilingual queries and corpora.

## Datasets

- **Legal Information Retrieval** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `NDCG` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain measures ranking quality by weighting relevance scores logarithmically by position to reward higher-ranked relevant documents.
- `Capped Recall@k` — range: [0, 1]
  - Measures the proportion of relevant documents retrieved within the top-k results, capped at a maximum threshold to handle varying ground-truth sizes.

## Input / output format

**Input**: Multilingual legal queries (avg. 847 words) and a corpus of Swiss legal documents (rulings/legislation, avg. 4K-7K words).

**Output**: Ranked list of document IDs or scores corresponding to the query.

## Scoring recipe

```python
def score_ir(predictions, golds, k_values=[1, 5, 10]):
    ndcg_scores = []
    recall_scores = []
    for q in queries:
        pred_ranking = predictions[q]
        rel_labels = golds[q]
        ndcg_scores.append(ndcg_at_k(pred_ranking, rel_labels))
        recall_scores.append(capped_recall_at_k(pred_ranking, rel_labels, k_values))
    return {'NDCG': mean(ndcg_scores), 'Capped_Recall@k': mean(recall_scores)}
```

## Common pitfalls

- BM25 is used for scalability but lacks contextual/multilingual understanding, potentially skewing baselines.
- Cross-encoder models were excluded due to computational cost on long documents, limiting state-of-the-art comparison.

## Evidence (verbatim from paper)

> We evaluate models with Normalized Discounted Cumulative Gain (NDCG) [67] and Capped Recall@k [11].

## Citation

```bibtex
@misc{stern2023legal,
  title={One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support},
  author={Stern et al. (2023)},
  year={2023},
  note={arXiv:2306.09237}
}
```

- arXiv: 2306.09237