Legal Bench Eval

---
name: legal-bench-eval
description: Evaluates the ability of state-space models (Mamba/SSD-Mamba) and transformers to perform statutory classification and case law retrieval on long-context legal documents. It probes how well models capture fine-grained semantic distinctions and maintain global coherence over thousands of tokens while balancing accuracy with computational throughput. Use when the user wants to benchmark on SCOTUS, ILDC, ECtHR, EUR-Lex, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.00141
  bibtex_key: maurya2025scalinglegalai
  confidence: medium
---

# legal-bench-eval

> Scaling Legal AI: Benchmarking Mamba and Transformers for Statutory Classification and Case Law Retrieval — Maurya et al. (2025) (arXiv:2509.00141, 2025)

## What this evaluates

Evaluates the ability of state-space models (Mamba/SSD-Mamba) and transformers to perform statutory classification and case law retrieval on long-context legal documents. It probes how well models capture fine-grained semantic distinctions and maintain global coherence over thousands of tokens while balancing accuracy with computational throughput.

## Datasets

- **SCOTUS** — total ?; splits: test (-1)
- **ILDC** — total ?; splits: test (-1)
- **ECtHR** — total ?; splits: test (-1)
- **EUR-Lex** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted labels out of total instances.
- `Micro-F1` — range: [0, 1]
  - F1 score computed globally by counting total true positives, false negatives, and false positives across all classes.
- `Macro-F1` — range: [0, 1]
  - F1 score computed per class and then averaged, giving equal weight to each category regardless of frequency.
- `Recall@10` — range: [0, 1]
  - Proportion of queries where at least one relevant document appears in the top-10 retrieved results.
- `nDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10, measuring ranking quality with logarithmic position discounting.
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank: average of 1/rank of the first relevant document for each query.
- `MAP` — range: [0, 1]
  - Mean Average Precision: average of precision values at each rank where a relevant document is retrieved.
- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve, measuring binary classification separability.

## Input / output format

**Input**: Legal documents (statutes or case law) provided as raw text sequences, ranging from moderate to extremely long context lengths.

**Output**: For classification: predicted category label(s). For retrieval: ranked list of candidate documents or relevance scores.

## Scoring recipe

```python
def compute_metrics(predictions, gold, k=10):
    # Classification
    accuracy = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    # F1 scores computed via standard confusion matrix aggregation (micro/macro)
    # Retrieval
    recall_k = sum(1 for p in predictions if any(g in p[:k] for g in gold)) / len(gold)
    # nDCG@k, MRR, MAP computed using standard ranking formulas with position discounting
    return {'accuracy': accuracy, 'recall@10': recall_k, 'nDCG@10': ndcg, 'MRR': mrr}
```

## Common pitfalls

- Confusing micro-averaged F1 with macro-averaged F1, which can mask performance on imbalanced legal categories.
- Comparing accuracy without accounting for context length limits or throughput differences, leading to unfair efficiency assessments.
- Treating Recall@10 and nDCG@10 as equivalent; Recall measures coverage while nDCG penalizes rank order errors.

## Evidence (verbatim from paper)

> On structured inputs such as SCOTUS issue classification (Table[IV]), DeBERTa achieves the strongest overall performance (Micro-F1 83.8, Accuracy 84.0), with Longformer close behind. Similarly, in ILDC retrieval (Table[VI]), both Longformer and DeBERTa outperform Mamba on Recall@10 and nDCG@10.

## Citation

```bibtex
@misc{maurya2025scalinglegalai,
  title={Scaling Legal AI: Benchmarking Mamba and Transformers for Statutory Classification and Case Law Retrieval},
  author={Maurya et al. (2025)},
  year={2025},
  note={arXiv:2509.00141}
}
```

- arXiv: 2509.00141