Legalbench Eval

---
name: legalbench-eval
description: This benchmark probes large language models' ability to perform diverse, real-world legal reasoning tasks, including rule-recall, issue-spotting, rule-application, interpretation, and rhetorical understanding. It evaluates how well models can apply legal frameworks, classify contractual clauses, and answer questions based on statutory or case law text. Use when the user wants to benchmark on LegalBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.11462
  bibtex_key: guha2023legalbench
  confidence: high
---

# legalbench-eval

> LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models — Guha et al. (2023) (arXiv:2308.11462, 2023)

## What this evaluates

This benchmark probes large language models' ability to perform diverse, real-world legal reasoning tasks, including rule-recall, issue-spotting, rule-application, interpretation, and rhetorical understanding. It evaluates how well models can apply legal frameworks, classify contractual clauses, and answer questions based on statutory or case law text.

## Datasets

- **LegalBench** — total ?; splits: test (-1); repo https://github.com/HazyResearch/legalbench

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Calculated as the proportion of correctly predicted labels or exact matches for generated outputs divided by the total number of instances in a task. For classification tasks, it measures the percentage of correct binary or multi-class predictions.

## Input / output format

**Input**: Task instructions, few-shot demonstrations, and a specific legal context (e.g., contract clause, court opinion excerpt, statutory text, or privacy policy) followed by a question or classification prompt.

**Output**: Model must produce a classification label (binary/multi-class), a multiple-choice option, or a generated text/citation string.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Context window constraints initially limited tasks to short documents, excluding long-form legal reasoning.
- Tasks are strictly objective and do not evaluate reasoning where 'reasonable minds may differ' or involve degrees of correctness.
- Benchmark is heavily skewed toward US law, contracts, and English, limiting generalizability to other jurisdictions or legal domains.
- Evaluates IRAC components (Issue, Rule, Application, Conclusion) independently rather than in multi-hop, sequential legal analysis.

## Evidence (verbatim from paper)

> Appendix F discusses each task in detail, providing a description of the reasoning that each task evaluates, how task data was constructed, task examples, and evaluation protocols. LEGALBENCH tasks also span different formats: multiple-choice questions (35 tasks), open-generation (7 tasks), binary classification (112 tasks), and multi-class/multi-label classification (8 tasks).

## Citation

```bibtex
@misc{guha2023legalbench,
  title={LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models},
  author={Guha et al. (2023)},
  year={2023},
  note={arXiv:2308.11462}
}
```

- arXiv: 2308.11462