Maud Eval

---
name: maud-eval
description: This benchmark evaluates a model's ability to perform legal reading comprehension on merger agreements by answering specialized deal point questions. It probes the model's capacity to interpret complex contractual clauses and handle imbalanced classification tasks across various legal categories. Use when the user wants to benchmark on MAUD, or asks about evaluating this task. Reports AUPR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.00876
  bibtex_key: wang2023maud
  confidence: high
---

# maud-eval

> MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding — Wang et al. (2023) (arXiv:2301.00876, 2023)

## What this evaluates

This benchmark evaluates a model's ability to perform legal reading comprehension on merger agreements by answering specialized deal point questions. It probes the model's capacity to interpret complex contractual clauses and handle imbalanced classification tasks across various legal categories.

## Datasets

- **MAUD** — total 39000; splits: train (-1), dev (-1), test (-1); repo http://github.com/TheAtticusProject/maud

## Metrics

- `AUPR` **(primary)** — range: [0, 1]
  - Area under the precision-recall curve, averaged across different questions and answers to account for imbalanced answer distributions.

## Input / output format

**Input**: A merger agreement text excerpt paired with a specific deal point question.

**Output**: A predicted class label for the deal point question.

## Scoring recipe

```python
def compute_aupr(y_true, y_prob):
    precisions, recalls, _ = precision_recall_curve(y_true, y_prob)
    return auc(recalls, precisions)
# Average AUPR across all deal point questions and answer types as detailed in Appendix A.5
```

## Common pitfalls

- Imbalanced answer distributions make standard accuracy or F1 misleading; AUPR is required.
- Over 50% of texts exceed 512 tokens, so truncation or long-context models (e.g., BigBird) are necessary for fair evaluation.
- Single-task models typically outperform multi-task models by ~4 pp AUPR, contrary to typical multi-task benefits.

## Evidence (verbatim from paper)

> Many MAUD questions have an imbalanced answer distribution, so we use area under the precision-recall curve (AUPR) as our primary metric. See Appendix [A.5](#A1.SS5 "A.5 Details on MAUD AUPR Score ‣ Appendix A Appendix ‣ MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding") for details on how we average AUPR across different questions and answers.

## Citation

```bibtex
@misc{wang2023maud,
  title={MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2301.00876}
}
```

- arXiv: 2301.00876