Cuad Eval

---
name: cuad-eval
description: Evaluates a model's ability to identify and extract relevant text spans from legal contracts corresponding to specific clause categories. It probes domain-specific information extraction and needle-in-a-haystack detection under severe class imbalance. Use when the user wants to benchmark on CUAD, or asks about evaluating this task. Reports Precision@80% Recall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.06268
  bibtex_key: hendrycks2021cuad
  confidence: high
---

# cuad-eval

> CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review — Hendrycks et al. (2021) (arXiv:2103.06268, 2021)

## What this evaluates

Evaluates a model's ability to identify and extract relevant text spans from legal contracts corresponding to specific clause categories. It probes domain-specific information extraction and needle-in-a-haystack detection under severe class imbalance.

## Datasets

- **CUAD** — total 500; splits: train (-1), test (-1)

## Metrics

- `AUPR` — range: percent
  - Area under the precision-recall curve computed by sweeping the model's confidence threshold across all predictions.
- `Precision@80% Recall` **(primary)** — range: percent
  - Precision achieved when the confidence threshold is tuned to yield exactly 80% recall.
- `Precision@90% Recall` — range: percent
  - Precision achieved when the confidence threshold is tuned to yield exactly 90% recall.

## Input / output format

**Input**: A legal contract document (context) and a target label category (question) with a short description. The model receives the text and predicts start/end token positions for relevant spans.

**Output**: Predicted start and end token positions for each label category, accompanied by a confidence probability.

## Scoring recipe

```python
def compute_metrics(predictions, ground_truth):
    # 1. Normalize strings: lowercase, remove punctuation, split by space
    # 2. For each prediction span, compute Jaccard similarity J(A,B) vs each GT span
    # 3. Match prediction to GT if max J(A,B) >= 0.5; else mark as False Positive
    # 4. Unmatched GT spans are False Negatives; matched are True Positives
    # 5. Vary confidence threshold to compute Precision and Recall at each step
    # 6. Compute AUPR by integrating the PR curve
    # 7. Extract Precision at 80% and 90% recall from the curve
    return AUPR, Precision_at_80, Precision_at_90
```

## Common pitfalls

- Severe class imbalance (>99% of sliding windows are negative) can cause models to trivially predict empty spans unless negative samples are downweighted during training.
- Span matching uses a Jaccard similarity threshold of 0.5 on normalized word sets, which may penalize minor tokenization or punctuation differences.
- Precision@X% Recall requires precise thresholding of confidence scores; curve interpolation or threshold selection methods can affect reported values.

## Evidence (verbatim from paper)

> We use the Area Under the Precision-Recall curve (AUPR) and Precision at 80% and 90% Recall as our primary metrics. ... We determine whether a highlighted text span matches the ground truth with the Jaccard similarity coefficient. ... We use the threshold 0.5≤J(A,B) for determining matches.

## Citation

```bibtex
@misc{hendrycks2021cuad,
  title={CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review},
  author={Hendrycks et al. (2021)},
  year={2021},
  note={arXiv:2103.06268}
}
```

- arXiv: 2103.06268