Billsum Eval

---
name: billsum-eval
description: This benchmark evaluates the ability of models to automatically generate concise and accurate summaries of complex, nested legislative texts. It probes extractive and abstractive summarization capabilities in a highly technical, domain-specific legal context. Use when the user wants to benchmark on BillSum, or asks about evaluating this task. Reports ROUGE F-Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.00523
  bibtex_key: [REDACTED]
  confidence: high
---

# billsum-eval

> BillSum: A Corpus for Automatic Summarization of US Legislation — Kornilova et al. (2019) (arXiv:1910.00523, 2019)

## What this evaluates

This benchmark evaluates the ability of models to automatically generate concise and accurate summaries of complex, nested legislative texts. It probes extractive and abstractive summarization capabilities in a highly technical, domain-specific legal context.

## Datasets

- **BillSum** — total 23455; splits: train (-1), test (-1); repo https://github.com/FiscalNote/BillSum

## Metrics

- `ROUGE F-Score` **(primary)** — range: percent
  - Computes the F-measure (harmonic mean of precision and recall) for overlapping n-grams between the generated summary and the reference summary. Expressed as a percentage.

## Input / output format

**Input**: Full text of a US Congressional or California state legislative bill.

**Output**: A generated summary of the bill, evaluated against a human-written reference summary.

## Scoring recipe

```python
def compute_rouge_f_score(generated, reference, n=2):
    gen_ngrams = ngrams(generated, n)
    ref_ngrams = ngrams(reference, n)
    overlap = len(gen_ngrams & ref_ngrams)
    precision = overlap / len(gen_ngrams) if gen_ngrams else 0
    recall = overlap / len(ref_ngrams) if ref_ngrams else 0
    if precision + recall == 0:
        return 0.0
    f_score = 2 * (precision * recall) / (precision + recall)
    return f_score * 100
```

## Common pitfalls

- ROUGE F-scores are reported as percentages, not decimals.
- The dataset features nested, bulleted legislative structures and technical legal language, which can cause extractive models to struggle with sentence-level salience.
- Models trained on US Congressional bills often show performance drops when transferred to California state bills due to differences in summary structure and content.

## Evidence (verbatim from paper)

> The Rouge F-Score is used because it considers both the completeness and conciseness of the summary method.[11,12]

## Citation

```bibtex
@misc{kornilova2019billsum,
  title={BillSum: A Corpus for Automatic Summarization of US Legislation},
  author={Kornilova et al. (2019)},
  year={2019},
  note={arXiv:1910.00523}
}
```

- arXiv: 1910.00523