Bigpatent Eval

---
name: bigpatent-eval
description: Evaluates abstractive summarization models on patent documents, probing their ability to capture global discourse structure, maintain entity coherence, and generate novel content without excessive repetition or fabrication. Use when the user wants to benchmark on BIGPATENT, or asks about evaluating this task. Reports ROUGE-1 F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1906.03741
  bibtex_key: sharma2019bigpatent
  confidence: high
---

# bigpatent-eval

> BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization — Sharma et al. (2019) (arXiv:1906.03741, 2019)

## What this evaluates

Evaluates abstractive summarization models on patent documents, probing their ability to capture global discourse structure, maintain entity coherence, and generate novel content without excessive repetition or fabrication.

## Datasets

- **BIGPATENT** — total ?; splits: test (-1)

## Metrics

- `ROUGE-1 F1` **(primary)** — range: percent
  - F1 score computed between the generated summary and the human-written reference summary using unigram overlap.
- `ROUGE-2 F1` — range: percent
  - F1 score computed between the generated summary and the human-written reference summary using bigram overlap.
- `ROUGE-L F1` — range: percent
  - F1 score based on the longest common subsequence between the generated summary and the reference summary.

## Input / output format

**Input**: Patent document text (input articles).

**Output**: Abstractive summary text (truncated to 100 words/tokens during evaluation).

## Scoring recipe

```python
def compute_rouge_f1(generated, reference):
    gen_tokens = tokenize(generated)
    ref_tokens = tokenize(reference)
    gen_ngrams = ngrams(gen_tokens, n=1)
    ref_ngrams = ngrams(ref_tokens, n=1)
    overlap = sum(min(gen_ngrams[g], ref_ngrams[g]) for g in set(gen_ngrams) & set(ref_ngrams))
    precision = overlap / len(gen_ngrams) if len(gen_ngrams) > 0 else 0
    recall = overlap / len(ref_ngrams) if len(ref_ngrams) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100
```

## Common pitfalls

- Models are evaluated with inputs and summaries truncated to 400 and 100 respectively, which may disadvantage models designed for long documents.
- ROUGE scores do not capture entity repetition or fabrication, which are significant failure modes in patent summarization.
- Extractive baselines like LEAD-3 perform poorly due to the uniform distribution of salient content in patents, unlike news articles.

## Evidence (verbatim from paper)

> Table 4 reports F1 scores of ROUGE-1, 2, and L (Lin and Hovy, 2003) for all models.

## Citation

```bibtex
@misc{sharma2019bigpatent,
  title={BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization},
  author={Sharma et al. (2019)},
  year={2019},
  note={arXiv:1906.03741}
}
```

- arXiv: 1906.03741