Long Doc Rouge Eval

---
name: long-doc-rouge-eval
description: Evaluates the quality of abstractive summaries for long scientific documents by measuring n-gram overlap between generated text and reference abstracts. Use when the user wants to benchmark on arXiv, PubMed, or asks about evaluating this task. Reports ROUGE-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1804.05685
  bibtex_key: cohan2018discourse
  confidence: high
---

# long-doc-rouge-eval

> A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents — Cohan et al. (2018) (arXiv:1804.05685, 2018)

## What this evaluates

Evaluates the quality of abstractive summaries for long scientific documents by measuring n-gram overlap between generated text and reference abstracts.

## Datasets

- **arXiv** — total ?; splits: test (-1)
- **PubMed** — total ?; splits: test (-1)

## Metrics

- `ROUGE-1` **(primary)** — range: percent
  - Full-length F-1 score based on unigram overlap between generated summary and reference abstract. Tokens are lowercased and tokenized using spaCy.
- `ROUGE-2` — range: percent
  - Full-length F-1 score based on bigram overlap, computed with the same lowercased spaCy tokenization.
- `ROUGE-3` — range: percent
  - Full-length F-1 score based on trigram overlap, computed with the same lowercased spaCy tokenization.
- `ROUGE-L` — range: percent
  - Full-length F-1 score based on the longest common subsequence between generated and reference summaries, computed with the same lowercased spaCy tokenization.

## Input / output format

**Input**: Long scientific documents (arXiv/PubMed papers), truncated to 2000 tokens per document, 500 tokens per section, max 4 sections.

**Output**: Abstractive summary (abstract), generated via beam search with beam size 4, max length 210 tokens.

## Scoring recipe

```python
def compute_rouge_f1(generated, reference):
    # 1. Lowercase both strings
    gen = generated.lower()
    ref = reference.lower()
    # 2. Tokenize using spaCy (sentence and word level)
    gen_tokens = [token.text for token in nlp(gen)]
    ref_tokens = [token.text for token in nlp(ref)]
    # 3. Compute full-length F-1 score for specified ROUGE variant
    score = rouge_score(ref, gen, use_stemmer=False)
    return score['rouge1'].fmeasure * 100  # Return as percent
```

## Common pitfalls

- ROUGE tokenization differs across implementations; this paper explicitly uses spaCy for both sentence and word tokenization, unlike the standard NLTK-based ROUGE.
- The paper lowercases all tokens before evaluation, which can inflate scores compared to case-sensitive ROUGE variants.
- ROUGE scores are inherently biased towards extractive methods that copy salient sentences, making direct comparison with abstractive models challenging.

## Evidence (verbatim from paper)

> evaluation was done using the ROUGE automatic summarization evaluation metric (Lin, 2004) with full-length F-1 ROUGE scores. We lowercase all tokens and perform sentence and word tokenization using spaCy (Honnibal and Johnson, 2015).

## Citation

```bibtex
@misc{cohan2018discourse,
  title={A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents},
  author={Cohan et al. (2018)},
  year={2018},
  note={arXiv:1804.05685}
}
```

- arXiv: 1804.05685