Wikicatsum Rouge Eval

---
name: wikicatsum-rouge-eval
description: Evaluates abstractive multi-document summarization models on their ability to generate coherent, content-adequate summaries across three domains (Company, Film, Animal). The protocol measures lexical and sentence-level overlap between generated summaries and reference summaries using ROUGE metrics, while also contextualizing scores against a baseline overlap between input documents and summaries. Use when the user wants to benchmark on WIKICATSUM, or asks about evaluating this task. Reports ROUGE-1, ROUGE-2, ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1906.04687
  bibtex_key: perezbeltrachini2019generatingsummariestopic
  confidence: high
---

# wikicatsum-rouge-eval

> Generating Summaries with Topic Templates and Structured Convolutional Decoders — Perez-Beltrachini et al. (2019) (arXiv:1906.04687, 2019)

## What this evaluates

Evaluates abstractive multi-document summarization models on their ability to generate coherent, content-adequate summaries across three domains (Company, Film, Animal). The protocol measures lexical and sentence-level overlap between generated summaries and reference summaries using ROUGE metrics, while also contextualizing scores against a baseline overlap between input documents and summaries.

## Datasets

- **WIKICATSUM** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `ROUGE-1, ROUGE-2, ROUGE-L` **(primary)** — range: [0, 1]
  - ROUGE computes recall-based n-gram overlap (ROUGE-1 for unigrams, ROUGE-2 for bigrams) and longest common subsequence similarity (ROUGE-L) between the generated summary and the reference summary. Scores are reported as decimals between 0 and 1.

## Input / output format

**Input**: First 800 tokens of multi-document input paragraphs concatenated with the Wikipedia lead section.

**Output**: Generated abstractive summary text.

## Scoring recipe

```python
def compute_rouge_metrics(generated_summary, reference_summary):
    import rouge
    scorer = rouge.Rouge()
    scores = scorer.get_scores(generated_summary, reference_summary, avg=True)
    return scores['rouge-1']['r'], scores['rouge-2']['r'], scores['rouge-l']['r']
```

## Common pitfalls

- The paper computes a baseline ROUGE score between the input documents and summaries to contextualize model performance; this should not be confused with the standard model-to-reference evaluation.
- ROUGE scores are reported as decimals (e.g., .260) rather than percentages, which can cause misinterpretation when comparing to other literature using a 0-100 scale.
- Generation uses beam size 5 and trigram blocking (discarding steps with >80% token overlap), which directly impacts ROUGE scores and must be replicated for fair comparison.

## Evidence (verbatim from paper)

> We compute recall ROUGE scores of the input documents against the summaries to assess the amount of overlap and as a reference for the interpretation of the scores achieved by the models. Across domains content overlap (R1) is ~ 50 points. However, R2 is much lower indicating that there is abstraction, paraphrasing, and content selection in the summaries with respect to the input.

## Citation

```bibtex
@misc{perezbeltrachini2019generatingsummariestopic,
  title={Generating Summaries with Topic Templates and Structured Convolutional Decoders},
  author={Perez-Beltrachini et al. (2019)},
  year={2019},
  note={arXiv:1906.04687}
}
```

- arXiv: 1906.04687