Multi News Eval

---
name: multi-news-eval
description: Evaluates abstractive models on multi-document summarization, measuring how well they condense multiple source articles into a single coherent summary. It probes coverage, redundancy control, and fluency under strict input length constraints. Use when the user wants to benchmark on Multi-News, DUC 2004, or asks about evaluating this task. Reports R-1, R-2, R-SU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1906.01749
  bibtex_key: [REDACTED]
  confidence: high
---

# multi-news-eval

> Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model — Fabbri et al. (2019) (arXiv:1906.01749, 2019)

## What this evaluates

Evaluates abstractive models on multi-document summarization, measuring how well they condense multiple source articles into a single coherent summary. It probes coverage, redundancy control, and fluency under strict input length constraints.

## Datasets

- **Multi-News** — total ?; splits: test (-1); repo https://github.com/Alex-Fabbri/Multi-News
- **DUC 2004** — total ?; splits: test (-1)

## Metrics

- `R-1` **(primary)** — range: percent
  - Unigram overlap between generated summary and reference summary.
- `R-2` **(primary)** — range: percent
  - Bigram overlap between generated summary and reference summary.
- `R-SU` **(primary)** — range: percent
  - Skip-gram overlap with a maximum gap of four words between tokens.
- `Human Pairwise Wins` — range: count
  - Count of times a system's summary was preferred over a baseline in pairwise comparisons across Informativeness, Fluency, and Non-Redundancy dimensions.

## Input / output format

**Input**: Concatenated source documents truncated to exactly 500 tokens total. Tokens are distributed evenly (500/S per document), with a special token inserted between documents to mark boundaries.

**Output**: Abstractive summary text.

## Scoring recipe

```python
# ROUGE scoring
rouge_scores = {}
for metric in ['R-1', 'R-2', 'R-SU']:
    rouge_scores[metric] = rouge_score(generated, reference, metric=metric) * 100

# Human evaluation scoring
human_wins = {'Informativeness': 0, 'Fluency': 0, 'Non-Redundancy': 0}
for pair in pairwise_comparisons:
    for dim in ['Informativeness', 'Fluency', 'Non-Redundancy']:
        if pair.winner == 'system':
            human_wins[dim] += 1
```

## Common pitfalls

- Truncation splits tokens evenly across documents (500/S) rather than taking full documents sequentially, which alters the input distribution.
- Cross-dataset evaluation trains on CNN/DailyMail and tests on DUC 2004, making results incomparable to in-domain Multi-News benchmarks.
- Human evaluation reports raw win counts rather than normalized scores or statistical significance, complicating direct metric comparison.

## Evidence (verbatim from paper)

> Following the setting from (Lebanoff et al., 2018), we report ROUGE (Lin, 2004) scores, which measure the overlap of unigrams (R-1), bigrams (R-2) and skip bigrams with a max distance of four words (R-SU). For the neural abstractive models, we truncate input articles to 500 tokens in the following way: for each example with S source input documents, we take the first 500 / S tokens from each source document.

## Citation

```bibtex
@misc{fabbri2019multi_news,
  title={Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model},
  author={Fabbri et al. (2019)},
  year={2019},
  note={arXiv:1906.01749}
}
```

- arXiv: 1906.01749