Wcep Mds Eval

---
name: wcep-mds-eval
description: Evaluates multi-document summarization systems on news event clusters by measuring how well generated summaries match human-written reference summaries. It probes the model's ability to extract or generate concise, informative summaries from highly redundant, large-scale document collections. Use when the user wants to benchmark on WCEP, or asks about evaluating this task. Reports ROUGE F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.10070
  bibtex_key: [REDACTED]
  confidence: high
---

# wcep-mds-eval

> A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal — Gholipour Ghalandari et al. (2020) (arXiv:2005.10070, 2020)

## What this evaluates

Evaluates multi-document summarization systems on news event clusters by measuring how well generated summaries match human-written reference summaries. It probes the model's ability to extract or generate concise, informative summaries from highly redundant, large-scale document collections.

## Datasets

- **WCEP** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/complementizer/wcep-mds-dataset

## Metrics

- `ROUGE F1-score` **(primary)** — range: [0, 1]
  - Computes n-gram overlap (1-gram, 2-gram, and longest common subsequence) between system and reference summaries, reporting F1-scores. Lowercased versions of summaries are used without further modification.

## Input / output format

**Input**: A cluster of news articles describing a single current event (truncated to a maximum of 100 articles per cluster in experiments).

**Output**: A summary of up to 40 tokens. Extractive methods return lists of full untruncated sentences; abstractive methods return generated text.

## Scoring recipe

```python
def compute_rouge_f1(predictions, references):
    preds = [p.lower() for p in predictions]
    refs = [r.lower() for r in references]
    scores = []
    for p, r in zip(preds, refs):
        r1 = rouge_1_f1(p, r)
        r2 = rouge_2_f1(p, r)
        rl = rouge_l_f1(p, r)
        scores.append((r1, r2, rl))
    return {
        'R1-F': sum(s[0] for s in scores) / len(scores),
        'R2-F': sum(s[1] for s in scores) / len(scores),
        'RL-F': sum(s[2] for s in scores) / len(scores)
    }
```

## Common pitfalls

- Summaries must be lowercased before scoring; otherwise ROUGE scores will be artificially low.
- The evaluation caps summary length at 40 tokens, but extractive methods must return full untruncated sentences that fit within this limit.
- The paper recommends evaluating with dynamic/longer output lengths and providing Recall results alongside the truncated F1 scores.

## Evidence (verbatim from paper)

> We evaluate lowercased versions of summaries and do not modify ground-truth or system summaries otherwise. We compare and evaluate systems using F1-score and Recall of ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004).

## Citation

```bibtex
@misc{gholipourghalandari2020wcep,
  title={A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal},
  author={Gholipour Ghalandari et al. (2020)},
  year={2020},
  note={arXiv:2005.10070}
}
```

- arXiv: 2005.10070