Samsum Summarization Eval

---
name: samsum-summarization-eval
description: Abstractive summarization of naturalistic, messenger-style dialogues. It probes a model's ability to generate concise, human-like summaries from informal, multi-turn conversations containing typos, emoticons, and multi-party interactions. Use when the user wants to benchmark on SAMSum Corpus, or asks about evaluating this task. Reports ROUGE F1 (ROUGE-1, ROUGE-2, ROUGE-L).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.12237
  bibtex_key: gliwa2019samsun
  confidence: high
---

# samsum-summarization-eval

> SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization — Gliwa et al. (2019) (arXiv:1911.12237, 2019)

## What this evaluates

Abstractive summarization of naturalistic, messenger-style dialogues. It probes a model's ability to generate concise, human-like summaries from informal, multi-turn conversations containing typos, emoticons, and multi-party interactions.

## Datasets

- **SAMSum Corpus** — total 16369; splits: train (-1), test (-1)

## Metrics

- `ROUGE F1 (ROUGE-1, ROUGE-2, ROUGE-L)` **(primary)** — range: [0, 1]
  - Standard ROUGE metric reporting F1 scores with stemming for ROUGE-1, ROUGE-2, and ROUGE-L. F1 is the harmonic mean of recall and precision for n-gram overlaps.

## Input / output format

**Input**: Dialogue conversation text (messenger-style, truncated to 400 tokens) with utterances separated by a special token (e.g., <EOU> or |).

**Output**: Abstractive summary of the dialogue (no length limit during generation).

## Scoring recipe

```python
def compute_rouge_f1(predictions, references):
    # Uses py-rouge package with stemming enabled
    scores = {}
    for ngram in ['rouge1', 'rouge2', 'rougeL']:
        r = count_ngram_overlap(predictions, references, ngram, mode='recall')
        p = count_ngram_overlap(predictions, references, ngram, mode='precision')
        f1 = 2 * r * p / (r + p) if (r + p) > 0 else 0.0
        scores[ngram] = f1
    return scores
```

## Common pitfalls

- ROUGE F1 scores on dialogue summarization often contradict human evaluator judgments, making them an imperfect proxy for summary quality.
- Models trained exclusively on news corpora (e.g., CNN/Daily Mail) fail to generalize to dialogue data without domain-specific training or joint training.
- The use of stemming in the ROUGE calculation differs from standard implementations, which can cause score discrepancies if not replicated exactly.

## Evidence (verbatim from paper)

> We evaluate models with the standard ROUGE metric (Lin, 2004), reporting the $F_{1}$ scores (with stemming) for ROUGE-1, ROUGE-2 and ROUGE-L following previous works (Chen and Bansal, 2018; See et al., 2017). We obtain scores using the py-roge package.

## Citation

```bibtex
@misc{gliwa2019samsun,
  title={SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization},
  author={Gliwa et al. (2019)},
  year={2019},
  note={arXiv:1911.12237}
}
```

- arXiv: 1911.12237