Dialogue Summarization Eval

---
name: dialogue-summarization-eval
description: Evaluates the quality of unsupervised abstractive dialogue summarization across multiple domains by comparing generated summaries against human references using standard n-gram and LCS overlap metrics. It tests the model's ability to compress and rephrase conversational transcripts into coherent summaries without training data. Use when the user wants to benchmark on AMI, ICSI, DialogSum, SAMSum, MediaSum, SummScreen, ADS, or asks about evaluating this task. Reports ROUGE-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.13108
  bibtex_key: park2022unsupervised
  confidence: high
---

# dialogue-summarization-eval

> Unsupervised Abstractive Dialogue Summarization with Word Graphs and POV Conversion — Park et al. (2022) (arXiv:2205.13108, 2022)

## What this evaluates

Evaluates the quality of unsupervised abstractive dialogue summarization across multiple domains by comparing generated summaries against human references using standard n-gram and LCS overlap metrics. It tests the model's ability to compress and rephrase conversational transcripts into coherent summaries without training data.

## Datasets

- **AMI** — total ?; splits: test (-1)
- **ICSI** — total ?; splits: test (-1)
- **DialogSum** — total ?; splits: test (-1)
- **SAMSum** — total ?; splits: test (-1)
- **MediaSum** — total ?; splits: test (-1)
- **SummScreen** — total ?; splits: test (-1)
- **ADS** — total ?; splits: test (-1)

## Metrics

- `ROUGE-1` **(primary)** — range: percent
  - F-1 score based on unigram overlap between the generated summary and the reference summary.
- `ROUGE-2` — range: percent
  - F-1 score based on bigram overlap between the generated summary and the reference summary.
- `ROUGE-L` — range: percent
  - F-1 score based on longest common subsequence (LCS) overlap between the generated summary and the reference summary.

## Input / output format

**Input**: Dialogue transcript (optionally pre-segmented by topic for long texts >5,000 characters).

**Output**: A single abstractive summary sentence or paragraph.

## Scoring recipe

```python
def compute_rouge_f1(pred, ref, ngram_type):
    pred_ngrams = extract_ngrams(pred, ngram_type)
    ref_ngrams = extract_ngrams(ref, ngram_type)
    overlap = count_overlap(pred_ngrams, ref_ngrams)
    prec = overlap / len(pred_ngrams) if len(pred_ngrams) > 0 else 0
    rec = overlap / len(ref_ngrams) if len(ref_ngrams) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
# ngram_type: 'unigram' for ROUGE-1, 'bigram' for ROUGE-2, 'lcs' for ROUGE-L
```

## Common pitfalls

- Topic segmentation is only applied to datasets with average transcription length greater than 5,000 characters (MediaSum, SummScreen), which changes the input structure for those splits.
- POV conversion is applied to all datasets to transform semi-extractive outputs into fully abstractive ones, meaning the raw graph path is not the final output.
- ROUGE scores are reported as percentages (e.g., 20.79), not decimals, which can cause confusion if standard 0-1 scaling is assumed.

## Evidence (verbatim from paper)

> We evaluate the quality of generated system summaries against reference summaries using standard ROUGE scores (Lin, 2004). Specifically, we use ROUGE-1 (R1), ROUGE-2 (R2), and ROUGE-L (RL) scores that respectively measure unigram, bigram, and longest common subsequence coverage.

## Citation

```bibtex
@misc{park2022unsupervised,
  title={Unsupervised Abstractive Dialogue Summarization with Word Graphs and POV Conversion},
  author={Park et al. (2022)},
  year={2022},
  note={arXiv:2205.13108}
}
```

- arXiv: 2205.13108