Multiwoz Dialog Summarization Eval

---
name: multiwoz-dialog-summarization-eval
description: Evaluates abstractive summarization models on their ability to preserve critical semantic slots, entities, and domain consistency across multi-domain dialog conversations. Use when the user wants to benchmark on MultiWOZ, or asks about evaluating this task. Reports ROUGE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.00825
  bibtex_key: yuan2019abstractive
  confidence: high
---

# multiwoz-dialog-summarization-eval

> Abstractive Dialog Summarization with Semantic Scaffolds — Lin Yuan et al. (2019) (arXiv:1910.00825, 2019)

## What this evaluates

Evaluates abstractive summarization models on their ability to preserve critical semantic slots, entities, and domain consistency across multi-domain dialog conversations.

## Datasets

- **MultiWOZ** — total ?; splits: test (-1)

## Metrics

- `ROUGE` **(primary)** — range: [0, 1]
  - Standard ROUGE-L recall/precision/f1 comparing generated summaries to reference summaries.
- `CIC` — range: [0, 1]
  - Critical Information Coverage; measures the preservation of critical slot information and entities in the generated summary.
- `Relevance` — range: [1, 5]
  - Human-assigned score (1-5) indicating how well the summary captures the dialog's core information.
- `Conciseness` — range: [1, 5]
  - Human-assigned score (1-5) indicating the summary's brevity and lack of redundancy.
- `Readability` — range: [1, 5]
  - Human-assigned score (1-5) indicating the naturalness and fluency of the summary.
- `Ranking Win/Lose/Tie` — range: percent
  - Pairwise comparison percentage where workers rank one summary over another (or tie).

## Input / output format

**Input**: Multi-domain dialog conversations with speaker-labeled utterances (e.g., Client/Help desk).

**Output**: Abstractive summary text.

## Scoring recipe

```python
# Automatic metrics
rouge = compute_rouge_l(predictions, references)
cic = compute_cic(predictions, references)  # Measures critical slot/entity preservation

# Human evaluation (100 test samples, 3 workers each)
human_scores = []
for sample in test_set:
    for worker in workers:
        rel, conc, read = worker.score(sample.summary, scale=1-5)
        human_scores.append({'relevance': rel, 'conciseness': conc, 'readability': read})
        rank = worker.rank_pair(sample.summary_a, sample.summary_b)  # Returns Win/Lose/Tie
```

## Common pitfalls

- ROUGE scores can be high while CIC is low, meaning standard metrics miss critical slot/entity preservation.
- Ground truth references may omit necessary information or sound unnatural, limiting their reliability as a gold standard.
- Semantic slot scaffolds require explicit annotations that are rarely available in standard dialog corpora.

## Evidence (verbatim from paper)

> We observe that SPNet reaches the highest score in both ROUGE and CIC. Both Pointer-Generator and Transformer achieve high ROUGE scores, but a relative low CIC scores. It suggests that the baselines have more room for improvement on preserving critical slot information.

## Citation

```bibtex
@misc{yuan2019abstractive,
  title={Abstractive Dialog Summarization with Semantic Scaffolds},
  author={Lin Yuan et al. (2019)},
  year={2019},
  note={arXiv:1910.00825}
}
```

- arXiv: 1910.00825