Multiwoz Dialogue Eval

---
name: multiwoz-dialogue-eval
description: Evaluates the quality, diversity, and goal adherence of task-oriented dialogue generation models. It measures how well a model generates natural, diverse responses while correctly incorporating specified dialogue goals and slot values. Use when the user wants to benchmark on MultiWOZ, or asks about evaluating this task. Reports BLEU-4.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1909.09220
  bibtex_key: lai2019goal
  confidence: high
---

# multiwoz-dialogue-eval

> Goal-Embedded Dual Hierarchical Model for Task-Oriented Dialogue Generation — Lai et al. (2019) (arXiv:1909.09220, 2019)

## What this evaluates

Evaluates the quality, diversity, and goal adherence of task-oriented dialogue generation models. It measures how well a model generates natural, diverse responses while correctly incorporating specified dialogue goals and slot values.

## Datasets

- **MultiWOZ** — total 10423; splits: train (8423), val (1000), test (1000)

## Metrics

- `BLEU-4` **(primary)** — range: percent
  - Standard BLEU score using 4-gram precision with a brevity penalty, computed against reference utterances.
- `D-1, D-2, D-U` — range: percent
  - Distinctiveness metrics: number of unique unigrams (D-1), bigrams (D-2), and full utterances (D-U) normalized by the total count of tokens/utterances in the generated text.
- `Multi-label F1` — range: percent
  - Precision, recall, and F1-score calculated for extracted slot values (e.g., address, booking status) compared to reference goal targets.

## Input / output format

**Input**: Delexicalized dialogue history with goal embeddings, capped at 22 turns and 36 tokens per sequence. Vocabulary size is 4258 including slots and special tokens.

**Output**: Generated utterance sequence produced via greedy decoding.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    # BLEU-4
    bleu4 = nltk.translate.bleu_score.sentence_bleu(references, predictions, weights=(0.25, 0.25, 0.25, 0.25))
    # Diversity
    d1 = len(set(predictions)) / len(predictions)
    d2 = len(set(ngrams(predictions, 2))) / len(predictions)
    du = len(set(predictions)) / total_utterances
    # Multi-label F1 for goal slots
    pred_slots = extract_slots(predictions)
    gold_slots = extract_slots(references)
    tp = len(pred_slots & gold_slots)
    fp = len(pred_slots - gold_slots)
    fn = len(gold_slots - pred_slots)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return bleu4, d1, d2, du, precision, recall, f1
```

## Common pitfalls

- Delexicalization strips entity names, so models cannot rely on memorizing specific values and must generate based on slot placeholders.
- Diversity metrics (D-1, D-2, D-U) are normalized by total token/utterance count, not vocabulary size, making them sensitive to sequence length.
- Goal focus is evaluated via multi-label slot extraction precision/recall rather than exact dialogue state tracking or intent classification.

## Evidence (verbatim from paper)

> We employ a number of automatic metrics as well as human evaluations to benchmark competing models on quality, diversity, and goal focus: Quality. BLEU (Papineni et al., 2002), as BLEU-4 by default, is a word-overlap measure against references and commonly used by dialogue generation works to evaluate quality (2015; 2016b; 2016a; 2017; 2018). Lower N-gram B1, B2, B3 are also reported. Diversity. D-1, D-2, D-U: The distinctiveness denotes the number of unique unigrams, bigrams, and utterances normalized by each total count (Li et al., 2016a; Xu et al., 2018). These metrics are commonly used to evaluate the dialogue diversity. Goal Focus. A set of slots such as address are extracted from reference dialogues as multi-label targets. Generated slots in model's output dialogues are the predictions. We use the multi-label precision, recall, and F1-score as surrogates to measure the goal focus and achievement.

## Citation

```bibtex
@misc{lai2019goal,
  title={Goal-Embedded Dual Hierarchical Model for Task-Oriented Dialogue Generation},
  author={Lai et al. (2019)},
  year={2019},
  note={arXiv:1909.09220}
}
```

- arXiv: 1909.09220