Multiwoz 2.1 Eval

---
name: multiwoz-2.1-eval
description: Evaluates a model's ability to track and predict the complete set of user intent slots (dialogue state) across multiple domains in a multi-turn conversation. Use when the user wants to benchmark on MultiWOZ 2.1, or asks about evaluating this task. Reports slot accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1907.01669
  bibtex_key: budzianowski2019multiwoz
  confidence: high
---

# multiwoz-2.1-eval

> MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Baselines — Budzianowski et al. (2019) (arXiv:1907.01669, 2019)

## What this evaluates

Evaluates a model's ability to track and predict the complete set of user intent slots (dialogue state) across multiple domains in a multi-turn conversation.

## Datasets

- **MultiWOZ 2.1** — total ?; splits: train (-1), test (-1); repo https://github.com/budzianowski/multiwoz

## Metrics

- `slot accuracy` **(primary)** — range: percent
  - The proportion of correctly predicted slot values across all slots in the dialogue. Calculated by dividing the total number of correctly predicted slot-value pairs by the total number of ground truth slot-value pairs.
- `joint accuracy` — range: percent
  - The proportion of turns where the model's predicted state exactly matches the ground truth state across all domains. Also referred to as turn-level accuracy in the text.

## Input / output format

**Input**: Multi-turn dialogue history containing user utterances and system responses, segmented by domain.

**Output**: A dictionary mapping each domain to a set of slot-value pairs representing the predicted dialogue state.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    slot_correct = 0
    slot_total = 0
    joint_correct = 0
    for turn_pred, turn_gold in zip(predictions, gold):
        turn_slots_correct = 0
        for domain in turn_gold:
            for slot, value in turn_gold[domain].items():
                slot_total += 1
                if turn_pred[domain].get(slot) == value:
                    slot_correct += 1
                    turn_slots_correct += 1
        if turn_slots_correct == len(turn_gold):
            joint_correct += 1
    return slot_correct / slot_total, joint_correct / len(predictions)
```

## Common pitfalls

- Models frequently misclassify the dontcare and none labels, which were intentionally increased in MultiWOZ 2.1 to better capture user ambiguity and penalize spurious predictions.
- Strict exact-match evaluation requires precise canonicalization of slot values; minor formatting or synonym differences result in hard penalties for both slot and joint accuracy.

## Evidence (verbatim from paper)

> For the Flat Joint State Tracker, we also observed that the largest slot accuracy decrease from MultiWOZ 2.0 to MultiWOZ 2.1 occurred for the restaurant.name slot (87.02%→ 83.33%).

## Citation

```bibtex
@misc{budzianowski2019multiwoz,
  title={MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Baselines},
  author={Budzianowski et al. (2019)},
  year={2019},
  note={arXiv:1907.01669}
}
```

- arXiv: 1907.01669