Multiwoz2.1 Eval

---
name: multiwoz2.1-eval
description: Evaluates a model's ability to track dialogue state (domain, slot, value triplets) across conversation turns, specifically probing its robustness to user mind-changes or 'turnback' utterances that modify previously stated intentions. Use when the user wants to benchmark on MultiWOZ 2.1, or asks about evaluating this task. Reports joint goal accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2108.12637
  bibtex_key: [REDACTED]
  confidence: high
---

# multiwoz2.1-eval

> Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances — Kim et al. (2021) (arXiv:2108.12637, 2021)

## What this evaluates

Evaluates a model's ability to track dialogue state (domain, slot, value triplets) across conversation turns, specifically probing its robustness to user mind-changes or 'turnback' utterances that modify previously stated intentions.

## Datasets

- **MultiWOZ 2.1** — total 10419; splits: train (8420), val (1000), test (999)

## Metrics

- `joint goal accuracy` **(primary)** — range: [0, 1]
  - For each turn, the model outputs a belief state (set of domain, slot, value triplets). The metric is marked as 1 if the predicted triplet set exactly matches the gold triplet set, and 0 otherwise. The final score is the average over all turns in the test set.

## Input / output format

**Input**: Dialogue history consisting of alternating System and User utterances up to the current turn.

**Output**: Belief state represented as a set of (domain, slot, value) triplets.

## Scoring recipe

```python
correct = 0
total = 0
for turn in test_set:
    pred_state = model.predict(turn.history)
    gold_state = turn.gold_state
    if set(pred_state) == set(gold_state):
        correct += 1
    total += 1
return correct / total
```

## Common pitfalls

- Joint goal accuracy requires exact match of all slots in a turn; a single mismatch yields 0 for that turn.
- Performance drops on turnback scenarios stem from dataset bias (lack of turnback examples in training) rather than model architecture limitations.
- Template-generated turnbacks may overestimate model robustness compared to natural human turnbacks.

## Evidence (verbatim from paper)

> As a performance metric, the joint goal accuracy was employed. The joint goal accuracy is a standard criterion used to check if the model tracks the triplet of (domain, slot, value) precisely. When tracked correctly, the joint goal accuracy is marked as 1, and is otherwise 0. The numbers of training, validation, and test sets are 8420, 1000, and 999, respectively.

## Citation

```bibtex
@misc{kim2021ohmymistake,
  title={Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances},
  author={Kim et al. (2021)},
  year={2021},
  note={arXiv:2108.12637}
}
```

- arXiv: 2108.12637