Multiwoz 2.1 Jga Eval

---
name: multiwoz-2.1-jga-eval
description: Evaluates a model's ability to perform Dialogue State Tracking (DST) by predicting the correct values and statuses for all requested slots across multi-domain conversations. It specifically probes robustness to long-range contextual noise and class imbalance in slot status prediction. Use when the user wants to benchmark on MultiWOZ 2.1, or asks about evaluating this task. Reports joint-goal-accuracy (JGA).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2103.00109
  bibtex_key: zhang2021improving
  confidence: high
---

# multiwoz-2.1-jga-eval

> Improving Longer-range Dialogue State Tracking — Ye Zhang et al. (arXiv:2103.00109, 2021)

## What this evaluates

Evaluates a model's ability to perform Dialogue State Tracking (DST) by predicting the correct values and statuses for all requested slots across multi-domain conversations. It specifically probes robustness to long-range contextual noise and class imbalance in slot status prediction.

## Datasets

- **MultiWOZ 2.1** — total ?; splits: test (7372)

## Metrics

- `joint-goal-accuracy (JGA)` **(primary)** — range: [0, 1]
  - Ratio of turns where all dialogue states (slot values and statuses) are correctly predicted to the total number of turns.

## Input / output format

**Input**: Multi-turn conversation history containing user and agent utterances across multiple domains.

**Output**: Dialogue state representation specifying the status (active/inactive) and value for each slot in the current turn.

## Scoring recipe

```python
correct_turns = 0
total_turns = 0
for turn in test_set:
    total_turns += 1
    pred_states = model.predict(turn.history)
    if pred_states == turn.gold_states:
        correct_turns += 1
jga = correct_turns / total_turns
```

## Common pitfalls

- JGA is computed at the turn level, so a single incorrect slot in a turn causes the entire turn to be marked wrong.
- Slot status prediction suffers from severe class imbalance (most slots are inactive), making high accuracy easy to achieve trivially without meaningful prediction.

## Evidence (verbatim from paper)

> We use the commonly used joint-goal-accuracy (JGA) as metric for DST, defined as the ratio between turns whose all states are correctly predicted and total number of turns.

## Citation

```bibtex
@misc{zhang2021improving,
  title={Improving Longer-range Dialogue State Tracking},
  author={Ye Zhang et al.},
  year={2021},
  note={arXiv:2103.00109}
}
```

- arXiv: 2103.00109