Dst Eval

---
name: dst-eval
description: Evaluates cross-lingual and zero-shot dialogue state tracking by measuring a model's ability to predict correct slot-value pairs in target languages using limited or translated training data. Use when the user wants to benchmark on Parallel MultiWoZ, Multilingual WoZ, or asks about evaluating this task. Reports Joint Goal Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.13620
  bibtex_key: [REDACTED]
  confidence: high
---

# dst-eval

> Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking — Moghe et al. (2021) (arXiv:2109.13620, 2021)

## What this evaluates

Evaluates cross-lingual and zero-shot dialogue state tracking by measuring a model's ability to predict correct slot-value pairs in target languages using limited or translated training data.

## Datasets

- **Parallel MultiWoZ** — total 10000; splits: train (-1), test (-1)
- **Multilingual WoZ** — total ?; splits: train (-1), test (-1)

## Metrics

- `Joint Goal Accuracy` **(primary)** — range: percent
  - Proportion of dialogue turns where the predicted dialogue state exactly matches the ground truth dialogue state.
- `Slot F1` — range: percent
  - Macro-average of F1 scores computed over individual slot-types and slot-values for every turn.

## Input / output format

**Input**: Current utterance concatenated with previous dialogue act, encoded by a multilingual transformer. For Parallel MultiWoZ, the state language is fixed (Chinese) while conversation language varies. For Multilingual WoZ, states are in the same language as the dialogue (German/Italian).

**Output**: Set of predicted slot-value pairs representing the dialogue state for the current turn.

## Scoring recipe

```python
def compute_jga(predictions, gold):
    correct = sum(1 for pred, gt in zip(predictions, gold) if pred == gt)
    return (correct / len(gold)) * 100

def compute_slot_f1(predictions, gold):
    # Compute precision and recall for each slot-value type across all turns
    # F1 = 2 * (precision * recall) / (precision + recall)
    # Return macro-average F1 across all slot-value combinations
    pass
```

## Common pitfalls

- Slot Accuracy is explicitly discouraged because it treats unpredicted slots as correct, artificially inflating scores (e.g., 130 unpredicted out of 135 yields 96% accuracy despite errors).
- Cross-lingual evaluation setups differ between datasets: Parallel MultiWoZ uses language-agnostic states (Chinese) with mixed-language conversations, while Multilingual WoZ uses zero-shot language-matched states (German/Italian).
- Only 10% of target language training data is used for Parallel MultiWoZ, making low-resource performance the key focus rather than full supervised training.

## Evidence (verbatim from paper)

> The metrics used for dialogue state tracking tasks are turn-level and generally include Slot Accuracy, Slot F1, and Joint Goal Accuracy (JGA). Their descriptions are as follows:

Slot Accuracy: Proportion of the correct slots predicted across all utterances.

Slot F1: Macro-average of F1 score computed over the individual slot-types and slot-values for every turn.

Joint Goal Accuracy: Proportion of examples (dialogue turns) where the predicted dialogue state matches exactly the ground truth dialogue state.

## Citation

```bibtex
@misc{moghe2021crosslingual,
  title={Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking},
  author={Moghe et al. (2021)},
  year={2021},
  note={arXiv:2109.13620}
}
```

- arXiv: 2109.13620