Crosswoz Dst Eval

---
name: crosswoz-dst-eval
description: Evaluates the ability of generative dialogue state tracking models to accurately predict and maintain the complete set of user intents (domain-slot-value triples) across dialogue turns. It specifically probes cross-lingual and cross-ontology transfer capabilities by measuring how well models trained on one language or ontology generalize to another. Use when the user wants to benchmark on CrossWOZ-en, or asks about evaluating this task. Reports Joint Goal Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2101.11360
  bibtex_key: lin2021crosslingual
  confidence: medium
---

# crosswoz-dst-eval

> An Empirical Study of Cross-Lingual Transferability in Generative Dialogue State Tracker — Lin et al. (2021) (arXiv:2101.11360, 2021)

## What this evaluates

Evaluates the ability of generative dialogue state tracking models to accurately predict and maintain the complete set of user intents (domain-slot-value triples) across dialogue turns. It specifically probes cross-lingual and cross-ontology transfer capabilities by measuring how well models trained on one language or ontology generalize to another.

## Datasets

- **CrossWOZ-en** — total ?; splits: test (-1)

## Metrics

- `Joint Goal Accuracy` **(primary)** — range: [0, 1]
  - The proportion of dialogue turns where the predicted dialogue states match entirely to the ground truth dialogue states.
- `Slot F1` — range: [0, 1]
  - The macro-averaged F1 score computed across all slots in each dialogue turn.

## Input / output format

**Input**: Dialogue context and current user utterance.

**Output**: A set of domain-slot-value triples representing the current dialogue state.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    # predictions/golds: list of sets of (domain, slot, value) tuples per turn
    jga_correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    jga = jga_correct / len(predictions)
    
    slot_f1_scores = []
    for p, g in zip(predictions, golds):
        all_slots = {s for _, s, _ in g}
        slot_f1s = []
        for slot in all_slots:
            gold_vals = {v for d, s, v in g if s == slot}
            pred_vals = {v for d, s, v in p if s == slot}
            tp = len(gold_vals & pred_vals)
            fp = len(pred_vals - gold_vals)
            fn = len(gold_vals - pred_vals)
            prec = tp / (tp + fp) if (tp + fp) > 0 else 0
            rec = tp / (tp + fn) if (tp + fn) > 0 else 0
            f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
            slot_f1s.append(f1)
        slot_f1_scores.append(sum(slot_f1s) / len(slot_f1s) if slot_f1s else 0)
    slot_f1 = sum(slot_f1_scores) / len(slot_f1_scores)
    return jga, slot_f1
```

## Common pitfalls

- Joint Goal Accuracy requires exact set equality per turn; partial slot matches do not contribute to the score.
- Slot F1 is macro-averaged across slots, meaning rare slots contribute equally to the final metric as frequent ones, which can mask performance on dominant slots.
- Cross-lingual transfer experiments often suffer from ontology misalignment, where identical slot names map to different semantic concepts across languages, unfairly penalizing the model.

## Evidence (verbatim from paper)

> We use joint goal accuracy and slot F1 as our metrics to evaluate our dialogue state tracking system.

- Joint Goal Accuracy: The proportion of dialogue turns where predicted dialogue states match entirely to the ground truth dialogue states.  
- Slot F1: The macro-averaged F1 score for all slots in each turn.

## Citation

```bibtex
@misc{lin2021crosslingual,
  title={An Empirical Study of Cross-Lingual Transferability in Generative Dialogue State Tracker},
  author={Lin et al. (2021)},
  year={2021},
  note={arXiv:2101.11360}
}
```

- arXiv: 2101.11360