Schema Guided Dstc8 Eval

---
name: schema-guided-dstc8-eval
description: Evaluates zero-shot dialogue state tracking across single and multi-domain conversations. It measures the model's ability to predict intents, extract slot values, and maintain accurate dialogue states over long contexts without prior exposure to unseen service domains. Use when the user wants to benchmark on Schema-Guided Dialogue (DSTC8 Track 4), or asks about evaluating this task. Reports Joint Goal Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2006.09035
  bibtex_key: li2020sppd
  confidence: high
---

# schema-guided-dstc8-eval

> The SPPD System for Schema Guided Dialogue State Tracking Challenge — Li et al. (2020) (arXiv:2006.09035, 2020)

## What this evaluates

Evaluates zero-shot dialogue state tracking across single and multi-domain conversations. It measures the model's ability to predict intents, extract slot values, and maintain accurate dialogue states over long contexts without prior exposure to unseen service domains.

## Datasets

- **Schema-Guided Dialogue (DSTC8 Track 4)** — total ?; splits: train (-1), dev (-1)

## Metrics

- `Joint Goal Accuracy` **(primary)** — range: percent
  - Exact match rate of all slot values across all domains in a dialogue turn.
- `Average Goal Accuracy` — range: percent
  - Mean of exact match rates computed per domain.
- `Intent Accuracy` — range: percent
  - Percentage of correctly predicted dialogue intents.
- `Requested Slot F1` — range: percent
  - F1 score for correctly identifying requested slots.

## Input / output format

**Input**: Lower-cased user utterances, system dialogue actions, and service schema definitions encoded via BERT.

**Output**: Predicted intent label, slot values for each domain, and requested slots.

## Scoring recipe

```python
def joint_goal_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:  # Exact match of all slot values across domains
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Evaluating only on seen domains ignores the zero-shot capability explicitly tested on unseen services.
- Using original system utterances instead of dialogue actions degrades performance, as the authors found actions yield better results.
- Joint goal accuracy requires exact match across all domains; partial matches or domain-wise averaging will overstate performance.

## Evidence (verbatim from paper)

> Table 2: The overall performance of our system  
<table><tr><td>Model</td><td>Intent Accuracy
All(Seen/Unseen)</td><td>Average Goal Accuracy
All(Seen/Unseen)</td><td>Joint Goal Accuracy
All(Seen/Unseen)</td><td>Requested Slot F1
All(Seen/Unseen)</td></tr>

## Citation

```bibtex
@misc{li2020sppd,
  title={The SPPD System for Schema Guided Dialogue State Tracking Challenge},
  author={Li et al. (2020)},
  year={2020},
  note={arXiv:2006.09035}
}
```

- arXiv: 2006.09035