Dstc11 Track3 Eval

---
name: dstc11-track3-eval
description: Evaluates a system's ability to track dialogue state in spoken conversations, specifically measuring robustness to ASR errors, disfluencies, and proper noun mismatches. Use when the user wants to benchmark on DSTC11 Track 3, or asks about evaluating this task. Reports JGA.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.15053
  bibtex_key: yoon2023adapting
  confidence: high
---

# dstc11-track3-eval

> Adapting Text-based Dialogue State Tracker for Spoken Dialogues — Yoon et al. (2023) (arXiv:2308.15053, 2023)

## What this evaluates

Evaluates a system's ability to track dialogue state in spoken conversations, specifically measuring robustness to ASR errors, disfluencies, and proper noun mismatches.

## Datasets

- **DSTC11 Track 3** — total ?; splits: tts-verbatim (-1), human-verbatim (-1), human-paraphrased (-1)

## Metrics

- `JGA` **(primary)** — range: percent
  - Joint Goal Accuracy: the percentage of dialogue turns where all slot values are predicted exactly correctly.
- `SER` — range: percent
  - Slot Error Rate: the percentage of individual slots across all turns that are predicted incorrectly.

## Input / output format

**Input**: Sequential dialogue turns provided as ASR transcripts or TTS-generated text.

**Output**: A set of slot-value pairs representing the current dialogue state.

## Scoring recipe

```python
def compute_jga(pred_states, gold_states):
    correct = sum(1 for p, g in zip(pred_states, gold_states) if p == g)
    return correct / len(gold_states)

def compute_ser(pred_states, gold_states):
    total, errors = 0, 0
    for p, g in zip(pred_states, gold_states):
        for slot in g:
            total += 1
            if slot not in p:
                errors += 1
    return errors / total
```

## Common pitfalls

- Confusing evaluation splits: TTS-verbatim (machine-generated) vs human-verbatim/paraphrased (human speech) drastically changes ASR error profiles and results.
- Proper nouns dominate error rates due to ASR hallucinations and lack of external ontologies, often masking performance on common slots.

## Evidence (verbatim from paper)

> Table 5 is the official results of the test submission by the participants. A total of 6 teams submitted, and each team could submit up to 2 systems, so a total of 11 systems were submitted. We submitted a model that recorded 42.4 in the validation set. Finally, our model achieved third place, with JGA 40.2 for tts-verbatim in the challenge.

## Citation

```bibtex
@misc{yoon2023adapting,
  title={Adapting Text-based Dialogue State Tracker for Spoken Dialogues},
  author={Yoon et al. (2023)},
  year={2023},
  note={arXiv:2308.15053}
}
```

- arXiv: 2308.15053