Dstc10 Spoken Eval

---
name: dstc10-spoken-eval
description: Evaluates task-oriented dialogue systems on spoken conversations to measure robustness against ASR errors and disfluencies. It probes multi-domain dialogue state tracking, knowledge-seeking turn detection, knowledge selection, and response generation capabilities under realistic speech conditions. Use when the user wants to benchmark on DSTC10, DSTC9, MultiWOZ 2.1, or asks about evaluating this task. Reports Joint Goal Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.13489
  bibtex_key: kim2021howrobust
  confidence: high
---

# dstc10-spoken-eval

> "How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations — Kim et al. (2021) (arXiv:2109.13489, 2021)

## What this evaluates

Evaluates task-oriented dialogue systems on spoken conversations to measure robustness against ASR errors and disfluencies. It probes multi-domain dialogue state tracking, knowledge-seeking turn detection, knowledge selection, and response generation capabilities under realistic speech conditions.

## Datasets

- **DSTC10** — total 263; splits: valid (263); repo https://github.com/alexa/alexa-with-dstc9-track1-dataset
- **DSTC9** — total 4181; splits: test (4181)
- **MultiWOZ 2.1** — total ?; splits: test (-1)

## Metrics

- `Joint Goal Accuracy` **(primary)** — range: [0, 1]
  - Percentage of dialogue turns where all slot values are predicted exactly correctly.
- `Slot F1` — range: [0, 1]
  - F1 score computed separately for value predictions and none predictions, reported independently or averaged.
- `Detection F1` — range: [0, 1]
  - F1 score for binary knowledge-seeking turn detection.
- `MRR@5` — range: [0, 1]
  - Mean Reciprocal Rank of the correct knowledge item within the top 5 retrieved candidates.
- `BLEU-4` — range: [0, 1]
  - 4-gram BLEU score comparing generated responses to reference responses.
- `Human Accuracy` — range: [1, 5]
  - Crowd-sourced score on a 1-5 scale assessing how accurate the system output is relative to reference knowledge.

## Input / output format

**Input**: Dialogue context and target user turn provided as ASR transcripts (1-best or n-best hypotheses), along with domain and knowledge information for response generation.

**Output**: For Task 1: predicted slot values and 'none' flags per slot. For Task 2: binary turn detection label, ranked list of knowledge items, and generated response text.

## Scoring recipe

```python
def score_dst(pred_slots, gold_slots):
    joint_acc = sum(1 for p, g in zip(pred_slots, gold_slots) if p == g) / len(gold_slots)
    val_f1 = f1_score(gold_values, pred_values)
    none_f1 = f1_score(gold_none, pred_none)
    return joint_acc, val_f1, none_f1

def score_detection(pred_det, gold_det):
    return f1_score(gold_det, pred_det)

def score_selection(pred_ranks, gold_idx):
    mrr = mean(1.0 / r for r in pred_ranks if r <= 5)
    rec1 = mean(1 if r == 1 else 0 for r in pred_ranks)
    return mrr, rec1

def score_generation(preds, refs):
    return compute_bleu(refs, preds, n=4), compute_rouge(refs, preds)

def score_human(preds, refs):
    scores = [crowd_worker_score(p, r) for p, r in zip(preds, refs)]
    return mean(scores)  # 1-5 scale
```

## Common pitfalls

- Comparing spoken DSTC10 directly with written MultiWOZ/DSTC9 without accounting for modality/locale differences, which causes significant performance drops independent of ASR errors.
- Evaluating only on the top-1 ASR result ignores the paper's finding that ensemble/n-best hypothesis aggregation is necessary for robust spoken dialogue modeling.
- Focusing solely on joint goal accuracy while overlooking slot-level precision/recall trade-offs, which masks false negative issues in value predictions.

## Evidence (verbatim from paper)

> It achieves extremely low performance on the spoken data in joint goal accuracy and also a significantly worse score in slot-level accuracy compared to that on MultiWOZ.

## Citation

```bibtex
@misc{kim2021howrobust,
  title={"How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken Conversations},
  author={Kim et al. (2021)},
  year={2021},
  note={arXiv:2109.13489}
}
```

- arXiv: 2109.13489