Spokenwoz Dst Eval

---
name: spokenwoz-dst-eval
description: Evaluates dialog state tracking performance by measuring how accurately an agent predicts and maintains the current state of a multi-turn spoken conversation. Use when the user wants to benchmark on SpokenWOZ, or asks about evaluating this task. Reports Joint Goal Accuracy (JGA).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.23049
  bibtex_key: maben2025aura
  confidence: high
---

# spokenwoz-dst-eval

> AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks — Maben et al. (2025) (arXiv:2506.23049, 2025)

## What this evaluates

Evaluates dialog state tracking performance by measuring how accurately an agent predicts and maintains the current state of a multi-turn spoken conversation.

## Datasets

- **SpokenWOZ** — total ?; splits: test (-1)

## Metrics

- `Joint Goal Accuracy (JGA)` **(primary)** — range: percent
  - Exact match between predicted and ground-truth dialog states across turns. Calculated as (number of turns with exact state match / total turns) * 100.

## Input / output format

**Input**: Spoken dialogue turns with domain classification and slot-filling requirements.

**Output**: Predicted dialog state (set of slot-value pairs) per turn.

## Scoring recipe

```python
def compute_jga(predictions, gold):
    exact_matches = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (exact_matches / len(gold)) * 100
```

## Common pitfalls

- JGA requires exact match of all slots, making it sensitive to minor formatting or ordering differences.
- Domain classification is done via prompt-based inference prior to DST, which can propagate errors into state tracking.
- No fine-tuning is used, relying entirely on prompt-based inference, which may not reflect fine-tuned system capabilities.

## Evidence (verbatim from paper)

> We evaluate AURA’s dialog state tracking (DST) on the SpokenWOZ benchmark using Joint Goal Accuracy (JGA), which measures the exact match between predicted and ground-truth dialog states across turns. AURA outperforms the best baseline by over 3 points, achieving a JGA of 28.76 using LLaMA3.3-70B with prompt-based DST.

## Citation

```bibtex
@misc{maben2025aura,
  title={AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks},
  author={Maben et al. (2025)},
  year={2025},
  note={arXiv:2506.23049}
}
```

- arXiv: 2506.23049