Granular Change Accuracy

---
name: granular-change-accuracy
description: Evaluates Dialogue State Tracking (DST) models by measuring performance based on per-turn belief state changes rather than raw slot or turn-level accuracy. It aims to provide partial credit for partially correct predictions and reduce bias from error timing and distribution across dialogue turns. Use when the user has predictions and gold and needs to compute Granular Change Accuracy.
metadata:
  skill_kind: metric
  source_arxiv: 2403.11123
  bibtex_key: aksu2024granularchangeaccuracy
  confidence: medium
---

# granular-change-accuracy

> Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking — Aksu et al. (2024) (arXiv:2403.11123, 2024)

## What this evaluates

Evaluates Dialogue State Tracking (DST) models by measuring performance based on per-turn belief state changes rather than raw slot or turn-level accuracy. It aims to provide partial credit for partially correct predictions and reduce bias from error timing and distribution across dialogue turns.

## Datasets

- **MultiWOZ 2.1** — total ?; splits: test (-1)
- **SGD** — total ?; splits: test (-1)

## Metrics

- `Granular Change Accuracy` **(primary)** — range: [0, 1]
  - Computes accuracy by comparing predicted versus gold belief state changes per turn. Assigns partial credit for partially correct slot-value updates and avoids double-counting errors when incorrect values persist across multiple turns. Aggregated as an average score over all turns.

## Input / output format

**Input**: A sequence of system and user utterance pairs per turn, along with the previous turn's belief state.

**Output**: Current turn's belief state represented as a set of active slot-value pairs, with inactive slots assigned a 'none' value.

## Scoring recipe

```python
score = 0.0
for turn in dialogue:
    gold_change = compute_belief_state_change(gold_bs, prev_gold_bs)
    pred_change = compute_belief_state_change(pred_bs, prev_pred_bs)
    turn_score = partial_match_score(gold_change, pred_change)
    score += turn_score
final_metric = score / total_turns
```

## Common pitfalls

- Treating all turns equally regardless of how many slots actually changed, which masks model performance on dynamic belief states.
- Double-counting prediction errors when a model incorrectly predicts a slot value and fails to update it to 'none' in subsequent turns.
- Ignoring partial correctness, as traditional metrics often use a hard binary pass/fail for each turn or slot.

## Evidence (verbatim from paper)

> Granular Change Accuracy (GCA) introduces a novel DST evaluation metric that addresses three key flaws in existing metrics: turn-centric uniform scoring, lack of partial credit for partially correct turns, and double-counting of prediction errors. By focusing on per-turn belief state changes rather than raw slot or turn accuracy, GCA reduces bias from error timing and error distribution, particularly in few-shot/zero-shot settings where model errors are more concentrated.

## Citation

```bibtex
@misc{aksu2024granularchangeaccuracy,
  title={Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking},
  author={Aksu et al. (2024)},
  year={2024},
  note={arXiv:2403.11123}
}
```

- arXiv: 2403.11123