Dvd Dst Eval

---
name: dvd-dst-eval
description: Evaluates a model's ability to track visual objects and their attributes across turns in video-grounded dialogues. It probes long-term cross-modal dependency resolution and precise state decoding under controlled, bias-free synthetic dialogue conditions. Use when the user wants to benchmark on DVD-DST, or asks about evaluating this task. Reports Joint Acc.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.07898
  bibtex_key: le2022multimodal
  confidence: high
---

# dvd-dst-eval

> Multimodal Dialogue State Tracking — Le et al. (2022) (arXiv:2206.07898, 2022)

## What this evaluates

Evaluates a model's ability to track visual objects and their attributes across turns in video-grounded dialogues. It probes long-term cross-modal dependency resolution and precise state decoding under controlled, bias-free synthetic dialogue conditions.

## Datasets

- **DVD-DST** — total 13992; splits: train (9295), val (3326), test (1371); repo https://github.com/henryhungle/mm.dst

## Metrics

- `Joint Acc` **(primary)** — range: percent
  - Exact match accuracy over all discrete slot values in a dialogue turn. A prediction is counted as correct only when every component value exactly matches the oracle values.
- `Acc IoU @0.5` — range: percent
  - Accuracy for continuous temporal slots (start/end time) using Intersection-over-Union. A prediction is correct if the IoU between the predicted and ground-truth segment exceeds 0.5.
- `Obj Ident. F1` — range: percent
  - F1 score computed over object identity predictions across the dataset.

## Input / output format

**Input**: Dialogue context (textual turns) and video context (object-level and segment-level visual features or frames).

**Output**: A dialogue state represented as a set of tuples (object_id, slot, value) for discrete attributes (size, color, material, shape) and (s_start, s_end) for temporal boundaries.

## Scoring recipe

```python
def compute_joint_acc(pred_state, gold_state):
    # pred_state and gold_state are dicts/lists of (slot, value) tuples
    discrete_pred = {k: v for k, v in pred_state if k != 'temporal'}
    discrete_gold = {k: v for k, v in gold_state if k != 'temporal'}
    discrete_match = discrete_pred == discrete_gold
    
    pred_temporal = pred_state.get('temporal', (0, 0))
    gold_temporal = gold_state.get('temporal', (0, 0))
    iou = intersection_over_union(pred_temporal, gold_temporal)
    temporal_match = iou >= 0.5
    
    return 1.0 if discrete_match and temporal_match else 0.0
```

## Common pitfalls

- Existing multimodal dialogue benchmarks suffer from strong distribution and annotator biases, allowing models to ignore dialogue context; DVD-DST was specifically designed to remove these biases.
- Continuous temporal slots require IoU thresholding (0.5 or 0.7) rather than exact match, which can mask poor temporal localization if thresholds are too lenient.
- Component F1 scores can be artificially inflated if models output all possible object identities and slot values, so joint exact-match accuracy is a stricter and more reliable metric.

## Evidence (verbatim from paper)

> We reported the joint state accuracy of discrete slots only ("Joint Acc") as well as all slot values ("Joint Acc IoU@p"). For continuous slots, we followed (Hu et al., 2016; Gao et al., 2017) by using Intersection-over-Union (IoU) between predicted temporal segment and ground-truth segment. The predicted segment is counted as correct if its IoU with the oracle is more than p, where we chose p = {0.5, 0.7}.

## Citation

```bibtex
@misc{le2022multimodal,
  title={Multimodal Dialogue State Tracking},
  author={Le et al. (2022)},
  year={2022},
  note={arXiv:2206.07898}
}
```

- arXiv: 2206.07898