Vdialogue Eval

---
name: vdialogue-eval
description: Evaluates visually-grounded dialogue systems across five core tasks: multi-modal intent prediction, dialog retrieval (text-to-image and image-to-text), dialog state tracking, and response generation. It provides a unified hierarchical scoring metric to compare cross-task performance and generalization. Use when the user wants to benchmark on VisDial, PhotoChat, MMDialog, Image-Chat, or asks about evaluating this task. Reports VDscore.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.07387
  bibtex_key: li2023vdialogue
  confidence: high
---

# vdialogue-eval

> VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialogue — Yunshui Li et al. (2023) (arXiv:2309.07387, 2023)

## What this evaluates

Evaluates visually-grounded dialogue systems across five core tasks: multi-modal intent prediction, dialog retrieval (text-to-image and image-to-text), dialog state tracking, and response generation. It provides a unified hierarchical scoring metric to compare cross-task performance and generalization.

## Datasets

- **VisDial** — total ?; splits: test (-1)
- **PhotoChat** — total ?; splits: test (-1)
- **MMDialog** — total ?; splits: test (-1)
- **Image-Chat** — total ?; splits: test (-1)

## Metrics

- `VDscore` **(primary)** — range: [0, 1]
  - A hierarchical evaluation metric based on the Analytic Hierarchy Process (AHP) that aggregates task-specific performance scores into a single comprehensive score using weighted criteria.
- `R@1, R@5, R@10` — range: percent
  - Recall at K, measuring the proportion of instances where the ground-truth item appears within the top K ranked predictions.

## Input / output format

**Input**: Multi-modal dialogue context comprising text history and associated images, with task-specific prompts (e.g., candidate sets for retrieval, intent/state labels for prediction, or generation targets).

**Output**: Task-dependent: ranked list of images or text, predicted intent/state labels, or generated dialogue responses.

## Scoring recipe

```python
def compute_vdscore(task_scores):
    # AHP-based weighted aggregation of task-specific metrics
    return ahp_aggregate(task_scores)

def compute_recall_at_k(predictions, gold, k):
    return 1.0 if gold in predictions[:k] else 0.0
```

## Common pitfalls

- VisDial exhibits a distribution bias towards image content, causing models to ignore dialogue context.
- Annotator bias can create spurious causal links between dialogue context and output responses.
- Models struggle to differentiate correct images from visually similar candidates in retrieval tasks.
- Concatenating long dialogue history with short candidate answers equally degrades text retrieval performance.

## Evidence (verbatim from paper)

> Specifically, we found that our model also achieved consistent improvement in the comprehensive evaluation of VDscore.

## Citation

```bibtex
@misc{li2023vdialogue,
  title={VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialogue},
  author={Yunshui Li et al. (2023)},
  year={2023},
  note={arXiv:2309.07387}
}
```

- arXiv: 2309.07387