Visdial Eval

---
name: visdial-eval
description: Evaluates an AI agent's ability to maintain conversational context, resolve co-references, and ground follow-up questions in visual content. The task requires ranking a set of candidate answers based on an image and dialog history. Use when the user wants to benchmark on VisDial v0.9, or asks about evaluating this task. Reports MRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1611.08669
  bibtex_key: das2016visualdialog
  confidence: high
---

# visdial-eval

> Visual Dialog — Abhishek Das et al. (arXiv:1611.08669, 2016)

## What this evaluates

Evaluates an AI agent's ability to maintain conversational context, resolve co-references, and ground follow-up questions in visual content. The task requires ranking a set of candidate answers based on an image and dialog history.

## Datasets

- **VisDial v0.9** — total 123000; splits: train (80000), val (3000), test (40000)

## Metrics

- `MRR` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank: the average of 1/rank for the correct answer across all instances, where rank is the position of the ground-truth answer in the model's ranked list of 10 options.

## Input / output format

**Input**: An image, a sequence of prior dialog turns (question-answer pairs), a current question, and a fixed set of 10 candidate answer options.

**Output**: A score or probability for each of the 10 candidate answers, used to produce a ranked list.

## Scoring recipe

```python
def compute_metrics(predictions, gold_indices):
    ranks = []
    for scores, gold in zip(predictions, gold_indices):
        sorted_indices = np.argsort(-scores)
        rank = np.where(sorted_indices == gold)[0][0] + 1
        ranks.append(rank)
    mrr = np.mean(1.0 / ranks)
    r1 = np.mean([r <= 1 for r in ranks])
    r5 = np.mean([r <= 5 for r in ranks])
    r10 = np.mean([r <= 10 for r in ranks])
    mean_rank = np.mean(ranks)
    return {'MRR': mrr, 'R@1': r1, 'R@5': r5, 'R@10': r10, 'Mean': mean_rank}
```

## Common pitfalls

- Mean Rank is inversely scaled (lower is better), unlike MRR and Recall@k which are higher-is-better.
- The task is closed-set ranking over 10 predefined options, not open-ended generation; models must output scores for all options.
- Ignoring dialog history significantly degrades performance, as shown by the gap between LF-Q and LF-QH/LF-QIH models.

## Evidence (verbatim from paper)

> Table 1: Performance of methods on VisDial v0.9, measured by mean reciprocal rank (MRR), recall@k and mean rank. Higher is better for MRR and recall@k, while lower is better for mean rank.

## Citation

```bibtex
@misc{das2016visualdialog,
  title={Visual Dialog},
  author={Abhishek Das et al.},
  year={2016},
  note={arXiv:1611.08669}
}
```

- arXiv: 1611.08669