Image Chat Eval

---
name: image-chat-eval
description: Evaluates multimodal conversational models on their ability to generate or retrieve engaging, style-conditioned responses grounded in images and dialogue history. It probes retrieval accuracy, generation quality, and human-perceived engagement in multi-turn image-grounded conversations. Use when the user wants to benchmark on IMAGE-CHAT, or asks about evaluating this task. Reports R@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1811.00945
  bibtex_key: shuster2018imagechat
  confidence: high
---

# image-chat-eval

> Image Chat: Engaging Grounded Conversations — Shuster et al. (2018) (arXiv:1811.00945, 2018)

## What this evaluates

Evaluates multimodal conversational models on their ability to generate or retrieve engaging, style-conditioned responses grounded in images and dialogue history. It probes retrieval accuracy, generation quality, and human-perceived engagement in multi-turn image-grounded conversations.

## Datasets

- **IMAGE-CHAT** — total ?; splits: train (-1), test (-1)

## Metrics

- `R@1` **(primary)** — range: percent
  - Recall at rank 1: fraction of samples where the true label is ranked first out of a fixed pool of 100 candidates (1 ground truth + 99 random test samples).
- `ROUGE-L` — range: [0, 1]
  - Standard ROUGE-L recall metric measuring the longest common subsequence between generated and reference utterances.
- `BLEU-4` — range: [0, 1]
  - Standard BLEU-4 metric for n-gram precision up to 4-grams.
- `Human Win Rate` — range: percent
  - Percentage of pairwise comparisons where the model's utterance is preferred over the human's utterance by annotators as 'more engaging'.

## Input / output format

**Input**: Image, style trait, and dialogue history (for turns 2+). For retrieval, a candidate pool of 100 utterances (1 ground truth + 99 random from test set) is provided.

**Output**: Retrieved utterance from the candidate pool, or a generated text utterance.

## Scoring recipe

```python
def calc_r_at_1(predictions, gold):
    # predictions: list of 100 candidate indices ranked by model
    # gold: index of true label in the 100 candidates
    return int(gold in predictions[:1]) / len(predictions)

def calc_rouge_l(generated, reference):
    return compute_rouge_l(generated, reference)

def calc_bleu_4(generated, reference):
    return compute_bleu_4(generated, reference)

def calc_human_win_rate(model_wins, total_comparisons):
    return (model_wins / total_comparisons) * 100
```

## Common pitfalls

- Automatic retrieval evaluation uses a fixed pool of 100 candidates (1 ground truth + 99 random test samples), not the full training set.
- Human evaluations are conducted on a separate set of 500 images not present in the IMAGE-CHAT training data.
- Transfer to the IGC task is performed zero-shot without fine-tuning on the target dataset.

## Evidence (verbatim from paper)

> We measure recall at 1 and 5 (R@1/100 and R@5/100) retrieval metrics, where for each sample there are 100 candidates to rank: 99 random candidates chosen from the test set, and the true label.

## Citation

```bibtex
@misc{shuster2018imagechat,
  title={Image Chat: Engaging Grounded Conversations},
  author={Shuster et al. (2018)},
  year={2018},
  note={arXiv:1811.00945}
}
```

- arXiv: 1811.00945