Image Captioning Eval

---
name: image-captioning-eval
description: Evaluates a model's ability to generate coherent natural language descriptions for images and align specific image regions with corresponding text segments. It measures both retrieval quality and generation fidelity against human-written references. Use when the user wants to benchmark on Flickr8K, Flickr30K, MSCOCO, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1412.2306
  bibtex_key: karpathy2014deep
  confidence: high
---

# image-captioning-eval

> Deep Visual-Semantic Alignments for Generating Image Descriptions — Karpathy et al. (2014) (arXiv:1412.2306, 2014)

## What this evaluates

Evaluates a model's ability to generate coherent natural language descriptions for images and align specific image regions with corresponding text segments. It measures both retrieval quality and generation fidelity against human-written references.

## Datasets

- **Flickr8K** — total 8000; splits: train (-1), val (1000), test (1000)
- **Flickr30K** — total 31000; splits: train (-1), val (1000), test (1000)
- **MSCOCO** — total 123000; splits: train (-1), val (5000), test (5000)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Computes n-gram precision (up to 4-grams) with a brevity penalty against up to 5 human reference sentences per image.
- `METEOR` — range: percent
  - Measures alignment between candidate and reference sentences using synonymy, stemming, and exact matches, normalized against 5 references.
- `CIDEr` — range: percent
  - Computes TF-IDF weighted n-gram similarity between candidate and reference sentences, normalized against 5 references.
- `Recall@K` — range: [0, 1]
  - Measures the fraction of times a correct image or sentence is found among the top K results in a ranked retrieval list.
- `Median Rank` — range: rank
  - Reports the median position of the ground truth item in the sorted list of retrieved candidates.

## Input / output format

**Input**: Image (full frame or cropped region) and optionally a reference sentence for alignment tasks.

**Output**: Generated natural language sentence (for captioning) or ranked list of image-sentence pairs with alignment scores (for alignment).

## Scoring recipe

```python
def compute_generation_metrics(predictions, references):
    # references is a list of 5 human sentences per image
    bleu = compute_bleu(predictions, references, max_n=4)
    meteor = compute_meteor(predictions, references)
    cider = compute_cider(predictions, references)
    return bleu, meteor, cider

def compute_alignment_metrics(scores, ground_truth_indices):
    ranked_indices = argsort(scores, descending=True)
    median_rank = median([ranked_indices.tolist().index(gt) + 1 for gt in ground_truth_indices])
    recall_k = mean([1 if gt in ranked_indices[:K] else 0 for gt in ground_truth_indices])
    return median_rank, recall_k
```

## Common pitfalls

- Using fewer than 5 reference sentences per image violates the coco-caption evaluation standard used in the paper.
- Confusing full-image captioning results with region-level captioning results, as the region model is evaluated on a separate 200-image AMT-annotated test split.
- Comparing BLEU scores across models without accounting for sentence length differences, as the paper notes longer sentences can artificially lower BLEU despite higher semantic accuracy.

## Evidence (verbatim from paper)

> We report the BLEU, METEOR and CIDEr scores computed with the coco-caption code 222https://github.com/tylin/coco-caption. Each method evaluates a candidate sentence by measuring how well it matches a set of five reference sentences written by humans.

## Citation

```bibtex
@misc{karpathy2014deep,
  title={Deep Visual-Semantic Alignments for Generating Image Descriptions},
  author={Karpathy et al. (2014)},
  year={2014},
  note={arXiv:1412.2306}
}
```

- arXiv: 1412.2306