Image Captioning Retrieval Eval

---
name: image-captioning-retrieval-eval
description: Evaluates vision-language models on image captioning and image-text retrieval tasks to measure zero-shot and fine-tuned generalization on long-tail visual concepts and out-of-domain data. Use when the user wants to benchmark on nocaps, COCO Captions, Flickr30K, LocNar Flickr30K, or asks about evaluating this task. Reports CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.08981
  bibtex_key: changpinyo2021conceptual12m
  confidence: high
---

# image-captioning-retrieval-eval

> Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts — Changpinyo et al. (2021) (arXiv:2102.08981, 2021)

## What this evaluates

Evaluates vision-language models on image captioning and image-text retrieval tasks to measure zero-shot and fine-tuned generalization on long-tail visual concepts and out-of-domain data.

## Datasets

- **nocaps** — total ?; splits: test (-1)
- **COCO Captions** — total ?; splits: val2017 (-1)
- **Flickr30K** — total ?; splits: test (-1)
- **LocNar Flickr30K** — total ?; splits: test (-1)

## Metrics

- `CIDEr` **(primary)** — range: other
  - Computes n-gram TF-IDF weighted cosine similarity between predicted and ground-truth captions against a reference corpus. Higher scores indicate better semantic and lexical alignment.
- `BLEU-4` — range: [0, 1]
  - Geometric mean of modified precision scores for 1- to 4-grams, with brevity penalty to discourage overly short outputs.
- `SPICE` — range: [0, 1]
  - Measures semantic proposition overlap between predicted and ground-truth captions using scene graph parsing.
- `R1` — range: [0, 1]
  - Fraction of queries where the ground-truth match appears in the top-1 ranked results.
- `R5` — range: [0, 1]
  - Fraction of queries where the ground-truth match appears in the top-5 ranked results.
- `R10` — range: [0, 1]
  - Fraction of queries where the ground-truth match appears in the top-10 ranked results.

## Input / output format

**Input**: Image file and optional text prompt/context for captioning; image-text pairs for retrieval.

**Output**: Natural language caption string for each image; ranked list of matching texts/images for retrieval.

## Scoring recipe

```python
def compute_cider(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        scores.append(cider_metric.compute([pred], [gold]))
    return sum(scores) / len(scores)

def compute_recall_at_k(predictions, golds, k):
    hits = sum(1 for pred_list, gold in zip(predictions, golds) if gold in pred_list[:k])
    return hits / len(predictions)
```

## Common pitfalls

- Zero-shot captioning scores are artificially low because models lack the COCO captioning style; fine-tuning is required to match automatic metric expectations.
- Over-fine-tuning on in-domain data (COCO) degrades out-of-domain generalization, causing a trade-off between in-domain metrics and novel concept recognition.
- BLEU-4 and SPICE are highly sensitive to exact lexical overlap, which may not reflect semantic alignment for long-tail concepts.

## Evidence (verbatim from paper)

> With a fine-tuned model, the benefit of transfer learning using pre-training on this task is clear (Row 1 vs. Rows 4,5,6), with CC12M outperforming CC3M by +14.2 CIDEr points and another +2.8 with CC3M+CC12M.

## Citation

```bibtex
@misc{changpinyo2021conceptual12m,
  title={Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts},
  author={Changpinyo et al. (2021)},
  year={2021},
  note={arXiv:2102.08981}
}
```

- arXiv: 2102.08981