Coco Caption Re Ranking Eval

---
name: coco-caption-re-ranking-eval
description: Evaluates the ability of vision-language models to re-rank candidate image captions based on their semantic alignment with extracted visual context. It probes how well models can leverage object-level visual information to improve caption relevance and accuracy. Use when the user wants to benchmark on COCO Captions (Karpathy test split), or asks about evaluating this task. Reports BERTScore (B-S).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.08784
  bibtex_key: [REDACTED]
  confidence: high
---

# coco-caption-re-ranking-eval

> Visual Semantic Relatedness Dataset for Image Captioning — Sabir et al. (2023) (arXiv:2301.08784, 2023)

## What this evaluates

Evaluates the ability of vision-language models to re-rank candidate image captions based on their semantic alignment with extracted visual context. It probes how well models can leverage object-level visual information to improve caption relevance and accuracy.

## Datasets

- **COCO Captions (Karpathy test split)** — total ?; splits: test (-1); repo https://github.com/ahmedssabir/Textual-Visual-Semantic-Dataset

## Metrics

- `BERTScore (B-S)` **(primary)** — range: [0, 1]
  - Computes the mean cosine similarity between BERT embeddings of the predicted caption and the ground-truth reference caption.
- `BLEU-4` — range: [0, 1]
  - Measures n-gram precision up to 4-grams between predicted and reference captions, with a brevity penalty.
- `METEOR` — range: [0, 1]
  - Aligns predicted and reference captions using exact, stem, synonym, and paraphrase matches, then computes a weighted harmonic mean with a penalty for fragmentation.
- `ROUGE-L` — range: [0, 1]
  - Measures the longest common subsequence between predicted and reference captions to capture sentence-level structure.
- `CIDEr` — range: [0, 1]
  - Computes TF-IDF weighted n-gram similarity between predicted and reference captions, emphasizing rare but informative words.
- `SPICE` — range: [0, 1]
  - Evaluates caption quality by matching scene graphs (objects, attributes, relationships) between predicted and reference captions.

## Input / output format

**Input**: Candidate image captions (typically top-9 from beam search) paired with extracted visual context (top-3 object contexts or overlapping object-caption terms).

**Output**: Re-ranked list of candidate captions, or a binary/continuous semantic relatedness score [0,1] used for filtering and ranking.

## Scoring recipe

```python
def evaluate(gold_captions, pred_captions):
    # Standard COCO metrics
    bleu4 = coco_bleu4(gold_captions, pred_captions)
    meteor = coco_meteor(gold_captions, pred_captions)
    rouge_l = coco_rouge(gold_captions, pred_captions)
    cider = coco_cider(gold_captions, pred_captions)
    spice = coco_spice(gold_captions, pred_captions)
    # BERTScore
    pred_embeds = bert_model.encode(pred_captions)
    gold_embeds = bert_model.encode(gold_captions)
    b_score = cosine_similarity(pred_embeds, gold_embeds).mean()
    return {'BLEU-4': bleu4, 'METEOR': meteor, 'ROUGE-L': rouge_l, 'CIDEr': cider, 'SPICE': spice, 'BERTScore': b_score}
```

## Common pitfalls

- The re-ranking pipeline applies a similarity threshold (e.g., ≥ 0.2, 0.3, 0.4) to filter captions before ranking, which drastically changes final metric scores and is not always reported consistently.
- Visual context is extracted using external off-the-shelf classifiers (ResNet-152, CLIP, Faster R-CNN), so evaluation results conflate the re-ranker's performance with the detector's accuracy.
- Human evaluation is limited to 19 images and 12 subjects, yielding only a ~50% agreement rate that does not statistically validate the re-ranking improvements.

## Evidence (verbatim from paper)

> Evaluation Metric. We use the official COCO offline evaluation suite, producing several widely used caption quality metrics: BLEU [34] METEOR [4], ROUGE [29], CIDEr [41], SPICE [2] and the semantic-similarity based metric BERTScore (B-S) [43].

## Citation

```bibtex
@misc{sabir2023visualsemantic,
  title={Visual Semantic Relatedness Dataset for Image Captioning},
  author={Sabir et al. (2023)},
  year={2023},
  note={arXiv:2301.08784}
}
```

- arXiv: 2301.08784