Mscoco Eval

---
name: mscoco-eval
description: Evaluates image captioning quality by scoring generated captions against crowdworker references and expert THumB 1.0 scores. It measures how well automatic metrics correlate with human judgments across different captioning systems. Use when the user wants to benchmark on MSCOCO, or asks about evaluating this task. Reports RefCLIP-S.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.04139
  bibtex_key: [REDACTED]
  confidence: high
---

# mscoco-eval

> Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand — First Author et al. (2021) (arXiv:2112.04139, 2021)

## What this evaluates

Evaluates image captioning quality by scoring generated captions against crowdworker references and expert THumB 1.0 scores. It measures how well automatic metrics correlate with human judgments across different captioning systems.

## Datasets

- **MSCOCO** — total ?; splits: test (-1)

## Metrics

- `RefCLIP-S` **(primary)** — range: [0, 1]
  - A reference-based metric that uses CLIP image and text features to compute similarity between image and caption.
- `RefOnlyC` — range: [0, 1]
  - A variant of RefCLIP-S that removes image features and only uses reference text features from CLIP.
- `CIDEr` — range: [0, 1]
  - A reference-based metric that computes TF-IDF weighted n-gram similarity between hypothesis and references.

## Input / output format

**Input**: Image and optionally reference captions.

**Output**: Generated caption.

## Scoring recipe

```python
scores = []
for img, refs, caption in test_images:
    metric_score = compute_metric(caption, refs, img)
    human_score = average(expert_THumB_scores[caption])
    scores.append((metric_score, human_score))
return pearson_correlation([s[0] for s in scores], [s[1] for s in scores])
```

## Common pitfalls

- Using RefCLIP-S instead of RefOnlyC when isolating text-only evaluation quality, as RefCLIP-S incorporates image features.
- Not following the standard Karpathy split for train/test partitioning.

## Evidence (verbatim from paper)

> Table 1: Summary of Billboards as of Jan. 10, 2022. ... Top Metric: RefCLIP-S ... Ensemble of Metrics: 2.08·RefCLIP-S+1.51·RefOnlyC+0.82·CIDEr ... We use the MSCOCO dataset... We apply the standard Karpathy split. For each of 500 test images, rubric-based evaluations (THumB 1.0) are available for five systems... we use all five crowdworker captions as a reference set for reference-based metrics.

## Citation

```bibtex
@misc{first2021billboard,
  title={Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand},
  author={First Author et al. (2021)},
  year={2021},
  note={arXiv:2112.04139}
}
```

- arXiv: 2112.04139