Crossmodal 3600 Eval

---
name: crossmodal-3600-eval
description: Evaluates multilingual image captioning models across 36 languages, probing their ability to generate stylistically coherent and culturally representative descriptions without relying on direct translation artifacts. It measures how well models generalize to low-resource and geographically diverse languages. Use when the user wants to benchmark on Crossmodal-3600, or asks about evaluating this task. Reports CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.12522
  bibtex_key: thapliyal2022crossmodal3600
  confidence: high
---

# crossmodal-3600-eval

> Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset — Thapliyal et al. (2022) (arXiv:2205.12522, 2022)

## What this evaluates

Evaluates multilingual image captioning models across 36 languages, probing their ability to generate stylistically coherent and culturally representative descriptions without relying on direct translation artifacts. It measures how well models generalize to low-resource and geographically diverse languages.

## Datasets

- **Crossmodal-3600** — total 3600; splits: test (3600)

## Metrics

- `CIDEr` **(primary)** — range: [0, 1]
  - CIDEr (Consensus-based Image Description Evaluation) computes n-gram overlap between generated captions and reference captions, weighted by TF-IDF scores to downweight common words and upweight distinctive terms.

## Input / output format

**Input**: An image and a target language code.

**Output**: A natural language caption in the specified target language.

## Scoring recipe

```python
def compute_cider(predictions, references):
    scores = []
    for pred, refs in zip(predictions, references):
        pred_ngrams = get_ngrams(pred, max_n=4)
        ref_ngrams = [get_ngrams(r, max_n=4) for r in refs]
        tfidf = compute_tfidf(ref_ngrams)
        sims = [cosine_similarity(pred_ngrams, ref_ngrams, tfidf) for ref_ngrams in ref_ngrams]
        scores.append(max(sims))
    return sum(scores) / len(scores)
```

## Common pitfalls

- COCO-dev scores in this benchmark are computed using machine-translated references rather than human-written ones, which can artificially inflate CIDEr scores compared to standard English evaluations.
- CIDEr values can exceed 1.0; scores above 1 indicate strong n-gram overlap and are valid, not errors.
- Evaluations are reported per language, so averaging across languages without weighting by dataset size or language difficulty can mask performance disparities.

## Evidence (verbatim from paper)

> Table 8: CIDEr on XM3600 and COCO-dev for the best performing model BB+CC on all 35 languages. (COCO-dev computed using machine-translated references).

## Citation

```bibtex
@misc{thapliyal2022crossmodal3600,
  title={Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset},
  author={Thapliyal et al. (2022)},
  year={2022},
  note={arXiv:2205.12522}
}
```

- arXiv: 2205.12522