Multimodal Retrieval Eval

---
name: multimodal-retrieval-eval
description: Evaluates fine-grained and coarse-grained cross-modal retrieval capabilities across text, image, and video modalities. It probes a model's ability to align and retrieve relevant visual or textual content given a query from a different modality, including instruction-based queries. Use when the user wants to benchmark on CaReBench, ShareGPT4V, Urban1K, DOCCI, WebVid-CoVR, MMEB, Flickr30K, MSR-VTT, MSVD, DiDeMo, or asks about evaluating this task. Reports Recall@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19650
  bibtex_key: kong2025modality
  confidence: high
---

# multimodal-retrieval-eval

> Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval — Kong et al. (2025) (arXiv:2505.19650, 2025)

## What this evaluates

Evaluates fine-grained and coarse-grained cross-modal retrieval capabilities across text, image, and video modalities. It probes a model's ability to align and retrieve relevant visual or textual content given a query from a different modality, including instruction-based queries.

## Datasets

- **CaReBench** — total ?; splits: test (-1)
- **ShareGPT4V** — total ?; splits: test (-1)
- **Urban1K** — total ?; splits: test (-1)
- **DOCCI** — total ?; splits: test (-1)
- **WebVid-CoVR** — total ?; splits: test (-1)
- **MMEB** — total ?; splits: test (-1)
- **Flickr30K** — total ?; splits: test (-1)
- **MSR-VTT** — total ?; splits: test (-1)
- **MSVD** — total ?; splits: test (-1)
- **DiDeMo** — total ?; splits: test (-1)

## Metrics

- `Recall@1` **(primary)** — range: percent
  - The fraction of queries for which the ground-truth matching item appears in the top-1 retrieved result.
- `Recall@5` — range: percent
  - The fraction of queries for which the ground-truth matching item appears in the top-5 retrieved results.
- `Recall@10` — range: percent
  - The fraction of queries for which the ground-truth matching item appears in the top-10 retrieved results.
- `Recall@50` — range: percent
  - The fraction of queries for which the ground-truth matching item appears in the top-50 retrieved results.
- `Average Score` — range: percent
  - The arithmetic mean of in-distribution (IND) and out-of-distribution (OOD) scores across meta-tasks on the MMEB benchmark.

## Input / output format

**Input**: Text queries paired with candidate images or videos, or visual queries paired with candidate text descriptions.

**Output**: A ranked list of retrieved candidates or similarity scores for each query-candidate pair.

## Scoring recipe

```python
def recall_at_k(retrieved_indices, gold_index, k):
    top_k = retrieved_indices[:k]
    return 1.0 if gold_index in top_k else 0.0
# Average across all queries in the test set
```

## Common pitfalls

- Confusing text-to-visual (T→V) versus visual-to-text (V→T) retrieval directions, which often show asymmetric performance.
- Failing to distinguish between fine-grained (e.g., CaReBench) and coarse-grained (e.g., MSR-VTT) tasks, as fine-grained benchmarks demand significantly higher precision.
- Reporting only R@1 when the protocol explicitly notes that R@5, R@10, and R@50 are also standard for robustness evaluation.

## Evidence (verbatim from paper)

> All experimental results are reported in Recall@1 unless otherwise specified.

## Citation

```bibtex
@misc{kong2025modality,
  title={Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval},
  author={Kong et al. (2025)},
  year={2025},
  note={arXiv:2505.19650}
}
```

- arXiv: 2505.19650