Mmeb Eval

---
name: mmeb-eval
description: Evaluates the cross-modal alignment and generalization capabilities of multimodal embedding models across classification, VQA, retrieval, and visual grounding tasks. Use when the user wants to benchmark on MMEB, or asks about evaluating this task. Reports Precision@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.08468
  bibtex_key: chen2025mme5
  confidence: high
---

# mmeb-eval

> mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data — Chen et al. (2025) (arXiv:2502.08468, 2025)

## What this evaluates

Evaluates the cross-modal alignment and generalization capabilities of multimodal embedding models across classification, VQA, retrieval, and visual grounding tasks.

## Datasets

- **MMEB** — total ?; splits: test (-1)

## Metrics

- `Precision@1` **(primary)** — range: [0, 1]
  - The fraction of queries where the top-1 retrieved image matches the ground truth label or description.
- `Recall@10` — range: [0, 1]
  - The fraction of queries where the ground truth image appears in the top-10 retrieved results.

## Input / output format

**Input**: Image and corresponding text query/prompt for each task type (classification, VQA, retrieval, visual grounding).

**Output**: Dense embedding vectors for the image and text, used for similarity matching.

## Scoring recipe

```python
def compute_precision_at_1(img_embs, txt_embs, labels):
    sims = cosine_similarity(img_embs, txt_embs)
    preds = argmax(sims, axis=1)
    return mean(preds == labels)

def compute_recall_at_10(img_embs, txt_embs, labels):
    sims = cosine_similarity(img_embs, txt_embs)
    top_k = argsort(sims, axis=1, descending=True)[:, :10]
    return mean(labels in top_k)
```

## Common pitfalls

- Distinguishing between zero-shot evaluation (using only synthetic data) and supervised evaluation (using in-distribution training data) is critical for fair comparison.
- The synthetic dataset distribution is intentionally skewed 1:1:2 for classification:VQA:retrieval, which affects generalization compared to models trained on pure retrieval data.

## Evidence (verbatim from paper)

> We evaluate the general embedding performance in terms of Precision@1 on the MMEB benchmark*Jiang et al. ([2024b])*. This benchmark comprises 36 multimodal embedding tasks across four categories: classification (10), VQA (10), retrieval (12), and visual grounding (4). ... we conduct experiments on seven languages of XTD and report Recall@10 results.

## Citation

```bibtex
@misc{chen2025mme5,
  title={mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2502.08468}
}
```

- arXiv: 2502.08468