Mmbe Eval

---
name: mmbe-eval
description: Evaluates the ability of vision-language models to generate unified multimodal embeddings for diverse tasks including classification, visual question answering, retrieval, and visual grounding. It probes zero-shot generalization to unseen datasets and the model's capacity to follow task-specific instructions for cross-modal alignment. Use when the user wants to benchmark on MMEB, or asks about evaluating this task. Reports Precision@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.05160
  bibtex_key: jiang2024vlm2vec
  confidence: high
---

# mmbe-eval

> VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks — Jiang et al. (2024) (arXiv:2410.05160, 2024)

## What this evaluates

Evaluates the ability of vision-language models to generate unified multimodal embeddings for diverse tasks including classification, visual question answering, retrieval, and visual grounding. It probes zero-shot generalization to unseen datasets and the model's capacity to follow task-specific instructions for cross-modal alignment.

## Datasets

- **MMEB** — total ?; splits: train (662000), test (-1)

## Metrics

- `Precision@1` **(primary)** — range: percent
  - Measures the ratio of positive candidates being ranked in the top place for all queries.

## Input / output format

**Input**: Image and text pairs (query and target), optionally accompanied by task-specific instructions.

**Output**: Fixed-dimensional embedding vectors representing the image-text combination.

## Scoring recipe

```python
def precision_at_1(predictions, gold):
    correct = 0
    for pred, gold_idx in zip(predictions, gold):
        if pred[0] == gold_idx:
            correct += 1
    return correct / len(gold)
```

## Common pitfalls

- Long text queries or targets may be truncated due to the text encoder's length limitations in baseline models.
- Instruction usage significantly impacts performance differently for CLIP vs VLM2Vec backbones, so results must be reported with and without instructions.
- Shallow fusion baselines (e.g., score-level addition) do not leverage deep cross-modal integration, making direct comparison with deep-fusion models potentially unfair without careful configuration.

## Evidence (verbatim from paper)

> We report Precision@1 for all models in Table [2]. It measures the ratio of positive candidates being ranked in the top place for all queries.

## Citation

```bibtex
@misc{jiang2024vlm2vec,
  title={VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks},
  author={Jiang et al. (2024)},
  year={2024},
  note={arXiv:2410.05160}
}
```

- arXiv: 2410.05160