Mmmeb Eval

---
name: mmmeb-eval
description: Evaluates cross-lingual and cross-modal embedding alignment for image-text retrieval, classification, visual question answering, and visual grounding. Probes whether multilingual adaptation preserves semantic consistency across languages and modalities without degrading English performance. Use when the user wants to benchmark on MMMEB, or asks about evaluating this task. Reports P@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.09313
  bibtex_key: [REDACTED]
  confidence: high
---

# mmmeb-eval

> xVLM2Vec: Adapting LVLM-based embedding models to multilinguality using Self-Knowledge Distillation — Musacchio et al. (2025) (arXiv:2503.09313, 2025)

## What this evaluates

Evaluates cross-lingual and cross-modal embedding alignment for image-text retrieval, classification, visual question answering, and visual grounding. Probes whether multilingual adaptation preserves semantic consistency across languages and modalities without degrading English performance.

## Datasets

- **MMMEB** — total ?; splits: test (-1); repo https://github.com/swapUniba/xVLM2Vec

## Metrics

- `P@1` **(primary)** — range: percent
  - Precision at 1: for each query, similarity scores are computed between the query embedding and all candidate target embeddings. The metric returns 1 if the ground-truth target is ranked first, and 0 otherwise. Results are averaged across queries and reported as AVG-3 (I2T, T2I, C) or AVG (all tasks).

## Input / output format

**Input**: Multimodal inputs formatted as specific instruction prompts. Images are represented via the special token <$|$image_1$|$> within text strings. Queries and targets follow task-specific templates (e.g., I2T: <$|$image_1$|$>\nFind an image caption...). Plain formatting removes trailing punctuation; punctuation formatting adds a period or question mark.

**Output**: A dense embedding vector extracted from the last token of the model's output sequence (not the special EOS token).

## Scoring recipe

```python
def compute_p1(query_emb, target_embs, correct_idx):
    sims = cosine_similarity(query_emb, target_embs)
    rank = argsort(descending=True)[correct_idx]
    return 1.0 if rank == 0 else 0.0
# Average over all queries in the dataset
```

## Common pitfalls

- Input formatting drastically affects performance; plain vs punctuation formatting can change P@1 by 10-40 points.
- CLIP/SigLIP models cannot be evaluated on VQA/VG tasks due to missing instruction-tuning and modality constraints.
- Non-English tasks rely on translated prompts/targets, so translation quality directly impacts scores.
- Embeddings are extracted from the last token, not a pooled or special token, making tokenization sensitive.

## Evidence (verbatim from paper)

> We use P@1 as the evaluation metric, following the same setup used in the MMEB benchmark. We report two averages, one for the tasks that are covered by all models (AVG-3), that are I2T, T2I and C, and one for all tasks (AVG).

## Citation

```bibtex
@misc{musacchio2025xvlm2vec,
  title={xVLM2Vec: Adapting LVLM-based embedding models to multilinguality using Self-Knowledge Distillation},
  author={Musacchio et al. (2025)},
  year={2025},
  note={arXiv:2503.09313}
}
```

- arXiv: 2503.09313