M Beir Eval

---
name: m-beir-eval
description: Evaluates multimodal information retrieval models across eight heterogeneous query-to-candidate modalities (text, image, image-text pairs) using instruction-tuned and fine-tuned vision-language models. Probes zero-shot generalization, cross-modality alignment, and the impact of instruction tuning on retrieval accuracy in large-scale candidate pools. Use when the user wants to benchmark on M-BEIR, or asks about evaluating this task. Reports Recall@5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.17136
  bibtex_key: wei2023uniir
  confidence: high
---

# m-beir-eval

> UniIR: Training and Benchmarking Universal Multimodal Information Retrievers — Wei et al. (2023) (arXiv:2311.17136, 2023)

## What this evaluates

Evaluates multimodal information retrieval models across eight heterogeneous query-to-candidate modalities (text, image, image-text pairs) using instruction-tuned and fine-tuned vision-language models. Probes zero-shot generalization, cross-modality alignment, and the impact of instruction tuning on retrieval accuracy in large-scale candidate pools.

## Datasets

- **M-BEIR** — total 5600000; splits: train (-1), test (-1)

## Metrics

- `Recall@5` **(primary)** — range: [0, 1]
  - Fraction of queries where the ground-truth candidate appears in the top-5 retrieved results. For Fashion200K and FashionIQ, Recall@10 is reported instead.

## Input / output format

**Input**: Query consisting of text, image, or image-text pair, accompanied by a task-specific instruction string. Candidate pool containing images, texts, or image-text pairs.

**Output**: Ranked list of candidate IDs/embeddings, typically returning the top-5 (or top-10 for specific datasets) retrieved candidates.

## Scoring recipe

```python
def recall_at_k(predictions, gold, k=5):
    top_k = set(predictions[:k])
    return len(top_k.intersection(gold)) / len(gold)
```

## Common pitfalls

- Zero-shot models fail on heterogeneous global pools without task instructions, often retrieving wrong modalities.
- Score-level fusion (CLIP) generally outperforms feature-level fusion (BLIP) for instruction-tuned models.
- Recall@10 is used instead of Recall@5 for Fashion200K and FashionIQ datasets.

## Evidence (verbatim from paper)

> We report the main results on M-BEIR in Table 2, where models retrieve candidates from the 5.6M pool. ... The average Recall@5 has increased by 12.8 and 10.9, respectively.

## Citation

```bibtex
@misc{wei2023uniir,
  title={UniIR: Training and Benchmarking Universal Multimodal Information Retrievers},
  author={Wei et al. (2023)},
  year={2023},
  note={arXiv:2311.17136}
}
```

- arXiv: 2311.17136