Mieb Eval

---
name: mieb-eval
description: MIEB evaluates the diverse capabilities of image and image-text embedding models across 130 tasks spanning retrieval, document understanding, classification, clustering, compositionality, and visual question answering. It probes zero-shot generalization, multilingual understanding, spatial/depth reasoning, and the model's ability to encode visual representations of text. Use when the user wants to benchmark on MIEB, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.10471
  bibtex_key: xiao2025mieb
  confidence: high
---

# mieb-eval

> MIEB: Massive Image Embedding Benchmark — Xiao et al. (2025) (arXiv:2504.10471, 2025)

## What this evaluates

MIEB evaluates the diverse capabilities of image and image-text embedding models across 130 tasks spanning retrieval, document understanding, classification, clustering, compositionality, and visual question answering. It probes zero-shot generalization, multilingual understanding, spatial/depth reasoning, and the model's ability to encode visual representations of text.

## Datasets

- **MIEB** — total ?; splits: test (-1)

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. It measures the quality of a ranked list of retrieved items by comparing the actual relevance scores to an ideal sorted list, discounting lower-ranked relevant items logarithmically.
- `accuracy` — range: [0, 1]
  - The proportion of correctly predicted class labels or top-1 matches between image embeddings and text prompt embeddings.
- `Normalized Mutual Information (NMI)` — range: [0, 1]
  - A clustering evaluation metric that measures the agreement between predicted cluster assignments and ground truth labels, normalized by the entropy of both distributions.
- `Spearman correlation` — range: [-1, 1]
  - A rank-based correlation coefficient measuring the monotonic relationship between predicted embedding similarity scores and human annotations.

## Input / output format

**Input**: Pairs or sets of images and texts (queries and documents), which may be interleaved or multilingual. For classification/clustering tasks, images with class labels or text prompts are provided.

**Output**: Fixed-dimensional embedding vectors for each image and text instance. For classification tasks, predicted class labels or similarity scores between image embeddings and text prompt embeddings.

## Scoring recipe

```python
def compute_ndcg_at_k(gold_relevance, k=10):
    import math
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(gold_relevance[:k]))
    ideal = sorted(gold_relevance, reverse=True)[:k]
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0

# Usage: predictions = model.encode(query), gold = relevance_scores
# metric_value = compute_ndcg_at_k(predictions, k=10)
```

## Common pitfalls

- Models trained with non-representation losses (e.g., autoregressive models) often lack good off-the-shelf zero-shot performance but may still perform well in linear probing.
- Visual STS tasks primarily measure OCR capabilities of vision encoders, which are often overlooked when evaluating semantic similarity.
- Linear probing uses a fixed few-shot setting (16 shots per class) rather than full-dataset training, which can affect performance trends compared to standard classification benchmarks.

## Evidence (verbatim from paper)

> We use nDCG@10 as the primary metric, and recall@1/map@5 for some tasks to align with prior work or adjust for difficulty.

## Citation

```bibtex
@misc{xiao2025mieb,
  title={MIEB: Massive Image Embedding Benchmark},
  author={Xiao et al. (2025)},
  year={2025},
  note={arXiv:2504.10471}
}
```

- arXiv: 2504.10471