Mmdocir Eval

---
name: mmdocir-eval
description: Evaluates multimodal retrieval systems on long documents by measuring their ability to retrieve relevant pages and fine-grained layout elements given a natural language query. Use when the user wants to benchmark on MMDocIR, or asks about evaluating this task. Reports similarity scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.08828
  bibtex_key: dong2025mmdocir
  confidence: medium
---

# mmdocir-eval

> MMDocIR: Benchmarking Multimodal Retrieval for Long Documents — Dong et al. (2025) (arXiv:2501.08828, 2025)

## What this evaluates

Evaluates multimodal retrieval systems on long documents by measuring their ability to retrieve relevant pages and fine-grained layout elements given a natural language query.

## Datasets

- **MMDocIR** — total 73800; splits: train (73800), test (-1)

## Metrics

- `similarity scores` **(primary)** — range: [0, 1]
  - Relevance of a query Q to a page p or layout l is computed as Sim(Q,p) or Sim(Q,l), used to rank items for top-k retrieval.

## Input / output format

**Input**: Natural language query Q and a multimodal document corpus consisting of pages P and layout elements L.

**Output**: A ranked list of top-k pages and/or layouts most relevant to Q.

## Scoring recipe

```python
def score(predictions, gold, k):
    retrieved = predictions[:k]
    hits = sum(1 for pred in retrieved if pred in gold)
    return hits / len(gold)
```

## Common pitfalls

- Converting multimodal documents to text via OCR or VLMs before retrieval causes information loss and reduces accuracy.
- Layout detection must correctly segment pages into 5-15 layout elements; errors here propagate to layout-level retrieval.

## Evidence (verbatim from paper)

> The relevance of pages (p) and layouts (l) to Q is measured by similarity scores, Sim(Q,p) and Sim(Q,l) respectively. The retrieval system consists of two phases: (1) an offline indexing phase, where pages and layouts from P and L are encoded into vectors, and (2) an online querying phase, in which a query Q is encoded into a vector, which is then compared against the offline-indexed vectors using similarity scores Sim(Q,p) for pages and Sim(Q,l) for layouts.

## Citation

```bibtex
@misc{dong2025mmdocir,
  title={MMDocIR: Benchmarking Multimodal Retrieval for Long Documents},
  author={Dong et al. (2025)},
  year={2025},
  note={arXiv:2501.08828}
}
```

- arXiv: 2501.08828