Mrmr Eval

---
name: mrmr-eval
description: Evaluates reasoning-intensive multimodal retrieval across 23 expert domains using interleaved image-text queries and documents. It probes a model's ability to perform knowledge-based matching, theorem linking, and logical contradiction detection in complex, real-world scenarios. Use when the user wants to benchmark on MRMR, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.09510
  bibtex_key: zhang2025mrmr
  confidence: high
---

# mrmr-eval

> MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval — Zhang et al. (2025) (arXiv:2510.09510, 2025)

## What this evaluates

Evaluates reasoning-intensive multimodal retrieval across 23 expert domains using interleaved image-text queries and documents. It probes a model's ability to perform knowledge-based matching, theorem linking, and logical contradiction detection in complex, real-world scenarios.

## Datasets

- **MRMR** — total 1502; splits: test (1502)

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Measures the quality of the ranked list of retrieved documents by discounting relevance by position and normalizing by the ideal ranking.
- `Hit@1` — range: [0, 1]
  - Binary metric indicating whether the single gold document appears at rank 1 in the retrieved list. Used exclusively for the Negation subtask.

## Input / output format

**Input**: Interleaved image-text queries paired with multimodal documents. Evaluation setups vary: text retrievers use LLM-generated image captions, two-stream models use vector fusion of text and image embeddings, and document-as-image models encode entire documents as screenshots.

**Output**: A ranked list of retrieved documents or candidate documents for each query.

## Scoring recipe

```python
def compute_metrics(predictions, gold, task='default'):
    if task == 'negation':
        return 1.0 if predictions[0] == gold else 0.0
    k = 10
    dcg = sum(1.0 / math.log2(i + 2) for i, p in enumerate(predictions[:k]) if p == gold)
    idcg = 1.0 / math.log2(2)
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- The Negation subtask uses exactly 4 candidates per query, so Hit@1 scores below 25% indicate performance at or below random guessing.
- Models often rely on surface-level semantic matching rather than deep reasoning, leading to poor performance on Theorem and Contradiction subtasks despite strong results on Knowledge subtasks.
- Multimodal models struggle with expert-level query images (e.g., microscopic tissue sections) that require domain-specific reasoning beyond superficial visual similarity.

## Evidence (verbatim from paper)

> Following prior work (Xiao et al., 2025; Su et al., 2025), we use nDCG@10 as the main evaluation metric except Negation. Since each query in Negation has exactly one gold document among four candidates, we adopt Hit@1 as the main metric for this task.

## Citation

```bibtex
@misc{zhang2025mrmr,
  title={MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.09510}
}
```

- arXiv: 2510.09510