M3retrieve Eval

---
name: m3retrieve-eval
description: Evaluates the ability of multimodal retrieval models to accurately rank relevant medical documents in response to text-and-image queries. It probes domain-specific alignment, handling of complex clinical terminology, and cross-specialty generalization in safety-critical healthcare settings. Use when the user wants to benchmark on M3Retrieve, or asks about evaluating this task. Reports nNDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.06888
  bibtex_key: acharya2025m3retrieve
  confidence: high
---

# m3retrieve-eval

> M3Retrieve: Benchmarking Multimodal Retrieval for Medicine — Acharya et al. (2025) (arXiv:2510.06888, 2025)

## What this evaluates

Evaluates the ability of multimodal retrieval models to accurately rank relevant medical documents in response to text-and-image queries. It probes domain-specific alignment, handling of complex clinical terminology, and cross-specialty generalization in safety-critical healthcare settings.

## Datasets

- **M3Retrieve** — total 1200000; splits: test (-1); repo https://github.com/AkashGhosh/M3Retrieve

## Metrics

- `nNDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. It measures the quality of the predicted ranking by computing the DCG@10 of the retrieved documents and normalizing it by the ideal DCG@10 for the same set of relevant documents.

## Input / output format

**Input**: Multimodal queries consisting of text and associated medical images, paired with a large corpus of text documents to be retrieved.

**Output**: A ranked list of retrieved document IDs or text snippets, typically evaluated at the top 10 positions.

## Scoring recipe

```python
def compute_nndcg_at_10(gold_relevant_set, predicted_ranking):
    dcg = 0.0
    for i, doc_id in enumerate(predicted_ranking[:10]):
        if doc_id in gold_relevant_set:
            dcg += 1.0 / math.log2(i + 2)
    ideal_dcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(gold_relevant_set), 10)))
    return dcg / ideal_dcg if ideal_dcg > 0 else 0.0
```

## Common pitfalls

- Medical domain queries often contain complex or ambiguous terminology that can cause lexicon-based models like BM25 to fail without careful preprocessing or domain-specific tuning.
- Multimodal alignment issues arise when image-text relationships are weak or highly domain-specific, leading to poor cross-modal retrieval performance that standard CLIP-style models may not capture.
- nNDCG@10 assumes binary relevance; if the benchmark uses graded relevance scores, the formula must be adapted accordingly.

## Evidence (verbatim from paper)

> We used nNDCG@10 as the primary metric for evaluation. For both FLMR and BM25, the evaluation metrics were computed using the pytrec_eval Python library, following the implementation in the MTEB library.

## Citation

```bibtex
@misc{acharya2025m3retrieve,
  title={M3Retrieve: Benchmarking Multimodal Retrieval for Medicine},
  author={Acharya et al. (2025)},
  year={2025},
  note={arXiv:2510.06888}
}
```

- arXiv: 2510.06888