R2med Eval

---
name: r2med-eval
description: Evaluates retrieval models on reasoning-driven medical tasks where document relevance is determined by alignment with inferred clinical diagnoses or multi-step reasoning paths rather than lexical or semantic overlap. Covers three task types—Q&A reference, clinical evidence, and clinical case retrieval—spanning eight medical sub-domains. Use when the user wants to benchmark on R2MED, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.14558
  bibtex_key: li2025r2med
  confidence: high
---

# r2med-eval

> R2MED: A Benchmark for Reasoning-Driven Medical Retrieval — Li et al. (2025) (arXiv:2505.14558, 2025)

## What this evaluates

Evaluates retrieval models on reasoning-driven medical tasks where document relevance is determined by alignment with inferred clinical diagnoses or multi-step reasoning paths rather than lexical or semantic overlap. Covers three task types—Q&A reference, clinical evidence, and clinical case retrieval—spanning eight medical sub-domains.

## Datasets

- **R2MED** — total ?; splits: test (-1); repo https://github.com/R2MED/R2MED

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. It measures the quality of the ranked list of retrieved documents by discounting the relevance score logarithmically with position, normalized by the ideal DCG. Formula: nDCG@10 = DCG@10 / IDCG@10, where DCG@10 = Σ (rel_i / log2(i+1)) for i=1 to 10.

## Input / output format

**Input**: A clinical query (question, case description, or evidence request) and a candidate corpus of medical documents.

**Output**: A ranked list of candidate documents (typically top-10 or top-100) ordered by predicted relevance to the query.

## Scoring recipe

```python
def compute_ndcg_at_10(retrieved_rel_scores, k=10):
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(retrieved_rel_scores[:k]))
    ideal_scores = sorted(retrieved_rel_scores, reverse=True)[:k]
    idcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(ideal_scores))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Assuming standard semantic retrieval models will perform well without explicit reasoning alignment, as lexical overlap is insufficient for clinical diagnosis inference.
- Reranking over top-100 candidates often degrades performance or yields inconsistent gains compared to reranking over top-10, especially for strong base retrievers.
- Medical-domain pretraining alone does not guarantee superior performance on reasoning-driven tasks if the training corpus lacks explicit reasoning paths.

## Evidence (verbatim from paper)

> Following prior work[nguyen2016ms] ; [thakur2021beir] ; [su2024bright], we use nDCG@10 as the primary evaluation metric.

## Citation

```bibtex
@misc{li2025r2med,
  title={R2MED: A Benchmark for Reasoning-Driven Medical Retrieval},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2505.14558}
}
```

- arXiv: 2505.14558