Medical Multimodal Eval

---
name: medical-multimodal-eval
description: Evaluates cross-modal understanding and generation capabilities in medical imaging, specifically image-report retrieval, radiology report generation, and multi-label disease diagnosis from chest X-rays. Use when the user wants to benchmark on MIMIC-CXR, IU-Xray, ChestX-ray 14, or asks about evaluating this task. Reports CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.14204
  bibtex_key: [REDACTED]
  confidence: high
---

# medical-multimodal-eval

> Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining — Bingqian Lin et al. (arXiv:2304.14204, 2023)

## What this evaluates

Evaluates cross-modal understanding and generation capabilities in medical imaging, specifically image-report retrieval, radiology report generation, and multi-label disease diagnosis from chest X-rays.

## Datasets

- **MIMIC-CXR** — total ?; splits: test (-1)
- **IU-Xray** — total ?; splits: test (-1)
- **ChestX-ray 14** — total ?; splits: test (-1)

## Metrics

- `R@K` — range: [0, 1]
  - Recall at K: the fraction of ground-truth items successfully retrieved within the top K predictions for both image-to-report and report-to-image retrieval.
- `CIDEr` **(primary)** — range: [0, 1]
  - TF-IDF weighted n-gram similarity between the generated report and reference reports, emphasizing rare but informative words.
- `AUROC` — range: [0, 1]
  - Area under the receiver operating characteristic curve computed per disease class and averaged across all 14 categories.

## Input / output format

**Input**: Per task: (1) IRR: chest X-ray image or radiology report; (2) MRG: chest X-ray image; (3) Diagnosis: chest X-ray image.

**Output**: Per task: (1) IRR: ranked list of reports or images; (2) MRG: free-text radiology report; (3) Diagnosis: multi-label binary vector over 14 disease categories.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # R@K for retrieval
    r_at_k = sum(1 for p, g in zip(predictions, gold) if g in p[:K]) / len(predictions)
    # CIDEr for generation
    cidec = compute_tfidf_ngram_similarity(predictions, gold)
    # AUROC for classification
    auroc = mean(roc_auc_score(gold[:, c], predictions[:, c]) for c in range(14))
    return r_at_k, cidec, auroc
```

## Common pitfalls

- MIMIC-CXR is used for both pretraining and IRR evaluation, so zero-shot and finetuning results must be reported separately to avoid data leakage confusion.
- CIDEr is explicitly marked as the main metric for report generation; relying solely on BLEU or ROUGE may misrepresent model performance.
- Diagnosis AUROC is averaged over 14 classes; class imbalance in medical datasets means macro-averaging can mask poor performance on rare conditions.

## Evidence (verbatim from paper)

> The R@K (recall with top k predictions) metric is used for the performance evaluation for both subtasks. ... The widely-used BLEU [26], METEOR [27], ROUGE-L [28], and CIDEr [29] for MRG are adopted as the evaluation metrics.

## Citation

```bibtex
@misc{lin2023towardsmedical,
  title={Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining},
  author={Bingqian Lin et al.},
  year={2023},
  note={arXiv:2304.14204}
}
```

- arXiv: 2304.14204