Fleming Vl Medical Eval

---
name: fleming-vl-medical-eval
description: Evaluates a multimodal LLM's ability to perform visual reasoning across heterogeneous medical modalities (2D images, 3D volumes, videos) and generate clinical reports. It probes diagnostic accuracy, cross-modal generalization, temporal understanding, and structured medical knowledge integration. Use when the user wants to benchmark on OmniMedVQA, PMC-VQA, VQA-RAD, PathVQA, SLAKE, MIMIC-CXR, IU-Xray, M3D-VQA, MedVideoBench, or asks about evaluating this task. Reports accuracy, ROUGE-L, CIDEr.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.00916
  bibtex_key: shu2025flemingvl
  confidence: high
---

# fleming-vl-medical-eval

> Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs — Shu et al. (2025) (arXiv:2511.00916, 2025)

## What this evaluates

Evaluates a multimodal LLM's ability to perform visual reasoning across heterogeneous medical modalities (2D images, 3D volumes, videos) and generate clinical reports. It probes diagnostic accuracy, cross-modal generalization, temporal understanding, and structured medical knowledge integration.

## Datasets

- **OmniMedVQA** — total ?; splits: test (-1)
- **PMC-VQA** — total 227000; splits: test (-1)
- **VQA-RAD** — total 315; splits: test (-1)
- **PathVQA** — total 32799; splits: test (-1)
- **SLAKE** — total ?; splits: test (-1)
- **MIMIC-CXR** — total 377110; splits: test (-1)
- **IU-Xray** — total 7470; splits: test (-1)
- **M3D-VQA** — total 13791; splits: test (-1)
- **MedVideoBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered multiple-choice or open-ended questions. Computed as the ratio of exact matches between predicted and ground-truth answers.
- `ROUGE-L` **(primary)** — range: percent
  - Longest common subsequence overlap between generated and reference text, scaled by 100 in reported tables.
- `CIDEr` **(primary)** — range: percent
  - Consensus-based image description evaluation using TF-IDF weighted n-grams, scaled by 100.
- `RaTE` — range: percent
  - Clinical fidelity metric based on Radiology Text Embedding similarity between generated and reference reports.
- `Semb` — range: percent
  - Semantic clinical fidelity metric measuring embedding-level alignment between predictions and gold reports.
- `RadCliQ-1` — range: percent
  - Clinical quality scoring metric evaluating the diagnostic accuracy and clinical coherence of generated reports.

## Input / output format

**Input**: Medical image (2D/3D/video frame) or video sequence paired with a natural language question or instruction to generate a report.

**Output**: Natural language text: either a short answer/option for VQA, or a structured clinical report (findings, impressions, diagnosis).

## Scoring recipe

```python
def evaluate(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower()) / len(golds)
    rouge_l = rouge_l_score(predictions, golds) * 100
    cider = cider_score(predictions, golds) * 100
    rate = clinical_embedding_similarity(predictions, golds, model='RaTE') * 100
    semb = clinical_embedding_similarity(predictions, golds, model='Semb') * 100
    radcliq = clinical_quality_score(predictions, golds, model='RadCliQ-1') * 100
    return {'accuracy': acc, 'ROUGE-L': rouge_l, 'CIDEr': cider, 'RaTE': rate, 'Semb': semb, 'RadCliQ-1': radcliq}
```

## Common pitfalls

- Tables scale all scores by a factor of 100 for clarity; evaluation code must multiply or divide accordingly to match reported values.
- MedVideoBench is a newly proposed benchmark by the authors; external reproduction requires accessing their supplementary code or data release.
- Clinical fidelity metrics (RaTE, Semb, RadCliQ-1) rely on specific pretrained clinical NLP models that must be correctly initialized and aligned with the evaluation script.

## Evidence (verbatim from paper)

> For holistic video understanding, we employ ROUGE-L and CIDEr metrics to evaluate the quality and clinical fidelity of generated video summaries, while multiple-choice questions are evaluated using accuracy.

## Citation

```bibtex
@misc{shu2025flemingvl,
  title={Fleming-VL: Towards Universal Medical Visual Reasoning with Multimodal LLMs},
  author={Shu et al. (2025)},
  year={2025},
  note={arXiv:2511.00916}
}
```

- arXiv: 2511.00916