Medlvr Medical Vqa Eval

---
name: medlvr-medical-vqa-eval
description: Evaluates a model's ability to answer medical visual questions across diverse imaging modalities (CT, MRI, X-ray, etc.) and generalizes to out-of-domain benchmarks. It probes the model's capacity for latent visual reasoning and robust cross-modality transfer without relying on external tools or retrieval augmentation. Use when the user wants to benchmark on OmniMedVQA, SLAKE, VQA-RAD, PMC-VQA, MMMU (Health & Medicine), MedXpertQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.09757
  bibtex_key: xi2026medlvr
  confidence: high
---

# medlvr-medical-vqa-eval

> MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering — Xi et al. (2026) (arXiv:2604.09757, 2026)

## What this evaluates

Evaluates a model's ability to answer medical visual questions across diverse imaging modalities (CT, MRI, X-ray, etc.) and generalizes to out-of-domain benchmarks. It probes the model's capacity for latent visual reasoning and robust cross-modality transfer without relying on external tools or retrieval augmentation.

## Datasets

- **OmniMedVQA** — total 88996; splits: train (71196), test (17800)
- **SLAKE** — total ?; splits: test (-1)
- **VQA-RAD** — total ?; splits: test (-1)
- **PMC-VQA** — total ?; splits: test (-1)
- **MMMU (Health & Medicine)** — total ?; splits: test (-1)
- **MedXpertQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions, computed as (number of correct predictions / total number of questions) * 100. For benchmarks like SLAKE and VQA-RAD, a normalized answer protocol is applied for binary yes/no questions.

## Input / output format

**Input**: A medical image (from one of eight modalities: CT, MRI, X-ray, Ultrasound, Dermoscopy, Fundus, OCT, Microscopy) paired with a natural language question.

**Output**: A text answer corresponding to the question. For multiple-choice or binary questions, the model outputs the selected option or yes/no.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
accuracy = (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Using the full OmniMedVQA dataset for evaluation instead of the specified 80/20 train-test split.
- Comparing results with models that use external tools, retrieval augmentation, or agentic pipelines, as MedLVR is evaluated under a unified direct-decoding protocol.
- Ignoring modality-specific breakdowns; overall accuracy can mask significant performance drops on specific modalities like Ultrasound or Microscopy.

## Evidence (verbatim from paper)

> MedLVR improves over its base model Qwen2.5-VL-7B on all five benchmarks, increasing accuracy from 63.7% to 66.4% on SLAKE, from 59.9% to 65.9% on VQA-RAD, from 49.0% to 53.6% on PMC-VQA, from 46.4% to 56.6% on MMMU (Health & Medicine), and from 22.5% to 24.3% on MedXpertQA.

## Citation

```bibtex
@misc{xi2026medlvr,
  title={MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering},
  author={Xi et al. (2026)},
  year={2026},
  note={arXiv:2604.09757}
}
```

- arXiv: 2604.09757