Medical Vqa Eval

---
name: medical-vqa-eval
description: Evaluates the medical visual question answering capabilities of multimodal large language models across diverse imaging modalities and general medical knowledge domains. Use when the user wants to benchmark on VQA-RAD, SLAKE (English CLOSED), PathVQA, PMC-VQA, MMMU (Health & Medicine track), OmniMedVQA (open access), or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.19280
  bibtex_key: chen2024huatuogptvision
  confidence: high
---

# medical-vqa-eval

> HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale — Chen et al. (2024) (arXiv:2406.19280, 2024)

## What this evaluates

Evaluates the medical visual question answering capabilities of multimodal large language models across diverse imaging modalities and general medical knowledge domains.

## Datasets

- **VQA-RAD** — total ?; splits: test (-1)
- **SLAKE (English CLOSED)** — total ?; splits: test (-1)
- **PathVQA** — total ?; splits: test (-1)
- **PMC-VQA** — total ?; splits: test (-1)
- **MMMU (Health & Medicine track)** — total ?; splits: test (-1)
- **OmniMedVQA (open access)** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of test instances. Calculated as (number of exact matches / total instances) × 100.

## Input / output format

**Input**: A medical image paired with a natural language question (VQA format).

**Output**: A text string containing the predicted answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if normalize_answer(pred) == normalize_answer(gold))
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Must use the English CLOSED segment for SLAKE, not the full dataset.
- All evaluations must use zero-shot inference with LLaVA's default question template.
- MMMU Health & Medicine results require official website submission rather than local evaluation.

## Evidence (verbatim from paper)

> Notably, the use of the PubMedVision led to an 11.7% increase in overall accuracy, significantly outperforming the earlier LLaVA_Med dataset.

## Citation

```bibtex
@misc{chen2024huatuogptvision,
  title={HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2406.19280}
}
```

- arXiv: 2406.19280