Gmai Vl Eval

---
name: gmai-vl-eval
description: Evaluates a vision-language model's ability to understand and reason over diverse medical imaging modalities (CT, MRI, X-ray, pathology slides) to answer clinical questions, diagnose diseases, and perform anatomical or lesion recognition tasks. Use when the user wants to benchmark on PMCVQA, PathVQA, VQA-RAD, SLAKE, OmniMedVQA, GMAI-MMBench, MMMU Health & Medicine track, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.14522
  bibtex_key: li2024gmaivl
  confidence: high
---

# gmai-vl-eval

> GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI — Li et al. (2024) (arXiv:2411.14522, 2024)

## What this evaluates

Evaluates a vision-language model's ability to understand and reason over diverse medical imaging modalities (CT, MRI, X-ray, pathology slides) to answer clinical questions, diagnose diseases, and perform anatomical or lesion recognition tasks.

## Datasets

- **PMCVQA** — total ?; splits: test (-1)
- **PathVQA** — total ?; splits: test (-1)
- **VQA-RAD** — total ?; splits: test (-1)
- **SLAKE** — total ?; splits: test (-1)
- **OmniMedVQA** — total ?; splits: test (-1)
- **GMAI-MMBench** — total ?; splits: val (-1), test (-1)
- **MMMU Health & Medicine track** — total ?; splits: val (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions in the benchmark. Reported as an overall score and broken down by clinical sub-tasks (e.g., Modality Recognition, Disease Diagnosis).

## Input / output format

**Input**: A medical image (e.g., X-ray, CT, MRI, fundus, or pathology slide) paired with a clinical or diagnostic question.

**Output**: A text answer or selected option corresponding to the question.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if normalize_answer(pred) == normalize_answer(gold))
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Data leakage must be strictly prevented by using only public test sets for evaluation and verifying image MD5 hashes against training data.
- Evaluation relies on VLMEvalKit with default settings; deviations in inference parameters or prompt formatting may cause non-reproducible results.
- Benchmarks report both overall accuracy and sub-task accuracies, which should be reported separately to match the paper's breakdown.

## Evidence (verbatim from paper)

> Overall, with an average accuracy of 88.48%, GMAI-VL surpasses models like HuatuoGPT-Vision-34B and InternVL2-40B, establishing itself as a leading model in multimodal medical image understanding and setting a new benchmark for medical VQA tasks.

## Citation

```bibtex
@misc{li2024gmaivl,
  title={GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2411.14522}
}
```

- arXiv: 2411.14522