Unified Med Vlm Benchmark

---
name: unified-med-vlm-benchmark
description: Evaluates medical vision-language models across a broad capability surface including visual diagnosis, medical imaging, clinical reasoning, text-based QA, report generation, and instruction following. It emphasizes protocol reproducibility, deployment relevance (safety, consistency, faithfulness), and robustness to real-world clinical imagery and OCR conditions. Use when the user wants to benchmark on Public Med-VLM Benchmarks (30+ subsets), Inhouse VQA, Inhouse OCR, Inhouse Caption, or asks about evaluating this task. Reports MCQ/Short QA Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.12705
  bibtex_key: shi2026medxiaohe
  confidence: high
---

# unified-med-vlm-benchmark

> MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMs — Shi et al. (2026) (arXiv:2602.12705, 2026)

## What this evaluates

Evaluates medical vision-language models across a broad capability surface including visual diagnosis, medical imaging, clinical reasoning, text-based QA, report generation, and instruction following. It emphasizes protocol reproducibility, deployment relevance (safety, consistency, faithfulness), and robustness to real-world clinical imagery and OCR conditions.

## Datasets

- **Public Med-VLM Benchmarks (30+ subsets)** — total ?; splits: test (-1)
- **Inhouse VQA** — total 100000; splits: test (-1)
- **Inhouse OCR** — total ?; splits: test (-1)
- **Inhouse Caption** — total ?; splits: test (-1)

## Metrics

- `MCQ/Short QA Accuracy` **(primary)** — range: [0, 1]
  - Binary correctness score (1.0 if deterministic option extraction or normalized numeric prediction matches gold, 0.0 otherwise).
- `rubric-style structured judging` — range: other
  - Structured evaluation scoring clinical correctness, completeness, and absence of unsupported claims for open-form tasks (dialogue, reports, captions).
- `Key-point Coverage Score` — range: other
  - Reward-and-penalty metric where correctly covered key points receive positive credit and incorrect statements are penalized.

## Input / output format

**Input**: Multimodal inputs (clinical images, medical documents, OCR-distorted reports) paired with standardized capability-aware prompt templates. Instructions specify task type (MCQ, short QA, long QA/dialogue, report/caption, OCR, agentic search) and include auxiliary context where applicable.

**Output**: Deterministic option extraction for closed-form tasks; structured text outputs for open-form tasks (dialogue, reports, captions) constrained by prompt templates. For Inhouse Caption, outputs are evaluated against curated key points.

## Scoring recipe

```python
def score_instance(task_type, pred, gold):
    if task_type in ['MCQ', 'short_QA']:
        return 1.0 if normalize(pred) == gold else 0.0
    elif task_type == 'caption':
        covered = count_covered_key_points(pred, gold['key_points'])
        penalized = count_unsupported_claims(pred)
        return covered - penalized
    else:
        return rubric_judge(pred, gold, ['clinical_correctness', 'completeness', 'no_unsupported_claims'])

def evaluate_suite(predictions, golds, task_types):
    scores = [score_instance(t, p, g) for t, p, g in zip(task_types, predictions, golds)]
    return macro_average_by_category(scores)
```

## Common pitfalls

- Cross-paper comparisons are brittle due to protocol-level mismatches (different benchmark subsets, prompting styles, scoring scripts, and data hygiene assumptions).
- Open-form task scoring is highly sensitive to protocol choices, particularly distinguishing omissions from incorrect additions in report generation.
- Real-world OCR and VQA benchmarks require handling of perspective distortion, blur, glare, and partial occlusion, which standard clean datasets do not capture.

## Evidence (verbatim from paper)

> Across benchmarks, we normalize evaluation with a consistent harness: Task normalization: each dataset is mapped into a small set of task families (MCQ, short QA, long QA/dialogue, report/caption generation, OCR, agentic search/decision). Prompt templates: we use capability-aware templates that standardize instruction phrasing, answer format constraints, and the placement of auxiliary context. Answer parsing: for closed-form tasks (MCQ/short QA), we enforce deterministic parsing (option extraction, numeric normalization, whitespace/punctuation normalization). For open-form tasks (dialogue/report/caption), we score with task-appropriate metrics and/or rubric-style structured judging to distinguish clinical correctness, completeness, and unsupported claims.

## Citation

```bibtex
@misc{shi2026medxiaohe,
  title={MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMs},
  author={Shi et al. (2026)},
  year={2026},
  note={arXiv:2602.12705}
}
```

- arXiv: 2602.12705