Medthinkvqa Eval

---
name: medthinkvqa-eval
description: Evaluates vision-language models' ability to interpret multiple medical images, integrate cross-view evidence, and perform stepwise clinical reasoning for differential diagnosis. It probes visual grounding, evidence alignment, and reasoning depth beyond simple answer matching. Use when the user wants to benchmark on MedThinkVQA, or asks about evaluating this task. Reports Stepwise Reasoning Evaluation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.16506
  bibtex_key: yao2026medicalthinking
  confidence: high
---

# medthinkvqa-eval

> Medical thinking with multiple images — Yao et al. (2026) (arXiv:2604.16506, 2026)

## What this evaluates

Evaluates vision-language models' ability to interpret multiple medical images, integrate cross-view evidence, and perform stepwise clinical reasoning for differential diagnosis. It probes visual grounding, evidence alignment, and reasoning depth beyond simple answer matching.

## Datasets

- **MedThinkVQA** — total 8067; splits: test (-1); repo https://github.com/benluwang/MedThinkVQA

## Metrics

- `ROUGE` — range: [0, 1]
  - Standard ROUGE-N lexical overlap between the generated clinical report/summary and the expert-written ground truth.
- `RadCliQ` — range: [0, 1]
  - A learned evaluation metric that correlates strongly with radiologist preferences for clinical report quality.
- `Stepwise Reasoning Evaluation` **(primary)** — range: percent
  - LLM-judged factual correctness per atomic reasoning step, plus classification into four error types (Image Understanding Err, Reasoning Err, Medical Knowledge Err, Clinical-Scenario Misunderstanding).

## Input / output format

**Input**: A set of multiple medical images (average 6.62 per case) paired with a clinical question or case context.

**Output**: Per-image clinical findings, a case-level integrated imaging summary, and a stepwise reasoning explanation leading to a final diagnosis.

## Scoring recipe

```python
# 1. Report-level metrics
rouge = compute_rouge(model_report, expert_report)
radcliq = compute_radcliq(model_report, expert_report)

# 2. Stepwise reasoning evaluation
steps = split_explanation_into_atomic_steps(model_output)
judge = load_llm_judge("GPT-5")
step_results = []
for step in steps:
    label = judge.evaluate_step(step, gold_diagnosis)
    # label contains: factual_correctness (bool), is_critical (bool), error_type (str)
    step_results.append(label)

step_accuracy = sum(r.factual_correctness for r in step_results) / len(step_results)
error_counts = Counter(r.error_type for r in step_results if not r.factual_correctness)
```

## Common pitfalls

- Relying solely on final answer accuracy hides clinically meaningful reasoning failures; step-level auditing is explicitly required.
- Models may appear to scale well with reasoning length, but gains only materialize when early visual grounding is robust; extra tokens cannot compensate for weak multi-image evidence extraction.
- Automatic LLM-judged step evaluation requires validation; human-LLM agreement varies (κ=0.70–0.84), so error type distributions should be cross-checked.

## Evidence (verbatim from paper)

> For the per-image findings and the case-level integrated imaging summary (Steps 1–2 in Fig.[1](#S0.F1 "Figure 1 ‣ Medical thinking with multiple images")), we follow recent radiology-report evaluation work*(Yu et al., [2023](#bib.bib28 "Evaluating progress in automatic chest x-ray radiology report generation"); Ostmeier et al., [2024](#bib.bib29 "Green: generology report evaluation and error notation"))* and compute ROUGE as a lexical-overlap baseline together with RadCliQ, which correlates more strongly with radiologist preferences.

## Citation

```bibtex
@misc{yao2026medicalthinking,
  title={Medical thinking with multiple images},
  author={Yao et al. (2026)},
  year={2026},
  note={arXiv:2604.16506}
}
```

- arXiv: 2604.16506