Multimodal Medical Stress Test Eval

---
name: multimodal-medical-stress-test-eval
description: This evaluation probes the robustness and genuine multimodal reasoning capabilities of large language models in clinical settings. It measures how model accuracy degrades when visual inputs are removed, answer options are perturbed, or distractors are replaced, revealing reliance on textual shortcuts and memorization rather than true visual-textual integration. Use when the user wants to benchmark on NEJM, JAMA, VQA-RAD, OmniMedVQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.18234
  bibtex_key: gu2025illusion
  confidence: high
---

# multimodal-medical-stress-test-eval

> The Illusion of Readiness in Health AI — Yu Gu et al. (2025) (arXiv:2509.18234, 2025)

## What this evaluates

This evaluation probes the robustness and genuine multimodal reasoning capabilities of large language models in clinical settings. It measures how model accuracy degrades when visual inputs are removed, answer options are perturbed, or distractors are replaced, revealing reliance on textual shortcuts and memorization rather than true visual-textual integration.

## Datasets

- **NEJM** — total ?; splits: test (-1)
- **JAMA** — total ?; splits: test (-1)
- **VQA-RAD** — total ?; splits: test (-1)
- **OmniMedVQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Proportion of correctly answered multiple-choice questions out of the total evaluated items. Calculated as (number of correct predictions) / (total number of items).

## Input / output format

**Input**: Multiple-choice clinical vignette paired with one or more medical images (radiology, dermatology, pathology). Inputs may be provided as text-only, image+text, or with perturbed options (shuffled order, replaced distractors, or swapped images).

**Output**: A single selected answer choice (e.g., A, B, C, D) corresponding to the correct diagnosis or finding. Optionally accompanied by a Chain-of-Thought reasoning trace.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

# For stress tests, compute delta:
baseline_acc = compute_accuracy(predictions_baseline, gold_labels)
perturbed_acc = compute_accuracy(predictions_perturbed, gold_labels)
accuracy_drop = baseline_acc - perturbed_acc
```

## Common pitfalls

- High leaderboard accuracy often masks severe brittleness when visual inputs are removed or perturbed.
- Models frequently exploit dataset artifacts like answer position bias, memorized QA pairs, or distractor co-occurrence patterns instead of genuine clinical reasoning.
- Model refusal behavior (e.g., declining to answer without an image) can artificially deflate accuracy scores and should be interpreted cautiously.

## Evidence (verbatim from paper)

> Accuracy drops when images are removed from two diagnostic benchmarks (NEJM and JAMA), revealing hidden differences in modality sensitivity across models. Sharp drops on NEJM—but minimal impact on JAMA—suggest inconsistent visual dependence across benchmarks and question types.

## Citation

```bibtex
@misc{gu2025illusion,
  title={The Illusion of Readiness in Health AI},
  author={Yu Gu et al. (2025)},
  year={2025},
  note={arXiv:2509.18234}
}
```

- arXiv: 2509.18234