Medical Vqa Grounding Eval

---
name: medical-vqa-grounding-eval
description: Evaluates whether multimodal medical vision-language models actually rely on image content to answer questions, or if they exploit text-only shortcuts. It measures visual grounding by comparing model performance and prediction stability across real, blank, and shuffled image conditions. Use when the user wants to benchmark on PathVQA, PMC-VQA, SLAKE, VQA-RAD, or asks about evaluating this task. Reports VRS (Visual Reliance Score), IS (Image Sensitivity).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.03437
  bibtex_key: [REDACTED]
  confidence: high
---

# medical-vqa-grounding-eval

> Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning — Zafar et al. (2026) (arXiv:2603.03437, 2026)

## What this evaluates

Evaluates whether multimodal medical vision-language models actually rely on image content to answer questions, or if they exploit text-only shortcuts. It measures visual grounding by comparing model performance and prediction stability across real, blank, and shuffled image conditions.

## Datasets

- **PathVQA** — total 100; splits: test (100)
- **PMC-VQA** — total 100; splits: test (100)
- **SLAKE** — total 100; splits: test (100)
- **VQA-RAD** — total 100; splits: test (100)

## Metrics

- `VRS (Visual Reliance Score)` **(primary)** — range: [-1, 1]
  - Difference in accuracy between real images and shuffled images: Acc_real - Acc_shuffle. Positive values indicate visual reliance; negative values indicate text-shortcut exploitation.
- `IS (Image Sensitivity)` **(primary)** — range: [0, 1]
  - Percentage of predictions that change when the input image is shuffled. Measures answer-level dependence on visual content regardless of correctness.
- `BD (Blank Drop)` — range: [0, 1]
  - Difference in accuracy between real images and blank images: Acc_real - Acc_blank. Measures performance degradation when visual input is removed.
- `VBR (Visual Benefit Rate)` — range: [0, 1]
  - Fraction of cases where the model answers correctly with the real image but incorrectly with a shuffled image.
- `VHR (Visual Harm Rate)` — range: [0, 1]
  - Fraction of cases where the model answers incorrectly with the real image but correctly with a shuffled image.
- `HVRR (Hallucinated VR Rate)` — range: [0, 1]
  - Fraction of responses that contain visual claims but produce invariant answers across real and shuffled image conditions.

## Input / output format

**Input**: Medical VQA question paired with a medical image (e.g., pathology slide, X-ray, CT scan).

**Output**: Textual answer (and optionally a rationale/visual claim describing image content).

## Scoring recipe

```python
def compute_metrics(pred_real, pred_blank, pred_shuffle, rationales_real, gold):
    acc_real = mean(pred_real == gold)
    acc_blank = mean(pred_blank == gold)
    acc_shuffle = mean(pred_shuffle == gold)
    vrs = acc_real - acc_shuffle
    bd = acc_real - acc_blank
    is_score = mean(pred_real != pred_shuffle)
    vbr = mean((pred_real == gold) & (pred_shuffle != gold))
    vhr = mean((pred_real != gold) & (pred_shuffle == gold))
    nvcr = mean(has_visual_claim(rationales_real))
    hvrr = mean(has_visual_claim(rationales_real) & (pred_real == pred_shuffle))
    return {'VRS': vrs, 'BD': bd, 'IS': is_score, 'VBR': vbr, 'VHR': vhr, 'NVCR': nvcr, 'HVRR': hvrr}
```

## Common pitfalls

- Accuracy can increase while visual grounding degrades, as models learn text shortcuts that correlate with correct answers without using image content.
- VRS and IS can diverge: VRS may improve via better text pattern matching while IS drops, meaning accuracy-based metrics alone cannot detect grounding collapse.
- Blank/shuffled image baselines are often overlooked, leading to false confidence in multimodal capabilities.

## Evidence (verbatim from paper)

> The disconnect between accuracy and grounding is most evident in image sensitivity (Table 2): RL(image) changes predictions only 39.8% of the time when images are shuffled, meaning 60.2% of answers ignore image content entirely. This contrasts sharply with the baseline model (48.2% IS), which despite having no medical fine-tuning shows stronger visual dependence.

## Citation

```bibtex
@misc{zafar2026beyondaccuracy,
  title={Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning},
  author={Zafar et al. (2026)},
  year={2026},
  note={arXiv:2603.03437}
}
```

- arXiv: 2603.03437