Medframeqa Eval

---
name: medframeqa-eval
description: This benchmark evaluates multi-image medical visual question answering and clinical reasoning. It probes a model's ability to integrate diagnostic evidence across temporally coherent medical images, detect salient findings, and propagate reasoning chains to answer single-choice questions. Use when the user wants to benchmark on MedFrameQA, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.16964
  bibtex_key: yu2025medframeqa
  confidence: high
---

# medframeqa-eval

> MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning — Yu et al. (2025) (arXiv:2505.16964, 2025)

## What this evaluates

This benchmark evaluates multi-image medical visual question answering and clinical reasoning. It probes a model's ability to integrate diagnostic evidence across temporally coherent medical images, detect salient findings, and propagate reasoning chains to answer single-choice questions.

## Datasets

- **MedFrameQA** — total 2851; splits: test (2851)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly answered single-choice questions out of the total number of VQA pairs. Calculated as (number of correct predictions / total questions) * 100.

## Input / output format

**Input**: A set of 2 to 5 medical images (key-frames from instructional videos) paired with a single-choice question that requires integrating information across all provided frames.

**Output**: A single selected option (letter or text) corresponding to the correct answer for the multiple-choice question.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Models frequently overlook or misinterpret intermediary frames, causing reasoning errors to cascade through subsequent steps.
- Performance is highly sensitive to imaging modality and anatomical system, so averaging across all tasks may mask significant capability gaps.
- Increasing the number of frames does not linearly decrease accuracy; performance fluctuates based on visual complexity and redundancy rather than frame count alone.

## Evidence (verbatim from paper)

> Table 2 presents the evaluation of 10 advanced MLLMs on MedFrameQA. In general, all assessed models demonstrate persistently low accuracy, with the peak accuracy remaining below 55.00%.

## Citation

```bibtex
@misc{yu2025medframeqa,
  title={MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2505.16964}
}
```

- arXiv: 2505.16964