Kormedmcqa V Eval

---
name: kormedmcqa-v-eval
description: This benchmark evaluates vision-language models on multimodal medical reasoning using questions derived from the Korean Medical Licensing Examination. It probes the models' ability to integrate textual and visual evidence across diverse clinical imaging modalities, including cross-image reasoning when multiple scans are provided. Use when the user wants to benchmark on KorMedMCQA-V, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.13650
  bibtex_key: [REDACTED]
  confidence: high
---

# kormedmcqa-v-eval

> KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examination — Choi et al. (2026) (arXiv:2602.13650, 2026)

## What this evaluates

This benchmark evaluates vision-language models on multimodal medical reasoning using questions derived from the Korean Medical Licensing Examination. It probes the models' ability to integrate textual and visual evidence across diverse clinical imaging modalities, including cross-image reasoning when multiple scans are provided.

## Datasets

- **KorMedMCQA-V** — total 1534; splits: test (1534); repo https://github.com/baeseongsu/kormedmcqa_v

## Metrics

- `accuracy` **(primary)** — range: percent
  - Exact-match accuracy calculated as the fraction of correctly predicted option labels (A–E) out of the total number of questions. Reported as a percentage.

## Input / output format

**Input**: A question stem, five multiple-choice answer options (A–E), and one or more associated medical images (e.g., X-ray, CT, ECG, ultrasound, endoscopy).

**Output**: The selected option label (A–E) formatted in JSON.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p.strip().upper() == g.strip().upper())
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Answer-position bias: The dataset has a strong frequency bias toward option 'E', as shown by the majority-label baseline scoring 22.4%.
- Multi-image degradation: Models consistently underperform on questions requiring evidence integration across 2 or 3+ images compared to single-image items.
- Modality-specific variance: Overall accuracy masks large performance gaps across imaging types (e.g., MRI/PBS vs. NST/Endoscopy), requiring per-modality analysis.

## Evidence (verbatim from paper)

> We score predictions by exact match between the predicted and gold option labels and report accuracy. For open-source models, we run three random seeds and report the average.

## Citation

```bibtex
@misc{choi2026kormedmcqa,
  title={KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examination},
  author={Choi et al. (2026)},
  year={2026},
  note={arXiv:2602.13650}
}
```

- arXiv: 2602.13650