Med Mim Eval

---
name: med-mim-eval
description: Evaluates medical vision-language models on multi-image reasoning tasks, including temporal understanding, cross-modal comparison, multi-view diagnosis, and co-reference resolution across longitudinal and multi-modality medical imaging data. Use when the user wants to benchmark on Med-MIM Benchmark, or asks about evaluating this task. Reports closed-type accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19031
  bibtex_key: yang2025medmim
  confidence: high
---

# med-mim-eval

> Medical Large Vision Language Models with Multi-Image Visual Ability — Xikai Yang et al. (2025) (arXiv:2505.19031, 2025)

## What this evaluates

Evaluates medical vision-language models on multi-image reasoning tasks, including temporal understanding, cross-modal comparison, multi-view diagnosis, and co-reference resolution across longitudinal and multi-modality medical imaging data.

## Datasets

- **Med-MIM Benchmark** — total 4424; splits: held-in (3224), held-out (1200)

## Metrics

- `closed-type accuracy` **(primary)** — range: [0, 1]
  - Exact match between the model's predicted answer and the ground truth for closed-type questions. Open-type questions use separate open-ended scoring protocols.

## Input / output format

**Input**: Interleaved sequence of up to three medical images and a text question. Images are formatted as `(image {id}: <Image> image embeddings </Image>)`.

**Output**: Text response answering the medical question.

## Scoring recipe

```python
def score(predictions, gold, is_closed):
    if is_closed:
        return 1.0 if predictions.strip().lower() == gold.strip().lower() else 0.0
    else:
        return open_ended_scoring(predictions, gold)
```

## Common pitfalls

- Models must handle sequences of up to three images, not just single images.
- Composed dataset samples use location-specific prefixes (e.g., 'In the first image') that must be parsed correctly.
- Temporal tasks require chronological ordering of multi-visit images.
- Closed-type and open-type examples require different evaluation protocols.

## Evidence (verbatim from paper)

> Our constructed Med-MIM Benchmark comprises two parts. (a) Held-in part: To comprehensively evaluate four multi-image visual abilities, we construct the held-in Med-MIM Benchmark derived from the Med-MIM instruction dataset, which includes 2,968 closed-type examples (903, 454, 208, and 1,403 for temporal, reasoning, comparison, co-reference abilities, respectively) and 256 open-type examples (30, 30, 136, and 60, respectively).

## Citation

```bibtex
@misc{yang2025medmim,
  title={Medical Large Vision Language Models with Multi-Image Visual Ability},
  author={Xikai Yang et al. (2025)},
  year={2025},
  note={arXiv:2505.19031}
}
```

- arXiv: 2505.19031