Mmxu Test Eval

---
name: mmxu-test-eval
description: Evaluates multi-modal vision-language models on their ability to perform visual question answering across two temporal X-ray images to detect regional disease progression. It probes temporal reasoning, subtle change detection, and bias mitigation in medical imaging diagnostics. Use when the user wants to benchmark on MMXU-test, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.11651
  bibtex_key: mu2025mmxu
  confidence: high
---

# mmxu-test-eval

> MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression — Linjie Mu et al. (2025) (arXiv:2502.11651, 2025)

## What this evaluates

Evaluates multi-modal vision-language models on their ability to perform visual question answering across two temporal X-ray images to detect regional disease progression. It probes temporal reasoning, subtle change detection, and bias mitigation in medical imaging diagnostics.

## Datasets

- **MMXU-test** — total 3000; splits: test (3000); repo https://github.com/linjiemu/MMXU

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Accuracy is calculated as the proportion of correctly answered single-choice questions out of the total number of questions. It is reported overall and stratified by disease progression category: Worsen, Improved, and No change.

## Input / output format

**Input**: Two X-ray images (representing different time points) paired with a single-choice question about regional disease progression or patient history.

**Output**: A single categorical answer choice corresponding to the correct option for the question.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- Models exhibit strong bias toward predicting 'Worsen' outcomes, especially smaller open-source models, which inflates overall accuracy if not stratified.
- Single-image models are evaluated by concatenating two X-rays, which may degrade performance compared to native multi-image architectures.
- Human expert evaluation shows variability (e.g., Expert 3 at 90.6%), indicating potential ambiguity in some progression questions.

## Evidence (verbatim from paper)

> Since all the questions from our benchmark are single-choice, we use accuracy as the metric. The data from the MMXU-test benchmark was randomly divided into five parts, containing 500, 500, 500, 750, and 750 questions, respectively. We ensured that the three question categories were distributed as evenly as possible within each subset.

## Citation

```bibtex
@misc{mu2025mmxu,
  title={MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progression},
  author={Linjie Mu et al. (2025)},
  year={2025},
  note={arXiv:2502.11651}
}
```

- arXiv: 2502.11651