Medq Bench Eval

---
name: medq-bench-eval
description: Probes multimodal large language models' ability to assess medical image quality through low-level visual attribute detection and no-reference or comparative reasoning. It evaluates how well models identify image degradations, describe clinical attributes, and compare quality across different imaging modalities. Use when the user wants to benchmark on MedQ-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.01691
  bibtex_key: liu2025medqbench
  confidence: high
---

# medq-bench-eval

> MedQ-Bench: Evaluating and Exploring Medical Image Quality Assessment Abilities in MLLMs — Liu et al. (2025) (arXiv:2510.01691, 2025)

## What this evaluates

Probes multimodal large language models' ability to assess medical image quality through low-level visual attribute detection and no-reference or comparative reasoning. It evaluates how well models identify image degradations, describe clinical attributes, and compare quality across different imaging modalities.

## Datasets

- **MedQ-Bench** — total 3308; splits: dev (-1), test (-1); repo https://github.com/liujiyaoFDU/MedQBench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions across Yes-or-No, What, and How subtasks in the perception phase.

## Input / output format

**Input**: A medical image paired with a task-specific prompt (multiple-choice question for perception; open-ended or structured prompt for no-reference or comparative quality reasoning).

**Output**: For perception: a single selected option from the provided choices. For reasoning: a generated text response describing image quality attributes, which is subsequently scored by an automated rubric.

## Scoring recipe

```python
perception_correct = sum(1 for pred, gold in perception_preds if pred == gold)
perception_acc = (perception_correct / len(perception_preds)) * 100

# Reasoning scoring (automated rubric validated against humans)
comp = score_dimension(predictions, gold, 'completeness') # 0-2
prec = score_dimension(predictions, gold, 'preciseness') # 0-2
cons = score_dimension(predictions, gold, 'consistency') # 0-2
qual = score_dimension(predictions, gold, 'quality')     # 0-2
reasoning_overall = comp + prec + cons + qual
```

## Common pitfalls

- Assuming medical-specialized MLLMs inherently outperform generalist models on IQA tasks (the study shows the opposite).
- Overestimating model robustness to subtle artifacts; performance actually drops for mild degradations compared to severe ones.
- Treating reasoning outputs as binary correct/incorrect; they require multi-dimensional rubric scoring (0–2 per dimension) rather than exact-match accuracy.

## Evidence (verbatim from paper)

> The confusion matrices in the appendix (Figure[13]) demonstrate strong alignment between human expert scores and GPT-4o automated evaluation across all three dimensions, with consistently high accuracy rates: 83.3% for completeness, 87.0% for preciseness, and 90.5% for consistency, with all individual class recall rates exceeding 80%.

## Citation

```bibtex
@misc{liu2025medqbench,
  title={MedQ-Bench: Evaluating and Exploring Medical Image Quality Assessment Abilities in MLLMs},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2510.01691}
}
```

- arXiv: 2510.01691