Medq Deg Eval

---
name: medq-deg-eval
description: Evaluates multimodal large language models' robustness and metacognitive reliability when processing medical images with various quality degradations (e.g., blur, noise, motion, artifacts) across different clinical capability dimensions. Use when the user wants to benchmark on MedQ-Deg, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.07769
  bibtex_key: liu2026medqdeg
  confidence: high
---

# medq-deg-eval

> MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations — Liu et al. (2026) (arXiv:2603.07769, 2026)

## What this evaluates

Evaluates multimodal large language models' robustness and metacognitive reliability when processing medical images with various quality degradations (e.g., blur, noise, motion, artifacts) across different clinical capability dimensions.

## Datasets

- **MedQ-Deg** — total 24894; splits: test (24894)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered clinical questions. Computed as the number of correct predictions divided by the total number of QA pairs.
- `Calibration Shift` — range: other
  - Measures the change in confidence-accuracy alignment between clean (L0) and degraded (L1/L2) images, quantifying the AI Dunning-Kruger effect where models maintain high confidence despite accuracy drops.

## Input / output format

**Input**: A medical image paired with a clinical question requiring a text answer.

**Output**: A text response answering the clinical question, optionally accompanied by a confidence score.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100
```

## Common pitfalls

- Performance is reported as a drop from clean (L0) to degraded (L1/L2) baselines, so clean accuracy must be tracked separately to compute the delta.
- Degradations are synthetically simulated rather than captured from real clinical devices, though distribution alignment is validated.
- Confidence scores used for Calibration Shift may not be natively output by all MLLMs and might require prompting or logit extraction.

## Evidence (verbatim from paper)

> Average accuracy across the mid-level capability dimensions, computed over L1 and L2 degradations. Numbers in parentheses show performance drop from L0 to L1&L2.

## Citation

```bibtex
@misc{liu2026medqdeg,
  title={MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradations},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2603.07769}
}
```

- arXiv: 2603.07769