Medrcube Eval

---
name: medrcube-eval
description: Probes multimodal large language models' fine-grained capabilities in medical imaging across anatomical regions, imaging modalities, and cognitive hierarchies. It assesses reasoning reliability, shortcut behavior, and foundational perceptual skills to reveal how models handle clinical VQA beyond aggregate performance. Use when the user wants to benchmark on MedRCube, or asks about evaluating this task. Reports MedRCube Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.13756
  bibtex_key: bao2026medrcube
  confidence: high
---

# medrcube-eval

> MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging — Bao et al. (2026) (arXiv:2604.13756, 2026)

## What this evaluates

Probes multimodal large language models' fine-grained capabilities in medical imaging across anatomical regions, imaging modalities, and cognitive hierarchies. It assesses reasoning reliability, shortcut behavior, and foundational perceptual skills to reveal how models handle clinical VQA beyond aggregate performance.

## Datasets

- **MedRCube** — total ?; splits: test (-1); repo https://github.com/F1mc/MedRCube

## Metrics

- `MedRCube Score` **(primary)** — range: percent
  - Aggregate accuracy score computed as the average of per-task accuracies across the multidimensional task hierarchy (anatomical region, modality, and cognitive level). Reported on a 0–100 scale.

## Input / output format

**Input**: Paired medical imaging scans (CT, X-ray, MRI, Ultrasound) with multiple-choice clinical questions requiring perception, semantic understanding, or cognitive reasoning.

**Output**: A single selected option/answer per question.

## Scoring recipe

```python
def compute_medrcube_score(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if p == g)
    total = len(gold_answers)
    return (correct / total) * 100 if total > 0 else 0
```

## Common pitfalls

- Models may achieve high scores by relying on text-only priors or shortcut heuristics rather than genuine visual reasoning, as shown by the text-only baseline dropping to near random guessing.
- Assuming parameter scaling guarantees performance; the evaluation shows models >10B fail to establish a decisive advantage over <10B models in medical imaging VQA.
- Overlooking foundational perceptual blind spots (e.g., imaging protocol recognition) that are assumed to be prerequisites for higher-level clinical reasoning but actually score poorly.

## Evidence (verbatim from paper)

> Lingshu-32B achieves the best performance (62.55), followed by Lingshu-7B (59.86), Gemini-3-Pro (59.35), and InternVL3.5-38B (58.14). Notably, the top-performing models span both proprietary and open-source paradigms, as well as general-purpose and medically specialized training regimes. At the same time, a large fraction of models—including several medically fine-tuned systems—cluster in a relatively narrow performance band (approximately 45 - 55), indicating that flat aggregate metrics compress substantial underlying heterogeneity.

## Citation

```bibtex
@misc{bao2026medrcube,
  title={MedRCube: A Multidimensional Framework for Fine-Grained and In-Depth Evaluation of MLLMs in Medical Imaging},
  author={Bao et al. (2026)},
  year={2026},
  note={arXiv:2604.13756}
}
```

- arXiv: 2604.13756