Omnibrainbench Eval

---
name: omnibrainbench-eval
description: Evaluates multimodal large language models' ability to perform visual-to-clinical reasoning on brain imaging data. It probes capabilities ranging from basic anatomical identification to complex multi-stage clinical decision-making and prognosis prediction. Use when the user wants to benchmark on OmniBrainBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.00846
  bibtex_key: [REDACTED]
  confidence: high
---

# omnibrainbench-eval

> OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks — Peng et al. (2025) (arXiv:2511.00846, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to perform visual-to-clinical reasoning on brain imaging data. It probes capabilities ranging from basic anatomical identification to complex multi-stage clinical decision-making and prognosis prediction.

## Datasets

- **OmniBrainBench** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions across all 15 clinical subtasks. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: Brain imaging images (spanning 15 clinical modalities) paired with clinical questions or prompts requiring visual-to-clinical reasoning.

**Output**: Text-based answers to clinical questions, evaluated for correctness against ground-truth physician responses.

## Scoring recipe

```python
correct = 0
total = 0
for task in tasks:
    for question, gold in task.items():
        pred = model.generate(image, question)
        if pred == gold:
            correct += 1
        total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Assuming high scores in visual perception tasks translate to reliable clinical reasoning capabilities.
- Treating benchmark accuracy as a substitute for real-world clinical validation, despite the paper's explicit warning that it cannot replace final clinical evaluation.
- Overlooking task-specific variability; models may excel in structured tasks but fail significantly in complex reasoning tasks like risk stratification.

## Evidence (verbatim from paper)

> We conduct extensive experiments on MLLMs across five specialized clinical tasks with 15 secondary subtasks to summarize their current capabilities and limitations... The physician achieves an average accuracy of 91.35% across all tasks, whereas the highest-performing model, Gemini-2.5-Pro, attained only 66.58%—reflecting a substantial performance gap of approximately 24.77%.

## Citation

```bibtex
@misc{peng2025omnibrainbench,
  title={OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2511.00846}
}
```

- arXiv: 2511.00846