Neural Medbench Eval

---
name: neural-medbench-eval
description: Neural-MedBench probes the clinical reasoning and multimodal synthesis capabilities of vision-language models in neurology diagnostics. It specifically tests whether models can move beyond superficial classification to perform uncertainty resolution, generate clinically justified rationales, and maintain logical coherence when interpreting patient histories and medical imaging. Use when the user wants to benchmark on Neural-MedBench, or asks about evaluating this task. Reports Diagnostic Accuracy (pass@1).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.22258
  bibtex_key: miao2025neuralmedbench
  confidence: high
---

# neural-medbench-eval

> Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks — Miao Jing et al. (2025) (arXiv:2509.22258, 2025)

## What this evaluates

Neural-MedBench probes the clinical reasoning and multimodal synthesis capabilities of vision-language models in neurology diagnostics. It specifically tests whether models can move beyond superficial classification to perform uncertainty resolution, generate clinically justified rationales, and maintain logical coherence when interpreting patient histories and medical imaging.

## Datasets

- **Neural-MedBench** — total 200; splits: test (200)

## Metrics

- `Diagnostic Accuracy (pass@1)` **(primary)** — range: [0, 1]
  - Binary metric indicating whether the model's top-1 predicted diagnosis exactly matches the gold-standard diagnosis.
- `Diagnostic Accuracy (pass@5)` — range: [0, 1]
  - Binary metric indicating whether the correct diagnosis appears anywhere within the model's top-5 differential diagnosis list.
- `Semantic Fidelity (BERTScore)` — range: [0, 1]
  - Computes the BERT-based F1 score between the model's generated rationale and the expert reference rationale to measure semantic alignment.
- `Reasoning Fidelity (LLM Grader)` — range: [0, 1]
  - A clinically calibrated GPT-4o model scores the output on correctness, logical coherence, and evidence grounding against clinical standards.

## Input / output format

**Input**: A structured prompt initializing the model as an 'experienced neurologist', followed by a clinical task containing textual narratives, structured patient history, and medical imaging (MRI/CT encoded in base64).

**Output**: A differential diagnosis list (up to 5 options) and a free-form clinical rationale explaining the diagnostic reasoning.

## Scoring recipe

```python
def score(predictions, gold):
    # Diagnostic Accuracy
    pass1 = 1.0 if predictions['top1_diag'] == gold['diag'] else 0.0
    pass5 = 1.0 if gold['diag'] in predictions['top5_diag'] else 0.0
    
    # Semantic Fidelity
    bert_score = compute_bertscore(predictions['rationale'], gold['rationale'])
    
    # Reasoning Fidelity
    grader_prompt = build_clinical_grader_prompt(predictions['rationale'], gold['diag'])
    reasoning_score = gpt4o_grader.evaluate(grader_prompt) # Returns 0-1 or categorical
    
    return {'pass@1': pass1, 'pass@5': pass5, 'bertscore': bert_score, 'reasoning_fidelity': reasoning_score}
```

## Common pitfalls

- Assuming high performance on broad classification benchmarks implies strong clinical reasoning competence, which the paper explicitly debunks.
- Relying solely on exact-match diagnosis accuracy without evaluating the quality of the generated rationale or differential diagnosis breadth.
- Failing to distinguish between perceptual errors (misreading images) and reasoning/knowledge gaps when analyzing model failures.

## Evidence (verbatim from paper)

> Diagnostic Accuracy (pass@k): For tasks with definitive outcomes, we report top-1 (pass@1) and top-5 (pass@5) accuracy. The latter measures whether the correct diagnosis appears within a model’s differential. Semantic Fidelity (BERTScore): Used for free-form rationale generation, capturing semantic alignment between model outputs and expert references. Reasoning Fidelity (LLM Grader): A clinically calibrated GPT-4o grader evaluates correctness, logical coherence, and evidence grounding (see Section[5.2]).

## Citation

```bibtex
@misc{miao2025neuralmedbench,
  title={Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks},
  author={Miao Jing et al. (2025)},
  year={2025},
  note={arXiv:2509.22258}
}
```

- arXiv: 2509.22258