Bj Benchmark Eval

---
name: bj-benchmark-eval
description: This benchmark evaluates vision-language and large language models on clinical reasoning for musculoskeletal disorders. It probes capabilities ranging from medical knowledge recall and unimodal interpretation to open-ended multimodal diagnosis, treatment planning, and text-image inconsistency detection. The protocol highlights the performance gap between structured multiple-choice questions and complex, free-form clinical reasoning tasks. Use when the user wants to benchmark on B&J benchmark, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.22275
  bibtex_key: wang2025illusion
  confidence: high
---

# bj-benchmark-eval

> The Illusion of Clinical Reasoning: A Benchmark Reveals the Pervasive Gap in Vision-Language Models for Clinical Competency — Wang et al. (2025) (arXiv:2512.22275, 2025)

## What this evaluates

This benchmark evaluates vision-language and large language models on clinical reasoning for musculoskeletal disorders. It probes capabilities ranging from medical knowledge recall and unimodal interpretation to open-ended multimodal diagnosis, treatment planning, and text-image inconsistency detection. The protocol highlights the performance gap between structured multiple-choice questions and complex, free-form clinical reasoning tasks.

## Datasets

- **B&J benchmark** — total 1245; splits: test (1245)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correct responses out of total instances. For MCQs, it is exact match to the correct option. For open-ended tasks, it is determined via GPT-4-assisted scoring and manual expert review against ground truth diagnostic/treatment plans or reasoning chains.

## Input / output format

**Input**: For MCQ tasks: text-only or image-only prompts with five answer options. For multimodal open-ended tasks: patient medical history, physical examination findings, and radiological images (or radiology reports for LLMs). Models must generate diagnoses, treatment plans, or chain-of-thought reasoning.

**Output**: For MCQs: a single selected option letter. For open-ended tasks: free-text responses containing diagnostic conclusions, therapeutic recommendations, and/or step-by-step clinical reasoning.

## Scoring recipe

```python
def compute_accuracy(predictions, gold, task_type):
    if task_type == 'MCQ':
        return 1.0 if predictions.strip() == gold.strip() else 0.0
    else:
        # Open-ended tasks scored via GPT-4-assisted evaluation and manual expert review
        # Returns 1.0 if prediction aligns with expert ground truth in clinical validity,
        # 0.0 otherwise.
        return 1.0 if matches_expert_criteria(predictions, gold) else 0.0
```

## Common pitfalls

- Models exhibit a strong 'text shortcut' tendency, generating diagnoses based on linguistic cues rather than analyzing visual evidence.
- Performance drops drastically on open-ended tasks compared to structured MCQs, even for models fine-tuned on medical data.
- LLMs are evaluated using radiology reports instead of actual images due to modality limitations, which inflates their apparent multimodal reasoning capability.

## Evidence (verbatim from paper)

> In the MCQs requiring medical knowledge recall (Task I) and clinical note interpretation (Task II), SOTA VLMs such as GPT-4o achieved accuracy of 86.8% and 95.0%, respectively. In stark contrast, their performance on radiological image interpretation MCQs (Task III) was significantly poor, with accuracy ranging from 20.0% to 40.0%... On these more complex tasks, even the best-performing VLMs, including GPT-4o and Claude 3.5 Sonnet, failed to achieve an accuracy of 60.0%.

## Citation

```bibtex
@misc{wang2025illusion,
  title={The Illusion of Clinical Reasoning: A Benchmark Reveals the Pervasive Gap in Vision-Language Models for Clinical Competency},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2512.22275}
}
```

- arXiv: 2512.22275