Jama Clinical Challenge Eval

---
name: jama-clinical-challenge-eval
description: Evaluates medical multimodal models on real-world diagnostic reasoning using clinical case images and questions. It probes both factual accuracy in close-ended QA and the model's ability to generate clinically sound reasoning across key points, inference steps, and evidence citation. Use when the user wants to benchmark on JAMA Clinical Challenge, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.14948
  bibtex_key: wang2024semihvision
  confidence: high
---

# jama-clinical-challenge-eval

> SemiHVision: Enhancing Medical Multimodal Models with a Semi-Human Annotated Dataset and Fine-Tuned Instruction Generation — Wang et al. (2024) (arXiv:2410.14948, 2024)

## What this evaluates

Evaluates medical multimodal models on real-world diagnostic reasoning using clinical case images and questions. It probes both factual accuracy in close-ended QA and the model's ability to generate clinically sound reasoning across key points, inference steps, and evidence citation.

## Datasets

- **JAMA Clinical Challenge** — total ?; splits: test (-1); repo https://github.com/believewhat/SemiHVision

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Standard close-ended QA accuracy calculated as the proportion of correct predictions over total instances.
- `UMLS Factuality` — range: [0, 1]
  - Measures factual consistency by computing the overlap between model outputs and gold answers using UMLS medical concept identifiers.
- `GPT-4 Overall` — range: other
  - Automated LLM-judge score evaluating the holistic quality and clinical appropriateness of the diagnostic response.
- `GPT-4 Key-Points` — range: other
  - LLM-judge score assessing the completeness of critical diagnostic points in the model's output.
- `GPT-4 Inference` — range: other
  - LLM-judge score measuring the logical reasoning steps and diagnostic deduction quality.
- `GPT-4 Evidence` — range: other
  - LLM-judge score evaluating the citation and relevance of supporting clinical evidence.

## Input / output format

**Input**: Medical image(s) paired with a clinical question or case description requiring diagnosis or reasoning.

**Output**: Text response containing the diagnosis/answer, followed by reasoning steps, key points, inferences, and cited evidence.

## Scoring recipe

```python
def compute_metrics(predictions, golds, prompts):
    acc = sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(predictions)
    umls_f = compute_uMLS_overlap(predictions, golds)
    gpt_scores = {dim: [] for dim in ['Overall', 'Key-Points', 'Inference', 'Evidence']}
    for p, g, prompt in zip(predictions, golds, prompts):
        judge_output = llm_judge(prompt, p, g, dimensions=['Overall', 'Key-Points', 'Inference', 'Evidence'])
        for dim in gpt_scores:
            gpt_scores[dim].append(parse_score(judge_output[dim]))
    return acc, umls_f, {k: mean(v) for k, v in gpt_scores.items()}
```

## Common pitfalls

- The exact GPT-4 automated scoring prompts and rubrics are detailed in Section 4.4, which is not provided in this excerpt, making exact reproduction difficult.
- Traditional benchmarks (SLAKE, VQA-RAD) are heavily knowledge-recall biased (~70-78% knowledge questions), whereas JAMA emphasizes inference (~55% inference questions), so comparing raw accuracy across datasets is misleading.
- PathVQA dataset classification required expert intervention due to GPT-4o's domain knowledge gaps, highlighting potential bias in automated dataset analysis.

## Evidence (verbatim from paper)

> Accuracy was measured using standard methodologies for close-ended QA tasks, while diagnostic reasoning was evaluated through the automatic scoring pipeline described in Section 4.4, which measures performance across three key dimensions: Key Points, Inference, and Evidence.

## Citation

```bibtex
@misc{wang2024semihvision,
  title={SemiHVision: Enhancing Medical Multimodal Models with a Semi-Human Annotated Dataset and Fine-Tuned Instruction Generation},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2410.14948}
}
```

- arXiv: 2410.14948