Diagnosisarena Eval

---
name: diagnosisarena-eval
description: Clinical diagnostic reasoning capability of LLMs, requiring them to generate plausible diagnoses from patient case descriptions and imaging/symptom details. It probes the model's ability to perform complex, multi-step medical deduction and generalize across 28 clinical specialties. Use when the user wants to benchmark on DiagnosisArena, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.14107
  bibtex_key: [REDACTED]
  confidence: high
---

# diagnosisarena-eval

> DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models — Zhu et al. (2025) (arXiv:2505.14107, 2025)

## What this evaluates

Clinical diagnostic reasoning capability of LLMs, requiring them to generate plausible diagnoses from patient case descriptions and imaging/symptom details. It probes the model's ability to perform complex, multi-step medical deduction and generalize across 28 clinical specialties.

## Datasets

- **DiagnosisArena** — total 1113; splits: test (-1); repo https://github.com/SPIRAL-MED/DiagnosisArena

## Metrics

- `accuracy` **(primary)** — range: percent
  - GPT-4o judges the model's open-ended diagnostic output against the ground truth, classifying it as identical, relevant, or irrelevant. Only identical counts as correct. Accuracy is the percentage of cases classified as identical.
- `top-k accuracy` — range: percent
  - Hit rate of the correct diagnosis within the top k predicted outcomes generated by the model in descending order of confidence.
- `multi-choice accuracy` — range: percent
  - Rule-based extraction of the model's selected option from a four-choice question, compared against the fixed ground truth option.

## Input / output format

**Input**: Clinical case descriptions including patient symptoms, imaging findings, and incidental observations, presented via a unified prompt instructing the model to generate diagnostic outcomes.

**Output**: Open-ended: Five possible diagnostic outcomes in descending order of confidence. Multi-choice: A single selected option from four predefined choices.

## Scoring recipe

```python
def score_open_ended(predictions, gold):
    judge = GPT4oJudge()
    correct = 0
    for pred, g in zip(predictions, gold):
        if judge.classify(pred, g) == 'identical':
            correct += 1
    return correct / len(predictions)

def score_topk(predictions, gold, k=3):
    correct = 0
    for preds, g in zip(predictions, gold):
        if g in preds[:k]:
            correct += 1
    return correct / len(predictions)

def score_mc(predictions, gold):
    correct = 0
    for pred, g in zip(predictions, gold):
        if extract_option(pred) == g:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Multiple-choice formats significantly inflate performance by narrowing the problem space, allowing models to rely on superficial cues rather than full diagnostic reasoning.
- Data leakage from pretraining corpora (e.g., medical journals) can artificially boost scores if cases overlap with training data.
- Open-ended evaluation relies on an LLM-as-a-judge (GPT-4o), which may introduce bias or inconsistency compared to rule-based exact matching.

## Evidence (verbatim from paper)

> For open-ended questions, the ground truth is a clear diagnostic conclusion. In clinical scenarios, medical diagnoses are generally classified into three categories: “identical”, “relevant”, and “irrelevant”. Therefore, we use GPT-4o as the judge to evaluate the output results into these three categories. Among them, only when the model’s result is judged to be “identical” is it considered correct. Additionally, we instruct the LLM to generate the $k$ possible diagnostic outcomes in descending order of confidence, and then calculate the Top $k$ accuracy, which is the hit rate of the correct answer within the top $k$ predicted results.

## Citation

```bibtex
@misc{zhu2025diagnosisarena,
  title={DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2505.14107}
}
```

- arXiv: 2505.14107