Ddxplus Diagnostic Reasoning Eval

---
name: ddxplus-diagnostic-reasoning-eval
description: Probes large language models' ability to perform iterative medical diagnostic reasoning through a simulated patient-doctor dialogue. It evaluates whether models can gather clinical evidence, generate differential diagnoses, and converge on the correct ground-truth pathology within a limited number of interaction turns. Use when the user wants to benchmark on DDXPlus, or asks about evaluating this task. Reports diagnostic accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.08922
  bibtex_key: wu2023largellmsperformdiagnosticreasoning
  confidence: high
---

# ddxplus-diagnostic-reasoning-eval

> Large Language Models Perform Diagnostic Reasoning — Cheng-Kuang Wu et al. (2023) (arXiv:2307.08922, 2023)

## What this evaluates

Probes large language models' ability to perform iterative medical diagnostic reasoning through a simulated patient-doctor dialogue. It evaluates whether models can gather clinical evidence, generate differential diagnoses, and converge on the correct ground-truth pathology within a limited number of interaction turns.

## Datasets

- **DDXPlus** — total ?; splits: test (-1)

## Metrics

- `diagnostic accuracy` **(primary)** — range: [0, 1]
  - The proportion of instances where the model's final predicted diagnosis exactly matches the ground-truth PATHOLOGY. Computed as correct predictions divided by total test instances.

## Input / output format

**Input**: Patient profile constructed from AGE, SEX, and INITIAL_EVIDENCE, followed by a dialogue history of clinical findings (EVIDENCES) gathered turn-by-turn.

**Output**: Final diagnosis prediction (PATHOLOGY), optionally preceded by a differential diagnosis list and evidence summary. Forced at turn 8 if not generated earlier.

## Scoring recipe

```python
def compute_diagnostic_accuracy(predictions, gold_pathologies):
    correct = 0
    for pred, gold in zip(predictions, gold_pathologies):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return correct / len(gold_pathologies)
```

## Common pitfalls

- Ignoring the hard turn limit (T_max=8), which forces a diagnosis via an '[inform]' prefix and artificially inflates accuracy if models are allowed to continue indefinitely.
- Misclassifying domain splits: in-domain vs out-domain is defined strictly by whether the INITIAL_EVIDENCE (IE) matches the few-shot exemplars, not by random patient ID or symptom category.
- Using non-zero temperature, which introduces unnecessary randomness in a deterministic diagnostic reasoning task where the paper explicitly sets temperature to 0.

## Evidence (verbatim from paper)

> The AGE, SEX, IE, and EVIDENCES are used to construct the patient profile P based on a rule-based template, and the PATHOLOGY is the ground truth diagnosis for computing the diagnostic accuracy. To evaluate the diagnosis accuracy of dialogue turn $T$, if the final diagnosis has not been generated, we instruct the DSAD to make the final diagnosis by inserting a prefix "[inform]" in turn $T$'s prompt (see Appendix A.1, A.2). And to avoid lengthy dialogue, we set a pre-defined $T_{max}$ as the maximum number of turn allowed for making the final diagnosis. We determine $T_{max}$ by inviting a physician to interact with our patient bot, and find that $6\sim 8$ turns are required on average. Thus, we set $T_{max} = 8$ for our experiments.

## Citation

```bibtex
@misc{wu2023largellmsperformdiagnosticreasoning,
  title={Large Language Models Perform Diagnostic Reasoning},
  author={Cheng-Kuang Wu et al. (2023)},
  year={2023},
  note={arXiv:2307.08922}
}
```

- arXiv: 2307.08922