Drbench Eval

---
name: drbench-eval
description: Evaluates generative models' ability to perform clinical diagnostic reasoning, including medical knowledge representation, evidence synthesis, and diagnosis generation. The benchmark spans sentence-level to full-note tasks to probe abstractive reasoning and clinical knowledge inference. Use when the user wants to benchmark on DR.BENCH, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.14901
  bibtex_key: gao2022drbench
  confidence: high
---

# drbench-eval

> DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing — Gao et al. (2022) (arXiv:2209.14901, 2022)

## What this evaluates

Evaluates generative models' ability to perform clinical diagnostic reasoning, including medical knowledge representation, evidence synthesis, and diagnosis generation. The benchmark spans sentence-level to full-note tasks to probe abstractive reasoning and clinical knowledge inference.

## Datasets

- **DR.BENCH** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Proportion of correctly predicted labels or answers out of total instances.
- `macro F1` — range: percent
  - Unweighted mean of recall or precision across all classes, typically used for relation labeling tasks.
- `ROUGE-L` — range: percent
  - Longest common subsequence-based recall and precision score between generated summary and reference summary.

## Input / output format

**Input**: Clinical text inputs such as progress notes, assessment sections, or medical board exam questions, sometimes with retrieved context paragraphs.

**Output**: Sequence generation: diagnostic labels, assessment-plan relations, problem list summaries, or multiple-choice answers.

## Scoring recipe

```python
def score(predictions, golds, metric):
    if metric == 'accuracy':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif metric == 'macro F1':
        return f1_score(golds, predictions, average='macro')
    elif metric == 'ROUGE-L':
        return rouge_score(golds, predictions, rouge_types=['rougeL'])
```

## Common pitfalls

- Different tasks use different evaluation metrics (accuracy, F1, ROUGE-L), complicating direct cross-task comparison.
- 95% confidence intervals are computed via bootstrapping, not analytical formulas.
- Summarization tasks report very low ROUGE-L scores, which may not fully capture clinical reasoning quality.

## Evidence (verbatim from paper)

> The models achieved the best performance on MedNLI with an accuracy range between 79.75% and 84.88% (Table 4). Problem summarization (SUMM-NOTE), which was intended as the most challenging task, had the lowest performance across all models, with Rouge-L scores between 2.14% and 5.66% (Table 7).

## Citation

```bibtex
@misc{gao2022drbench,
  title={DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing},
  author={Gao et al. (2022)},
  year={2022},
  note={arXiv:2209.14901}
}
```

- arXiv: 2209.14901