Er Reason Eval

---
name: er-reason-eval
description: Evaluates LLMs on longitudinal clinical reasoning across five emergency room workflow stages, including acuity assessment, case summarization, treatment planning, final diagnosis, and patient disposition. It probes the models' ability to integrate sparse clinical notes, perform rule-out differential diagnosis, and align outputs with real-world clinical decision-making and safety constraints. Use when the user wants to benchmark on ER-Reason, or asks about evaluating this task. Reports Accuracy, ROUGE-F1, cTAKES CUI Overlap Ratio.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.22919
  bibtex_key: [REDACTED]
  confidence: high
---

# er-reason-eval

> ER-REASON: A Benchmark Dataset for LLM-Based Clinical Reasoning in the Emergency Room — Mehandru et al. (2025) (arXiv:2505.22919, 2025)

## What this evaluates

Evaluates LLMs on longitudinal clinical reasoning across five emergency room workflow stages, including acuity assessment, case summarization, treatment planning, final diagnosis, and patient disposition. It probes the models' ability to integrate sparse clinical notes, perform rule-out differential diagnosis, and align outputs with real-world clinical decision-making and safety constraints.

## Datasets

- **ER-Reason** — total 3984; splits: test (-1); repo https://github.com/AlaaLab/ER-Reason

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted categorical labels (acuity level or disposition) out of total instances.
- `ROUGE-F1` **(primary)** — range: [0, 1]
  - F1 score computed over unigram (ROUGE-1), bigram (ROUGE-2), and longest common subsequence (ROUGE-L) overlaps between generated summaries and reference summaries.
- `cTAKES CUI Overlap Ratio` **(primary)** — range: [0, 1]
  - Average ratio of overlapping Clinical Unit Identifiers (CUIs) extracted via cTAKES between model-generated treatment plans and physician-annotated rationales across three reasoning dimensions.
- `ICD-10 Match` — range: percent
  - Exact match percentage of predicted ICD-10 codes against gold-standard diagnostic codes.
- `HCC Match` — range: percent
  - Percentage of predictions matching the gold standard when aggregated into broader Hierarchical Condition Category groups to mitigate lexical coding variations.

## Input / output format

**Input**: Longitudinal de-identified clinical notes across ER stages, often accompanied by task-specific prompts detailing the intended clinical workflow context.

**Output**: Task-dependent: categorical acuity level, one-line patient summary, structured treatment plan/reasoning rationale, ICD-10 diagnosis code, and final disposition (e.g., discharge vs. admission).

## Scoring recipe

```python
acc = sum(pred == gold for pred, gold in zip(predictions, golds)) / len(golds)
rouge_scores = rouge.compute(predictions=predictions, references=golds, rouge_types=['rouge1', 'rouge2', 'rougeL'])
model_cuis = ctakes_extract(model_output)
gold_cuis = ctakes_extract(gold_rationale)
overlap = len(model_cuis & gold_cuis) / len(gold_cuis)
icd_match = sum(pred == gold for pred, gold in zip(icd_preds, icd_golds)) / len(golds)
hcc_match = sum(map_hcc(pred) == map_hcc(gold) for pred, gold in zip(icd_preds, icd_golds)) / len(golds)
```

## Common pitfalls

- Models exhibit a 'regression to the mean' or risk-averse bias, over-classifying patients as Urgent or recommending admission, which compresses the acuity spectrum and skews resource allocation predictions.
- Exact ICD-10 code matching is heavily penalized by free-text documentation variations (e.g., 'sepsis' vs 'acute sepsis'), making HCC category matching a more robust diagnostic fidelity metric.
- Summarization ROUGE scores remain modest because the task requires workflow-aware reasoning about which sparse clinical details are actionable, rather than general text compression.

## Evidence (verbatim from paper)

> Table 2: ER acuity assessment

| Model | Accuracy |
| --- | --- |
| LLaMA 3.2 | 53.63 |
| GPT-3.5 | 62.11 |
| GPT-4o | 61.98 |
| o3-mini | 62.70 |

## Citation

```bibtex
@misc{mehandru2025erreason,
  title={ER-REASON: A Benchmark Dataset for LLM-Based Clinical Reasoning in the Emergency Room},
  author={Mehandru et al. (2025)},
  year={2025},
  note={arXiv:2505.22919}
}
```

- arXiv: 2505.22919