Ehrnoteqa Eval

---
name: ehrnoteqa-eval
description: Evaluates large language models' ability to perform patient-specific clinical reasoning by synthesizing information from multiple electronic health record (EHR) discharge summaries to answer medical questions. It specifically tests multi-document clinical analysis and automated medical model evaluation using structured multi-choice or free-text formats. Use when the user wants to benchmark on EHRNoteQA, or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.16040
  bibtex_key: [REDACTED]
  confidence: high
---

# ehrnoteqa-eval

> EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries — Kweon et al. (2024) (arXiv:2402.16040, 2024)

## What this evaluates

Evaluates large language models' ability to perform patient-specific clinical reasoning by synthesizing information from multiple electronic health record (EHR) discharge summaries to answer medical questions. It specifically tests multi-document clinical analysis and automated medical model evaluation using structured multi-choice or free-text formats.

## Datasets

- **EHRNoteQA** — total 962; splits: test (962); repo https://github.com/ji-youn-kim/EHRNoteQA

## Metrics

- `score` **(primary)** — range: [0, 100]
  - Models receive 1 point for a correct answer and 0 for incorrect. The final score is the average points across all questions, normalized to a 100-point scale.

## Input / output format

**Input**: Patient discharge summaries (clinical notes), a clinical question, and multiple-choice answer options (for multi-choice evaluation) or just the question (for free-text evaluation).

**Output**: Model-generated text response, either selecting an answer choice letter/sequence or providing a free-text explanation.

## Scoring recipe

```python
def compute_score(predictions, gold_answers):
    total_score = 0.0
    for pred, gold in zip(predictions, gold_answers):
        iter_results = []
        for _ in range(5):
            judge = call_gpt4turbo(f"Evaluate if prediction '{pred}' matches gold '{gold}'.")
            iter_results.append(1 if judge == 'correct' else 0)
        total_score += sum(iter_results) / 5
    return (total_score / len(predictions)) * 100
```

## Common pitfalls

- Probability-based scoring (log-probs) is infeasible because multi-note EHR contexts exceed the context windows of most models, making few-shot prompting impractical.
- Free-text evaluation yields high variance in model rankings and scores compared to multi-choice, making it unreliable for automated benchmarking.
- GPT-4-turbo automated scoring can be inconsistent for ambiguous responses, requiring repeated evaluations and averaging to stabilize results.

## Evidence (verbatim from paper)

> For each model, correct outputs were assigned of 1 point, while incorrect responses were given 0 points. Across 962 questions, the scores of each model were normalized to a 100-point scale for comparison.

## Citation

```bibtex
@misc{kweon2024ehrnoteqa,
  title={EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries},
  author={Kweon et al. (2024)},
  year={2024},
  note={arXiv:2402.16040}
}
```

- arXiv: 2402.16040