Clinical Note Eval

---
name: clinical-note-eval
description: This evaluation probes the clinical reasoning, safety, and instruction-following capabilities of large language models on real-world medical datasets. It measures how well models generate accurate and appropriate responses to clinical prompts compared to baseline systems and GPT-3.5-turbo. Use when the user wants to benchmark on MIMIC-III, MIMIC-IV, i2b2, MTSamples, CASI (AE), CASI (CR), DisCQ, or asks about evaluating this task. Reports scores.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.00237
  bibtex_key: kweon2023asclepius
  confidence: medium
---

# clinical-note-eval

> Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes — Kweon et al. (2023) (arXiv:2309.00237, 2023)

## What this evaluates

This evaluation probes the clinical reasoning, safety, and instruction-following capabilities of large language models on real-world medical datasets. It measures how well models generate accurate and appropriate responses to clinical prompts compared to baseline systems and GPT-3.5-turbo.

## Datasets

- **MIMIC-III** — total 413; splits: test (413)
- **MIMIC-IV** — total 500; splits: test (500)
- **i2b2** — total 619; splits: test (619)
- **MTSamples** — total 101; splits: test (101)
- **CASI (AE)** — total 100; splits: test (100)
- **CASI (CR)** — total 100; splits: test (100)
- **DisCQ** — total 100; splits: test (100)

## Metrics

- `scores` **(primary)** — range: [1, 5]
  - Average rating assigned by evaluators (GPT-4 and/or clinicians) on a 1–5 scale for each generated response. Higher scores indicate better clinical accuracy, safety, and helpfulness.

## Input / output format

**Input**: Clinical instruction or case description paired with a reference answer (gold), used to prompt the model for a response.

**Output**: A natural language clinical response or answer generated by the model.

## Scoring recipe

```python
scores = []
for pred, gold in zip(predictions, golds):
    rating = evaluator.evaluate(pred, gold)  # 1-5 scale
    scores.append(rating)
return sum(scores) / len(scores)
```

## Common pitfalls

- Small test set sizes for several datasets (100–101 samples), which may lead to high variance in average scores.
- Reliance on LLM-as-a-judge (GPT-4) and limited human clinicians introduces potential bias and inconsistency in scoring criteria.
- Scores are reported per model size (7B, 13B, 175B) but not aggregated across sizes, making cross-architecture comparison difficult.

## Evidence (verbatim from paper)

> Table 2: Individual Experimental Results. Bolded entries denote the best scores, while underscored ones represent the second-best results for each model size.

## Citation

```bibtex
@misc{kweon2023asclepius,
  title={Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes},
  author={Kweon et al. (2023)},
  year={2023},
  note={arXiv:2309.00237}
}
```

- arXiv: 2309.00237