Discharge Note Generation Eval

---
name: discharge-note-generation-eval
description: Evaluates the ability of fine-tuned LLMs to generate clinically accurate, complete, and readable discharge summaries for cardiac patients from raw medical records. It probes domain-specific medical summarization, factual consistency, and adherence to clinical documentation standards. Use when the user wants to benchmark on Cardiology Clinical Dataset, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.05144
  bibtex_key: [REDACTED]
  confidence: high
---

# discharge-note-generation-eval

> Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients — Jung et al. (2024) (arXiv:2404.05144, 2024)

## What this evaluates

Evaluates the ability of fine-tuned LLMs to generate clinically accurate, complete, and readable discharge summaries for cardiac patients from raw medical records. It probes domain-specific medical summarization, factual consistency, and adherence to clinical documentation standards.

## Datasets

- **Cardiology Clinical Dataset** — total ?; splits: test (-1)

## Metrics

- `ROUGE-1 / ROUGE-2 / ROUGE-L` — range: [0, 1]
  - Overlap of unigrams, bigrams, and longest common subsequence between generated and reference text.
- `BLEU` — range: [0, 1]
  - Geometric mean of modified n-gram precisions with brevity penalty.
- `BERTScore` — range: [0, 1]
  - Cosine similarity between contextual embeddings of generated and reference tokens, aggregated via F1.
- `Perplexity` — range: [0, inf)
  - Exponential of the average negative log-likelihood of the reference text under the model.
- `Accuracy` **(primary)** — range: [1, 5]
  - Expert-rated score (1-5) for factual correctness of the generated discharge note, averaged across 5 test samples.

## Input / output format

**Input**: Raw patient medical records / clinical notes from a cardiology center.

**Output**: Structured discharge note containing sections: [Chief Complaint], [Operation and Procedure], [Hospital Course], [Condition at Discharge], [Type of Discharge].

## Scoring recipe

```python
# Quantitative metrics (per sample)
rouge = compute_rouge(reference, generated)
bleu = compute_bleu(reference, generated)
bertscore = compute_bertscore(reference, generated)
ppl = compute_perplexity(reference, model)

# Qualitative expert assessment (5 test samples)
scores = []
for sample in test_samples[:5]:
    scores.append(expert_rating(sample.generated_note, criteria=[
        "Accuracy", "Completeness", "Readability & Comprehensibility",
        "Consistency", "Utility"
    ]))
total_qualitative = sum(scores)  # Max 25
```

## Common pitfalls

- Expert qualitative assessment was only performed on 5 test samples, limiting statistical power and generalizability.
- Metrics like ROUGE and BERTScore are optimized for English and may not accurately reflect quality for Korean clinical notes without language-specific tokenization.
- Perplexity measures language model likelihood, not clinical factual accuracy or safety.

## Evidence (verbatim from paper)

> Five sample notes from the test set were evaluated by a cardiology expert across five criteria: Accuracy, Completeness, Readability and Comprehensibility, Consistency, and Utility, with each aspect rated on a 5-point scale.

## Citation

```bibtex
@misc{jung2024discharge,
  title={Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients},
  author={Jung et al. (2024)},
  year={2024},
  note={arXiv:2404.05144}
}
```

- arXiv: 2404.05144