Soap Note Generation Eval

---
name: soap-note-generation-eval
description: Evaluates the ability of audio and text models to generate clinically accurate, well-structured SOAP notes from long-form doctor-patient conversations. Probes long-context audio reasoning, fact-grounding, and clinical documentation quality. Use when the user wants to benchmark on Doctor-Patient SOAP Conversations, or asks about evaluating this task. Reports Faithfulness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.06138
  bibtex_key: labrak2026generatingsynthetic
  confidence: high
---

# soap-note-generation-eval

> Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarization — Labrak et al. (2026) (arXiv:2604.06138, 2026)

## What this evaluates

Evaluates the ability of audio and text models to generate clinically accurate, well-structured SOAP notes from long-form doctor-patient conversations. Probes long-context audio reasoning, fact-grounding, and clinical documentation quality.

## Datasets

- **Doctor-Patient SOAP Conversations** — total 8800; splits: dev (-1), test (-1)

## Metrics

- `Faithfulness` **(primary)** — range: other
  - LLM-as-a-judge scores claim support and contradictions on a 1–5 scale.
- `Coverage` — range: other
  - LLM-as-a-judge scores completeness of documentation relative to transcript evidence on a 1–5 scale.
- `ROUGE-2 F1` — range: percent
  - Standard ROUGE-2 F1 score computed between generated and reference SOAP notes.
- `ROUGE-3 F1` — range: percent
  - Standard ROUGE-3 F1 score computed between generated and reference SOAP notes.
- `ROUGE-L F1` — range: percent
  - Standard ROUGE-L F1 score computed between generated and reference SOAP notes.
- `Open Medical Concept F1` — range: percent
  - F1 score over overlap of medical concepts extracted via MeSH keyword matching and NER (en_core_sci_md) from reference and hypothesis notes.
- `Unsupported claim rate` — range: [0, 1]
  - Proportion of atomic claims in the generated note that lack grounding in the transcript.
- `Contradiction rate` — range: [0, 1]
  - Proportion of atomic claims in the generated note that contradict the transcript.

## Input / output format

**Input**: Multi-speaker doctor-patient audio conversation (for end-to-end models) or speaker-attributed transcript (for cascaded models).

**Output**: SOAP note text containing History of Present Illness (HPI), Review of Systems (ROS), Assessment, and Plan sections.

## Scoring recipe

```python
def evaluate(generated_note, reference_note, transcript):
    claims = extract_atomic_claims(generated_note)
    faithfulness = 0
    for claim in claims:
        evidence = transcript.search(claim)
        if evidence.supports: faithfulness += 1
        elif evidence.contradicts: faithfulness -= 1
    faithfulness = scale_1_to_5(faithfulness / len(claims))
    rouge2 = compute_rouge_n(generated_note, reference_note, n=2)
    rouge3 = compute_rouge_n(generated_note, reference_note, n=3)
    rouge_l = compute_rouge_l(generated_note, reference_note)
    med_f1 = compute_f1(
        extract_medical_concepts(generated_note, method="MeSH+scispaCy"),
        extract_medical_concepts(reference_note, method="MeSH+scispaCy")
    )
    unsupported_rate = count_unsupported_claims(claims, transcript) / len(claims)
    contradiction_rate = count_contradicting_claims(claims, transcript) / len(claims)
    return faithfulness, rouge2, rouge3, rouge_l, med_f1, unsupported_rate, contradiction_rate
```

## Common pitfalls

- Reference notes are LLM-generated rather than physician-authored, which may introduce systematic biases in both content and evaluation signals.
- Acoustic simulation uses low-WER synthetic audio (<3% for Whisper), so results may not generalize to real clinical recordings with high background noise or overlapping speech.
- End-to-end models are evaluated on raw audio while cascaded models use transcripts, making direct comparison sensitive to ASR error propagation and modality gaps.

## Evidence (verbatim from paper)

> We score SOAP notes along 12 dimensions. Four dimensions are scored on a 1–5 scale (5 = best): Faithfulness (claim support and contradictions), Structure (SOAP formatting and section placement), Coverage (completeness of documentation relative to transcript evidence), and Conciseness (redundancy and low-value content). We additionally evaluate two rates: unsupported claim rate and contradiction rate. In addition to the reference-free LLM judge, we evaluate generated notes against the reference using standard ROUGE F1 metrics (R-2, R-3, R-L) and an Open Medical Concept metric, which extracts medical concepts from both reference and hypothesis notes and computes F1 over their overlap.

## Citation

```bibtex
@misc{labrak2026generatingsynthetic,
  title={Generating Synthetic Doctor-Patient Conversations for Long-form Audio Summarization},
  author={Labrak et al. (2026)},
  year={2026},
  note={arXiv:2604.06138}
}
```

- arXiv: 2604.06138