Soap Note Hallucination Eval

---
name: soap-note-hallucination-eval
description: Evaluates the hallucination rate of LLM-generated medical SOAP notes against physician-patient transcripts. It compares a literal, inference-unaware evaluation framework against a clinically informed, inference-aware framework to measure how often valid clinical reasoning is incorrectly flagged as hallucination. Use when the user wants to benchmark on Physician-Patient Transcripts, or asks about evaluating this task. Reports Mean Hallucination Rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.14829
  bibtex_key: vachhani2026beyond
  confidence: high
---

# soap-note-hallucination-eval

> Beyond Literal Summarization: Redefining Hallucination for Medical SOAP Note Evaluation — Vachhani et al. (2026) (arXiv:2604.14829, 2026)

## What this evaluates

Evaluates the hallucination rate of LLM-generated medical SOAP notes against physician-patient transcripts. It compares a literal, inference-unaware evaluation framework against a clinically informed, inference-aware framework to measure how often valid clinical reasoning is incorrectly flagged as hallucination.

## Datasets

- **Physician-Patient Transcripts** — total 100; splits: test (100)

## Metrics

- `Mean Hallucination Rate` **(primary)** — range: percent
  - The percentage of generated claims or samples flagged as hallucinations by the evaluation judge. Calculated as (number of flagged hallucinations / total evaluated claims or samples) × 100.

## Input / output format

**Input**: Physician-patient transcripts paired with the corresponding LLM-generated SOAP notes.

**Output**: Binary hallucination flag per claim, aggregated into a mean hallucination rate percentage.

## Scoring recipe

```python
flagged = 0
total = 0
for transcript, soap_note in dataset:
    claims = extract_claims(soap_note)
    for claim in claims:
        total += 1
        if judge_flag_hallucination(claim, transcript, stage=2):
            flagged += 1
return (flagged / total) * 100
```

## Common pitfalls

- Literal lexical matching incorrectly flags valid clinical inferences (e.g., diagnosing GERD from heartburn symptoms) as hallucinations.
- Failing to recognize medical terminology translations (e.g., 'dyspnea on exertion' for 'gets breathless when climbing stairs') leads to false positive hallucination flags.
- Over-penalizing guideline-based care plans that require clinical reasoning rather than direct textual extraction.

## Evidence (verbatim from paper)

> We evaluate the two judge configurations Stage 1 (inference unaware) and Stage 2 (inference aware) across 100 physician patient transcripts, measuring hallucination rates and comparing these against human annotator judgments. Table 3 summarizes the aggregate hallucination rates across the three evaluation conditions, and Figure 2 visualizes the per sample distributions. The results demonstrate a substantial reduction in flagged hallucinations when the judge is equipped with clinically informed criteria. Under Stage 1, the mean hallucination rate reached 35.2%, nearly three times the human annotator baseline of 10.4%. Stage 2 reduced this to 9.1%, achieving near parity with human judgment and confirming that the excess flagging under Stage 1 reflects evaluation design artifacts rather than genuine model errors.

## Citation

```bibtex
@misc{vachhani2026beyond,
  title={Beyond Literal Summarization: Redefining Hallucination for Medical SOAP Note Evaluation},
  author={Vachhani et al. (2026)},
  year={2026},
  note={arXiv:2604.14829}
}
```

- arXiv: 2604.14829