Mimic Iii Clinical Fact Checking Eval

---
name: mimic-iii-clinical-fact-checking-eval
description: Evaluates the factual consistency and logical coherence of LLM-generated clinical discharge summaries against ground-truth Electronic Health Records (EHRs) at a granular propositional level. It measures how accurately a model's extracted propositions align with clinician-validated EHR facts. Use when the user wants to benchmark on MIMIC-III, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.16189
  bibtex_key: zeba2025mitigating
  confidence: high
---

# mimic-iii-clinical-fact-checking-eval

> Mitigating Hallucinations in Healthcare LLMs with Granular Fact-Checking and Domain-Specific Adaptation — Musarrat Zeba et al. (2025) (arXiv:2512.16189, 2025)

## What this evaluates

Evaluates the factual consistency and logical coherence of LLM-generated clinical discharge summaries against ground-truth Electronic Health Records (EHRs) at a granular propositional level. It measures how accurately a model's extracted propositions align with clinician-validated EHR facts.

## Datasets

- **MIMIC-III** — total 26104; splits: train (20883), val (2610), test (2611)

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall for binary proposition classification (Supported vs. Not Supported). Calculated as 2 * (precision * recall) / (precision + recall).
- `precision` — range: [0, 1]
  - Ratio of correctly identified supported propositions to all propositions labeled as supported by the model.
- `recall` — range: [0, 1]
  - Ratio of correctly identified supported propositions to all actual supported propositions in the ground truth EHR.

## Input / output format

**Input**: Paired clinical discharge summaries and corresponding Electronic Health Records (EHRs), with propositions extracted from both for comparison.

**Output**: Binary classification label per proposition ('Supported' or 'Not Supported') and generated clinical summary text.

## Scoring recipe

```python
def compute_metrics(gold_labels, pred_labels):
    tp = sum(1 for g, p in zip(gold_labels, pred_labels) if g == 'Supported' and p == 'Supported')
    fp = sum(1 for g, p in zip(gold_labels, pred_labels) if g == 'Not Supported' and p == 'Supported')
    fn = sum(1 for g, p in zip(gold_labels, pred_labels) if g == 'Supported' and p == 'Not Supported')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return {'precision': precision, 'recall': recall, 'f1': f1}
```

## Common pitfalls

- Proposition-level annotation requires significant clinician time, limiting the fact-checking evaluation to a small stratified subset (104 records) rather than the full 26k dataset.
- Binary 'Supported/Not Supported' labels may oversimplify nuanced clinical inconsistencies or rare condition representations that clinicians noted during human evaluation.

## Evidence (verbatim from paper)

> We evaluated the correctness of each proposition of our fact-checking module using precision, recall, F1-score, and confusion matrix analysis. These metrics allow us to understand how well the generated summaries align with their respective EHR data.

## Citation

```bibtex
@misc{zeba2025mitigating,
  title={Mitigating Hallucinations in Healthcare LLMs with Granular Fact-Checking and Domain-Specific Adaptation},
  author={Musarrat Zeba et al. (2025)},
  year={2025},
  note={arXiv:2512.16189}
}
```

- arXiv: 2512.16189