Mimic Iii Clinical Notes Eval

---
name: mimic-iii-clinical-notes-eval
description: Evaluates multi-modal deep learning models for predicting patient outcomes (decompensation, in-hospital mortality, phenotyping) using electronic health records (EHR) and clinical notes. It probes the model's ability to fuse tabular/time-series physiological data with unstructured clinical text to improve clinical decision support. Use when the user wants to benchmark on MIMIC-III, or asks about evaluating this task. Reports AUPRC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.03044
  bibtex_key: husmann2022clinicalnotes
  confidence: high
---

# mimic-iii-clinical-notes-eval

> On the Importance of Clinical Notes in Multi-modal Learning for EHR Data — Severin Husmann et al. (arXiv:2212.03044, 2022)

## What this evaluates

Evaluates multi-modal deep learning models for predicting patient outcomes (decompensation, in-hospital mortality, phenotyping) using electronic health records (EHR) and clinical notes. It probes the model's ability to fuse tabular/time-series physiological data with unstructured clinical text to improve clinical decision support.

## Datasets

- **MIMIC-III** — total ?; splits: test (-1)

## Metrics

- `AUPRC` **(primary)** — range: [0, 1]
  - Area under the precision-recall curve. Computed as the weighted mean of precisions achieved at each threshold.
- `AUROC` — range: [0, 1]
  - Area under the receiver operating characteristic curve. Measures the trade-off between true positive rate and false positive rate across thresholds.
- `Macro-AUC` — range: [0, 1]
  - Macro-averaged area under the ROC curve for multi-class phenotyping, computed by averaging AUC per class.
- `Micro-AUC` — range: [0, 1]
  - Micro-averaged area under the ROC curve for multi-class phenotyping, computed by aggregating contributions of all classes.

## Input / output format

**Input**: Time-series EHR measurements (timesteps) concatenated with clinical notes (text) for each patient.

**Output**: Continuous probability scores or binary predictions for decompensation, in-hospital mortality (IHM), and phenotyping tasks.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, task):
    auprc = average_precision_score(y_true, y_pred)
    auroc = roc_auc_score(y_true, y_pred)
    if task == 'phenotyping':
        macro_auc = roc_auc_score(y_true, y_pred, average='macro')
        micro_auc = roc_auc_score(y_true, y_pred, average='micro')
        return {'AUPRC': auprc, 'AUROC': auroc, 'Macro-AUC': macro_auc, 'Micro-AUC': micro_auc}
    return {'AUPRC': auprc, 'AUROC': auroc}
```

## Common pitfalls

- Assuming high-frequency note types (e.g., nurses' notes) drive performance gains without verifying attention weights.
- Reporting only point estimates without averaging over multiple random seeds or providing 95% confidence intervals.
- Assuming clinician notes are redundant; the evaluation shows they add little value compared to non-clinician notes like radiology reports.

## Evidence (verbatim from paper)

> All results reported are from the test set and obtained over 5 random seeds. Table 1: Performances on MIMIC-III Benchmark [15] for uni-modal (top, middle) and cross-modal models (bottom). Reported errors, if provided, are 95% confidence intervals on the mean. Metric: AUPRC, AUROC, Macro-AUC, Micro-AUC.

## Citation

```bibtex
@misc{husmann2022clinicalnotes,
  title={On the Importance of Clinical Notes in Multi-modal Learning for EHR Data},
  author={Severin Husmann et al.},
  year={2022},
  note={arXiv:2212.03044}
}
```

- arXiv: 2212.03044