Medfuse Eval

---
name: medfuse-eval
description: Evaluates a multi-modal fusion model's ability to predict patient phenotypes and in-hospital mortality using partially paired clinical time-series data and chest X-ray images. It probes robustness to missing modalities and temporal alignment in ICU settings. Use when the user wants to benchmark on MIMIC-IV / MIMIC-CXR, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.07027
  bibtex_key: hayat2022medfuse
  confidence: high
---

# medfuse-eval

> MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images — Hayat et al. (2022) (arXiv:2207.07027, 2022)

## What this evaluates

Evaluates a multi-modal fusion model's ability to predict patient phenotypes and in-hospital mortality using partially paired clinical time-series data and chest X-ray images. It probes robustness to missing modalities and temporal alignment in ICU settings.

## Datasets

- **MIMIC-IV / MIMIC-CXR** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/nyuad-cai/MedFuse

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve. Measures the trade-off between true positive rate and false positive rate across all classification thresholds.
- `AUPRC` — range: [0, 1]
  - Area Under the Precision-Recall curve. Measures the trade-off between precision and recall across all classification thresholds, particularly sensitive to class imbalance.

## Input / output format

**Input**: Clinical time-series data vector x_ehr in R^(t x 76) (regularly sampled every 2 hours, standardized continuous features, one-hot encoded categorical features) paired with a chest X-ray image x_cxr.

**Output**: Binary or multi-label vector y_ehr (25 binary labels for phenotype classification, or 1 binary label for in-hospital mortality).

## Scoring recipe

```python
import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score

def compute_metrics(y_true, y_pred, task):
    if task == 'phenotype':
        auroc = roc_auc_score(y_true, y_pred, average='macro', multi_label=True)
        auprc = average_precision_score(y_true, y_pred, average='macro')
    else:
        auroc = roc_auc_score(y_true, y_pred)
        auprc = average_precision_score(y_true, y_pred)
    return auroc, auprc

# Bootstrap 1000 iterations for 95% CI
scores = [compute_metrics(y_true, y_pred, task) for _ in range(1000)]
```

## Common pitfalls

- Splits are performed at the patient level to prevent data leakage across ICU stays, not at the sample level.
- The 25 phenotype labels require mapping ICD-10 codes to ICD-9 and then to CCS categories, differing from the original Harutyunyan et al. (2019) benchmark.
- Models must handle partially paired data where chest X-rays are missing, often requiring learnable substitute vectors or masking strategies.

## Evidence (verbatim from paper)

> We evaluate this task using the Area Under the Receiver Operating Characteristic (AUROC) curve and the Area Under the Precision Recall curve (AUPRC). Using the patient identifier of the clinical time-series data, we randomly split the dataset into 70% for training, 10% for validation, and 20% for test set, as shown in Figure 1. We report final results on the test sets and compute 95% confidence intervals with 1000 iterations via the bootstrap method (Efron and Tibshirani, 1994).

## Citation

```bibtex
@misc{hayat2022medfuse,
  title={MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images},
  author={Hayat et al. (2022)},
  year={2022},
  note={arXiv:2207.07027}
}
```

- arXiv: 2207.07027