Mimic Iii Multitask Eval

---
name: mimic-iii-multitask-eval
description: Evaluates clinical time series models on four interrelated ICU prediction tasks: in-hospital mortality, physiologic decompensation, length of stay, and phenotype classification. It probes a model's ability to handle heterogeneous multitask learning with varying temporal structures and output types. Use when the user wants to benchmark on MIMIC-III, or asks about evaluating this task. Reports Test score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1703.07771
  bibtex_key: harutyunyan2017multitask
  confidence: high
---

# mimic-iii-multitask-eval

> Multitask learning and benchmarking with clinical time series data — Harutyunyan et al. (2017) (arXiv:1703.07771, 2017)

## What this evaluates

Evaluates clinical time series models on four interrelated ICU prediction tasks: in-hospital mortality, physiologic decompensation, length of stay, and phenotype classification. It probes a model's ability to handle heterogeneous multitask learning with varying temporal structures and output types.

## Datasets

- **MIMIC-III** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Test score` **(primary)** — range: other
  - Not explicitly named in the provided section; standard for this benchmark is AUC-ROC for classification tasks and MAE for length-of-stay regression. Scores are reported with 95% bootstrap confidence intervals.

## Input / output format

**Input**: Longitudinal clinical time series data (vital signs, lab measurements) discretized into 1-hour intervals.

**Output**: Predictions for four tasks: binary mortality, binary decompensation, continuous length of stay, and multi-class phenotype classification.

## Scoring recipe

```python
# 1. Split predefined dataset: 85% train, 15% val
# 2. Tune hyperparameters on val set; select best model
# 3. Evaluate best model on test set to get base score
# 4. Bootstrap resample test set K times (K=10000 for mortality/phenotype, K=1000 for decomp/LOS)
# 5. Compute score on each resample
# 6. 95% CI = [percentile(scores, 2.5), percentile(scores, 97.5)]
```

## Common pitfalls

- Test set leakage: The test set must be used sparingly during development to avoid unintentional overfitting.
- Overfitting on public benchmarks: The authors explicitly use bootstrapping to estimate confidence intervals and mitigate known test-set overfitting issues.
- Hyperparameter tuning must be strictly confined to the validation set; the test set is only for final reporting.

## Evidence (verbatim from paper)

> For all algorithms we use the data of the same 15% patients of the predefined training set as validation data and train the models on the remaining 85%. ... The final scores are reported on the test set, which we used sparingly during model development in order to avoid unintentional test set leakage. ... To estimate a 95% confidence interval we resample the test set K times; calculate the score on the resampled sets; and use 2.5 and 97.5 percentiles of these scores as our confidence interval estimate. For in-hospital mortality and phenotype prediction K is 10000, while for decompensation and length-of-stay prediction K is 1000...

## Citation

```bibtex
@misc{harutyunyan2017multitask,
  title={Multitask learning and benchmarking with clinical time series data},
  author={Harutyunyan et al. (2017)},
  year={2017},
  note={arXiv:1703.07771}
}
```

- arXiv: 1703.07771