Mimic Iv Icd Eval

---
name: mimic-iv-icd-eval
description: Predicts ICD-9 and ICD-10 medical billing codes from clinical discharge notes under extreme multi-label classification settings. It probes a model's ability to handle long-tailed label distributions, high cardinality, and code hierarchy variations in electronic health records. Use when the user wants to benchmark on MIMIC-IV-ICD9, MIMIC-IV-ICD10, MIMIC-IV-ICD9-50, MIMIC-IV-ICD10-50, or asks about evaluating this task. Reports Macro-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.13998
  bibtex_key: nguyen2023mimicivicd
  confidence: high
---

# mimic-iv-icd-eval

> Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification — Thanh-Tung Nguyen et al. (2023) (arXiv:2304.13998, 2023)

## What this evaluates

Predicts ICD-9 and ICD-10 medical billing codes from clinical discharge notes under extreme multi-label classification settings. It probes a model's ability to handle long-tailed label distributions, high cardinality, and code hierarchy variations in electronic health records.

## Datasets

- **MIMIC-IV-ICD9** — total 209352; splits: train (188533), val (7110), test (13709); repo https://github.com/thomasnguyen92/MIMIC-IV-ICD-data-processing
- **MIMIC-IV-ICD10** — total 122310; splits: train (110442), val (4017), test (7851); repo https://github.com/thomasnguyen92/MIMIC-IV-ICD-data-processing
- **MIMIC-IV-ICD9-50** — total 189475; splits: train (170664), val (6406), test (12405); repo https://github.com/thomasnguyen92/MIMIC-IV-ICD-data-processing
- **MIMIC-IV-ICD10-50** — total 115250; splits: train (104077), val (3805), test (7368); repo https://github.com/thomasnguyen92/MIMIC-IV-ICD-data-processing

## Metrics

- `Macro-F1` **(primary)** — range: [0, 1]
  - Standard macro-averaged F1 score computed over all unique ICD codes. Calculated as the unweighted mean of F1 scores for each label, where F1 = 2 * (Precision * Recall) / (Precision + Recall).
- `Precision@K` — range: [0, 1]
  - Fraction of predicted codes in the top-K ranked list that are correct. K is typically set to 5 or 10 for extreme multi-label settings.

## Input / output format

**Input**: Clinical discharge note text (string). Optional patient metadata (age, gender, hospital stay duration) may be appended depending on the model variant.

**Output**: A ranked list or set of predicted ICD-9 or ICD-10 codes (strings) corresponding to the input note.

## Scoring recipe

```python
def compute_macro_f1(preds, golds):
    all_labels = set(golds) | set(preds)
    f1_scores = []
    for label in all_labels:
        tp = sum(1 for p in preds if label in p and label in golds)
        fp = sum(1 for p in preds if label in p and label not in golds)
        fn = sum(1 for p in preds if label not in p and label in golds)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores) if f1_scores else 0
```

## Common pitfalls

- Splitting data by hospital admission (hadm_id) instead of patient (subject_id) causes patient-level data leakage across train/val/test sets.
- Ignoring the long-tailed distribution of ICD codes leads models to only predict frequent codes and fail on rare diagnoses.
- Inconsistent mapping of parent/child code hierarchies between ICD-9 and ICD-10 versions can cause evaluation mismatches if not standardized.

## Evidence (verbatim from paper)

> Evaluating existing methods from MIMIC-III in the MIMIC-IV context is advantageous for determining their performance in larger and more complex multilabel classification scenarios.

## Citation

```bibtex
@misc{nguyen2023mimicivicd,
  title={Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification},
  author={Thanh-Tung Nguyen et al. (2023)},
  year={2023},
  note={arXiv:2304.13998}
}
```

- arXiv: 2304.13998