Multi Disease Cxr Eval

---
name: multi-disease-cxr-eval
description: Evaluates the cross-institutional generalizability of deep learning models for multi-disease chest X-ray classification. It probes whether training on diverse, weakly-labeled radiology datasets improves prediction performance for specific pathologies when tested on held-out medical sites. Use when the user wants to benchmark on NIH, CheXpert, Shifa International Hospital (SIH), or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.12065
  bibtex_key: bajwa2020generalized
  confidence: high
---

# multi-disease-cxr-eval

> A generalized deep learning model for multi-disease Chest X-Ray diagnostics — Bajwa et al. (2020) (arXiv:2010.12065, 2020)

## What this evaluates

Evaluates the cross-institutional generalizability of deep learning models for multi-disease chest X-ray classification. It probes whether training on diverse, weakly-labeled radiology datasets improves prediction performance for specific pathologies when tested on held-out medical sites.

## Datasets

- **NIH** — total ?; splits: test (-1)
- **CheXpert** — total ?; splits: val (-1), test (-1)
- **Shifa International Hospital (SIH)** — total ?; splits: test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic (ROC) curve. Computed independently for each of the four disease classes (Atelectasis, Cardiomegaly, Pleural Effusion, Pneumonia) and averaged across classes to report overall performance.

## Input / output format

**Input**: Chest X-ray radiographic images.

**Output**: Binary classification scores/probabilities for four disease classes: Atelectasis, Cardiomegaly, Pleural Effusion, and Pneumonia.

## Scoring recipe

```python
def compute_auc(preds, labels):
    fpr, tpr, _ = roc_curve(labels, preds)
    return auc(fpr, tpr)

aucs = []
for disease in ['Atelectasis', 'Cardiomegaly', 'Pleural Effusion', 'Pneumonia']:
    aucs.append(compute_auc(model_predictions[disease], ground_truth[disease]))
average_auc = mean(aucs)
std_auc = std(aucs)
# Statistical significance assessed via t-test and Wilcoxon signed-rank test
```

## Common pitfalls

- The CheXpert validation set contains only nine pneumonia-positive cases, making AUC scores on that specific split statistically unreliable.
- The reported 'average AUC' is a simple arithmetic mean of per-class AUCs, not a macro-averaged AUC computed over all samples jointly.
- Models are evaluated on cross-site generalization by training on one or two sites and testing on held-out sites, but label noise from weak supervision is not corrected during evaluation.

## Evidence (verbatim from paper)

> We first discuss the overall performance, calculated as the average AUC over all four diseases. This is followed by a detailed performance evaluation and discussion of the models on each disease class. Table 6 demonstrates the average AUC scores of the models on the three datasets.

## Citation

```bibtex
@misc{bajwa2020generalized,
  title={A generalized deep learning model for multi-disease Chest X-Ray diagnostics},
  author={Bajwa et al. (2020)},
  year={2020},
  note={arXiv:2010.12065}
}
```

- arXiv: 2010.12065