Chestxray Pneumonia Eval

---
name: chestxray-pneumonia-eval
description: Evaluates deep learning models for binary classification of chest X-rays into normal versus pneumonia categories, while also assessing the spatial interpretability of model predictions using Grad-CAM heatmaps. Use when the user wants to benchmark on Chest X-Rays dataset, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.00456
  bibtex_key: shahi2025weakly
  confidence: high
---

# chestxray-pneumonia-eval

> Weakly Supervised Pneumonia Localization from Chest X-Rays Using Deep Neural Network and Grad-CAM Explanations — Shahi et al. (2025) (arXiv:2511.00456, 2025)

## What this evaluates

Evaluates deep learning models for binary classification of chest X-rays into normal versus pneumonia categories, while also assessing the spatial interpretability of model predictions using Grad-CAM heatmaps.

## Datasets

- **Chest X-Rays dataset** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/kiranshahi/pneumonia-analysis

## Metrics

- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified instances: (TP + TN) / (TP + TN + FP + FN).
- `ROC-AUC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring trade-off between true positive rate and false positive rate across thresholds.
- `PR-AUC` — range: [0, 1]
  - Area under the Precision-Recall curve, measuring trade-off between precision and recall across thresholds.
- `Best F1` — range: [0, 1]
  - Maximum F1-score achieved across all classification thresholds, calculated as 2 * (Precision * Recall) / (Precision + Recall).
- `Lung Attention Ratio (LAR)` — range: [0, 1]
  - Ratio of Grad-CAM heatmap activation pixels falling within the anatomical lung mask to the total heatmap activation pixels.

## Input / output format

**Input**: Single chest X-ray image with an associated patient-level binary label (Normal or Pneumonia).

**Output**: Binary classification prediction (Normal/Pneumonia) and a Grad-CAM heatmap overlay indicating spatial attention regions.

## Scoring recipe

```python
y_pred = model.predict(X)
accuracy = (y_pred == y_true).mean()
f1 = f1_score(y_true, y_pred, average='binary')
roc_auc = roc_auc_score(y_true, y_prob)
pr_auc = average_precision_score(y_true, y_prob)
heatmap = compute_gradcam(model, X)
lung_mask = get_anatomical_lung_mask(X)
lar = (heatmap * lung_mask).sum() / heatmap.sum()
```

## Common pitfalls

- Using image-level splits instead of patient-level splits causes data leakage due to multiple views per patient.
- Grad-CAM heatmaps are post-hoc explanations, not supervised localization targets, so high classification accuracy does not guarantee anatomically correct attention.
- Early stopping is based on validation ROC-AUC, not accuracy, which may shift the optimal threshold for F1/Best F1 reporting.

## Evidence (verbatim from paper)

> Evaluation metrics include Accuracy, ROC-AUC, PR-AUC and Best F1. ... Quantitative Grad-CAM localization using Lung Attention Ratio (LAR) on a representative subset of the test set.

## Citation

```bibtex
@misc{shahi2025weakly,
  title={Weakly Supervised Pneumonia Localization from Chest X-Rays Using Deep Neural Network and Grad-CAM Explanations},
  author={Shahi et al. (2025)},
  year={2025},
  note={arXiv:2511.00456}
}
```

- arXiv: 2511.00456