Nyu Breast Cancer Eval

---
name: nyu-breast-cancer-eval
description: This benchmark evaluates a model's ability to classify breast cancer findings (benign vs. malignant vs. none) from high-resolution screening mammograms using only image-level labels. It also probes weakly supervised localization by measuring how well the model's generated saliency maps align with radiologist-annotated lesion segmentations. Use when the user wants to benchmark on NYU Breast Cancer Screening Dataset, or asks about evaluating this task. Reports AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2002.07613
  bibtex_key: shen2020interpretable
  confidence: high
---

# nyu-breast-cancer-eval

> An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization — Shen et al. (2020) (arXiv:2002.07613, 2020)

## What this evaluates

This benchmark evaluates a model's ability to classify breast cancer findings (benign vs. malignant vs. none) from high-resolution screening mammograms using only image-level labels. It also probes weakly supervised localization by measuring how well the model's generated saliency maps align with radiologist-annotated lesion segmentations.

## Datasets

- **NYU Breast Cancer Screening Dataset** — total 229426; splits: train (186816), val (28462), test (14148); repo https://github.com/nyukat/GMIC

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve computed on breast-level predictions. It measures the trade-off between true positive rate and false positive rate across all classification thresholds.
- `PRAUC` — range: [0, 1]
  - Area under the Precision-Recall Curve used in the reader study to compare model and radiologist performance, particularly useful for imbalanced datasets where the positive class (malignant findings) is rare.
- `DSC` — range: [0, 1]
  - Dice Similarity Coefficient calculated as the average overlap between predicted saliency maps and ground truth segmentation masks for biopsied, non-occult lesions. Formula: 2 * |prediction ∩ ground_truth| / (|prediction| + |ground_truth|).

## Input / output format

**Input**: Four high-resolution mammogram images per exam (R-CC, L-CC, R-MLO, L-MLO), cropped to 2944×1920 pixels and normalized. The model processes the two views per breast (CC and MLO) independently.

**Output**: Breast-level probability prediction for malignant/benign findings (averaged from two image-level predictions), plus a pixel-level saliency map highlighting suspected lesion regions.

## Scoring recipe

```python
# Breast-level AUC
img_preds = [model.predict(img) for img in [img_cc, img_mlo]]
breast_pred = np.mean(img_preds)
auc = roc_auc_score(gold_label, breast_pred)

# Localization DSC
pred_mask = generate_saliency(model, img)
gt_mask = gold_segmentation
intersection = np.logical_and(pred_mask, gt_mask).sum()
dsc = 2.0 * intersection / (pred_mask.sum() + gt_mask.sum())
```

## Common pitfalls

- Model predictions are computed per image but must be averaged across the two views (CC and MLO) to form the final breast-level prediction.
- Segmentation masks are strictly for evaluation only; the primary training protocol uses only image-level labels (weak supervision).
- Approximately 32.8% of biopsied exams contain mammographically occult lesions, meaning ground truth masks exist but are invisible on the input images, which can artificially lower localization metrics if not filtered.

## Evidence (verbatim from paper)

> For classification performance, we report area under the ROC curve (AUC) on the breast-level. In the reader study, we also use area under the precision-recall curve (PRAUC) to compare radiologists and the proposed model. We computed the radiologists' sensitivity which served as prediction threshold to derive the specificity of GMIC. To assess statistical significance, we performed Student's t-test and used binomial proportion confidence intervals for specificity. To quantitatively evaluate our model's localization ability, we calculate the Dice similarity coefficient (DSC). The DSC values we report are computed as an average over images for which segmentation labels are available (i.e. images from breasts which have biopsied findings which were not mammographically occult).

## Citation

```bibtex
@misc{shen2020interpretable,
  title={An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization},
  author={Shen et al. (2020)},
  year={2020},
  note={arXiv:2002.07613}
}
```

- arXiv: 2002.07613