Breast Cancer Prototype Eval

---
name: breast-cancer-prototype-eval
description: This evaluation probes the classification accuracy and prototype-based interpretability of deep learning models on mammography datasets. It measures how well models predict malignancy while ensuring that their learned prototypes align with domain-specific radiological features (e.g., mass/calcification types and BIRADS descriptors). Use when the user wants to benchmark on CBIS-DDSM, CMMD, VinDr-Mammo, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.20260
  bibtex_key: pathak2024prototype
  confidence: high
---

# breast-cancer-prototype-eval

> Prototype-based Interpretable Breast Cancer Prediction Models: Analysis and Challenges — Pathak et al. (2024) (arXiv:2403.20260, 2024)

## What this evaluates

This evaluation probes the classification accuracy and prototype-based interpretability of deep learning models on mammography datasets. It measures how well models predict malignancy while ensuring that their learned prototypes align with domain-specific radiological features (e.g., mass/calcification types and BIRADS descriptors).

## Datasets

- **CBIS-DDSM** — total 3103; splits: train (2458), test (645)
- **CMMD** — total 5202; splits: train (3199), test (2002)
- **VinDr-Mammo** — total 20000; splits: train (16000), test (4000)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Reported as mean and standard deviation over three random seeds.
- `AUC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the model's ability to discriminate between malignant and benign classes across all classification thresholds.
- `IoU` — range: [0, 1]
  - Intersection over Union between the activated prototype region and the ground truth Region of Interest (ROI). Calculated for top-1, top-10, and all activated prototypes.
- `DSC` — range: [0, 1]
  - Dice Similarity Coefficient: 2 * |A ∩ B| / (|A| + |B|), measuring spatial overlap between prototype activations and annotated ROIs.

## Input / output format

**Input**: Mammography images (mediolateral oblique and craniocaudal views) resized to 1536x768 pixels.

**Output**: Binary classification prediction (malignant/benign) and activated prototype image patches/regions for interpretability.

## Scoring recipe

```python
def compute_metrics(preds, gold, prototype_masks, gt_masks):
    f1 = f1_score(gold, preds)
    auc = roc_auc_score(gold, preds)
    iou_scores = [iou(p_mask, g_mask) for p_mask, g_mask in zip(prototype_masks, gt_masks)]
    dsc_scores = [2 * np.sum(p * g) / (np.sum(p) + np.sum(g)) for p, g in zip(prototype_masks, gt_masks)]
    return {'F1': f1, 'AUC': auc, 'IoU': np.mean(iou_scores), 'DSC': np.mean(dsc_scores)}
```

## Common pitfalls

- Using image-level random splits instead of patient-wise splits, which causes data leakage since multiple views from the same patient appear in both train and test sets.
- Comparing prototype visualization patch sizes across different architectures without normalization, as ProtoPNet uses percentile-based upsampling while PIP-Net uses fixed 130x130 patches.
- Defining prototype purity categories flatly instead of following the hierarchical BIRADS lexicon (abnormality type -> shape/margin/morphology/distribution), leading to inaccurate coverage and specialization scores.

## Evidence (verbatim from paper)

> We calculate the intersection over union (IoU) and Dice Similarity Coefficient (DSC) for the top-1 (IoU1, DSC1), top-10 (IoU10, DSC10) and all (IoUAll, DSCAll) activated prototypes with the annotated ROI.

## Citation

```bibtex
@misc{pathak2024prototype,
  title={Prototype-based Interpretable Breast Cancer Prediction Models: Analysis and Challenges},
  author={Pathak et al. (2024)},
  year={2024},
  note={arXiv:2403.20260}
}
```

- arXiv: 2403.20260