Mammo Fm Diagnostic Eval

---
name: mammo-fm-diagnostic-eval
description: Evaluates a breast-specific foundational model's ability to generalize across in-distribution and out-of-distribution mammographic datasets for zero-shot diagnosis, linear probing, full fine-tuning, and pathology localization. It probes the model's robustness, data efficiency, and representation quality for clinical tasks like cancer detection and risk prediction. Use when the user wants to benchmark on EMBED, VinDr, RSNA, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.00198
  bibtex_key: ghosh2025mammofm
  confidence: high
---

# mammo-fm-diagnostic-eval

> Mammo-FM: Breast-specific foundational model for Integrated Mammographic Diagnosis, Prognosis, and Reporting — Ghosh et al. (2025) (arXiv:2512.00198, 2025)

## What this evaluates

Evaluates a breast-specific foundational model's ability to generalize across in-distribution and out-of-distribution mammographic datasets for zero-shot diagnosis, linear probing, full fine-tuning, and pathology localization. It probes the model's robustness, data efficiency, and representation quality for clinical tasks like cancer detection and risk prediction.

## Datasets

- **EMBED** — total 255039; splits: train (-1), val (-1), test (-1)
- **VinDr** — total ?; splits: test (-1)
- **RSNA** — total ?; splits: test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve. Measures the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance across all classification thresholds.
- `mAP` — range: [0, 1]
  - Mean average precision across detection classes. Computed by averaging the area under the precision-recall curve for each class, typically at a fixed IoU threshold for object detection.

## Input / output format

**Input**: Mammographic images (CC and MLO views) at native resolution. For zero-shot classification, inputs are paired with descriptive text prompts specifying finding subtype, laterality, depth, and positional descriptors.

**Output**: Classification labels or probabilities (mass, suspicious calcification, architectural distortion, cancer), detection bounding boxes with confidence scores, or continuous risk scores.

## Scoring recipe

```python
def compute_auroc(y_true, y_scores):
    # y_true: binary ground truth labels
    # y_scores: predicted probabilities or logits
    return roc_auc_score(y_true, y_scores)

def compute_map(boxes_pred, scores_pred, boxes_true, labels_true, iou_thresh=0.5):
    # Standard object detection mAP calculation
    # Matches predictions to ground truth based on IoU and confidence thresholds
    # Returns mean average precision across all classes
    pass
```

## Common pitfalls

- Patient-level data splitting is strictly enforced to prevent information leakage between train/val/test sets; images from the same patient must not appear in multiple splits.
- Zero-shot evaluation relies on clinically grounded text prompts (including subtype, laterality, depth) rather than simple class names, which drastically changes image-text similarity scores.
- OOD generalization is tested on datasets with different imaging protocols and label definitions (e.g., RSNA only provides binary cancer labels, not 1-5 year outcomes), requiring proxy metrics like 1-year risk scores.

## Evidence (verbatim from paper)

> We report classification and detection performance using AUROC and mAP, respectively. ... In zero-shot classification setup, Mammo-FM achieves 13 - 38% higher AUROC than MedSigLIP across tasks ... For mass detection, the fine-tuned multi-institution Mammo-FM achieves a mAP of 0.58 (95% CI: 0.55–0.62)

## Citation

```bibtex
@misc{ghosh2025mammofm,
  title={Mammo-FM: Breast-specific foundational model for Integrated Mammographic Diagnosis, Prognosis, and Reporting},
  author={Ghosh et al. (2025)},
  year={2025},
  note={arXiv:2512.00198}
}
```

- arXiv: 2512.00198