Sam Zero Shot Medical Eval

---
name: sam-zero-shot-medical-eval
description: Evaluates the zero-shot segmentation capability of the Segment Anything Model (SAM) across diverse medical imaging modalities and anatomical structures. It probes the model's ability to generalize without task-specific fine-tuning to structured medical targets like organs, lesions, and retinal layers. Use when the user wants to benchmark on Skin Lesion Analysis Toward Melanoma Detection (ISIC), DoFE (Drishiti-GS, RIM-ONE-r3, REFUGE amalgamated), AMOS, MICCAI 2017 Robotic Instrument Segmentation, Chest X-ray, Rat Colon, AROI, or asks about evaluating this task. Reports Dice similarity coefficient.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.12637
  bibtex_key: [REDACTED]
  confidence: high
---

# sam-zero-shot-medical-eval

> Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation — Peilun Shi et al. (2023) (arXiv:2304.12637, 2023)

## What this evaluates

Evaluates the zero-shot segmentation capability of the Segment Anything Model (SAM) across diverse medical imaging modalities and anatomical structures. It probes the model's ability to generalize without task-specific fine-tuning to structured medical targets like organs, lesions, and retinal layers.

## Datasets

- **Skin Lesion Analysis Toward Melanoma Detection (ISIC)** — total 1000; splits: test (1000)
- **DoFE (Drishiti-GS, RIM-ONE-r3, REFUGE amalgamated)** — total 271; splits: test (271)
- **AMOS** — total 15361; splits: val (15361)
- **MICCAI 2017 Robotic Instrument Segmentation** — total 1200; splits: test (1200)
- **Chest X-ray** — total 704; splits: test (704)
- **Rat Colon** — total 130; splits: test (130)
- **AROI** — total 113; splits: test (113)

## Metrics

- `Dice similarity coefficient` **(primary)** — range: [0, 1]
  - Measures overlap between ground truth Y and predicted mask Ŷ: Dice = 2|Y ∩ Ŷ| / (|Y| + |Ŷ|). Ranges from 0 (no overlap) to 1 (perfect overlap). Averaged across samples.
- `Intersection over Union (IoU)` — range: [0, 1]
  - Computes ratio of intersection to union of two sets: IoU = |Y ∩ Ŷ| / (|Y| + |Ŷ| - |Y ∩ Ŷ|). Ranges from 0 to 1. Averaged across samples.

## Input / output format

**Input**: Single medical image (dermoscopy, fundus, CT, MRI, endoscopy, X-ray, or OCT) with a point prompt corresponding to the centroid of the ground truth mask.

**Output**: Binary segmentation mask (predicted mask Ŷ) of the target anatomical structure or lesion.

## Scoring recipe

```python
for each image in dataset:
  Y = ground_truth_mask
  Y_hat = sam_predict(image, prompt=centroid(Y))
  dice = 2 * np.sum(Y * Y_hat) / (np.sum(Y) + np.sum(Y_hat))
  iou = np.sum(Y * Y_hat) / (np.sum(Y) + np.sum(Y_hat) - np.sum(Y * Y_hat))
  store dice, iou
avg_dice = np.mean(all_dices)
avg_iou = np.mean(all_ious)
```

## Common pitfalls

- SAM is prompted with the ground-truth centroid, which assumes oracle knowledge and does not reflect realistic user interaction or automated prompt generation.
- Some datasets (e.g., AMOS) lack released test labels, forcing the use of validation sets for evaluation, which may inflate or bias results.
- Small sample sizes in certain subsets (e.g., 259 vs 1000 for ISIC) lead to high variance in reported scores.
- SAM's zero-shot performance heavily depends on image contrast rather than semantic understanding, particularly in fundus images, making cross-domain comparisons sensitive to lighting/contrast variations.

## Evidence (verbatim from paper)

> To assess the zero-shot segmentation capability of SAM on medical images, two quantitative metrics were employed: Dice similarity coefficient and Intersection over Union (IoU). The Dice coefficient measures the overlap between two sets of data and ranges from 0 (no overlap) to 1 (perfect overlap). Similarly, IoU computes the ratio of the intersection over the union of two sets and ranges from 0 to 1. Both metrics were then averaged across multiple samples to obtain an overall measure of the segmentation accuracy.

## Citation

```bibtex
@misc{shi2023generalist,
  title={Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation},
  author={Peilun Shi et al. (2023)},
  year={2023},
  note={arXiv:2304.12637}
}
```

- arXiv: 2304.12637