Cold Start Al 3d Medical Seg Eval

---
name: cold-start-al-3d-medical-seg-eval
description: Evaluates cold-start active learning sample selection strategies for 3D medical image segmentation by comparing how well diversity-based, uncertainty-based, and random methods perform when annotation budgets are extremely limited. Use when the user wants to benchmark on Medical Segmentation Decathlon (MSD), or asks about evaluating this task. Reports Dice score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.12004
  bibtex_key: liu2023colossal
  confidence: high
---

# cold-start-al-3d-medical-seg-eval

> COLosSAL: A Benchmark for Cold-start Active Learning for 3D Medical Image Segmentation — Liu et al. (2023) (arXiv:2307.12004, 2023)

## What this evaluates

Evaluates cold-start active learning sample selection strategies for 3D medical image segmentation by comparing how well diversity-based, uncertainty-based, and random methods perform when annotation budgets are extremely limited.

## Datasets

- **Medical Segmentation Decathlon (MSD)** — total ?; splits: test (-1); repo https://github.com/MedICL-VU/COLosSAL

## Metrics

- `Dice score` **(primary)** — range: [0, 1]
  - 2 * |A ∩ B| / (|A| + |B|), measuring the voxel-wise overlap between the predicted segmentation mask and the ground truth mask.
- `HD95` — range: mm
  - 95th percentile of the Hausdorff distance between the predicted and ground truth boundary surfaces, measuring the worst-case boundary localization error.

## Input / output format

**Input**: 3D medical image volumes (and optionally local ROIs) used as input for cold-start active learning sample selection strategies.

**Output**: Segmentation masks for each selected volume, evaluated against ground truth to compute Dice and HD95 scores.

## Scoring recipe

```python
def compute_metrics(pred_mask, gt_mask):
    intersection = np.sum(pred_mask & gt_mask)
    dice = 2.0 * intersection / (np.sum(pred_mask) + np.sum(gt_mask))
    # HD95 computed via distance transforms
    dist_map = hausdorff_distance_transform(pred_mask, gt_mask)
    hd95 = np.percentile(dist_map, 95)
    return dice, hd95
```

## Common pitfalls

- Assuming active learning strategies consistently outperform random selection; the benchmark shows random selection remains a strong baseline in cold-start scenarios.
- Expecting local ROI-based selection to outperform global volume-based selection; results show local strategies fail to consistently surpass global ones.
- Overestimating performance on tumor-containing tasks; uncertainty-based methods struggle significantly here due to poor uncertainty estimation on pathological regions.

## Evidence (verbatim from paper)

> In Fig. 2, with a fixed budget of 5 samples (except for Heart, where 3 samples are used), we compare the uncertainty-based and diversity-based strategies against the random selection on five different segmentation tasks. Note that the selections made by each of our evaluated AL strategies are deterministic. For random selection, we visualize the individual Dice scores (red dots) of all 15 runs as well as their mean (dashed line). HD95 results (Supp. Tab. 1) follow the same trends.

## Citation

```bibtex
@misc{liu2023colossal,
  title={COLosSAL: A Benchmark for Cold-start Active Learning for 3D Medical Image Segmentation},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2307.12004}
}
```

- arXiv: 2307.12004