Segbook Eval

---
name: segbook-eval
description: Evaluates the transfer learning and fine-tuning capabilities of volumetric medical image segmentation models across diverse imaging modalities, anatomical targets, and dataset sizes. It probes how well pre-trained models generalize to downstream segmentation tasks in clinical imaging scenarios and reveals non-linear performance scaling with dataset scale. Use when the user wants to benchmark on SegBook, or asks about evaluating this task. Reports Dice Score (DSC).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.14525
  bibtex_key: ye2024segbook
  confidence: high
---

# segbook-eval

> SegBook: A Simple Baseline and Cookbook for Volumetric Medical Image Segmentation — Jin Ye et al. (2024) (arXiv:2411.14525, 2024)

## What this evaluates

Evaluates the transfer learning and fine-tuning capabilities of volumetric medical image segmentation models across diverse imaging modalities, anatomical targets, and dataset sizes. It probes how well pre-trained models generalize to downstream segmentation tasks in clinical imaging scenarios and reveals non-linear performance scaling with dataset scale.

## Datasets

- **SegBook** — total 87; splits: (unstated)

## Metrics

- `Dice Score (DSC)` **(primary)** — range: percent
  - Computes the overlap between predicted and ground-truth segmentation masks: DSC = 2|X ∩ Y| / (|X| + |Y|). Reported as a percentage (0–100) averaged across datasets or stratified by modality, target type, and dataset size.

## Input / output format

**Input**: 3D volumetric medical images (CT, MRI, PET, or Ultrasound) with corresponding segmentation masks for fine-tuning; evaluation uses the test/validation splits of 87 downstream datasets.

**Output**: 3D volumetric segmentation masks predicting the target anatomical structures or lesions.

## Scoring recipe

```python
def compute_dice(pred_mask, gt_mask):
    intersection = np.sum(pred_mask & gt_mask)
    union = np.sum(pred_mask) + np.sum(gt_mask)
    if union == 0:
        return 100.0 if intersection == 0 else 0.0
    return 200.0 * intersection / union

# Average across all 87 datasets or specific subsets (e.g., by modality or size)
final_score = np.mean([compute_dice(pred, gt) for pred, gt in zip(predictions, golds)])
```

## Common pitfalls

- The paper reports DSC on a 0–100 scale, not 0–1, which can cause confusion when comparing to other benchmarks that use fractional values.
- Performance exhibits a non-linear bottleneck effect: fine-tuning yields ~3% gains on small and large datasets but only ~1% on medium-sized datasets, so averaging across scales without stratification masks this behavior.
- Evaluations are heavily stratified by modality (CT, MRI, US, CT&PET) and target type (seen/unseen structures, lesions, bone, vessel); results must be reported per stratum to avoid misleading cross-task averages.

## Evidence (verbatim from paper)

> To investigate the overall transfer learning performance of STU-Net, we conducted experiments on 87 downstream datasets. As shown in Table 2, we reported average DSC for various models and detailed DSC for three data scales: small (S), medium (M), and large (L).

## Citation

```bibtex
@misc{ye2024segbook,
  title={SegBook: A Simple Baseline and Cookbook for Volumetric Medical Image Segmentation},
  author={Jin Ye et al. (2024)},
  year={2024},
  note={arXiv:2411.14525}
}
```

- arXiv: 2411.14525