Isic Ham Segmentation Eval

---
name: isic-ham-segmentation-eval
description: Evaluates dermatologic image segmentation models by measuring how training on real versus synthetic data affects performance on held-out real test sets, and how model accuracy correlates with controllable synthetic image parameters like skin tone and lesion shape. Use when the user wants to benchmark on ISIC, HAM, or asks about evaluating this task. Reports Dice score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.00191
  bibtex_key: kim2024ssynth
  confidence: high
---

# isic-ham-segmentation-eval

> S-SYNTH: Knowledge-Based, Synthetic Generation of Skin Images — Kim et al. (2024) (arXiv:2408.00191, 2024)

## What this evaluates

Evaluates dermatologic image segmentation models by measuring how training on real versus synthetic data affects performance on held-out real test sets, and how model accuracy correlates with controllable synthetic image parameters like skin tone and lesion shape.

## Datasets

- **ISIC** — total ?; splits: train (-1), test (-1)
- **HAM** — total ?; splits: train (-1), test (-1)

## Metrics

- `Dice score` **(primary)** — range: [0, 1]
  - Measures overlap between predicted and ground-truth segmentation masks. Calculated as 2 * |A ∩ B| / (|A| + |B|), where A and B are the predicted and ground-truth masks respectively.

## Input / output format

**Input**: Dermoscopic or clinical skin images (real or synthetic).

**Output**: Binary segmentation mask indicating the lesion region.

## Scoring recipe

```python
def dice_score(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = pred_mask.sum() + gt_mask.sum()
    return 2.0 * intersection / union if union > 0 else 0.0
```

## Common pitfalls

- Domain shift between real and synthetic images causes performance drops when training exclusively on synthetic data.
- Absolute performance on synthetic test images is lower than on real images, though relative trends remain comparable.

## Evidence (verbatim from paper)

> We systematically evaluated the effect of the training data composition on the real test set performance (similar to the popular Train-Synthetic-Test-Real (TSTR) protocol) on both ISIC and HAM. ... More importantly, each model that was trained on a particular subset of the ISIC (or HAM) and supplemented with synthetic images resulted in a higher Dice score than the model that was trained on the same subset of only the real images.

## Citation

```bibtex
@misc{kim2024ssynth,
  title={S-SYNTH: Knowledge-Based, Synthetic Generation of Skin Images},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2408.00191}
}
```

- arXiv: 2408.00191