Kits21 Segmentation Eval

---
name: kits21-segmentation-eval
description: This benchmark evaluates the ability of deep learning models to perform multi-organ and multi-lesion semantic segmentation on 3D medical imaging data. It specifically probes a model's capacity to accurately delineate kidneys, renal tumors, and renal cysts from corticomedullary-phase CT scans, testing both volumetric overlap and boundary precision. Use when the user wants to benchmark on KiTS21, or asks about evaluating this task. Reports dice.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.01984
  bibtex_key: heller2023kits21
  confidence: high
---

# kits21-segmentation-eval

> The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT — Heller et al. (2023) (arXiv:2307.01984, 2023)

## What this evaluates

This benchmark evaluates the ability of deep learning models to perform multi-organ and multi-lesion semantic segmentation on 3D medical imaging data. It specifically probes a model's capacity to accurately delineate kidneys, renal tumors, and renal cysts from corticomedullary-phase CT scans, testing both volumetric overlap and boundary precision.

## Datasets

- **KiTS21** — total ?; splits: test (100)

## Metrics

- `dice` **(primary)** — range: [0, 1]
  - Volumetric Dice similarity coefficient: 2 * |prediction ∩ ground_truth| / (|prediction| + |ground_truth|). The final ranking uses the mean across all test cases.
- `surface_dice` — range: [0, 1]
  - Surface Dice: Fraction of surface points in the prediction within a specified distance threshold of the ground truth surface. Also averaged across all test cases for ranking.

## Input / output format

**Input**: 3D corticomedullary-phase CT scan volumes (voxel data).

**Output**: 3D segmentation masks containing binary labels for kidneys, renal tumors, and renal cysts.

## Scoring recipe

```python
def score_kits21(predictions, ground_truths):
    dice_scores = [dice_score(p, g) for p, g in zip(predictions, ground_truths)]
    surface_dice_scores = [surface_dice_score(p, g) for p, g in zip(predictions, ground_truths)]
    avg_dice = np.mean(dice_scores)
    avg_surface_dice = np.mean(surface_dice_scores)
    return {'dice': avg_dice, 'surface_dice': avg_surface_dice}
# Final ranking uses a rank-then-aggregate procedure on the mean scores across all test cases.
```

## Common pitfalls

- The leaderboard ranking is determined by a rank-then-aggregate procedure using both volumetric and surface Dice scores, not a single metric.
- A higher leaderboard rank does not guarantee statistical superiority; the authors explicitly recommend bootstrap sampling and Holm-Bonferroni corrections to assess performance differences.
- Ground truth is derived from three independent expert annotations per region, so models must align with a multi-annotator consensus protocol rather than a single label.

## Evidence (verbatim from paper)

> The final leaderboard ranking was determined with a rank-then-aggregate procedure using the respective means across HECs of the two chosen varieties of dice scores.

## Citation

```bibtex
@misc{heller2023kits21,
  title={The KiTS21 Challenge: Automatic segmentation of kidneys, renal tumors, and renal cysts in corticomedullary-phase CT},
  author={Heller et al. (2023)},
  year={2023},
  note={arXiv:2307.01984}
}
```

- arXiv: 2307.01984