Conformal Lesion Segmentation Eval

---
name: conformal-lesion-segmentation-eval
description: This evaluation protocol assesses the ability of 3D medical image segmentation models to control false negative rates under user-specified risk constraints while maintaining spatial precision. It benchmarks a model-agnostic conformal prediction calibration method against fixed heuristic thresholds across multiple anatomical datasets. Use when the user wants to benchmark on KiTS21, LiTS, NIH-LN ABD, LIDC-IDRI, MDSC-Colon, MDSC-Pancreas, or asks about evaluating this task. Reports ECR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.17897
  bibtex_key: tan2025conformal
  confidence: high
---

# conformal-lesion-segmentation-eval

> Conformal Lesion Segmentation for 3D Medical Images — Tan et al. (2025) (arXiv:2510.17897, 2025)

## What this evaluates

This evaluation protocol assesses the ability of 3D medical image segmentation models to control false negative rates under user-specified risk constraints while maintaining spatial precision. It benchmarks a model-agnostic conformal prediction calibration method against fixed heuristic thresholds across multiple anatomical datasets.

## Datasets

- **KiTS21** — total ?; splits: calibration (-1), test (-1)
- **LiTS** — total ?; splits: calibration (-1), test (-1)
- **NIH-LN ABD** — total ?; splits: calibration (-1), test (-1)
- **LIDC-IDRI** — total ?; splits: calibration (-1), test (-1)
- **MDSC-Colon** — total ?; splits: calibration (-1), test (-1)
- **MDSC-Pancreas** — total ?; splits: calibration (-1), test (-1)

## Metrics

- `ECR` **(primary)** — range: [0, 1]
  - Empirical Compliance Rate: the proportion of test samples whose FNR-specific loss is controlled below a user-specified tolerance ε. It is computed as the fraction of samples satisfying the FNR constraint across the test set.
- `PC` — range: [0, 1]
  - Prediction Compactness: the ratio of the number of predicted lesion voxels to the total number of voxels in the input image. Used to measure spatial precision under risk constraints.
- `FNR` — range: [0, 1]
  - False Negative Rate: the proportion of ground-truth lesion voxels incorrectly predicted as background. The protocol enforces FNR ≤ ε for compliance.

## Input / output format

**Input**: 3D medical image volumes (e.g., CT or MRI scans) representing specific anatomical regions or organs.

**Output**: 3D binary segmentation masks indicating predicted lesion voxels for each input volume.

## Scoring recipe

```python
def compute_ecr(predictions, gold, epsilon):
    compliant = 0
    for pred, gt in zip(predictions, gold):
        fnr = np.sum((gt == 1) & (pred == 0)) / max(np.sum(gt == 1), 1)
        if fnr <= epsilon:
            compliant += 1
    return compliant / len(predictions)

def compute_pc(predictions):
    pred_voxels = np.sum(predictions > 0)
    total_voxels = predictions.size
    return pred_voxels / total_voxels
```

## Common pitfalls

- Assuming ECR strictly equals 1−α on finite samples; the paper notes minor violations occur due to finite-sample variability.
- Using ground-truth lesion volume as the denominator for PC; the paper explicitly uses total input volume to avoid constant scaling factors per sample.
- Comparing against a fixed 0.5 threshold without accounting for distributional shifts; CLS adapts thresholds per sample based on calibration data.

## Evidence (verbatim from paper)

> We check whether the empirical compliance rate (ECR)*(Angelopoulos and Bates [2021])*, defined as the proportion of test samples whose FNR-specific loss is controlled below ε, exceeds 1−α. Beyond FNR control, we also emphasize spatial precision by encouraging compact lesion predictions, as smaller predicted regions are generally more accurate and clinically preferable. We introduce prediction compactness (PC), defined as the ratio of the number of predicted lesion voxels to the total number of voxels in the input image.

## Citation

```bibtex
@misc{tan2025conformal,
  title={Conformal Lesion Segmentation for 3D Medical Images},
  author={Tan et al. (2025)},
  year={2025},
  note={arXiv:2510.17897}
}
```

- arXiv: 2510.17897