Nuinsseg Eval

---
name: nuinsseg-eval
description: Evaluates the ability of models to perform instance-level segmentation of cell nuclei in H&E-stained histological images. It specifically probes robustness to tissue variability, high cell density, and ambiguous boundaries where manual annotation is difficult. Use when the user wants to benchmark on NuInsSeg, or asks about evaluating this task. Reports Dice score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.01760
  bibtex_key: mahbod2023nuinsseg
  confidence: high
---

# nuinsseg-eval

> NuInsSeg: A Fully Annotated Dataset for Nuclei Instance Segmentation in H&E-Stained Histological Images — Amirreza Mahbod et al. (2023) (arXiv:2308.01760, 2023)

## What this evaluates

Evaluates the ability of models to perform instance-level segmentation of cell nuclei in H&E-stained histological images. It specifically probes robustness to tissue variability, high cell density, and ambiguous boundaries where manual annotation is difficult.

## Datasets

- **NuInsSeg** — total 665; splits: 5-fold cross-validation (665); repo https://github.com/masih4/NuInsSeg

## Metrics

- `Dice score` **(primary)** — range: percent
  - Computes the overlap between predicted and ground truth masks: 2|P∩G|/(|P|+|G|). Averages the per-instance scores across the fold.
- `Aggregate Jaccard Index (AJI)` — range: percent
  - Measures instance-level overlap by dividing the intersection of all predicted and ground truth instances by their union, penalizing over- and under-segmentation.
- `Panoptic Quality (PQ)` — range: percent
  - Combines detection quality (F1 score of matched instances) and segmentation quality (mean Dice score of matched instances): PQ = F1 × Dice.

## Input / output format

**Input**: 512×512 pixel RGB PNG images cropped from 2048×2048 field-of-view histological slides.

**Output**: Binary and labeled instance segmentation masks (PNG format) indicating the precise boundary of each nucleus.

## Scoring recipe

```python
def evaluate_nuinsseg(pred_masks, gt_masks):
    dice_scores, aji_scores, pq_scores = [], [], []
    for pred, gt in zip(pred_masks, gt_masks):
        matches = match_instances(pred, gt, iou_thresh=0.5)
        dice = mean([dice_score(p, g) for p, g in matches])
        aji = aggregate_jaccard_index(pred, gt)
        pq = compute_f1(matches) * dice
        dice_scores.append(dice)
        aji_scores.append(aji)
        pq_scores.append(pq)
    return {
        'Dice': mean(dice_scores) * 100,
        'AJI': mean(aji_scores) * 100,
        'PQ': mean(pq_scores) * 100
    }
```

## Common pitfalls

- The dataset uses 5-fold cross-validation rather than a fixed train/val/test split; results must be averaged across all folds to match the paper's baseline.
- Ambiguous area masks are provided but are not used in the primary Dice/AJI/PQ scoring; models are evaluated only on deterministic nuclei masks.
- Metrics are reported as averages across folds, which can mask performance variation across specific tissue types or organs.

## Evidence (verbatim from paper)

> For evaluation, we utilized similarity Dice score, aggregate Jaccard index (AJI), and panoptic quality (PQ) scores as suggested in former studies.

## Citation

```bibtex
@misc{mahbod2023nuinsseg,
  title={NuInsSeg: A Fully Annotated Dataset for Nuclei Instance Segmentation in H&E-Stained Histological Images},
  author={Amirreza Mahbod et al. (2023)},
  year={2023},
  note={arXiv:2308.01760}
}
```

- arXiv: 2308.01760