Misaw Seg Eval

---
name: misaw-seg-eval
description: Evaluates pixel-wise and instance-level segmentation performance for microsurgical instruments, with a specific focus on accurately delineating extremely thin and sparse structures (e.g., wires, needles) under low-contrast, high-magnification conditions. The benchmark probes a model's ability to maintain fine boundaries and avoid background bias when object classes are severely imbalanced. Use when the user wants to benchmark on MISAW-Seg, or asks about evaluating this task. Reports mcIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.11727
  bibtex_key: jeong2025microsurgical
  confidence: high
---

# misaw-seg-eval

> Microsurgical Instrument Segmentation for Robot-Assisted Surgery — Jeong et al. (2025) (arXiv:2509.11727, 2025)

## What this evaluates

Evaluates pixel-wise and instance-level segmentation performance for microsurgical instruments, with a specific focus on accurately delineating extremely thin and sparse structures (e.g., wires, needles) under low-contrast, high-magnification conditions. The benchmark probes a model's ability to maintain fine boundaries and avoid background bias when object classes are severely imbalanced.

## Datasets

- **MISAW-Seg** — total 2999; splits: train (2433), test (566); HF `KIST-HARILAB/MISAW-Seg`

## Metrics

- `mcIoU` **(primary)** — range: [0, 1]
  - Arithmetic mean of per-class Intersection over Union (IoU) scores across all 6 classes. Averaging across classes prevents frequent categories from dominating the score.
- `ISI-IoU` — range: [0, 1]
  - Instrument-sensitive IoU that reduces background bias and better reflects errors along tool boundaries.
- `mDice` — range: [0, 1]
  - Dice coefficient averaged over all classes, directly measuring region overlap and providing stability when evaluating very small structures.
- `mAP@50` — range: [0, 1]
  - Mean Average Precision computed at an IoU threshold of 0.50.
- `mAP@95` — range: [0, 1]
  - Mean Average Precision computed at an IoU threshold of 0.95.

## Input / output format

**Input**: 460×540 pixel images augmented to 5 channels (RGB + two luminance channels).

**Output**: Pixel-wise segmentation masks assigning one of six classes (LAV, RAV, LNH, RNH, ND, WR) to each pixel.

## Scoring recipe

```python
def compute_mcIoU(pred_masks, gt_masks, num_classes=6):
    class_ious = []
    for c in range(num_classes):
        pred_c = (pred_masks == c)
        gt_c = (gt_masks == c)
        intersection = np.logical_and(pred_c, gt_c).sum()
        union = np.logical_or(pred_c, gt_c).sum()
        iou = intersection / union if union > 0 else 0.0
        class_ious.append(iou)
    return np.mean(class_ious)
```

## Common pitfalls

- Thin objects (wires, needles) occupy only 0.4–3% of pixels, causing severe class imbalance that can easily dominate standard metrics if not averaged per-class.
- mAP@95 heavily favors large instruments; thin structures rarely achieve 0.95 IoU, so a model may score lower on mAP@95 while still outperforming on thin-object-sensitive metrics like mcIoU or mDice.

## Evidence (verbatim from paper)

> The mean class IoU(mcIoU) is the arithmetic mean of per-class IoUs by averaging across classes. It prevents frequent categories from dominating the score and is therefore appropriate under the severe class imbalance and pixel sparsity of thin objects. The ISI-IoU *[[35]]* follows prior surgical segmentation work and computes an instrument-sensitive IoU that reduces background bias and better reflects errors along tool boundaries. We also report mean Dice(mDice), the Dice coefficient averaged over classes, which directly measures region overlap and is more stable when evaluating very small structures*[[36]]*.

## Citation

```bibtex
@misc{jeong2025microsurgical,
  title={Microsurgical Instrument Segmentation for Robot-Assisted Surgery},
  author={Jeong et al. (2025)},
  year={2025},
  note={arXiv:2509.11727}
}
```

- arXiv: 2509.11727