Imis Eval

---
name: imis-eval
description: Evaluates the ability of vision models to perform interactive medical image segmentation using user prompts like clicks, bounding boxes, or text. It probes how well models generalize across different imaging modalities, anatomical structures, and interaction strategies (single vs. multi-turn). Use when the user wants to benchmark on IMed-361M, TotalSegmentator MRI dataset, ISLES dataset, or asks about evaluating this task. Reports Dice score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.12814
  bibtex_key: cheng2024imis
  confidence: high
---

# imis-eval

> Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline — Cheng et al. (2024) (arXiv:2411.12814, 2024)

## What this evaluates

Evaluates the ability of vision models to perform interactive medical image segmentation using user prompts like clicks, bounding boxes, or text. It probes how well models generalize across different imaging modalities, anatomical structures, and interaction strategies (single vs. multi-turn).

## Datasets

- **IMed-361M** — total 6400000; splits: test (-1)
- **TotalSegmentator MRI dataset** — total ?; splits: test (-1)
- **ISLES dataset** — total ?; splits: test (-1)

## Metrics

- `Dice score` **(primary)** — range: [0, 1]
  - Measures the overlap between the predicted segmentation mask and the ground truth mask. Calculated as 2 * |A ∩ B| / (|A| + |B|), where A and B are the predicted and ground truth masks respectively.

## Input / output format

**Input**: A medical image paired with a user prompt, which can be a single point (click), a bounding box, or a text label.

**Output**: A binary segmentation mask corresponding to the prompted target.

## Scoring recipe

```python
def dice_score(pred_mask, gt_mask):
    intersection = np.logical_and(pred_mask, gt_mask).sum()
    union = pred_mask.sum() + gt_mask.sum()
    if union == 0:
        return 1.0 if intersection == 0 else 0.0
    return 2.0 * intersection / union
```

## Common pitfalls

- Models are evaluated on both single-interaction and multi-interaction (up to 9 turns) settings, but results are often reported only for single-turn, missing the interactive refinement capability.
- Performance heavily depends on prompt placement; points near the centroid yield significantly higher Dice scores than random placements, which can skew benchmark results if not standardized.

## Evidence (verbatim from paper)

> Additionally, under the single-point prompt condition, SAM and SAM-2 achieve only Dice scores of 60.26% and 59.57%, respectively, likely due to the absence of medical knowledge in the pretraining data and limited interactive information constraining model performance.

## Citation

```bibtex
@misc{cheng2024imis,
  title={Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2411.12814}
}
```

- arXiv: 2411.12814