Segment Anything Arxiv 2304 02643v1

# Segment Anything

## One-line decision
Use this skill when you need to build a billion-scale segmentation mask dataset using a model-in-the-loop interactive annotation approach. Avoid it when you need text-grounded segmentation rather than prompt-based mask generation.

## Skill metadata
- **Skill type**: interactive-annotation-at-scale
- **Paper kind**: operational-method
- **Actionability**: high
- **Evidence quality**: full_paper

## Goal
Build SA-1B, a dataset of 1.1 billion segmentation masks on 11 million images, using a model-in-the-loop data annotation engine where the SAM model assists human annotators and eventually generates masks automatically.

## Problem signature
- Modality: images with segmentation masks generated through model-assisted annotation.
- Data state: progressively annotated: manual → model-assisted → fully automatic mask generation.
- Scale regime: 1.1 billion masks on 11 million images.
- Model requirement: SAM (ViT-H) promptable segmentation model.

## Use when
- You need a large-scale segmentation dataset for pretraining.
- You want to use model-in-the-loop annotation to scale data collection.
- You need masks for diverse objects beyond standard detection categories.

## Do not use when
- You need text-grounded or semantic segmentation with class labels.
- You only need a few thousand annotated images.
- You need 3D or temporal segmentation.

## Required inputs
- **licensed_images**: 11M diverse, high-resolution images.
- **annotation_tool**: Interactive annotation tool with SAM model assistance.
- **sam_model**: Progressively trained SAM model for mask generation.

## Optional inputs
- **quality_review**: Human review of automatically generated masks.

## Outputs
- **sa_1b_dataset**: 1.1 billion segmentation masks on 11M images.
- **sam_model**: Trained promptable segmentation model.

## Assumptions and prerequisites
- Model-in-the-loop annotation can scale to billions of masks.
- Progressive automation (manual → assisted → automatic) maintains quality while increasing speed.
- Diverse prompts (points, boxes, masks) enable comprehensive segmentation.

## Procedure
1. **Phase 1: Manual annotation with SAM**
   Action: Human annotators create masks with SAM's interactive segmentation.
   Why: Bootstraps initial training data and validates the tool.
   Note: See paper for details.
2. **Phase 2: Model-assisted annotation**
   Action: SAM proposes masks that annotators refine.
   Why: Increases annotation speed 6x while maintaining quality.
   Note: See paper for details.
3. **Phase 3: Fully automatic annotation**
   Action: SAM generates masks automatically with a grid of point prompts.
   Why: Scales to billions of masks without human involvement.
   Note: See paper for details.
4. **Train SAM iteratively**
   Action: Retrain SAM on accumulated annotations between phases.
   Why: Progressive training improves mask quality.
   Note: See paper for details.
5. **Release SA-1B dataset**
   Action: Package and release the 1.1B mask dataset.
   Why: Enables downstream research and applications.
   Note: See paper for details.

## Parameters to set
- **prompt_grid_density** — Role: Density of automatic point prompts. How to set: 32x32 grid per image for comprehensive coverage. Default/range: 32x32. Effect: Denser grids find more objects but increase compute.
- **mask_quality_threshold** — Role: Minimum predicted IoU for keeping a mask. How to set: Filter low-confidence masks. Default/range: Model-dependent. Effect: Higher threshold keeps fewer but better masks.
- **nms_threshold** — Role: Non-maximum suppression overlap threshold. How to set: Standard NMS settings. Default/range: 0.7. Effect: Controls overlap between final masks.

## Validation checks
- Mask quality should be comparable to manually annotated datasets.
- Coverage should include diverse object types beyond standard categories.
- The dataset should enable strong zero-shot segmentation.

## Failure modes
- Automatic masks may miss small or occluded objects.
- Over-segmentation may produce too many fragmented masks.
- Class-agnostic masks lack semantic labels.

## Adaptation notes for VLM training
- SA-1B masks are widely used for VLM training data augmentation.
- Combine SAM masks with VLM-generated captions for grounded data.
- Use SAM as an annotation tool for constructing region-level VLM data.

## Implementation notes
- Use the SAM model in ONNX format for efficient inference.
- Process images in batches for the automatic phase.
- Store masks in the efficient RLE format.

## Evidence from the paper
- SA-1B contains 1.1 billion masks on 11M images, 400x larger than any existing segmentation dataset.
- Model-in-the-loop annotation achieves 6x speedup in the assisted phase.
- SAM achieves strong zero-shot segmentation across diverse image types.
- The dataset and model have become foundational tools for computer vision.

## Source paper
- **Title**: Segment Anything
- **Year**: 2023
- **Venue**: ICCV
- **Paper ID**: arxiv-2304.02643v1
- **URL**: http://arxiv.org/abs/2304.02643v1
- **arXiv ID**: 2304.02643v1