Auto Annotation

---
name: auto-annotation
description: "Implement CVAT local auto-annotation functions and reason about
  serverless native-function deployment, label matching, ROI, and tracking
  flows."
metadata:
  disco-role: operating
disable-model-invocation: true
license: MIT
---

# CVAT auto-annotation

Use this sub-skill when the user needs to automate annotation with a Python function or a CVAT native function: build detection/interaction/tracking/re-identification logic, run `annotate_task`, create native functions, run agents, or understand the CPU/GPU serverless deployment story for built-in Nuclio models.

## Route first

- Read `references/function-protocols.md` for the confirmed AA function protocols, helper factories, prompt/result constraints, and CLI/API mappings.
- Read `references/serverless-models.md` for CPU/GPU serverless deployment patterns, model categories, and deployment hazards.
- Read `references/troubleshooting.md` for label mapping, shape validation, backend, import, and deployment errors.
- Use `scripts/aa_detection_template.py` for a safe starter implementation and `scripts/cvat_serverless_deploy_notes.py` for a non-executing command checklist.

## Choose the right mode

- **Immediate mode**: call `cvat_sdk.auto_annotation.annotate_task(client, task_id, function, ...)` from Python or use `cvat-cli task auto-annotate`.
- **Native function mode**: create a server-side function with `cvat-cli function create-native` and run an agent process with `cvat-cli function run-agent`.
- **Serverless deployment mode**: deploy built-in CVAT Nuclio functions when the user wants reusable server-side detectors/interactors.

If the task is primarily general SDK scripting, route to `../sdk-automation/SKILL.md`. If it is primarily task/project shell automation, route to `../cli-automation/SKILL.md`.

## Protocols at a glance

### Detection functions

Implement `spec` and `detect`.

- `spec` returns a `DetectionFunctionSpec` with label definitions.
- `detect(context, image)` returns a sequence of tags/shapes.
- Helper factories such as `label_spec`, `skeleton_label_spec`, `keypoint_spec`, `attribute_spec`, `rectangle`, `mask`, `polygon`, `skeleton`, `keypoint`, and `tag` make valid objects easier to construct.
- `frame_id` must be 0 and `id`/`source` must not be set in returned annotations.

### Interaction functions

Implement `spec`, `detect`, and optionally `preprocess_image`.

- `spec` is an `InteractionFunctionSpec` with minimum positive points and optional negative points/bounding boxes.
- `detect(context, pp_image, prompts)` returns `InteractionResultShape` objects.
- CVAT UI currently only processes returned shapes of type `mask`.
- `InteractionResultAttributes.CONFIDENCE` is the only supported result attribute spec id.

### Tracking functions

Implement `spec`, `init_tracking_state`, `track`, and optionally `preprocess_image`.

- `spec.supported_shape_types` declares the trackable shapes.
- `init_tracking_state` analyzes the first image and shape.
- `track` predicts a later position or returns `None` when tracking fails.

## CLI and API entry points

- `task auto-annotate` uses a local function module or source file and can clear existing annotations, allow unmatched labels, set a confidence threshold, and convert masks to polygons.
- `function create-native` registers a local function implementation with the server.
- `function run-agent` polls the server for requests and executes them against the local function.

## Safety defaults

- Keep model loading local and deterministic; avoid hidden downloads in the function path unless the user explicitly wants them.
- Put heavyweight model weights, checkpoints, or Nuclio build contexts outside the runtime skill and describe them as deployment inputs.
- Require the operator to match labels and shapes before running automatic annotation on real data.
- For GPU serverless functions, treat driver/runtime compatibility as a hard prerequisite and do not imply CPU validation covers GPU execution.