Rtvi Cv Scaffold Vss Service

---
name: rtvi-cv-scaffold-vss-service
description: >
  Scaffold a standalone RTVI CV microservice that plugs into VSS Search and
  Alerts profiles via Kafka mdx-raw. The shipped scaffold script is a YOLO26
  reference implementation (ONNX, labels, custom parser required). Use when
  building a new perception microservice repo, validating the VSS integration
  contract, extending that scaffold for segmentation frame-mask payloads, or
  scaffolding with placeholders before customer YOLO26 assets exist. For
  swapping the detector in the stock vss-rt-cv container, use
  rtvi-cv-customize-model instead. Live DeepStream integration cannot run
  until the customer-supplied ONNX, labels file, and parser library exist.
metadata:
  author: "NVIDIA CORPORATION <info@nvidia.com>"
  tags:
    - rtvi-cv
    - vss
    - deepstream
    - yolo26
    - kafka
    - perception
    - custom-microservice
    - segmentation
    - frame-mask
  languages:
    - python
    - bash
  domain: accelerated-microservices
  team: team-mind-hub
  owner: "merylm"
  service: "rtvi-cv"
  version: "2.0.0"
  reviewed: "2026-05-07"
  compatibility: >
    Scaffolding requires Python 3.10+. The generated microservice runs on a
    Linux host with NVIDIA GPU, Docker with the NVIDIA container runtime, and
    DeepStream 9.1 (image nvcr.io/nvidia/deepstream:9.1-triton-multiarch).
    Plugging into VSS requires a working VSS deployment -- Search Profile,
    Alerts Profile, or both -- with Kafka reachable through its standard host
    listener.
  data_classification: "internal"
license: "NVIDIA Proprietary"
---

# Construct RTVI VSS CV Service

Scaffolds a deployable custom perception microservice that:

1. runs a DeepStream pipeline with **YOLO26** primary inference + tracker,
2. converts detection metadata to the VSS protobuf schema and publishes it
   to the **`mdx-raw`** Kafka topic, and
3. drops into the existing VSS compose stack via the
   `bp_developer_search_2d` and `bp_developer_alerts_2d_cv` profile flags so
   the downstream Search Workflow, Alert Verification, and Behavior
   Analytics services consume it without further changes.

The scaffolded output is a runnable repo, not a design document.

## Scope boundaries

This skill name describes the **service type** (a VSS-bound RTVI CV
microservice), not a model family. What is generic vs YOLO26-specific:

| Layer | Scope |
|-------|-------|
| VSS integration | Generic — compose profiles, host networking, `mdx-raw`, protobuf-2 payload, tests, smoketest |
| Scaffold script output | **YOLO26 reference** — `pgie-yolo26-config.txt`, `YOLO26_*` mount paths, `NvDsInferParseYolo26` |
| Other ONNX detectors | Adapt the scaffold manually, or use `rtvi-cv-customize-model` to swap the model inside the stock `vss-rt-cv` perception container |
| Segmentation frame masks | Documented in `integration-contract.md`; not auto-generated by the scaffold — extend msgconv/wrapper after scaffolding |

Do not treat the scaffold as a model-agnostic generator. Agents should either
run the YOLO26 scaffold as-is or consciously edit pgie/compose paths for
another detector while keeping the VSS contract fixed.

## When to use

Use this skill when the user wants to:

- replace VSS's default perception service (RT-DETR / GDINO / YOLOv11) with
  a custom YOLO26-based detector,
- add new object classes or domain-specific tracking logic while keeping
  the VSS Search / Alerts / Behavior Analytics workflows intact,
- ship a perception microservice on a customer's hardware that integrates
  with a VSS deployment they manage.

Do **not** use this skill to:

- deploy or operate VSS itself (use VSS deployment runbooks),
- swap the detector in the default `vss-rt-cv` container without a new
  microservice (use `rtvi-cv-customize-model`).

This skill assumes a target VSS deployment already exists or will be brought
up separately.

## Instructions

- Read [references/integration-contract.md](references/integration-contract.md) first and keep `mdx-raw`, `msg-conv-payload-type=2`, broker reachability, and the protobuf-2 contract fixed unless the user explicitly wants to break VSS compatibility.
- Treat all VSS deployment paths as relative to a separate checkout of the public [VSS Blueprint repository](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization), not the DeepStream repository. Clone or reuse a VSS checkout that is **v3.2.1 or compatible**, then deploy that release (see [VSS Quickstart](https://docs.nvidia.com/vss/latest/quickstart.html#download-the-deployment-package)).
- If the user does **not** have the YOLO26 ONNX, labels file, or custom parser `.so`, say so plainly: scaffolding and unit tests can proceed with placeholders, but the live DeepStream app cannot run yet. Point them at Ultralytics/Hugging Face for pretrained weights and ONNX export (see [references/yolo26-deepstream.md](references/yolo26-deepstream.md)).
- When assets are missing, stop the "live validation" path at scaffold generation plus host-only unit tests. Do not imply that `docker compose up`, `deepstream-app`, or `kafka_smoketest.py` can succeed without the customer-supplied ONNX, labels matching `num-detected-classes`, and a parser exposing `NvDsInferParseYolo26`.
- Start the generated service only after the VSS Kafka topic initializer has completed. Keep it as a separate Compose application using host networking. Set `KAFKA_BOOTSTRAP` to Kafka's host-reachable advertised listener (default: `localhost:9092`). Compose cannot resolve `depends_on` across separate invocations.

## Examples

- "Create a custom RTVI CV microservice that runs YOLO26 in DeepStream and publishes object metadata so VSS Search and Alerts can consume it."
- "Validate the generated RTVI VSS service scaffold locally, then explain how to verify the live integration on a GPU host with a deployed VSS stack."
- "I do not have a trained YOLO26 ONNX, labels file, or parser `.so` yet. Can the live VSS integration still run?"

## VSS source location

This skill ships the scaffold generator, but not the VSS deployment. Clone the
public VSS Blueprint repository separately or reuse an existing checkout, then
set `VSS_ROOT` to that checkout explicitly. Do not search for or clone VSS
relative to the generated service directory:

```bash
# Customer-specific path to an existing VSS v3.2.1-compatible checkout.
export VSS_ROOT=/absolute/path/to/video-search-and-summarization
VSS_DEPLOY_DIR="${VSS_ROOT}/deploy/docker"
test -f "${VSS_DEPLOY_DIR}/compose.yml" || {
  echo "Missing VSS compose file: ${VSS_DEPLOY_DIR}/compose.yml" >&2
  exit 1
}
echo "Using VSS deployment: ${VSS_DEPLOY_DIR}"
```

Relevant customer-accessible VSS locations include:

- [`deploy/docker/scripts/dev-profile.sh`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/blob/v3.2.1/deploy/docker/scripts/dev-profile.sh) — brings up Search or Alerts using the documented Quickstart flow
- [`deploy/docker/developer-profiles/dev-profile-search/`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/tree/v3.2.1/deploy/docker/developer-profiles/dev-profile-search)
- [`deploy/docker/developer-profiles/dev-profile-alerts/`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/tree/v3.2.1/deploy/docker/developer-profiles/dev-profile-alerts)
- [`deploy/docker/services/rtvi/rtvi-cv/`](https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization/tree/v3.2.1/deploy/docker/services/rtvi/rtvi-cv) — stock RTVI-CV service definition

The custom service generated by this skill remains in `<target-dir>` and runs as a separate, host-networked Compose application alongside VSS. Set `KAFKA_BOOTSTRAP` to Kafka's host-reachable advertised listener (default: `localhost:9092`). Do not look for generated YOLO26 files inside either the DeepStream or stock VSS checkout.

## Required reads

Read these before generating code so the implementation matches the
documented VSS data path rather than a generic Kafka producer:

1. [references/integration-contract.md](references/integration-contract.md) — the exact mdx-raw protobuf contract. For segmentation/frame-mask services, read its segmentation frame-mask payload section before implementing msgconv, wrapper, Kafka, or protobuf changes.
2. [references/yolo26-deepstream.md](references/yolo26-deepstream.md) — DeepStream nvinfer config for YOLO26.
3. [references/vss-profile-integration.md](references/vss-profile-integration.md) — how the service plugs into Search / Alerts / Behavior Analytics.
4. VSS Object Detection and Tracking — `https://docs.nvidia.com/vss/latest/object-detection-tracking.html`
5. VSS Behavior Analytics — `https://docs.nvidia.com/vss/latest/behavior-analytics.html`
6. VSS Search Workflow — `https://docs.nvidia.com/vss/latest/agent-workflow-search.html`
7. DeepStream `Gst-nvmsgconv` — `https://docs.nvidia.com/metropolis/deepstream/9.1/text/DS_plugin_gst-nvmsgconv.html`
8. DeepStream `Gst-nvmsgbroker` — `https://docs.nvidia.com/metropolis/deepstream/9.1/text/DS_plugin_gst-nvmsgbroker.html`

## Inputs to confirm

Ask the user for these. Use placeholders if not provided — do not block.

- `service-name` (slug; becomes container name and python package)
- `yolo26-onnx-path` (host path to the YOLO26 ONNX export — not `.pt`)
- `yolo26-labels-path` (host path to a one-class-per-line labels file)
- `yolo26-parser-lib` (host path to the YOLO26 custom-parser `.so`, or the
  parser function name if compiled into the customer image)
- `num-classes` (integer; must match the labels file)
- `target-vss-profiles` — any subset of:
  - `bp_developer_search_2d` (Search Profile)
  - `bp_developer_alerts_2d_cv` (Alerts Profile + Behavior Analytics consumer)
- `kafka-bootstrap` (Kafka's host-reachable advertised listener; default `localhost:9092`)
- `kafka-topic` (default `mdx-raw` — VSS consumers expect this name)
- `input-rtsp-uri` or input video file path

## Workflow

1. Read [references/integration-contract.md](references/integration-contract.md). The fixed parts of the
   contract (topic name, payload type, schema library) are not negotiable
   if the goal is to plug into an existing VSS deployment.
   For segmentation/frame-mask services, follow the segmentation payload
   contract in that reference before changing msgconv or wrapper code.
2. Scaffold the **YOLO26 reference** service (do not generalize pgie paths unless
   the user explicitly needs another detector):

   ```bash
   python3 scripts/scaffold_rtvi_vss_service.py \
     --service-name <service-name> \
     --output-dir <target-dir> \
     --num-classes <N> \
     --vss-profiles bp_developer_search_2d bp_developer_alerts_2d_cv
   ```

3. Drop the customer's YOLO26 ONNX, labels file, and custom-parser library
   into the paths the generated `service_config.json` references.
   If those artifacts do not exist yet, stop after scaffolding and host-only
   unit tests; the live DeepStream service cannot start correctly without
   them.
4. Build the image:

   ```bash
   docker build -t <service-name>:dev <target-dir>
   ```

5. From the separate VSS checkout, bring up the matching Search or Alerts
   profile using the VSS Quickstart and `deploy/docker/scripts/dev-profile.sh`.
   Confirm that VSS and its one-shot Kafka topic initializer are ready, then
   start the generated service as a separate Compose application:

   For Search or Alerts, find the matching one-shot initializer without
   assuming a Compose project or container name:

   ```bash
   docker ps -a \
     --filter label=com.docker.compose.service=kafka-topic-init-container \
     --format 'table {{.Names}}\t{{.Status}}'
   ```

   Identify the selected deployment's initializer and confirm it is
   `Exited (0)`, then start the generated service with the matching gate:

   ```bash
   cd <target-dir>
   docker compose -f compose/service.compose.yml \
                  --profile bp_developer_search_2d up -d

   # Or for Alerts:
   docker compose -f compose/service.compose.yml \
                  --profile bp_developer_alerts_2d_cv up -d
   ```

   Do not add a cross-file `depends_on` entry. The generated service uses
   `network_mode: host`. Set `KAFKA_BOOTSTRAP` to Kafka's host-reachable
   advertised listener (default: `localhost:9092`). The readiness check
   enforces startup order.

6. Verify the metadata flow:

   ```bash
   cd <target-dir>
   python3 tools/kafka_smoketest.py --describe-only
   python3 tools/kafka_smoketest.py --timeout 120
   ```

   The smoke test auto-detects Kafka only when exactly one matching container
   is running. If multiple deployments are active, pass the selected running
   name with `--kafka-container`. Its `--bootstrap-server` is resolved inside
   that Kafka container and normally remains `localhost:9092`, even when the
   detector's host-facing `KAFKA_BOOTSTRAP` uses a different port.

   A successful consume confirms at least one non-empty message arrived on
   `mdx-raw` within the timeout. It does not decode the protobuf payload or
   validate sensorId, objects, or bbox fields.

7. Confirm downstream pickup:
   - **Search Profile**: query the Video Analytics API for ingested events,
     run an embed query and an attribute query against the search workflow.
   - **Alerts Profile**: watch `mdx-incidents` for behavior-analytics
     output; the `alert-bridge` service should generate VLM-verified
     incidents.
   - **Behavior Analytics**: the selected VSS Search or Alerts deployment starts its own behavior consumer. This is part of the VSS stack, not another profile for the generated service. It consumes `mdx-raw` and emits behavior windows to `mdx-incidents`.

## Service shape (generated)

The scaffolder emits this layout:

```
<service-name>/
├── Dockerfile                            DeepStream 9.1 + custom parser hook
├── README.md                             customer-facing build/run/plug-in guide
├── service_config.json                   declarative config (topic, profile flags, paths)
├── compose/
│   └── service.compose.yml               service def with profiles for VSS plug-in
├── pipeline/
│   ├── ds-app-config.txt                 deepstream-test5 derived; msgconv→msgbroker→mdx-raw
│   ├── ds-start.sh                       container entrypoint
│   └── configs/
│       ├── pgie-yolo26-config.txt        nvinfer config skeleton for YOLO26
│       ├── tracker-nvdcf.yml             NvDCF tracker config
│       ├── cfg_kafka.txt                 librdkafka producer overrides
│       ├── msgconv_config.txt            mega2d sensor context
│       └── labels.txt                    class labels placeholder
├── app/
│   ├── __init__.py
│   ├── contracts.py                      NvDsEventMsgMeta-aligned event envelope
│   ├── pipeline_plan.py                  declarative stage list (used by tests)
│   └── service.py                        adapter helpers for extension/derived events
├── tools/
│   └── kafka_smoketest.py                consumes one mdx-raw message and asserts it is non-empty
└── tests/
    ├── test_pipeline_config.py           asserts the Kafka adapter contract and msgconv→msgbroker→mdx-raw wiring
    ├── test_service.py                   exercises the python adapter without GPU
    └── test_compose.py                   asserts profile gates, host networking, and liveness healthcheck
```

## Non-negotiable contract elements

The customer can change almost everything *except* these — they're what
the VSS stack consumes:

- Kafka topic `mdx-raw`, using the host-reachable advertised listener selected by `KAFKA_BOOTSTRAP` (default: `localhost:9092`).
- `msg-conv-payload-type=2` (`NVDS_PAYLOAD_DEEPSTREAM_PROTOBUF`) with `msg-conv-msg2p-new-api=1` so the generated `deepstream-app` serializes frame/object metadata directly: protobuf serialized by
  `msg-conv-msg2p-lib` (`libnvds_msgconv_mega2d.so`, or `libnvds_msgconv.so`
  on DGX-SPARK/THOR); `libnvds_kafka_proto.so` is the `msg-broker-proto-lib`
  Kafka transport adapter only.
- A protobuf `Frame` payload with `sensorId`, `timestamp`, and `objects[]`
  carrying `id`, `bbox`, `type`, `confidence`. This is what
  `vss-search-analytics-*`, `vss-behavior-analytics-*`, and
  `vss-video-analytics-api-*` deserialize.
- `network_mode: host`.
- An explicit deployment precondition that the VSS `kafka-topic-init-container`
  completed successfully before this separate Compose application starts.

If any of these change, the customer is no longer plugging into VSS — they
are running an isolated CV service.

## Validation

Run unit tests on any host (no GPU required):

```bash
cd <target-dir>
python3 -m unittest discover -v -s tests -p 'test_*.py'
```

Run end-to-end on a host with GPU + a deployed VSS stack:

```bash
cd <target-dir>
python3 tools/kafka_smoketest.py --describe-only
python3 tools/kafka_smoketest.py --timeout 120
```

A successful smoke test confirms non-empty bytes arrived on `mdx-raw` within the timeout. It does not decode the protobuf payload or prove the message came from this service specifically. To verify downstream pickup, check the Search / Alerts / Behavior Analytics services via their own APIs.

Do not present local unit-test success as proof that live VSS integration is
ready. Without the customer ONNX, labels file, and parser library, the live
DeepStream path remains blocked even if scaffolding and unit tests succeed.

## Implementation guardrails

- Do not invent topic names. `mdx-raw` is the single perception ingress.
  Per-service prefixes break the existing VSS consumers.
- Do not switch to `msg-conv-payload-type=0` or `=1`. VSS deserializers
  expect type 2.
- Do not bridge through a custom Python Kafka producer. The DeepStream
  `nvmsgconv → nvmsgbroker` boundary is what produces correctly-framed
  protobuf with timestamps in nanoseconds. A Python producer drift will
  silently degrade Behavior Analytics.
- Keep `track_id` stable across frames. Behavior Analytics derives dwell
  and direction from track continuity; a per-frame regenerated id makes
  every frame look like a new object.
- Do not embed credentials, NGC tokens, or absolute customer paths in the
  scaffolded files. The generated tree must be portable.