Vss Search Archive

---
name: vss-search-archive
description: Use this skill when a user wants to search archived VSS video or ingest or delete a source for search. Do not use it for visual Q&A, live captioning, or video summarization.
license: Apache-2.0
metadata:
  author: "NVIDIA Video Search and Summarization team"
  version: "3.3.0-rc0"
  github-url: "https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization"
  tags: "nvidia blueprint operational"
  # What a live deployment must expose for this skill to be usable, as the vss CLI
  # names it: a command group (search, summarize, vlm, vios, memory), "alerts"
  # (Alert Bridge), or "always" for a skill every VSS deployment gets. The
  # OpenClaw harness image ships and activates skills by it.
  vss-requires: "search"
---

## Purpose

Operate archive search from the caller's host. Compose and Kubernetes use the
same `vss configure` and `vss search run` commands; only the deployment origin
differs. Source ingestion and deletion are Agent-backed **when the deployment has
an agent `/api` route**; on a build without one, they belong to
`vss-manage-video-io-storage` `references/provision-vios-source.md`.

## When to Use

- Search archived VSS video by natural-language or similarity query
- Ingest a source for search, or delete a previously ingested source (Agent-backed when an agent `/api` route exists)

Not for visual Q&A, live captioning, or video summarization.

## Hard boundaries

- Run the project-local CLI on the host. Never use `docker exec`, `kubectl
  exec`, a pod shell, or a globally installed `vss` as a substitute.
- Never improvise a mutation against Elasticsearch, RTVI-CV, RTVI-Embed,
  storage-ms, or VST. Two paths are sanctioned, and the deployment picks which:
  the Agent upload/delete lifecycle where an agent `/api` route answers, and
  `vss-manage-video-io-storage` `references/provision-vios-source.md` where none
  does. That recipe owns VIOS registration and the hand-driven RT-VLM legs;
  nothing sanctions a direct call to RTVI-CV or RTVI-Embed, which receive every
  source from VIOS.
- Never remove, broaden, or silently substitute a requested source constraint.
- Similarity is retrieval evidence, not proof of visual presence.
- The CLI attempts critic verification by default. Do not separately inspect
  screenshots or call another verifier during the initial search turn.
- Offer delegated verification only when every displayed result is
  `unverified`, and only after displaying them and receiving explicit user
  confirmation. If any result is `confirmed` or `rejected`, do not hand off
  any result to another verifier.

## Prerequisites

- A running VSS `search` profile and its host-reachable Compose or Ingress
  origin.
- The `vss` CLI on `PATH`. The OpenClaw and Hermes harness images ship it; anywhere else, install it from the same checkout as this skill so the CLI and the skill match: `uv tool install <checkout>/libs/vss/cli`.
- `curl` and `jq`.
- `vss vios list` for source listing and inspection (same CLI, same recorded origin).

Check the CLI once:

```bash
vss search run --help >/dev/null || exit 1
```

Resolve the deployment through its one public/host origin:

```bash
if [ -z "${VSS_ORIGIN:-}" ]; then
  VSS_ORIGIN=$(vss configure show 2>/dev/null |
    jq -er '.base_url | select(type == "string" and length > 0)') || {
      echo "Provide the Compose or Ingress origin" >&2
      exit 1
    }
fi
VSS_ORIGIN="${VSS_ORIGIN%/}"
VST_URL="${VSS_ORIGIN}"
VSS_VIOS_URL="${VSS_ORIGIN}/vst"
vss configure --base-url "${VSS_ORIGIN}" || exit 1
```

In a persisted multi-step workflow, reuse the origin recorded by the prepared
deployment as above. Do not repeat public-origin selection, edit routing, or
redeploy merely because the next agent turn did not inherit shell variables.

See [deployment resolution](../../vss-build-vision-ai/references/deployment_resolution.md)
for the deployment-owned `VSS_PUBLIC_URL` contract. On Kubernetes, never use
port-forwarding, Service DNS, NodePorts, or a guessed Helm release. Routes not
exposed through the Ingress are recorded as absent and a search path needing
one exits 4.

For deployment readiness, ingestion, fixture cleanup, index checks, RTSP, or
deletion, read [source lifecycle](references/source_lifecycle.md) completely
before acting. Re-run `vss configure` after the first ingestion: the recorded
raw family is what enables frame-level lookups (it gates `frames_index`, which
attribute and fusion need for frame enrichment). Only source-type selection is
independent of the index inventory.

## Mandatory search workflow

1. Confirm the selected deployment is the `search` profile. If required routes
   are unavailable, ask whether to reconnect or deploy it with
   `the`/vss-build-vision-ai`stock Search workflow`; do not target another profile.

2. When the user names a file, camera, or sensor, list registered sources with
   `vss vios list` before invoking the search CLI — it reads the origin
   `vss configure` recorded, so it takes no endpoint. Accept only an exact
   source, stream ID, or one unambiguous normalized substring match.

   - No match: report the missing source, list available names, and ask the
     user to clarify or explicitly request ingestion. Stop without probing the
     search CLI, deploying, or ingesting. **Never continue with a different
     source.** Answering about `warehouse_sample` when the request named
     `warehouse-ladder` returns a confident answer about the wrong video, and
     nothing downstream can tell it was substituted.
   - Several matches: ask the user to choose and stop.
   - Never substitute another video or run an unrestricted search as a probe.

   Preserve both the matched source's `.sensorId` and `.name`. The
   `--video-source` value depends on the search path, not the source type (optional for every path):
   `embed` matches the sensor ID literally; `attribute` and `object` match the
   name literally; only `tag` resolves a source name to its VST sensor ID (passing an already-id through). `fusion` does **not** resolve — its embedding leg filters by sensor ID literally — so hand fusion the preserved sensor ID (the tag leg accepts IDs too). For every path an unknown source yields an empty, narrowed result, not an error.
   Set `--source-type video_file` for uploads or `--source-type rtsp` for live
   streams. This selects the index partition for that media kind from a fixed
   uploads anchor (not a discovered index), independently of the identifier, so
   it is correct regardless of ingestion order.

3. Decompose the request before choosing a path; do not pick by surface form.
   Before changing or splitting it, preserve the user's exact sentence as
   `ORIGINAL_QUERY`. Retrieval may use the decomposed query, attributes, or
   object IDs, but critic verification must receive this original wording.
   `run embed` accepts any sentence, so being one sentence is not evidence for
   embed. Separate each specific detectable property (`white jacket`, `red hard
   hat`) from the actions/relations only embeddings capture, then choose:

   - a detectable property plus an action or relation is present → `run fusion` (even within one sentence)
   - free-text intent with no detectable property → `run embed`
   - detectable properties only, no action or relation → `run attribute`
   - explicit tracked object IDs → `run object`
   - explicit keyword or tag intent — lexical (BM25) match against indexed VLM tags, with no detectable property and no semantic free-text → `run tag`

   `--attribute` is for specific detectable properties, not generic nouns or
   actions. A property counts only when RT-CV detects it on the subject (attire,
   PPE, color-on-person), not object identity or an object's own color; keep
   `red forklift` wholly in `--query`. `worker in a hard hat carrying a cone` has
   a property (`hard hat`) and an action (`carrying a cone`): `run fusion --query
   "worker in a hard hat carrying a cone" --attribute "hard hat"`. Reserve embed
   for genuinely attribute-free intent. `run tag` is for explicit lexical
   intent — matching indexed VLM tag keywords by BM25 — not semantic similarity;
   reserve it for keyword/tag queries that name no detectable property.

4. Construct the invocation as a Bash array and validate only its exact
   stdout. Read [CLI usage](references/cli_usage.md) for every supported flag.

```bash
: "${SEARCH_PATH:?set embed|attribute|fusion|object|tag}"
: "${SOURCE_TYPE:?set video_file or rtsp}"
: "${ORIGINAL_QUERY:?set the exact pre-decomposition user question}"
TOP_K="${TOP_K:-3}"
VIDEO_SOURCES=() # sensor IDs for embed/fusion; names for attribute/object/tag
: "${SOURCE_SCOPED:?set true for a resolved scope; false only when unrestricted}"
if [ "${SOURCE_SCOPED}" = true ] && [ "${#VIDEO_SOURCES[@]}" -eq 0 ]; then
  echo "Resolved source scope is empty; refusing an unrestricted search" >&2
  exit 1
fi
SEARCH_COMMAND=(
  vss search run "${SEARCH_PATH}"
  --source-type "${SOURCE_TYPE}" --top-k "${TOP_K}"
  --original-query "${ORIGINAL_QUERY}" --raw
)
for source in "${VIDEO_SOURCES[@]}"; do
  SEARCH_COMMAND+=(--video-source "${source}")
done
# Append --query, repeatable --attribute, --object-id, and time bounds as needed.
if ! SEARCH_JSON=$("${SEARCH_COMMAND[@]}"); then
  echo "Search command failed" >&2
  exit 1
fi
printf '%s' "${SEARCH_JSON}" |
  jq -e 'type == "object" and (.data | type == "array")' >/dev/null || {
    echo "Search did not return a SearchOutput object with a data array" >&2
    exit 1
  }
```

Do not pass endpoint, index, model, deployment, profile, or base-URL flags to
`search run`; `vss configure` owns those values. Do not replace a failed CLI
call with `/api/v1/search` or private backend access.

1. Validate each nonempty hit's exact returned `screenshot_url` with a bounded
GET for availability only. Its normalized scheme, host, and effective port
always match the origin recorded by `vss configure`, because the CLI stamps
that origin into every hit — a localhost media URL means the deployment was
configured against a localhost origin, not that the URL is malformed. On Brev,
prefer the public HTTPS secure-link origin. If setup used the documented
host-reachable fallback after its one bounded public probe failed, accept only
that exact recorded origin and label its media URLs host-local; do not restart
routing diagnosis. Reject credentials in the URL and never rewrite the URL or
add a `streamId` routing header. Discard the response body; availability is not
visual evidence.

2. Read every hit's `verification` object:

   - `confirmed`: the critic found all requested visual criteria in that clip.
   - `rejected`: the critic found a visual criterion was not met.
   - `unverified`: no usable critic verdict was produced. This includes a
     missing VLM, inaccessible media, and malformed or inconclusive output.

The CLI is fail-open: verification failure must not discard or fail retrieval.
Never derive a verdict from similarity, filenames, object IDs, or screenshot
availability. Treat boolean `criteria_met` values as critic evidence only.

1. Format nonempty results without raw JSON. The final reply is user-facing,
   not a diagnostic trace: identify a hit by the source name the user supplied
   or its display filename, never a raw `sensor_id` or stream UUID. Never expose
   a job ID, model or service name, endpoint, CLI flag, or implementation terms
   such as "VLM" or "critic". Say "visual verification" when it is relevant,
   and report only its `confirmed`, `rejected`, or `unverified` result.

```text
## Video Search Results
<each hit's exact source, start/end, similarity, complete media URL,
verification result, and criteria when present>

Similarity scores are retrieval evidence; the separate verification result
records whether the bounded clip satisfied the visual request.

## Verification Step
Would you like me to verify the unverified search results?
```

Include `## Verification Step` only when the nonempty displayed result set is
entirely `unverified`. If any displayed result is `confirmed` or `rejected`,
omit it even when other hits are unverified. Never deploy a VLM or call
`vss-ask-video` automatically during this results turn.

1. If the user explicitly confirms, read
[search-result verification](references/result_verification.md) completely and
delegate the displayed hits only after confirming again that every one is
still `unverified`. Preserve their exact bounded intervals and the complete
original visual intent. Keep at most three delegations in flight. Never hand
off a partially verified result set.

2. If `.data` is empty, report zero candidates faithfully — a fact about
retrieval, not about the video. Do not claim the object is absent, describe
what the footage contains, or argue it is not something you would expect
there: a threshold or embedding gap yields the same empty result as a genuine
absence. Offer a specific query or similarity-threshold refinement while
preserving the source. Never broaden the search silently.

## Natural-language Agent responses

Use the host CLI for deterministic structured search. If a caller explicitly
requires the deployment Agent to decompose a natural-language request, its
`/api/v1/search` response is conversational text, not `SearchOutput`. Validate
the known text field and present it as prose; never run `.data[]`, screenshot,
or verification parsing against that response or invent structured hit rows.

## Troubleshooting

- CLI unavailable: `vss` is not on `PATH`; install it as *Prerequisites* says, or report the image problem, and stop.
- Exit 2: read the selected path's `--help`; do not guess flags.
- Exit 3: a recorded backend is unreachable; repair routing and reconfigure.
- Exit 4: run `vss configure --base-url <origin>` or choose a path whose
  required services are actually routed.
- Exit 5: ingest the source, wait for readiness, and re-run `vss configure`.
- Missing/ambiguous source: stop for clarification; never substitute.
- Missing RT-VLM: retrieval remains valid and results remain `unverified`.
- Authentication: use the operator-approved route. Never place secrets in
  prompts, flags, generated files, logs, or skill output.