Workbench Reference Workflows

---
name: workbench-reference-workflows
description: Use when working on NPA reference workflow specs, runner scripts, cookbooks, customer-adaptable pipeline implementations, or the guarded examples that are not workflow authoring surfaces.
---

# Workbench Reference Workflows

> The supported, customer-facing catalog and source of truth is the `npa.workflow` spec set under
> `workflows/`. The old raw SkyPilot task catalog has
> no remaining templates. Raw SkyPilot YAMLs may still exist only as guarded
> tool-specific examples or resource profiles, not as workflow authoring
> surfaces. SkyPilot remains the engine that executes rendered specs.

The catalog keeps the three principal pipelines in `workflows/main/`:
`sim2real.yaml`, `paidf-cosmos3.yaml`, and `nurec-reconstruct.yaml`.
Partner integrations live in `workflows/partners/<partner>/`; general reference
workflows, component tests, and fixtures live in `workflows/testing/`.
Keep catalog documentation in `workflows/README.md` and partner runbooks beside
their specs.

## When To Use

Use this skill for repository workflow YAMLs, runner scripts, cookbooks,
artifact contracts, and customer-adaptable pipeline implementations.

## Procedure

1. Start from the closest checked-in `npa.workflow` spec under
   `workflows/`.
2. Reuse a toolRef from
   `npa/src/npa/orchestration/npa_workflow/catalog.py`; add missing behavior to
   the workbench tool rather than implementing it again in a runner.
3. Keep the runner thin. Python runners materialize config, call the workflow
   submission helper, and report artifacts; the spec owns the stage graph.
4. Keep all input and output paths configurable and run-scoped through S3.
   Stages run in separate pods and cannot depend on a repository-relative path.
5. Declare the output the tool actually writes. Extend
   `test_spec_declared_outputs.py` when a tool exposes a result-URI helper.
6. Run `validate-spec`, then `plan-spec --run-id preview`, before live submit.
   Register every shipped spec in `SUBMIT_LIVE_MATRIX`.

For `field-failure-policy-improvement.yaml`, the native reference adapters join
metric RGB-D reconstruction and shared-scene Isaac navigation. Follow the
[native adapter contract](../../../docs/workbench/cookbooks/field-failure-policy-improvement.md#native-metric-capture-and-isaac-reference).
Include both components in the reviewed source distribution, pin the navigation
module inventory separately from the immutable image, and preserve source-overlay
provenance. TSDF reconstruction is CPU work; native renderer/learner execution on
RTX supplies the GPU evidence. Retain before/after replays of training-exposed
failures separately from the untouched paired cohort. Only the latter may drive
promotion; valid losing baseline episodes must not be discarded by a standalone
success threshold. Public reference results do not establish compatibility with
an undisclosed proprietary policy or calibration.

For the public scan-to-policy sample, derive the parked isolation-control pose
from dense rotated stance support and exact source topology before learning.
Preserve its measured selection evidence in `support.json`; five corner/center
rays or a large center clearance alone cannot exclude a small hole or a source
boundary beneath a foot. Keep the free/obstacle controls, action sequence,
native contact classification, cohort sizes and success gates unchanged when
repairing initial parking placement. Initial CPU support evidence does not
replace native control and full training/evaluation qualification.

For `field-failure-reference-demo.yaml`, freeze training, development and final
cohorts and the simulation-failure admission rule before learning. Evaluate the
exact baseline on every frozen office training route before admitting its public
capture; recompute complete native outcomes, bind admission into the public bundle
reference, and reverify it before reconstruction and training. A crashed or
control-failed observation cannot admit a capture; zero failures publish an honest
terminal report without continuation or final evaluation. Preserve the distinction
between these observed simulation failures and operator-supplied physical logs.
Before admitting final evaluation, development selection
must pair the exact case IDs and seeds and apply every per-case regression bound
from the frozen `plan.metrics`, using the final comparator's strict boundary.
Compute success gain as the mean paired success delta so an improvement of exactly
one percentage point passes without an epsilon or a weakened threshold. Failed
development selection must publish the measured violations in the HTML report
and leave final evaluation untouched.
Interrupted selection/report publication may accept identical retained bytes
to finish missing report files. Reject different bytes without overwriting
evidence; report recovery must not change eligibility or consume final cases.

## Current Reference YAMLs

For a first BEHAVIOR DEV evaluation, use
[the one-file onboarding guide](../../../docs/workbench/challenge-onboarding.md)
and `npa workbench workflow challenge init`, `check`, and `prepare`.
The helper materializes the canonical evaluation workflow and preserves the ten
prescribed cases. A local `ready-to-prepare` result does not prove GPU readiness.
The simulator runtime, licensed assets and task-configured policy remain
operator prerequisites. Keep generated kits private; use the campaign path from
the outset for durable case recovery. Other challenges are not supported by
this helper yet.

The retired catalog path is machine-checked by
`npa/tests/guardrails/test_skypilot_catalog_retirement.py`, so a raw template
cannot quietly reappear there.

No raw SkyPilot templates remain in the retired catalog. Author workflow examples
as `npa.workflow/v0.0.1` specs under `workflows/testing/` or, for partner
integrations, `workflows/partners/<partner>/`.

## Retired Templates

These raw templates were retired once their `npa.workflow` spec had a live run
(run ids in `EVIDENCE.md`). Use the spec under
`workflows/`:

- `isaac-lab-rl-sweep.yaml` — parallel GPU sweep (`--runtime`).
- `cosmos3-reason.yaml` — Cosmos3 reason-stage manifest.
- `sonic-export.yaml`, `sonic-eval.yaml`, `sonic-export-eval.yaml` — SONIC
  export/eval. The tools now accept `s3://` inputs and outputs directly, which is
  what the templates' inline download/upload bash used to do.
- `token-factory-caption.yaml`, `token-factory-generate.yaml`,
  `token-factory-cosmos-reason.yaml` — hosted Token Factory stages.
- `mjlab-eval.yaml` — MJLab locomotion evaluation.
- `retargeting.yaml` — motion retargeting. The harness synthesizes a SOMA-CSV clip
  (`npa.workflows.motion_fixture`) when no real motion set is staged.
- `vlm-eval.yaml`, `vlm-eval-benchmark.yaml` — self-hosted VLM scoring and the labeled
  sweep. The renderer now starts and health-checks the vLLM server the spec asks for, so
  no prebuilt serving image is needed.
- `cosmos3-text-to-image-inference.yaml` — retired to
  `workflows/testing/cosmos3-text-to-image.yaml`. The procedure it carried as bash inside an `envs:`
  block is now `npa workbench cosmos3 text-to-image`.
- `bdd100k-pipeline.yaml` — retired to `workflows/testing/bdd100k-pipeline.yaml`. A live run needs
  both in-cluster services (`lancedb` and `detection-training`) deployed first.
- `dataset-ingest-curate.yaml` — retired to `workflows/testing/dataset-ingest-curate.yaml`, whose
  `register` stage reads back what `ingest` wrote to the in-cluster LanceDB service
  (`npa workbench lancedb deploy --runtime kubernetes --namespace workbench`).
- `sim-to-real-pipeline.yaml` / `sim-to-real-trigger.yaml` — retired. The pipeline ran the
  deprecated `npa.workflows.sim_to_real real-loop`; the maintained path and the
  watcher's submit target are the single staged-engine YAML,
  `workflows/main/sim2real.yaml`.
- `cosmos2-transfer.yaml` — retired to `workflows/testing/cosmos2-transfer.yaml`, which runs the
  REAL Cosmos-Transfer2.5 model (`--execute`) instead of printing a `contract_ready` payload.
- `isaac-franka-capture-reason.yaml` — retired to
  `workflows/testing/isaac-franka-capture-reason.yaml`. The capture code moved into the package
  (`npa.workflows.isaac_capture`), so the stage no longer needs a repo mounted into the pod.
- `sim2real-actions.yaml` — retired into `workflows/testing/sim2real-envgen-shards.yaml` as its
  fourth stage, which conditions the train slice the `split` stage just wrote.
- `tokenfactory-scene-to-rollout-judge.yaml` — hosted reasoner, GPU rollout, hosted judge. Its
  twin keeps the chain: `vlm-eval run --task-from` reads the reasoner's artifact, so the judge
  scores the rollout against the plan rather than a literal string.
- `tokenfactory-rollout-judge.yaml` — GPU rollout then a hosted VLM judge. Its twin is
  `workflows/testing/tokenfactory-rollout-judge-combo.yaml`; note the older same-named spec is a
  *different* workflow (a Cosmos reasoner feeding a judge over externally-seeded rollouts).
- `tokenfactory-train-triage.yaml` — GPU LeRobot training then a hosted triage report. Its twin
  `workflows/testing/tokenfactory-train-triage.yaml` trains in the stage's own pod (the renderer
  switches to the vendor image's interpreter) and triages with
  `npa.workflows.token_factory_triage`. Needs a SkyPilot-hostable LeRobot image; 0.5.1 ships a
  torch/torchcodec ABI mismatch, 0.6.0 does not.
- `cosmos3-ea-fetch.yaml` — Cosmos source/checkpoint fetch. Its twin
  `workflows/testing/cosmos-fetch.yaml` is the two CLI commands the template wrapped in ~60 lines
  of setup bash; the renderer installs `huggingface_hub[cli]`, which was the only load-bearing
  line of that preamble.
- `cosmos3-generate.yaml` — Cosmos 3 omni-model generation in the `npa-cosmos3`
  image. Its twin `workflows/testing/cosmos3-generate.yaml` ran through the live
  submit matrix and produced `generated/generate.json` plus a non-flat 960x960
  `generated/vision.jpg`.
- `nurec-reconstruct.yaml` — **relocated**, not retired: #234 deliberately
  shipped and live-verified a single-pod NuRec/NRE SkyPilot task in addition to
  the multi-pod `npa.workflow` spec. It now lives at
  `npa/src/npa/workbench/nurec/examples/` with its own README and guardrail.
- `sim2real-envgen-split.yaml` — raw env generation + 80/20 split. Its twin
  `workflows/testing/sim2real-envgen-shards.yaml` declares the shard fan-out as a `parallel:`
  group instead of relying on a Kubernetes Job completion index, and runs on CPU.
- `scenario-gen-adversarial.yaml` — adversarial scenario mining. Its twin
  `workflows/testing/scenario-gen-smoke.yaml` runs the same two CLI commands; the template's GPU
  image advertised an RL adversary the CLI cannot select.
- `sim-to-real-loop.yaml` — the rollout-SET loop. Retired via a new tool capability
  (`npa workbench vlm-eval loop`), because nothing else produced
  `task_success_report.json`; the spec is `workflows/testing/vlm-eval-loop.yaml`.
- `isaac-lab-cosmos-sdg-burst-smoke.yaml` — **relocated**, not retired: a single-task input to
  `npa burst submit-yaml`, now at `npa/src/npa/burst/examples/`. Burst is scoped to one
  executable task, so there is no plan or stage graph for a spec to describe.
- `isaac-lab-rl-train-rtxpro.yaml`, `isaac-lab-rl-train-rtxpro-smoke.yaml`,
  `isaac-lab-rl-train.yaml`, `byof-datagen-rtxpro-smoke.yaml`,
  `byof-container-smoke-rtxpro.yaml` — **relocated**, not retired: they are BYOF
  *resource profiles* (a pod shape), not workflows, and now live beside their
  runner at `npa/src/npa/workflows/byof/profiles/`.

The retired catalog is pinned empty in
`npa/tests/guardrails/test_skypilot_catalog_retirement.py`; do not add new raw
workflow templates.

## Three-Tier Contract

- CLI: use `npa workbench workflow ...` and tool-specific workflow commands
  such as `npa workbench mjlab workflow` or `npa workbench sonic retargeting workflow`.
- SDK: route through shared workflow submission helpers rather than shelling out
  from business logic.
- Workflow: the `npa.workflow` spec is the executable source of truth for stage
  order, resources, configuration, and artifact paths. ToolRef argv templates
  are the source of truth for commands; SkyPilot is the rendered execution
  layer.

## Gotchas

- The native Ray CLIP recipe has an optional completed-result archive companion
  at `npa/workflows/workbench/ray-clip-development/archive.py`; follow
  `docs/testing/ray-clip-archive.md` for quiescent source inventories, conditional
  S3 publication and hash-bound private restore. It does not submit Jobs or
  resume incomplete checkpoints, and its local rsync path needs no S3 credentials.

- Customer-provided raw SkyPilot `envs` does not support self-referencing
  interpolation; repository specs use resolved `config` tokens.
- `sky jobs launch` has no dry-run flag. Use `workflow submit --plan-only` for a
  rendered spec preflight.
- Keep SONIC locomotion orchestration in its spec; do not add a Python runner
  that re-implements the graph.

## Verify

```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q
```

The smoke test parses the listed workflow YAMLs and invokes workflow CLI help.