Isaac Lab

---
name: isaac-lab
description: Use when working on Isaac Lab RL simulation, deployment, SkyPilot workflows, or customer custom-fork support.
---

# Isaac Lab

Isaac Lab is the RL simulation framework. The canonical workbench pins the
latest published 3.Y beta point, `v3.0.0-beta2.patch1` / wheel
`3.0.0b2.post1`, with Isaac Sim `6.0.1.0`. Upstream has not labeled this a GA
release; keep the beta limitation explicit. Use L40S or RTX Pro 6000 for the
validated graphics/PhysX path. Rendering, camera-bearing tasks, and deployed
workbenches require RT cores and cannot target B200, H100, or H200.

The standalone training routing guard allows state-based, camera-free tasks to
select datacenter GPUs. This is routing eligibility, not proof that the selected
Isaac image, PhysX runtime, and driver work there. Require real training and a
checkpoint from that exact runtime before claiming support; historical Isaac
Lab 2.x results do not validate the generation 3 beta image.

Generation 3 training must invoke `--visualizer none`. A compatibility path
that can also receive a generation 2 image must select `--headless` only when
the image's `ISAAC_LAB_VERSION` starts with `2.`.

Before provisioning, building, downloading, or submitting an Isaac workload,
load `skills/atomic/third-party-eula-preflight/SKILL.md`. NPA defaults the
run-scoped `ACCEPT_EULA=Y` value for non-interactive use and preserves a
recognized negative value or `--no-accept-eula` opt-out. Legacy affirmative
spellings normalize case-insensitively; unrecognized values are errors.

## Runtime Isaac bootstrap (the container ships no Isaac Sim)

The `npa-isaac-lab` image contains **no NVIDIA Isaac Sim, Isaac Lab, Omniverse
Client, or other proprietary NVIDIA runtime code**. It used to bake Omniverse
Kit, which made it non-redistributable; the exact runtime wheel set is now
downloaded on first use of `/isaac-sim/python.sh` from
`https://pypi.nvidia.com`, hash-verified, and installed into a cache volume
under the **operator's own EULA acceptance**. Full rationale:
`docs/workbench/container-packaging.md` and `skills/atomic/solution-licensing/SKILL.md`.

What this changes in practice:

- **Unset acceptance succeeds; explicit opt-out refuses.** NPA defaults
  `ACCEPT_EULA=Y`. Empty, `N`, `NO`, `0`, `FALSE`, or `--no-accept-eula`
  exit **78** before download. `Y`, `YES`, `1`, and `TRUE` are accepted
  case-insensitively; other values are invalid. The launcher derives
  `OMNI_KIT_ACCEPT_EULA=YES` internally; do not expose duplicate user plumbing.
  Keep `PRIVACY_CONSENT` and telemetry off.
- **Reach Isaac through `/isaac-sim/python.sh`** (the value of `ISAAC_LAB_PYTHON`). That is
  the bootstrap shim, and it is what every SkyPilot template, the sim2real engine and the
  workbench CLI already use. A bare `python3` is the *system* interpreter and will not
  find Isaac.
- **Never invoke the shim from a Dockerfile `RUN`.** It would download and bake
  the multi-gigabyte Isaac runtime into a layer. Build-time work uses the
  image's own venv python.
- **Budget the first start.** Use the generation-specific measurements in
  `docs/workbench/isaac-lab-3.md`; do not reuse generation 2 cache numbers for
  generation 3. Pre-warm a shared volume once per node/PVC with
  `npa/docker/workbench/common/warm-isaac-cache.yaml`, then run workload pods
  with `NPA_ISAAC_CACHE_READONLY=1`. Otherwise every pod pays the cold fetch.
- `isaac-bootstrap status` reports what is cached without needing acceptance or network;
  `isaac-bootstrap verify` additionally launches Isaac Sim headless (needs a GPU).
- No NGC credentials are needed to build or run this image. Native
  `--runtime vm` installation is intentionally unsupported for generation 3;
  managed and BYOVM deployments use the payload-clean container contract.

## Interfaces

API:

- `POST /train`
- `POST /eval`
- `GET /status`
- `GET /system-info`
- `GET /list`

CLI:

```bash
npa workbench isaac-lab deploy --image <reviewed-image@sha256:digest>
npa workbench isaac-lab train
npa workbench isaac-lab eval
npa workbench isaac-lab status
npa workbench isaac-lab system-info
npa workbench isaac-lab list
```

The previously accepted public Isaac Lab image is publication-quarantined: its
layers contain package-generated SSH host private keys. The default container
resolver therefore refuses it even though the historical tag may still pull.
For container/BYOVM deployment, pass `--image` with a rebuilt, byte-scanned
immutable digest; for workflows, use the exact toolRef `--image-override`.
Never override with the withdrawn public tag. VM-native deployment is a
different path and does not establish a replacement container release.

### Standalone checkpoint eval

`npa workbench isaac-lab eval` runs the supplied RSL-RL policy headlessly; it
does not invoke a VLM. Checkpoint loading is fail-closed: a load error produces
a structured failure artifact and a non-zero command result, never a
random-action substitute.

Choose the success predicate to match the task:

- `--success-metric survival` for locomotion, with termination as failure;
- `--success-metric goal-distance --success-distance-m <metres>` for
  manipulation or reach tasks;
- `--success-metric auto` to prefer a simulator-native success signal, then a
  measurable goal distance, and otherwise survival.

Use `--seed`, `--num-episodes`, `--max-steps-per-episode`, and
`--min-success-rate` for a repeatable held-out evaluation. The output is
`npa_isaac_lab_eval_summary.json` with format `npa.isaac_lab.eval.v1`; it
records checkpoint provenance, `policy_loaded`, per-episode metrics,
`success_rate`, and `passed`. An S3 output prefix is uploaded on both runtime
success and failure so failed evaluations remain diagnosable. `passed=false`
does not turn a completed rollout into a runtime error; automation should gate
on `passed` (the Sim2Real workflow does this in Stage 11). With
`--output-format json`, `eval_status`, `policy_loaded`, `success_rate`, and
`passed` are top-level structured CLI fields; callers do not need to scrape
the remote log tail.

### Trained-policy visual export

`npa workbench isaac-lab train --export-trajectories` captures genuine Isaac
Sim RGB by default during the post-training policy rollout. The exporter must
load the trained RSL-RL checkpoint fail-closed, launch Isaac with cameras
enabled, call the environment's `rgb_array` renderer, and write one `rgb.npy`
array per episode with exactly the same frame count as `state.npy` and
`actions.npy`. It records runtime version, checkpoint hash, renderer, image
dimensions/count, `policy_loaded`, and the shared
episode/frame/timestamp timeline in `trajectories/meta.json`.

The Isaac-to-LeRobot adapter automatically encodes these frames as the
`observation.images.workspace` video feature and preserves their provenance.
The LeRobot-to-Rerun adapter maps each episode to its own video asset and
`VideoFrameReference` timeline, makes the trained-policy environment the
prominent view, and omits evaluation panes when no evaluation entities exist.
Metadata-only rollouts remain valid but must not be described as containing an
environment visual. `--no-export-rgb` is the explicit scalar-only opt-out.

## Custom Forks

Canonical onboarding starts at `docs/workbench/getting-started.md`; do not
duplicate credential, S3, Kubernetes, registry, or SkyPilot bootstrap setup here.

Customers can bring their own Isaac Lab fork through an `image_id` override in the SkyPilot YAML. The workbench provides a validated base container; the customer layers their fork on top.

The replacement image must preserve the expected Isaac Lab entry point or runner contract.

Cookbook: `docs/workbench/cookbooks/byof-isaac-lab/README.md`.

Validated BYOF surfaces:

- image override through `npa/scripts/run_isaac_lab_rl.py --image`
  (`w10-byof-image-only-20260520T232650Z`);
- command override through a SkyPilot YAML `run:` block variant passed with
  `--yaml`, invoking `/opt/byof/custom_train.py`
  (`w10-byof-image-and-cmd-20260520T233113Z`).

The runner exposes `--image` directly. It does not expose a `--run-cmd` flag, so
custom entrypoints should use a customer-owned YAML variant that preserves the
runtime contract, checkpoint discovery, manifest creation, and S3 upload block.

## Sim2Real Held-Out Backend

Isaac Lab is also the default sim engine for the Sim2Real loop's non-VLM
held-out rollout eval. The held-out eval is backend-pluggable:

- `sim_backend=isaac` (default): the held-out rollout runs headless Isaac Sim
  inside the Isaac Lab image as the eval component Job. It uses the Isaac Lab
  manipulation task (`Isaac-Lift-Cube-Franka-v0` by default) for a Franka
  pick/lift rollout.
- `sim_backend=genesis`: the existing Genesis `FrankaPickPlaceEnv` path, kept
  fully intact.

Select with `--sim-backend`, env `NPA_SIM2REAL_SIM_BACKEND`, or the runbook
YAML. Both backends emit the identical `npa.sim2real.heldout_eval.v1` per-env
schema (`env_id`/`score`/`success`/`details`), so `report.json` and the
outer-loop gate are backend-agnostic. The VLM eval (Cosmos-Reason) is unchanged.

When Stage 10 has a genuine Isaac trainer checkpoint, object storage, and a
registry-qualified Isaac image, it selects `byo_isaac_eval`. That vectorized
adapter uses the standalone evaluator's shared `load_rsl_rl_policy`, metric,
`npa.isaac_lab.eval.v1`, and failure-summary implementation while retaining
generated-environment IDs/seeds and held-out camera capture. It writes
`eval/heldout/isaac-eval-summary.json` and nests the same evidence in
`eval/heldout/report.json`. Runtime, checkpoint, or policy-load failure aborts
Stage 10; `passed=false` means the eval completed below its quality bar and is
handled by the Stage 11 threshold gate. Stage 14 remains responsible for the
run-level RRD/MCAP visualization artifacts.

Asset handling mirrors the Genesis no-fallback provenance discipline:

- Stock: the built-in Isaac lift-cube manipuland, recorded as
  `asset_source=isaac_stock` (no sha256).
- BYO mesh: a customer mesh/URDF imported to USD via Isaac Lab's offline
  converters (`isaaclab.sim.converters.MeshConverter` / `UrdfConverter`),
  recorded as `asset_source=byo_mesh` with a sha256. A mesh that fails to
  import or load raises; there is no silent fallback to the stock asset.

The Isaac Lab image bakes no `npa` code, and Isaac Sim is only importable via
its bundled interpreter `/isaac-sim/python.sh`. The eval component injects
branch `npa` code into that interpreter at start from an S3 source tarball
(`NPA_SIM2REAL_SOURCE_TARBALL_URI`) or, when the repo is reachable, a git clone
(`NPA_SOURCE_REPO`/`NPA_SOURCE_REF`), and ensures `boto3` for the S3 client.

Architecture + licensing rationale: `docs/architecture/sim-backend-selection.md`.

## Shared-scene navigation BYOF

Use `workflows/testing/shared-scene-navigation.yaml` and
`docs/workbench/guides/shared-scene-navigation.md` for an existing registered
navigation task in an operator-owned Isaac image. The operator supplies a
source-hashed task adapter, exact image digest, self-contained scene USDZ and
deterministic train/evaluation/probe cases. Native RSL-RL trains and reloads an
actual checkpoint; no stock task or CPU fallback proves this integration.
GPU acceptance is unverified until the opt-in native test runs with those inputs.
The development extra pins CPU OpenUSD so scene and visibility regressions run
in CI; those checks do not substitute for native RTX sensor/physics acceptance.

Validate peer-contact exclusion separately from observation isolation. The
supported RGB-D mode is `all_robot_geometry_hidden`: hide every robot's render
geometry, including self and attachments, without altering collision physics.
Require live USD coverage, fresh paired RGB/depth invariance and a changing
camera positive control. Reject unsupported proxies, effects and sensor modes.
State/raycast support still requires real isolation and obstacle-contact probes.
Use RT-core resources. Do not claim thousands-of-camera throughput from the
focal-camera probe. Stage publications use immutable attempt objects and
provider-conditional claim/completion records; failed claims require a new run
prefix. Keep operator artifact locations and runtime logs private.

## Operational Safety

For a captured static environment, use
`workflows/testing/scan-to-isaac-navigation.yaml` and the contract in
`docs/workbench/guides/scan-to-isaac-navigation.md`. Its stateless native probe
runs with `/isaac-sim/python.sh` and tests PhysX ray intersections against the
assembled collision geometry. It verifies sealed scene/provenance bytes and
rejects raw OmniScripting/OmniGraph declarations before Kit opens the stage,
including hidden variants and nested archives. It records actual Isaac/PhysX
versions separately from the operator-declared immutable image reference;
runtime introspection does not attest a registry digest. This path has committed
opt-in live coverage but no completed GPU acceptance, navigation-policy result,
or NuRec-rendering proof. Stock robot tasks do not qualify these scene inputs.

Managed VM `deploy` defaults to in-place updates for existing aliases. Terraform
plans that would destroy or replace critical infrastructure are blocked unless
the operator passes `--replace` and confirms with `--yes` for automation.

## Workflows

### Calibrated RGB-D sensor rigs

Use `workflows/testing/multicamera-rgbd-capture.yaml` for a supplied static USD
scene plus calibrated camera rig and sampled trajectory. It calls
`npa.workflows.isaac_rgbd.cli capture` in the supported Isaac interpreter and
validates decoded S3 RGB/depth/masks/optional world points in a separate CPU
stage. The graph stays in the standard workflow runtime; no robot is needed.
The default four-camera procedural room is a sensor qualification fixture, not
evidence of an industrial scan, navigation task, or encoder-training quality.
GPU acceptance has not been run for this adapter.

Keep `config.source_overlay: true` and submit with `--stage-src` (or a verified
`NPA_SRC_S3_URI`) so both workers receive the new adapter even when the selected
Isaac image already contains an older NPA package. Before opening any supplied
stage in Kit, inspect every hashed USD layer through raw Sdf traversal, including
inactive and unselected variants. Reject scripting APIs/properties, graph
content, time samples and value clips, then reject composition errors after
opening. Portable USDZ roots and nested packages are supported: inspect every
archive member in private audit directories, including unused layers, and reject
traversal, symlinks, duplicate/colliding names, compression and renamed packages.
Explicit package-relative member paths remain unsupported. Author capture cameras
into the writable session layer. Relative dependencies may use parent components
only when the resolved path stays in the hashed bundle or package. Preserve ordinary
textures. Do not describe these checks as a sandbox for arbitrary USD plugins.

`workflows/testing/multicamera-rgbd-warehouse.yaml` runtime-collects the public
NVIDIA full warehouse with native `omni.kit.usd.collect` (USD and MDL dependencies),
then captures 265 poses over 66 meters using four 1280x720 cameras. Preparation is
`npa.workflows.isaac_rgbd.cli prepare-reference --output-path <s3-prefix>` in the
Isaac interpreter. Missing assets, native collection errors and content-audit
failures stop preparation. Vendor assets stay in operator storage. This is a
public authored reference, not a scan reconstruction or robot navigation result;
GPU acceptance remains pending. Point-cloud output includes a per-frame fused
world cloud retaining camera and pixel indices; validation recomputes it from
the original RGB-D bytes rather than trusting cloud hashes alone.

The contract requires Z-up meter scenes, finite rigid transforms, zero-skew
pinhole calibration, and strictly increasing sampled timestamps. Depth is
axial optical Z in meters; invalid values become zero with a boolean mask.
Never publish successful capture evidence without decoding actual artifacts,
matching all camera render reference times, and verifying calibration and
backprojection. See `docs/workbench/multicamera-rgbd-capture.md` for S3 schemas,
limits, pinned API references and opt-in live acceptance.

- Single RL job: `npa/src/npa/workflows/byof/profiles/isaac-lab-rl-train.yaml`.
- Parameter sweep: `workflows/testing/isaac-lab-rl-sweep.yaml`.
- Runner: `npa/scripts/run_isaac_lab_rl.py`.

E2E is pending the training command fix tracked by `W9-isaac-lab-e2e-fix`.