Isaac Lab

---
name: isaac-lab
description: Use when working on Isaac Lab RL simulation, deployment, SkyPilot workflows, or customer custom-fork support.
---

# Isaac Lab

Isaac Lab is the RL simulation framework. The canonical workbench pins the
latest published 3.Y beta point, `v3.0.0-beta2.patch1` / wheel
`3.0.0b2.post1`, with Isaac Sim `6.0.1.0`. Upstream has not labeled this a GA
release; keep the beta limitation explicit. Use L40S or RTX Pro 6000 for the
validated graphics/PhysX path. Rendering, camera-bearing tasks, and deployed
workbenches require RT cores and cannot target B200, H100, or H200.

The standalone training routing guard allows state-based, camera-free tasks to
select datacenter GPUs. This is routing eligibility, not proof that the selected
Isaac image, PhysX runtime, and driver work there. Require real training and a
checkpoint from that exact runtime before claiming support; historical Isaac
Lab 2.x results do not validate the generation 3 beta image.

Generation 3 training must invoke `--visualizer none`. A compatibility path
that can also receive a generation 2 image must select `--headless` only when
the image's `ISAAC_LAB_VERSION` starts with `2.`.

Before provisioning, building, downloading, or submitting an Isaac workload,
load `skills/atomic/third-party-eula-preflight/SKILL.md`. NPA defaults the
run-scoped `ACCEPT_EULA=Y` value for non-interactive use and preserves a
recognized negative value or `--no-accept-eula` opt-out. Legacy affirmative
spellings normalize case-insensitively; unrecognized values are errors.

## Runtime Isaac bootstrap (the container ships no Isaac Sim)

The `npa-isaac-lab` image contains **no NVIDIA Isaac Sim, Isaac Lab, Omniverse
Client, or other proprietary NVIDIA runtime code**. It used to bake Omniverse
Kit, which made it non-redistributable; the exact runtime wheel set is now
downloaded on first use of `/isaac-sim/python.sh` from
`https://pypi.nvidia.com`, hash-verified, and installed into a cache volume
under the **operator's own EULA acceptance**. Full rationale:
`docs/workbench/container-packaging.md` and `skills/atomic/solution-licensing/SKILL.md`.

What this changes in practice:

- **Unset acceptance succeeds; explicit opt-out refuses.** NPA defaults
  `ACCEPT_EULA=Y`. Empty, `N`, `NO`, `0`, `FALSE`, or `--no-accept-eula`
  exit **78** before download. `Y`, `YES`, `1`, and `TRUE` are accepted
  case-insensitively; other values are invalid. The launcher derives
  `OMNI_KIT_ACCEPT_EULA=YES` internally; do not expose duplicate user plumbing.
  Keep `PRIVACY_CONSENT` and telemetry off.
- **Reach Isaac through `/isaac-sim/python.sh`** (the value of `ISAAC_LAB_PYTHON`). That is
  the bootstrap shim, and it is what every SkyPilot template, the sim2real engine and the
  workbench CLI already use. A bare `python3` is the *system* interpreter and will not
  find Isaac.
- **Never invoke the shim from a Dockerfile `RUN`.** It would download and bake
  the multi-gigabyte Isaac runtime into a layer. Build-time work uses the
  image's own venv python.
- **Budget the first start.** Use the generation-specific measurements in
  `docs/workbench/isaac-lab-3.md`; do not reuse generation 2 cache numbers for
  generation 3. Pre-warm a shared volume once per node/PVC with
  `npa/docker/workbench/common/warm-isaac-cache.yaml`, then run workload pods
  with `NPA_ISAAC_CACHE_READONLY=1`. Otherwise every pod pays the cold fetch.
- `isaac-bootstrap status` reports what is cached without needing acceptance or network;
  `isaac-bootstrap verify` additionally launches Isaac Sim headless (needs a GPU).
- No NGC credentials are needed to build or run this image. Native
  `--runtime vm` installation is intentionally unsupported for generation 3;
  managed and BYOVM deployments use the payload-clean container contract.

## Interfaces

API:

- `POST /train`
- `POST /eval`
- `GET /status`
- `GET /system-info`
- `GET /list`

CLI:

```bash
npa workbench isaac-lab deploy
npa workbench isaac-lab train
npa workbench isaac-lab eval
npa workbench isaac-lab status
npa workbench isaac-lab system-info
npa workbench isaac-lab list
```

### Standalone checkpoint eval

`npa workbench isaac-lab eval` runs the supplied RSL-RL policy headlessly; it
does not invoke a VLM. Checkpoint loading is fail-closed: a load error produces
a structured failure artifact and a non-zero command result, never a
random-action substitute.

Choose the success predicate to match the task:

- `--success-metric survival` for locomotion, with termination as failure;
- `--success-metric goal-distance --success-distance-m <metres>` for
  manipulation or reach tasks;
- `--success-metric auto` to prefer a simulator-native success signal, then a
  measurable goal distance, and otherwise survival.

Use `--seed`, `--num-episodes`, `--max-steps-per-episode`, and
`--min-success-rate` for a repeatable held-out evaluation. The output is
`npa_isaac_lab_eval_summary.json` with format `npa.isaac_lab.eval.v1`; it
records checkpoint provenance, `policy_loaded`, per-episode metrics,
`success_rate`, and `passed`. An S3 output prefix is uploaded on both runtime
success and failure so failed evaluations remain diagnosable. `passed=false`
does not turn a completed rollout into a runtime error; automation should gate
on `passed` (the Sim2Real workflow does this in Stage 11). With
`--output-format json`, `eval_status`, `policy_loaded`, `success_rate`, and
`passed` are top-level structured CLI fields; callers do not need to scrape
the remote log tail.

### Trained-policy visual export

`npa workbench isaac-lab train --export-trajectories` captures genuine Isaac
Sim RGB by default during the post-training policy rollout. The exporter must
load the trained RSL-RL checkpoint fail-closed, launch Isaac with cameras
enabled, call the environment's `rgb_array` renderer, and write one `rgb.npy`
array per episode with exactly the same frame count as `state.npy` and
`actions.npy`. It records runtime version, checkpoint hash, renderer, image
dimensions/count, `policy_loaded`, and the shared
episode/frame/timestamp timeline in `trajectories/meta.json`.

The Isaac-to-LeRobot adapter automatically encodes these frames as the
`observation.images.workspace` video feature and preserves their provenance.
The LeRobot-to-Rerun adapter maps each episode to its own video asset and
`VideoFrameReference` timeline, makes the trained-policy environment the
prominent view, and omits evaluation panes when no evaluation entities exist.
Metadata-only rollouts remain valid but must not be described as containing an
environment visual. `--no-export-rgb` is the explicit scalar-only opt-out.

## Custom Forks

Canonical onboarding starts at `docs/workbench/getting-started.md`; do not
duplicate credential, S3, Kubernetes, registry, or SkyPilot bootstrap setup here.

Customers can bring their own Isaac Lab fork through an `image_id` override in the SkyPilot YAML. The workbench provides a validated base container; the customer layers their fork on top.

The replacement image must preserve the expected Isaac Lab entry point or runner contract.

Cookbook: `docs/workbench/cookbooks/byof-isaac-lab/README.md`.

Validated BYOF surfaces:

- image override through `npa/scripts/run_isaac_lab_rl.py --image`
  (`w10-byof-image-only-20260520T232650Z`);
- command override through a SkyPilot YAML `run:` block variant passed with
  `--yaml`, invoking `/opt/byof/custom_train.py`
  (`w10-byof-image-and-cmd-20260520T233113Z`).

The runner exposes `--image` directly. It does not expose a `--run-cmd` flag, so
custom entrypoints should use a customer-owned YAML variant that preserves the
runtime contract, checkpoint discovery, manifest creation, and S3 upload block.

## Sim2Real Held-Out Backend

Isaac Lab is also the default sim engine for the Sim2Real loop's non-VLM
held-out rollout eval. The held-out eval is backend-pluggable:

- `sim_backend=isaac` (default): the held-out rollout runs headless Isaac Sim
  inside the Isaac Lab image as the eval component Job. It uses the Isaac Lab
  manipulation task (`Isaac-Lift-Cube-Franka-v0` by default) for a Franka
  pick/lift rollout.
- `sim_backend=genesis`: the existing Genesis `FrankaPickPlaceEnv` path, kept
  fully intact.

Select with `--sim-backend`, env `NPA_SIM2REAL_SIM_BACKEND`, or the runbook
YAML. Both backends emit the identical `npa.sim2real.heldout_eval.v1` per-env
schema (`env_id`/`score`/`success`/`details`), so `report.json` and the
outer-loop gate are backend-agnostic. The VLM eval (Cosmos-Reason) is unchanged.

When Stage 10 has a genuine Isaac trainer checkpoint, object storage, and a
registry-qualified Isaac image, it selects `byo_isaac_eval`. That vectorized
adapter uses the standalone evaluator's shared `load_rsl_rl_policy`, metric,
`npa.isaac_lab.eval.v1`, and failure-summary implementation while retaining
generated-environment IDs/seeds and held-out camera capture. It writes
`eval/heldout/isaac-eval-summary.json` and nests the same evidence in
`eval/heldout/report.json`. Runtime, checkpoint, or policy-load failure aborts
Stage 10; `passed=false` means the eval completed below its quality bar and is
handled by the Stage 11 threshold gate. Stage 14 remains responsible for the
run-level RRD/MCAP visualization artifacts.

Asset handling mirrors the Genesis no-fallback provenance discipline:

- Stock: the built-in Isaac lift-cube manipuland, recorded as
  `asset_source=isaac_stock` (no sha256).
- BYO mesh: a customer mesh/URDF imported to USD via Isaac Lab's offline
  converters (`isaaclab.sim.converters.MeshConverter` / `UrdfConverter`),
  recorded as `asset_source=byo_mesh` with a sha256. A mesh that fails to
  import or load raises; there is no silent fallback to the stock asset.

The Isaac Lab image bakes no `npa` code, and Isaac Sim is only importable via
its bundled interpreter `/isaac-sim/python.sh`. The eval component injects
branch `npa` code into that interpreter at start from an S3 source tarball
(`NPA_SIM2REAL_SOURCE_TARBALL_URI`) or, when the repo is reachable, a git clone
(`NPA_SOURCE_REPO`/`NPA_SOURCE_REF`), and ensures `boto3` for the S3 client.

Architecture + licensing rationale: `docs/architecture/sim-backend-selection.md`.

## Operational Safety

Managed VM `deploy` defaults to in-place updates for existing aliases. Terraform
plans that would destroy or replace critical infrastructure are blocked unless
the operator passes `--replace` and confirms with `--yes` for automation.

## Workflows

- Single RL job: `npa/src/npa/workflows/byof/profiles/isaac-lab-rl-train.yaml`.
- Parameter sweep: `workflows/testing/isaac-lab-rl-sweep.yaml`.
- Runner: `npa/scripts/run_isaac_lab_rl.py`.

E2E is pending the training command fix tracked by `W9-isaac-lab-e2e-fix`.