Neural Reconstruction

---
name: neural-reconstruction
description: Use when reconstructing real sensor captures into renderable 3D scenes with NVIDIA Omniverse NuRec / the Neural Reconstruction Engine (NRE) on Nebius — COLMAP source conversion to NCore V4, existing NCore V4 input, 3DGUT Gaussian training, renderable USDZ, novel-view rendering, and the Rerun recording the NPA agent displays. Also use when an NCore sequence will not load in NRE, when picking the GPU for a reconstruction, or when changing the nurec workbench tool, CLI, or SkyPilot workflow.
---

# Neural Reconstruction (NuRec / NRE)

## Source And Attribution

Adapted from the NVIDIA Omniverse NuRec agent skills at
<https://github.com/NVIDIA/nurec-skills> (`skills/nre`,
`skills/physical-ai-datasets`, `skills/ncore`) and the NVIDIA NCore data library
at <https://github.com/NVIDIA/ncore>.

The capability routing table, the easy mix-ups, the safe secret-verification
pattern, and several troubleshooting rows below are adapted from the NVIDIA
router skill
<https://github.com/NVIDIA/skills/tree/main/skills/physical-ai-neural-reconstruction>
(Apache-2.0), pinned at commit `0122ea0` (2026-08-01). That skill is a *router*:
it never runs anything, it decides which upstream sibling skill answers a
question. This skill is the opposite — it is the workbench implementation — so
the router's picker table is re-pointed at real `npa workbench nurec` verbs, and
each row upstream owns is marked as such rather than reproduced.

Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. Upstream licenses are
Apache-2.0 and CC-BY-4.0. Trademarks (NVIDIA, Omniverse, NuRec, NRE, Isaac Sim,
Cosmos) belong to NVIDIA. See `skills/NOTICE-NVIDIA-SKILLS`.

The NCore conversion image packages pinned Apache-2.0 NVIDIA source and its
MIT COLMAP reader; it contains no dataset, weights or proprietary NRE payload.
NRE remains a separately licensed NGC runtime. The new COLMAP ingestion path is
**not yet live validated** and has no accepted public image release. See
`skills/NOTICE-NVIDIA-NCORE-COLMAP` and
`docs/workbench/guides/nurec-colmap-reconstruct.md`. Existing preconverted-NCore
results below do not validate the new conversion path.

## When To Use

Load this skill when the user wants to:

- turn a real sensor capture (photographs, multi-camera clips) into a renderable
  3D Gaussian scene and render **novel views** from it;
- run, modify, or debug `npa workbench nurec` or
  `npa/src/npa/workbench/nurec/examples/nurec-reconstruct.yaml`;
- work out why an NCore sequence fails to load in NRE;
- choose the GPU for a reconstruction or rendering job;
- make a reconstruction run show up in the NPA agent's Rerun panel.

**Do NOT use this skill for:** Cosmos video augmentation (use
`skills/workflows/physical-ai-data-factory/SKILL.md`), per-object asset
extraction from sparse views (upstream `asset-harvester`, not implemented here),
or generic Rerun visualization of a Sim2Real run
(`skills/workbench/sim2real-engine/SKILL.md`).

## GPU Routing (hard constraint)

Gaussian reconstruction and rasterization are **RT-core** work. Route these jobs
at:

- **RTX PRO 6000 Blackwell** — `RTXPRO-6000-BLACKWELL-SERVER-EDITION:1` on the
  RT-core Kubernetes context (what the shipped workflow uses), or
- **L40S** — `L40S:1`.

Never route the reconstruct/render path at **H100 / H200 / A100 / B200**: they
have no RT cores. `npa workbench nurec check` fails with an explicit error when
the visible GPU is one of those (`has_rt_cores`). NRE additionally requires
driver **R580+** on Blackwell and >= 24 GB VRAM (48 GB+ recommended). See
`skills/atomic/gpu-selection/SKILL.md`.

## Container Access (read this before debugging a pull failure)

NRE ships only as a closed-source NGC container, so the container *is* the
runtime — there is no source/pip path.

| Repository | With a standard `NGC_API_KEY` |
| --- | --- |
| `nvcr.io/nvidia/nre/nre` | **402 Payment Required** — needs an extra entitlement |
| `nvcr.io/nvidia/nre/nre-ga` | **pullable** — the General Availability channel |
| `nvcr.io/nvidia/nre/nre-tools-ga` | pullable (auxiliary seg/depth data only) |

Use the **`-ga`** repositories. `npa workbench nurec check` reports
`ngc_image: entitlement-required` rather than failing opaquely when a
non-GA reference is configured.

Image startup preflight overrides the vendor entrypoint with `/bin/bash`, as
SkyPilot does, and installs missing SSH, rsync, and service helpers before
checking worker capabilities. NRE's `/app/run` entrypoint cannot forward an
arbitrary shell probe. Prepared results are never cached as image attestations;
they depend on package access from the selected cluster. First-party image
attestations and strict runtime byte probes keep their existing requirements.

The live regression is `test_vendor_image_runtime_bootstrap_live` in
`npa/tests/e2e/test_image_bootstrap_terminal_probe_live.py`. Select the exact
preflighted image/digest, `NPA_E2E_KUBECONTEXT`, `KUBECONFIG`, and any existing
pull Secret names through `NPA_E2E_IMAGE_PULL_SECRETS` (comma separated). It
requires compatible worker capabilities and verified deletion of its probe pod.

```bash
NPA_INTEGRATION_E2E=1 npa/.venv/bin/python -m pytest \
  npa/tests/e2e/test_image_bootstrap_terminal_probe_live.py \
  -k vendor_image_runtime_bootstrap -q
```

## Real Entrypoints

Every stage is a real command; nothing here is a manifest stub.

```bash
npa workbench nurec check       # NGC pullability + HF download rights + RT-core GPU
npa workbench nurec convert-colmap # CPU: official Apache-2.0 COLMAP -> NCore V4
npa workbench nurec fetch       # real NCore V4 shards + derived rig pose edge
npa workbench nurec reconstruct # NRE 3DGUT training -> renderable USDZ + metrics
npa workbench nurec render      # `nre render` novel views (rig offset, not training views)
npa workbench nurec visualize   # reports/sim2real.rrd for the agent's Rerun panel
npa workbench nurec finalize    # reports/final.json aggregate
npa workbench nurec status      # what a run prefix holds, stage by stage
```

| Concern | Implementation |
| --- | --- |
| Pure logic + argv builders | `npa/src/npa/workbench/nurec/nurec.py` |
| COLMAP ingestion + complete V4 verification | `npa/src/npa/workbench/nurec/colmap.py` |
| NCore rig-pose derivation | `npa/src/npa/workbench/nurec/ncore_rig.py` |
| CLI | `npa/src/npa/cli/nurec/__init__.py` |
| SDK | `npa.sdk.workbench.nurec` (`check`, `convert_colmap`, `fetch`, `reconstruct`, `render`, `visualize`, `finalize`, `status`); the framework-free API is re-exported from `npa.workbench.nurec` |
| SkyPilot workflow | `npa/src/npa/workbench/nurec/examples/nurec-reconstruct.yaml` |
| Main declarative workflow | `workflows/main/nurec-reconstruct.yaml` |
| COLMAP source workflow (not yet live validated) | `workflows/testing/nurec-colmap-reconstruct.yaml` |
| Rerun recording | `npa.workflows.data_factory_viz.build_run_rrd` |

## Input Data

Default: **`nvidia/PhysicalAI-NuRec-PPISP`** — ungated, CC-BY-4.0, real
photographic captures of four outdoor object-centric scenes shipped **already in
NCore V4**, which is what NRE consumes. Scene `struktur28`, variant `auto`, is
the small default (59 images across two cameras, ~1.1 GB archive).

The `nvidia/PhysicalAI-Autonomous-Vehicles*` family (raw clips, `-NCore`, and the
pre-built `-NuRec` USDZ scenes) is **gated**: the account that owns `HF_TOKEN`
must accept the NVIDIA AV Dataset License on each dataset page before any token
can pull a byte. `npa workbench nurec check` reports `hf_dataset: gated` for
those until it is accepted — it probes real download authorization, not just
metadata visibility, because a gated repo still answers 200 for
`/api/datasets/<id>`.

### COLMAP source ingestion (not yet live validated)

Use `workflows/testing/nurec-colmap-reconstruct.yaml` for original COLMAP images,
camera calibration/poses and sparse points. Its five states are conversion ->
existing NRE reconstruction -> render -> visualize -> finalize; the workflow
owns the graph, with no separate Python orchestrator. The exact CLI/toolRef
contract is documented in `docs/workbench/guides/nurec-colmap-reconstruct.md`.

- Select the full `struktur28` directory in the PPISP ZIP pinned at
  `2521064a3af6ab1c1caa2ba1b01ddde7eecded69`: 518 images, 3 cameras and 163,453
  sparse source points. The separate `struktur28_auto` directory is not the
  selected capture. Never replace the full source with that smaller variant.
- `--input-path` takes an S3 ZIP or prefix; `--output-path` is the exact portable
  sequence prefix. Feed the same trailing-slash prefix to NRE's `--ncore-uri`.
  Use `--dataset-root struktur28 --rig-mode derive` and `npa_rig` poses.
- Choose a fresh destination prefix for every publication attempt. The permanent
  provider-conditional claim prevents overlapping writers and replacement; after
  interruption, retry into a new prefix rather than deleting or expiring a claim.
- The CPU image runs NVIDIA/ncore
  `59c698d206da92b406a4f72619fce3b3a2c64bfd`, with its pinned MIT
  trueprice/pycolmap reader. Scope the immutable development image override to
  `workbench.nurec.convert_colmap`; no accepted/public availability is implied.
- Preserve virtual per-camera **1 FPS** timestamps as photographic ordering,
  not synchronized capture time. Sparse SfM points are not physical LiDAR.
  Record upstream's near-origin point filtering and any derived rig changes.
- Reopen every image/calibration/pose/point, compare full source counts and
  finite geometry, and retain hashes, source notices and CC-BY-4.0 attribution
  with runtime artifacts. No datasets or weights may be baked.
- Compare the complete camera model and distortion coefficients, including each
  downsampled camera, against source calibration. Matching counts and intrinsics
  alone miss equal-intrinsics cameras with different distortion. Keep the
  drift-checked upstream compatibility patches and post-patch source inventory.
- Verify converted member hashes and the publication claim before the NuRec
  reader opens a sequence. Return only the fresh download generation, never an
  overlay into an old cache: even two valid inventories can select the wrong
  capture after merging. Reject an explicit source without sequence metadata
  instead of falling back to a previously fetched dataset.
- NRE remains proprietary and separately gated. Keep its full native recipe
  (`max_epochs: "0"`) on the explicit RTX PRO 6000 profile; never remap to B200.
- Live matrix: real executing `gpu` case, with complete hash-verified source
  seeding and S3 conversion-member verification. Supply the development digest
  through `NPA_E2E_IMAGE_OVERRIDE_NCORE`. Report actual numeric GPU results
  before changing the pending validation status; older NuRec results do not
  qualify.

## Static Navigation Scene Handoff

For actual calibrated metric RGB-D rather than an existing Gaussian artifact,
use `workflows/testing/rgbd-scan-to-isaac.yaml` and the
`docs/workbench/guides/rgbd-scan-to-isaac.md` contract. Real Open3D TSDF integration
derives colored visual and exact collision surfaces from measured depth/poses;
held-out depth frames gate geometry before the existing USDZ/PhysX handoff.
The manifest uses column-vector optical-camera poses in metric Z-up world,
hashes every pair, and requires disjoint integration/validation membership.
Unknown space stays unknown. CPU reconstruction uses the existing SONIC Open3D
runtime without Isaac bootstrap, followed by Content Agents OpenUSD assembly and
RTX Isaac physics. It needs no new weights or image. Runtime/data provenance and
private capture attribution stay with the sealed artifacts. GPU, industrial
scene, robot clearance, and policy learning qualification remain separate.

Use `workflows/testing/scan-to-isaac-navigation.yaml` to combine an existing
NuRec visual USD/USDZ with independently supplied, triangulated collision USD.
The workflow reuses reconstruction outputs; it does not duplicate NCore or
COLMAP ingress. See `docs/workbench/guides/scan-to-isaac-navigation.md` for the
input manifest, module adapters, calibration convention, and live test.

- Require explicit source units, coordinate registration, capture-provenance
  digest, collision geometry, and expected physics ray intersections. Gaussian
  splats alone are visual assets, never collision surfaces.
- Audit raw Sdf layers before composition, recursively including nested USDZ,
  inactive prims and unselected variants. Reject external dependencies,
  OmniScripting/OmniGraph declarations, time samples and value clips; preserve
  ordinary shaders and contained neural volume payloads.
- Export portable USDZ with static triangle colliders and geometry/input hashes.
  Publish only to fresh prefixes using permanent conditional claims, immutable
  member writes, and a verified completion seal. Retry failures in a new prefix.
- Run the native PhysX probe through the Isaac interpreter on RTX infrastructure.
  Supply an immutable Isaac image declaration; reports bind both scene and exact
  assembly-provenance hashes, actual Isaac/PhysX versions, and the declaration's
  explicitly unverified image-attestation scope.
- CPU tests prove assembly and containment. Native GPU acceptance, NuRec
  rendering, metric RGB-D GPU execution, robot clearance, and
  navigation-policy quality remain unverified. The live test requires operator
  assets and must run before claiming physics acceptance.

## The rig -> world Pose Edge (the thing that breaks first)

NRE's NCore data source requires a `("rig", "world")` pose-graph edge:

```text
# nre/datasets/ncore.py, nre-ga 26.04
# TODO: frame-pose only data might fail here as there are no rig poses ...
rig_world_edge = unpack_optional(
    sequence_loader.pose_graph.get_edge("rig", "world"),
    msg="Rig-to-world poses are currently required to determine scene extend")
```

Object-centric captures have no vehicle rig, and NVIDIA's own COLMAP -> NCore
converter stores per-camera `<camera> -> world` poses with **no rig node**
(`tools/data_converter/colmap/converter.py`: `reference_frame = "world"`). So
every COLMAP-derived NCore sequence — including PPISP's own export — fails to
load with:

```text
ValueError
Rig-to-world poses are currently required to determine scene extend
```

Some captures carry camera-to-world transforms only in the per-frame
`T_sensor_worlds` generic-data field. NPA reads that representation when no
dynamic camera pose edges exist; dynamic edges remain preferred.

`npa workbench nurec fetch` fixes this by default. For a single-camera capture
the rig **is** the camera, so `rig -> world` is exactly that camera's pose
trajectory. `ncore_rig.derive_rig_poses` writes one small extra component store
plus a new sequence meta-file that symlinks the original shards, so the source
data is never modified:

- the derived poses live in their own component instance **`npa_rig`**, not
  `default` — `open_component_readers` asserts instance names are unique across a
  sequence's stores, and re-using `default` raises
  `Component instance default encountered multiple times`;
- selecting a poses group **replaces** the pose set rather than merging, so the
  derived component carries a complete copy of the original edges plus the rig
  edge;
- the frame-pose fallback validates every camera frame before writing: timestamps
  must be strictly increasing and each `T_sensor_worlds` value must be a finite
  4×4 transform. Missing, malformed, duplicate, inconsistent, or non-finite data
  fails closed without publishing a partial derived sequence;
- the sidecar records the selected camera and pose source. An explicit reference
  camera wins; otherwise the longest trajectory wins with camera ID as a stable
  tie-breaker;
- `reconstruct` then passes `dataset.poses_component_group=npa_rig`.

The fallback makes exactly one geometric claim: `rig` is the selected camera.
It does not infer multi-camera extrinsics, and compatibility tests do not prove
reconstruction quality. Local regression coverage establishes conversion
compatibility only; recording live workflow readiness still requires a decoded
NRE reconstruction on applicable RT-core hardware and inspected retained
artifacts.

Pass `--no-derive-rig` for AV-style sequences that already ship a rig edge (the
derivation short-circuits with `already_present: true` anyway), and
`--reference-camera <id>` to pin which camera becomes the rig.

## Recipe Selection

The container resolves `--config-name` against its own `configs/` tree. Pick by
capture shape:

| Capture | Recipe |
| --- | --- |
| Object-centric / static / camera-only (PPISP, COLMAP-style) | `configs/experimental/3dgut/3dgut_colmap.yaml` (**the default**) |
| Waymo Open Dataset | `configs/apps/AV/Waymo/3dgut_dynamic*.yaml` |
| PhysicalAI Autonomous Vehicles (Hyperion-8.1) | `configs/apps/prod/Hyperion-8.1/car2sim_6cam.yaml` |
| PandaSet / NV / Tesla / Alpasim | `configs/apps/AV/{PandaSet,NV,Tesla}/...`, `configs/apps/Alpasim/...` |

The default recipe already composes `options/artifact: default` (which is what
sets `checkpoint.artifact.enabled`, i.e. the renderable USDZ), MCMC
densification, SfM-point-cloud initialization, and disables difix/mesh/ground.
For a derived-rig capture with multiple selected cameras, NPA replaces only the
background initializer through the native
`model/gaussians/initialization@model.layers.background.initialization=accumulated_point_cloud`
config group. The native SfM initializer asserts one camera on the entire data
source; restricting an initialization-only camera list cannot fix it. Keep all
discovered cameras, export all verified NCore world XYZ and uint8 RGB points to
the reconstruction output's `initialization/ncore-sfm.ply`, and let NRE apply
its world-to-NRE transform. Match the complete point count and disable optional
random near/far seed points to preserve source-point-only initialization.
Never mutate the conversion generation or change the native training budget.
Single-camera selections retain native SfM; custom recipes and explicit
initialization overrides remain the caller's responsibility. The dry run shows
the planned native arguments without writing initialization artifacts.

Retain `initialization/ncore-sfm.json` with the source/conversion and PLY hashes,
selected cameras, count, and recipe/image identity. Native `parsed.yaml` records
effective configuration, while USDZ `data_info.json` is input sequence metadata.
Available frames and exported ground truth do not establish split membership or
actual sampled training frames. Require separate native split/sampler evidence
before claiming that every source image participated in training. The COLMAP
multi-camera path still requires a real full RTX run after integration; offline
PLY tests do not validate native reconstruction quality.

Enumerate what a given release actually ships with:

```bash
find /app/run.runfiles/_main/configs -name '*.yaml' | sort
/app/run --help          # sub-command inventory
/app/run render --help   # authoritative flag surface
```

## Novel Views vs Training Views

`nre render` defaults to `--replicate-training-views`, which re-renders views the
model was trained on. That is **not** a novel view. The tool therefore emits
`--no-replicate-training-views` plus a rig offset by default:

- `--rig-translation-offset` and `--rig-rotation-offset` are `FLOAT...` (three
  values) upstream; the CLI accepts one `"x,y,z"` string and expands it;
- a zero offset with no custom trajectory is rejected rather than silently
  producing training views;
- `--renderer default` (the artifact's own trained renderer) is the default.
  `nrend` is faster but needs the nrend model dictionary embedded in the USDZ,
  which the object-centric recipe disables.

## Artifact Layout

One S3 run prefix per run, so the agent's artifact browser picks it up:

```text
s3://<bucket>/<prefix>/neural-reconstruction/<run_id>/
  ncore/manifest.json                    # dataset, scene, sensors, rig derivation
  input/camera_images/<camera>/*.jpg     # real capture frames (export-ncore-benchmark-gt)
  reconstruction/last.usdz               # renderable Gaussian scene
  reconstruction/parsed.yaml
  reconstruction/metrics.yaml            # test/psnr, test/ssim, test/lpips
  reconstruction/val/...                 # NRE validation renders + videos
  novel_views/<camera>/*.png             # rig-offset novel views
  novel_views/<camera>.mp4
  reports/final.json
  reports/sim2real.rrd                   # preferred artifact for the Rerun panel
```

`<run_id>` must be a single safe segment embedding the submit timestamp, e.g.
`neural-reconstruction-struktur28-20260731t050118z`, so the run picker dates the
run by when it **started** (`npa.workflows.artifacts._run_started_at`).

`.usdz` classifies as `download`, which is correct: no browser renders USDZ and
the agent has no USDZ viewer. Viewability comes from the `.rrd`, `.png`, `.mp4`
and `.json`.

For a COLMAP-derived run, visualization fails closed unless the source
attribution, conversion report, and rig-derivation sidecar are all readable.
Detection uses both the pre-download S3 inventory and local files, so a partial
subtree download cannot produce an RRD backed only by novel-view media. This
strict trio applies only after a COLMAP lineage marker is present; optional
data-factory subtrees and legacy preconverted-NCore runs remain best effort.

## Which Capability Answers This?

Adapted from the NVIDIA router skill's picker table, re-pointed at what this
repo actually implements. "Upstream" means the workbench has no verb for it: read
the named sibling skill at <https://github.com/NVIDIA/nurec-skills> and run it
yourself; do not invent a workbench command for it.

| I want to... | Where |
| --- | --- |
| Check NGC/HF access and that the GPU has RT cores, before pulling 14 GB | `npa workbench nurec check` |
| Download a published NVIDIA NuRec/PhysicalAI capture in NCore V4 | `npa workbench nurec fetch` |
| Train a reconstruction from an NCore clip and get a USDZ | `npa workbench nurec reconstruct` |
| Render novel views along a shifted rig trajectory | `npa workbench nurec render` |
| Get a Rerun recording the NPA agent will display | `npa workbench nurec visualize` |
| Run all of the above on a GPU as one pipeline | `workflows/main/nurec-reconstruct.yaml` |
| Measure PSNR / SSIM / LPIPS | Already emitted -- `reconstruction/metrics.yaml`, and `gaussians/summary` in the `.rrd` |
| Convert a COLMAP reconstruction to NCore V4 | `npa workbench nurec convert-colmap` on CPU; not yet live validated |
| Convert a non-COLMAP recording (drone, RGB-D, ROS 2 bag, ScanNet++) | Upstream `ncore`; these input-specific converters are not wired here |
| Serve frames to CARLA / Isaac Sim / a custom simulator | Upstream `nre` (`serve-grpc`) -- not wired, see Limitations |
| Render LiDAR sweeps from a USDZ | Upstream `nre` (`render-grpc --lidar`) -- not wired |
| Extract individual 3D objects (cars, pedestrians) from a clip | Upstream `asset-harvester` -- not wired |
| Clean up ghosting / floaters / flicker in rendered frames | Upstream `nurec-fixer` (DiffusionHarmonizer), or NRE's inline `--enable-difix` -- neither wired |
| Generate segmentation / depth / ego-mask auxiliary inputs | Upstream `nre` via the `nre-tools` image -- not wired, see Limitations |
| Package CAD or source meshes for simulation | Not NuRec at all -- that is SimReady, a different pipeline |

## Easy Mix-Ups

Adapted from the router skill's `references/mix-ups.md`; the last row is
workbench-specific.

- **NuRec vs NRE.** NuRec is the product, NRE ("Neural Reconstruction Engine") is
  the engine that trains and renders. Used interchangeably in most docs.
- **`ncore` then `nre`, never instead of.** NCore V4 is the input format; NRE
  reads it. They run in order. If NRE says a clip "is not valid NCore V4", the
  conversion step is missing, not a training bug.
- **3DGUT vs 3DGRT.** Two Gaussian-splatting flavours inside NRE. The Hydra
  recipe picks one; you should not normally set it by hand.
- **`PhysicalAI-Autonomous-Vehicles-NuRec` vs `Cosmos-Drive-Dreams`.** Both AV
  datasets on Hugging Face and easy to confuse. The former is *real* driving
  footage under the gated AV license; the latter is *synthetic* weather-augmented
  video under CC-BY-4.0.
- **NRE's inline `--enable-difix` vs the standalone `nurec-fixer`.** Upstream
  documents `--enable-difix` as a built-in cleanup pass during rendering, while
  `nurec-fixer` wraps the public DiffusionHarmonizer release for frames already
  rendered. Neither is wired into a workbench verb, and the flag has not been
  exercised against `nre-ga 26.04` here -- treat it as upstream-documented, not
  as a verified workbench feature.
- **A missing `rig -> world` edge is not a corrupt download.** The most common
  first failure is a pose-graph gap this workflow derives for you. See
  [The rig -> world Pose Edge](#the-rig---world-pose-edge-the-thing-that-breaks-first).

## Troubleshooting

Rows marked (upstream) are adapted from the router skill's troubleshooting
table; the rest were hit for real while landing this capability.

| Symptom | Cause | Fix |
| --- | --- | --- |
| `402 Payment Required` pulling `nvcr.io/nvidia/nre/nre` | That repo needs an extra entitlement (upstream: `denied: requested access ...`) | Use the `-ga` channel, `nvcr.io/nvidia/nre/nre-ga:26.04`. `nurec check` reports `entitlement-required` |
| `401`/`403` on a gated `nvidia/PhysicalAI-Autonomous-Vehicles*` override (upstream) | Gated dataset access is absent, or `HF_TOKEN` lacks `read` | Accept/request access on Hugging Face as the token owner, then re-run `nurec check`; the default `PhysicalAI-NuRec-PPISP` remains anonymous |
| NRE will not load a clip: "not valid NCore V4" (upstream) | The recording was never converted | For COLMAP, use `nurec convert-colmap`; other source formats still need upstream conversion |
| `KeyError: ('rig', 'world')` / no scene extent | The clip has camera poses but no rig edge -- NVIDIA's own COLMAP converter omits it | Automatic: `reconstruct` derives it. See the pose-edge section |
| `Requested lidars not present in the data: dummy_lidar` | The recipe ships placeholder sensor ids | Automatic: the sequence's real ids are adopted |
| `Requested cameras not present: camera_front_wide_120fov` | Same, for AV camera names | Automatic: same adoption path |
| `Only one camera sensor is currently supported` | SfM point-cloud init is single-camera | Automatic: trains on the recorded reference camera and warns which cameras were dropped |
| Cluster never finishes provisioning, `sudo: command not found` | The image ships no `sudo`; SkyPilot's K8s bootstrap calls it unconditionally | Automatic: `pod_config` initContainer installs a shim |
| OOM / bus error early in training | `/dev/shm` defaults to 64 MB | Automatic: 64 Gi `emptyDir{medium: Memory}` |
| USDZ looks like an early preview | `<run>/artifacts/<step>.usdz`, first-alphabetical picks step 1000 | Automatic: `latest_usdz()` picks the newest by step |
| Renders look identical to the input frames | `nre render` defaults to `--replicate-training-views` | Automatic: the negation plus a non-zero rig offset is always emitted; a zero offset is rejected |
| Output files owned by `root` after a local `docker run` (upstream) | `-u $(id -u):$(id -g)` was omitted | `sudo chown -R "$(id -u):$(id -g)" <dir>`, and pass `-u` next time. Not an issue in-pod, which runs as root by design |
| Ghosting / floaters / flicker in rendered frames (upstream) | No cleanup pass | Upstream `nurec-fixer`, or NRE's inline `--enable-difix`. Neither is wired here |
| A stage runs but publishes nothing to S3 | The stage ran in its own pod and wrote only to `/tmp` | Pass the handoff URIs. Every declarative stage is a separate pod |

## Verifying Secrets Safely

From the router skill's `references/secrets-handling.md`. Never interpolate a
token into an ad-hoc shell check. In particular this common line **prints the
token**:

```bash
echo "HF_TOKEN: ${HF_TOKEN:+yes}${HF_TOKEN:-no}"   # WRONG: emits yes<token>
```

`${VAR:-no}` only falls back when the variable is *empty*, so a set token is
echoed straight into the log. If you suspect one was printed, rotate it at
<https://huggingface.co/settings/tokens> or
<https://org.ngc.nvidia.com/setup/api-key>.

Safe checks:

```bash
hf auth whoami
[ -n "${HF_TOKEN:-}" ]    && echo "HF_TOKEN length=${#HF_TOKEN}"       || echo "HF_TOKEN unset"
[ -n "${NGC_API_KEY:-}" ] && echo "NGC_API_KEY length=${#NGC_API_KEY}" || echo "NGC_API_KEY unset"
```

Better, because it probes real *download authorization* rather than mere
visibility (a gated HF repo still answers 200 on `/api/datasets/<id>`):

```bash
npa workbench nurec check --json
```

Every `nurec` failure payload is redacted before it is rendered, so a token
cannot reach a log or a `--json` body.

## Running The Workflow

```bash
# 1. Confirm access and GPU suitability first (seconds, no image pull).
npa workbench nurec check --require-gpu --output json

# 2. Submit the real GPU workflow. The wrapper substitutes ${...}; SkyPilot 0.12.2
#    does not interpolate them itself.
RUN_ID="neural-reconstruction-struktur28-$(date -u +%Y%m%dt%H%M%S)z"
npa workbench workflow submit npa/src/npa/workbench/nurec/examples/nurec-reconstruct.yaml \
  --run-id "$RUN_ID" \
  --infra k8s/<rt-core-context> \
  --var NPA_NUREC_IMAGE=nvcr.io/nvidia/nre/nre-ga:26.04 \
  --var NPA_NUREC_RUN_ID="$RUN_ID" \
  --var NPA_NUREC_RUN_URI="s3://<bucket>/<prefix>/neural-reconstruction/$RUN_ID" \
  --var NPA_SRC_S3_URI="s3://<bucket>/npa-src/<tag>" \
  --var AWS_ENDPOINT_URL="<s3-endpoint>" \
  --var AWS_ACCESS_KEY_ID="<key>" --var AWS_SECRET_ACCESS_KEY="<secret>" \
  --var NGC_API_KEY="<ngc>"

# 3. Watch it.
npa workbench workflow status "$RUN_ID" --watch
npa workbench workflow logs "$RUN_ID" --follow

# 4. Inspect the run tree.
npa workbench nurec status --run-uri "s3://<bucket>/<prefix>/neural-reconstruction/$RUN_ID/" --output json
```

Budget roughly: image pull (~14 GB compressed) on a cold node, a few minutes of
setup, ~1 min fetch, ~20 min for 30k 3DGUT steps on one RTX PRO 6000, then
rendering, the `.rrd`, and the upload.

## Two Container Quirks The Workflow Already Handles

1. **No `sudo`.** SkyPilot's runtime setup calls bare `sudo` when it writes the
   nofile limits and `/etc/fuse.conf`
   (`sky/templates/kubernetes-ray.yml.j2`). The NRE image has no `sudo`, so that
   step exits 127 and the cluster never becomes usable even though provisioning
   otherwise succeeds. The workflow's `pod_config` runs an initContainer that
   drops a shim which execs its arguments into `/usr/local/sbin` (on the image's
   PATH, and empty) — equivalent to SkyPilot's own `alias sudo=""` root path.
2. **64 MB `/dev/shm`.** NRE wants tens of GB; the workflow mounts a 64 Gi
   `emptyDir{medium: Memory}` at `/dev/shm`.

Also: the image ships no `unzip`, `git`, or `ffmpeg`. Extraction uses stdlib
`zipfile`; `setup` installs `ffmpeg` for `render --export-video`.

## Artifact Layout Surprise

Upstream docs describe `<run>/usd-out/last.usdz`. nre-ga 26.04 actually writes
one artifact per checkpoint as `<run>/artifacts/<step>.usdz` plus
`<run>/artifacts/last.usdz`. `latest_usdz()` handles both and picks the newest,
because taking the first alphabetical match would ship the 1000-step preview
instead of the trained scene.

## Verify

### Real-GPU (live) verification

The committed live e2e provisions a real RT-core GPU and asserts the whole
capability against S3. It skips unless the environment is supplied:

```bash
NPA_INTEGRATION_E2E=1 \
NPA_NUREC_E2E_BUCKET=<bucket> \
NPA_NUREC_E2E_NPA_SRC_S3_URI=s3://<bucket>/npa-src/<tag> \
NPA_NUREC_E2E_INFRA=k8s/<rt-core-context> \
NPA_NUREC_E2E_PREFIX=checkpoints \
  npa/.venv/bin/python -m pytest \
    npa/tests/e2e/test_nurec_reconstruct_live_e2e.py -v -s
```

Also needs `AWS_ENDPOINT_URL`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and
`NGC_API_KEY` exported. `HF_TOKEN` is optional for the public PPISP default and
required only for a gated/private dataset override. Budget ~30 min end to end (image pull,
30k 3DGUT steps, render, upload); raise `NPA_NUREC_E2E_MAX_WAIT_SECONDS` (default
7200) for a slower cluster. Measured: **1 passed in 28m47s**, with the underlying
SkyPilot job taking 26m10s and reporting `test/psnr 31.19`, `test/ssim 0.833`,
`test/lpips 0.267`.

### Offline verification

```bash
npa/.venv/bin/python -m pytest \
  npa/tests/workbench/test_nurec.py \
  npa/tests/workbench/test_nurec_access.py \
  npa/tests/workflows/test_nurec_viz.py \
  npa/tests/workflows/test_nurec_artifacts.py \
  npa/tests/orchestration/npa_workflow/test_real_components.py -q

npa/.venv/bin/python -m ruff check npa/src/npa/workbench/nurec npa/src/npa/cli/nurec
npa workbench nurec reconstruct --help
```

GREEN when: the workflow name equals its file stem, the accelerator is an RT-core
GPU with no H100/H200 reference, every stage in the YAML is a real
`npa workbench nurec` call, the `.rrd` carries the `novel_view` /
`reconstruction` / `gaussians` entities, and a synthetic run prefix is listed by
`list_runs` with `has_viewable=True` and `preferred == reports/sim2real.rrd`.

## Limitations

- **Linux x86_64 + NVIDIA RT-core GPU only.** aarch64 is unsupported upstream.
- **NGC entitlement.** Only the `-ga` repositories are pullable with a standard
  key.
- **Multi-GPU with `dataset.aux_data=false` is a known upstream crash.** The
  camera-only default therefore stays single-GPU; raise `--world-size` only with
  aux data enabled.
- **No gRPC sensor-sim path yet.** `serve-grpc` / `render-grpc` (LiDAR sweeps,
  actor editing, simulator loops) exist in the container but are not wired into
  a workbench verb.
- **`nre-tools` auxiliary data (segmentation, depth, DINOv2) is not wired in.**
  It is a second ~22 GB image and the object-centric default does not need it.
- **No object harvesting or actor editing.** Upstream `asset-harvester` extracts
  per-object `.ply` splats and `nre export-external-assets` packages them; the
  workbench does neither. Asset Harvester always runs *before* USDZ packaging.
- **No frame-cleanup pass.** Upstream `nurec-fixer` (DiffusionHarmonizer) and
  NRE's inline `--enable-difix` are both unwired and unverified here.
- **COLMAP ingestion is not yet live validated.** Other source-format converters
  and novel sensor-rig integrations remain upstream `ncore` work.