Health Preflight

---
name: health-preflight
description: Use before any deploy, image build, provisioning, or GPU submit to prove service credentials with `npa workbench health preflight`, request `--checks nebius` before provisioning, and verify gated-model access with `npa workbench health access`.
---

# Health preflight (the doctor before you spend)

There is no `npa doctor`. The readiness surface is `npa workbench health`, and it
has exactly two public commands. Run them before anything that costs GPU time or
provisions cloud resources.

```bash
npa workbench health preflight --json
npa workbench health preflight --checks nebius --json
npa workbench health access --json
```

All three exit non-zero on any FAIL. That is the point: they are gates, not reports.
Use `--warn-only` only when you deliberately want the report without the gate
(for example while collecting a status snapshot), never to make a red run green.

## What each command answers

`preflight` answers *"do I hold the credentials nearly every workbench tool
needs?"* — one PASS/WARN/FAIL/SKIP row per check over Hugging Face, NGC, S3, and
Token Factory. Request the optional `nebius` check before provisioning:

```bash
npa workbench health preflight --checks all
npa workbench health preflight --checks nebius --json
npa workbench health preflight --checks s3,token_factory
npa workbench health preflight --project <alias> --checks s3,nebius
npa workbench health preflight --checks all --offline  # presence only; Nebius is SKIP
```

Valid `--checks` values are `all`, `hf`, `ngc`, `s3`, `token_factory`,
`encord`, `nebius`. The default remains `hf,ngc,s3,token_factory`, so hosted-inference
work does not require a Nebius Cloud profile. Explicit `all` includes `encord` and `nebius`.
Empty selections and unknown check names are errors, including unknown names
combined with `all`. Repeated names run once.

Select `--project <alias>` (or `-p`) when checking a configured project's S3
storage. This reads the project's bucket, endpoint, and credential pair through
the project storage resolver, without adopting host credential files or shell S3 settings, or
changing saved configuration. A complete exact-project record wins over ambient
S3 settings. Unknown projects and missing or deselected project storage produce
a JSON-compatible FAIL and a nonzero exit, including offline; `--warn-only`
preserves that FAIL while returning zero. The flag scopes only the S3 check;
other checks and Nebius auth profile selection are unchanged.

The S3 probe issues one `ListObjectsV2` request with `MaxKeys=1`. An empty bucket
or prefix is valid. It discards object names and continuation tokens instead of
enumerating checkpoint directories, so a large dataset does not increase the
number of requests. Listing permission still does not prove write access.

Online `nebius` runs the selected Nebius CLI profile through `iam whoami` and
`iam get-access-token` with browser launch and update checks disabled. It
discards both commands' output and removes ambient `NEBIUS_IAM_TOKEN` and
`NEBIUS_IAM_TOKEN_FILE` values so a stale token cannot mask profile readiness.
PASS requires both calls to succeed. In offline mode it returns SKIP because a
local profile file is not proof of usable authentication.

A failed Nebius identity probe does not establish that a token expired. Profile
configuration may be missing, or access or connectivity may have failed. For
diagnosis or authorized human login on an operator VM, see
`skills/atomic/vm-nebius-auth/SKILL.md`. A readiness-only request does not
authorize login or credential changes. Authentication PASS also leaves project
permissions, quota, and capacity unverified.

Profile selection uses `NPA_NEBIUS_PROFILE`, then `NEBIUS_PROFILE`, then the
CLI's active/default profile. The project selector does not choose an auth
profile. PASS proves authentication; permission to provision a particular
resource still depends on the selected identity's access to that project.

Online `hf` authenticates against Hugging Face `whoami-v2`; public repository
metadata is not sufficient. Online `ngc` performs a registry token exchange;
that proves the key, not entitlement to every NGC artifact. Online `encord`
authenticates and performs the cheapest read-only storage-folder listing.
`access` performs the capability-specific repository/artifact probe.

`access` answers the different and more specific question *"is my token actually
entitled to fetch bytes from the gated assets this capability pulls?"* It probes
a representative payload path at the exact catalog revision with HEAD or a
one-byte Range request. Metadata access is not proof. Holding an HF token is not
the same as having account entitlement, and `Ready` is technical fetch evidence,
not proof of legal acceptance.

```bash
npa workbench health access --capability all
npa workbench health access --capability cosmos3,groot
```

Capabilities: `all`, `cosmos`, `cosmos3`, `cosmos3-serving`, `groot`, `lerobot`,
`nurec`, `paidf`, `sim2real`, `sonic`, `token_factory`, `vlm_eval`.

For anything still gated, `access` prints the exact "Agree and access
repository" URL. **Hugging Face gated licenses must be accepted interactively on
the model page.** There is no API that accepts them for you, so no amount of
retrying or re-tokenizing will clear a gate. Open the printed URL, accept, then
re-run. `scripts/accept-model-access.sh` collects the URLs for a batch.

## Ordering: preflight before provisioning, not after

The recurring cold-start failure this prevents is a mid-run stop after you have
already paid for a cluster. Run the checks in this order:

1. `npa configure --show`: confirm the project stanza, bucket, and endpoint you
   think you are using are the ones on disk.
2. `npa workbench health preflight` — credentials exist and authenticate.
3. `npa workbench health preflight --checks nebius`: the selected Nebius CLI
   profile can resolve identity and mint an IAM token before provisioning.
4. `npa workbench health access --capability <the one you will run>`: gated
   model entitlements for that capability only; `all` is slower and reports
   failures you do not care about today.
5. Only then `npa provision-if-absent`, `npa cluster up`, or
   `npa workbench workflow submit`.

Image pullability is a *separate* gate that health does not cover; see
`npa workbench workflow preflight-images <spec.yaml>` in
`skills/atomic/submit-workflow/SKILL.md`. A green health report with an
unpullable image still hangs in `ImagePullBackOff`.

**A green `s3` row does not mean the submit is execution-ready.** The generic
`s3` check lists its configured bucket. Submit resolves one effective target,
verifies provider project/tenant/region and exact bucket ownership before any
write, then writes and reads a unique object in each actual workload output,
ledger and source-staging prefix using the executing credential pair. It also
checks the exact Kubernetes context and requested GPU product/shape, including
single-node requests. An explicit destination never falls back to another
writable bucket. A mismatch or unknown ownership blocks creation, even with
`--skip-preflight`. A denied task prefix can therefore follow a green generic
health check:

```
Error: execution preflight storage_access: exact output prefix write failed (authorization)
```

Use submit's execution diagnosis. Reconcile the selected project and the exact
endpoint/bucket/prefix before changing IAM. `--s3-bucket` / `--s3-prefix` bind
the actual spec and ledger; `--s3-endpoint` binds both probe and worker endpoint.
S3 keys are selected as a pair from one source, with non-secret provenance;
an incomplete explicit pair cannot borrow a saved principal's other key.

Without `--project`, credential preflight resolves its bucket from `NPA_CHECKPOINT_BUCKET`, then
`NEBIUS_S3_BUCKET`, then saved credentials. Setting only `NPA_S3_BUCKET` changes
the workflow destination but does not select the credential preflight bucket.
For configured project storage, prefer `--project <alias>`. When checking an
explicitly authorized destination through environment credentials, set
`NPA_CHECKPOINT_BUCKET` to an unsigned `s3://` URI for that same bucket in the
private process environment and use the matching endpoint and credentials.
The list probe requires a URI, so a bare bucket name fails before S3 access.
Verify bucket ownership first;
do not change shared configuration or provision storage to repair a stale
default selection.

## Persisting credentials you already hold

If the tokens are in your environment but not in `~/.npa/credentials.yaml`,
persist them instead of re-exporting them in every shell:

```bash
npa workbench health access --save-env-credentials
npa configure --save-env-credentials
```

Both perform an atomic `0600` write and never print the values. This is the fix
for "it worked in my shell but the submit could not resolve the secret": submit
resolves each requested secret from the explicit process environment first, then
the selected project's configured NPA credentials.

## Sim2real preflight check

`npa workbench health sim2real` is specific to the 14-stage Sim2Real graph
rather than general readiness. It adds cluster-shaped checks (`config`,
`coherence`, `s3`, `registry`, `tokens`, `cluster`) including schedulable GPU
count and kube-context pinning:

```bash
npa workbench health sim2real --checks all --json
```

Use it from `skills/workflows/sim2real-operate/SKILL.md`, not as the generic
preflight. Its cluster check counts schedulable `nvidia.com/gpu` but does not yet
match the *requested GPU product* against available node products, so it can pass
on a cluster that has GPUs of the wrong kind for your spec. Confirm the product
with `npa workbench workflow gpus --cluster <name>`.

## Gotchas

- **`--offline` proves presence, not validity.** It skips live provider probes.
  Existing credentials may pass presence checks, while `nebius` returns SKIP
  because profile configuration is not authentication proof. Use offline only
  where there is genuinely no egress.
- **A SKIP is not a PASS.** Checks skip when the corresponding capability is not
  configured. If you are about to run a Cosmos stage and the NGC row says SKIP,
  you have not verified anything about NGC.
- **`preflight` does not check Kubernetes or SkyPilot.** Cluster readiness is
  `npa skypilot verify --cluster <exact-context>` and `npa cluster status`;
  registry pullability is `workflow preflight-images`, so run each command to
  verify its separate dependency.
- **Stale `NEBIUS_IAM_TOKEN` defeats provider calls even when health is green.**
  The Nebius provider prefers an ambient, often expired token over the fresh CLI
  token, so `unset NEBIUS_IAM_TOKEN NPA_NEBIUS_IAM_TOKEN` before provisioning
  or submitting.

## Verify

```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q
```

With an authenticated operator profile, exercise the real CLI's successful,
missing-profile, and offline paths (including stale ambient token scrubbing):

```bash
NPA_INTEGRATION_E2E=1 npa/.venv/bin/python -m pytest \
  npa/tests/e2e/test_nebius_auth_preflight.py -q
```

For selected-project S3 coverage, use an explicitly configured disposable test
project. The live tests use private configuration copies and clean up their
unique fixture objects:

```bash
NPA_INTEGRATION_E2E=1 NPA_E2E_PROJECT=<alias> npa/.venv/bin/python -m pytest \
  npa/tests/e2e/test_health_project_preflight_live.py -q
```