Health Preflight

---
name: health-preflight
description: Use before any deploy, image build, provisioning, or GPU submit to prove service credentials with `npa workbench health preflight`, request `--checks nebius` before provisioning, and verify gated-model access with `npa workbench health access`.
---

# Health preflight (the doctor before you spend)

There is no `npa doctor`. The readiness surface is `npa workbench health`, and it
has exactly two public commands. Run them before anything that costs GPU time or
provisions cloud resources.

```bash
npa workbench health preflight --json
npa workbench health preflight --checks nebius --json
npa workbench health access --json
```

All three exit non-zero on any FAIL. That is the point: they are gates, not reports.
Use `--warn-only` only when you deliberately want the report without the gate
(for example while collecting a status snapshot), never to make a red run green.

## What each command answers

`preflight` answers *"do I hold the credentials nearly every workbench tool
needs?"* — one PASS/WARN/FAIL/SKIP row per check over Hugging Face, NGC, S3, and
Token Factory. Request the optional `nebius` check before provisioning:

```bash
npa workbench health preflight --checks all
npa workbench health preflight --checks nebius --json
npa workbench health preflight --checks s3,token_factory
npa workbench health preflight --checks all --offline  # presence only; Nebius is SKIP
```

Valid `--checks` values are `all`, `hf`, `ngc`, `s3`, `token_factory`,
`nebius`. The default remains `hf,ngc,s3,token_factory`, so hosted-inference
work does not require a Nebius Cloud profile. Explicit `all` includes `nebius`.
Empty selections and unknown check names are errors, including unknown names
combined with `all`. Repeated names run once.

Online `nebius` runs the selected Nebius CLI profile through `iam whoami` and
`iam get-access-token` with browser launch and update checks disabled. It
discards both commands' output and removes ambient `NEBIUS_IAM_TOKEN` and
`NEBIUS_IAM_TOKEN_FILE` values so a stale token cannot mask profile readiness.
PASS requires both calls to succeed. In offline mode it returns SKIP because a
local profile file is not proof of usable authentication.

A failed Nebius identity probe does not establish that a token expired. Profile
configuration may be missing, or access or connectivity may have failed. For
diagnosis or authorized human login on an operator VM, see
`skills/atomic/vm-nebius-auth/SKILL.md`. A readiness-only request does not
authorize login or credential changes. Authentication PASS also leaves project
permissions, quota, and capacity unverified.

Profile selection uses `NPA_NEBIUS_PROFILE`, then `NEBIUS_PROFILE`, then the
CLI's active/default profile. The project selector does not choose an auth
profile. PASS proves authentication; permission to provision a particular
resource still depends on the selected identity's access to that project.

Online `hf` authenticates against Hugging Face `whoami-v2`; public repository
metadata is not sufficient. Online `ngc` performs a registry token exchange;
that proves the key, not entitlement to every NGC artifact. `access` performs
the capability-specific repository/artifact probe.

`access` answers the different and more specific question *"is my token actually
entitled to fetch bytes from the gated assets this capability pulls?"* It probes
a representative payload path at the exact catalog revision with HEAD or a
one-byte Range request. Metadata access is not proof. Holding an HF token is not
the same as having account entitlement, and `Ready` is technical fetch evidence,
not proof of legal acceptance.

```bash
npa workbench health access --capability all
npa workbench health access --capability cosmos3,groot
```

Capabilities: `all`, `cosmos`, `cosmos3`, `cosmos3-serving`, `groot`, `lerobot`,
`nurec`, `paidf`, `sim2real`, `sonic`, `token_factory`, `vlm_eval`.

For anything still gated, `access` prints the exact "Agree and access
repository" URL. **Hugging Face gated licenses must be accepted interactively on
the model page** — there is no API that accepts them for you, so no amount of
retrying or re-tokenizing will clear a gate. Open the printed URL, accept, then
re-run. `scripts/accept-model-access.sh` collects the URLs for a batch.

## Ordering: preflight before provisioning, not after

The recurring cold-start failure this prevents is a mid-run stop after you have
already paid for a cluster. Run the checks in this order:

1. `npa configure --show` — confirm the project stanza, bucket, and endpoint you
   think you are using are the ones on disk.
2. `npa workbench health preflight` — credentials exist and authenticate.
3. `npa workbench health preflight --checks nebius`: the selected Nebius CLI
   profile can resolve identity and mint an IAM token before provisioning.
4. `npa workbench health access --capability <the one you will run>`: gated
   model entitlements for that capability only; `all` is slower and reports
   failures you do not care about today.
5. Only then `npa provision-if-absent`, `npa cluster up`, or
   `npa workbench workflow submit`.

Image pullability is a *separate* gate that health does not cover; see
`npa workbench workflow preflight-images <spec.yaml>` in
`skills/atomic/submit-workflow/SKILL.md`. A green health report with an
unpullable image still hangs in `ImagePullBackOff`.

**A green `s3` row does not mean the submit is execution-ready.** The generic
`s3` check lists its configured bucket. Submit resolves one effective target,
verifies provider project/tenant/region and exact bucket ownership before any
write, then writes and reads a unique object in each actual workload output,
ledger and source-staging prefix using the executing credential pair. It also
checks the exact Kubernetes context and requested GPU product/shape, including
single-node requests. An explicit destination never falls back to another
writable bucket. A mismatch or unknown ownership blocks creation, even with
`--skip-preflight`. A denied task prefix can therefore follow a green generic
health check:

```
Error: execution preflight storage_access: exact output prefix write failed (authorization)
```

Use submit's execution diagnosis. Reconcile the selected project and the exact
endpoint/bucket/prefix before changing IAM. `--s3-bucket` / `--s3-prefix` bind
the actual spec and ledger; `--s3-endpoint` binds both probe and worker endpoint.
S3 keys are selected as a pair from one source, with non-secret provenance;
an incomplete explicit pair cannot borrow a saved principal's other key.

The credential preflight resolves its bucket from `NPA_CHECKPOINT_BUCKET`, then
`NEBIUS_S3_BUCKET`, then saved credentials. Setting only `NPA_S3_BUCKET` changes
the workflow destination but does not select the credential preflight bucket.
When checking an explicitly authorized workflow destination, set
`NPA_CHECKPOINT_BUCKET` to an unsigned `s3://` URI for that same bucket in the
private process environment and use the matching endpoint and credentials.
The list probe requires a URI, so a bare bucket name fails before S3 access.
Verify bucket ownership first;
do not change shared configuration or provision storage to repair a stale
default selection.

## Persisting credentials you already hold

If the tokens are in your environment but not in `~/.npa/credentials.yaml`,
persist them instead of re-exporting them in every shell:

```bash
npa workbench health access --save-env-credentials
npa configure --save-env-credentials
```

Both perform an atomic `0600` write and never print the values. This is the fix
for "it worked in my shell but the submit could not resolve the secret": submit
resolves each requested secret from the explicit process environment first, then
the selected project's configured NPA credentials.

## Hidden sim2real check

`npa workbench health sim2real` exists but is hidden, and is specific to the
14-stage Sim2Real graph rather than general readiness. It adds cluster-shaped
checks (`config`, `coherence`, `s3`, `registry`, `tokens`, `cluster`) including
schedulable GPU count and kube-context pinning:

```bash
npa workbench health sim2real --checks all --json
```

Use it from `skills/workflows/sim2real-operate/SKILL.md`, not as the generic
preflight. Its cluster check counts schedulable `nvidia.com/gpu` but does not yet
match the *requested GPU product* against available node products, so it can pass
on a cluster that has GPUs of the wrong kind for your spec. Confirm the product
with `npa workbench workflow gpus --cluster <name>`.

## Gotchas

- **`--offline` proves presence, not validity.** It skips live provider probes.
  Existing credentials may pass presence checks, while `nebius` returns SKIP
  because profile configuration is not authentication proof. Use offline only
  where there is genuinely no egress.
- **A SKIP is not a PASS.** Checks skip when the corresponding capability is not
  configured. If you are about to run a Cosmos stage and the NGC row says SKIP,
  you have not verified anything about NGC.
- **`preflight` does not check Kubernetes or SkyPilot.** Cluster readiness is
  `npa skypilot verify --cluster <exact-context>` and `npa cluster status`;
  registry pullability is `workflow preflight-images`. Three separate gates,
  three separate commands.
- **Stale `NEBIUS_IAM_TOKEN` defeats provider calls even when health is green.**
  The Nebius provider prefers an ambient (often expired) token over the fresh CLI
  token. `unset NEBIUS_IAM_TOKEN NPA_NEBIUS_IAM_TOKEN` before provisioning or
  submitting.

## Verify

```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q
```

With an authenticated operator profile, exercise the real CLI's successful,
missing-profile, and offline paths (including stale ambient token scrubbing):

```bash
NPA_INTEGRATION_E2E=1 npa/.venv/bin/python -m pytest \
  npa/tests/e2e/test_nebius_auth_preflight.py -q
```