Teardown And Cost

---
name: teardown-and-cost
description: Use to stop spend safely and audit orphaned resources — the mandatory cancel-before-destroy ordering, which npa commands touch cloud versus local state, and how to find leaked clusters, agent VMs, controllers, and buckets.
---

# Teardown and cost hygiene

There is no `npa cost` and no `npa quota` command. Spend control here is
procedural: run things in an order that cannot orphan resources, and audit for
residue afterwards. The single most expensive mistake is destroying
infrastructure while managed jobs are still running, which leaves jobs billing
against resources nothing is tracking any more.

## The ordering is not optional

Cancel workloads, then remove the things that host them, outermost last:

```bash
npa workbench workflow cancel <run-id> --project <alias> --json   # every run
npa agent destroy --project <alias> --name <name> --yes
npa skypilot cleanup-controller --project <alias> --context <context> --yes
npa cluster down --project <alias> --force
npa storage bucket delete --project <alias> --yes --wait
npa storage service-account delete --project <alias> --dry-run
npa storage service-account delete --project <alias> --yes
npa configure --forget-project <alias>
npa cleanup --full --yes --project <alias>
```

Cancel is repeat-safe — a run that never launched is a successful no-op — so
there is no reason to skip it. `cleanup-controller` refuses while managed jobs
are in progress and retries that specific refusal after the queue drains; treat
the refusal as correct, not as something to force.

If exact queue evidence says a recorded non-terminal job is absent, standalone
workflow cancellation stays non-terminal and exits 2; stale durable state is not
rewritten as success. An explicit `npa destroy --all --yes` transaction may
continue owned controller/cluster teardown only when cancel's structured result
says `owned_teardown_allowed: true`. That allowance means every live exact job
was cancelled cleanly and every remaining error is an exact verified-absence
contradiction. The workflow phase remains degraded for audit. Provider
unavailability, malformed state, missing IDs, or cleanup errors still block all
dependent destroy phases.

Managed-job terminality is closed-world during cleanup. `SUBMITTED`,
`WINDING_DOWN`, and any unrecognized future status remain non-terminal until
the exact queue reports a terminal state or verified absence; never infer
terminality merely because a status is absent from a known-active list.

For a unique per-run workflow API, controller cleanup does not finish local
shutdown: only its temporary transaction API is stopped automatically. Wait for
the submit driver and all other clients of that API to exit, preserve exact-run
cancellation and fully verified controller-cleanup receipts, then stop the
original API with the existing `stop_isolated_api(Path(owned_run_directory))`
helper. Follow `docs/teardown.md#owned-local-workflow-api` and the receipt-checked
R7 example in `workflows/guides/paidf-cosmos3.md`. Keep the final local stop
independently recoverable from its successful receipts; do not repeat cloud
deletion solely because the API is still alive. Verify the retained daemon
record is stopped with null PID/start ticks, retain run data, and never substitute
`sky api stop` or stop a shared API because one workflow finished.

Deleting the project itself is separately opt-in and stays retained by default:

```bash
npa destroy --project <alias> --all --json          # plan only
npa destroy --project <alias> --all --yes           # execute, project retained
npa destroy --project <alias> --all --delete-project --yes --json
```

`npa destroy --all` without `--delete-project --yes` never deletes the project.
Full details of the ownership gating live in `skills/tools/nebius-infra/SKILL.md`;
this skill is the operational ordering and the audit.

## What is cloud spend and what is only local clutter

Confusing these two wastes time and money in opposite directions.

| Command | Touches cloud spend | Notes |
|---|---|---|
| `npa cleanup` | No | Local caches and state only, even with `--yes` |
| `npa cleanup --full --yes` | No | Also removes saved HF/Token Factory/NGC creds; read-only IAM verification |
| `npa workbench workflow cancel` | Yes — stops it | Repeat-safe |
| `npa skypilot cleanup-controller` | Yes | Shared; drains first |
| `npa cluster down` | Yes | Terraform-managed cluster and its nodes |
| `npa agent destroy` | Yes | Agent VM and its resources |
| `npa storage bucket delete` | Yes | Contents and versions included |
| `npa destroy --all --delete-project --yes` | Yes | Ownership-gated; all three flags required |

`npa cleanup` never deletes cloud resources. It *reports* what teardown left
behind and prints the ordered runbook, which makes it the right first command
when you inherit a machine and do not know what is still running:

```bash
npa cleanup --json                 # report only
npa cleanup --list-receipts        # durable audit trail, no secrets
```

## Auditing for orphans

Residue accumulates when a teardown was interrupted or run out of order. Check
each class explicitly — nothing sweeps them for you:

```bash
npa workbench workflow list --project <alias> --json    # non-terminal runs
npa skypilot status --project <alias> --context <ctx>   # shared controller
npa cluster list --project <alias>                      # clusters, incl. stale
npa agent list                                          # agent VMs in config
npa storage bucket list --project <alias>               # marks the configured one
npa fleet status --spec <spec.yaml>                     # cross-project fleets
```

Cross-project fleets are the easiest thing to lose track of, because a fleet spec
can create clusters in projects that never appear in your default alias. Audit
fleets from the spec that created them.

`npa cleanup` queries the SkyPilot managed-job queue as part of its report. Do
not reach for `--skip-jobs` to make it quiet: skipping the queue with
`--attest-no-active-jobs` is an explicit attestation that you verified terminal
state some other way, not a shortcut.

## Receipts: how to finish a teardown you can no longer address by alias

Teardown receipts live under `~/.npa/teardown-receipts/`, contain no secrets, and
survive removal of the project config. If you already ran
`configure --forget-project` and later discover residue, the receipt is the
recovery identity:

```bash
npa cleanup --list-receipts
npa destroy --receipt <id> --all --delete-project --yes --json
```

Prune only aged terminal receipts, and only explicitly:

```bash
npa cleanup --prune-receipts --receipt-retention-days 90 --yes
```

## Choosing a cheap tier in the first place

Most verification does not need a GPU, and the tiers are ordered by cost:

1. **Dry run / plan.** `npa workbench workflow plan-spec`, `submit --plan-only`,
   `golden-eval run <name>` (dry-run is the default), `provision-if-absent
   --dry-run`. No cloud resources.
2. **Zero-GPU hosted inference.** Token Factory (`skills/tools/token-factory`)
   answers real captioning/generation/reasoning questions with no cluster at all.
3. **Serverless single job.** `golden-eval run <name> --serverless` runs one
   container on one GPU and returns PASS/FAIL, without standing up a cluster.
4. **Cluster.** Only when the work genuinely needs a persistent multi-stage graph.

Preemptible GPU nodes change the capacity pool but **not** hard instance, disk,
or IP quotas, and a reclaim stops the node mid-run. They lower price, not
requirements.

## Gotchas

- **A PENDING job is still billing the cluster it is stuck on.** `ImagePullBackOff`
  and `Unschedulable` are retried forever and never self-terminate. Cancel them;
  see `skills/atomic/debug-failed-run/SKILL.md`.
- **`npa cluster down` going quiet for minutes is expected.** It previews the
  PodDisruptionBudgets that hold up the node drain before it destroys anything.
- **The controller is shared.** Never tear it down because *your* run finished;
  it may be hosting someone else's managed jobs. That is why `cleanup-controller`
  requires the exact project and saved context rather than an ambient
  kube-context.
- **Full cleanup exiting 2 is a real result, not a flake.** It means storage IAM
  is present or unverified: cleanup was partial and something still exists.
- **Never run a live GPU tier on a schedule.** Scheduled GPU runs are the classic
  way to discover a five-figure bill; keep GPU rotations manually dispatched.

## Verify

```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q
```