Paidf Orchestration Setup

---
name: paidf-orchestration-setup
description: Audit, prepare, and deploy PAIDF Orchestration on a Kubernetes GPU cluster - single-GPU H100/L40S hosts, managed Kubernetes, kubeadm, and similar. Select for requests to set up, install, deploy, configure, or check a PAIDF Orchestration environment; run a workflow on a new or unverified GPU host; connect via kubeconfig; validate GPU compute; deploy the Airflow controller; or choose external versus in-cluster model services. A plain SSH host is not a supported backend.
version: "1.0.0"
license: CC-BY-4.0 AND Apache-2.0
metadata:
  owner: NVIDIA
  service: physical-ai-data-factory
  version: 1.0.0
  reviewed: '2026-09-02'
  author: NVIDIA
  tags:
    - physical-ai
    - paidf-orchestration
    - kubernetes
    - airflow
---

# PAIDF Orchestration Environment Setup

Prepare a Kubernetes GPU environment for PAIDF Orchestration without assuming a cloud provider.
Treat Kubernetes—not the host vendor—as the integration contract.

## Safety boundary

Start with a read-only audit. Before any `prepare`, Helm install, Airflow connection change, or
remote mutation, summarize the exact changes and obtain user approval. Never print kubeconfig,
NGC keys, Hugging Face tokens, AWS secrets, or Kubernetes Secret bodies.

## Select two independent axes

1. Select controller placement: use an existing controller, or deploy Airflow into the current
   Kubernetes cluster.
2. Select model-service placement separately: use external VLM/LLM/image-edit endpoints, or deploy
   those services in-cluster. Never infer one choice from the other.
3. Prefer external endpoints on a one-GPU H100 node. The controller and augmentation worker may
   still run in that node's cluster.
4. Reject a Docker-only or SSH-only host until a supported Kubernetes distribution and NVIDIA
   device plugin expose `nvidia.com/gpu`.

Read [topologies.md](references/topologies.md) before changing infrastructure.

## Audit the compute cluster

The cluster is reached only through a kubeconfig the user supplies. It carries a cluster address
and admin credentials, so it is never part of the repository. Resolve it in this order:

1. Use `$KUBECONFIG` if it is already set in the environment.
2. Otherwise **ask the user for the path** and export it.

```bash
echo "${KUBECONFIG:-unset}"   # ask the user for a path when this is unset
export KUBECONFIG=/path/the/user/gave
```

Never guess a path, assume a repository-relative location, or fall back to `~/.kube/config`. If
the path the user names does not exist, say so and ask again.

Run locally when the agent already has the kubeconfig (`remote_k8s.py audit` has no
`--kubeconfig` flag; pass it via the env var):

```bash
python scripts/remote_k8s.py audit --service-mode external --json
```

Alternatively, pass it inline through `--kubectl-command`:

```bash
python scripts/remote_k8s.py audit \
  --kubectl-command "kubectl --kubeconfig $KUBECONFIG" \
  --service-mode external --json
```

Run through SSH when Kubernetes tooling exists only on the remote host:

```bash
python scripts/remote_k8s.py audit \
  --ssh-target ubuntu@host \
  --kubectl-command "kubectl" \
  --service-mode external --json
```

`remote_k8s.py` is bundled with this skill — run it from the skill directory, not the repository
`scripts/` directory. Use `--kubectl-command "k3s kubectl"` when appropriate. Do not pass SSH
passwords or private-key contents in the prompt; use SSH configuration or an agent.

Before a Helm install the audit exits non-zero with `ready: false` and blocker
`NGC image-pull secret is missing`. That is the expected first-install state, because the chart
creates that secret itself — read the `facts` block and continue. Do not resolve it with
`--create-registry-secret`, which makes the subsequent install fail on ownership metadata.

Interpret capacity conservatively:

- **Image Attribute Augmentation:** External endpoints deploy no in-cluster inference services, and
  the checked-in augmentation and attribute-search tasks use CPU profiles. Internal mode deploys
  VLM, LLM, and image-edit services, each claiming one GPU; require at least three allocatable GPUs
  for one replica of each, plus one for each additional replica.
- **Event Video Generation:** External endpoints deploy no in-cluster inference services, but
  detection/tracking, captioning, and visual-QA auto-labeling task pods each claim one GPU while
  active. Internal mode additionally needs one GPU per VLM and LLM replica plus two per image-to-video
  replica — at least four allocatable GPUs for one replica of each service.
- A single-GPU node (for example, one H100) can use external model endpoints, subject to Event Video
  Generation's GPU auto-labeling capacity.

**The compute cluster is shared.** Other users' DAG runs may be active in the same namespace.
Always report GPUs as free-versus-total (check running pods for GPU requests, not just node
allocatable), and never issue broad destructive commands (`delete pods --all`) against the compute
namespace without first checking pod ownership via `dag_id` and `run_id` labels.

## Deploy a controller on the current cluster

When the user requests setup (not audit-only), confirm which steps to run before executing
anything. Present the exact commands you plan to run and obtain explicit approval:

> "I will run the following commands in order:
> 1. `make setup` — validates secrets from `secrets.env` and generates the Helm values for install
> 2. `make install sdg-controller` — packages Airflow runtime dependencies, uploads DAGs and
>    plugins to S3, and installs/upgrades the Helm release
>
> Proceed?"

There are exactly two install-related targets: `make setup` and `make install sdg-controller`.
There is no bare `make install` and no `make install nfs` unless NFS storage is also needed.

**Always run `make setup` first** on any deploy, install, or redeploy request — even if a previous
run already generated the Helm values. Secrets rotate; `make setup` is cheap and safe. Only skip it
mid-session when the agent itself just ran it moments earlier.

**Missing namespace** — if `kubectl get ns sdg-workflow` returns `NotFound`, this is a normal
first-install condition, not an error to diagnose. Route directly to `make install sdg-controller`.

Read [deploy-controller.md](references/deploy-controller.md) for the full `make setup` /
`make install sdg-controller` walkthrough: required `secrets.env` variables, the sandbox DNS
failure signature, post-install cluster-state verification, capacity pre-flight, and storage
requirements — before running either command.

## Prepare missing cluster prerequisites

After explicit approval, create only the requested resources. `remote_k8s.py prepare` reads
`NGC_API_KEY` from the **environment** — use `set -a` to export variables from `secrets.env`
before running, otherwise `source` alone does not export them to child processes:

```bash
set -a && source secrets.env && set +a   # sets KUBECONFIG when `make setup` has already run
export KUBECONFIG=/path/the/user/gave    # otherwise set it explicitly, after the source above
python skills/paidf-orchestration-setup/scripts/remote_k8s.py prepare \
  --create-registry-secret
```

The secret is sent as a manifest over stdin; `NGC_API_KEY` never appears in command arguments or
output.

**Do not pre-create `ngc-docker-registry-secret` when you intend to run `make install
sdg-controller`.** The Helm chart manages that secret itself, and a manually created one has no
Helm ownership metadata, so the install aborts before deploying anything:

```
Error: unable to continue with install: Secret "ngc-docker-registry-secret" in namespace
"sdg-workflow" exists and cannot be imported into the current release: invalid ownership
metadata; label validation error: missing key "app.kubernetes.io/managed-by"...
```

Use `--create-registry-secret` only to validate NGC credentials against a cluster that will not be
Helm-managed. If the conflict occurs, delete the secret and let Helm recreate it:

```bash
kubectl --kubeconfig "$KUBECONFIG" delete secret ngc-docker-registry-secret -n sdg-workflow
```

For internal services, create the model-cache PVC only after selecting a valid storage class:

```bash
python scripts/remote_k8s.py prepare \
  --create-registry-secret \
  --create-model-cache-pvc \
  --storage-class nfs \
  --pvc-access-mode ReadWriteMany
```

`prepare` has no `--kubeconfig` flag (neither does `audit`); like `audit`, it relies on ambient
`kubectl` picking up `$KUBECONFIG` from the environment.

Do not install a GPU operator, device plugin, or Kubernetes distribution automatically. Report
those as infrastructure prerequisites, because the correct installation is provider- and
distro-specific.

## Connect the SDG controller

Read [controller-connection.md](references/controller-connection.md). After install, verify using
the Airflow token obtained in [Connect to the deployed controller](#connect-to-the-deployed-controller):

```bash
# 1. kubernetes_remote connection exists.
# It is injected as an env var, not stored in the metadata database, so
# GET /api/v2/connections/kubernetes_remote returns 404 on a healthy controller.
# Check the env var instead — a 404 here is not a failure.
kubectl exec -n sdg-workflow deploy/sdg-workflow-controller-scheduler -c scheduler -- \
  printenv AIRFLOW_CONN_KUBERNETES_REMOTE >/dev/null 2>&1 \
  && echo "kubernetes_remote: present" \
  || echo "kubernetes_remote: MISSING"

# 2. Required pools have slots. default_pool is Airflow's built-in pool (not chart-created);
# the rest come from deploy/values.yaml airflowPools.pools and are workflow-specific — include
# every workflow you intend to run, not just one.
POOLS_JSON=$(curl -s -H "Authorization: Bearer $TOKEN" "$AIRFLOW_URL/api/v2/pools")
python3 -c "
import sys,json
required = ('k8s_gpu_1','default_pool',
    'external_image_edit_service_pool','iaa_internal_image_edit_service_pool',  # image-attribute-augmentation-workflow
    'external_image2video_service_pool','internal_image2video_service_pool')   # event-video-generation-workflow
pools = {p['name']: p for p in json.load(sys.stdin).get('pools',[])}
for n in required:
    p = pools.get(n)
    print(n, '- OK slots:', p['slots'] if p else 'MISSING')
" <<< "$POOLS_JSON"

# 3. The DAG(s) you intend to run are loaded and unpaused
IAA_DAG_JSON=$(curl -s -H "Authorization: Bearer $TOKEN" \
  "$AIRFLOW_URL/api/v2/dags/image_attribute_augmentation_dag_k8s")
python3 -c "import sys,json; d=json.load(sys.stdin); print('is_paused:', d.get('is_paused'), '| found:', 'dag_id' in d)" <<< "$IAA_DAG_JSON"
EVG_DAG_JSON=$(curl -s -H "Authorization: Bearer $TOKEN" \
  "$AIRFLOW_URL/api/v2/dags/event_video_generation_dag_k8s")
python3 -c "import sys,json; d=json.load(sys.stdin); print('is_paused:', d.get('is_paused'), '| found:', 'dag_id' in d)" <<< "$EVG_DAG_JSON"

# 4. Multistorage config secret exists
SECRET_JSON=$(kubectl get secret -n sdg-workflow multistorageclient-configuration-secret \
  -o jsonpath='{.data}' 2>/dev/null)
if [ -n "$SECRET_JSON" ]; then
  python3 -c "import sys,json; print('keys:', list(json.load(sys.stdin).keys()))" <<< "$SECRET_JSON"
else
  echo "multistorageclient-configuration-secret NOT FOUND"
fi
```

Return `controller readiness: unverified` unless these were checked. For a newly deployed
controller, run all four checks above before reporting ready.

## Connect to the deployed controller

After `make install sdg-controller` succeeds, establish the `AIRFLOW_URL`. The ClusterIP is
always routable from the host machine (even without port-forward) and is the most reliable choice
for agent use:

```bash
AIRFLOW_URL="http://$(kubectl get svc -n sdg-workflow \
  sdg-workflow-controller-api-server \
  -o jsonpath='{.spec.clusterIP}'):8080"
echo "AIRFLOW_URL=$AIRFLOW_URL"
```

Then obtain a JWT token. Credentials are in `deploy/values.yaml` under
`airflow.createUserJob.defaultUser` (default `admin`/`admin` — change before production use). Note
the path is `createUserJob`, not `webserver`, which does not exist in this chart:

```bash
AUTH_RESPONSE=$(curl -s -X POST "$AIRFLOW_URL/auth/token" \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin"}')
TOKEN=$(AUTH_RESPONSE="$AUTH_RESPONSE" python3 -c "import sys,json,os; print(json.loads(os.environ['AUTH_RESPONSE'])['access_token'])")

test -n "$TOKEN" && echo "auth OK" || echo "auth FAILED"
```

To also expose the UI in a browser from another machine, start a port-forward. It binds
`0.0.0.0:8080` on the host, so the UI is reachable at the host's own address on port 8080:

```bash
make port-forward   # blocks until interrupted — run it in a terminal you own
HOST_IP=$(hostname -I | awk '{print $1}')
echo "Airflow UI: http://$HOST_IP:8080"
```

The Kubernetes ClusterIP and the host's own network address are separate address spaces. The
ClusterIP is reachable from the host but is not externally routable; the host address via
port-forward is what a browser on another machine should use. Resolve both at runtime — never
assume or hard-code either.

`make port-forward` never exits. The agent may start it as a background job using the harness's
native background-job mechanism (not a raw shell `&`) to verify connectivity or serve a short-lived
need — this keeps the shell responsive for follow-up commands. Tell the user it will stop when the
agent session ends, and prefer a terminal the user owns for anything that must persist beyond this
conversation. Before starting a new forward, check for and clean up any stray prior
`make port-forward` / `kubectl port-forward ... 8080` processes so they don't compete for the port:

```bash
ps -ef | grep "port-forward" | grep -v grep
kill <pid>   # or kill -9 if it doesn't respond
```

Verify with a bounded probe against both addresses:

```bash
curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 http://localhost:8080
curl -s -o /dev/null -w "%{http_code}\n" --max-time 5 http://<host-ip>:8080
```

To update DAGs or plugins after the initial install without reinstalling (the dag-synchronizer
picks up S3 changes within the configured interval, default 30 s):

```bash
make sync-dag
```

## Handoff to the augmentation run

Produce a readiness report containing topology, Kubernetes context, ready GPU count, service mode,
Airflow URL, missing resources, controller checks, and safe remediation. Before continuing to a
workflow run, present the planned install commands (`make setup`, `make install sdg-controller`)
and wait for explicit approval — even if controller pods appear healthy. If the original request
also asks to run a workflow, continue with that workflow's own skill procedure (for example
`image-attribute-augmentation-workflow` or `event-video-generation-workflow`) only after the user
approves or declines the install steps and compute and controller readiness are established; do
not ask the user to name or re-invoke another skill. Never submit a workflow solely because
`kubectl get nodes` succeeds.