Project Deploy

---
name: project-deploy
description: Use when deploying to production, running terraform apply against the App Platform app, verifying what version is live, or diagnosing a deploy that appears to have succeeded and changed nothing.
---

# Deploying

Production is a DigitalOcean App Platform app pulling `ghcr.io/ubermuda/loupe:prod`.
Infrastructure lives in `terraform/`, and `terraform/terraform.tfvars` is
gitignored, so it holds the real values.

## The two halves, and why they are not the same event

A deploy is two independent things, and confusing them is the failure this skill
exists to prevent.

1. **The image.** `just build-prod` builds it, `just push-prod` pushes it to the
   `prod` tag. Nothing about this changes production.
2. **The app spec.** `terraform apply` changes environment variables, instance
   sizes and routes. It creates a deployment.

**A `terraform apply` deployment does not re-pull the image.** The tag is fixed
at `prod`, so App Platform sees the same tag it already has and reuses the layers
it holds. The spec change goes live and the code does not.

That is not a bug in the platform. `variable "image_tag"` says so: "A fixed tag
means App Platform's deployment history is the only record of what ran."

## What deploy notes hold

A card can link a Loupe document titled `Deploy notes: <card title>`, tagged `deploy-notes`. The tech design and implementation stages of the Loupe plugin write it. They read this section through `.loupe/lifecycle.md`, "Instruction files", item 8.

Record each item of the change that the deploy must handle:

- An environment variable, with its value or its source. `DeploymentConfigParityCheck` cross-checks `.env`, Terraform, the prod compose files and `docs/reference/environment.md`. Set a `terraform.tfvars` value before the deploy.
- A migration and its risk. The pre-deploy job runs it. Name long locks, backfills and irreversible steps.
- A new feature flag. `/install` seeds flags once, so ship a per-flag migration, as `migrations/Version20260906142530.php` does. Otherwise, name the manual switch-on.
- A new messenger transport or schedule. It must reach `worker_command` in `terraform/main.tf` and the `worker` service in `docker/compose/prod.yaml`.
- A new CSP origin in `config/packages/nelmio_security.yaml`.
- A new external service and its `DiagnosticInterface` check.
- The order of the steps, for example `terraform apply` before or after the image push.
- The checks to do after the deploy.

## Check the release diff

Do this check together with the deploy notes. It finds a deploy item that has no note. Get the last deployed sha from `/healthz`, as "Verify what is actually live" shows.

```bash
git fetch -q origin main
git diff --name-only <last deployed sha>..origin/main -- migrations/ .env terraform/ \
  config/packages/messenger.yaml config/packages/nelmio_security.yaml docker/compose/prod.yaml
```

Read each file in this list that no deploy note covers. Say whether the release ships a migration.

## Read the deploy notes of the release

1. Run `bin/agents/release-cards`. With no argument, it reads the live version from `/healthz`. Otherwise, give it the deployed sha.
2. Read the output. Each line is `<pr number>`, `<card ids>` and `<subject>`, separated by tabs. A card column can hold several ids, separated by commas.
3. Call `card_get` through the `loupe` MCP for each card id.
4. For an epic, also call `card_get` for each child in `children`. A standalone child owns its own deploy notes.
5. Call `document_get` for each linked document. Keep the documents tagged `deploy-notes`.
6. Make a deploy checklist from those documents.
7. Find each line with `-` or `?` as the card. A `?` means that `gh pr view` failed. Read the diff of that pull request, or of that commit, yourself. Never guess a card.

## Ship code

```bash
just deploy          # build-prod, push-prod, then create-deployment --force-rebuild --wait
```

Or, when the image is already pushed:

```bash
doctl apps create-deployment "$(cd terraform && terraform output -raw app_id)" --force-rebuild --wait
```

`--force-rebuild` is what makes the platform fetch the tag again. Without it a
deployment can complete, report `ACTIVE`, and run the previous image.

The working tree must be clean. `APP_VERSION` comes from `git describe --tags
--always --dirty`, so an uncommitted change ships an image whose version cannot
be traced to a commit. `just release` refuses outright; `just deploy` does not,
so check yourself.

## Ship an infrastructure change

```bash
cd terraform
terraform plan -out=/tmp/tf.plan     # read it, then apply the file you read
terraform apply /tmp/tf.plan
```

Apply the saved plan rather than re-planning, so what lands is what you reviewed.

**Read the plan as JSON, not as text.** The DigitalOcean provider collapses the
app resource into "(N unchanged blocks hidden)", and an added environment
variable hides in there. Grepping the human-readable output for a variable name
finds nothing and reads as "no change":

```bash
terraform show -json /tmp/tf.plan | python3 -c "
import json,sys
for rc in json.load(sys.stdin)['resource_changes']:
    if rc['change']['actions'] == ['no-op']: continue
    print(rc['address'], rc['change']['actions'])
    print('  before:', 'MY_VAR' in json.dumps(rc['change']['before'] or {}))
    print('  after :', 'MY_VAR' in json.dumps(rc['change']['after'] or {}))
"
```

To ship code and an infrastructure change together, push the image first and then
apply. One rollout carries both, because the apply's deployment pulls the tag you
just pushed.

## Verify what is actually live

**Ask the app, not the platform.** `doctl apps get` reporting `ACTIVE` says a
deployment finished. It does not say which image is running, and the previous
deployment is also `ACTIVE` until the new one replaces it.

```bash
TOKEN=$(grep -E "^health_probe_token" terraform/terraform.tfvars | sed -E 's/.*= *"([^"]*)".*/\1/')
curl -sS -H "X-Probe-Token: $TOKEN" https://loupe.ac/healthz
```

It answers `{"status":"ok","version":"<short sha>"}`. Compare that against
`git rev-parse --short HEAD`. Without the header the endpoint reports liveness
alone and no metadata, which is deliberate.

Do not scrape `/about` for this. It renders the same version behind a session and
an edge cache, so it is both harder to read and less trustworthy.

`/admin/status` runs the `DiagnosticInterface` checks and needs an admin session.
It is where `TrustedProxyCheck` reports, so read it after any change to
`TRUSTED_PROXIES` or to the edge.

## Which deployment is running, and why

```bash
doctl apps list-deployments "$(cd terraform && terraform output -raw app_id)" -o json \
  | python3 -c "
import json,sys
for d in json.load(sys.stdin)[:3]:
    print(d['id'][:8], d['phase'], d['created_at'][:19], d.get('cause',''))
"
```

`cause` is the tell. `app spec updated` came from a `terraform apply` and may be
running old code. `manual` came from `create-deployment`.

## Migrations

`enable_predeploy_migrations = true`, so a pre-deploy job runs
`doctrine:migrations:migrate` before the new containers take traffic. A failed
migration fails the deployment and the previous one keeps serving.

"Check the release diff" lists the migrations of the release. A release with no
migration is a much smaller risk than one with. Say which you are shipping.

## Trusted proxies

`loupe.ac` sits behind Cloudflare, which sits in front of App Platform's private
ingress. `TRUSTED_PROXIES` must list Cloudflare's ranges, or Symfony stops the
address walk at the ingress and every visitor resolves to a Cloudflare address.

The failure is quiet and it is not what you would test for. The immediate peer is
private and therefore trusted, so `isFromTrustedProxy()` answers true and the
configuration looks right, while `getClientIp()` still returns the edge. Two rate
limiters key on that address, so they collapse to one bucket per edge node.

Terraform prepends `PRIVATE_SUBNETS` to whatever `trusted_proxies` holds, so set
the Cloudflare ranges alone. Refresh them from `cloudflare.com/ips-v4` and
`/ips-v6` when they change. `TrustedProxyCheck` on `/admin/status` is what tells
you the list has gone stale.

## After a deploy

`docs/operating/post-deploy-checks.md` is the operator-facing list. Follow it.

## Symptoms and causes

| Symptom | Cause |
|---|---|
| Deployment `ACTIVE`, `/healthz` reports the old version | A `terraform apply` deployment reused the fixed tag. Run `create-deployment --force-rebuild`. |
| `terraform plan` shows no environment change you know you made | The text output hid it. Read the JSON plan. |
| `/healthz` returns liveness with no version | The `X-Probe-Token` header is missing or wrong. |
| Rate limits fire for unrelated visitors | `TRUSTED_PROXIES` does not cover the edge, so they share one bucket. |
| The image reports a `-dirty` version | It was built from an unclean tree. Commit, rebuild, push. |