Gitlab Best Practices

---
name: gitlab-best-practices
description: >-
  Operate self-managed GitLab on the official Helm chart — multi-hop upgrade
  campaigns across required stops, the chart-10 wall (bundled PostgreSQL,
  Redis and MinIO removed and now mandatory), PostgreSQL 17 being both the
  minimum and the maximum for GitLab 19.x, default flips invisible to a values
  diff, air-gapped image sets the chart will not reveal, backup/restore with
  external dependencies and the secrets that are deliberately not in the
  backup, Sidekiq migration-loss and HPA-flap traps, and what high
  availability the chart actually provides (standalone Gitaly on Kubernetes is
  a documented single point of failure; Praefect on Kubernetes is beta).
when_to_use: >-
  Use for any self-managed GitLab task: "upgrade GitLab", "GitLab required
  stops", "upgrade_path.yml", "GitLab 19", "chart 10", "gitlab helm values",
  "GitLab air-gap image list", "gitlab-rails-secret", "backup-utility",
  "restore GitLab", "GitLab HA", "Gitaly Cluster", "Praefect", "GitLab Duo
  self-hosted", "GitLab zero-downtime upgrade". Symptoms: background
  migrations stuck or finalizing forever, `PG::UndefinedTable` mid-upgrade,
  `deduplicated: dropped` in Sidekiq logs, Sidekiq pods rescaling constantly
  or jobs SIGKILLed, Ingress objects vanishing after a chart bump, a default
  render failing with "external PostgreSQL became required", `helm upgrade`
  timing out on a migration. NOT for GitLab CI pipeline authoring, and NOT for
  Zalando postgres-operator mechanics (use postgres-operator-best-practices).
argument-hint: "[upgrade|ha|air-gap|backup|duo] (optional focus area)"
---

# gitlab-best-practices

Run and upgrade a **self-managed GitLab on the official Helm chart** without
unplanned downtime. The hard part is never `helm upgrade`. It is that the
chart changes defaults the site never set, ships prerequisites written for a
configuration the site does not run, and removed its own bundled databases in a
release that breaks the *tooling* before it breaks the deployment.

Facts here were verified **2026-08-29** against machine-readable upstream
sources, the unpacked chart, and a live install taken through a multi-hop
18.x → 19.x campaign. Re-verify anything version-bearing.

**Version anchor (2026-08-29):** latest chart **10.3.1** / app **19.3.1**.
Chart major = app major − 9. Required stops through 19.x: **18.2 · 18.5 ·
18.8 · 18.11 · 19.2 · 19.5 · 19.8 · 19.11** — enumerate them fresh, never
from memory.

## Symptom index

| Symptom | Where |
|---|---|
| Planning a multi-version upgrade; which stops, which order | § The ladder below, then `references/upgrade-campaign.md` |
| Ingress objects gone after a chart bump; instance off the network | `references/upgrade-campaign.md` § the chart-10 wall |
| Default render fails: "external PostgreSQL became required" | `references/upgrade-campaign.md` § chart 10.0 makes external deps mandatory |
| Background migrations stuck, finalizing forever, `PG::UndefinedTable` | `references/failure-modes.md` |
| `deduplicated: dropped` in Sidekiq logs; migrations never run, install looks green | `references/failure-modes.md` § Sidekiq |
| Sidekiq rescaling constantly; jobs SIGKILLed mid-flight | `references/failure-modes.md` § the HPA flaps by construction |
| PostgreSQL version, extensions, `amcheck`, scaling down for a DB cutover | `references/external-deps.md` |
| Redis / Sentinel / Valkey, object storage on non-AWS S3, registry `s3_v2` | `references/external-deps.md` |
| Building an air-gapped image list; what the render will not show | `references/air-gap.md` |
| Backup, restore, `gitlab-rails-secret`, rehearsing a restore | `references/backup-restore.md` |
| Backup takes hours; toolbox out of disk; "how do I speed up backups" | `references/backup-restore.md` § Performance and scale |
| Gitaly server-side backups, `goCloudUrl`, Object Lock on a backup bucket | `references/backup-restore.md` § Gitaly server-side backups |
| "Is this HA?", Gitaly Cluster, Praefect, zero-downtime upgrades | `references/ha-and-topology.md` |
| GitLab Duo, AI features, pointing it at a local model | `references/duo-ai.md` |

## The ladder

**Stops are minor-level, and the `.0` of a new major is not one.** Read them
from the machine-readable source, never from prose docs or memory:

```bash
curl -sS https://gitlab.com/gitlab-org/gitlab/-/raw/master/config/upgrade_path.yml
helm search repo gitlab/gitlab --versions | head -30   # chart <-> app, unfiltered
```

There is no `19.0` entry. The path is **18.11 → 19.2**, which crosses every
19.0 and 19.1 breaking change in a single hop with no intermediate resting
point. Satisfy 19.0's prerequisites *before leaving 18.11*.

A stop is a minor, satisfied by any patch of it — take the highest. Chart and
app patch numbers drift apart (chart 9.11.12 carries app 18.11.11); name both
in every artifact.

## The nine facts that cause unplanned downtime

**1. PostgreSQL 17 is both the minimum and the maximum for 19.x.** There is no
running ahead to PG 18. Because 18.x already tolerates PG 17, that overlap is
the only ordering with a supported resting point on both sides: **move the
database to 17 while still on 18.x, then upgrade GitLab.** Extensions are the
quiet half — `amcheck` became required at 18.4, migrations do not install
extensions, and the extensions docs page does not carry the table.
→ `references/external-deps.md`

**2. Chart 10.0 makes external PostgreSQL, Redis and object storage
mandatory — and breaks the tooling first.** The bundled databases are gone
and `NOTES.txt` carries hard `fail`s, so any "render the chart with defaults"
step dies, taking the all-features image sweep with it. Installs already on
external dependencies are insulated at the *deployment* layer and still lose
the *diffing* layer. → `references/upgrade-campaign.md`

**3. Default flips are invisible to a values diff.** A site inheriting a
default cannot see it change by looking at what it sets. Crossing chart
9.11.12 → 10.2.5 flipped five defaults; a hand-built breaking-change list built
from the release notes missed one. **Diff stock values every hop, including
against a hand-built list.** Left unpinned,
`global.ingress.enabled: false` alone takes the instance off the network.

**4. A prerequisite in release notes is written for the DEFAULT
configuration.** Chart 10.3.0's "apply the Gateway API CRDs before upgrading or
it fails on the GitLab Shell TCPRoute" reads as mandatory for everyone. It
fires only where Gateway API objects actually render. Render the target with the
real values, grep for the object kind the prerequisite names, and if it does not
render, record *why* it cannot apply. Complying anyway is not free — Gateway
API CRDs are cluster-scoped and permanent.

**5. Offline `helm template` corrupts exactly the artifacts the diff gates read.**
A cluster-less render cannot populate `.Capabilities.APIVersions`, so the chart
falls to its **oldest** apiVersion branch and silently drops blocks. `helm
upgrade` is unaffected because it talks to the cluster — so only the diffs are
wrong. Pass `-a/--api-versions` from `kubectl api-versions`; `--kube-version`
does not fix it.

**6. A single Sidekiq replica removes a silent migration-loss race.** With
multiple replicas, an old-version pod can take a background-migration job
before terminating, crash, and leave an orphaned dedup key in Redis — after
which **migrations never run while the install looks completely green**. Scale
Sidekiq to 0 for the upgrade, or run one replica. Note the tension: the chart's
zero-downtime procedure *requires* multiple replicas.
→ `references/failure-modes.md`

**7. There is no GA high-availability path for Git data on Kubernetes.**
Standalone Gitaly went GA at 18.11 and is *by design a single point of
failure*; Gitaly Cluster (Praefect) on Kubernetes is **beta**, and its
first-class-solution epic publicly reports "No progress" at ~5 contributor
hours a week. The stateless tier is genuinely HA-capable. Plan around a
**restore plan for Git data**, not replication.
→ `references/ha-and-topology.md`

**8. Rollback is a restore plan.** Reverting means reverting the database
schema, which needs a backup taken at the exact version and edition being
downgraded to, and the restore overwrites all newer content. **There is no
supported `helm rollback` after migrations have run against production data** —
the documented remedy for a bad state is to go *forward* to a required stop.
`helm rollback` recovers a bad values change, nothing more.

**9. Do not tune the toolbox backup — split it by data class.** Every blob
round-trips object storage → pod disk → gzip → tar → back on *every* run, and
the three obvious levers are all unavailable there: `COMPRESS_CMD` never reaches
the blob path (gzip is hardcoded), `SKIP=tar` is refused with object storage,
and `backup-utility` has no incremental mode at all. Move PostgreSQL to its own
tooling, blobs to bucket-to-bucket replication, and repositories to **Gitaly
server-side backups**. **Upstream's documented `cron.extraArgs` recipe is
defective** — it pairs `--skip repositories` with `--repositories-server-side`,
and skip wins, so the scheduled backup contains **no Git repository data** and
nothing says so until restore. Drop `--skip repositories`.
→ `references/backup-restore.md`

## AI, in three lines

- **Free tier gets no Duo at all**, and **CE can never hold a licence**, so no
  Duo path exists there ever. The chart defaults to `edition: ee` — which is
  what keeps an unlicensed install upgradeable later.
- **Duo against a self-hosted vLLM needs Premium/Ultimate plus the Duo
  Enterprise add-on.** Owning the GPUs does not buy out the licence. The
  mandatory AI Gateway defaults to calling `customers.gitlab.com`, and when
  that is unreachable and unoverridden it costs **20 seconds per request** —
  presenting as "the AI is slow", not as an egress error.
- **The MCP server is free** (`Tier: Free, Premium, Ultimate`) and was
  decoupled from Duo in **19.2** — no add-on, no AI Gateway, no egress. On
  ≥19.2 that is the zero-licence answer; below it, the REST/GraphQL API.

→ `references/duo-ai.md`

## Working method

- **Run all six diff layers, every hop.** Stock values, stock render, site
  values, set-difference verify, site render, and site-render-vs-live-cluster.
  Each catches what the others structurally cannot — the stock-render layer is
  the only one that sees RBAC, probes and container args, because those never
  appear in `values.yaml`. → `references/upgrade-campaign.md`
- **Never `patch` a values delta onto a new version; use `diff3`.** `patch` has
  two inputs and cannot know what the file looked like when it was edited — it
  has already fuzz-matched a hunk onto an adjacent identically-shaped key and
  silently changed cluster-wide behaviour, exit 0. Merge mechanics live in the
  **`helm`** skill (its own `references/values-porting.md`), not here.
- **`helm lint` is not a typo check**, and **never `kubectl apply` a rendered
  template** — it rotates lookup-generated TLS/CA secrets.
- **Use `glab`, not a scraper, for gitlab.com.** The notes endpoint 401s to
  unauthenticated fetchers and comments are JS-rendered. A scraped pass once
  concluded a fix had no backport; `glab api .../related_merge_requests`
  returned three and proved the opposite. **A milestone says when a fix landed
  on master, not whether it reached the target version** — only backport MRs and
  their `target_branch` answer that.
- **Get background migrations to zero at every stop, not just the last.**
  Active is fine; paused or failed is not.
- **A failed `helm upgrade` is usually a timeout on a long migration**, not a
  broken release. Check before reacting.
- **Verify in order of what each check proves**, cheapest first: `helm list` →
  Ingress objects exist with an external address → HTTP 200 on the web host →
  registry `/v2/` returns 401 → background migrations clean.
- Citations, per-claim, with issue states: `references/sources.md`.