Kubernetes Operator

---
name: kubernetes-operator
description: Use when building a Kubernetes Operator — custom controllers that reconcile CRD state. Triggers on "build an operator", "CRD design", "reconcile loop", "controller-runtime", "kubebuilder", "operator-sdk", "custom resource", or "operator capability levels". NOT a generic k8s skill — specifically the Operator pattern.
stack: [kubernetes]
area: infra
---

# Build operators that reconcile correctly

An operator is a reconcile loop, not a script. Most operator bugs are not Kubernetes bugs — they are reconcile-loop bugs: missing finalizers, blocking calls, no requeue on transient errors, status drift, RBAC over-grants.

## The reconcile mental model

```
observe(actual) → desired = read(spec) → diff(actual, desired) → act → update(status)
                                                                         ↓
                                                                  requeue / done
```

**Idempotent, not imperative.** The reconcile function must be callable any number of times with the same outcome. It must not assume what happened on the previous call.

## Common operator bugs

| Bug | Prevention |
|-----|-----------|
| Blocking HTTP calls in reconcile | Use async clients or move to a goroutine |
| No requeue on transient error | Always `return ctrl.Result{}, err` for transient; `RequeueAfter` for polling |
| Missing finalizer | Add finalizer on creation; remove only after cleanup complete |
| Mutating spec instead of status | Spec is user-owned; status is controller-owned |
| No status subresource | Status updates without subresource trigger spec reconcile → loop |
| RBAC over-grant | Principle of least privilege; use `ClusterRole` only when namespace-scope is insufficient |
| No leader election | Multi-replica deploy without leader election → split-brain |

## Operator capability levels (OLM)

Level 1 (Basic install) → Level 2 (Seamless upgrades) → Level 3 (Full lifecycle) → Level 4 (Deep insights) → Level 5 (Auto pilot)

Start at Level 1 and promote only when the lower levels are tested and stable.

## Steps

1. **Read the lore + existing operators.** `search_lore` for CRD conventions, RBAC policy, and framework choice. Match the existing toolchain (kubebuilder vs operator-sdk vs KOPF vs metacontroller).
2. **Design the CRD API surface.** `spec` = desired state (user-owned); `status` = observed state (controller-owned). Define `conditions` as standard status fields. Validate with admission webhooks for types that have invariants.
3. **Implement the reconcile loop.** Read → diff → act → update status. Every `return` is either `Result{}, nil` (done, no requeue) or `Result{}, err` / `Result{RequeueAfter: d}, nil` (retry/poll).
4. **Add finalizers.** Register on object creation; execute cleanup in the finalizer block; remove only after cleanup succeeds.
5. **RBAC.** Generate from controller-gen annotations; scope to namespace where possible; document every `ClusterRole` grant.
6. **Leader election.** Enable for any operator that will run with >1 replica.
7. **Verify.** Deploy to a local cluster (kind/minikube); create/update/delete a CR; confirm status conditions reflect the current state; test error injection (unavailable dependency); record evidence.

## Build / Test

- `go vet` + `golangci-lint` clean.
- Unit tests for the reconcile function with a fake client — test every `return` path.
- E2E tests against a real cluster (kind): create, update (in-place upgrade), delete with finalizer cleanup.
- `controller-gen` generates CRD + RBAC manifests — commit generated files, do not hand-edit them.

## Review checklist

- **Reconcile is idempotent** — safe to call N times with the same result.
- **Every error path requeues** — transient errors return `err`; expected polls use `RequeueAfter`.
- **Finalizer present** — and cleanup is tested on deletion.
- **Status uses conditions** — standard Kubernetes condition format (`Type`, `Status`, `Reason`, `Message`).
- **RBAC minimal** — no `ClusterRole` without justification; generated by controller-gen.
- **Leader election enabled** for multi-replica deploys.

## Capture lore

Framework choice, CRD naming conventions, cluster version, and RBAC policy are high-value lore — call `suggest_lore` with `tags: [kubernetes, operator, crd]`.