Kubernetes Ops

---
name: kubernetes-ops
description: Kubernetes operations, manifest generation, Helm charts, Karpenter autoscaling, GitOps (ArgoCD/Flux), network policies, pod security, and troubleshooting. Auto-activates on Kubernetes, K8s, kubectl, pod, deployment, service, ingress, helm, karpenter, eks, gke, aks, node, cluster, namespace, HPA, autoscaling.
---

# Kubernetes Operations

## Overview

Production-ready Kubernetes patterns with security-hardened defaults. Covers manifest generation, autoscaling, GitOps, networking, and troubleshooting.

## Manifest Templates

### Deployment (Security-Hardened)

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: APP_NAME
  labels:
    app.kubernetes.io/name: APP_NAME
    app.kubernetes.io/version: "1.0.0"
    app.kubernetes.io/managed-by: helm
spec:
  # No `replicas` here on purpose — the HPA below owns this field. Pinning it while an
  # HPA is active flaps forever under GitOps; see the warning under the HPA template.
  # No HPA for this workload? Then add `replicas: 2` (omitting it defaults to 1).
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app.kubernetes.io/name: APP_NAME
  template:
    metadata:
      labels:
        app.kubernetes.io/name: APP_NAME
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      serviceAccountName: APP_NAME
      containers:
        - name: APP_NAME
          image: REGISTRY/APP_NAME:TAG
          ports:
            - containerPort: 8080
              protocol: TCP
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: APP_NAME
```

### PodDisruptionBudget

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: APP_NAME
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: APP_NAME
```

### NetworkPolicy (Default Deny + Allow Specific)

```yaml
# Default deny all ingress in namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes:
    - Ingress
---
# Allow specific traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-APP_NAME
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: APP_NAME
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: frontend
      ports:
        - port: 8080
          protocol: TCP
```

### HorizontalPodAutoscaler

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: APP_NAME
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: APP_NAME
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60
```

**WARNING — pinned `replicas` + HPA + GitOps self-heal flap forever.** The Deployment above
composes safely with this HPA *only because it omits `spec.replicas`*. Put that field back and
the HPA scales the Deployment up while the GitOps engine reverts it to the pinned number, in a
loop, with **no error raised anywhere** — the only symptom is an oscillating replica count and a
storm of sync / `ScalingReplicaSet` events. Kubernetes says removing it is recommended, since
re-applying a pinned value while an HPA is active risks "thrashing or flapping behavior"
([HPA docs](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#migrating-deployments-and-statefulsets-to-horizontal-autoscaling)).

| Engine | Why it reverts the HPA | Fix |
|--------|------------------------|-----|
| Argo CD `selfHeal: true` | live replicas ≠ Git replicas ⇒ OutOfSync ⇒ self-heal patches it back | omit `replicas`; if a Helm chart forces it, use `ignoreDifferences` **plus** `RespectIgnoreDifferences=true` (see below) |
| Argo CD `ServerSideApply=true` | applies with `--force-conflicts`, seizing `spec.replicas` from the HPA's field manager | omit `replicas` — force-conflicts overrides field ownership |
| Flux `Kustomization` | every `interval` re-applies each field that diverges from Git; Flux has **no** field-level ignore | omit `replicas` (the [Flux FAQ](https://fluxcd.io/flux/faq/) requires it) |

Removing `replicas` costs a one-time dip to 1 pod on first apply (the API default) before the HPA
scales up to `minReplicas`. Avoid the dip by dropping the field from the live object first —
`kubectl apply edit-last-applied deployment/APP_NAME`, delete `spec.replicas`, then commit. The
same rule covers anything a controller owns: never pin in Git a field HPA, KEDA, or VPA writes.

## Karpenter (AWS Node Autoscaling)

Karpenter replaces Cluster Autoscaler with workload-aware, faster node provisioning.

### NodePool

```yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    metadata:
      labels:
        managed-by: karpenter
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h
  limits:
    cpu: "100"
    memory: 400Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
```

### EC2NodeClass

```yaml
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiSelectorTerms:
    - alias: al2023@latest
  role: KarpenterNodeRole-CLUSTER_NAME
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: CLUSTER_NAME
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: CLUSTER_NAME
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 50Gi
        volumeType: gp3
        encrypted: true
```

### Autoscaling Tiers

| Tier | Tool | What It Scales | When to Use |
|------|------|---------------|-------------|
| 1 | HPA | Pod replicas | CPU/memory-based, always start here |
| 2 | VPA | Pod resource requests | Right-sizing, don't combine with HPA on same metric |
| 3 | Karpenter | Nodes | Workload-aware node provisioning, replaces Cluster Autoscaler |
| 4 | KEDA | Pod replicas | Event-driven (queue depth, Kafka lag, custom metrics) |

## GitOps Patterns

### ArgoCD Application

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: APP_NAME
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/ORG/REPO.git
    targetRevision: main
    path: k8s/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: APP_NAME
  # Belt-and-braces for charts that always render `replicas` and cannot omit it.
  # Needs BOTH of these: ignoreDifferences alone only silences the diff — Argo still
  # applies the desired state as-is during a sync, so replicas snap back on the next
  # commit or manual sync. RespectIgnoreDifferences pre-patches it out of the apply.
  # `name:` is REQUIRED here, not optional tidiness. Omit it and the rule matches EVERY
  # Deployment in this Application, so replica drift goes both undetected and unhealed
  # across all of them while the UI still reports Synced.
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: APP_NAME
      jsonPointers:
        - /spec/replicas
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
      - RespectIgnoreDifferences=true
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
```

### Flux Kustomization

```yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: APP_NAME
  namespace: flux-system
spec:
  interval: 5m
  path: ./k8s/overlays/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: APP_NAME
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: APP_NAME
      namespace: APP_NAME
```

Flux has no equivalent of Argo's `ignoreDifferences`: this Kustomization re-applies every field
that diverges from Git on each `interval`, so omitting `spec.replicas` from the Deployment is the
*only* way to let an HPA hold a replica count here. See the HPA warning above.

## Helm Chart Scaffold

```
mychart/
├── Chart.yaml
├── values.yaml
├── values-dev.yaml
├── values-staging.yaml
├── values-prod.yaml
├── templates/
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── hpa.yaml
│   ├── pdb.yaml
│   ├── networkpolicy.yaml
│   ├── serviceaccount.yaml
│   └── configmap.yaml
└── .helmignore
```

Always: pin chart versions, validate with `helm lint` and `helm template` in CI, use values files per environment.

## Troubleshooting Decision Trees

### Pod Not Starting

```
Pod status?
├── Pending
│   ├── Check events: kubectl describe pod POD
│   ├── "Insufficient cpu/memory" → check resource requests, node capacity
│   ├── "no nodes match" → check nodeSelector, tolerations, affinity
│   └── "PVC not bound" → check storage class, PV availability
├── CrashLoopBackOff  (the container keeps exiting non-zero; kubelet backs off
│   │                  10s → 20s → 40s …, capped at 300s, and resets the timer
│   │                  once the container has run 10 min without a problem)
│   ├── Check logs: kubectl logs POD --previous
│   ├── OOMKilled → increase memory limits
│   ├── Application error → fix code, check config
│   └── Liveness or startup probe failing → kubelet killed the container; compare
│       probe config against actual startup time (readiness CANNOT cause this)
├── ImagePullBackOff
│   ├── Check image name and tag exist
│   ├── Check imagePullSecrets for private registries
│   └── Check network access to registry
└── Error / Unknown
    ├── Check events: kubectl describe pod POD
    └── Check node status: kubectl get nodes
```

**Which probe can restart a container** — only two of the three can, and readiness is not
one of them ([pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/)):

| Probe | kubelet action on failure | Restarts? | What you observe |
|-------|---------------------------|-----------|------------------|
| liveness | Kills the container; the Pod's restart policy applies | Yes | RESTARTS climbing → CrashLoopBackOff while it keeps exiting |
| startup | Kills the container; runs before liveness/readiness begin | Yes | Same, but only during startup |
| readiness | Kills nothing — the EndpointSlice controller removes the Pod IP from every matching Service | **No** | `Running`, `READY 0/1`, `RESTARTS 0`, endpoints empty |

So `READY 0/1` with `RESTARTS 0` is a readiness failure, never CrashLoopBackOff — diagnose it
under "Service Not Reachable" below, not here. The backoff numbers above are the defaults; the
`ReduceDefaultCrashLoopBackOffDecay` (1s start, 60s cap) and `KubeletCrashLoopBackOffMax`
feature gates change them, so confirm against your cluster before quoting a timing to anyone.

### Service Not Reachable

```
1. Verify pod is Running: kubectl get pods -l app=APP_NAME
2. Verify service exists: kubectl get svc APP_NAME
3. Check endpoints: kubectl get endpoints APP_NAME
   └── Empty? Either the Service and Pod selectors don't match, or every Pod is
       failing its readiness probe (kubectl get pods → READY 0/1, RESTARTS 0)
4. Test from within cluster: kubectl run debug --rm -it --image=busybox -- wget -qO- http://APP_NAME:PORT
5. Check network policies: kubectl get networkpolicy -n NAMESPACE
6. Check ingress/load balancer: kubectl describe ingress APP_NAME
```

## Security Checklist

- [ ] Pods run as non-root (`runAsNonRoot: true`)
- [ ] Read-only root filesystem (`readOnlyRootFilesystem: true`)
- [ ] All capabilities dropped (`drop: ["ALL"]`)
- [ ] No privilege escalation (`allowPrivilegeEscalation: false`)
- [ ] Seccomp profile set (`seccompProfile: RuntimeDefault`)
- [ ] Resource requests AND limits set on every container
- [ ] NetworkPolicies enforce least-privilege network access
- [ ] PodDisruptionBudgets protect availability during disruptions
- [ ] ServiceAccounts use least-privilege RBAC
- [ ] Images from trusted registries only, scanned with Trivy
- [ ] Secrets mounted as volumes, not environment variables
- [ ] No `latest` image tag — use immutable digests or semantic versions