Container Security

---
name: container-security
description: "Routing and expertise agent for container and Kubernetes security. Covers image scanning, admission control (OPA Gatekeeper, Kyverno), Pod Security Standards, runtime protection, RBAC, network policies, secrets management, supply chain security (cosign/SLSA), and service mesh mTLS. WHEN: \"container security\", \"Kubernetes security\", \"K8s security\", \"image scanning\", \"admission control\", \"OPA Gatekeeper\", \"Kyverno\", \"Pod Security Standards\", \"RBAC Kubernetes\", \"Kubernetes network policy\", \"supply chain security\", \"container runtime security\". Do NOT use for platform-specific questions -- use the `aqua`, `falco`, or `sysdig` skill."
license: MIT
---

# Container and Kubernetes Security

This skill covers container and Kubernetes security. It provides deep knowledge of container security concepts, Kubernetes security architecture, and the tools used to secure containerized workloads across the full lifecycle — from build to runtime.

## When to Use This Skill vs. a Technology Skill

**Use this skill for concepts and cross-tool questions:**
- "How does image scanning work?"
- "What is the difference between OPA Gatekeeper and Kyverno?"
- "How do I implement Pod Security Standards?"
- "Design a container security strategy for our Kubernetes environment"
- "How do I secure Kubernetes RBAC?"
- "What is supply chain security for containers?"
- "What's the difference between Aqua, Sysdig, and Falco?"

**Read a sibling skill for platform-specific guidance:**
- "Configure Aqua image scanning policies and vShield" --> `aqua`
- "Write Sysdig alerts for container runtime anomalies" --> `sysdig`
- "Author Falco rules for syscall monitoring" --> `falco`

## How to Approach Tasks

When you receive a request:

1. **Classify** the lifecycle phase:
   - **Build security** -- Image scanning, base image selection, Dockerfile best practices, SCA, secrets in images
   - **Registry security** -- Registry authentication, image signing, policy enforcement at registry
   - **Admission control** -- OPA Gatekeeper/Kyverno policies enforced at deployment time
   - **Runtime security** -- Syscall monitoring, behavioral detection, anomaly detection in running containers
   - **Infrastructure (KSPM)** -- Kubernetes configuration, RBAC, network policies, cluster hardening
   - **Supply chain** -- SLSA, Sigstore, cosign, SBOM, provenance
   - **Secrets management** -- Secrets in K8s, external secrets operator, sealed secrets, Vault

2. **Identify environment** -- Managed Kubernetes (EKS, AKS, GKE) or self-managed? Number of clusters? What's already deployed (existing tools, admission controllers, CNI)? Compliance requirements?

3. **Load context** -- Read `references/concepts.md` for deep conceptual knowledge.

4. **Analyze** -- Apply container security reasoning. Container security differs from VM security: immutable infrastructure, ephemeral containers, sidecar patterns, orchestrator-level controls.

5. **Recommend** -- Provide layered, defense-in-depth recommendations across the full lifecycle.

## Container Security Lifecycle

### 1. Build Phase Security

**Base image selection:**
- Use minimal base images (distroless, scratch, Alpine) to reduce attack surface
- Pin to specific digest (not `latest` or mutable tags): `FROM ubuntu@sha256:abc123...`
- Use only images from trusted registries with content trust enabled
- Prefer official images from Docker Hub verified publishers or vendor-provided images

**Dockerfile best practices:**
```dockerfile
# Good practices
FROM gcr.io/distroless/java17-debian11:nonroot  # minimal + nonroot

# Drop capabilities at build time if using non-distroless
USER nonroot:nonroot

# Don't store secrets in build args or env vars
# BAD:
ARG DB_PASSWORD
ENV DB_PASS=${DB_PASSWORD}

# GOOD: Use secrets mounts (build-time only, not in image)
RUN --mount=type=secret,id=db_password,target=/run/secrets/db_password \
    do-something-with-secret

# Multi-stage builds to minimize final image size
FROM golang:1.21 AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o app .

FROM gcr.io/distroless/static-debian11
COPY --from=builder /app/app /app
CMD ["/app"]
```

**Secrets in images:**
- NEVER store secrets in image layers (even if deleted in later layer — history persists)
- Scan images for secrets before pushing (Trivy, Wiz CLI, Prisma Cloud, Aqua)
- Use runtime secrets injection: Kubernetes Secrets, Vault Agent Sidecar, AWS Secrets Manager

### 2. Image Scanning

**What image scanning detects:**
- **OS package vulnerabilities:** CVEs in DEB/RPM/APK packages (libc, openssl, curl, etc.)
- **Language library vulnerabilities:** npm, pip, gem, Maven, NuGet, Cargo vulnerabilities
- **Hardcoded secrets:** API keys, tokens, passwords in image layers
- **Misconfigurations:** Running as root, capabilities granted, SUID binaries
- **License compliance:** GPL, LGPL licenses in image dependencies
- **Malware:** Known malicious binaries in the image filesystem

**Scanning tools:**

| Tool | Type | Notes |
|---|---|---|
| Trivy | Open source (Aqua) | Comprehensive, fast, widely adopted; integrates with many platforms |
| Grype (Anchore) | Open source | Alternative to Trivy; strong language package detection |
| Snyk Container | Commercial | Strong developer experience, CI/CD focus |
| AWS Inspector | AWS-native | Scans ECR images; integrates with Security Hub |
| Wiz CLI | Commercial (Wiz) | Part of Wiz platform; used in CI/CD |
| Aqua Trivy/Scanner | Commercial (Aqua) | Enterprise version with policies and reporting |
| Sysdig Image Scanner | Commercial (Sysdig) | Part of Sysdig Secure; CI/CD integration |

**Trivy quick reference:**
```bash
# Scan an image
trivy image nginx:latest

# Scan with severity filter
trivy image --severity CRITICAL,HIGH nginx:latest

# Scan for secrets
trivy image --scanners secret nginx:latest

# Scan for misconfigs
trivy image --scanners misconfig nginx:latest

# Output formats: table, json, sarif, cyclonedx (SBOM), spdx (SBOM)
trivy image --format json --output results.json nginx:latest

# Scan filesystem (CI/CD)
trivy fs --scanners vuln,secret,config .

# Scan IaC directory
trivy config ./kubernetes/
```

**CI/CD gate policies:**
Define thresholds for blocking deployments:
- Block on CRITICAL CVEs with public exploits
- Warn on HIGH CVEs
- Allow with exception if no fix available and risk accepted
- Block if hardcoded secrets detected (no threshold — always block)

### 3. Admission Control

Admission controllers intercept Kubernetes API requests at deployment time and can validate, mutate, or reject workload specifications.

**Built-in admission controllers (no extra deployment):**
- `LimitRanger` — Enforces resource limits on pods
- `ResourceQuota` — Enforces namespace-level resource quotas
- `PodSecurity` — Enforces Pod Security Standards (replaces PodSecurityPolicy, deprecated in 1.21, removed in 1.25)
- `NodeRestriction` — Limits what kubelets can modify

**Pod Security Standards (PSS) — built-in admission:**

Three enforcement levels:
```yaml
# Applied per namespace via labels
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    # enforce: fail creation of pods violating this profile
    pod-security.kubernetes.io/enforce: restricted
    # audit: log violations but don't reject
    pod-security.kubernetes.io/audit: restricted
    # warn: warn users but don't reject
    pod-security.kubernetes.io/warn: restricted
```

**Pod Security Standard profiles:**

| Profile | Description | Restrictions |
|---|---|---|
| Privileged | Unrestricted | No restrictions |
| Baseline | Minimally restrictive | Blocks known privilege escalations; allows many defaults |
| Restricted | Heavily restricted | Requires: non-root user, no privilege escalation, drop capabilities, seccomp |

**OPA Gatekeeper:** Policy enforcement via Open Policy Agent (Rego language) — a `ConstraintTemplate` defines policy logic, a `Constraint` applies it with parameters.

**Kyverno:** Policy enforcement with Kubernetes-native YAML syntax (no Rego needed) — supports `validate` (block) and `mutate` (patch) rules.

See `references/admission-examples.md` for full OPA Gatekeeper ConstraintTemplate/Constraint YAML and Kyverno validate/mutate policy examples.

**Gatekeeper vs. Kyverno:**

| Aspect | OPA Gatekeeper | Kyverno |
|---|---|---|
| Policy language | Rego (powerful but complex) | YAML/JSON (Kubernetes-native) |
| Learning curve | Steep (Rego requires learning) | Gentle (familiar YAML) |
| Mutation support | Limited (via assign mutations) | Strong (strategic merge patch, JSON patch) |
| Audit mode | Yes | Yes |
| Ecosystem | Extensive Rego library | Growing Kyverno policy library |
| Generate resources | No | Yes (create resources as side effect) |
| Best for | Complex policy logic, existing OPA investment | Kubernetes-native teams, simpler policies |

### 4. Kubernetes RBAC

Role-Based Access Control is Kubernetes' authorization mechanism.

**RBAC objects:**
- `Role` — Grants permissions within a single namespace
- `ClusterRole` — Grants permissions cluster-wide or to non-namespaced resources
- `RoleBinding` — Binds a Role or ClusterRole to users/groups/service accounts within a namespace
- `ClusterRoleBinding` — Binds a ClusterRole cluster-wide

**RBAC best practices:**
```yaml
# Least privilege: specific verbs, specific resources, specific resource names
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: my-app
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
  # Restrict to specific pods if possible:
  resourceNames: ["my-specific-pod"]

---
# Service account per workload (not shared)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app-sa
  namespace: my-app
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-app-role  # IRSA for AWS
```

**Common RBAC security mistakes:**
- `cluster-admin` bound to service accounts or developers
- `wildcards` in verbs or resources (`*`)
- `secrets` get/list access to service accounts that don't need it
- Shared service accounts across multiple workloads
- ClusterRoleBindings instead of namespace-scoped RoleBindings

**RBAC audit query:**
```bash
# Find all cluster-admin bindings
kubectl get clusterrolebindings -o json | \
  jq '.items[] | select(.roleRef.name == "cluster-admin") | {name: .metadata.name, subjects: .subjects}'

# Find all subjects with secrets access
kubectl auth can-i list secrets --as system:serviceaccount:my-namespace:my-sa
```

### 5. Network Policies

Kubernetes Network Policies control pod-to-pod and pod-to-external traffic at the network layer. Requires a CNI that supports network policies (Calico, Cilium, Weave, Canal).

**Default deny all traffic (recommended starting point):**
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: my-app
spec:
  podSelector: {}    # selects all pods in namespace
  policyTypes:
  - Ingress
  - Egress
```

**Allow specific ingress:**
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: my-app
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
```

**Cilium network policies (eBPF-based, richer policy):**
- Layer 7 (HTTP) policy enforcement
- DNS-based policy (allow outbound to specific FQDNs)
- Identity-based policy (not just label-based)
- Better observability (Hubble for network flow visualization)

### 6. Secrets Management in Kubernetes

**Kubernetes Secrets (native):**
- Base64-encoded (NOT encrypted) by default in etcd
- Enable etcd encryption at rest (EncryptionConfiguration)
- RBAC: restrict `get`/`list` secrets to only pods that need them
- Environment variable injection exposes to all processes in container — prefer volume mounts

**External Secrets Operator (ESO):**
Syncs secrets from external stores (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, GCP Secret Manager) into Kubernetes Secrets:
```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-app-secrets
  namespace: my-app
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: my-app-secret    # creates/updates this K8s Secret
  data:
  - secretKey: database-password
    remoteRef:
      key: my-app/prod/database
      property: password
```

**Sealed Secrets (Bitnami):**
Encrypts secrets before storing in Git (GitOps-friendly):
```bash
# Encrypt a secret with the cluster's public key
kubectl create secret generic my-secret --dry-run=client -o yaml | \
  kubeseal --format yaml > sealed-secret.yaml

# Commit sealed-secret.yaml to Git (safe to store in version control)
# SealedSecrets controller in cluster decrypts and creates the K8s Secret
```

**Vault Agent Sidecar / Vault Secrets Operator:**
Injects secrets from HashiCorp Vault directly into pods without touching Kubernetes Secrets.

### 7. Runtime Protection

Container runtime security monitors behavior inside running containers:

**What runtime security detects:**
- Shell spawned in container (unexpected bash/sh execution)
- Unexpected process execution (binary not in allowed list)
- Sensitive file access (/etc/shadow, /etc/passwd, SSH keys)
- Outbound network connections to unexpected destinations
- Container escape attempts (namespace manipulation, dangerous syscalls)
- Privilege escalation attempts (setuid, capability abuse)
- Crypto mining patterns (CPU usage + network to mining pools)

**Tools:** Falco (open source CNCF), Aqua Enforcer, Sysdig Secure, Prisma Cloud Defender, Wiz Defend, Datadog Cloud Security

**seccomp profiles:**
```yaml
# Apply seccomp profile to pod
spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault    # use container runtime's default seccomp
      # type: Localhost        # use a custom profile from node
      # localhostProfile: profiles/fine-grained.json
```

**AppArmor:**
```yaml
# Apply AppArmor profile to container
metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/my-container: runtime/default
```

### 8. Supply Chain Security

**SLSA Framework (Supply-chain Levels for Software Artifacts):**
Four levels of supply chain integrity:
- Level 1: Provenance generated by build system
- Level 2: Build service generates signed provenance
- Level 3: Build environment is hardened, ephemeral
- Level 4: All dependencies verified at Level 3 or higher

**Sigstore / cosign (image signing):**
```bash
# Sign an image with cosign
cosign sign --key cosign.key registry.example.com/myapp:v1.0.0

# Verify an image signature
cosign verify --key cosign.pub registry.example.com/myapp:v1.0.0

# Keyless signing via OIDC (Sigstore Fulcio CA)
# No key management needed — signs using GitHub Actions identity
cosign sign --identity-token $ACTIONS_ID_TOKEN_REQUEST_TOKEN \
  registry.example.com/myapp:v1.0.0
```

**SBOM (Software Bill of Materials):**
```bash
# Generate SBOM with Syft
syft registry.example.com/myapp:v1.0.0 -o spdx-json > sbom.spdx.json

# Generate SBOM with Trivy
trivy image --format spdx-json --output sbom.spdx registry.example.com/myapp:v1.0.0

# Attach SBOM to image with cosign
cosign attach sbom --sbom sbom.spdx.json registry.example.com/myapp:v1.0.0
```

**Policy enforcement with Sigstore (Kyverno):** A `verifyImages` rule can require images from a given registry to carry a valid cosign signature before admission. See `references/admission-examples.md` for the full `require-signed-images` ClusterPolicy example.

## Subcategory Routing

| Request Pattern | Route To |
|---|---|
| Aqua Security platform, Aqua Enterprise, vShield, DTA, Aqua Trivy | `aqua` |
| Sysdig Secure, Sysdig runtime, CDR, Sysdig posture, Sysdig monitor | `sysdig` |
| Falco rules, eBPF, Falcosidekick, falco-talon, Falco plugins | `falco` |

## Reference Files

Load for deep conceptual knowledge:

- `references/concepts.md` -- Container security fundamentals: full detail on image scanning, admission control, runtime protection, Pod Security Standards, RBAC security, network policies, secrets management, service mesh security, and supply chain security.
- `references/admission-examples.md` -- Full OPA Gatekeeper ConstraintTemplate/Constraint YAML, Kyverno validate/mutate ClusterPolicy examples, and a Sigstore image-signature-verification policy.