Kubernetes Operations

---
created: 2025-12-16
modified: 2026-09-24
reviewed: 2026-04-25
name: kubernetes-operations
description: "Kubernetes operations — deployment, management, troubleshooting, kubectl mastery. Use when the user mentions K8s, kubectl, pods, deployments, services, ingress, or cluster stability."
user-invocable: false
allowed-tools: Glob, Grep, Read, Bash(kubectl *), Bash(helm *), Bash(kustomize *), Edit, Write, TodoWrite, WebFetch
---

# Kubernetes Operations

Expert knowledge for Kubernetes cluster management, deployment, and troubleshooting with mastery of kubectl and cloud-native patterns.

## When to Use This Skill

| Use this skill when... | Use <sibling> instead when... |
|---|---|
| Working with kubectl against pods, deployments, services, ingress, ConfigMaps, or Secrets | Use kubectl-debugging when you specifically need `kubectl debug` ephemeral containers or node sessions |
| Applying or inspecting raw Kubernetes manifests and kustomize overlays | Use helm-release-management when the workload is delivered as a Helm chart |
| Diagnosing cluster-level networking, storage, or workload health | Use argocd-login when the issue is authenticating to ArgoCD before any cluster operation |

## Core Expertise

**Kubernetes Operations**
- **Workload Management**: Deployments, StatefulSets, DaemonSets, Jobs, and CronJobs
- **Networking**: Services, Ingress, NetworkPolicies, and DNS configuration
- **Configuration & Storage**: ConfigMaps, Secrets, PersistentVolumes, and PersistentVolumeClaims
- **Troubleshooting**: Debugging pods, analyzing logs, and inspecting cluster events

## Cluster Operations Process

1. **Manifest First**: Always prefer declarative YAML manifests for resource management
2. **Validate & Dry-Run**: Use `kubectl apply --dry-run=client` to validate changes
3. **Inspect & Verify**: After applying changes, verify with `kubectl get`, `kubectl describe`, `kubectl logs`
4. **Monitor Health**: Continuously check status of nodes, pods, and services
5. **Clean Up**: Ensure old or unused resources are properly garbage collected

## Essential Commands

```bash
# Resource management
kubectl apply -f manifest.yaml
kubectl get pods -A
kubectl describe pod <pod-name>
kubectl logs -f <pod-name>
kubectl exec -it <pod-name> -- /bin/bash

# Debugging
kubectl get events --sort-by='.lastTimestamp'
kubectl top nodes
kubectl top pods --containers
kubectl port-forward <pod-name> 8080:80

# Deployment management
kubectl rollout status deployment/<name>
kubectl rollout history deployment/<name>
kubectl rollout undo deployment/<name>

# Cluster inspection
kubectl cluster-info
kubectl get nodes -o wide
kubectl api-resources
```

## Key Debugging Patterns

**Pod Debugging**
```bash
# Pod inspection
kubectl describe pod <pod-name>
kubectl get pod <pod-name> -o yaml
kubectl logs <pod-name> --previous

# Interactive debugging
kubectl exec -it <pod-name> -- /bin/bash
kubectl debug <pod-name> -it --image=busybox
kubectl port-forward <pod-name> 8080:80
```

**Networking Troubleshooting**
```bash
# Service debugging
kubectl get svc -o wide
kubectl get endpoints
kubectl describe svc <service>

# Network connectivity
kubectl run test-pod --image=busybox -it --rm -- sh
# Inside pod: nslookup, wget, nc commands
```

**Common Issues**
```bash
# CrashLoopBackOff debugging
kubectl logs <pod> --previous
kubectl describe pod <pod>
kubectl get events --field-selector involvedObject.name=<pod>

# Resource constraints
kubectl top pod <pod>
kubectl describe pod <pod> | grep -A 5 Limits

# State management
kubectl state list
kubectl state show <resource>
```

## Best Practices

**Context Safety (CRITICAL)**
- **Always specify `--context`** explicitly in every kubectl command
- Never rely on the current context - it may have been changed by another process
- Use `kubectl --context=<context-name> get pods` format for all operations
- This prevents accidental operations on the wrong cluster (e.g., running production commands against staging)

```bash
# CORRECT: Explicit context
kubectl --context=gke_myproject_us-central1_prod get pods
kubectl --context=staging-cluster apply -f deployment.yaml

# WRONG: Relying on current context
kubectl get pods  # Which cluster is this targeting?
```

**GKE: kubectl 403 as the wrong Google account**

`gke-gcloud-auth-plugin` caches its token in
`~/.kube/gke_gcloud_auth_plugin_cache`, and that cache survives a
`gcloud auth login` account switch. kubectl keeps authenticating as the old
account (often a personal one) and gets 403s. Clear the cache, then refresh the
credentials:

```bash
rm -f ~/.kube/gke_gcloud_auth_plugin_cache
gcloud container clusters get-credentials <cluster> --region <region> --project <project>
```

`gcloud auth application-default login` sets Application Default Credentials for
SDKs and Terraform. It does not change the account kubectl uses.

**Resource Definitions**
- Use declarative YAML manifests
- Implement proper labels and selectors
- Define resource requests and limits
- Configure health checks (liveness/readiness probes)

**Security**
- Use NetworkPolicies to restrict traffic
- Implement RBAC for access control
- Store sensitive data in Secrets
- Run containers as non-root users

**Monitoring**
- Configure proper logging and metrics
- Set up alerts for critical conditions
- Use health checks and readiness probes
- Monitor resource usage and quotas

## Agentic Optimizations

| Context | Command |
|---------|---------|
| Pod status (structured) | `kubectl get pods -n <ns> -o json \| jq '.items[] \| {name:.metadata.name, status:.status.phase}'` |
| Quick overview | `kubectl get pods -n <ns> -o wide` |
| Events (compact) | `kubectl get events -n <ns> --sort-by='.lastTimestamp' -o json` |
| Resource details | `kubectl get <resource> -o json` |
| Logs (bounded) | `kubectl logs <pod> -n <ns> --tail=50` |

For detailed debugging commands, troubleshooting patterns, Helm workflows, and advanced K8s operations, see REFERENCE.md.