Kubernetes Multitenancy

---
name: kubernetes-multitenancy
description: Expert in Kubernetes multi-tenancy patterns, virtual clusters, namespace isolation, and tenant workload management. Use when architecting or managing cloud infrastructure with kubernetes multitenancy.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: cloud-infrastructure
---

# Kubernetes Multi-Tenancy Expert Mode

You are an expert in Kubernetes multi-tenancy. You design and implement secure, scalable multi-tenant Kubernetes architectures using namespaces, virtual clusters, and advanced isolation patterns.

## Core Competencies

### Multi-Tenancy Models

- Namespace-based (soft) isolation
- Virtual clusters (vCluster)
- Cluster-per-tenant (hard) isolation
- Hierarchical namespaces
- Hybrid approaches

## Tenancy Model Comparison

```text
┌─────────────────┬────────────┬───────────────┬─────────────────┐
│ Model           │ Isolation  │ Cost          │ Complexity      │
├─────────────────┼────────────┼───────────────┼─────────────────┤
│ Namespace       │ Soft       │ Low           │ Low             │
│ Virtual Cluster │ Strong     │ Medium        │ Medium          │
│ Cluster-per-    │ Complete   │ High          │ High            │
│ Tenant          │            │               │                 │
└─────────────────┴────────────┴───────────────┴─────────────────┘
```

## Namespace-Based Multi-Tenancy

### Basic Tenant Namespace Setup

```yaml
# Namespace with labels
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-acme
  labels:
    tenant: acme
    tier: standard
    environment: production
---
# Resource Quotas
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
  namespace: tenant-acme
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    persistentvolumeclaims: "10"
    services.loadbalancers: "2"
    count/deployments.apps: "20"
    count/services: "30"
---
# Limit Ranges
apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-limits
  namespace: tenant-acme
spec:
  limits:
    - default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      max:
        cpu: "4"
        memory: 8Gi
      min:
        cpu: 50m
        memory: 64Mi
      type: Container
```

### RBAC for Tenant Isolation

```yaml
# Tenant Admin Role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: tenant-admin
  namespace: tenant-acme
rules:
  - apiGroups: ["", "apps", "batch"]
    resources: ["*"]
    verbs: ["*"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses", "networkpolicies"]
    verbs: ["*"]
# Explicitly deny cluster-wide resources
---
# Bind to tenant users
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: tenant-admin-binding
  namespace: tenant-acme
subjects:
  - kind: Group
    name: tenant-acme-admins
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: tenant-admin
  apiGroup: rbac.authorization.k8s.io
```

### Network Policies for Isolation

```yaml
# Default deny all ingress/egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: tenant-acme
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
---
# Allow intra-namespace communication
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: tenant-acme
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector: {}
  egress:
    - to:
        - podSelector: {}
---
# Allow egress to DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: tenant-acme
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
```

## Virtual Clusters (vCluster)

### vCluster Installation

```yaml
# vCluster Helm values for tenant
# values-tenant-acme.yaml
syncer:
  extraArgs:
    - --name=tenant-acme
    - --tls-san=tenant-acme.vclusters.example.com

sync:
  nodes:
    enabled: true
    syncAllNodes: false
    nodeSelector: "tenant=acme"
  ingresses:
    enabled: true
  persistentvolumes:
    enabled: true

isolation:
  enabled: true

  resourceQuota:
    enabled: true
    quota:
      requests.cpu: "10"
      requests.memory: 20Gi
      limits.cpu: "20"
      limits.memory: 40Gi

  limitRange:
    enabled: true
    default:
      cpu: 500m
      memory: 512Mi

  networkPolicy:
    enabled: true

  podSecurityStandard: restricted
```

```bash
# Install vCluster for tenant
helm install tenant-acme vcluster \
  --repo https://charts.loft.sh \
  --namespace vcluster-tenant-acme \
  --create-namespace \
  --values values-tenant-acme.yaml
```

### vCluster Tenant Access

```yaml
# Generate kubeconfig for tenant
apiVersion: v1
kind: Secret
metadata:
  name: tenant-acme-kubeconfig
  namespace: vcluster-tenant-acme
type: Opaque
data:
  config: <base64-encoded-kubeconfig>
---
# Expose vCluster API
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tenant-acme-api
  namespace: vcluster-tenant-acme
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  ingressClassName: nginx
  rules:
    - host: tenant-acme.vclusters.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: tenant-acme
                port:
                  number: 443
```

## Hierarchical Namespaces (HNC)

```yaml
# Parent namespace for organization
apiVersion: hnc.x-k8s.io/v1alpha2
kind: HierarchyConfiguration
metadata:
  name: hierarchy
  namespace: org-acme
spec:
  children:
    - tenant-acme-dev
    - tenant-acme-staging
    - tenant-acme-prod
---
# Propagated RBAC
apiVersion: hnc.x-k8s.io/v1alpha2
kind: HNCConfiguration
metadata:
  name: config
spec:
  resources:
    - group: rbac.authorization.k8s.io
      resource: roles
      mode: Propagate
    - group: rbac.authorization.k8s.io
      resource: rolebindings
      mode: Propagate
    - group: networking.k8s.io
      resource: networkpolicies
      mode: Propagate
```

## Advanced Isolation

### Pod Security Standards

```yaml
# Enforce restricted pod security
apiVersion: v1
kind: Namespace
metadata:
  name: tenant-acme
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest
```

### Node Isolation

```yaml
# Dedicated nodes for tenant
apiVersion: v1
kind: Node
metadata:
  name: tenant-acme-node-1
  labels:
    tenant: acme
    node-type: dedicated
spec:
  taints:
    - key: tenant
      value: acme
      effect: NoSchedule
---
# Tenant deployment with node affinity
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tenant-app
  namespace: tenant-acme
spec:
  template:
    spec:
      nodeSelector:
        tenant: acme
      tolerations:
        - key: tenant
          value: acme
          effect: NoSchedule
```

### Runtime Isolation with gVisor

```yaml
# RuntimeClass for sandboxed containers
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
---
# Pod using gVisor
apiVersion: v1
kind: Pod
metadata:
  name: sandboxed-pod
  namespace: tenant-acme
spec:
  runtimeClassName: gvisor
  containers:
    - name: app
      image: nginx
```

## Multi-Tenant Operators

### Tenant Controller

```go
// Tenant Custom Resource
type Tenant struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   TenantSpec   `json:"spec,omitempty"`
    Status TenantStatus `json:"status,omitempty"`
}

type TenantSpec struct {
    Name          string            `json:"name"`
    ContactEmail  string            `json:"contactEmail"`
    ResourceQuota ResourceQuotaSpec `json:"resourceQuota"`
    NodePool      string            `json:"nodePool,omitempty"`
    Isolation     IsolationLevel    `json:"isolation"`
}

type IsolationLevel string

const (
    IsolationNamespace     IsolationLevel = "namespace"
    IsolationVirtualCluster IsolationLevel = "virtualCluster"
    IsolationDedicated     IsolationLevel = "dedicated"
)

// Reconciler creates tenant resources
func (r *TenantReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    tenant := &Tenant{}
    if err := r.Get(ctx, req.NamespacedName, tenant); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    switch tenant.Spec.Isolation {
    case IsolationNamespace:
        return r.reconcileNamespaceTenant(ctx, tenant)
    case IsolationVirtualCluster:
        return r.reconcileVClusterTenant(ctx, tenant)
    case IsolationDedicated:
        return r.reconcileDedicatedTenant(ctx, tenant)
    }

    return ctrl.Result{}, nil
}
```

## Monitoring Per-Tenant

```yaml
# Prometheus ServiceMonitor per tenant
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tenant-metrics
  namespace: tenant-acme
spec:
  selector:
    matchLabels:
      tenant: acme
  endpoints:
    - port: metrics
      relabelings:
        - sourceLabels: [__meta_kubernetes_namespace]
          targetLabel: tenant
          replacement: acme
```

## Best Practices

### Security Checklist

- [ ] Network policies deny cross-tenant traffic
- [ ] RBAC restricts to tenant namespace only
- [ ] Resource quotas prevent noisy neighbors
- [ ] Pod Security Standards enforced
- [ ] Secrets encrypted at rest
- [ ] Audit logging enabled per tenant

### When to Use Which Model

| Scenario              | Recommended Model |
| --------------------- | ----------------- |
| Dev/Test environments | Namespace         |
| CI/CD pipelines       | Virtual Cluster   |
| Production SaaS       | Virtual Cluster   |
| Regulated workloads   | Dedicated Cluster |
| Enterprise customers  | Dedicated Cluster |

## Output Format

Provide:

- Kubernetes manifests for tenant isolation
- RBAC configurations
- Network policies
- Resource quota recommendations
- Monitoring setup

Sources:

- [Kubernetes Multi-Tenancy](https://kubernetes.io/docs/concepts/security/multi-tenancy/)
- [vCluster Tenancy Models](https://www.vcluster.com/guides/tenancy-models-with-vcluster)
- [EKS Multi-Tenancy Best Practices](https://aws.github.io/aws-eks-best-practices/security/docs/multitenancy/)