Consul

---
name: consul
description: "Expert coverage of Consul Connect service mesh: Consul Dataplane, intentions, ServiceRouter, ServiceSplitter, mesh gateways, multi-datacenter federation, transparent proxy, and hybrid platform support (Kubernetes + VMs). Use for \"Consul Connect\", \"Consul mesh\", \"intentions\", \"ServiceRouter\", \"Consul Dataplane\", \"mesh gateway\", \"Consul service mesh\", \"Consul on Kubernetes\". Do NOT use for HashiCorp Vault secrets management, dynamic secrets, or PKI/certificate issuance -- that's the `vault` skill (in the `security` plugin), even though both are HashiCorp products often deployed together."
license: MIT
---

# Consul Connect

This skill covers Consul Connect, HashiCorp Consul's service mesh capability. Consul Connect is unique among service meshes: it works across Kubernetes, VMs, bare metal, and cloud-native services, making it the choice for hybrid and multi-platform environments. Covers:

- Consul Dataplane (lightweight sidecar, replaced client agent model)
- Intentions (service-to-service access control at L4 and L7)
- L7 traffic management (ServiceRouter, ServiceSplitter, ServiceResolver)
- Mesh gateways for multi-datacenter and multi-cluster communication
- Transparent proxy (automatic traffic interception on Kubernetes)
- Consul server cluster (Raft consensus, service catalog, KV store)
- Vault integration for CA and secrets management
- Kubernetes Helm deployment and CRD-based configuration

## How to Approach Tasks

1. **Classify** the request:
   - **Access control** -- Intentions (L4/L7 service-to-service authorization)
   - **Traffic management** -- ServiceRouter, ServiceSplitter, ServiceResolver patterns
   - **Architecture** -- Load `references/architecture.md` for Dataplane, mesh gateways, multi-DC
   - **Installation** -- Helm chart configuration, Kubernetes CRDs
   - **Multi-platform** -- VM enrollment, catalog sync, hybrid mesh

2. **Determine platform** -- Is the user on Kubernetes, VMs, bare metal, or a mix? This changes the deployment model significantly.

3. **Load context** -- Read `references/architecture.md` for deep architectural knowledge.

4. **Analyze** -- Apply Consul-specific reasoning. Consul is not Kubernetes-only like Istio or Linkerd.

5. **Recommend** -- Provide actionable guidance with Helm values, CRD manifests, and CLI commands.

6. **Verify** -- Suggest validation steps (`consul intention check`, Consul UI, proxy status).

## Core Architecture

```
Consul Server Cluster (3 or 5 nodes, Raft consensus)
  |-- Service Catalog: registry of all services and their health
  |-- Intentions: access control rules (L4/L7)
  |-- Configuration: centralized KV store, ServiceRouter, ServiceSplitter
  |-- Certificate Authority: built-in CA or Vault integration
  |-- Service Discovery: DNS and HTTP API
  |
  Per Service Instance (Kubernetes or VM):
    Consul Dataplane (lightweight sidecar process)
    |-- Manages local Envoy proxy configuration
    |-- Talks to Consul servers via xDS and gRPC
    |-- No full Consul client agent needed (since 1.14+)
    |
    Envoy Proxy (sidecar)
    |-- Enforces intentions (L4 and L7)
    |-- Handles mTLS (certificate exchange, encryption)
    |-- Reports telemetry (Prometheus metrics, access logs)
    |-- Applies L7 routing rules
```

### Consul Dataplane vs Legacy Client Agent

| Aspect | Client Agent (Legacy) | Consul Dataplane |
|---|---|---|
| Architecture | Full Consul agent per node | Lightweight process per service |
| Resource usage | High (gossip, health checks, cache) | Low (gRPC to servers only) |
| Networking | Agent port exposure (8301, 8500, etc.) | gRPC only |
| Kubernetes | DaemonSet of agents | Sidecar per pod |
| Startup time | Slower (gossip join, sync) | Fast (direct gRPC) |
| Introduced | Original | Consul 1.14+ |

**Recommendation**: Always use Consul Dataplane for new deployments. Client agent model is legacy.

## Installation (Kubernetes)

```bash
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install consul hashicorp/consul \
  --namespace consul \
  --create-namespace \
  --values consul-values.yaml
```

### Production Helm Values

```yaml
global:
  name: consul
  datacenter: dc1
  image: hashicorp/consul:1.20
  tls:
    enabled: true
    enableAutoEncrypt: true
  acls:
    manageSystemACLs: true
  metrics:
    enabled: true
    enableAgentMetrics: true
    enableGatewayMetrics: true

server:
  replicas: 3
  storage: 20Gi
  storageClass: fast-ssd
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: 2000m
      memory: 2Gi

connectInject:
  enabled: true
  default: true              # auto-inject sidecars into all pods
  transparentProxy:
    defaultEnabled: true     # capture all traffic via iptables
  metrics:
    defaultEnabled: true
    defaultPrometheusScrapePort: 20200
  consulNamespaces:
    mirrorK8S: true          # mirror K8s namespaces to Consul

meshGateway:
  enabled: true
  replicas: 2
  service:
    type: LoadBalancer
  wanAddress:
    source: Service

ingressGateway:
  enabled: true
  defaults:
    replicas: 2
    service:
      type: LoadBalancer

terminatingGateway:
  enabled: true
  defaults:
    replicas: 1

ui:
  enabled: true
  service:
    type: ClusterIP

dns:
  enabled: true
```

## Intentions (Access Control)

Intentions are Consul's service-to-service authorization rules.

### L4 Intentions (Connection-Level)

```bash
# CLI: Allow frontend to call api
consul intention create -allow frontend api

# CLI: Default deny all
consul intention create -deny '*' '*'

# List intentions
consul intention list
```

### L7 Intentions (Request-Level)

```yaml
# CRD: L7 intention with HTTP permissions
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: api
  namespace: production
spec:
  destination:
    name: api
  sources:
  - name: frontend
    permissions:
    - action: allow
      http:
        methods: ["GET"]
        pathPrefix: /api/v1/
    - action: allow
      http:
        methods: ["POST"]
        pathPrefix: /api/v1/orders
    - action: deny       # deny everything else from frontend

  - name: admin-service
    action: allow         # full access for admin

  - name: '*'
    action: deny          # deny all other services
```

**L4 vs L7**: L4 intentions evaluate per-connection (allow/deny the TCP connection). L7 intentions evaluate per-request (allow/deny based on HTTP method, path, headers). L7 requires `protocol: http` in ServiceDefaults.

### Intention Precedence

1. **Exact source + exact destination** (most specific)
2. **Exact source + wildcard destination**
3. **Wildcard source + exact destination**
4. **Wildcard source + wildcard destination** (least specific, default deny)

## L7 Traffic Management

### ServiceDefaults (Protocol Configuration)

```yaml
# Required: set protocol for L7 features
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: api
spec:
  protocol: http          # http | grpc | tcp | http2
  meshGateway:
    mode: local           # local | remote | none
  expose:
    checks: true          # expose health check endpoints
  maxInboundConnections: 1000
```

### ServiceRouter (Path-Based Routing)

```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
  name: api
spec:
  routes:
  - match:
      http:
        pathPrefix: /api/v2
    destination:
      service: api-v2
      requestTimeout: 10s
      numRetries: 3
      retryOnStatusCodes: [503]

  - match:
      http:
        header:
        - name: x-canary
          exact: "true"
    destination:
      service: api-canary

  # Default route (no match = catch-all)
  - destination:
      service: api
```

### ServiceSplitter (Traffic Splitting)

```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceSplitter
metadata:
  name: api
spec:
  splits:
  - weight: 90
    service: api
    serviceSubset: v1
  - weight: 10
    service: api
    serviceSubset: v2
```

### ServiceResolver (Subsets and Failover)

```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceResolver
metadata:
  name: api
spec:
  defaultSubset: v1
  subsets:
    v1:
      filter: "Service.Meta.version == v1"
    v2:
      filter: "Service.Meta.version == v2"
  failover:
    '*':
      datacenters: ["dc2", "dc3"]
  connectTimeout: 5s
  requestTimeout: 10s
```

### L7 Traffic Chain

```
Request --> ServiceRouter (path/header matching)
              |
              v
           ServiceSplitter (weight-based distribution)
              |
              v
           ServiceResolver (subset selection, failover)
              |
              v
           Endpoint (actual service instance)
```

## Multi-Datacenter / Multi-Cluster

### WAN Federation

Consul natively supports multi-datacenter federation:

```yaml
# consul-values.yaml for federated cluster
global:
  datacenter: dc2
  tls:
    enabled: true
    caCert:
      secretName: consul-federation    # shared CA across DCs
  federation:
    enabled: true
    primaryDatacenter: dc1

server:
  extraVolumes:
  - type: secret
    name: consul-federation
    load: true
```

### Mesh Gateways

Mesh gateways route mTLS traffic between datacenters:

```
DC1:  Service A --> Envoy sidecar --> Mesh Gateway (DC1)
                                          |
                                    (mTLS over WAN)
                                          |
DC2:                                 Mesh Gateway (DC2) --> Envoy sidecar --> Service B
```

Gateway modes:
- **local**: Traffic exits through the local datacenter's mesh gateway (default)
- **remote**: Traffic enters through the remote datacenter's mesh gateway
- **none**: Direct pod-to-pod (requires flat network)

### Service Discovery Across DCs

```bash
# DNS query for service in another datacenter
dig @consul-dns api.service.dc2.consul

# HTTP API
curl http://consul:8500/v1/health/service/api?dc=dc2
```

## Gateway Types

| Gateway | Purpose | Use Case |
|---|---|---|
| Mesh Gateway | Cross-datacenter service mesh traffic | Multi-DC communication |
| Ingress Gateway | Expose mesh services to external clients | External access without sidecar |
| Terminating Gateway | Allow meshed services to reach external services | Database, API outside mesh |

### Terminating Gateway

```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: TerminatingGateway
metadata:
  name: terminating-gateway
spec:
  services:
  - name: external-db
    caFile: /consul/tls/ca.pem
  - name: external-api
```

## Transparent Proxy (Kubernetes)

Transparent proxy captures all TCP traffic from pods via iptables, routing it through the Envoy sidecar without application changes:

```yaml
connectInject:
  transparentProxy:
    defaultEnabled: true
```

Benefits:
- No service URL changes (applications use Kubernetes DNS as normal)
- All traffic is automatically encrypted with mTLS
- Intentions are enforced on all connections, not just explicitly configured ones

## Vault Integration

```yaml
# Use Vault as CA instead of built-in
global:
  secretsBackend:
    vault:
      enabled: true
      consulServerRole: consul-server
      consulClientRole: consul-client
      connectCA:
        address: https://vault.example.com
        rootPKIPath: connect-root
        intermediatePKIPath: connect-intermediate
        authMethodPath: kubernetes
```

## Common Pitfalls

1. **Forgetting ServiceDefaults protocol**: L7 intentions and routing require `protocol: http` in ServiceDefaults. Without it, everything is L4 TCP.
2. **ACL token management**: Production requires ACLs. Ensure `manageSystemACLs: true` for automatic bootstrap.
3. **Mesh gateway sizing**: Mesh gateways handle all cross-DC traffic. Under-provisioning causes bottlenecks.
4. **Transparent proxy + external services**: External services not in the mesh need a terminating gateway or explicit ServiceEntry.
5. **Consul server resource limits**: Raft consensus requires consistent performance. Use SSDs and adequate CPU/memory for server pods.
6. **Namespace mirroring**: Enable `consulNamespaces.mirrorK8S` to keep Kubernetes and Consul namespaces aligned.
7. **Health check ports**: With transparent proxy, ensure health check endpoints are exposed via `expose.checks: true` in ServiceDefaults.

## Reference Files

Read these for implementation depth:

- `references/architecture.md` -- Consul Dataplane, intentions, ServiceRouter/Splitter/Resolver, mesh gateways, multi-DC federation. Read for architecture questions.