Cisco Aci

---
name: cisco-aci
description: "Expert coverage of Cisco ACI across all versions: APIC, OpFlex, policy model (tenant/VRF/BD/EPG/contract), spine-leaf fabric, L3Out, service graphs, Multi-Site NDO, and fabric troubleshooting. Use for \"Cisco ACI\", \"APIC\", \"OpFlex\", \"EPG\", \"contract\", \"bridge domain\", \"L3Out\", \"service graph\", \"Nexus Dashboard Orchestrator\", \"ACI fabric\". Do NOT use for cross-vendor comparison, platform selection, or category-wide architecture -- use the `dc-fabric` skill."
license: MIT
---

# Cisco ACI

This skill covers Cisco Application Centric Infrastructure (ACI) across all supported versions. Areas of expertise include:

- APIC cluster management and fabric discovery
- OpFlex declarative southbound protocol
- ACI policy model: tenants, VRFs, bridge domains, EPGs, contracts, filters, L3Outs
- Spine-leaf CLOS fabric on Nexus 9000 in ACI mode
- Service graphs for L4-L7 service insertion (firewall, load balancer)
- Multi-Pod and Multi-Site with Nexus Dashboard Orchestrator (NDO)
- Cloud ACI for AWS and Azure
- Micro-segmentation with uSeg EPGs
- Fabric troubleshooting: health scores, faults, ELAM, contract hit counts
- REST API, moquery, Cobra SDK, Terraform ACI provider, Ansible

When a question is version-specific, consult the matching `references/versions/` file. When the version is unknown, provide general guidance and note where behavior differs.

## How to Approach Tasks

1. **Classify** the request:
   - **Troubleshooting** -- Load `references/diagnostics.md` for fault analysis, ELAM, contract debugging
   - **Policy design** -- Apply EPG/contract model guidance below
   - **Architecture** -- Load `references/architecture.md` for APIC, fabric topology, OpFlex, policy model
   - **Multi-site** -- Apply NDO/Multi-Pod/Multi-Site guidance below
   - **Automation** -- Apply REST API, moquery, Cobra SDK, or Terraform guidance

2. **Identify version** -- Determine which APIC version. If unclear, ask. Version matters for feature availability (ESGs require 6.0+, Cloud ACI requires 5.0+).

3. **Load context** -- Read the relevant reference file for deep knowledge.

4. **Analyze** -- Apply ACI-specific reasoning, not generic networking advice. ACI's policy model is fundamentally different from traditional networking.

5. **Recommend** -- Provide actionable guidance with GUI paths, REST API calls, or moquery examples.

6. **Verify** -- Suggest validation steps (health scores, contract hit counts, endpoint learning checks).

## Core Architecture

### APIC Cluster

The APIC is the centralized management, policy, and automation engine:
- **Deployment**: Minimum 3 physical or virtual APICs (odd number for quorum)
- **Production**: 3 physical APICs in separate availability zones; up to 9 for scale
- **Not in data path**: APIC distributes policies via OpFlex; fabric switches enforce locally
- **Resilience**: If all APICs go offline, existing policies continue to be enforced by switches
- **Functions**: Policy repository, REST API, GUI/CLI, fabric discovery, health monitoring, firmware management

### OpFlex Protocol

Declarative southbound protocol between APIC and fabric nodes:
- JSON-RPC over TCP
- Intent-based: APIC declares desired state; nodes implement locally
- Decentralized enforcement: each node processes its policy copy independently
- Nodes pull policies and report state back to APIC
- Also used by ACI Virtual Edge and Virtual Pod

### Spine-Leaf Fabric

Two-tier CLOS topology on Nexus 9000 series in ACI mode:
- **Leaf switches**: Connect servers, storage, service appliances, external networks. Policy enforcement points.
- **Spine switches**: Backbone interconnect. Carry VXLAN-encapsulated traffic. Run IS-IS underlay. No endpoint connections.
- Every leaf connects to every spine; no direct leaf-to-leaf or spine-to-spine links
- Fabric discovery is automated: switches register to APIC on boot via LLDP/IS-IS

**Supported platforms:**
- Nexus 9300-FX/EX/GX leaf series (1/10/25/100 GbE access)
- Nexus 9500 modular spine series (100/400 GbE backbone)
- Nexus 9336C-FX2 (dual-role: leaf or spine in smaller fabrics)

### ACI Mode vs NX-OS Standalone

Nexus 9000 runs in one of two mutually exclusive modes:
- **ACI mode**: Managed by APIC. No direct CLI configuration. Read-only diagnostics only.
- **NX-OS standalone mode**: Traditional CLI-driven. Full NX-OS feature set.
- Cannot mix modes within a single fabric. Switching modes requires a full wipe.

## Policy Model

### Hierarchy

```
Tenant
  VRF (L3 routing domain)
    Bridge Domain (L2 forwarding domain)
      Subnet(s)
      EPG (Endpoint Group)
        Endpoints (physical, virtual, container)
  Application Profile
    EPG
  Contract
    Subject
      Filter (L3/L4 match)
  L3Out (External Routed Network)
    External EPG
```

### Tenants

Top-level administrative container providing namespace isolation:
- **common**: Shared objects accessible by all tenants (shared L3Outs, contracts)
- **infra**: Fabric infrastructure (access policies, VLAN pools, domains)
- **mgmt**: Management network (in-band and out-of-band)
- Custom tenants for application/business isolation

### VRFs

Layer-3 routing domain within a tenant:
- Isolated routing table per VRF
- Multiple VRFs per tenant supported
- Inter-VRF communication requires shared contracts or L3Out route leaking
- VRF policy enforcement: ingress (recommended) or egress

### Bridge Domains

Layer-2 forwarding domain replacing traditional VLANs:
- Subnets defined on the BD (not on interfaces)
- One BD belongs to exactly one VRF
- BD settings control: ARP flooding, unicast routing, unknown unicast handling, L2/L3 forwarding modes
- Multiple subnets per BD supported (secondary IPs)
- **Hardware proxy mode** (recommended): Unknown unicast sent to spine proxy, not flooded

### EPGs (Endpoint Groups)

Core policy construct -- logical grouping of endpoints sharing the same security and forwarding policies:
- Endpoints classified by: static port binding, VLAN, VMM integration (vCenter, SCVMM), or container CNI
- EPGs belong to an Application Profile within a Tenant
- **Communication between EPGs is denied by default** -- whitelisting via contracts is required
- An EPG can span multiple leaf switches and even multiple sites (via NDO)

### Contracts

Define permitted communication between EPGs:
- **Filter**: L3/L4 match criteria (protocol, source/destination port)
- **Subject**: Groups filters; specifies directionality and QoS
- **Contract**: Groups subjects; applied between provider EPG and consumer EPG
- **Provider**: EPG offering a service
- **Consumer**: EPG accessing the service
- **vzAny**: Apply a contract to all EPGs in a VRF (simplifies shared services)
- **Preferred Group**: EPGs in a preferred group communicate freely without contracts; only non-preferred EPGs require explicit contracts

### L3Outs

Connect ACI fabric to external Layer-3 networks:
- Logical interface + external routing protocol (OSPF, BGP, EIGRP, static)
- External EPG (ExtEPG) classifies external subnets for policy
- Contracts between fabric EPGs and ExtEPGs control external access
- VRF-Lite for multi-VRF external connectivity
- BGP multi-hop support (APIC 6.1+)

### Service Graphs

Insert L4-L7 service appliances (firewalls, load balancers) in the traffic path between EPGs:
- **Function nodes**: Go-Through (transparent) or Go-To (routed)
- **Device clusters**: Physical or virtual appliance pools (ASA, F5, Palo Alto)
- **Managed mode**: APIC configures the appliance via device package
- **Unmanaged mode**: Appliance configured independently; ACI handles stitching only
- **PBR (Policy-Based Redirect)**: Steers traffic to service appliances at the leaf level

## Multi-Site Architecture

### Multi-Pod

- Single APIC cluster across geographically separated pods
- Inter-Pod Network (IPN) runs OSPF + PIM/BGP
- Single administrative domain with consistent policy
- Use case: adjacent buildings or campus-scale DC

### Multi-Site (NDO)

- Separate APIC clusters per site; centrally orchestrated by Nexus Dashboard Orchestrator
- Inter-site VXLAN overlay using BGP EVPN on Inter-Site Network (ISN)
- Stretched schemas push EPGs, BDs, VRFs, and contracts to multiple sites
- Independent failure domains per site
- Supports ACI on-premises, Cloud ACI (AWS/Azure), and NDFC sites

### Cloud ACI

- **AWS**: Cloud APIC as EC2 instance; VPCs mapped to ACI tenants; security groups managed via policy
- **Azure**: Cloud APIC as VM; VNets mapped to ACI tenants; NSGs managed via policy
- Cisco Catalyst 8000V cloud router for inter-site IPsec connectivity
- NDO provides unified policy across on-prem + cloud

## Automation

### REST API

```
Base URL: https://<apic>/api/

# Login
POST /api/aaaLogin.json
Body: {"aaaUser":{"attributes":{"name":"admin","pwd":"password"}}}

# Query all tenants
GET /api/node/class/fvTenant.json

# Query specific tenant
GET /api/node/mo/uni/tn-Production.json?rsp-subtree=full

# Create an EPG
POST /api/node/mo/uni/tn-Production/ap-WebApp/epg-FrontEnd.json
Body: {"fvAEPg":{"attributes":{"name":"FrontEnd"}}}
```

### moquery (CLI Object Query)

```
moquery -c fvTenant                              # List all tenants
moquery -c fvAEPg                                # List all EPGs
moquery -c fvCEp -f 'fv.CEp.ip=="10.1.1.5"'     # Find endpoint by IP
moquery -c vzFilter                              # List all filters
moquery -c faultInst -f 'fault.Inst.severity=="critical"'  # Critical faults
```

### Cobra SDK (Python)

```python
from cobra.mit.access import MoDirectory
from cobra.mit.session import LoginSession
from cobra.model.fv import Tenant, Ctx, BD, Ap, AEPg

ls = LoginSession('https://apic', 'admin', 'password')
md = MoDirectory(ls)
md.login()

# Query
tenants = md.lookupByClass('fvTenant')
for t in tenants:
    print(t.name)
```

### Terraform

```hcl
provider "aci" {
  username = "admin"
  password = var.apic_password
  url      = "https://apic.example.com"
}

resource "aci_tenant" "prod" {
  name = "Production"
}

resource "aci_vrf" "main" {
  tenant_dn = aci_tenant.prod.id
  name      = "Main-VRF"
}
```

## Common Pitfalls

1. **Forgetting that inter-EPG traffic is denied by default** -- New ACI operators expect connectivity after creating EPGs. Contracts must be explicitly configured between provider and consumer EPGs. Use `vzAny` for shared services like DNS/NTP.

2. **BD flooding mode vs hardware proxy** -- Hardware proxy (recommended) reduces broadcast but requires proper endpoint learning. Flooding mode works like a traditional VLAN but wastes fabric bandwidth. Default is hardware proxy in recent versions.

3. **L3Out route leaking without contracts** -- Even with correct routing, traffic between external networks and fabric EPGs requires contracts on the External EPG. Missing contracts result in silent drops.

4. **Service graph PBR misconfigurations** -- PBR requires correct consumer/provider bridge domain configuration, health check policies for the service appliance, and proper IP addressing on the service device. The most common issue is a misconfigured health check that marks the service device as down.

5. **Multi-Site schema conflicts** -- When stretching objects via NDO, local APIC modifications to stretched objects can cause schema drift. Always make changes for stretched objects through NDO, not directly on APIC.

6. **Endpoint learning issues** -- Rogue endpoint detection can mark legitimate endpoints as rogue during VM migrations. Tune rogue EP parameters for environments with frequent vMotion.

7. **VLAN pool exhaustion** -- ACI allocates VLANs from pools configured in access policies. If the pool range is too small, EPG deployment fails silently. Monitor VLAN pool utilization.

## Version-Specific Guidance

| Version | Reference | What's version-specific |
|---|---|---|
| 6.1 | `references/versions/6.1.md` | Endpoint Security Groups (ESGs), enhanced inter-VRF shared services, BGP multi-hop L3Out, streaming telemetry |

## Reference Files

Load these when you need deep knowledge:

- `references/architecture.md` -- APIC internals, OpFlex, policy model details, fabric discovery, access policies, VMM integration. Read for "how does X work" questions.
- `references/diagnostics.md` -- Faults, health scores, ELAM, contract hit counts, endpoint reachability, Central CLI. Read when troubleshooting.