To configure a multipod fabric, follow these guidelines:
All Cisco Nexus 9000 Series ACI-mode switches and all of the Cisco Nexus 9500 platform ACI-mode switch line cards and fabric
modules support multipod. With Cisco APIC, release 3.1(x) and higher, this includes the N9K-C9364C switch.
Create the associated node group and Layer 3 Out policies.
Before you make any changes to a spine switch, ensure that there is at least one operationally “up” external link that is
participating in the multipod topology. Failure to do so could bring down the multipod connectivity.
If a multipod setup needs to be downgraded, and it is required to convert the setup to a single pod (containing only pod1),
first shrink the controllers to the number of controllers only in pod-1 and then decommission all nodes from other pods before
performing the downgrade. The TEP pool configuration should not be deleted. Please note that with this downgrade, all nodes
in other pods will go down.
Only OSPF regular area is supported under the
Up to APIC release 2.0(2), multipod is not supported with Cisco ACI GOLF. In APIC release 2.0 (2) the two features are supported in the same fabric only over Cisco Nexus 9000 switches without “EX”
on the end of the switch name; for example, N9K-9312TX. Since the 2.1(1) release, the two features can be deployed together
over all switches used in the multipod topologies.
In a multipod fabric, if a spine in POD 1 uses the infra tenant L3extOut-1, the TORs of the other pods ( POD 2, POD 3) cannot
use the same infra L3extOut (L3extOut-1) for Layer 3 EVPN control plane connectivity. Each pod must use its own spine switch
and infra L3extOut, because it is not supported to use a pod as a transit for WAN connectivity of other pods.
In a multipod fabric setup, if a new spine switch is added to a pod, it must first be connected to at least one leaf switch
in the pod. This enables the APIC to discover the spine switch and join it to the fabric.
After a pod is created and nodes are added in the pod, deleting the pod results in stale entries from the pod that are active
in the fabric. This occurs because the APIC uses open source DHCP, which creates some resources that the APIC cannot delete
when a pod is deleted
Forward Error Correction (FEC) is enabled for all 100G transceivers by default. Do not use QSFP-100G-LR4-S / QSFP-100G-LR4
transceivers for multipod configuration.
The following is required when deploying a pair of Active/Standby Firewalls (FWs) across pods:
Scenario 1: Use of PBR to redirect traffic through the FW:
Mandates the use of Service Graphs and enables connecting the FW inside/outside interfaces to the ACI Fabric. This feature
is fully supported from the 2.1(1) release.
Flows from all the compute leaf nodes are always sent to the border leaf nodes connected to the Active FW.
Scenario 2: Use of L3Out connections between the Fabric and the FW:
Fully supported starting from 2.0(1) release.
Only supported with dynamic routing (no static routing) and with Cisco ASA (not with FWs using VRRP).
Active FW only peers with the BL nodes in the local Pod. The leafs inject external routing information into the fabric.
Dynamic peering sessions must be re-established in the new Pod, due to longer traffic outages after FW failover.
Scenario 3: Use of a single L3Out stretched across Pods.
Active and Standby FWs connected to a single leaf node with a physical link or (local port-channel) is supported in releases
2.1(2e) and 2.2(2e) on all ACI leaf nodes (E, EX, FX).
Active and Standby FWs connected in vPC mode in each Pod to a pair of leaf nodes is supported from release 2.3(1) and only
for EX, FX or newer ACI leaf nodes.
If you delete and recreate the Multipod L3out, for example to change the name of a policy, a clean reload of some of the spine
switches in the fabric must be performed. The deletion of the Multipod L3Out causes one or more of the spine switches in the
fabric to lose connectivity to the APICs and these spine switches are unable to download the updated policy from the APIC.
Which spine switches get into such a state depends upon the deployed topology. To recover from this state, a clean reload
must be performed on these spine switches. The reload is performed using the setup-clean-config.sh command, followed by the reload command on the spine switch.
Cisco ACI does not support IP fragmentation. Therefore, when you configure Layer 3 Outside (L3Out) connections to external
routers, or multipod connections through an Inter-Pod Network (IPN), it is critical that the MTU is set appropriately on both
sides. On some platforms, such as ACI, Cisco NX-OS, and Cisco IOS, the configurable MTU value takes into account the IP headers (resulting in a max packet size to be set as
9216 bytes for ACI and 9000 for NX-OS and IOS). However, other platforms such as IOS-XR configure the MTU value exclusive
of packet headers (resulting in a max packet size of 8986 bytes).
For the appropriate MTU values for each platform, see the relevant configuration guides.
Cisco highly recommends you test the MTU using CLI-based commands. For example, on the Cisco NX-OS CLI, use a command such as
ping 18.104.22.168 df-bit packet-size 9000 source-interface ethernet 1/1.
You can set the global MTU for control plane (CP) packets sent by the nodes (APIC and the switches) in the fabric at
In a multipod topology, the MTU set for the fabric external ports must be greater than or equal to the CP MTU value set. Otherwise,
the fabric external ports might drop the CP MTU packets.
If you change the IPN or CP MTU, Cisco recommends changing the CP MTU value first, then changing the MTU value on the spine
of the remote pod. This reduces the risk of losing connectivity between the pods due to MTU mismatch.
To decommission a pod, decommission all the nodes in the pod. For instructions, see Decommissioning and Recommissioning a Pod in Cisco APIC Troubleshooting Guide.