Guidelines for Setting Up a Cisco ACI Multi-Pod Fabric

Cisco ACI Multi-Pod is supported on all ACI-mode spine switches, all Cisco Nexus 9000 Series ACI-mode leaf switches, and all of the Cisco Nexus 9500 platform ACI-mode switch line cards and fabric modules.

Create the associated node group and Layer 3 Outside (L3Out) policies.

Before you make any changes to a spine switch, ensure that there is at least one operationally "up" external link that is participating in the Cisco ACI Multi-Pod topology. Failure to do so could bring down the Cisco ACI Multi-Pod connectivity.

If you have to convert a Cisco ACI Multi-Pod setup to a single pod (containing only Pod 1), the Cisco Application Policy Infrastructure Controllers (APICs) connected to the pods that are decommissioned should be re-initialized and connected to the leaf switches in Pod 1, which will allow them to re-join the cluster after going through the initial setup script. See Move an APIC from one pod to another pod for those instructions. The TEP pool configuration should not be deleted.

Cisco ACI GOLF (also known as Layer 3 EVPN Services for Fabric WAN) and Cisco ACI Multi-Pod can be deployed together over all the switches used in the Cisco ACI Multi-Pod and EVPN topologies. For more information on GOLF, see Cisco ACI GOLF.

In a Cisco ACI Multi-Pod fabric, the Pod 1 configuration (with the associated TEP pool) must always exist on Cisco APIC, as the Cisco APIC nodes are always addressed from the Pod 1 TEP pool. This remains valid also in the scenario where the Pod 1 is physically decommissioned (which is a fully supported procedure) so that the original Pod 1 TEP pool is not re-assigned to other pods that may be added to the fabric.

In a Cisco ACI Multi-Pod fabric setup, if a new spine switch is added to a pod, it must first be connected to at least one leaf switch in the pod. This enables the Cisco APIC to discover the spine switch and join it to the fabric.

After a pod is created and nodes are added in the pod, deleting the pod results in stale entries from the pod that are active in the fabric. This occurs because the Cisco APIC uses open source DHCP, which creates some resources that the Cisco APIC cannot delete when a pod is deleted

If you connect spine switches belonging to separate pods with direct back-to-back links, an OSPF neighborship might get established on the peer interface between the two spine switches. If there is a mismatch between the peer interfaces, with one of the peers having the Cisco ACI Multi-Pod direct flag disabled, the session won't be up and forwarding will not happen. Even though the system will throw a fault in this situation, this is expected behavior.

Beginning with Cisco APIC release 5.2(3), the IPN underlay protocol can be external BGP (eBGP). Internal BGP (iBGP) is not supported as the underlay protocol.

When preparing to migrate a Cisco ACI Multi-Pod fabric between OSPF and BGP as the IPN underlay, follow these guidelines:

• A BGP underlay is not supported if the Cisco ACI fabric is connected to a cloud site or to a GOLF router.

• A BGP underlay supports only an IPv4 address family, not an IPv6 address family.

When deploying Cisco APIC cluster connectivity to the fabric over a Layer 3 network, which was introduced in Cisco APIC release 5.2(1), the IPN network can use OSPF as the underlay protocol, or a BGP underlay if the Cisco APIC connects to the fabric using the same network that provides Cisco ACI Multi-Pod or remote leaf switch connectivity.

Cisco ACI 5.2(3) and later supports BGP for underlay connectivity. When enabling BGP underlay on the infra L3Out for IPN or ISN connectivity, the spines BGP router ID will change to the router ID configured in the L3Out. This will result in all BGP sessions on the spines configured in the infra tenant L3Out being reset and reestablished. To avoid traffic interruption due to router ID changes, add spines to the infra tenant L3Out individually.

From Cisco ACI release 6.1(1) and later, spines switches configured in infra tenant L3Outs for IPN or ISN connectivity will have their router IDs changed to the router ID configured in the infra L3Out when either OSPF or BGP is configured for the underlay protocol. The router ID change for OSPF is due to DDTS CSCwu13120.

If you delete and recreate the Cisco ACI Multi-Pod L3Out, for example to change the name of a policy, a clean reload of some of the spine switches in the fabric must be performed. The deletion of the Cisco ACI Multi-Pod L3Out causes one or more of the spine switches in the fabric to lose connectivity to the Cisco APICs and these spine switches are unable to download the updated policy from the Cisco APIC. Which spine switches get into such a state depends upon the deployed topology. To recover from this state, a clean reload must be performed on these spine switches. The reload is performed using the setup-clean-config.sh command, followed by the reload command on the spine switch.

You can set the global MTU for control plane (CP) packets sent by the nodes (Cisco APIC and the switches) in the fabric at System > System Settings > Control Plane MTU.

In a Cisco ACI Multi-Pod topology, the MTU set for the fabric external ports must be greater than or equal to the CP MTU value set. Otherwise, the fabric external ports might drop the CP MTU packets.

If you change the IPN or CP MTU, we recommend changing the CP MTU value first, then changing the MTU value on the spine of the remote pod. This reduces the risk of losing connectivity between the pods due to MTU mismatch. This is to ensure that the MTU across all the interfaces of the IPN devices between the pods is large enough for both control plane and VXLAN data plane traffic at any given time. For data traffic, keep in mind the extra 50 bytes due to VXLAN.

To decommission a pod, decommission all the nodes in the pod. For instructions, see Decommissioning and Recommissioning a Pod in Cisco APIC Troubleshooting Guide.

In the Cisco APIC 6.0(2) release and later, when you configure an OSPF Cisco ACI Multi-Pod session and the session is running, do not configure a passive interface for the L3Out interface properties.