Traffic Flow and Fabric Architecture

Traditional data center designs rely on north-south traffic patterns and Spanning Tree Protocol, which often results in blocked links and suboptimal paths. The ACI fabric improves upon these designs through the following mechanisms:

• Encapsulation and policy application at the fabric entry point.

• Forwarding across the fabric through a spine switch with a maximum of two hops.

• Utilization of IS-IS and COOP for endpoint-to-endpoint communication.

• Implementation of MP-BGP for propagating routing information between software-defined networks and external routers.

The following figure illustrates the ACI fabric architecture.

VXLAN Encapsulation and Policy Enforcement in Cisco ACI

VXLAN in Cisco ACI encapsulates various packet types and enforces policy attributes, allowing for distributed and consistent policy enforcement across the fabric.

• Encapsulates external VLAN, VXLAN, and NVGRE packets into VXLAN packets at ingress.

• Supports isolated broadcast and failure bridge domains in the overlay.

• Decouples application policy EPG identity from forwarding, enabling flexible endpoint placement.

1. Ingress packets are encapsulated as VXLAN packets.

2. Policy attributes are carried in every packet for distributed enforcement.

3. Forwarding is not constrained by encapsulation type or overlay network.

• Layer 2 MAC address: Source and destination fields for efficient forwarding.

• Layer 3 IP address: Source and destination fields for scalable routing.

• VNID: 24-bit field for up to 16 million unique segments.

Note	VXLAN in Cisco ACI allows for scalable, policy-driven Layer 2 overlays across Layer 3 infrastructure, supporting large multitenant data centers.

How Layer 3 VNIDs Enable Inter-subnet Routing in Cisco ACI

The Cisco ACI fabric provides tenant default gateway functionality that routes between VXLAN networks. For each tenant, the fabric provides a virtual default gateway that spans all of the leaf switches assigned to the tenant. This gateway operates at the ingress interface of the first leaf switch connected to the endpoint, and all ingress interfaces share the same router IP and MAC address for a given tenant subnet.

The Cisco ACI fabric decouples the tenant endpoint address (its identifier) from the location of the endpoint, which is defined by its VXLAN tunnel endpoint (VTEP) address. Forwarding within the fabric is performed between VTEPs.

• Each VTEP has a switch interface on the local LAN segment for local endpoint communication through bridging.

• Each VTEP has an IP interface to the transport IP network, with a unique IP address identifying the VTEP device on the infrastructure VLAN.

The VTEP device uses its IP interface to encapsulate Ethernet frames and transmit them to the transport network. It also discovers remote VTEPs and learns remote MAC Address-to-VTEP mappings through its IP interface.

The VTEP in Cisco ACI maps the internal tenant MAC or IP address to a location using a distributed mapping database. After a lookup, the VTEP sends the original data packet encapsulated in VXLAN to the destination VTEP. The destination leaf switch de-encapsulates the packet and delivers it to the receiving host. This model enables a full mesh, single hop, loop-free topology without the need for spanning-tree protocol.

The VXLAN segments are independent of the underlying network topology, and the underlying IP network between VTEPs is independent of the VXLAN overlay. Routing is based on the outer IP address header, with the initiating VTEP as the source and the terminating VTEP as the destination.

For each tenant VRF, Cisco ACI assigns a single L3 VNID. Traffic is transported across the fabric according to the L3 VNID, and at the egress leaf switch, ACI routes the packet from the L3 VNID to the VNID of the egress subnet.

Traffic arriving at the fabric ingress and sent to the Cisco ACI default gateway is routed into the Layer 3 VNID, providing efficient forwarding for intra-tenant routed traffic. For example, traffic between two VMs in the same tenant but on different subnets only needs to reach the ingress switch interface before being routed to the correct destination.

To distribute external routes within the fabric, Cisco ACI route reflectors use multiprotocol BGP (MP-BGP). The fabric administrator provides the autonomous system (AS) number and specifies the spine switches that become route reflectors.

Note

Cisco ACI does not support IP fragmentation. Therefore, when you configure Layer 3 Outside (L3Out) connections to external routers, or Multi-Pod connections through an Inter-Pod Network (IPN), it is recommended that the interface MTU is set appropriately on both ends of a link. IGP Protocol Packets (EIGRP, OSPFv3) are constructed by components based on the Interface MTU size. In Cisco ACI , if the CPU MTU size is less than the Interface MTU size and if the constructed packet size is greater than the CPU MTU, then the packet is dropped by the kernel, especially in IPv6. To avoid such control packet drops always configure the same MTU values on both the control plane and on the interface. On some platforms, such as Cisco ACI , Cisco NX-OS, and Cisco IOS, the configurable MTU value does not take into account the Ethernet headers (matching IP MTU, and excluding the 14-18 Ethernet header size), while other platforms, such as IOS-XR, include the Ethernet header in the configured MTU value. A configured value of 9000 results in a max IP packet size of 9000 bytes in Cisco ACI , Cisco NX-OS, and Cisco IOS, but results in a max IP packet size of 8986 bytes for an IOS-XR untagged interface. For the appropriate MTU values for each platform, see the relevant configuration guides. We highly recommend that you test the MTU using CLI-based commands. For example, on the Cisco NX-OS CLI, use a command such as ping 1.1.1.1 df-bit packet-size 9000 source-interface ethernet 1/1.