Explains how Layer 3 VNIDs in the ACI fabric enable efficient routing and transport of inter-subnet tenant traffic by decoupling endpoint identity from location and leveraging VXLAN encapsulation.
Layer 3 VNIDs are identifiers assigned to each tenant VRF in the ACI fabric that enable routing and transport of inter-subnet tenant traffic across the fabric.
-
Each tenant receives a single L3 VNID, which is used to route traffic between subnets within the tenant.
-
The ACI fabric provides a distributed virtual default gateway, sharing the same IP and MAC address across all ingress interfaces for a tenant subnet.
-
VXLAN encapsulation and VTEP devices decouple endpoint identity from location, allowing efficient forwarding and routing without spanning-tree protocol.
How Layer 3 VNIDs Enable Inter-subnet Routing in ACI
The Cisco ACI fabric provides tenant default gateway functionality that routes between VXLAN networks. For each tenant, the fabric provides a virtual default gateway that spans all of the leaf switches assigned to the tenant. This gateway operates at the ingress interface of the first leaf switch connected to the endpoint, and all ingress interfaces share the same router IP and MAC address for a given tenant subnet.
The Cisco ACI fabric decouples the tenant endpoint address (its identifier) from the location of the endpoint, which is defined by its VXLAN tunnel endpoint (VTEP) address. Forwarding within the fabric is performed between VTEPs.
-
Each VTEP has a switch interface on the local LAN segment for local endpoint communication through bridging.
-
Each VTEP has an IP interface to the transport IP network, with a unique IP address identifying the VTEP device on the infrastructure VLAN.
The VTEP device uses its IP interface to encapsulate Ethernet frames and transmit them to the transport network. It also discovers remote VTEPs and learns remote MAC Address-to-VTEP mappings through its IP interface.
The VTEP in Cisco ACI maps the internal tenant MAC or IP address to a location using a distributed mapping database. After a lookup, the VTEP sends the original data packet encapsulated in VXLAN to the destination VTEP. The destination leaf switch de-encapsulates the packet and delivers it to the receiving host. This model enables a full mesh, single hop, loop-free topology without the need for spanning-tree protocol.
The VXLAN segments are independent of the underlying network topology, and the underlying IP network between VTEPs is independent of the VXLAN overlay. Routing is based on the outer IP address header, with the initiating VTEP as the source and the terminating VTEP as the destination.
For each tenant VRF, ACI assigns a single L3 VNID. Traffic is transported across the fabric according to the L3 VNID, and at the egress leaf switch, ACI routes the packet from the L3 VNID to the VNID of the egress subnet.
Traffic arriving at the fabric ingress and sent to the ACI default gateway is routed into the Layer 3 VNID, providing efficient forwarding for intra-tenant routed traffic. For example, traffic between two VMs in the same tenant but on different subnets only needs to reach the ingress switch interface before being routed to the correct destination.
To distribute external routes within the fabric, ACI route reflectors use multiprotocol BGP (MP-BGP). The fabric administrator provides the autonomous system (AS) number and specifies the spine switches that become route reflectors.
Cisco ACI does not support IP fragmentation. Therefore, when you configure Layer 3 Outside (L3Out) connections to external routers, or Multi-Pod connections through an Inter-Pod Network (IPN), it is recommended that the interface MTU is set appropriately on both ends of a link.
IGP Protocol Packets (EIGRP, OSPFv3) are constructed by components based on the Interface MTU size. In Cisco ACI , if the CPU MTU size is less than the Interface MTU size and if the constructed packet size is greater than the CPU MTU, then the packet is dropped by the kernel, especially in IPv6. To avoid such control packet drops always configure the same MTU values on both the control plane and on the interface.
On some platforms, such as Cisco ACI , Cisco NX-OS, and Cisco IOS, the configurable MTU value does not take into account the Ethernet headers (matching IP MTU, and excluding the 14-18 Ethernet header size), while other platforms, such as IOS-XR, include the Ethernet header in the configured MTU value. A configured value of 9000 results in a max IP packet size of 9000 bytes in Cisco ACI , Cisco NX-OS, and Cisco IOS, but results in a max IP packet size of 8986 bytes for an IOS-XR untagged interface.
For the appropriate MTU values for each platform, see the relevant configuration guides.
We highly recommend that you test the MTU using CLI-based commands. For example, on the Cisco NX-OS CLI, use a command such as ping 1.1.1.1 df-bit packet-size 9000 source-interface ethernet 1/1.
