Layer 3 VNIDs are identifiers assigned to each tenant VRF in the ACI fabric that enable routing and transport of inter-subnet tenant traffic across the fabric.
Each tenant receives a single L3 VNID, which is used to route traffic between subnets within the tenant.
The ACI fabric provides a distributed virtual default gateway, sharing the same IP and MAC address across all ingress interfaces for a tenant subnet.
VXLAN encapsulation and VTEP devices decouple endpoint identity from location, allowing efficient forwarding and routing without spanning-tree protocol.
How Layer 3 VNIDs Enable Inter-subnet Routing in ACI
The Cisco ACI fabric provides tenant default gateway functionality that routes between VXLAN networks. For each tenant, the fabric provides a virtual default gateway that spans all of the leaf switches assigned to the tenant. This gateway operates at the ingress interface of the first leaf switch connected to the endpoint, and all ingress interfaces share the same router IP and MAC address for a given tenant subnet.
The Cisco ACI fabric decouples the tenant endpoint address (its identifier) from the location of the endpoint, which is defined by its VXLAN tunnel endpoint (VTEP) address. Forwarding within the fabric is performed between VTEPs.
Each VTEP has a switch interface on the local LAN segment for local endpoint communication through bridging.
Each VTEP has an IP interface to the transport IP network, with a unique IP address identifying the VTEP device on the infrastructure VLAN.
The VTEP device uses its IP interface to encapsulate Ethernet frames and transmit them to the transport network. It also discovers remote VTEPs and learns remote MAC Address-to-VTEP mappings through its IP interface.
The VTEP in Cisco ACI maps the internal tenant MAC or IP address to a location using a distributed mapping database. After a lookup, the VTEP sends the original data packet encapsulated in VXLAN to the destination VTEP. The destination leaf switch de-encapsulates the packet and delivers it to the receiving host. This model enables a full mesh, single hop, loop-free topology without the need for spanning-tree protocol.
The VXLAN segments are independent of the underlying network topology, and the underlying IP network between VTEPs is independent of the VXLAN overlay. Routing is based on the outer IP address header, with the initiating VTEP as the source and the terminating VTEP as the destination.
For each tenant VRF, ACI assigns a single L3 VNID. Traffic is transported across the fabric according to the L3 VNID, and at the egress leaf switch, ACI routes the packet from the L3 VNID to the VNID of the egress subnet.
Traffic arriving at the fabric ingress and sent to the ACI default gateway is routed into the Layer 3 VNID, providing efficient forwarding for intra-tenant routed traffic. For example, traffic between two VMs in the same tenant but on different subnets only needs to reach the ingress switch interface before being routed to the correct destination.
To distribute external routes within the fabric, ACI route reflectors use multiprotocol BGP (MP-BGP). The fabric administrator provides the autonomous system (AS) number and specifies the spine switches that become route reflectors.
Note
Cisco ACI does not support IP fragmentation. Therefore, when you configure Layer 3 Outside (L3Out) connections to external routers, or Multi-Pod connections through an Inter-Pod Network (IPN), it is recommended that the interface MTU is set appropriately on both ends of a link.
IGP Protocol Packets (EIGRP, OSPFv3) are constructed by components based on the Interface MTU size. In Cisco ACI , if the CPU MTU size is less than the Interface MTU size and if the constructed packet size is greater than the CPU MTU, then the packet is dropped by the kernel, especially in IPv6. To avoid such control packet drops always configure the same MTU values on both the control plane and on the interface.
On some platforms, such as Cisco ACI , Cisco NX-OS, and Cisco IOS, the configurable MTU value does not take into account the Ethernet headers (matching IP MTU, and excluding the 14-18 Ethernet header size), while other platforms, such as IOS-XR, include the Ethernet header in the configured MTU value. A configured value of 9000 results in a max IP packet size of 9000 bytes in Cisco ACI , Cisco NX-OS, and Cisco IOS, but results in a max IP packet size of 8986 bytes for an IOS-XR untagged interface.
For the appropriate MTU values for each platform, see the relevant configuration guides.
We highly recommend that you test the MTU using CLI-based commands. For example, on the Cisco NX-OS CLI, use a command such as ping 1.1.1.1 df-bit packet-size 9000 source-interface ethernet 1/1.
Figure 1. ACI Decouples Identity and Location
Figure 2. Layer 3 VNIDs Transport ACI Inter-subnet Tenant Traffic
Router peering and route distribution
Summary
Router peering and route distribution involve the following components:
Leaf switches establish routing protocol peering sessions with external routers and maintain forwarding tables containing the learned routes.
External routers establish routing protocol peering sessions and advertise WAN routes.
Spine switches act as route reflectors and distribute external routes throughout the fabric.
Tenant endpoints send packets to the default gateway in the Cisco ACI fabric.
Workflow
Figure 3. Router peering
Router peering and route distribution occur as follows:
The leaf switch interface is configured to establish a routing protocol peering session with the external router.
The leaf switch sends the routes learned through the peering session to the spine switches.
The spine switches act as route reflectors and distribute the external routes to all leaf switches that have interfaces belonging to the same tenant.
The leaf switches install the routes as longest prefix match (LPM) entries in their forwarding tables. Each entry identifies the virtual tunnel endpoint (VTEP) IP address of the remote leaf switch to which the external router is connected.
WAN routes do not use a forwarding proxy. If the WAN routes exceed the capacity of a leaf switch forwarding table, the leaf switch drops the traffic.
Because the external router is not the default gateway, tenant endpoints send their packets to the default gateway in the Cisco ACI fabric.
Bridged interface to an external router
With a bridged interface, an external router provides the default gateway for the tenant virtual network identifier (VNID).
The Cisco ACI fabric does not maintain routing information about the external router. The APIC statically assigns the leaf switch interface to its endpoint group (EPG).
Bridged interface topology
Figure 4. Bridged external router
Route reflectors
An ACI fabric route reflector is a spine switch that
uses MP-BGP to distribute external routes throughout the fabric
requires you to select spine switches for the route reflector role
requires an autonomous system number, and
provides redundancy when you configure at least two spine switches as route reflectors in each pod.
External connectivity and route distribution
After you enable route reflectors, configure L3Outs on leaf switches to connect the fabric to external networks. Leaf switches that provide this connectivity are called border leaf switches.
Each border leaf switch exchanges routes with external devices through dynamic routing protocols or static routes. The spine route reflectors then distribute these routes to all leaf switches in the fabric.
Note
For the maximum number of routes supported on each leaf switch, refer to the Verified Scalability Guide for Cisco APIC for your release.
External connectivity through a Layer 3 Outside
A Layer 3 Outside (L3Out) is a tenant configuration object that provides connectivity between the Cisco ACI fabric and an external routed network.
Supports routing protocols such as Open Shortest Path First (OSPF).
Uses multiprotocol Border Gateway Protocol (MP-BGP) route reflectors to distribute external routes within the fabric.
Operates within a specific tenant, such as the mgmt tenant.
External connectivity configuration requirements
Configuring external connectivity requires the following components:
Define the L3Out in the target tenant.
Configure a routing protocol, such as OSPF, to exchange routes with the external network.
Configure MP-BGP route reflectors to distribute the external routes within the fabric.
Configure MP-BGP route reflector using the GUI
Procedure
From the menu bar, choose System > System Settings.
In the Navigation pane, right-click BGP Route Reflector, and choose Create Route Reflector Node.
In the Create Route Reflector Node dialog box, choose the spine switch that you want to configure from the Spine Node drop-down list, and click Submit.
Note
Repeat the node-creation process for each additional spine switch.
The APIC designates the spine switch as a route reflector.
In the BGP Route Reflector properties area, enter the autonomous system number in the Autonomous System Number field, and click Submit.
Note
If the external router uses BGP, the autonomous system number must match the autonomous system number configured on the router connected to the leaf switch. If the fabric learns routes through static routing or OSPF, you can enter any valid autonomous system number.
From the menu bar, choose Fabric > Fabric Policies > Pods > Policy Groups.
In the Navigation pane, right-click Policy Groups, and choose Create Pod Policy Group.
In the Create Pod Policy Group dialog box, enter a name for the pod policy group in the Name field.
From the BGP Route Reflector Policy drop-down list, choose the BGP route reflector policy, and click Submit.
The APIC associates the BGP route reflector policy with the pod policy group and enables the BGP process on the leaf switches.
From the menu bar, choose Fabric > Fabric Policies > Profiles > Pod Profile default > default.
In the Work pane, choose the pod policy group that you created from the Fabric Policy Group drop-down list, and click Submit.
The APIC applies the pod policy group to the fabric policy group.
MP-BGP route reflectors for the Cisco ACI fabric
A multiprotocol Border Gateway Protocol (MP-BGP) route reflector simplifies internal BGP scaling by enabling a spine switch to distribute routes among leaf switches in the Cisco ACI fabric.
Requires an active MP-BGP process.
Uses designated spine switches as route reflectors.
MP-BGP route reflector configuration example
Note
In this example, the BGP fabric autonomous system number (ASN) is 100, and spine switches 104 and 105 are the MP-BGP route reflectors.
The following commands configure the BGP fabric ASN and designate the route reflectors:
Use Secure Shell (SSH) to log in to each leaf switch as an administrator.
Enter the show processes | grep bgp command and verify that the process state is S.
A state of NR indicates that the process is not running and that the configuration was unsuccessful.
Verify the autonomous system number on each spine switch.
Use SSH to log in to each spine switch as an administrator.
Enter these commands in the shell:
Example:
cd /mit/sys/bgp/inst
grep asn summary
The command displays the configured autonomous system number. A value of 0 indicates that the configuration was unsuccessful.
Create an OSPF L3Out for the management tenant by using the GUI
Verify that the router ID and logical interface profile IP address are different and do not overlap.
This procedure creates an OSPF L3Out for the management tenant. To create an OSPF L3Out for another tenant, select that tenant and create a VRF for it.
For more information, see Cisco APIC and Transit Routing.
Procedure
On the menu bar, choose Tenants > mgmt.
In the Navigation pane, expand Networking > L3Outs.
Right-click L3Outs and choose Create L3Out.
The Create L3Out wizard appears.
In the Identity window of the Create L3Out wizard, configure the L3Out.
In the Name field, enter a name, such as RtdOut.
From the VRF drop-down list, choose the in-band VRF, such as inb.
Note
This selection associates the L3Out with the in-band VRF.
From the L3 Domain drop-down list, choose the appropriate domain.
Check the OSPF check box.
In the OSPF Area ID field, enter the area ID.
For OSPF Area Control, check the appropriate check boxes.
For OSPF Area Type, choose the appropriate area type.
For OSPF Area Cost, enter the appropriate value.
Click Next.
The Nodes and Interfaces window appears.
In the Nodes and Interfaces window, configure the nodes and interfaces.
Uncheck the Use Defaults check box.
You can now edit the Node Profile Name field.
In the Node Profile Name field, enter a name, such as borderLeaf.
From the Node ID drop-down list, choose the first node, such as leaf1.
In the Router ID field, enter a unique router ID.
In the Loopback Address field, retain the router ID, enter a different IP address, or clear the field if you do not want to use a loopback address.
Note
The Loopback Address field is automatically populated with the value from the Router ID field. This behavior is equivalent to the Use Router ID for Loopback Address option in previous releases.
As required, configure the Interface, IP Address, Interface Profile Name, and MTU fields for the first node.
In the Nodes area, click the + icon to add another node.
From the Node ID drop-down list, choose the second node, such as leaf2.
In the Router ID field, enter a unique router ID for the second node.
In the Loopback Address field, retain the router ID, enter a different IP address, or clear the field if you do not want to use a loopback address.
Note
The Loopback Address field is automatically populated with the value from the Router ID field. This behavior is equivalent to the Use Router ID for Loopback Address option in previous releases.
As required, configure the Interface, IP Address, Interface Profile Name, and MTU fields for the second node.
Click Next.
The Protocols window appears.
In the Protocols window, choose default in the Policy area, and then click Next.
The External EPG window appears.
In the External EPG window, configure the external EPG.
In the Name field, enter a name for the external network, such as extMgmt.
Uncheck the Default EPG for all external networks check box.
The Subnets area appears.
Click the + icon.
The Create Subnet dialog appears.
In the IP address field, enter the subnet IP address and prefix length.
For Scope, check the appropriate check boxes, and then click OK.
In the External EPG window, click Finish.
The L3Out, such as RtdOut, appears in the L3Outs area of the Work pane.
Create an OSPF external routed network for a tenant by using the NX-OS-style CLI
Configuring external routed connectivity requires the following tasks:
Create a VRF in the tenant.
Configure Layer 3 connectivity for the VRF on the border leaf switches that connect to the external routed network. This configuration can include interfaces, routing protocols such as BGP, OSPF, or EIGRP, protocol parameters, and route maps.
Create external Layer 3 EPGs in the tenant and deploy them on the border leaf switches. External routed subnets in a VRF that share the same policy within the Cisco ACI fabric form one external Layer 3 EPG, also called a prefix EPG.
Complete the configuration in the following modes:
Tenant configuration mode: Create the VRF and configure the external Layer 3 EPG.
Leaf configuration mode: Configure Layer 3 connectivity and deploy the external Layer 3 EPG.
Before you create an OSPF external routed network, select a tenant and create a VRF in that tenant.
Note
The examples provide external routed connectivity to the web EPG in the OnlineStore application profile for the exampleCorp tenant.
apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant exampleCorp vrf exampleCorp_v1
apic1(config-leaf-ospf-vrf)# area 0.0.0.1 route-map map100 out
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
Assign the VRF to the subinterface and enable OSPF for the area.
Note
Before you configure the subinterface, use the no switchport command to convert the main interface, Ethernet 1/11 in this example, to a Layer 3 interface. Assign the main interface to the dom_exampleCorp VLAN domain, which contains the encapsulation VLAN used by the subinterface. In the interface name ethernet 1/11.500, 500 is the encapsulation VLAN.
Example:
apic1(config-leaf)# interface ethernet 1/11
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vlan-domain member dom_exampleCorp
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet 1/11.500
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf exampleCorp_v1
apic1(config-leaf-if)# ip address 157.10.1.1/24
apic1(config-leaf-if)# ip router ospf default area 0.0.0.1
Configure the external Layer 3 EPG policy.
The policy identifies the external subnet and consumes the contract that provides connectivity to the web EPG.
Example:
apic1(config)# tenant exampleCorp
apic1(config-tenant)# external-l3 epg l3epg100
apic1(config-tenant-l3ext-epg)# vrf member exampleCorp_v1
apic1(config-tenant-l3ext-epg)# match ip 145.10.1.0/24
apic1(config-tenant-l3ext-epg)# contract consumer web
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit
Deploy the external Layer 3 EPG on the border leaf switch.
