A border leaf switch is a leaf switch that provides routed connectivity between the ACI fabric and the rest of the datacenter network. Best practice is to use two leaf switches for this purpose. In a stretched fabric, border leaf switches are typically placed in different datacenters to maximize availability if there are outages in one datacenter.
Two important design choices related to border-leaf switch placement in a stretched fabric are the following:
The use of one or two border leaf switches per datacenter.
The choice between the use of dedicated border leaf switches or the use of border leaf switches for both server connectivity and routed connectivity to an outside network.
The design choice must consider the specific ACI software version and hardware capabilities of the leaf switches.
In ACI, an L3Out provides routed connectivity of VRFs to a network outside the ACI fabric. The stretched-fabric border leaf switch topology you choose relates to the type of ACI L3Out configuration you deploy. Border leaf switches support three types of interfaces to connect to an external router:
Layer 3 (routed) interface
Sub-interface with IEEE 802.1Q tagging
Switched virtual interface (SVI)
When configuring an SVI on an L3Out, you specify a VLAN encapsulation. Specifying the same VLAN encapsulation on multiple border leaf nodes in the same L3Out results in the configuration of an external bridge domain. You can configure static or dynamic routing protocol peering over a vPC for an L3Out connection by specifying the same SVI encapsulation on both vPC peers.
Considerations for Using More Than Two Border Leaf Switches
According to the hardware used for the leaf switches and the software release, one has to consider that having more than two border leaf switches as part of the same L3Out in ACI may have restrictions in the following situations:
The L3Out consists of more than two leaf switches with SVI using the same encapsulation (VLAN).
The border leaf switches are configured with static routing to the external device.
The connectivity from the outside device to the fabric is vPC-based.
This is because traffic may be routed from one datacenter to the local L3Out and then bridged on the external bridge domain to the L3Out in the other datacenter.
In the two topologies below, for connectivity to an external active/standby firewall pair, ACI is configured for static routing. The dotted line identifies the border leaf switches. The first topology, shown in Figure 10, is supported with any version of ACI leaf switches. The second topology, shown in Figure 11, is only supported with Cisco Nexus 9300-EX Cloud Scale or newer leaf switches.
Figure 10. Static Routing L3 Out with SVI and vPC Supported with Any Version ACI Leaf Switches
Figure 10 illustrates a topology that works with both first and second-generation ACI leaf switches. The L3out uses the same encapsulation on all the border leaf switches to allow static routing from either border leaf switch to the active firewall. In this diagram, both the split fabric transit function and the border leaf switch function are configured on the same leaf switches, which should be planned according to the guidelines in Transit Leaf Switch Guidelines in this article.
First-generation ACI leaf switches are:
Cisco Nexus 9332PQ Switch
Cisco Nexus 9372PX-E Switch
Cisco Nexus 9372TX-E Switch
Cisco Nexus 9372PX Switch
Cisco Nexus 9372TX Switch
Cisco Nexus 9396PX Switch
Cisco Nexus 9396TX Switch
Cisco Nexus 93120TX Switch
Cisco Nexus 93128TX Switch
Figure 11. Static Routing L3 Out with SVI and vPC Supported with Only ACI 9300-EX Cloud Scale Switches
Figure 11 illustrates a topology that is only supported with Cisco Nexus 9300-EX Cloud Scale or newer leaf switches. In this topology, ACI is configured for static routing to an external active/standby firewall pair. To allow static routing from any border leaf switch to the active firewall, the L3Out uses the same encapsulation on all the border leaf switches.
If the border leaf switches are not Cisco Nexus 9300-EX Cloud Scale or newer, and for topologies consisting of more than two border leaf switches, use dynamic routing and a different VLAN encapsulation per vPC pair on the L3Out SVI.
In Figure 12, there are four border leaf switches, two in each datacenter. There are two L3Outs or a single L3Out that uses different VLAN encapsulations for DC1 and DC2. The L3Out configuration uses dynamic routing with an external device. For this design, there are no specific restrictions related to routing to the outside. This is because with dynamic routing, the fabric routes the traffic to the L3Out that has reachability to the external prefix without the need to perform bridging on an outside bridge domain.
Figure 12 illustrates a topology that is supported with both first and second-generation leaf switches.
Figure 12. Dynamic Routing L3 Out with SVI and vPC Supported with Any Version ACI Leaf Switches
Design Considerations for Attaching Endpoints to Border Leaf Switches
Dedicated border leaf switches offer increased availability. As an example, failure scenarios related to routing protocols do not impact server-to-server connectivity and a compute leaf switch failure does not impact outside reachability from another compute leaf switch.
However, for smaller scale datacenters it is sometimes necessary to use border-leaf switches also to connect workloads as depicted in Figure 13.
Figure 13. Design Considerations for Attaching Endpoints to Border Leaf Switches
The recommendations for this design need to take into account the policy-cam filtering optimization called "ingress filtering”, that is controlled by the configurable option "Policy Control Enforcement Direction" in the VRF configuration on the APIC:
The following considerations also apply to this design:
For best-practices related to the use of external routers or firewalls connected in vPC mode to the border leaf, please refer to the previous section.
This design is fully supported when the leaf switches in DC1 and DC2 are all Cisco Nexus 9300-EX Cloud Scale or newer.
If servers are connected to first-generation leaf switches, consider either of the following options:
If a VRF ingress policy is enabled (which is the default and the recommended practice), observe these guidelines:
Ensure that the software is ACI, Release 2.2(2e) or newer.
Configure the option to disable IP endpoint learning on the border leaf switches. You can disable endpoint learning on the border leaf switches by navigating to Disable Remote EP Learn., by selecting
Disable ingress filtering on the VRFs that have an L3Out configured. You can perform this configuration by navigating to Policy Control Enforcement Direction with the option Egress., by clicking the VRF and selecting