Have an account?

  •   Personalized content
  •   Your products and support

Need an account?

Create an account

Cisco ACI Remote Leaf Architecture White Paper

Networking Solutions White Paper

Available Languages

Download Options

  • PDF
    (6.2 MB)
    View with Adobe Reader on a variety of devices
Updated:January 22, 2020

Available Languages

Download Options

  • PDF
    (6.2 MB)
    View with Adobe Reader on a variety of devices
Updated:January 22, 2020
 

 

Introduction

With the increasing adoption of Cisco® Application Centric Infrastructure (Cisco ACI) as pervasive fabric technology, enterprises and service providers commonly need to interconnect separate Cisco ACI fabrics. Business requirements (business continuance, disaster avoidance, etc.) lead to the deployment of separate data center fabrics, and these need to be interconnected with each other.

Note:      To best understand the design presented in this document, readers should have at least a basic understanding of Cisco ACI and how it works and is designed for operation in a single site or pod. For more information, see the Cisco ACI white papers available at the following link: https://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/white-paper-listing.html.

Following figure shows the various Cisco ACI fabric and policy domain extension options that have been offered from the launch of Cisco ACI up to today. Cisco ACI Remote leaf is the latest addition to the ACI policy domain extension to satellite datacenters with consistent policy and centralized management.

Title: ACI fabric and policy domain evolution

Figure 1.         

ACI fabric and policy domain evolution

     The first option, available from Cisco ACI Release 1.0, consists of a classic leaf-and-spine two-tier fabric (a single Pod) in which all the deployed leaf nodes are fully meshed with all the deployed spine nodes. A single instance of Cisco ACI control-plane protocols runs between all the network devices within the pod. The entire Pod is under the management of a single Cisco Application Policy Infrastructure Controller (APIC) cluster, which also represents the single point of policy definition.

     The next step in the evolution of Cisco ACI geographically stretches a pod across separate physical data center locations, usually deployed in the same metropolitan area. Given the common limited availability of fiber connections between those locations, the stretched fabric uses a partial mesh topology, in which some leaf nodes (called transit leaf nodes) are used to connect to both the local and remote spine nodes, and the rest of the leaf nodes connect only to the local spine nodes. Despite the use of partial mesh connectivity, functionally the stretched fabric still represents a single-pod deployment, in which a single instance of all the Cisco ACI control-plane protocols run across all the interconnected data center sites, and so creates a single failure domain.

For more information about the Cisco ACI stretched-fabric deployment option, refer to the following link: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_kb-aci-stretched-fabric.html.

     To address the concerns about extending a single network fault domain across the entire stretched-fabric topology, Cisco ACI Release 2.0 introduced the Cisco ACI Multipod architecture. This model calls for the deployment of separate Cisco ACI Pods, each running separate instances of control-plane protocols and interconnected through an external IP routed network (or interpod network [IPN]). The Cisco ACI Multipod design offers full resiliency at the network level across pods, even if the deployment remains functionally a single fabric, with all the nodes deployed across the Pods under the control of the same APIC cluster. The main advantage of the Cisco ACI Multipod design is hence operational simplicity, with separate Pods managed as if they were logically a single entity.

For more information about the Cisco ACI Multipod design, refer to the following link: https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-737855.html.

Maximum latency of 50 msec RTT can be supported between Pods.

     The need for complete isolation across separate Cisco ACI networks led to the Cisco ACI Multisite architecture, introduced in Cisco ACI Release 3.0. Cisco ACI Multi-Site offers connectivity between the two completely separate ACI fabrics (Sites) that are managed by a Multisite Orchestrator (MSO). Each ACI fabric has an independent APIC cluster, and control plane to provide complete fault isolation. BGP EVPN is used to exchange control plane information and VXLAN is used for data-plane communication between ACI Sites and to extend the policy domain by carrying to policy information in the VXLAN header. Cisco ACI Multisite Orchestrator provides centralized policy definition (intent) and management by.

    Monitoring the health-state of the different ACI Sites.

    Provisioning of day-0 configuration to establish inter-site EVPN control plane.

    Defining and provisioning policies across sites (scope of changes).

    Inter-site troubleshooting.

For more information about the Cisco ACI Multisite design, refer to the following link: https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739609.html.

     The need to extend connectivity and consistent policies to remote locations where it is not possible or desirable to deploy a full ACI Pod (with leaf and spine nodes) led to the development of the Remote leaf solutions. With Cisco Remote leaf solutions, the APIC controller deployed in the ACI main Datacenter (DC) manages the Remote leaf switches connected over a generic IP Network (IPN). Remote Location DCs may not be large in size, hence the Remote leaf solution allows to deploy only Leaf switches, while the spines remain in ACI main DC. The APIC controller manages and operates the Remote leaf nodes as if those were local leaf switches, pushing there all the centrally defined ACI policies. This architecture is the main focus of this document and will be discussed in great detail in the following sections.

Business values of Remote leaf

The Remote leaf solution provides multiple business values since it can manage remote data centers without investing in APIC controllers and spine switches at remote locations. Following are key business values of Remote leaf Solution-

     Extension of the ACI policy model outside the main data center to remote sites distributed over IP backbone.

     Extension of ACI fabric to a small data center without investing in full-blown ACI Fabric.

     Centralized policy management and control plane for remote locations.

     Small form factor solution at locations with space constraints.

Title: Remote leaf use cases

Figure 2.         

Remote leaf use cases

There are multiple use-cases of Remote leaf in the DC, following are some example of these:

     Satellite/Co-Lo Data Center

Due to business demands, customers often have main DCs and many small satellite DCs distributed across multiple locations. There is a demand for centralized management of all these DCs to simplify operations. Customers also need to ensure that satellite DCs have same networking, security, monitoring, telemetry, and troubleshooting policy as main DC. ACI Remote leaf provides the solution for these demands.

Customers can use Remote leaf switches at Co-Lo facility to manage and apply policies in the same way as on prem DCs.

     Extension and Migration of DCs

Remote leaf deployment simplifies the provisioning of L2 and L3 multi-tenant connectivity between locations with consistent policy. During the migration of DCs, there is a need to build DCI over which workload can be migrated between DCs. Remote leaf allows the extension of L2 domain from ACI main DC to remote DCs. Customers can migrate workloads using vMotion using L2 extension built through Remote leaf solution.

     Telco 5G distributed DCs

Typically Telecom operators build DCs at central and edge locations, but now due to increasing demands, and provide better experience to subscribers, some services are moving to aggregation layer. DCs are becoming smaller, but lot more in number, there is a demand to have centralized management and consistent policy for these DCs. Cisco Remote leaf solutions allows the centralized management of these DC by providing full day-0 and day-1 automation, consistent day-1 policy and end to end troubleshooting across any location.

     Disaster Recovery

With the recent enhancement in Remote leaf, traffic forwarding from Remote leaf will continue to work even when it loses all the spines of a logically connected DC. This solution works for smaller Disaster Recovery (DR) DCs where the full fabric can’t be deployed. Details about this are explained in this white paper in the section “Failure handling in Remote leaf deployment.”

Architecture overview

Title: Remote leaf architecture overview

Figure 3.         

Remote leaf architecture overview

In Remote leaf solution, APIC controllers and spines remain at the main DC, while leaf switches at remote location (Remote leaf) logically connect to spines in main DC over IP network. Discovery, configuration and pushing policies to the Remote leaf is done by the APIC cluster at main DC. Remote leaf connects to the Spines of one of the Pod in Main DC over VXLAN tunnels.

Just like local leaf, Remote leaf can be used to connect virtual servers, physical servers and containers. Traffic to the end points connected to Remote leaf is locally forwarded through remote leaves. Remote location firewalls, routers, load-balancers or other service devices can be connected to Remote leaf switches similar to local switches. Customers can use ACI service graphs to perform service chaining for remote location DC with ACI Remote leaf solution. At the time of writing of this document, only unmanaged service graph mode is supported when connecting service nodes to Remote leaf switches. Local L2Out or L3Out can also be used to connect networking nodes at Remote locations using the Remote leaf solution.

Starting from Cisco ACI Release 4.1(2), traffic from a Remote leaf to any other Remote leaf within the same Pod or across Pods is directly forwarded instead of being hair-pinned to the spines of the ACI main DC Pod to which the RL nodes are associated (as shown in Figure 5 below).

Title: Traffic forwarding between Remote leaf Pairs before Cisco ACI Release 4.1(2)

Figure 4.         

Traffic forwarding between Remote leaf Pairs before Cisco ACI Release 4.1(2)

Title: Traffic forwarding between Remote leaf Pairs after Cisco ACI Release 4.1(2)

Figure 5.         

Traffic forwarding between Remote leaf Pairs after Cisco ACI Release 4.1(2)

In ACI Multi-Pod solution, Remote leaf nodes get logically associated with one specific Pod. In the figure below, Remote leaf pair is associated with ACI Pod1.

Title: Logical connectivity of Remote leaf to a spines of a Pod in a Multipod fabric

Logical connectivity of Remote leaf to a spines of a Pod in a Multipod fabric

Hardware and software support

Remote leaf solution is supported from ACI 3.1(1) release. Remote leaf solution is supported with cloud scale ASICs. Following table has the list of hardware that supports Remote leaf as of writing of this document. Please check latest release note for hardware support for specific release.

Spine

Leaf

Fixed Spine:

  N9364C
  N9332C
  N9K-C9316D-GX

Modular Spine with following Line cards

  N9732C-EX
  N9736C-FX
  N9736Q-FX
  N93180YC-EX
  N93180YC-FX
  N93108TC-EX
  N93108TC-FX
  N93180LC-EX
  N9348GC-FXP
  N9336C-FX2
  N93240YC-FX2
  N9358GY-FXP
  N93360YC-FX2
  N93216TC-FX2
  N9K-C9358GY-FXP
  N9K-C93600CD-GX
  N9K-C9364C-GX

Customers may already be using first generation Spines that do not support Remote leaf solution. In this case, first generation spines and next-gen spines can be part of the same ACI fabric. However, only next-gen spines as shown below should connect to the IPN to support Remote leaf. Following diagram shows how to connect mixed generation spines within fabric.

Title: Different generation of spines in ACI main DC

Different generation of spines in ACI main DC

IP Network (IPN) requirements for Remote leaf

Remote leaves connect to the ACI main DC over an IP Network (IPN). Below is the list of requirements for IPN:

1.     VXLAN is used as an overlay between ACI main DC and Remote leaf. To take care of the overhead of the VXLAN encapsulation, it’s important to increase at least 50B MTU in the IPN network to allow data-plane communication between endpoints in the main DC and at the remote location (this is valid assuming the endpoints source traffic of the default 1500B MTU size). Additionally, the MTU for control plane traffic between the spines in the main DC and the Remote leaf nodes should also be tuned (by default it tries to use up to 9000B MTU). The following snapshot from APIC controller highlights how to tune the MTU of control plane traffic to 1500 Bytes.

Title: System Settings

2.     Up to 300 msec latency between ACI main DC and Remote Location.

3.     Minimum 100 Mbps bandwidth in IP Network.

4.     Dedicated links between the RL nodes and the Uplink routers for RL discovery and VXLAN traffic destined to other RL nodes or the main fabric (that is, L3Out links must be separate from the fabric discovery link).

There are some changes done with respect to reachability requirements from Remote leaf to the ACI main DC starting from Cisco ACI Release 4.1(2) to solve the challenge of advertising the ACI main DC Pod’s TEP pool to remote locations. The following section explains this.

Reachability requirements from Remote leaf to ACI main DC before Cisco ACI Release 4.1(2)

Remote leaf is logically associated to one of the Pods of the ACI main DC. As previously mentioned, the Remote leaf nodes should have reachability to the VTEP pool of its logically associated Pod. This could be achieved via the backbone if the TEP pool is enterprise routable. Otherwise, the customer can use dedicated VRF or a tunneling mechanism to provide reachability to the ACI main DC Pod TEP pool from RL.

Reachability requirements from Remote leaf to ACI main DC from Cisco ACI Release 4.1(2)

To solve the challenge of advertising VTEP pool of the Pod to remote locations, we added the support of a routable subnet for the Pod, in addition to the private VTEP pool subnet. This allows customers to advertise only the routable subnet to remote locations from the ACI main DC, because all communication from Remote leaf to the ACI main DC happens using routable subnet IP addresses assigned to Spine, APIC controllers, and Border Leaf. ACI automatically assigns IP addresses to APIC controllers, spines, and border leaves from the routable subnet in addition to the private addresses.

A private VTEP pool is not required to be advertised outside of the ACI main DC. Customers can configure an additional routable subnet later when they need to add more ACI leaves, spines, or APIC controllers, instead of providing a larger routable subnet pool on day 0. The minimum supported number for a routable subnet is /27. Also, there is no requirement for the various routable TEP pools to be adjacent.

Title: Routable subnet reachability between RL and the ACI main DC

Figure 8.         

Routable subnet reachability between RL and the ACI main DC

Title: Routable subnet configuration

Figure 9.         

Routable subnet configuration

Please note that Routable subnet configuration is mandatory for supporting the establishment of direct communication between Remote leaf pairs and for deploying Remote leaf nodes with ACI Multi-Site. It is strongly recommended to deploy the routable subnet configuration for any new deployment.

The following configuration is needed on the upstream router connected to Remote leaf.

1.     OSPF with VLAN-4 sub-interface at the upstream router connected to Remote leaf.

2.     DHCP relay to the APIC controller’s IP address. This will be the routable IP address of the APIC if the routable subnet is configured.

Title: Configuration for the upstream router connected to Remote leaf with a private VTEP pool

Figure 10.     

Configuration for the upstream router connected to Remote leaf with a private VTEP pool

The following figure shows how to check the routable IP address of the APIC. The upstream router connected to RL needs to have a DHCP relay to the routable IP address of the APIC.

Title: Routable IP address of the APIC

Figure 11.     

Routable IP address of the APIC

Title: Configuration for the upstream router connected to Remote leaf with a routable subnet VTEP pool

Figure 12.     

Configuration for the upstream router connected to Remote leaf with a routable subnet VTEP pool

The following configuration is needed on the upstream router connected to Spine.

1.     OSPF with VLAN-4 sub-interface at the upstream router connected to the spines. Please note that starting from Cisco ACI Release 4.1(2), there is no need any more to deploy a separate VLAN-5 sub-interface for integrating RL nodes with Multi-Pod.

Following diagram provides the example configuration for upstream routers connected to spines.

Title: Configuration for upstream router connected to spine

Configuration for upstream router connected to spine

Recommended QOS configuration for Remote leaf

There may be a requirement to preserve the COS values coming from remote endpoints connected to EPGs and carry the same value to the ACI main DC Pod. To achieve this goal, it is possible to enable the preservation of the COS value (“Dot1p Preserve” flag) as part of the “External Access Policies” on APIC, as shown in the snapshot below. With this configuration, the COS value of the original frame received from the endpoint connected to the Remote leaf node is translated to a corresponding DSCP value that is then copied to outer IP header of the VXLAN encapsulated frame. This value can then be propagated across the IPN and converted back to the original COS value before sending the traffic to the endpoints connected in the main ACI DC (and vice versa).

Title: Recommended QOS configuration for Remote leaf

Another common requirement is to classify ACI QOS levels and default classes to a DSCP value within IPN, so that the user can prioritize traffic based on DSCP. To achieve this requirement, ACI Fabric should be enabled with “DSCP class-cos translation policy for L3 traffic.” The following snapshot from the APIC controller shows how to map ACI QOS levels and default classes to DSCP values in IPN. The traffic between RL and Spine in IPN is marked with DSCP based on the following mapping.

Title: DSCP class-cos translation policy for L3 traffic

Following are key points to remember while configuring QOS in ACI fabric for Remote leaf.

1.     ACI Fabric expects that the DSCP values of the packet exiting from Remote leaf and entering the spine and vice versa are the same, therefore the DSCP values of the packets cannot be modified inside the IPN.

2.     Dot1P preserve and DSCP translation policies cannot be enabled together.

3.     The following table explains the COS values that are used within ACI Fabric for the default QOS classes and recommended actions for these QOS classes in IPN.

 

COS value within fabric

Recommendation

Control plane

COS5

Prioritize in IPN

Policy plane

COS3

Prioritize in IPN

Traceroute

COS6

Prioritize in IPN

SPAN

COS4

De-prioritize in IPN

1G or 10G connectivity from Remote leaf switches to upstream Router

Upstream Routers at remote locations may not have 40G/100G connectivity. It might be required to connect Remote leaf switches with upstream routers with 1G or 10G connectivity. Remote leaf switches with QSA adapter can be connected with 1G or 1G connectivity. Please check latest optics compatibility matrix for QSA support.

Title: Options for connecting 1G or 10G links from Remote leaf to upstream router

Options for connecting 1G or 10G links from Remote leaf to upstream router

Discovery of Remote leaf

Remote leaf gets discovered and configured automatically as soon as it gets powered up at Remote Location. APIC controller at ACI main DC Pod and IPN needs to be pre-configured for this. For complete discovery of Remote leaf, following two process takes place.

1.     IP address allocation to uplink interfaces, and configuration push to Remote leaf.

2.     TEP IP address allocation to Remote leaf.

Title: IP address assignment for Remote leaf uplink, and configuration push to Remote leaf

Figure 15.     

IP address assignment for Remote leaf uplink, and configuration push to Remote leaf

1.     When Remote leaf gets powered on, it sends a broadcast DHCP discover message out of its uplink interfaces.

2.     When the DHCP discover message is received by upstream router, it relays the DHCP discover message to APIC controller. Upstream router’s interfaces towards Remote leaf are configured to relay DHCP discover to APIC controller.

3.     APIC controllers send DHCP offer for the uplink interface of Remote leaf along with bootstrap configuration file location in the DHCP offer message.

4.     Remote leaf picks up one of the APIC controller and sends a DHCP request for the uplink interface of the Remote leaf to the APIC controller.

5.     APIC controller sends DHCP ACK to complete the IP address allocation for the uplink IP address.

This process gets repeated for all the uplink addresses. To get bootstrap configuration file from APIC controller, Remote leaf automatically configures a static route with next-hop of upstream router.

After receiving configuration file, this static route is removed and Remote leaf is configured as per configuration file.

Next step in Remote leaf discovery is the assignment of TEP address to the Remote leaf switches Following diagram and steps explains the assignment on TEP address to Remote leaf.

Title: TEP address assignment to Remote leaf

Figure 16.     

TEP address assignment to Remote leaf

1.     Remote leaf gets full IP connectivity to the APIC controllers once it’s uplinks get ip addresses, it sends a DHCP discover message to APICs to receive the TEP address.

2.     APIC controllers sends the DHCP offer for TEP IP address.

3.     Remote leaf picks up one of the APIC controller and sends the DHCP request for the TEP IP.

4.     APIC controller sends DHCP ack to complete the DHCP discovery process for the TEP IP address.

Once this process is over, the Remote leaf becomes active into the Fabric and it’s shown in topology diagram at APIC controller.

Title: Fabric membership

Remote leaf switches can be connected to ACI Multipod Fabric. Following snapshot from APIC controller shows the ACI Fabric with two Pods. Remote leaf switches are part of Pod2.

Title: Topology - Pods: 2

 

Title: Pod - 2

Remote leaf software upgrade

Remote leaf switches can be upgraded in the same way as Local Leaf using APIC controllers. Customers can divide the switches into odd and even groups and upgrade one group at a time.

Following is the snapshot from APIC controller that divides Remote leaf switch along with other switches into odd and even groups for upgrades.

Title: Firmware Management

ACI connectivity and policy extension to Remote leaf

One of the main use-case of Remote leaf is to provide centralized management and consistent ACI Policy to remote locations DC. Customers can define ACI networking policies such as Tenant, VRF, BD, EPG, L3Outs, security policies such as contracts and micro-segmentation, telemetry policies, monitoring policies, troubleshooting policies for the main DC on APIC controller and extend the same policy to Remote DC using Remote leaf.

Remote leaf and Local Leaf have same ACI features and functionality. Single EPG can have EPs located in Local Leaf as well as Remote leaf. This functionality seamlessly offers L2 extension between ACI main DC and Remote Locations without much complexity and extra protocol overheads.

Following snapshot from APIC controller shows that EPG1 is stretched to Remote leaf. It has end points connected to Remote leaf (Node-110 and Node-111) and Local leaf (Node-201).

Title: APIC_Tenants

Endpoint connectivity option

Endpoints can be connected to Remote leaf switches in different ways, however Remote leaf switches should always be configured as part of vPC domain. The following options are possible (also shown in the diagram below):

1.     End points connected to Remote leaf switches using vPC port-channel (usually leveraging LACP).

2.     Endpoints are single home connected to one of the Remote leaf switch as ‘orphan port’.

3.     Endpoints are dual home connected to Remote leaf switches leveraging any form of active/standby NIC redundancy functionality (MAC pinning, NIC teaming, etc.).

Notice that option 2 and 3 is only supported starting from ACI 3.2 release.

Title: Options to connect end points to Remote leaf

Options to connect end points to Remote leaf

Remote leaf Control Plane and data plane

In a Cisco ACI fabric, information about all the endpoints connected to leaf nodes is stored in the COOP database available in the spine nodes. Every time an endpoint is discovered as locally connected to a given leaf node, the leaf node originates a COOP control-plane message to communicate the endpoint information (IPv4/IPv6 and MAC addresses) to the spine nodes. COOP is also used by the spines to synchronize this information between them.

In a Cisco ACI Remote leaf deployment as well, host information for discovered endpoints must be exchanged with spine nodes. Remote leaf builds a COOP session with the Spines and updates the Spine with the information of locally attached host.

There are specific TEP IP addresses that are defined on Spine and Remote leaf to exchange control plane and data plane information.

RL-DP-TEP (Remote leaf Data-Plane Tunnel End Point) - This is an automatically (and unique) IP address assigned to each Remote leaf switch from the TEP pool that is allocated to the Remote Location. VXLAN packets from a Remote leaf node are originated using this TEP as the source IP address when the endpoints are connected without vPC.

RL-vPC-TEP (Remote leaf vPC Tunnel End Point) – This is the anycast IP address automatically assigned to the vPC pair of Remote leaf nodes from the TEP pool that is allocated to the Remote location. VXLAN packets sourced from Remote leaf switches are originated using this TEP as the source IP address when endpoints are connected using vPC.

RL-Ucast-TEP (Remote leaf Unicast Tunnel End Point) – This is the anycast IP address automatically assigned to all the spines to which the Remote leaf switches are being associated. It is automatically assigned from the routable TEP pool if configured; otherwise, it is assigned from the ACI Pod TEP pool. When unicast packets are sent from endpoints connected to the RL nodes to an endpoint connected to the ACI main Pod local leaf, VXLAN encapsulated packets from RL are sent, with the RL-Ucast-TEP address as the destination and RL-DP-TEP or RL-vPC-TEP as the source. Any Spine in the ACI main DC Pod can then receive the traffic, de-encapsulate it, perform the required L2 or L3 lookup, and finally re-encapsulate it and forward it to a leaf in the local Pod.

RL-Mcast-TEP (Remote leaf Multicast Tunnel End Point) – This is another anycast IP address assigned to all the spines to which the Remote leaf switches are being associated. It is automatically assigned from the routable TEP pool if configured; otherwise, it is assigned from the ACI Pod TEP pool. When BUM (Layer 2 Broadcast, Unknown Unicast or Multicast) traffic is generated by an endpoint connected to the Remote leaf nodes, it gets VXLAN-encapsulated by the RL node and sent with the RL-Mcast-TEP address as the destination and RL-DP-TEP or RL-vPC-TEP as the source. Any of the spines in the ACI Pod can receive BUM traffic and forward it inside the fabric.

The following diagram shows the EP learning process on Spine through COOP session. It also shows the TEP addresses present on Remote leaf and Spines.

Title: TEP addresses on spine and Remote leaf

Figure 18.     

TEP addresses on spine and Remote leaf

Let’s understand how traffic forwards under different scenarios for a single Pod with Remote leaf:

     Unicast Traffic between endpoints dual-homed to a pair of Remote leaf nodes

     Unicast traffic between endpoints single-homed to a pair of Remote leaf nodes (orphan ports)

     Broadcast, Unknown Unicast and Multicast (BUM) traffic generated by an endpoint connected to a pair of Remote leaf nodes and sent to a Local Leaf (in the ACI Pod) when BD is configured in flood mode

     Broadcast, Unknown Unicast and Multicast (BUM) traffic generated by an endpoint connected to a pair of Remote leaf nodes and sent to a Local Leaf when BD has the default proxy mode configuration

     Unicast traffic from an endpoint connected to a Local Leaf to an endpoint connected to the Remote leaf nodes

     Unicast traffic from an endpoint connected to the Remote Leaf to an endpoint connected to a Local Leaf

Unicast traffic between Remote leaf Nodes (Dual-Homed Endpoints)

It’s always recommended to configure Remote leaf switches as part of a vPC domain even if the endpoints are single homed. When Remote Leaves are configured as a vPC pair, they will establish a vPC control plane session over the upstream router. Endpoint information in the same EPG is synchronized over the vPC control plane session.

Communication between dual-homed endpoints is usually handled locally on the RL node receiving the traffic from the source endpoint, a behavior named “Greedy Forwarding” and highlighted in the figure below.

Title: Unicast RL to RL packet flow when end point is connected to RL with vPC

Figure 19.     

Unicast RL to RL packet flow when end point is connected to RL with vPC

Unicast traffic between Remote Leaves with Orphan port

Communication between endpoints single attached to Remote Leaf nodes (orphan ports) is supported from ACI 3.2 release. EP information is synced between the Remote leaf switches over the vPC control plane session. In the following diagram EP1 and EP2 are part of same EPG. Since the EPG is deployed on both the RL nodes, they synchronize EP1 and EP2 information over the vPC control plane and traffic between the single-homed endpoints is forwarded establishing a VXLAN tunnel through the upstream router.

Title: Unicast RL to RL packet flow when the endpoint is connected to RL as an orphan port

Figure 20.     

Unicast RL to RL packet flow when the endpoint is connected to RL as an orphan port

Broadcast, Unknown Unicast and Multicast (BUM) traffic from Remote leaf to Local Leaf when BD in flood mode

Title: Broadcast, Unknown Unicast and Multicast traffic (BUM) traffic flow from RL to ACI main DC when bridge domain is in flood mode

Figure 21.     

Broadcast, Unknown Unicast and Multicast traffic (BUM) traffic flow from RL to ACI main DC when bridge domain is in flood mode

The ACI Remote leaf solution uses Head End Point Replication (HREP) for Broadcast, Unknown and Multicast (BUM) traffic when BD is configured in flood mode. In the above example, the endpoint (EP1) connected to the Remote leaf nodes sends an ARP request for the endpoint connected to local leaf (EP3), and the ARP request is first flooded within the Remote leaf pair. Remote leaf learns about the EP1 and send a COOP update to the spine about EP1’s location; with this COOP update, Spine learns that EP1 is attached to RL.

The Remote leaf that gets the ARP packet from the host also encapsulates the ARP packet in the VXLAN header and forwards a single packet to the Spines with the destination TEP address in the VXLAN header as the anycast address of the spine (RL-Mcast-TEP). The Remote leaf uses the source TEP address in the VXLAN header as the anycast address of the RL vPC pair (RL-vPC-TEP), since EP1 is connected to RL using vPC.

The Spine floods the packet to all the leaves using the BD multicast address. Every configured BD has a unique multicast address assigned inside the ACI Pod. When the Spine floods the ARP packet with BD multicast IP addresses, the ARP packet will reach only the leaves that have endpoints actively connected to this BD. Leaf switches in the ACI main DC make an entry for EP1 with next-hop as RL-vPC-TEP and flood the ARP request to all interfaces that are part of the BD.

The above process ensures that the following ARP request generated by EP1 and received by the spines can then be directly forwarded to the leaf where the EP3 is connected, allowing the completion of the ARP exchange between EP1 and EP3.

Broadcast, Unknown Unicast and Multicast (BUM) traffic from Remote leaf to Local Leaf when BD in proxy mode

Title: Broadcast, Unknown Unicast and Multicast traffic (BUM) traffic flow from RL to ACI main DC when bridge domain is in proxy mode

Figure 22.     

Broadcast, Unknown Unicast and Multicast traffic (BUM) traffic flow from RL to ACI main DC when bridge domain is in proxy mode

In the above example, the EP1 connected to Remote leaf switches sends the ARP request for the EP3 that is connected to Local Leaf. EP1 and EP3 are part of same BD, BD is in proxy mode, and ARP flooding is disabled. On receiving the ARP request, Remote leaf learns about the EP1 and sends a COOP update to Spine about the EP1 location; with this COOP update, Spine thus learns that EP1 is attached to RL. Remote leaf forwards the packet to the spine switches with RL-vPC-TEP as the source TEP address and RL-Ucast-TEP as the destination TEP address.

Spine forwards the ARP request to the destination leaf if it has information about EP3 in its database; otherwise, it sends a glean message to all of the leaves (including other Remote leaf nodes associated to the same ACI Fabric). The glean message triggers each leaf to send an ARP request on all of the local interfaces in the BD that received the ARP request.

The process described above ensures that the following ARP request generated by EP1 and received by the spines can then be directly forwarded to the leaf where EP3 is connected, allowing completion of the ARP exchange between EP1 and EP3.

Unicast traffic from Local Leaf to Remote leaf

Title: Unicast traffic flow from ACI main DC to RL

Figure 23.     

Unicast traffic flow from ACI main DC to RL

In the above example EP3 respond to ARP request sent from EP1. Following are the sequence of event when ARP response packet is forwarded to EP1 from EP3.

1.     EP3 sends a unicast ARP response to EP1 with sender MAC as EP3 and target MAC as EP1.

2.     Local Leaf in main DC receives the ARP response packet and looks for the EP1 MAC address in its hardware table. It finds EP1 with next-hop of RL-vPC-TEP. It encapsulates ARP response packet in the VXLAN header with Local Leaf TEP (LL TEP) as the source TEP address and RL-vPC-TEP as the destination TEP. As previously explained, the destination TEP would be RL-DP-TEP if the Remote leaf nodes were not configured as part of a vPC domain.

3.     When Spine receives the packet with RL-vPC-TEP as the destination TEP, it will update the source TEP as the anycast IP address of the spines (RL-Ucast-TEP) and forward the packet to RL.

4.     One of the RL receives the packet, de-encapsulates the VXLAN header and forwards the ARP response to the locally attached host. RL learns the EP3 with next-hop of the anycast IPaddress of the Spine (RL-Ucast-TEP).

Unicast traffic from Remote leaf to Local Leaf

Title: Unicast traffic flow from RL to ACI main DC

Figure 24.     

Unicast traffic flow from RL to ACI main DC

In the above example EP1 is sending a unicast packet to EP3. Following sequence of events happen when packet is sent to EP3 from EP1.

1.     EP1 sends a unicast packet to EP3 with source IP as local IP EP1 and destination as EP3 (same applies to the MAC addresses). EP1 picks one of the link towards Remote leaf for forwarding the packet based on hash.

2.     Packet from EP1 is received on one of the Remote leaf. Receiving Remote leaf does a Layer 2 look-up for EP3’s MAC. It finds the next-hop for EP3’s MAC as the anycast IPaddress of Spine (RL-Ucast-TEP). Remote leaf encapsulates the packet into VXLAN header with source TEP as RL-vPC-TEP and destination TEP as RL-Ucast-TEP.

3.     One of the Spine receives this packet. Spine does a look up for inner header destination (EP3’s MAC). It updates the destination TEP as Local Leaf TEP (TEP) and forward the packet to the leaf connected to EP3.

4.     Local Leaf forwards the packet to connected End point (EP3).

Traffic forwarding between Remote leaf pair before Cisco ACI 4.1(2)

The behavior shown in the figure below is applicable only to deployments running a code before Cisco ACI Release 4.1(2) or when “Remote Leaf direct traffic forwarding” is not enabled. Before Release 4.1(2), traffic between the Remote leaf pair gets forwarded through the Spines in the ACI main DC Pod as shown in the figure below.

Title: Traffic flow between Remote leaf pairs before Cisco ACI Release 4.1(2) without “Remote Leaf direct” feature

Figure 25.     

Traffic flow between Remote leaf pairs before Cisco ACI Release 4.1(2) without “Remote Leaf direct” feature

The following diagram and steps explain this behavior.

Title: Endpoint learning between Remote leaf pairs before Cisco ACI 4.1(2) without “Remote Leaf direct” feature

Figure 26.     

Endpoint learning between Remote leaf pairs before Cisco ACI 4.1(2) without “Remote Leaf direct” feature

In Figure 26, EP1 connected to Remote leaf Pair 1 sends unicast traffic to EP2 connected to Remote leaf Pair 2.

1.     When one of the Remote leaf pair 1 nodes receives this packet, assuming it does not have any information about EP2’s MAC in its hardware table, it forwards the packet to the spine. When doing so, the Remote leaf encapsulates the packet into VXLAN header with source TEP as RL Pair 1 TEP and destination as the anycast IP address of the Spine (RL-Ucast-TEP).

2.     This packet can be received by any spine of the main DC. Spine in Main DC looks up for the MAC address of EP2, and assuming it knows where it is located, it then forwards the packet to EP2 at Remote leaf Pair 2. When doing so, and this is the critical point to understand, it changes the source TEP to RL-Ucast-TEP and destination to RL-Pair 2 TEP.

3.     As a consequence, the Remote leaf Pair 2 will associate EP1’s MAC to the anycast IP address of the spine (RL-Ucast-TEP). Hence Remote leaf Pair 2 will always send packets to Spine in the ACI main DC before sending it to Remote leaf Pair 1.

This behavior is changed from Cisco ACI 4.1(2) if the customer has enabled “Remote Leaf direct traffic forwarding,” as explained in the following section.

Remote Leaf to Remote Leaf direct traffic forwarding from Cisco ACI 4.1(2)

Starting from Cisco ACI release 4.1(2), communication between endpoints connected to separate RL pairs can be established directly through the IPN without hair-pinning to the main DC Pod. This applies to both scenarios where different RL pairs are associated to the same Pod or to separate Pods that are part of a Multi-Pod fabric.

In the below figure, Remote location 1 is logically attached to Pod1, and Remote location 2 is logically attached to Pod2. Traffic between the endpoints in Remote location 1 and Remote location 2 is directly forwarded without being hair-pinned to Spine. Similarly, traffic from Remote location of Pod1 to the local leaves of Pod2 or Remote location of Pod2 to the local leaves of Pod1 is also directly forwarded instead of going to the Spines.

Title: Remote leaf to Remote leaf direct traffic forwarding

Figure 27.     

Remote leaf to Remote leaf direct traffic forwarding

The following key changes were made in Remote Leaf architecture to achieve Remote leaf to Remote Leaf direct traffic forwarding:

     Remote Leaf automatically builds VXLAN tunnels with all the other Remote Leaf switches within the same Pod or across Pods within a single ACI Fabric.

     Remote leaf builds VXLAN tunnels with its logically connected Pod’s spine as before, but from Cisco ACI 4.1(2),it will also form VXLAN tunnels with the spines of all the other Pods within a single ACI Fabric.

     Remote leaf learns the endpoints of all the other Remote leaves and local leaves with the next-hop of the tunnel toward other RLs or toward the spines of other Pods within a single ACI Fabric.

     Remote leaf forwards traffic over these tunnels. Since traffic is being directly forwarded over these tunnels, traffic forwarding from RL to other RLs and local leaves within the ACI Fabric is direct instead of hair-pinning to spines.

Details on traffic forwarding are covered in the individual sections on RL to RL traffic forwarding within a single Pod, across Pods, and across ACI Fabrics.

Control plane for RL direct traffic forwarding

The following section explains the changes to the control plane that were made to support Remote leaf to Remote leaf direct traffic forwarding.

1.     When an endpoint is learned in the fabric, the COOP database on all of the spines is updated. Remote leaf builds a local software database (SW DB) from the COOP database of spines. This SW DB will have information about all of the endpoints (EPs) for all of the VRFs deployed on it. For example, in the figure below, Remote leaf in Remote location 1 has VRF1 deployed on it; therefore, it will build a local software database (SW DB) of all of the endpoints (EP1, EP2, EP3) in VRF1.

2.     Remote leaf also has a hardware endpoint manager (HW EPM) database where it keeps information about all of the remote endpoints to which local endpoints have established active communication.

3.     Remote leaf updates the hardware endpoint manager (HW EPM) database from the software database based on data plane activity.

4.     When Remote leaf needs to send a packet toward a remote endpoint, it first performs a lookup for the destination in its HW EPM DB. If it finds an entry, it will forward the packet.

5.     If HW EPM DB does not have an entry for the destination, it will check the local SW DB. If the destination entry is available in SW DB, Remote leaf updates its HW EPM DB and forwards the packet. Keeping the SW DB locally allows Remote leaf to forward the packets even when the COOP connection to all of the Spines has failed.

6.     If the destination of the packet is not present in SW DB as well, RL will either flood or send it to the spine proxy, based on BD settings to discover the silent host.

7.     Once the endpoint is discovered, COOP will be updated on the spine and EP information in SW DB will be updated on all of the Remote leaves within the ACI site.

Title: Control plane for Remote leaf to Remote leaf direct traffic forwarding

Figure 28.     

Control plane for Remote leaf to Remote leaf direct traffic forwarding

Remote leaf to Remote leaf direct traffic forwarding within or across Pods in a fabric

This section provides details about the process of traffic forwarding between Remote leaf pairs within or across Pods in a fabric. As explained in the previous section, to achieve direct traffic forwarding between Remote leaf pairs, Remote leaf builds VXLAN tunnels automatically to the spines of all of the Pods that are part of the same fabric. A Remote leaf forwards packets to all of the other Remote leaves using these VXLAN tunnels.

In the figure below, RL1, RL2, RL3, and RL4 are part of the same ACI Pod. Each RL builds a tunnel to all of the other RLs with the destination “anycast TEP for vPC” to send traffic to EPs that are connected using vPC. For example, in the figure below, RL1 and RL2 build a tunnel with a destination of RL34-vPC-TEP. Similarly, RL3 and RL4 build a tunnel with a destination of RL12-vPC-TEP.

Each RL also forms tunnels to all of the other RLs with RL-DP-TEP as the destination address to send traffic to EPs that are connected through orphan ports. For example, in the figure below, RL1 builds tunnels with destinations RL2-DP-TEP, RL3-DP-TEP, and RL4-DP-TEP. Similarly, all the other RLs also build tunnels with each other’s DP-TEPs.

Title: VXLAN tunnels between Remote leaf pairs for Remote leaf to Remote leaf traffic forwarding within a single Pod

Figure 29.     

VXLAN tunnels between Remote leaf pairs for Remote leaf to Remote leaf traffic forwarding within a single Pod

Let’s understand how traffic forwards under different scenarios between Remote leaf pairs within and across Pods:

     Unicast traffic between Remote Leaf pairs associated to the same Pod

     Unicast traffic between Remote Leaf pairs associated to separate Pods of the same fabric

Unicast traffic between Remote leaf pairs associated to the same Pod

In the figure below, EP1, EP2, EP3, and EP4 are part of the same BD. All of these EPs are not silent hosts; therefore, COOP on the spines has information about the location and next-hops of these EPs. As explained in the section “Control plane for RL direct traffic forwarding,” since the same VRF/BD is deployed on all of the RLs, each RL downloads information of all of the EPs into the local software DB (SW DB).

Title: Traffic forwarding from EP2 to EP3 across different Remote leaves

Traffic forwarding from EP2 to EP3 across different Remote leaves

4.     Assume that EP2 is sending a packet to EP3. EP2 picks one of the links toward the Remote leaf for forwarding the packet based on the hash.

5.     The packet from EP2 is received by one of the Remote leaves. The receiving Remote leaf performs a Layer 2 lookup of EP3’s MAC address in the HW EPM DB. It finds the next-hop for EP3’s MAC address as DP-TEP of RL3 (RL3-DP-TEP) in the SW DB. The Remote leaf encapsulates the packet in the VXLAN header with the source TEP as RL12-vPC-TEP and the destination TEP as RL3-DP-TEP, and forwards to RL3.

6.     RL3 de-encapsulates this packet, performs a Layer 2 lookup in the MAC address table associated to the BD, and forwards it to EP2. RL3 also updates its HW EPM DB with EP2’s IP and MAC addresses with a next-hop of RL12-vPC-TEP.

Title: Traffic forwarding from EP3 to EP2 across different Remote leaves

Figure 31.     

Traffic forwarding from EP3 to EP2 across different Remote leaves

7.     Similarly, let’s look at EP3 sending a packet to EP2.

8.     RL3 performs a Layer2 lookup of the EP2 MAC address in its HW EPM DB. It finds the EP2 MAC address with the next-hop of RL12-vPC-TEP. RL3 then encapsulates and forwards the packet with the source TEP as RL3-DP-TEP and the destination TEP as vPC TEP of RL12 (RL12-vPC-TEP).

9.     The packet from RL3 will be received by either RL1 or RL2. The receiving RL updates its HW EPM DB with EP3’s IP and MAC addresses with a next-hop of RL3-DP-TEP. It also updates its vPC peer with the EP3 location. The receiving RL forwards the packet to EP2 after performing a Layer2 lookup.

When EP1, EP2, EP3, and EP4 are forwarding packets to each other, the resulting HW EPM DB table on RL1, RL2, RL3, and RL4 is captured in the figure below.

Title: EP learning on RL within a Pod

Figure 32.     

EP learning on RL within a Pod

Unicast traffic between Remote leaf pairs associated to separate Pods of the same fabric

Traffic forwarding between Remote leaf pairs works in the same way whether it is within a Pod or across Pods, as shown in the figure below. For example, in the figure, traffic between Remote leaf pairs across Pods is forwarded directly.

Title: Traffic forwarding between Remote leaves across Pods

Figure 33.     

Traffic forwarding between Remote leaves across Pods

Remote leaf switches form VXLAN tunnels and learn from the EPs connected to Remote leaf pairs in different Pods.

In the figure below, RL1 and RL2 are logically associated to Pod1, while RL3 and RL4 are logically associated to Pod2.

When EP1, EP2, EP3, and EP4 are forwarding packets to each other, the resulting EP table for RL1, RL2, RL3, and RL4 is captured in the figure below. Since EP2 and EP4 are connected through vPC, these EPs are learned with the next-hop as vPC-TEP. Because EP1 and EP3 are connected through orphan ports, these EPs are learned with next-hop as DP-TEP.

Title: EP learning on RL across Pods

Figure 34.     

EP learning on RL across Pods

Unicast traffic forwarding between Remote and local leaf across Pods with RL direct forwarding

Remote leaf switches forward the traffic to the local leaves of all of the other Pods directly, without forwarding it to logically connected spines. The following figure shows this behavior.

Title: Traffic forwarding between Remote leaves and local leaves across Pods

Figure 35.     

Traffic forwarding between Remote leaves and local leaves across Pods

To achieve direct traffic forwarding to the local leaves of other Pods, a Remote leaf automatically forms VXLAN tunnels to the Spines of all Pods, including the Pod where it is logically connected.

In the figure below, EP1, EP2, EP3, EP4, EP5, and EP6 are in same BD. On each RL, there will be a tunnel with the destination of the anycast IP address of the spines (RL-Ucast-TEP) of each Pod. This tunnels on the Remote leaf is used to forward traffic from the Remote leaf to the local leaves of all Pods.

Similarly, Spines will have tunnels to each RL with a destination of RL-DP-TEP or RL vPC-TEP, based on whether the EP on the RL is connected using vPC or an orphan port. This tunnel on the spine is used to forward Spine proxy–related packets from Spine to RL.

Title: VXLAN tunnels between RL and Spine across Pods

Figure 36.     

VXLAN tunnels between RL and Spine across Pods

Let’s understand how traffic forwards under different scenarios between Remote leaves and local leaves across Pods:

     Unicast traffic from local leaves to Remote leaves across Pods

     Unicast traffic from Remote leaves to local leaves across Pods

Unicast traffic from local leaves to Remote leaves across Pods

Unicast traffic from local leaves to Remote leaves across Pods works similarly to how it works in a single Pod. In the figure below, EP1, EP2, EP3, EP4, EP5, and EP6 are part of same BD. All these EPs are not silent hosts; therefore, COOP on the spines has information about the location and next-hops of these EPs. All the Remote leaves will also download all information from the EPs into its local SW DB because the same VRF/BD is deployed on all of the Remote leaves and local leaves.

Title: Traffic forwarding from local leaves to Remote leaves across Pods

Figure 37.     

Traffic forwarding from local leaves to Remote leaves across Pods

1.     Assume that EP6 connected to Pod2’s local leaf is sending a packet to EP2 connected to a Remote leaf of Pod1. EP6 picks one of the links toward the local leaf for forwarding the packet based on the hash.

2.     The receiving local leaf performs a Layer 2 look-up of EP2’s MAC address. It finds the next-hop for EP2’s MAC address as RL12-vPC-TEP in the HW EPM DB. The receiving leaf encapsulates the packet with the source TEP address as Local Leaf TEP (LL TEP) and the destination TEP as RL12-vPC-TEP.

3.     When the spine receives the packet with the destination TEP as RL12-vPC-TEP, it will update the source TEP as the anycast IP address of the Spine (RL-Ucast-TEP) of Pod2. Spine forwards the packet to the anycast IP address of RL vPC pair (RL12-vPC-TEP).

4.     One of the RL receives the packet, de-encapsulates the VXLAN header, and forwards the packet to the locally attached host. RL learns the EP6 with the next-hop of the anycast IP address of the Spine (RL-Ucast-TEP) of Pod2 and updates the information in HW EPM DB.

Unicast traffic from Remote leaf to local leaf across Pods

Title: Traffic forwarding from Remote leaves to local leaves across Pods

Figure 38.     

Traffic forwarding from Remote leaves to local leaves across Pods

1.     When EP2 sends a packet to EP6, EP2 picks one of the links to the Remote leaf for forwarding the packet based on the hash.

2.     The packet from EP2 is received by one of the Remote leaves. The receiving Remote leaf performs a Layer 2 lookup of EP6’s MAC address in the HW EPM DB. It finds the RL-Ucast-TEP of Pod2 as the next-hop for EP6. The Remote leaf encapsulates the packet in the VXLAN header with the source TEP as RL12-vPC-TEP and the destination as RL-Ucast-TEP of Pod2, and forwards it to the spines of Pod2.

3.     One of the spines receives this packet. The spine performs a lookup for the inner header destination (EP2’s MAC address). It updates the destination TEP as the Local Leaf TEP (TEP) and forwards the packet to the leaf connected to EP6.

4.     The local leaf forwards the packet to the locally connected endpoint (EP6) and updates its hardware EPM DB with the EP2 with a next-hop of RL12-vPC-TEP.

When EP1, EP2, EP3, and EP4 begin bidirectional communications with EP6, the resulting EP table on the local leaf and Remote leaf will look as shown in the following figure.

Title: EP learning between Remote leaves and local leaves across

EP learning between Remote leaves and local leaves across

Remote leaf with Cisco ACI Multi-Site

Remote leaf and Cisco ACI Multi-Site are supported together, starting from Cisco ACI 4.1(2). Packets from RL to another site’s Remote leaf or local leaf are forwarded through the spines of the logically connected Pod. RL does not build VXLAN tunnels with the RLs or the spines of the other site, to avoid building VXLAN tunnels across sites. For this reason, packets from RL to other sites are forwarded through the spines of the logically attached Pod’s spines, as shown in the figure below.

Title: Traffic between Remote Leaf pairs across Sites

Figure 40.     

Traffic between Remote Leaf pairs across Sites

Let’s understand how traffic forwards between Remote leaf pairs across sites. In the example below, RL1 and RL2 are part of ACI Site1, and RL3 and RL4 are part of ACI Site2. The Remote leaf in each site forms a VXLAN tunnel with spines of the Pod where the Remote leaf is logically connected; the destination TEP of this tunnel would be the Anycast IP address of the spine used for Cisco ACI Multi-Site (DCI-Ucast-TEP).

When the Remote leaf is deployed with Cisco ACI Multi-Site, this tunnel is used to communicate to all of the EPs connected in the other sites. This tunnel on the Remote leaf is also used to communicate with EPs connected to the local leaves within the same site when the Remote leaf is deployed with Cisco ACI Multi-Site.

In the figure below, RL1 and RL2 form VXLAN tunnels using the anycast IP address of the spine for the Multi-Site (Site1-DCI-Ucast-TEP) of its Pod. Similarly, RL3 and RL4 form VXLAN tunnels with the anycast IP address of the spine for the Multi-Site (Site2-DCI-Ucast-TEP) of its Pod. Please note that each Pod in the ACI site has a separate DCI-Ucast-TEP address. This is the anycast IP address assigned to all spines in that Pod. In the figure below, VXLAN tunnels are shown only with single spines, but same tunnels are formed with all of the spines.

Title: VXLAN tunnels from RL to ACI main DC with Cisco ACI Multi-Site

Figure 41.     

VXLAN tunnels from RL to ACI main DC with Cisco ACI Multi-Site

In the figure above EP1, EP2, EP3, EP4, EP5, and EP6 are part of the same VRF. All of these EPs are not silent hosts; therefore, COOP on the spines has information about the location and next-hop of these EPs. All of the RLs will also download all of the EPs’ information into its local SW DB because the same VRF/BD is deployed on all of the RLs and local leaves.

Unicast traffic between RL pairs across Sites

Title: Traffic forwarding between Remote leaf pairs across Sites

Figure 42.     

Traffic forwarding between Remote leaf pairs across Sites

1.     Assume that EP2 is sending a packet to EP3 with the source IP as the local IP EP2 and the destination as EP3’s IP. EP2 picks one of the links toward the Remote leaf for forwarding the packet based on the hash.

2.     The packet from EP2 is received by one of the Remote leaves. The receiving Remote leaf performs a Layer 3 lookup of EP3’s IP in the HW EPM DB. It finds the next-hop for EP3’s IP as Site1-DCI-Ucast-TEP. Each Pod in the ACI site has a separate DCI-Ucast-TEP address. This is the anycast IP address assigned to all of the spines in that Pod. The Remote leaf encapsulates the packet in the VXLAN header with the source TEP as RL12-vPC-TEP and the destination TEP as Site1-DCI-Ucast-TEP. This packet can be received by the spines of the logically connected Pods of RL1 and RL2.

3.     The receiving Spine performs a lookup of EP3’s IP. It finds the next-hop for EP3’s IP as Site2-DCI-Ucast-TEP. It changes the source TEP to Site1-DCI-Ucast-TEP and the destination TEP to Site2-DCI-Ucast-TEP and forwards the packet to the spines of the logically connected Pods of RL3 and RL4.

4.     The receiving Spine of Site2 performs a Layer3 lookup of EP3’s IP. It finds the next-hop for EP3’s IP as RL3-DP-TEP. It changes the source TEP to Site2-DCI-Ucast-TEP and the destination TEP to RL3-DP-TEP.

5.     RL3 receives this packet, de-encapsulates it, and sends it to EP2. It will also update its HW EPM DB with EP2, with a next-hop of Site2-DCI-Ucast-TEP.

A similar process is performed when a packet is sent from EP3 to EP2.

When EP1, EP2, EP3, and EP4 are forwarding packets to each other, the resulting EP tables on RL1, RL2, RL3, and RL4 is captured in the figure below.

Title: EP Learning between Remote leaf pairs across Pods

Figure 43.     

EP Learning between Remote leaf pairs across Pods

Broadcast, Unknown Unicast and Multicast (BUM) traffic with RL direct when BD is in flood mode

The section below covers how two endpoints that are connected to Remote leaves forward Broadcast, Unknown Unicast and Multicast (BUM) traffic with RL direct enabled and when BD is in flood mode. This section assumes that EP2 and EP3 are silent hosts, and that each is connected to a different Remote leaf pair. The following section takes an example of an ARP request being forwarded from EP2 to EP3. The following figure shows EP2 and EP3 connected to different Pods, but the section explains the behavior whether endpoints are connected within same Pod or across Pods. BD stretching is not supported across ACI sites.

Title: BUM traffic forwarding for silent host when BD is in flood mode and RL direct is enabled

Figure 44.     

BUM traffic forwarding for silent host when BD is in flood mode and RL direct is enabled

1.     When EP2 sends an ARP request for EP3, it picks one of the links toward the Remote leaf for forwarding the ARP request packet based on the hash.

2.     Since EP3 is a silent host, EP3’s information is not available in the COOP DB, SW DB on RL, and HW EPM DB on the receiving Remote leaf. As explained in the section “Control plane for RL direct traffic forwarding”, the receiving Remote leaf tries to resolve the ARP for EP3.

3.     The receiving Remote leaf will encapsulate the ARP request packet in the VXLAN header and forward it to all other RLs within and across Pods using VXLAN tunnels. It will encapsulate the ARP packet with the source TEP as RL-vPC-TEP (RL12-vPC-TEP) and the destination TEP as DP-TEP for each RL. All of the RLs that receive this packet will make an entry for EP2 with a next-hop of RL12-vPC-TEP.

4.     Each receiving Remote leaf will de-encapsulate the VXLAN packet and flood the ARP request in the locally attached segment to resolve the ARP for EP3.

5.     The Remote leaf that receives the ARP request directly from EP2 will also send a single packet to the spines in its Pod with the destination TEP address in the VXLAN header as the anycast address of the spine (RL-Mcast-TEP) and with the source as RL12-vPC-TEP.

6.     The spines flood the packet to all of the leaves in its local Pod using the BD multicast address. Every configured BD has a unique multicast address assigned inside the ACI Pod. When a spine floods the ARP packet with BD multicast IP, it reaches only the leaves that have endpoints actively connected to this BD. Each receiving leaf de-encapsulates the VXLAN packet and floods ARP request in the locally attached segment to resolve the ARP for EP3.

7.     One of the spines of the Pod where Remote leaf is logically attached also floods the ARP request to the spines in the other Pods using inter-pod multicast (IPN). This will make sure the ARP request is received by all of the Pods within a single site. Each receiving leaf in the Pod de-encapsulates the VXLAN packet and floods the ARP request in the locally attached segment to resolve the ARP for EP3. Each receiving leaf will learn EP3’s information with a next-hop of RL12-vPC-TEP.

Please note that spines are configured with an ACL entry that blocks forwarding BUM traffic to any other RL if BUM traffic is received from an RL. This will make sure that RLs do not receive duplicate copies of BUM traffic from an RL as well as from the spines. Duplicate copies of the same packet would have created EP flap on RLs.

The process described above ensures that the following ARP request generated by EP2 and received by the spines can then be directly forwarded to the leaf where EP3 is connected, allowing the completion of the ARP exchange between EP1 and EP3 (as discussed in the unicast traffic forwarding section).

Broadcast, Unknown Unicast and Multicast (BUM) traffic with RL direct when BD is in proxy mode

The section below covers how two endpoints that are connected to Remote leaves forward Broadcast, Unknown Unicast and Multicast (BUM) traffic with RL direct enabled and when BD is in proxy mode. This section assumes that EP2 and EP3 are silent hosts. and that each is connected to a different Remote leaf pair. The following section takes an example of an ARP request being forwarded from EP2 to EP3. The following figure shows EP2 and EP3 connected to different Pods, but the section explains the behavior whether endpoints are connected within the same Pod or across Pods. BD stretching is not supported across ACI sites.

Title: BUM traffic forwarding for silent host when BD is in proxy mode and RL direct is enabled

Figure 45.     

BUM traffic forwarding for silent host when BD is in proxy mode and RL direct is enabled

1.     When EP2 sends an ARP request for EP3, it picks one of the links toward the Remote leaf for forwarding the ARP request packet based on the hash.

2.     Since EP3 is a silent host, EP3’s information is not available in the COOP DB, SW DB on RL, and HW EPM DB on the receiving RL. As explained in the section “Control plane for RL direct traffic forwarding,” the receiving RL tries to resolve the ARP for EP3.

3.     The receiving Remote leaf will encapsulate the ARP request packet in the VXLAN header and forward it to the spine proxy with a source TEP as RL-vPC-TEP (RL12-vPC-TEP) and a destination TEP as Pod1-RL-Ucast-TEP.

4.     The spine forwards the ARP request to the destination leaf if it has information about EP3 in its database; otherwise, it sends a glean message to all of the leaves, including the Remote leaves in its site.

5.     The glean message triggers each leaf to send an ARP request on all of the local interfaces in the BD that received the ARP request. The spine will also send one copy of the glean message to the spines of the other Pods. Those Pods’ spines will forward the glean packet to the all of the leaves, including the Remote leaves.

6.     This triggers an ARP reply from EP3. When the leaf receives the ARP response, it updates its hardware table and sends a COOP update to the spine with the EP3 location.

The process described above ensures that the following ARP request generated by EP2 and received by the spines can then be directly forwarded to the leaf where EP3 is connected, allowing the completion of the ARP exchange between EP1 and EP3 (as discussed in the unicast traffic forwarding section).

Inter-VRF traffic between Remote leaf switches

In ACI, inter-VRF traffic always gets forwarded to the spine proxy function to find the policy information of the packet destined to a different VRF; therefore, when the endpoints are connected to the Remote leaf nodes, inter-VRF traffic gets forwarded through the spines in the main data center. However, an optimization is introduced from Cisco ACI Release 4.0(1), allowing inter-VRF traffic between endpoints connected to Remote leaf switches to be locally forwarded.

Title: Inter-VRF traffic within a Remote leaf pair before Cisco ACI Release 4.0

Figure 46.     

Inter-VRF traffic within a Remote leaf pair before Cisco ACI Release 4.0

However, from Cisco ACI Release 4.0(1), inter-VRF traffic between endpoints connected to Remote leaf switches is locally forwarded through a WAN router. The reason packets are sent to an upstream router is that, when an incoming packet from EP1 in VRF1 arrives at RL, it will not have the policy information of EP2, which is in VRF2; because of this, RL cannot forward the packet to EP2 directly. To overcome this, RL forms a new tunnel to itself through the WAN router. All inter-VRF packets are sent, through this tunnel, from one VRF to a second, receiving VRF. In the figure below, RL sends packets in VRF1 through this tunnel, to be received by VRF2, so that it can discover the policy and contract information of EP2.

Title: Inter-VRF traffic within a Remote-leaf pair from Cisco ACI Release 4.0(1)

Inter-VRF traffic within a Remote-leaf pair from Cisco ACI Release 4.0(1)

VMM Domain Integration and vMotion with Remote leaf

ACI Fabric allows the integration with Multiple VMM (Virtual Machine Manager) domains. With this integration, APIC controller pushes the ACI policy configuration such as networking, telemetry monitoring and troubleshooting to switches based on the location of virtual instances. APIC controller can pushes the ACI policy in the same way as Local Leaf. A single VMM domain can be created for compute resources connected to both the ACI main DC Pod and Remote leaf switches. VMM-APIC integration is also used to push a VDS to those hosts managed by the VMM and dynamically create port-groups as a result of the creation of EPGs and their association to the VMM domain. This allows to enable mobility (‘live’ or ‘cold’) for virtual endpoints across different compute hypervisors.

Note:      It is worth noticing that mobility for virtual endpoints could also be supported if VMM domain is not created (i.e. VMs are treated as physical resources).

Virtual instances in same EPG or L2 domain (VLAN) can be behind local leaf as well as Remote leaf. When Virtual instance moves from Remote leaf to Local Leaf or vice versa, APIC controller detects the leaf switches where virtual instance is moved, and pushes the associated policies to new leaf. All VMM and container domain integration supported for Local leaf are supported for Remote leaf as well.

Title: vMotion between RL to ACI main DC

Figure 48.     

vMotion between RL to ACI main DC

Above example shows the process of vMotion with ACI Fabric. The following events happen during a vMotion event:

1.     The VM has IP address “100.1.1.100” and a default gateway of 100.1.1.1 in VLAN 100. When the VM comes up, ACI fabric configures the VLAN and the default gateway of the Leaf switches where the VM is connected. The APIC controller also pushes the contract and other associated policies based on the location of the VM.

2.     When the VM moves from a Remote leaf to a Local Leaf, the ACI detects the location of VM through VMM integration.

3.     Depending on the EPG specific configuration, the APIC controller may need to pushes the ACI policy on the Leaf for successful VM mobility or policy may already be existing on destination leaf.

Following is the snapshot from APIC controller where single VMM domain is extended from Local Leaf in ACI main DC to Remote leaf.

Related image, diagram or screenshot

External Connectivity from Remote leaf

External connectivity to Remote DC can be provided through Local L3Out connections on Remote leaf switches. Following diagram provides an example of how local L3Out on Remote leaf works.

Title: Control plane with Local L3Out on RL

Figure 49.     

Control plane with Local L3Out on RL

In the above example, Remote leaf has local L3Out connection to external router. External router is connected to Remote leaf switches over vPC with SVI. External prefixes on Remote leaf switches are learnt with next-hop of Router learnt over SVI.

Remote leaf advertises the externally learnt prefixes to spines through the MP-BGP VPNv4 session between RLs and Spines. Spines in ACI main DC act as BGP Route Reflector (RR) for both Remote leaf and Local Leaf. Spines advertises the prefixes received from the Remote leaf nodes to the local leaf switches through Intra-Pod MP-BGP VPNv4 sessions.

ACI main DC can also have a local L3Out connection to connect to the external Layer 3 domain. Server Leaves in ACI main DC learns the external prefixes with the next-hop of local Border Leaf TEP addresses. ACI Main Pod prefers BL1-TEP and BL-2-TEP compared to RL1-DP-TEP and RL2-DP-TEP due to better is metric.

As a consequence, endpoints connected to either Remote leaf nodes or Local Leaf nodes use by default the local L3Out for external connectivity. However, ACI main DC L3Out can act as a backup for RL, and vice versa.

Title: Data plane with Local L3Out on RL

Figure 50.     

Data plane with Local L3Out on RL

Both Remote DC and ACI main DC may have Local L3Outs for WAN connectivity and there could be cases where hosts belonging to the same IP subnet are deployed in both locations. In this case, as explained in the previous section, the local L3Out connection will normally be preferred for outbound traffic from DC to WAN. However, since the Border Leaf nodes in the ACI main DC Pod and the Remote leaf switches may be advertising the same IP subnet prefixes toward the external Layer 3 network domain, incoming traffic may take a sub-optimal path. In such scenario, incoming traffic destined to an endpoint connected to the Remote leaf switches might in fact ingress the L3Out in the ACI main DC, if endpoints in the same IP subnet of the remote endpoint are connected in the Main DC. The traffic would then be forwarded from the ACI main DC to the endpoint connected to the Remote leaf switches via the IPN; when the endpoint replies, the outbound traffic will take the path via the local L3Out creating an asymmetric traffic path behavior that may cause traffic drop if perimeter stateful devices (like firewalls) are deployed between the leaf switches and the external routers.

Title: Possible traffic asymmetry with same prefix being advertised from RL and ACI main DC

Figure 51.     

Possible traffic asymmetry with same prefix being advertised from RL and ACI main DC

A possible solution to this problem is providing a more granular advertisement of routing information into the WAN, whereby the BL nodes in the ACI main DC would advertise not only the IP subnet but also the specific endpoints belonging to that IP subnet and discovered in the main Pod. At the same way, the RL nodes would advertise into the WAN the IP subnet and the specific host routes for endpoints belonging to that IP subnet and locally discovered.

This host routes advertisement capability will be available on ACI leaf switches starting from ACI software release 4.0, and ensures that both ingress and egress paths are symmetric and use the same Local L3Out connection (as shown in the diagram below). With this capability, it is hence possible to deploy independent pairs of perimeter firewalls in the main DC and at the Remote leaf location.

Title: Solution for traffic asymmetry with same prefix being advertised from RL and ACI main DC

Solution for traffic asymmetry with same prefix being advertised from RL and ACI main DC

Failure handling in Remote leaf deployment

This section captures the different failure scenarios and explains how the network behaves during those failures.

Spine failure in main DC

Remote leaf switches uses the anycast IP address of the Spine to send unicast or BUM traffic toward the ACI Pod. If a spine fails, alternatively available spine can accept traffic from Remote leaf and forward to final destination. Leaf switches in main DC are attached to multiple spine, and uses ECMP to pick a spine to forward traffic. When one spine fails, Local leaf switches can pick alternative available spine to forward traffic to Remote leaf.

Title: Traffic forwarding when spine fails in ACI main DC

Figure 53.     

Traffic forwarding when spine fails in ACI main DC

Remote leaf failure

It’s always recommended to use vPC for Remote leaf switches. When the Remote leaf nodes are part of a vPC domain, packets from the spines are forwarded to the anycast IP of the Remote leaf vPC pair. If one Remote leaf fails, traffic will be rerouted toward the second Remote leaf that can accept and forward packets to attached host.

Similarly, when a host is dual-homed to the Remote leaf pair through vPC, the failure of a Remote leaf node would simply cause the re-shuffling of flows on the link connecting to the second RL switch.

Title: Traffic forwarding when one Remote leaf fails

Figure 54.     

Traffic forwarding when one Remote leaf fails

Upstream router on RL is lost

It’s recommended to use multiple upstream router for the redundancy purpose. When one of the upstream router fails, Remote leaf switches can use alternative available upstream router for traffic forwarding.

Title: Upstream router failure at Remote location

Figure 55.     

Upstream router failure at Remote location

All Upstream routers connected to RL are lost

When Remote leaf switches are configured with vPC domain, and end host/L3Out is connected to Remote leaf switches with vPC, if all upstream routers fail, then all the uplinks of the RL nodes will fail, and as a consequence the vPC links toward the endpoints are brought down. This behavior is to avoid traffic blackholing when the EP attached to the Remote leaf can send packets, but the Remote leaf cannot forward due to uplink failure.

Title: Failure scenario when RL loses all upstream routers, or all uplink connections

Figure 56.     

Failure scenario when RL loses all upstream routers, or all uplink connections

Failure of APIC controller reachability in a single Pod

When all APIC controllers in a single Pod are lost, Remote leaf switches will not experience any data-plane or control plane failures. They will continue to forward traffic to all destinations; however, new configurations cannot be applied since APIC controller connectivity is down. Since all APIC controllers are down, operations and visibility to the Remote leaf will be lost; however, users can login to the Remote leaf using an out-of-band or console connection.

Title: Remote leaf with failure of all APIC controllers

Figure 57.     

Remote leaf with failure of all APIC controllers

Failure of APIC controller reachability, with Multi-Pod and RL-direct

Starting from Cisco ACI Release 4.2(2), Remote leaf can communicate to all the ACI Pods directly without depending on the spines in the logically connected DC. Multi-Pod will typically be used with APIC controllers distributed across Pods. In case the Remote leaf loses reachability to all of the APIC controllers of its associated Pod, it can still be managed from the APIC controllers of the other available Pods as shown in the figure below.

Title: Remote leaves managed by an alternate Pod’s APIC controller

Figure 58.     

Remote leaves managed by an alternate Pod’s APIC controller

Failure of all-spine reachability, with Multi-Pod and RL direct enabled

Deployment of Remote leaf with Multi-Pod is recommended to build a resilient design. Reachability to the spine could fail either due to the failure of the IP Network between Remote leaf and Spine, or due to the failure of all of the spines in a logically attached Pod of the Remote leaf.

In such a case, starting from Cisco ACI Release 4.2(2), the Remote leaf will keep forwarding traffic to all destinations, as shown in the figure below.

Title: Traffic forwarding behavior on RL during spine failure with Multi-pod and RL direct enabled

Figure 59.     

Traffic forwarding behavior on RL during spine failure with Multi-pod and RL direct enabled

During a spine-connectivity failure, the Remote leaf continues to forward traffic using local SW DB and EPM DB; however, it uses an alternate Pod’s spine as a control plane. When the Remote leaf loses connectivity to the spines of the Pod it was originally associated with, it will try to build COOP and spine proxy connection to the alternative available spines. Using an alternate COOP connection, Release leaf downloads the SW DB and gets the latest EP information. An alternate spine proxy connection is used for BUM traffic forwarding, forwarding traffic toward silent hosts, ARP, etc.

Title: Remote leaf building COOP/Spine proxy

Figure 60.     

Remote leaf building COOP/Spine proxy

In a Multi-Pod setup, the user must configure the spines in more than one Pod as external BGP Route Reflectors (RRs). The Remote leaf builds an MP-BGP session with spines that are configured as external RRs. In a steady state, the Remote leaf builds a BGP relationship with the spines in multiple Pods. If the Remote leaf loses connectivity to its logically connected Pod’s spine, it will continue to have MP-BGP sessions with other Pod’s spines that are configured as external RRs. This allows the Remote leaf to receive the external prefixes from the spines of the alternate Pod in the event of failure of its logically connected Pod.

Title: Remote leaf building BGP connections with multiple spines across Pods

Figure 61.     

Remote leaf building BGP connections with multiple spines across Pods

Failures of all-spine reachability, with single Pod and RL direct enabled

When a Remote leaf loses reachability to its logically connected Pod’s spines either due to spine failure or due to reachability failure in the IPN, it will have the following traffic forwarding behavior:

1.     The Remote leaf continues to forward traffic between the existing and the new EPs connected directly to it.

2.     It can forward traffic to destinations that are connected to other Remote leaf pairs if those EPs are learned in the SW DB or HW EPM of the Remote leaf before the spine’s reachability failure. However, if the existing learned remote EPs time out from the SW DB, communication will fail. The SW DB timeout on the Remote leaf is 15 minutes.

3.     If there are new EPs that are learned on one Remote leaf pair, new EPs cannot be synced to other Remote leaf pairs when reachability to the spines is down. As explained in the section “Control plane for RL direct traffic forwarding,” the Remote leaf depends on Spine Proxy/COOP to learn new EPs connected to other Remote leaf pairs or ACI main DC.

4.     Because the L3Out prefixes of one Remote leaf pair are learnt on other Remote leaf pairs using spines that act as MP-BGP RRs, spine-reachability failure causes failure of L3Out to L3Out traffic between Remote Leaf pairs.

5.     EPs belonging to different VRFs are learned on the Remote leaf using the COOP connection; therefore, inter-VRF communication across RL pairs will break down when spine reachability is down.

Title: Traffic forwarding behavior on RL during spine failure with single Pod and RL direct enabled

Figure 62.     

Traffic forwarding behavior on RL during spine failure with single Pod and RL direct enabled

Failure of all-spine reachability, with Multi-Site

When the Remote leaf solution is deployed with Multi-Site, redundancy during spine-reachability failure will depend on whether the solution is deployed with single Pod or Multi-Pod. If Multi-Site is deployed with Multi-Pod, and the Remote leaf loses connectivity to the spines of its logically connected Pod, then the Remote leaf continues to forward traffic to all destinations within and across ACI sites. Details of Remote leaf redundancy behavior are the same as explained in the section “Failure of all-spine reachability, with Multi-Pod and RL direct enabled”.

Title: Traffic forwarding on Remote leaves during all-spine failure with Multi-Site and RL direct enabled

Figure 63.     

Traffic forwarding on Remote leaves during all-spine failure with Multi-Site and RL direct enabled

If Multi-Site is deployed with Single Pod, and the Remote leaf loses connectivity to spines of its logically connected Pod, then RL redundancy behavior will be the same as explained in section “Failure of all-spine reachability, with single Pod and RL direct enabled.”

Failure of all-spine reachability, with single Pod and without RL direct

Reachability between Remote leaves and spines can fail for multiple reasons, mentioned below; therefore, building redundancy for connectivity between Remote leaves and the spine is recommended. Reachability to the spine can fail either due to an IP network failure between Remote leaves and a spine or to a failure of all spines in the ACI main data center.

Title: No reachability between Remote leaf and spine

Figure 64.     

No reachability between Remote leaf and spine

When Remote leaf switches are configured with the vPC domain and end-host/L3Out is connected to the Remote leaf switches with vPC, the following behavior will be seen during failure:

     Traffic to known endpoints/prefixes connected to the Remote leaf will continue to work.

     Traffic destined to all EPs connected to local leaves and other remote leaves will be dropped, because this traffic gets forwarded through the spine without RL-direct.

     Traffic to the L3Out prefixes locally learned on the Remote leaves will continue to work fine.

     No configuration can be added to Remote leaves since connectivity to APIC is lost.

Complete failure of Local L3Out at Remote leaf (WAN Isolation Scenario)

Remote leaf switches receive BGP updates for the prefix learned through L3Out in the ACI main DC using an MP-BGP session with the spines. However, when the same prefix is learned over Local L3Out, the endpoints connected to the Remote leaf switches prefer Local L3Out over the L3Out connection in the ACI main DC if the Remote leaf is peering with peering OSPF/EIGRP or when it is receiving a prefix with the same BGP attributes from both local L3Out and the ACI main DC.

When the local L3Out connection fails, outbound traffic can instead use the L3Out connection in the ACI main DC, as shown in the figure below.

Title: Traffic forwarding when RL loses local

Figure 65.     

Traffic forwarding when RL loses local

L3Out interface failure when L3Out on Remote leaf is configured over vPC

Title: Control plane learning when L3Out is configured over SVI with vPC on RL

Figure 66.     

Control plane learning when L3Out is configured over SVI with vPC on RL

Like local switches, Remote leaf switches can also be configured with L3Out over SVI with vPC. In this configuration, each RL uses local next-hop reachable via SVI to reach the external prefix.

When the local interface toward the external router goes down, the SVI interface on RL does not go down, and next-hop (that is, the external router’s connected interface) becomes reachable via the upstream router.

Title: Traffic forwarding when L3Out interface fails on RL

Figure 67.     

Traffic forwarding when L3Out interface fails on RL

Title: Traffic forwarding when L3Out interface fails on RL along with connectivity to ACI main DC

Figure 68.     

Traffic forwarding when L3Out interface fails on RL along with connectivity to ACI main DC

When L3Out on Remote leaf is configured over SVI with vPC, it does not depend upon spine MP-BGP neighborship to provide reachability information for locally learned external prefixes. This ensures that even if IP Network connectivity is lost, external prefixes are reachable via local L3Out.

L3Out interface failure when L3Out on Remote leaf is configured over Routed interface

Title: Control-plane learning when L3Out is configured on RL over routed interface/sub-interface

Figure 69.     

Control-plane learning when L3Out is configured on RL over routed interface/sub-interface

Like local switches, Remote leaf switches can also be configured with L3Out over a routed interface or sub-interface. In this configuration, each RL will use local next-hop reachable to reach the external prefix.

When the local interface toward the external router goes down, a Remote leaf can use its peer RL to reach the external prefix. The Remote leaf receives the external prefix update via the MP-BGP session with the spine.

Title: Traffic forwarding when L3Out interface fails on RL

Figure 70.     

Traffic forwarding when L3Out interface fails on RL

Title: Traffic forwarding when L3Out interface fails on RL along with connectivity to ACI main DC

Figure 71.     

Traffic forwarding when L3Out interface fails on RL along with connectivity to ACI main DC

If both local leaf and IPN connectivity fail at the same time, then the Remote leaf cannot receive BGPs update from the spine for the external prefix. In this case, RL will not have reachability to the external prefix.

The problem shown in above the figure will be seen only in a single Pod scenario from Cisco ACI 4.2(2).

Starting from Cisco ACI 4.2(2), in a Multi-Pod scenario, with RL-direct enabled, Remote leaf builds BGP sessions with spines across different Pods. This will ensure that, even when all spines of the logically connected Pod go down, BGP sessions on RL will be up with alternate Pods.

Please note that if a Remote leaf loses connectivity to all Pods’ spines due to a problem in the IP Network, then the Remote leaf cannot receive BGP updates from any spine for the external prefix if both the local L3Out interface and IP Network fail together. In that case, the Remote leaf won’t have reachability to the external prefix.

Summary

Cisco Remote leaf solution allows the centralized management and consistent policy for Remote DCs without the investment of APIC controllers and Spines for these smaller size DCs. Networking, Security, Monitoring, Telemetry and troubleshooting Policies that are defined for Main DC can be used for Remote DCs as well that are connected to ACI main DC over IP Network.

For more information

For more information about the ACI architecture, refer to the documentation available at the following links: https://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/index.html.

https://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/white-paper-listing.html.

Appendix-A

ACI Multipod and Remote leaf integration before Cisco ACI 4.1(2)

Before Cisco ACI 4.1(2), when Remote leaf switches were used in the Multipod solution, there was a need to configure a separate sub-interface on the spine switches and a separate VRF in IPN. This was to make sure that unicast and multicast traffic took the same path and that next-hops of endpoints did not flap. The following section provides more details about Remote leaf with ACI Multipod before Cisco ACI 4.1(2).

Title: Remote leaf with ACI Multipod

Figure 72.     

Remote leaf with ACI Multipod

When Remote leaf switches are used in Multipod solution, there is a need to configure separate sub-interface on spine switches and a separate VRF in IPN. This is to make sure unicast and multicast traffic takes the same path and the next-hop of endpoint does not flap. The following section provides more details about Remote leaf with ACI Multipod.

Reasons to keep separate sub-interfaces on spines and a separate VRF on IPN

To understand the reason for this requirement, we need to analyze what would happen when using a normal configuration on the spines required for building a Multipod fabric (that is, leveraging a single L3Out in the infra tenant with only sub-interfaces tagging traffic on VLAN 4, and connecting the spines with the IPN devices). In this scenario, we are going to consider how those different traffic flows would be handled.

In the example below, a Remote leaf is associated to Pod1 (“pod1” in the figure below). The Multipod local leaf term is being used for a leaf in the remote Pod (pod2).

Unicast traffic within the BD from the Remote leaf to Multipod Local leaf

1.     Unicast traffic within BD from Multipod local leaf to Remote leaf.

2.     L2 multicast traffic from Multipod local leaf to Remote leaf.

Title: Unicast traffic within BD from Remote leaf to Multipod local leaf

Figure 73.     

Unicast traffic within BD from Remote leaf to Multipod local leaf

The figure above highlights the unicast traffic flow between EP1, which is connected to the Remote leaf switches, and endpoint EP2, which is part of the ACI Multipod fabric but connected in a Pod different from the one to which the RL nodes are logically associated (in this above example, the Remote leaf switches are part of Pod1). The following sequence of events happens to establish a unicast traffic flow between EP1 and EP2. The assumption here is that EP1 and EP2 are part of same BD and IP subnet.

1.     EP1 sends a unicast packet to EP2 with a source MAC address of EP1 and a destination MAC address of EP2.

2.     The packet is received by one of the Remote leaf nodes, which performs a Layer 2 lookup of EP2’s MAC address in its hardware table and finds that EP2 is reachable via the anycast VTEP address enabled on all the spines of its Pod (Pod1). The Remote leaf switch encapsulates this packet in the VXLAN header and forwards it toward the Multipod fabric with the source IP being the anycast IP address of the Remote leaves (RL-vPC-TEP), and destination IP the anycast IP address of the spines in Pod1 (RL-Ucast-TEP).

3.     One of the spines in Pod1 receives the packet. The spine performs a lookup in the COOP database and finds that EP2 is part of Pod2. As a consequence, it changes the destination IP address to the anycast IP address of the spines in Pod2.

4.     One of the spines in Pod2 receives this packet, performs the lookup in the COOP database and forwards it to the local leaf where EP2 is located. At the same time, the spine changes the destination IP to the TEP address of local leaf (Multipod LL TEP) of Pod2 where EP2 is located.

5.     The local leaf of Pod2 receives the packet and updates the hardware table with EP1 information with next-hop of RL-vPC-TEP, based on the source TEP of the packet, and forwards traffic to EP1.

Title: Unicast traffic within BD from Multipod Local leaf to Remote

Figure 74.     

Unicast traffic within BD from Multipod Local leaf to Remote

In the figure above is shown, instead, the return unicast traffic flow from EP2 to EP1, which includes the following sequence of events:

1.     EP2 sends a unicast packet to EP1, with the source MAC address of EP2 and the destination MAC address of EP1.

2.     The Multipod local leaf receives this packet and does a Layer 2 lookup of EP1 in its hardware table and finds the next-hop as the anycast IP address of the Remote leaf (RL-vPC-TEP), based on the data-plane learning previously described. The leaf switch encapsulates the packet in the VXLAN header with the source TEP as its own TEP (Multipod LL TEP) and the destination as RL-vPC-TEP. Since Remote leaf and Multipod are sharing the same IP Network in the same VRF, Multipod local leaf can directly send the packet to Remote leaf switches over the IP Network.

3.     One of the Remote leaf switches receives the packet and updates the EP2 information in its local hardware table with next-hop of Multipod LL TEP learned from the source TEP of the VXLAN encapsulated packet.

4.     The Remote leaf then forwards the packet to the attached host EP1.

Title: L2 multicast traffic from a Multipod local leaf to Remote leaf

Figure 75.     

L2 multicast traffic from a Multipod local leaf to Remote leaf

Let’s see now what happens when a multicast traffic flow is sent from EP2 to EP1 without configuring a separate sub-interface and a separate VRF in IPN.

1.     EP2 sends multicast traffic with the source MAC address as EP2 and the destination as L2 multicast MAC address.

2.     Multipod leaf receives this packet and checks for the bridge domain (BD) of EP2; it then forwards the packet to the multicast IP address of the BD.

3.     Local spines in Pod2 receive the packet and forward it to the spines of Pod1 with the BD multicast group as the destination. Since Remote leaves are logically associated to Pod1, Pod2 spines cannot do HREP of L2 multicast traffic directly to Remote leaves.

4.     The spine in Pod1 forwards the packet to Remote leaf after encapsulating the multicast packet in a unicast VXLAN packet with the source as the anycast IP address of the spines (RL-Ucast-TEP) and the destination as the anycast IP address of the Remote leaves (RL-vPC-TEP).

5.     The Remote leaf now learns the EP2 with next-hop of RL-Ucast-TEP. It already learned EP2 with Multipod-LL TEP during the unicast traffic flow.

As a consequence, without configuring separate sub-interfaces and separate VRFs in the IP Network, unicast and multicast traffic will take different paths. This causes the EP information to flap on Remote leaves.

The Remote leaf solution with Multipod creates separate paths for Remote leaves and ensures that unicast and multicast traffic take the same path.

Title: Solution for EP flaps with RL and Multipod

Figure 76.     

Solution for EP flaps with RL and Multipod

The following figure highlights the configuration that is needed for supporting the integration of Remote leaf nodes with an ACI Multipod fabric to ensure that unicast and multicast traffic flows take the same path, avoiding flapping of EP information. It is critical to highlight that this configuration is not required without Multipod.

Title: Configuration details for RL and Multipod

Figure 77.     

Configuration details for RL and Multipod

     As part of the regular ACI Multipod configuration, an L3Out in the infra tenant is required, leveraging VLAN-4 sub-interfaces on the spines to peer via OSPF with the IPN routers. For integrating Remote leaf nodes with a Multipod fabric, it is additionally required to create a separate L3Out in the infra tenant leveraging VLAN-5 sub-interfaces (with OSPF enabled as well).

     The corresponding VLAN-5 sub-interfaces on the upstream IPN routers are configured in a different routing domain (VRF2) than the one used by VLAN-4 sub-interfaces (VRF1) for east-west Multipod traffic. Notice that either of the two VRFs can be represented in the global routing table of the IPN device, if desired.

     The IPN must extend connectivity across Pods for this second VRF (VRF2) to which VLAN-5 sub-interfaces are connected. This can be achieved in multiple ways, such as MPLS VPN, VXLAN, or VRF-lite.

     The Remote leaf nodes remain configured with only VLAN-4 sub-interfaces used to peer OSPF with the IPN. VLAN-4 sub-interfaces on upstream routers are configured in the VRF (VRF1). As a consequence, reachability information for the RL TEP pool is propagated across the IPN only in the context of VRF1.

     In addition, the APIC controller automatically applies the configuration shown in the figure below and required to implement the following key functionalities:

Title: Automatic configuration done by APIC for RL and Multipod

Figure 78.     

Automatic configuration done by APIC for RL and Multipod

1.     Spines that are part of the Pod to which the RL nodes are logically associated start advertising the RL TEP pool on VLAN-5 sub-interfaces to all other Pods.

2.     Spines are configured with a route-map to reject learning of that Remote leaf TEP pool of all other Pods on VLAN-4 sub-interfaces. Spines accept only their associated RL’s TEP pools on VLAN-4 sub-interfaces.

As a result of those two automatic configuration steps, all the spines that are part of the remote Pods have reachability information toward the RL TEP Pool (10.1.1.0/24) only via the second set of sub-interfaces (leveraging VLAN tag 5) and in the context of the second VRF (VRF2) inside the IP Network.

Let’s take a look at how the unicast traffic from Multipod local leaf to Remote leaf changes after applying this configuration.

Title: Unicast traffic from Multipod local leaves to Remote leaves

Figure 79.     

Unicast traffic from Multipod local leaves to Remote leaves

1.     EP2 sends a unicast packet to EP1 with the source MAC address of EP2 and the destination MAC address of EP1.

2.     The Multipod leaf receives this packet and performs a Layer 2 lookup of EP1 in its hardware table and finds the next-hop as the anycast IP address of the Remote leaf (RL-vPC-TEP). The leaf switch encapsulates the packet in the VXLAN header with the source TEP as its own TEP (Multipod LL TEP) and the destination as RL-vPC-TEP.

3.     The packet is received on one of the spines of Pod2, which has reachability information for RL-vPC-TEP via VLAN-5 sub-interfaces that are part of VRF2 in the IPN. The spines in Pod2 do not have direct reachability to the Remote leaf in the context of VRF2; therefore, the packet must first be routed toward the spines in Pod1. The spines in Pod2 change the destination TEP to the anycast IP address of the spines in Pod1 for Multipod purposes.

4.     One of the spines in Pod1 receives the packet and changes the source TEP to the anycast IP address of the spine (RL-Ucast-TEP) and the destination to RL-vPC-TEP.

5.     One of the Remote leaf switches receives the packet and updates the EP2 information in its local hardware table with a next-hop of RL-Ucast-TEP, learned from the source TEP of the VXLAN- encapsulated packet. This behavior is now the same as the L2 multicast traffic previously described.

Now traffic path is same for both Unicast and Multicast traffic from Multipod Local Leaf to Remote leaf, hence there is no EP information flap on Remote leaf.

Learn more