Remote leaf switch restrictions and limitations identify the guidelines, supported features, and configuration limitations that apply to remote leaf switches in the fabric. These constraints help ensure connectivity and traffic forwarding between the main data center and remote locations.
Remote leaf switches extend the fabric to remote locations. Remote leaf switch deployments must meet specific underlay and configuration requirements to maintain operational stability.
-
Advertisement of /32 tunnel endpoint (TEP) IP addresses without route summarization.
-
Deletion and recreation of the virtual port channel (vPC) when moving switches between sites that use the same node ID.
-
Use of specific ports for initial discovery on N9K-C9348GC-FXP and N9K-C9348GC-FX3 switches.
Operational guidelines and feature support
The following guidelines apply to remote leaf switch deployments:
-
The remote leaf solution requires the /32 TEP IP addresses of the remote leaf switches and the main data center leaf and spine switches to be advertised without route summarization.
-
If you move a remote leaf switch to another site within the same pod and the new site uses the same node ID as the original site, delete and recreate the vPC.
-
For Cisco N9K-C9348GC-FXP and N9K-C9348GC-FX3 switches, use only port 1/53 or 1/54 for initial remote leaf switch discovery. After discovery, you can use the other ports as fabric uplinks to the ISN or IPN.
Beginning with Cisco APIC Release 6.0(3), dynamic packet prioritization behavior varies based on CoS preservation and Cisco ACI Multi-Pod policy settings:
-
Mice flows egress the fabric with a VLAN CoS priority of 0 if you enable CoS preservation with dynamic packet prioritization on the physical leaf and remote leaf switches.
-
Mice flows egress the fabric with a VLAN CoS priority of 0 if you enable Cisco ACI Multi-Pod DSCP translation with dynamic packet prioritization on a physical leaf switch.
-
Mice flows egress the fabric with a VLAN CoS priority of 3 if you enable Cisco ACI Multi-Pod DSCP translation with dynamic packet prioritization on a remote leaf switch.
If you do not want mice flows to egress a remote leaf switch with a VLAN CoS priority of 3, use CoS preservation instead of Cisco ACI Multi-Pod DSCP translation.
The following sections describe the features and configurations that are supported and unsupported with remote leaf switches:
Supported features
Stretching of an L3Out SVI within a vPC remote leaf switch pair is supported.
Beginning with Cisco APIC release 4.2(4), the 802.1Q (Dot1q) tunnels feature is supported.
Beginning with Cisco APIC release 4.1(2), the following features are supported:
-
Remote leaf switches with ACI Multi-Site
-
Traffic forwarding directly across two remote leaf vPC pairs in the same remote data center or across data centers, when those remote leaf pairs are associated to the same pod or to pods that are part of the same multipod fabric
-
Transit L3Out across remote locations, which is when the main Cisco ACI data center pod is a transit between two remote locations (the L3Out in RL location-1 and L3Out in RL location-2 are advertising prefixes for each other)
Beginning with Cisco APIC release 4.0(1), the following features are supported:
-
Q-in-Q Encapsulation Mapping for EPGs
-
PBR Tracking on remote leaf switches (with system-level global GIPo enabled)
-
PBR Resilient Hashing
-
Netflow
-
MacSec Encryption
-
Troubleshooting Wizard
-
Atomic counters
Unsupported features
Full fabric and tenant policies are supported on remote leaf switches in this release with the exception of the following features, which are unsupported:
-
GOLF
-
vPod
-
Floating L3Out
-
Stretching of L3Out SVI between local leaf switches (ACI main data center switches) and remote leaf switches.
-
Copy service is not supported when deployed on local leaf switches and when the source or destination is on the remote leaf switch. In this situation, the routable TEP IP address is not allocated for the local leaf switch. For more information, see the section "Copy Services Limitations" in the "Configuring Copy Services" chapter in the Cisco APIC Layer 4 to Layer 7 Services Deployment Guide, available in the APIC documentation page.
-
Layer 2 Outside Connections (except Static EPGs)
-
Copy services with vzAny contract
-
FCoE connections on remote leaf switches
-
Flood in encapsulation for bridge domains or EPGs
-
Fast Link Failover policies are for ACI fabric links between leaf and spine switches, and are not applicable to remote leaf connections. Alternative methods are introduced in Cisco APIC Release 5.2(1) to achieve faster convergence for remote leaf connections.
-
Managed Service Graph-attached devices at remote locations
-
Traffic Storm Control
-
Cloud Sec Encryption
-
First Hop Security
-
Layer 3 Multicast routing on remote leaf switches
-
Maintenance mode
-
TEP to TEP atomic counters
The following scenarios are not supported when integrating remote leaf switches in a Multi-Site architecture in conjunction with the intersite L3Out functionality:
-
Transit routing between L3Outs deployed on remote leaf switch pairs associated to separate sites
-
Endpoints connected to a remote leaf switch pair associated to a site communicating with the L3Out deployed on the remote leaf switch pair associated to a remote site
-
Endpoints connected to the local site communicating with the L3Out deployed on the remote leaf switch pair associated to a remote site
-
Endpoints connected to a remote leaf switch pair associated to a site communicating with the L3Out deployed on a remote site
The limitations above do not apply if the different data center sites are deployed as pods as part of the same Multi-Pod fabric.
The following deployments and configurations are not supported with the remote leaf switch feature:
-
It is not supported to stretch a bridge domain between remote leaf nodes associated to a given site (APIC domain) and leaf nodes part of a separate site of a Multi-Site deployment (in both scenarios where those leaf nodes are local or remote) and a fault is generated on APIC to highlight this restriction. This applies independently from the fact that BUM flooding is enabled or disabled when configuring the stretched bridge domain on the Multi-Site Orchestrator (MSO). However, a bridge domain can always be stretched (with BUM flooding enabled or disabled) between remote leaf nodes and local leaf nodes belonging to the same site (APIC domain).
-
Spanning Tree Protocol across remote leaf switch location and main data center.
-
APICs directly connected to remote leaf switches.
-
Orphan port channel or physical ports on remote leaf switches, with a vPC domain (this restriction applies for releases 3.1 and earlier).
-
With and without service node integration, local traffic forwarding within a remote location is only supported if the consumer, provider, and services nodes are all connected to remote leaf switches are in vPC mode.
-
/32 loopbacks advertised from the spine switch to the IPN must not be suppressed/aggregated toward the remote leaf switch. The /32 loopbacks must be advertised to the remote leaf switch.
-
Service devices part of the same service graph must not be distributed across remote leaf location and main data center. The consumers and the provider endpoints can be spread across remote location and main data center
Changes for release 5.0(1)
Beginning with Cisco APIC release 5.0(1), the following changes have been applied for remote leaf switches:
-
The direct traffic forwarding feature is enabled by default and cannot be disabled.
-
A configuration without direct traffic forwarding for remote leaf switches is no longer supported. If you have remote leaf switches and you are upgrading to Cisco APIC Release 5.0(1), review the information provided in the section "About Direct Traffic Forwarding" and enable direct traffic forwarding using the instructions in that section.
Changes for release 5.2(3)
Beginning with Cisco APIC release 5.2(3), the following changes have been applied for remote leaf switches:
-
The IPN underlay protocol to peer between the remote leaf switches and the upstream router can be either OSPF or BGP. In previous releases, only an OSPF underlay is supported.