VXLAN EVPN Multi-Site has the following configuration guidelines and limitations:
-
The following switches support VXLAN EVPN Multi-Site:
-
Cisco Nexus 9300-EX and 9300-FX platform switches (except Cisco Nexus 9348GC-FXP platform switches)
-
Cisco Nexus 9300-FX2 platform switches
-
Cisco Nexus 9300-FX3 platform switches
-
Cisco Nexus 9300-GX platform switches
-
Cisco Nexus 9300-GX2 platform switches
-
Cisco Nexus 9332D-H2R switches
-
Cisco Nexus 93400LD-H1 switches
-
Cisco Nexus 9364C-H1 switches
-
Cisco Nexus 9800 platform switches with X9836DM-A and X98900CD-A line cards
-
Cisco Nexus 9500 platform switches with -EX or -FX or -GX line cards
Note
|
Cisco Nexus 9500 platform switches with -R/RX line cards don't support VXLAN EVPN Multi-Site.
|
-
The evpn multisite dci-tracking is mandatory for anycast BGWs and vPC BGW DCI links.
The evpn multisite fabric-tracking is mandatory only for anycast BGWs. For vPC based BGWs, this command is not mandatory. The NVE Interface will be brought
up with just the dci tracked link in the up state.
-
Cisco Nexus 9332C and 9364C platform switches can be BGWs.
-
In a VXLAN EVPN Multi-Site deployment, when you use the ttag feature, make sure that the ttag is stripped (ttag-strip ) on BGW's DCI interfaces that connect to the cloud. To elaborate, if the ttag is attached to non-Nexus 9000 devices that
do not support EtherType 0x8905, stripping of the ttag is required. However, BGW back-to-back model of DCI does not require
ttag stripping.
-
VXLAN EVPN Multi-Site and Tenant Routed Multicast (TRM) are supported between sources and receivers deployed across different
sites.
-
The Multi-Site BGW allows the coexistence of Multi-Site extensions (Layer 2 unicast/multicast and Layer 3 unicast) as well
as Layer 3 unicast and multicast external connectivity.
-
In TRM with multi-site deployments, all BGWs receive traffic from fabric. However, only the designated forwarder (DF) BGW
forwards the traffic. All other BGWs drop the traffic through a default drop ACL. This ACL is programmed in all DCI tracking
ports. Don't remove the evpn multisite dci-tracking configuration from the DCI uplink ports. If you do, you remove the ACL, which creates a nondeterministic traffic flow in
which packets can be dropped or duplicated instead of deterministically forwarded by only one BGW, the DF.
-
Anycast mode can support up to six BGWs per site.
-
BGWs in a vPC topology are supported.
-
Multicast Flood Domain between inter-site/fabric BGWs isn't supported.
-
iBGP EVPN Peering between BGWs of different fabrics/sites isn't supported.
-
The
peer-type fabric-external
command configuration is required only for VXLAN Multi-Site BGWs (this command must not be used when peering with non-Cisco
equipment).
Note
|
The
peer-type fabric-external
command configuration is not required for pseudo BGWs.
|
-
Anycast mode can support only Layer 3 services that are attached to local interfaces.
-
In Anycast mode, BUM is replicated to each border leaf. DF election between the border leafs for a particular site determines
which border leaf forwards the inter-site traffic (fabric to DCI and conversely) for that site.
-
In Anycast mode, all Layer 3 services are advertised in BGP via EVPN Type-5 routes with their physical IP as the next hop.
-
vPC mode can support only two BGWs.
-
vPC mode can support both Layer 2 hosts and Layer 3 services on local interfaces.
-
In vPC mode, BUM is replicated to either of the BGWs for traffic coming from the external site. Hence, both BGWs are forwarders
for site external to site internal (DCI to fabric) direction.
-
In vPC mode, BUM is replicated to either of the BGWs for traffic coming from the local site leaf for a VLAN using Ingress
Replication (IR) underlay. Both BGWs are forwarders for site internal to site external (fabric to DCI) direction for VLANs
using the IR underlay.
-
In vPC mode, BUM is replicated to both BGWs for traffic coming from the local site leaf for a VLAN using the multicast underlay.
Therefore, a decapper/forwarder election happens, and the decapsulation winner/forwarder only forwards the site-local traffic
to external site BGWs for VLANs using the multicast underlay.
-
Prior to NX-OS 10.2(2)F only ingress replication was supported between DCI peers across the core. Beginning with Cisco NX-OS
Release 10.2(2)F both ingress replication and multicast are supported between DCI peers across the core.
-
In vPC mode, all Layer 3 services/attachments are advertised in BGP via EVPN Type-5 routes with their virtual IP as next hop.
If the VIP/PIP feature is configured, they are advertised with PIP as the next hop.
-
If different Anycast Gateway MAC addresses are configured across sites, enable ARP suppression for all VLANs that have been
extended.
-
Bind NVE to a loopback address that is separate from loopback addresses that are required by Layer 3 protocols. A best practice
is to use a dedicated loopback address for the NVE source interface (PIP VTEP) and multi-site source interface (anycast and
virtual IP VTEP).
-
PIM BiDir is not supported for fabric underlay multicast replication with VXLAN Multi-Site.
-
PIM is not supported on Multi-Site VXLAN DCI links.
-
FEX is not supported on a vPC BGW and Anycast BGW.
-
Beginning with Cisco NX-OS Release 9.3(5), VTEPs support VXLAN-encapsulated traffic over parent interfaces if subinterfaces
are configured. This feature is supported for VXLAN EVPN Multi-Site and DCI. DCI tracking can be enabled only on the parent
interface.
-
Beginning with Cisco NX-OS Release 9.3(5), VXLAN EVPN Multi-Site supports asymmetric VNIs. For more information, see Multi-Site
with Asymmetric VNIs and Configuration Example for Multi-Site with Asymmetric VNIs.
-
The following guidelines and limitations apply to dual RD support for Multi-Site:
-
Dual RD are supported beginning with Cisco NX-OS Release 9.3(5).
-
Dual RD is enabled automatically for Cisco Nexus 9332C, 9364C, 9300-EX, and 9300-FX/FX2 platform switches and Cisco Nexus
9500 platform switches with -EX/FX line cards that have VXLAN EVPN Multi-Site enabled.
-
To use CloudSec or other features that require PIP advertisement for multi-site reoriginated routes, configure BGP additional
paths on the route server if dual RD are enabled on the BGW, or disable dual RD.
-
Sending secondary RD additional paths at the BGW node isn't supported.
-
During an ISSU, the number of paths for the leaf nodes might double temporarily while all BGWs are being upgraded.
-
Beginning with Cisco NX-OS Release 9.3(5), if you disable the host-reachability protocol bgp command under the NVE interface in a VXLAN EVPN Multi-Site topology, the NVE interface stays operationally down.
-
Beginning with Cisco NX-OS Release 9.3(5), Multi-Site Border Gateways re-originate incoming remote routes when advertising
to the site's local spine/leaf switches. These re-originated routes modify the following fields:
-
RD value changes to [Multisite Site ID:L3 VNID].
-
It is mandatory that Route-Targets are defined on all VTEP that are participating in a given VRF, this includes and is explicitly
required for the BGW to extend the given VRF. Prior to Cisco NX-OS Release 9.3(5), Route-Targets from intra-site VTEPs were
inadvertently kept across the site boundary, even if not defined on the BGW. Starting from Cisco NX-OS Release 9.3(5) the
mandatory behavior is enforced. By adding the necessary Route-Targets to the BGW, the change from inadvertent Route-Target
advertisement to explicit Route-Target advertisement can be performed.
-
Path type changes from external to local.
-
Beginning with Cisco NX-OS Release 10.2(3)F, the VXLAN EVPN Multi-Site is supported on the Cisco Nexus 9300-GX2 platform switches.
-
Beginning with Cisco NX-OS Release 10.4(1)F, the VXLAN EVPN Multi-Site is supported on the Cisco Nexus 9332D-H2R switches.
-
Beginning with Cisco NX-OS Release 10.4(2)F, the VXLAN EVPN Multi-Site is supported on the Cisco Nexus 93400LD-H1 switches.
-
Beginning with Cisco NX-OS Release 10.4(3)F, the VXLAN EVPN Multi-Site is supported on the Cisco Nexus 9364C-H1 switches.
-
Beginning with Cisco NX-OS Release 10.2(3)F, the dual RD support for Multi-Site is supported on the Cisco Nexus 9300-FX3 platform
switches.
-
Beginning with Cisco NX-OS Release 10.4(3)F, the VXLAN Multi-Site Anycast BGW is also supported on the Cisco Nexus 9808/9804
switches with X9836DM-A and X98900CD-A line cards.
-
To improve the convergence in case of fabric link failure and avoid issues in case of fabric link flapping, ensure to configure
multi-hop BFD between loopbacks of spines and BGWs.
In the specific scenario where a BGW node becomes completely isolated from the fabric due to all its fabric links failing,
the use of multi-hop BFD ensures that the BGP sessions between the spines and the isolated BGW can be immediately brought
down, without relying on the configured BGP hold-time value.
-
In a VXLAN Multi-Site environment, a border gateway device that uses ECMP for routing through both a VXLAN overlay and an
L3 prefix to access remote site subnets might encounter adjacency resolution failure for one of these routes. If the switch
attempts to use this unresolved prefix, it will result in traffic being dropped.