Guidelines and Limitations for VXLAN BGP EVPN
VXLAN BGP EVPN has the following guidelines and limitations:
-
The following guidelines and limitations apply to VXLAN/VTEP:
-
SPAN source or destination is supported on any port.
For more information, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide, Release 7.x.
-
-
When SVI is enabled on a VTEP (flood and learn, or EVPN) regardless of ARP suppression, make sure that ARP-ETHER TCAM is carved using the hardware access-list tcam region arp-ether 256 double-wide command. This is not applicable to the Cisco Nexus 9200 and 9300-EX platform switches and Cisco Nexus 9500 platform switches with 9700-EX line cards.
-
Beginning with Cisco NX-OS Release 7.0(3)F3(3), VXLAN Layer 2 Gateway is supported only on the 9636C-RX line card. VXLAN and MPLS cannot be enabled on the Cisco Nexus 9508 switch at the same time.
-
Beginning with Cisco NX-OS Release 7.0(3)F3(3), if VXLAN is enabled, the Layer 2 Gateway cannot be enabled when there is any line card other than the 9636C-RX.
-
Beginning with Cisco NX-OS Release 7.0(3)I6(1), you can configure EVPN over segment routing or MPLS. See the Cisco Nexus 9000 Series NX-OS Label Switching Configuration Guide, Release 7.x for more information.
-
Beginning with Cisco NX-OS Release 7.0(3)I6(1), you can use MPLS tunnel encapsulation using the new CLI encapsulation mpls command. You can configure the label allocation mode for the EVPN address family. See the Cisco Nexus 9000 Series NX-OS Label Switching Configuration Guide, Release 7.x for more information.
-
In VXLAN EVPN setup that has 2K VNI scale configuration, the control plane down time takes more than 200 seconds. To avoid BGP flap, configure the graceful restart time to 300 seconds.
-
SVI and subinterfaces as uplinks are not supported.
-
In a VXLAN EVPN setup, border leaves must use unique route distinguishers, preferably using auto rd command. It is not supported to have same route distinguishers in different border leaves.
-
ARP suppression is only supported for a VNI if the VTEP hosts the First-Hop Gateway (Distributed Anycast Gateway) for this VNI. The VTEP and the SVI for this VLAN have to be properly configured for the distributed Anycast Gateway operation, for example, global Anycast Gateway MAC address configured and Anycast Gateway feature with the virtual IP address on the SVI.
-
The show commands with the internal keyword are not supported.
-
DHCP snooping (Dynamic Host Configuration Protocol snooping) is not supported on VXLAN VLANs.
-
RACLs are not supported on Layer 3 uplinks for VXLAN traffic. Egress VACLs support is not available for de-capsulated packets in the network to access direction on the inner payload.
As a best practice, use PACLs/VACLs for the access to the network direction.
See the Cisco Nexus 9000 Series NX-OS Security Configuration Guide for other guidelines and limitations for the VXLAN ACL feature.
-
QoS classification is not supported for VXLAN traffic in the network to access direction on the Layer 3 uplink interface.
See the Cisco Nexus 9000 Series NX-OS Quality of Service Configuration Guide for other guidelines and limitations for the VXLAN QoS feature.
-
The QoS buffer-boost feature is not applicable for VXLAN traffic.
-
VTEP does not support Layer 3 subinterface uplinks that carry VXLAN encapsulated traffic.
-
Layer 3 interface uplinks that carry VXLAN encapsulated traffic do not support subinterfaces for non-VXLAN encapsulated traffic.
-
On Cisco Nexus 9000 PX/TX/PQ switches configured as VXLAN VTEPs, if any ALE 40G port is used as a VXLAN underlay port, configuring subinterfaces on either this or any other 40G port is not allowed and could lead to VXLAN traffic loss.
-
For Cisco NX-OS 7.0(3)I2(1) and later, a FEX HIF (FEX host interface port) is supported for a VLAN that is extended with VXLAN.
-
For eBGP, it is recommended to use a single overlay eBGP EVPN session between loopbacks.
-
EBGP peering from a VXLAN host to local VTEP is supported with loopback in tenant VRF as BGP update-source.
-
You must bind NVE to a loopback address that is separate from other loopback addresses that are required by Layer 3 protocols. NVE and other Layer 3 protocols using the same loopback is not supported.
-
VXLAN BGP EVPN does not support an NVE interface in a non-default VRF.
-
It is recommended to configure a single BGP session over the loopback for an overlay BGP session.
-
When configuring VXLAN BGP EVPN, only the "System Routing Mode: Default" is applicable for the following hardware platforms:
-
Cisco Nexus 9200, 9300-EX, and 9300-FX/FX2 platform switches
-
Cisco Nexus 9300 platform switches
-
Cisco Nexus 9500 platform switches with X9500 line cards
-
Cisco Nexus 9500 platform switches with -EX and -FX line cards
-
-
The “System Routing Mode: template-vxlan-scale” is not applicable to Cisco NX-OS Release 7.0(3)I5(2) and later.
-
When using VXLAN BGP EVPN with Cisco NX-OS Release 7.0(3)I4(x) or 7.0(3)I5(1), the “System Routing Mode: template-vxlan-scale” is required on the following hardware platforms:
-
Cisco Nexus 9300-EX platform switches
-
Cisco Nexus 9500 platform switches with -EX line cards
-
-
Changing the “System Routing Mode” requires a reload of the switch.
-
For Cisco NX-OS Release 7.0(3)I2(1) and later, VXLAN is supported on Cisco Nexus 9500 platform switches with the following line cards:
-
9500-R
-
9564PX
-
9564TX
-
9536PQ
-
9700-EX
-
9700-FX
-
-
When Cisco Nexus 9500 platform switches are used as VTEPs (7.0(3)I2(1) and later), 100G line cards are not supported on Cisco Nexus 9500 platform switches. This limitation does not apply to a Cisco Nexus 9500 platform switch with -EX or -FX line cards.
-
Cisco Nexus 9300 platform switches with 100G uplinks only support VXLAN switching/bridging. (7.0(3)I2(1) and later)
Cisco Nexus 9200, 9300-EX, and 9300-FX platform switches do not have this restriction.
Note
For VXLAN routing support, a 40G uplink module is required.
-
The VXLAN UDP port number is used for VXLAN encapsulation. For Cisco Nexus NX-OS, the UDP port number is 4789. It complies with IETF standards and is not configurable.
-
For Cisco NX-OS Release 7.0(3)I1(2) and earlier, a static route with next-hop reachable over the VXLAN BGP EVPN route is not supported.
-
For Cisco Nexus 9200 platform switches that have the Application Spine Engine (ASE2), there exists a Layer 3 VXLAN (SVI) throughput issue. There is a data loss for packets of sizes 99–122. (7.0(3)I3(1) and later).
-
For the Cisco NX-OS 7.0(3)I2(3) release, the VXLAN network identifier (VNID) 16777215 is reserved and should not be configured explicitly.
-
For Cisco NX-OS Release 7.0(3)I4(1) and later, VXLAN supports In-Service Software Upgrade (ISSU).
-
VXLAN does not support co-existence with the GRE tunnel feature or the MPLS (static or segment-routing) feature on Cisco Nexus 9000 Series switches with a Network Forwarding Engine (NFE).
-
The vpc orphan-ports suspend command must be enabled for orphan ports that are connected to Cisco Nexus 9000 vPC VTEPs.
-
VTEP connected to FEX host interface ports is not supported (7.0(3)I2(1) and later).
-
In Cisco NX-OS Release 7.0(3)I4(1), resilient hashing (port-channel load-balancing resiliency) and VXLAN configurations are not compatible with VTEPs using ALE uplink ports.
Note
Resilient hashing is disabled by default.
-
For SVI-related triggers (such as shut/unshut or PIM enable/disable), a 30-second delay was added, allowing the Multicast FIB (MFIB) Distribution module (MFDM) to clear the hardware table before toggling between L2 and L3 modes or vice versa.
Note |
For information about VXLAN BGP EVPN scalability, see the Verified Scalability Guide for your platform. |
Considerations for VXLAN BGP EVPN Deployment
-
A loopback address is required when using the source-interface config command. The loopback address represents the local VTEP IP.
-
During boot-up of a switch (7.0(3)I2(2) and later), you can use the source-interface hold-down-time hold-down-time command to suppress advertisement of the NVE loopback address until the overlay has converged. The range for the hold-down-time is 0 - 1000 seconds. The default is 180 seconds.
-
To establish IP multicast routing in the core, IP multicast configuration, PIM configuration, and RP configuration is required.
-
VTEP to VTEP unicast reachability can be configured through any IGP/BGP protocol.
-
If the anycast gateway feature is enabled for a specific VNI, then the anyway gateway feature must be enabled on all VTEPs that have that VNI configured. Having the anycast gateway feature configured on only some of the VTEPs enabled for a specific VNI is not supported.
-
It is a requirement when changing the primary or secondary IP address of the NVE source interfaces to shut the NVE interface before changing the IP address.
-
As a best practice, the RP for the multicast group should be configured only on the spine layer. Use the anycast RP for RP load balancing and redundancy.
-
Every tenant VRF needs a VRF overlay VLAN and SVI for VXLAN routing.
-
For scale environments, the VLAN IDs related to the VRF and Layer-3 VNI (L3VNI) must be reserved with the system vlan nve-overlay id command.
This is required to optimize the VXLAN resource allocation to scale the following platforms:
-
Cisco Nexus 9200 platform switches beginning with Cisco NX-OS Release 7.0(3)I1(2) through 7.0(3)I5(2)
-
Cisco Nexus 9300-EX, 9300-FX, and 9300-FX2 platform switches beginning with Cisco NX-OS Release 7.0(3)I1(2) through 7.0(3)I5(2)
-
Cisco Nexus 9300 platform switches beginning with Cisco NX-OS Release 7.0(3)I1(2)
Note
Beginning with Cisco NX-OS Release 7.0(3)I5(2), the Cisco Nexus 9200, 9300-EX, and 9300-FX/FX2 platform switches do not require this command. Beginning with Cisco NX-OS Release 7.0(3)I5(2), it is strongly recommended to remove the command on Cisco Nexus 9200, 9300-EX, and 9300-FX/FX2 platform switches as it would disable Tenant Routed Multicast functionality on the VRF.
The following example shows how to reserve the VLAN IDs related to the VRF and the Layer-3 VNI:
system vlan nve-overlay id 2000 vlan 2000 vn-segment 50000 interface Vlan2000 vrf member MYVRF_50000 ip forward ipv6 forward vrf context MYVRF_50000 vni 50000Note
The system vlan nve-overlay id command should be used for a VRF or a Layer-3 VNI (L3VNI) only. Do not use this command for regular VLANs or Layer-2 VNIs (L2VNI).
-
-
When configuring ARP suppression with BGP-EVPN, use the hardware access-list tcam region arp-ether size double-wide command to accommodate ARP in this region. (You must decrease the size of an existing TCAM region before using this command.)
vPC Considerations for VXLAN BGP EVPN Deployment
-
The loopback address used by NVE needs to be configured to have a primary IP address and a secondary IP address.
The secondary IP address is used for all VxLAN traffic that includes multicast and unicast encapsulated traffic.
-
Each vPC peer needs to have separate BGP sessions to the spine.
-
vPC peers must have identical configurations.
-
Consistent VLAN to VN-segment mapping.
-
Consistent NVE1 binding to the same loopback interface
-
Using the same secondary IP address.
-
Using different primary IP addresses.
-
-
Consistent VNI to group mapping.
-
The VRF overlay VLAN should be a member of the peer-link port-channel.
-
-
For multicast, the vPC node that receives the (S, G) join from the RP (rendezvous point) becomes the DF (designated forwarder). On the DF node, encap routes are installed for multicast.
Decap routes are installed based on the election of a decapper from between the vPC primary node and the vPC secondary node. The winner of the decap election is the node with the least cost to the RP. However, if the cost to the RP is the same for both nodes, the vPC primary node is elected.
The winner of the decap election has the decap mroute installed. The other node does not have a decap route installed.
-
On a vPC device, BUM traffic (broadcast, unknown-unicast, and multicast traffic) from hosts is replicated on the peer-link. A copy is made of every native packet and each native packet is sent across the peer-link to service orphan-ports connected to the peer vPC switch.
To prevent traffic loops in VXLAN networks, native packets ingressing the peer-link cannot be sent to an uplink. However, if the peer switch is the encapper, the copied packet traverses the peer-link and is sent to the uplink.
Note
Each copied packet is sent on a special internal VLAN (VLAN 4041).
-
When peer-link is shut, the loopback interface used by NVE on the vPC secondary is brought down and the status is Admin Shut. This is done so that the route to the loopback is withdrawn on the upstream and that the upstream can divert all traffic to the vPC primary.
Note
Orphans connected to the vPC secondary will experience loss of traffic for the period that the peer-link is shut. This is similar to Layer 2 orphans in a vPC secondary of a traditional vPC setup.
-
When the vPC domain is shut, the loopback interface used by NVE on the VTEP with shutdown vPC domain is brought down and the status is Admin Shut. This is done so that the route to the loopback is withdrawn on the upstream and that the upstream can divert all traffic to the other vPC VTEP.
-
When peer-link is no-shut, the NVE loopback address is brought up again and the route is advertised upstream, attracting traffic.
-
For vPC, the loopback interface has 2 IP addresses: the primary IP address and the secondary IP address.
The primary IP address is unique and is used by Layer 3 protocols.
The secondary IP address on loopback is necessary because the interface NVE uses it for the VTEP IP address. The secondary IP address must be same on both vPC peers.
-
The vPC peer-gateway feature must be enabled on both peers.
As a best practice, use peer-switch, peer gateway, ip arp sync, ipv6 nd sync configurations for improved convergence in vPC topologies.
In addition, increase the STP hello timer to 4 seconds to avoid unnecessary TCN generations when vPC role changes occur.
The following is an example (best practice) of a vPC configuration:
switch# sh ru vpc version 6.1(2)I3(1) feature vpc vpc domain 2 peer-switch peer-keepalive destination 172.29.206.65 source 172.29.206.64 peer-gateway ipv6 nd synchronize ip arp synchronize -
On a vPC pair, shutting down NVE or NVE loopback on one of the vPC nodes is not a supported configuration. This means that traffic failover on one-side NVE shut or one-side loopback shut is not supported.
-
Redundant anycast RPs configured in the network for multicast load-balancing and RP redundancy are supported on vPC VTEP topologies.
-
Enabling vpc peer-gateway configuration is mandatory. For peer-gateway functionality, at least one backup routing SVI is required to be enabled across peer-link and also configured with PIM. This provides a backup routing path in the case when VTEP loses complete connectivity to the spine. Remote peer reachability is re-routed over the peer-link in this case.
The following is an example of SVI with PIM enabled:
swithch# sh ru int vlan 2 interface Vlan2 description special_svi_over_peer-link no shutdown ip address 30.2.1.1/30 ip pim sparse-modeNote
The SVI must be configured on both vPC peers and requires PIM to be enabled.
-
As a best practice when changing the secondary IP address of an anycast vPC VTEP, the NVE interfaces on both the vPC primary and the vPC secondary should be shut before the IP changes are made.
-
To provide redundancy and failover of VXLAN traffic when a VTEP loses all of its uplinks to the spine, it is recommended to run a Layer 3 link or an SVI link over the peer-link between vPC peers.
-
If DHCP Relay is required in VRF for DHCP clients or if loopback in VRF is required for reachability test on a vPC pair, it is necessary to create a backup SVI per VRF with PIM enabled.
swithch# sh ru int vlan 20 interface Vlan20 description backup routing svi for VRF Green vrf member GREEN no shutdown ip address 30.2.10.1/30
Network Considerations for VXLAN Deployments
-
MTU Size in the Transport Network
Due to the MAC-to-UDP encapsulation, VXLAN introduces 50-byte overhead to the original frames. Therefore, the maximum transmission unit (MTU) in the transport network must be increased by 50 bytes. If the overlays use a 1500-byte MTU, the transport network must be configured to accommodate 1550-byte packets at a minimum. Jumbo-frame support in the transport network is required if the overlay applications tend to use larger frame sizes than 1500 bytes.
-
ECMP and LACP Hashing Algorithms in the Transport Network
As described in a previous section, Cisco Nexus 9000 Series Switches introduce a level of entropy in the source UDP port for ECMP and LACP hashing in the transport network. As a way to augment this implementation, the transport network uses an ECMP or LACP hashing algorithm that takes the UDP source port as input for hashing, which achieves the best load-sharing results for VXLAN encapsulated traffic.
-
Multicast Group Scaling
The VXLAN implementation on Cisco Nexus 9000 Series Switches uses multicast tunnels for broadcast, unknown unicast, and multicast traffic forwarding. Ideally, one VXLAN segment mapping to one IP multicast group is the way to provide the optimal multicast forwarding. It is possible, however, to have multiple VXLAN segments share a single IP multicast group in the core network. VXLAN can support up to 16 million logical Layer 2 segments, using the 24-bit VNID field in the header. With one-to-one mapping between VXLAN segments and IP multicast groups, an increase in the number of VXLAN segments causes a parallel increase in the required multicast address space and the number of forwarding states on the core network devices. At some point, multicast scalability in the transport network can become a concern. In this case, mapping multiple VXLAN segments to a single multicast group can help conserve multicast control plane resources on the core devices and achieve the desired VXLAN scalability. However, this mapping comes at the cost of suboptimal multicast forwarding. Packets forwarded to the multicast group for one tenant are now sent to the VTEPs of other tenants that are sharing the same multicast group. This causes inefficient utilization of multicast data plane resources. Therefore, this solution is a trade-off between control plane scalability and data plane efficiency.
Despite the suboptimal multicast replication and forwarding, having multitenant VXLAN networks to share a multicast group does not bring any implications to the Layer 2 isolation between the tenant networks. After receiving an encapsulated packet from the multicast group, a VTEP checks and validates the VNID in the VXLAN header of the packet. The VTEP discards the packet if the VNID is unknown to it. Only when the VNID matches one of the VTEP’s local VXLAN VNIDs, does it forward the packet to that VXLAN segment. Other tenant networks will not receive the packet. Thus, the segregation between VXLAN segments is not compromised.
Considerations for the Transport Network
The following are considerations for the configuration of the transport network:
-
On the VTEP device:
-
Enable and configure IP multicast.*
-
Create and configure a loopback interface with a /32 IP address.
(For vPC VTEPs, you must configure primary and secondary /32 IP addresses.)
-
Enable UP multicast on the loopback interface. *
-
Advertise the loopback interface /32 addresses throught the routing protocol (static route) that runs in the transport network.
-
Enable IP multicast on the uplink outgoing physical interface. *
-
-
Throughout the transport network:
-
Enable and configure IP multicast.*
-
Note |
* Not required for static ingress replication or BGP EVPN ingress replication. |
Considerations for Tunneling VXLAN
DC Fabrics with VXLAN BGP EVPN are becoming the transport infrastructure for overlays. These overlays, often originated on the server (Host Overlay), require integration or transport over the top of the existing transport infrastructure (Network Overlay).
Nested VXLAN (Host Overlay over Network Overlay) support has been added starting with Cisco NX-OS Release 7.0(3)I7(4) and Cisco NX-OS Release 9.2(2) on the Cisco Nexus 9200, 9300-EX, 9300-FX, and 9300-FX2 platform switches.
To provide Nested VXLAN support, the switch hardware and software must differentiate between two different VXLAN profiles:
-
VXLAN originated behind the Hardware VTEP for transport over VXLAN BGP EVPN (nested VXLAN)
-
VXLAN originated behind the Hardware VTEP to integrated with VXLAN BGP EVPN (BUD Node)
The detection of the two different VXLAN profiles is automatic and no specific configuration is needed for nested VXLAN. As soon as VXLAN encapsulated traffic arrives in a VXLAN enabled VLAN, the traffic is transported over the VXLAN BGP EVPN enabled DC Fabric.
The following attachment modes are supported for Nested VXLAN:
-
Untagged traffic (in native VLAN on a trunk port or on an access port)
-
Tagged traffic (tagged VLAN on a IEEE 802.1Q trunk port)
-
Untagged and tagged traffic that is attached to a vPC domain
-
Untagged traffic on a Layer 3 interface of a Layer 3 port-channel interface
BGP EVPN Considerations for VXLAN Deployment
Commands for BGP EVPN
The following describes commands to support BGP EVPN VXLAN control planes.
|
Command |
Description |
||||
|---|---|---|---|---|---|
|
member vni range [associate-vrf] |
Associate VXLAN VNIs (Virtual Network Identifiers) with the NVE interface The attribute associate- vrf is used to identify and separate processing VNIs that are associated with a VRF and used for routing.
|
||||
|
show nve vni show nve vni summary |
Displays information that determine if the VNI is configured for peer and host learning via the control plane or data plane. |
||||
|
show bgp l2vpn evpn show bgp l2vpn evpn summary |
Displays the Layer 2 VPN EVPN address family. |
||||
|
host-reachability protocol bgp |
Specifies BGP as the mechanism for host reachability advertisement. |
||||
|
suppress-arp |
Suppresses ARP under Layer 2 VNI. |
||||
|
fabric forwarding anycast-gateway-mac |
Configures anycast gateway MAC of the switch. |
||||
|
vrf context |
Creates the VRF and enter the VRF mode. |
||||
|
nv overlay evpn |
Enables/Disables the Ethernet VPN (EVPN). |
||||
|
router bgp |
Configures the Border Gateway Protocol (BGP). |
||||
|
system vlan nve-overlay id range |
For scale environments, the VLAN IDs related to the VRF and Layer-3 VNI (L3VNI) must be reserved with the system vlan nve-overlay id command. This is required to optimize the VXLAN resource allocation to scale the following platforms:
|
||||
|
suppress mac-route |
Suppresses the BGP MAC route so that BGP only sends the MAC/IP route for a host. Under NVE, the MAC updates for all VNIs are suppressed.
|
Feedback