Nexus 9000: Configure and Verify VXLAN Xconnect
Available Languages
Contents
Introduction
The document describes a quick reference on how to configure and verify VXLAN Xconnect on Nexus 9000 Switches.
Prerequisites
Requirements
Cisco recommends that you have knowledge of VXLAN EVPN.
Components Used
The information in this document is based on these software and hardware versions:
- N9K-C93180YC-EX
- NXOS 9.2(1)
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Overview
VXLAN Xconnect is a mechanism for a point-to-point tunnel for data and control packets from one Leaf to another. Inner Dot1q Tags are preserved and VXLAN encapsulated within the outer VNID which is specified as the Xconnect VNID. Layer 2 Control Frames such as Link Layer Discovery Protocol (LLDP), Cisco Discovery Protocol (CDP), Spanning Tree Protocol (STP) are VXLAN Encapsulated and sent over to other ends of the Tunnel.
Topology
VTEP1, VTEP2, VTEP3, and VTEP4 are two vPC VTEP pairs configured in such a way that the inner dot1q tags from downstream switches are preserved and when VXLAN encapsulated, use outer VLAN ID's VXLAN VNID in order to send over to the remote VTEP. All VTEPs are N9K-C93180YC-EX.
Downstream switches are Nexus 3ks which are configured with Switch Virtual Interface (SVIs) in respective VLANs to mimic the hosts.
Configure
1. Outer VLAN used in this Xconnect topology is 3000. This would be the one with the VNID and Xconnect configuration.
VTEP1# sh run vlan 3000 vlan 3000 vn-segment 1003000 xconnect
2. Feature NGOAM has to be enabled and needs this configuration.
VTEP1# sh run ngoam feature ngoam ngoam install acl ngoam xconnect hb-interval 5000
3. Dot1q tunnel configuration towards the downstream switch.
VTEP1# sh run int po10 interface port-channel10 switchport switchport mode dot1q-tunnel switchport access vlan 3000 speed 40000 no negotiate auto vpc 10
The vPC configurations are required only when VTEPs are deployed as vPC. Otherwise, skip the vPC configurations mentioned in this document. VXLAN Xconnect also is configurable on a standalone VTEP.
4. Multicast group has to be defined under the NVE interface to take care of the forwarding. Note to enable ip pim sparse-mode on relevant uplinks and define PIM RP as well so that multicast routing and PIM messages are exchanged appropriately. Typically PIM RP is defined at the Spine Layer.
VTEP1# sh run int nve1 no shutdown host-reachability protocol bgp source-interface loopback1 member vni 1003000 mcast-group 239.30.30.30
5. Infra VLAN needs to be specified and allowed as the native VLAN within the peer link. This step is needed for vPC VTEPs.
VTEP1# sh run span|infra no spanning-tree vlan 3000 system nve infra-vlans 999 VTEP1# sh run int po1 interface port-channel1 switchport switchport mode trunk switchport trunk native vlan 999 spanning-tree port type network vpc peer-link
6. BGP/EVPN Configuration: L2VPN EVPN neighborships are needed between leaf/spine to exchange the Type 3 routes that are required to establish the VXLAN Xconnect.
- Here, the IP addresses- 192.168.100.1 and 192.168.100.2 are the Spines in the topology. Typically the L2VPN EVPN neighborships are formed to the Spines. Spines configure all Leaf switches as Route reflector clients in an iBGP Scenario.
- It is recommended to use separate Loopbacks for BGP/OSPF and NVE purposes.
feature bgp
router bgp 65000
router-id 192.168.100.3
neighbor 192.168.100.1
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 192.168.100.2
remote-as 65000
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
evpn
vni 1003000 l2
rd auto
route-target import auto
route-target export auto
Verify
Use this section to confirm that your configuration works properly.
1. BGP neighborship.
VTEP1# sh bgp l2vpn evpn sum BGP summary information for VRF default, address family L2VPN EVPN BGP router identifier 192.168.100.3, local AS number 65000 BGP table version is 14, L2VPN EVPN config peers 2, capable peers 1 4 network entries and 5 paths using 756 bytes of memory BGP attribute entries [3/492], BGP AS path entries [0/0] BGP community entries [0/0], BGP clusterlist entries [2/8] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 192.168.100.1 4 65000 92 90 14 0 0 01:21:41 2
2. Receive Type 3 prefixes.
VTEP1# sh bgp l2vpn evpn
BGP routing table information for VRF default, address family L2VPN EVPN
BGP table version is 14, Local Router ID is 192.168.100.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 192.168.100.3:35767 (L2VNI 1003000)
*>l[3]:[0]:[32]:[172.16.1.100]/88
172.16.1.100 100 32768 i
* i[3]:[0]:[32]:[172.17.1.100]/88 <<< bgp type 3
172.17.1.100 100 0 i
*>i 172.17.1.100 100 0 i
Route Distinguisher: 192.168.100.5:35767
*>i[3]:[0]:[32]:[172.17.1.100]/88
172.17.1.100 100 0 i
Route Distinguisher: 192.168.100.6:35767
*>i[3]:[0]:[32]:[172.17.1.100]/88
172.17.1.100 100 0 i
3. NVE Peering.
VTEP1# sh nve peer
Interface Peer-IP State LearnType Uptime Router-Mac
--------- --------------- ----- --------- -------- -----------------
nve1 172.17.1.100 Up CP 00:58:06 n/a
VTEP1# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
SU - Suppress Unknown Unicast
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 1003000 239.30.30.30 Up CP L2 [3000] Xconn <<<
4. NGOAM checks.
VTEP1# show ngoam xconnect sess all
States: LD = Local interface down, RD = Remote interface Down
HB = Heartbeat lost, DB = Database/Routes not present
* - Showing Vpc-peer interface info
Vlan Peer-ip/vni XC-State Local-if/State Rmt-if/State
===============================================================================
3000 172.17.1.100 / 1003000 Active Po10 / UP Po10 / UP
VTEP1# show ngoam xconnect sess 3000
Vlan ID: 3000
Peer IP: 172.17.1.100 VNI : 1003000
State: Active <<< State should be active
Last state update: 12/10/2018 17:13:45.337
Local interface: Po10 State: UP
Local vpc interface Po10 State: UP
Remote interface: Po10 State: UP
Remote vpc interface: Po10 State: UP
Once the NGOAM session is up, the N3k's would see each other in CDP. STP BPDUs are also tunnelled so the switches agree upon the root bridge placement too.
5. Verifications at the downstream switches.
SW1(config)# sh span vl 10
VLAN0010
Spanning tree enabled protocol rstp
Root ID Priority 32778
Address 7079.b348.6cb7
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
Address 7079.b348.6cb7
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po10 Desg FWD 1 128.4105 P2p
SW2(config)# sh span vl 10
VLAN0010
Spanning tree enabled protocol rstp
Root ID Priority 32778
Address 7079.b348.6cb7
Cost 1
Port 4105 (port-channel10)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
Address 707d.b964.9441
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po10 Root FWD 1 128.4105 P2p
SW1(config)# show ip int b IP Interface Status for VRF "default"(1) Interface IP Address Interface Status Vlan10 192.168.10.100 protocol-up/link-up/admin-up Vlan20 192.168.20.100 protocol-up/link-up/admin-up Vlan30 192.168.30.100 protocol-up/link-up/admin-up SW2(config)# show ip int b IP Interface Status for VRF "default"(1) Interface IP Address Interface Status Vlan10 192.168.10.200 protocol-up/link-up/admin-up Vlan20 192.168.20.200 protocol-up/link-up/admin-up Vlan30 192.168.30.200 protocol-up/link-up/admin-up SW1(config)# ping 192.168.10.200 PING 192.168.10.200 (192.168.10.200): 56 data bytes 64 bytes from 192.168.10.200: icmp_seq=0 ttl=254 time=0.826 ms 64 bytes from 192.168.10.200: icmp_seq=1 ttl=254 time=0.531 ms 64 bytes from 192.168.10.200: icmp_seq=2 ttl=254 time=0.54 ms 64 bytes from 192.168.10.200: icmp_seq=3 ttl=254 time=0.522 ms 64 bytes from 192.168.10.200: icmp_seq=4 ttl=254 time=0.571 ms
Troubleshoot
There is currently no specific troubleshooting information available for this configuration.
Caveats
1. The dot1q tunnel interfaces will be stuck in error disabled state in an Xconnect VXLAN setup if the configurations within vPC switches are not consistent. Below are some of the cases wherein the interface will be in error disabled;
- If the VLAN to VN-segment is not defined on both the vPC switches.
- If the NVE to multicast group is not defined on both the vPC switches.
- If the NGOAM Heartbeats are not received (use ethanalyzer with filter=cfm to catch the NGOAM heartbeat packets).
- Even if the dot1q tunnel interface is orphan connected in a vPC setup, it is still required to configure the multicast group under the NVE Interface for the VN-segment which is part of Xconnect on both the switches.
- NGOAM heartbeats are processed/sent by the vPC Primary switch. Heartbeat messages that land on vPC secondary will be synced to the primary
2. When Xconnect is configured in a VLAN, the traffic from one site to another is encapsulated with the outer destination address=multicast address defined under the NVE interface for that particular vn segment. It is recommended to use a unique multicast group for the Xconnect VLANs. Multicast in the core/Spine must be functional.
3. Multicast traffic might hit both the vPC boxes on the remote side of Xconnect; However, the Decap winner (the box which can decapsulate the BUM) will be only one switch in a vPC pair. This can be verified using the command- show forwarding multicast route group <Group address> source <SRC IP>. If the Flag shown here is a lower case v, it means the box is decap loser and if it's an Upper case V, the box is the decap winner and so can decapsulate the multicast traffic and forward it further down.
4. On 93180YC based platforms, when the Host is orphan connected to 9k1 and if there is no OIL for S, G on 9k1, a copy of the multicast packet is sent to the vPC Peer using a special encapsulation of Source IP-> 127.0.0.1 and Destination IP-> shared NVE IP and if the 9k2 has OIL for S, G entry, then the traffic forwarding will be taken care by the 9k2 to the remote sites.
Packet Capture
Here is a screenshot of a packet capture that was taken at the spine switch:
- Inner dot1q header=10 is preserved
- VNI used is 1003000 (which is the outer VLAN's VNID)
- The destination IP address would be the multicast group that is defined under the nve interface
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
3.0 |
18-Nov-2021 |
Added the default Troubleshoot Section. |
2.0 |
17-Nov-2021 |
Minor edits. |
1.0 |
13-Dec-2018 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)