Guest

Cisco Nexus 1000V Switch for VMware vSphere

Nexus 1000v VXLAN Configuration Example

Document ID: 117618

Updated: Apr 14, 2014

Contributed by Joseph Ristaino, Cisco TAC Engineer.

   Print

Introduction

This document describes how to configure Virtual Extensible LAN (VXLAN) on a Cisco Nexus 1000V (N1kV) Series switch.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information

This section describes the VXLAN feature and how is can be implemented in order to address scalability limitations within data centers.

About VXLAN

Since the VLAN ID field in a frame is only 12 bits, it limits the number of VLANS to 4,096 with some reserved. The VXLAN feature introduces a 24-bit ID field, which scales the possibilities to include a potential of 16 million different LAN segments. This is similar to the transition from IPv4 to IPv6.

For comparison, this is the standard frame layout:



This is the VXLAN encapsulated frame layout:



With VXLAN transport, the initial Layer 2 (L2) frame is encapsulated in a Layer 3 (L3) packet. The destination then de-encapsulates the packet and sends the frame based on the destination Media Access Control (MAC) address that is contained within. This allows segmented traffic on a current infrastructure and allows networks to mirror L2 domains across data centers. Also, it enables enhancements such as cross data center vMotions.

Use of the VXLAN technology is practical in environments that require you to scale and provide infrastructures for customers. A good example of this is the VMware vCloud Director, where providers deploy resources for their customers. This includes computer resources from servers, networking resources that utilize VMware networking, or Cisco networking powered by the N1kV. A provider uses VLANs as the transport mechanism for the tenant VXLANs. Without VXLAN, tenants are given their own VLAN, which can scale up to the 4,096 limit quickly. Each tenant can now be assigned a VXLAN and can utilize the infrastructure VLANs for transport. This is scalable and still segmented.


VXLAN Requirements

In order for VXLAN to work, these requirements must be fulfilled:

  • The Maximum Transition Unit (MTU) size throughout the transport must be increased to 50 bytes or the MTU size must be decreased on the Virtual Machines (VMs).

  • Proxy Address Resolution Protocol (ARP) must be configured on the gateways for the transport VLANs (Ethernet header (14) + UDP header (8) + IP header (20) + VXLAN header (8) = 50 bytes).

  • Multicast routing must be configured (for multicast mode).

    Note: This is only for VXLANs prior to Version 1.5 and Version 1.5 with Multicast mode. Version 1.5 was packaged with N1kV Version 4.2.1.SV2(2.1). This version supports a Unicast method of transport as well.

  • Internet Group Management Protocol (IGMP) snooping querier or Protocol Independent Multicast (PIM) must be configured.

  • User Datagram Protocol (UDP) port 8472 must be allowed through any firewalls.

    Note: This is the port that is used for the encapsulated traffic.

  • VMkernel interfaces must be configured on each host.

    Note: These are called the VXLAN Tunnel Endpoints (VTEP). They encapsulate and de-encapsulate the VXLAN traffic in the environment. You can use the same VMkernel that you use for l3control.

VXLAN Modes

In VXLAN Version 1.5, packaged in N1kV Version 4.2.1.SV2(2.1), VXLAN can operate in either Multicast mode or Unicast mode. Both of these modes are described in this section.

Multicast Mode

Each VXLAN has an assigned multicast group-IP. When a VM joins the VXLAN, the Virtual Ethernet Module (VEM) sends IGMP-Join requests to the assigned group. Broadcast, multicast, and flood traffic is sent to all VTEPs; unicast traffic is sent to the destination VTEP.

Unicast Mode

For broadcast, multicast, and unknown unicast frames, each VXLAN sends traffic to the destination IP address of each VTEP that houses a VM in the same VXLAN. If more than one VTEP exists, only one of the VTEPs is chosen to receive the flood traffic, which is similar to a designated broadcast receiver on the Cisco Unified Computing System (UCS). The VEMs then use the IP address as the destination VTEP for encapsulation.

With Unicast mode, there is also a MAC Distribution feature. With this feature, the VSM learns all of the MAC addresses from all of the VEMs and maps them to the designated VTEP. Flooding and replication is eliminated because the VEM always knows the destination VTEP for the specific destination VM.

Note: This is only supported for VEMs that are managed by the same VSM.

Configure

Use this section in order to configure VXLAN on a N1kV Series switch.

Enable the VXLAN Feature

Enter these commands in order to enable the VXLAN feature:

Nexus1000v# conf t Enter configuration commands, one per line. End with CNTL/Z.
Nexus1000v(config)# feature segmentation

Verify that the Feature is Enabled

Enter the show feature | grep segmentation command in order to verify that the feature is enabled:

Nexus1000v(config)# show feature | grep segmentation
network-segmentation 1 enabled
segmentation 1 enabled

Configure the Default Mode

Note: The default mode is unicast-only, without MAC distribution. This example in this document configures Multicast mode as the default and transitions to Unicast mode later.

Enter the no segment mode unicast-only command in order to configure the default mode:

Nexus1000v(config)# no segment mode unicast-only

Configure the VTEP Port-Profile

Now you must configure the port-profile that the VTEPs use. The port-profile configuration is similar to other access port-profiles, with the VXLAN feature added. Once the capability vxlan is configured, the VEM uses the designated VMkernel for encapsulation and de-encapsulation. It also includes the information as part of the Designated Receiver election.

Nexus1000v# conf t
Enter configuration commands, one per line. End with CNTL/Z.
Nexus1000v(config)# port-profile type vethernet VTEP
Nexus1000v(config-port-prof)# vmware port-group
Nexus1000v(config-port-prof)# switchport mode access
Nexus1000v(config-port-prof)# switchport access vlan 168
Nexus1000v(config-port-prof)# capability vxlan
Nexus1000v(config-port-prof)# no shutdown
Nexus1000v(config-port-prof)# state enabled

Create the VTEP VMKernel Interfaces

Complete these steps in order to create the VTEP VMkernel interface:

  1. Create a VMkernel and move it to the VXLAN-enabled port-profile:





  2. Choose the VTEP port-profile that has the capability vxlan configured:



  3. Assign the VTEP an IP address in your external VLAN:



  4. Finish the wizard.

Create the Bridge Domain

The bridge domain defines the Segment ID (VXLAN ID) and the Multicast Group (IP address).

Enter these commands in order to create the bridge domain:

Nexus1000v# conf t
Enter configuration commands, one per line. End with CNTL/Z.
Nexus1000v(config)# bridge-domain 192.168.1.x
Nexus1000v(config-bd)# segment id 5000
Nexus1000v(config-bd)# group 239.1.1.1

Create the VXLAN Port-Profile for VMs

In order for VMs to use the VXLAN, you must create a port-profile for them. The port-profile configuration is the same but it accesses a bridge domain instead of a VLAN.

Enter these commands in order to create the VXLAN port-profile for the VMs:

Nexus1000v# conf t
Enter configuration commands, one per line. End with CNTL/Z.
Nexus1000v(config)# port-profile type vethernet vxlan-192.168.1.x
Nexus1000v(config-port-prof)# vmware port-group
Nexus1000v(config-port-prof)# switchport mode access
Nexus1000v(config-port-prof)# switchport access bridge-domain 192.168.1.x
Nexus1000v(config-port-prof)# no shutdown
Nexus1000v(config-port-prof)# state enabled

Verify the Bridge Domain

Once you place the VMs into the port-profile, you must verify that the settings of the bridge domain are correct and that the VMs are included.

Enter the show bridge-domain command in order to verify the bridge domain:

Nexus1000v# show bridge-domain 192.168.1.x

Bridge-domain 192.168.1.x (2 ports in all)
Segment ID: 5000 (Manual/Active)
Mode: Multicast (override)
MAC Distribution: DisableGroup IP: 239.1.1.1
State: UP Mac learning: Enabled
Veth18, Veth19

Verify Connectivity

This image illustrates how to verify that your VMs have connectivity on the internal VXLAN:

Switch to Unicast Mode

Now you must switch to Unicast-Only mode and verify that you still have connectivity.

Enter these commands in order to switch to Unicast mode:

Nexus1000v# conf t
Enter configuration commands, one per line. End with CNTL/Z.
Nexus1000v(config)# bridge-domain 192.168.1.x
Nexus1000v(config-bd)# segment mode unicast-only

Verify the Unicast Bridge Domain

Enter the show bridge-domain command in order to verify the unicast bridge domain:

Nexus1000v# show bridge-domain 192.168.1.x

Bridge-domain 192.168.1.x (2 ports in all)
Segment ID: 5000 (Manual/Active)
Mode: Unicast-only (override)
MAC Distribution: DisableGroup IP: 239.1.1.1
State: UP Mac learning: Enabled
Veth18, Veth19

Verify Connectivity In Unicast Mode

This image illustrates how to verify connectivity in Unicast mode:

Deploy the VXLAN Gateway

This section gives an overview of the VXLAN gateway and describes the process that is used in order to deploy it.

About the VXLAN Gateway

The VXLAN gateway was created in order to provide a way for VXLAN segments to communicate with regular VLAN segments. It allows VMs on VXLANs to communicate with physical servers on external VLANs. This is comparable to the VMware vShield appliance, which allows VMs on internal networks to communicate with other machines on the provider networks.

This gateway is a Virtual Service Blade (VSB) that is installed on the Cisco Nexus 1x10 Series appliances. When you install the VXLAN gateway on a Nexus 1x10 Series appliance, you must run the appliance with network type flexible (type 5).

Note: The VXLAN gateway requires advanced licensing on the N1kV, so you must ensure that you have sufficient licenses before you switch to the advanced edition with the svs switch edition advanced command in configuration mode.

Enable the Feature

In order to enable the VXLAN gateway, you must enable the feature on the N1kV. Enter these commands in order to enable the feature:

Nexus1000v(config)# feature vxlan-gateway
Nexus1000v(config)# 2013 Aug 1 18:34:20 Nexus1000v %SEG_BD-2-VXLAN_GATEWAY_ENABLED:
  Feature vxlan-gateway enabled

Nexus1000v(config)# show feature | grep gateway
vxlan-gateway 1 enabled

Prepare the VSM for the VXLAN Gateway

Port-profiles for the VXLAN gateway must be provisioned on the VSM prior to installation. These profiles include an uplink port-profile and a vEthernet port-profile for the VTEP, which are configured with these commands:

Nexus1000v(config)# feature vxlan-gateway
Nexus1000v(config)# 2013 Aug 1 18:34:20 Nexus1000v %SEG_BD-2-VXLAN_GATEWAY_ENABLED:
  Feature vxlan-gateway enabled


Nexus1000v(config)# show feature | grep gateway
vxlan-gateway 1 enabled

Now you must configure the VXLAN gateway VTEP port-profile, which includes the capability vxlan feature as well as the transport IP address that the VXLAN gateway uses in order to transport the traffic. Enter these commands in order to configure it:

Nexus1000v(config)# port-profile vxgw-vtep  <---No vmware port-group cmd;
  Not published to vCenter
Nexus1000v(config-port-prof)# switchport mode access
Nexus1000v(config-port-prof)# switchport access vlan 168
Nexus1000v(config-port-prof)# capability vxlan
Nexus1000v(config-port-prof)# no shutdown
Nexus1000v(config-port-prof)# state enabled
Nexus1000v(config-port-prof)# transport ip address 10.10.168.25 255.255.255.0
  gateway 10.10.168.254

The VXLAN gateway requires a port-profile for the two physical passthrough interfaces. These links must be configured in a port-channel trunk. Both Link Aggregation Control Protocol (LACP) and static port-channels are supported.

The gateway takes a VXLAN segment and maps it to an 802.1Q VLAN. This is configured in a service instance.

Note: You can also configure the service instance after the gateway is registered.

Enter these commands in order to configure the port-channel trunk:

Nexus1000v(config)#port-profile type ethernet vxlan-gw-uplink <---No vmware
  port-group cmd.
Nexus1000v(config-port-prof)# switchport mode trunk
Nexus1000v(config-port-prof)# switchport trunk allowed vlan 119,219,319
Nexus1000v(config-port-prof)# mtu 9000
Nexus1000v(config-port-prof)# channel-group auto mode active
Nexus1000v(config-port-prof)# no shutdown
Nexus1000v(config-port-prof)# description Virtual PP push in opaque data to vxgw
Nexus1000v(config-port-prof)# state enabled
Nexus1000v(config-port-prof)# service instance 168
Nexus1000v(config-port-prof-srv)# encapsulation dot1q 168 bridge-domain 192.168.1.x
Nexus1000v(config-port-prof-srv)#

In order to register the VXLAN gateway with the VSM, you must make note of the primary and secondary MAC addresses from the VSMs. Enter the show vms internal info command from the VSM:

Nexus1000v# show vms internal info  Global svs connection mode: ipv4
Cached IP address: 10.10.168.2
DVS INFO:
-------------
DVS name: [Nexus1000v]
UUID: [ee 63 3c 50 04 b1 6d d6-58 61 ff ba 56 05 14 fd]
Description: [(null)]
Config version: [91]
Max ports: [8192]
DC name: [jristain]
OPQ data&colon; size [723], data&colon; [data-version 1.0]

switch-domain 27
switch-name Nexus1000v
cp-version 4.2(1)SV2(2.1)
control-vlan 1
system-primary-mac 00:50:56:bc:6a:3d
active-vsm packet mac 00:50:56:bc:2a:5f
active-vsm mgmt mac 00:50:56:bc:57:4e
standby-vsm ctrl mac 0050-56bc-74f1
inband-vlan 1
svs-mode L3
l3control-ipaddr 10.10.168.2

You can now install the VXLAN gateway on the Nexus 1x10 Series appliance.

Install the VXLAN Gateway on the 1x10 Appliance

Once you copy the VXLAN gateway software to the bootflash:repository directory of the Nexus 1x10 Series appliance, you can create a VSB. Enter these commands in order to create it:

1010(config)# virtual-service-blade VXGW
1010(config-vsb-config)# virtual-service-blade-type
  new vxgw.4.2.1.SV2.2.1.iso

Enter the show virtual-service-blade command in order to verify the interfaces that you created in the VSB:

1010(config-vsb-config)# show virtual-service-blade name VXGW
virtual-service-blade VXGW
Description:
Slot id: 1
Host Name:
Management IP:
VSB Type Name : vx-gw-1.5
Configured vCPU: 3
Operational vCPU: 3
Configured Ramsize: 2048
Operational Ramsize: 2048
Disksize: 3
Heartbeat: 0


Legends: P - Passthrough
--------------------------------------------------------------------
Interface Type MAC VLAN State Uplink-Int
Pri Sec Oper Adm
--------------------------------------------------------------------
VsbEthernet1/1 gw-uplink1 up up
VsbEthernet1/2 management 168 up up
VsbEthernet1/3 gw-uplink2 up up
internal NA NA NA up up
HA Role: Primary
HA Status: NONE
Status: VSB NOT PRESENT
Location: PRIMARY
SW version:
HA Role: Secondary
HA Status: NONE
Status: VSB NOT PRESENT
Location: SECONDARY
SW version:
VSB Info:

Check the network summary in order to view the available uplinks. The VXLAN gateway interfaces must be placed into a passthrough mode and pinned to a Nexus 1x10 Series appliance uplink. Enter the show network summary command in order to view the network summary:

1010(config-vsb-config)# show network summary
Legends: P - Passthrough
------------------------------------------------------------------------------
Port State Uplink-Interface Speed RefCnt MTU Nat-Vlan
Oper Admin Oper Admin Oper Admin
------------------------------------------------------------------------------
Gi1 up up 1000 1 9000
Gi2 up up 1000 1 9000
Gi3 up up 1000 0 9000
Gi4 up up 1000 0 9000
Gi5 up up 1000 0 9000
Gi6 up up 1000 0 9000
control0 up up Gi1 Gi1 1000 9000
mgmt0 up up Gi2 Gi2 1000 9000

Pin the VSB interfaces to the Nexus 1x10 Series appliance uplinks and set them to Passthrough mode. You must also configure a VLAN ID for the management VSB interface. 

Note: Ensure that you have LACP enabled on the upstream interfaces. When you configure these sections, the uplink interfaces perform LACP.

Enter these commands in order to pin the VSB interfaces to the uplinks:

1010(config-vsb-config)# interface gw-uplink1 uplink GigabitEthernet3
1010(config-vsb-config)# interface gw-uplink2 uplink GigabitEthernet4
1010(config-vsb-config)# interface gw-uplink1 mode passthrough
1010(config-vsb-config)# interface gw-uplink2 mode passthrough
1010(config-vsb-config)# interface management uplink GigabitEthernet2
1010(config-vsb-config)# interface management vlan 168

Once the network uplinks are configured, enable the VSB and verify that the deployment is successful. There are numerous fields that you must enter information into when you deploy the VSB; you must add the primary and standby MAC addresses from the VSM that was previously described. The service module is the VXLAN gateway. Also, two IP addresses are needed in order to complete the installation.

1010(config-vsb-config)# enable
Enter vsb image: [vxgw.4.2.1.SV2.2.1.iso]
Enter the VSM domain id[1-4095]: 27
Enter Management IP version [V4]: [V4]
Enter Management IP address of service module on primary: 10.10.168.101
Enter Management subnet mask of service module on primary: 255.255.255.0
Enter default gateway IP address of service module on primary: 10.10.168.254
Enter management IP address of service module on secondary: 10.10.168.102
Enter management subnet mask of service module on secondary: 255.255.255.0
Enter default gateway IP address of service module on secondary: 10.10.168.254
Enter HostName: VXLANGW
Enter the password for 'admin': S0lT3st1ng
VSM L3 Ctrl IPv4 address : 10.10.168.2
VSM Primary MAC Address: 0050.56bc.6a3d
VSM Standby MAC Address: 0050.56bc.74f1
Enter VSM uplink port-profile name: vxgw-pc
Enter VTEP port-profile name: vxgw-vtep
----Details entered----
DomainId : 27
IPV4V6 : V4PriMgmtIpV4 : 10.10.168.101
PriMgmtIpV4Subnet : 255.255.255.0
PriGatewayIpV4 : 10.10.168.254
SecMgmtIpV4 : 10.10.168.102
SecMgmtIpV4Subnet : 255.255.255.0
SecGatewayIpV4 : 10.10.168.254
HostName : VXLANGW
Password : S0lT3st1ng
VSMIpV4 : 10.10.168.2
VSMPriMac : 0050.56bc.6a3d
VSMStdbyMac : 0050.56bc.74f1
UplinkPPName : vxgw-pcVSMEncapPPName : vxgw-vtep

Do you want to continue installation with entered details (Y/N)? [Y]
Note: VSB installation is in progress, please use show virtual-service-blade
  commands to check the installation status.
Note: VSB installation may take up to 5 minutes.

Enter the show virtual-service-blade summary command on the Nexus 1x10 Series appliance in order to verify that the VSB is deployed and powered on:

1010(config-vsb-config)# show virtual-service-blade summary

------------------------------------------------------------------
Name HA-Role HA-Status Status Location
------------------------------------------------------------------
VXGW PRIMARY ACTIVE VSB POWERED ON PRIMARY
VXGW SECONDARY NONE VSB DEPLOY IN PROGRESS SECONDARY

Verify the Modules in the VSM

The VXLAN gateways are now added as modules in the VSM configuration. Enter the show module command on the VSM in order to verify:

Nexus1000v# show module

Mod Ports Module-Type Model Status
--- ----- ------------------------- ------------- -----------
1 0 Virtual Supervisor Module Nexus1000V ha-standby
2 0 Virtual Supervisor Module Nexus1000V active *
3 248 Virtual Ethernet Module NA ok
4 248 Virtual Ethernet Module NA ok
5 332 Virtual Ethernet Module NA ok
6 332 Virtual Ethernet Module NA ok
7 4 Virtual Service Module VXLAN Gateway ok
8 4 Virtual Service Module VXLAN Gateway ok

Mod Sw Hw
--- -------------- --------------------------------------------
1 4.2(1)SV2(2.1) 0.0
2 4.2(1)SV2(2.1) 0.0
3 4.2(1)SV2(2.1) VMware ESXi 5.1.0 Releasebuild-799733 (3.1)
4 4.2(1)SV2(2.1) VMware ESXi 5.1.0 Releasebuild-799733 (3.1)
5 4.2(1)SV2(2.1) VMware ESXi 5.1.0 Releasebuild-799733 (3.1)
6 4.2(1)SV2(2.1) VMware ESXi 5.1.0 Releasebuild-799733 (3.1)
7 4.2(1)SV2(2.1) Linux 2.6.27.10
8 4.2(1)SV2(2.1) Linux 2.6.27.10

Mod Server-IP Server-UUID Server-Name
--- -------------- ------------------------------------ -----------
1 10.10.168.2 NA NA
2 10.10.168.2 NA NA
3 10.10.168.1 24266920-d498-11e0-0000-00000000000f 10.10.168.1
4 10.10.168.4 24266920-d498-11e0-0000-00000000000e 10.10.168.4
5 10.10.168.5 d54be571-831f-11df-aaa7-d0d0fd095a08 10.10.168.5
6 10.10.168.3 24266920-d498-11e0-0000-00000000000c 10.10.168.3
7 10.10.168.101 e6b86534-5d0c-4cde-a48e-2b555f929d2b VXLANGW
8 10.10.168.102 06cc2f30-bc2b-4b6f-a7d2-4e712c530761 VXLANGW

Form a High Availability

Now you can configure the modules in a High Availability (HA) pair.

Enter the show module service-module command on the VSM in order to verify the status of the modules:

Nexus1000v# show module service-module

Mod Cluster-id Role HA Mode Status
--- ---------- ----------- ---------- -------
7 0 Unconfigured Standalone Init
8 0 Unconfigured Standalone Init

In order to configure the HA, ensure that the cluster number matches the service instance in the VXLAN gateway uplink port-profile:

Nexus1000v# configure t
Enter configuration commands, one per line. End with CNTL/Z.
Nexus1000v(config)# service 7 role primary ha-cluster 1
Nexus1000v(config)# service 8 role secondary ha-cluster 1

Verify that the VXLAN gateway forms an HA pair. Also, verify that the port-channels are configured for LACP and that they are Active:

Nexus1000v# show module service-module

Mod Cluster-id Role HA Mode Status
--- ---------- ----------- -------- -------
7 1 Primary HA Active
8 1 Secondary HA Standby
Nexus1000v# show port-channel summary

Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)

-----------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
-----------------------------------------------------------
1 Po1(SU) Eth NONE Eth4/1(P) Eth4/2(P)
2 Po2(SU) Eth NONE Eth3/1(P) Eth3/2(P)
3 Po3(SU) Eth NONE Eth6/1(P) Eth6/2(P)
4 Po4(SU) Eth NONE Eth5/2(P)
5 Po5(SD) Eth NONE --
6 Po6(SU) Eth LACP Eth7/1(P) Eth7/3(P)
7 Po7(SU) Eth LACP Eth8/1(P) Eth8/3(P)

Enter the service ha-cluster command if you want to switchover the VXLAN gateway:

Nexus1000v# service ha-cluster 1 switchover

Note: This switchover command is different than a traditional Nexus Operating System (NXOS) switchover command because you must switchover the HA cluster that you created.

Verify

Use this section in order to confirm that your configuration works properly.

The Output Interpreter Tool (registered customers only) supports certain show commands. Use the Output Interpreter Tool in order to view an analysis of show command output.

VSM CLI

Verify that the VTEPs are programmed correctly on the VSM:

Nexus1000v# show bridge-domain 192.168.2.x vteps

D: Designated VTEP I:Forwarding Publish Incapable VTEP

Bridge-domain: 192.168.2.xVTEP Table Version: 9

Port Module VTEP-IP Address VTEP-Flags
------ ------- ---------------- -------------
Veth11 3 10.17.168.20 (D)
Veth28 4 10.17.168.22 (D)
Veth21 7 10.17.124.70 (DI*) <---- VXLAN GW
Veth22 8 10.17.124.70 (DI) <---- VXLAN GW (Standby)

VEM Commands

There are many VEM commands (VEMCMDs) that you can use in order to verify the configuration, view the statistics, and validate the setup of your VXLAN. In order to view the available commands, search the database with the vemcmd | grep vxlan command:

~ # vemcmd | grep vxlan

show vxlan interfaces Show the VXLAN Encap Interfaces
show vxlan-encap ltl <ltl> Show VXLAN Encap Information
show vxlan-encap mac <MAC.MAC.MAC>
show vxlan-stats Show VXLAN port stats for all ports
show vxlan-stats bd-all Show VXLAN port stats for all BDs
show vxlan-stats ltl <ltl> Show VXLAN port stats detail
show vxlan-stats ltl-detail Show all VXLAN ports stats detail
show vxlan-stats ltl <ltl> bd-all cookie <number>
show vxlan-stats ltl <ltl> bd-name <bd-name>
show vxlan-vteps Show VXLAN VTEPs
show vxlan-vteps bd-name <bd-name>
show vxlan threads Show the VXLAN thread stats
clear vxlan threads Clear the VXLAN thread stats
show vlan-vxlan mapping Show VXLAN VLAN mappings

Enter the vemcmd show vxlan interfaces command in order to verify that the VEM is programmed with the correct VTEP:

~ # vemcmd show vxlan interfaces
LTL VSM Port IP Seconds since Last Vem Port
IGMP Query Received
(* = IGMP Join Interface/Designated VTEP)
------------------------------------------------------------
51 Veth6 10.10.168.22 33 vmk2*

Note: The * in the output shows the VTEP that is the designated receiver on the host.

You should also verify that the Seconds since Last IGMP Query Received number transitions to 0 after some time. The default IGMP query time for NXOS is 125 seconds. This proves that IGMP queries are received on the VTEP and that the multicast transport is functional.

Note: You cannot see the VTEP vEthernet ports in the output of the show ip igmp snooping groups command on an N1kV. By default, all of the multicast traffic for the bridge domain groups is flooded on the transport VLAN; thus, IGMP snooping is not utilized on the N1kV.

Enter the vemcmd show vxlan-encap ltl <x> command in order to view encapsulation information for a specific VM:

~ # vemcmd show vxlan-encap ltl 53

Encapsulation details for LTL 53 in BD "192.168.1.x":
Source MAC: 00:50:56:bc:77:25
Segment ID: 5000
Multicast Group IP: 239.1.1.1

Encapsulating VXLAN Interface LTL: 51
Encapsulating Source IP: 10.10.168.22
Encapsulating Source MAC: 00:50:56:6d:7a:25

Pinning of VXLAN Interface to the Uplink:
LTL IfIndex PC_LTL VSM_SGID Eff_SGID iSCSI_LTL* Name
51 1c000050 561 32 0 0 vmk2= IGMP Join

Enter the vemcmd show vxlan-stats command on the host in order to verify the statistics for VXLAN. This command shows the number of encapsulations and de-encapsulations that occur for each VM Local Target Logic (LTL).

Note: All broadcast and multicast traffic is classified as an Mcast/Repl Encaps. This is because the traffic must be sent to all VTEPs. Unicast traffic is classified as Ucast Encaps. If you attempt to troubleshoot an issue where ARP does not complete, then verify that the Mcast/Repl Encaps column increments with every ARP request.

~ # vemcmd show vxlan-stats
LTL Ucast Mcast/Repl Ucast Mcast Total
Encaps Encaps Decaps Decaps Drops
51 7557 507 8012 0 0
53 7137 431 7512 0 0

Enter the vemcmd show 12 segment 5001 command in order to verify that the source host learns the MAC address of the destination dynamically:

~ # vemcmd show l2 segment 5001
Bridge domain 14 brtmax 4096, brtcnt 2, timeout 300
Segment ID 5001, swbd 4097, "192.168.2.x"
Flags: P - PVLAN S - Secure D - Drop
Type MAC Address LTL timeout Flags PVLAN Remote IP DSN
Static 00:50:56:bc:77:25 55 0 0.0.0.0 0
Dynamic 00:50:56:bc:19:5b 561 0 10.17.168.22 0

Troubleshoot

There is currently no specific troubleshooting information available for this configuration.

Updated: Apr 14, 2014
Document ID: 117618