Cisco Nexus 1000V provides Layer 2 switching functions in a virtualized server environment. Nexus 1000V replaces virtual switches within ESX servers and allows users to configure and monitor the virtual switch using the Cisco NX-OS command line interface. Nexus 1000V also gives you visibility into the networking components of the ESX servers and access to the virtual switches within the network.
The Nexus 1000V manages a data center defined by the vCenter Server. Each server in the Datacenter is represented as a linecard in Nexus 1000V and can be managed as if it were a line card in a physical Cisco switch. The Nexus 1000V implementation has two components:
Virtual supervisor module (VSM) – This is the control software of the Nexus 1000V distributed virtual switch. It runs on a virtual machine (VM) and is based on NX-OS.
Virtual Ethernet module (VEM) – This is the part of Cisco Nexus 1000V that actually switches data traffic. It runs on a VMware ESX 4.0 host. Several VEMs are controlled by one VSM. All the VEMs that form a switch domain should be in the same virtual Datacenter as defined by VMware vCenter Server.
Cisco Nexus 1000V Getting Started Guide
for a detailed overview of how the Nexus 1000V works with VMware ESX software.
General Restrictions for vCenter Server
When you are troubleshooting issues related to vCenter Server, make sure that you observe the following restrictions:
The name of a distributed virtual switch (DVS) name must be unique across Datacenters
You create a DVS in a network folder
A Datacenter cannot be removed unless the DVS folder or the underlying DVS is deleted.
A DVS can be deleted only with the help of VSM using the
no vmware dvs
command in config-svs-conn mode.
The no vmware dvs command can succeed only if there are no VMs using the DVS port-groups.
A port group on vCenter Server can be deleted only if there are no interfaces associated with it.
A sync operation performed in conjunction with the
command helps VSM keep in sync with vCenter Server.
Each VSM uses a unique extension key to communicate with vCenter Server and perform operations on a DVS.
The VSM uses the extension key when communicating with the vCenter Server. Each VSM has its own unique extension key, such as Cisco_Nexus_1000V_32943215
show vmware vc extension-key
command to find the extension key of the VSM. It is also listed in the .xml file.
If you have not previously saved a back up copy of the VSM configuration file, then you may try recreating the old port profiles before connecting to the VC. This procedure has a step for recreating port profiles. If you do not recreate these before connecting to VC, then all the port groups present on the VC are removed and all ports in use are moved to the quarantine port groups.
Make sure that the VSM VM switchname is the same as the DVS switchname on the vCenter Server. This allows the VSM configuration to synchronize with the correct DVS on the vCenter Server.
To change the VSM switchname use the
The host does not show up in the Add host to DVS screen.
Make sure that the Host is installed with VMware Enterprise plus
license containing the Distributed Virtual Switch feature.
Add host to DVS returns an error.
Confirm that the VEM software is installed on the ESX server,
The server name column of the
command output shows the IP address.
The server name shows the host-name or IP address, whichever was used to add the host to the DVS on the vCenter Server.
Example 21-1 shows the
show vms internal event-history errors
command that is useful for examining VC errors in detail. It shows whether an error is caused by a VSM (client) or the server.
Example 21-1 show vms internal event-history error Command
n1000v# show vms internal event-history errors
Event:E_DEBUG, length:239, at 758116 usecs after Tue Feb 3 18:21:58 2009
 convert_soap_fault_to_err(1179): SOAP 1.1 fault: "":ServerFaultCode [VMWARE-VIM] A DVS n1000v with spec.name as n1000v already exists, cannot create DVS n1000v. A specified parameter was not correct.spec.name
Event:E_DEBUG, length:142, at 824006 usecs after Tue Feb 3 18:18:30 2009
 convert_soap_fault_to_err(1179): SOAP 1.1 fault: SOAP-ENV:Client [VMWARE-VIM] Operation could not be completed due to connection failure.
Event:E_DEBUG, length:134, at 468208 usecs after Tue Feb 3 18:15:37 2009
 convert_soap_fault_to_err(1179): SOAP 1.1 fault: "":ServerFaultCode [VMWARE-VIM] Extension key was not registered before its use.
Connection Failure After ESX Reboot
To prevent a loss of connectivity between the VSM and VEM, and preserve a non-default MTU setting for a physical NIC across reboots of the ESX, you must configure a system MTU in the system port profile.
If you use an MTU other than 1500 (the default) for a physical NIC attached to the Cisco Nexus 1000V, then reboots of the ESX can result in a mismatch with the VMware kernel NIC MTU and failure of the VSM and VEM. For example, you may manually configure an MTU of other than 1500 in networks with jumbo frames. During a power cycle, the ESX reboots and the MTU of the physical NIC reverts to the default of 1500 but the VMware kernel NIC does not.
(Optional) Saves the running configuration persistently through reboots and restarts by copying it to the startup configuration.
Recovering Lost Connectivity Due to MTU Mismatch
Use this procedure to recover lost connectivity due to an MTU mismatch between the physical NIC and the VMware kernel NIC after an ESX reboot.
BEFORE YOU BEGIN
Before beginning this procedure, you must know or do the following:
You are logged in to the CLI in EXEC mode.
To verify the ESX MTU settings for corresponding PNICs, use the
Note Use vemcmds only as a recovery measure and then update the MTU value in the port profile configuration for system uplinks or in the interface configuration for non-system uplinks.
execute vemcmd show port
execute vemcmd set mtu
n1000v# config t
Enters global configuration mode.
execute vemcmd show port
Displays the port configuration including the LTL number needed for Step 3.
n1000v(config)# module vem 3 execute vemcmd show port 48
LTL IfIndex Vlan Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name
17 1a030100 1 T 304 1 32 PHYS UP UP 1 Trunk vmnic1
execute vemcmd set mtu
n1000v(config)# module vem 3 execute vemcmd set mtu 9000 ltl 17
Designates the MTU size for the port, using the LTL number obtained in Step 2.
The VSM VM is stuck at the boot prompt.
Make sure that you have three e1000 NICs.
The VSM VM cannot ping itself.
Configure the mgmt0 interface.
The VSM VM can ping itself, but not the gateway.
Make sure the NIC order is correct: control, mgmt, inband.
The VSM VM can ping the gateway, but not the outside subnet.
Configure vrf context management.
When creating a port profile, use the following commands to create the corresponding port groups on the vCenter Server:
Profiles that have the system VLAN configuration allow the VEM to communicate with the VSM.
Make sure that the system port-profile is defined with the right system VLANS.
show port-profile usage
commands to collect basic required information.
Problems with Port Profiles
You receive an error message “Possible failure in communication with vCenter Server.”
The VSM is not connected to the vCenter Server.
svs connection vc
command to connect to the vCenter Server.
The port group name is not unique.
Port group names must be unique within a vCenter Server Datacenter.
Port profile or port groups do not appear on the vCenter Server.
Make sure you have issued the
Problems with Hosts
You receive an error message, DVS Operation failed for one or more members.”
vem status -v
command to verify if the VEM is running on the host.
command to unload the VEM.
In the vSphere Client, remove the stale DVS:
1. Go to the
Distributed Virtual Switch
The host is visible on the vCenter Server, but not the VSM.
vemcmd show trunk
command to verify that there is an uplink carrying the control VLAN. The profile applied to the uplink must be a system profile with a control VLAN as a system VLAN.
Verify the control VLAN in the upstream switch port and the path to the VSM VM. Make sure that one uplink at most carries the control VLAN, or that all uplinks and upstream ports carrying the control VLAN are in port channels.
A module flap occurs.
The VSM may be overloaded. Make sure that you have 1 GB of memory and CPU shares for the VSM VM on the vCenter Server.
Problems with VM Traffic
When troubleshooting problems with intra-host VM traffic, follow these guidelines:
Make sure that at least one of the VMware virtual NICs is on the correct DVS port group and is connected.
If the VMware virtual NIC is down, determine if there is a conflict between the MAC address configured in the OS and the MAC address assigned by VMware. You can see the assigned MAC addresses in the vmx file.
When troubleshooting problems with inter-host VM traffic, follow these guidelines:
Determine if there is exactly one uplink sharing a VLAN with the VMware virtual NIC. If there is more than one, they must be in a port channel.
Ping a SVI on the upstream switch using the
show intX counters
VEM Troubleshooting Commands
Use the following commands to display VEM information:
– displays and controls VEM kernel logs
– displays configuration and status information
– collects support information
– collects status information
– collects version information
vemlog show last
– displays the circular buffer
Example 21-2 vemlog show last Command
[root@esx-cos1 ~]# vemlog show last 5
Timestamp Entry CPU Mod Lv Message
Oct 13 13:15:52.615416 1095 1 1 4 Warning vssnet_port_pg_data_ …
Oct 13 13:15:52.620028 1096 1 1 4 Warning vssnet_port_pg_data_ …
Oct 13 13:15:52.630377 1097 1 1 4 Warning svs_switch_state …
Oct 13 13:15:52.633201 1098 1 1 8 Info vssnet new switch …
Oct 13 13:16:24.990236 1099 1 0 0 Suspending log
vemlog show info
– displays information about entries in the log
Example 21-3 vemcmd show info Command
[root@esx-cos1 ~]# vemlog show info
Total Entries: 1092
Wrapped Entries: 0
Lost Entries: 0
Skipped Entries: 0
Available Entries: 6898
Stop After Entry: Not Specified
– displays the type of information you can display
Example 21-4 vemcmd help Command
[root@esx-cos1 ~]# vemcmd help
show card Show the card's global info
show vlan [vlan] Show the VLAN/BD table
show bd [bd] Show the VLAN/BD table
show l2 <bd-number> Show the L2 table for a given BD/VLAN
show l2 all Show the L2 table
show port [priv|vsm] Show the port table
show pc Show the port channel table
show portmac Show the port table MAC entries
show trunk [priv|vsm] Show the trunk ports in the port table
show stats Show port stats
VEM Log Commands
Use the following commands to control the vemlog:
– stops the log
– clear s the log
– starts the log and stops it after the specified number of entries
– stops the log after the next specified number of entries
– starts the log, but does not clear the stop value
On the vSphere Client, you can see error messages under the recent tasks tab. You can find detailed description of the error under the Tasks and Events tab. The same messages are also propagated to the VSM.
Table 21-1 lists error messages that you might see on the VSM.
Table 21-1 Error Messages on the VSM
ERROR: [VMWARE-VIM] Extension key was not registered before its use
This error indicates that VSM extension key is not registered.
ERROR: [VMWARE-VIM] A DVS n1000v with spec.name as n1000v already exists, cannot create DVS n1000v. A specified parameter was not correct. spec.name.
This error is displayed after you enter the first
command, and indicates that a DVS already exists with the same name.
ERROR: [VMWARE-VIM] A DVS n1000v with spec.extensionKey as Cisco_Nexus_1000V_2055343757 already exists, cannot create DVS new-n1000v. A specified parameter was not correct. spec.extensionKey
This error is displayed when the VSM tries to create a different DVS after changing the switch name.
ERROR: [VMWARE-VIM] A DVS n1000v with name as n1000v already exists, cannot reconfigure DVS test. A specified parameter was not correct. Spec.name
This error indicates that a DVS with the same name already exists.
Warning: Operation succeeded locally but update failed on vCenter server.[VMWARE-VIM] DVPortgroup test port 0 is in use. The resource vim.dvs.DistributedVirtualPort 0 is in use.
This warning is displayed when the VSM tries to delete the port profile if the VSM is not aware of the nics attached to the port groups.