Cisco Nexus 1000V Troubleshooting Guide, Release 4.0(4)SV1(3)
System
Downloads: This chapterpdf (PDF - 197.0KB) The complete bookPDF (PDF - 4.35MB) | Feedback

System

Table Of Contents

System

Information About the System

General Restrictions for vCenter Server

Extension Key

Recovering a DVS

Recovering a DVS With a Saved Copy of the VSM

Recovering a DVS Without a Saved Copy of the VSM

Problems Related to VSM and vCenter Server Connectivity

Connection Failure After ESX Reboot

Information About Changing the MTU

Verifying that an MTU Mismatch Caused a Connection Failure

Restoring Network Connectivity to the VEM

Displaying VC Event History

VSM Creation

Port Profiles

Problems with Port Profiles

Problems with Hosts

Problems with VM Traffic

VEM Troubleshooting Commands

VEM Log Commands

Error Messages


System


This chapter describes the Cisco Nexus 1000V system and how to identify and correct problems related to the system.

This chapter includes the following sections:

Information About the System

General Restrictions for vCenter Server

Recovering a DVS

Problems Related to VSM and vCenter Server Connectivity

VSM Creation

Port Profiles

Problems with Hosts

Problems with VM Traffic

Connection Failure After ESX Reboot

VEM Troubleshooting Commands

VEM Log Commands

Error Messages

Information About the System

Cisco Nexus 1000V provides Layer 2 switching functions in a virtualized server environment. you can configure and monitor the virtual switch in an ESX server using the Cisco NX-OS command line interface. You also have visibility into the networking components of the ESX servers and access to the virtual switches within the network.

The Cisco Nexus 1000V manages a data center defined by the vCenter Server. Each server in the Datacenter is represented as a linecard and can be managed as if it were a line card in a physical Cisco switch. The Cisco Nexus 1000V has two components:

Virtual supervisor module (VSM) - This is the control software of the Cisco Nexus 1000V distributed virtual switch. It runs on a virtual machine (VM) and is based on Cisco NX-OS.

Virtual Ethernet module (VEM) - This is the part of Cisco Nexus 1000V that actually switches data traffic. It runs on a VMware ESX 4.0 host. Several VEMs are controlled by one VSM. All the VEMs that form a switch domain should be in the same virtual Datacenter as defined by VMware vCenter Server.

For more information, see the Cisco Nexus 1000V Getting Started Guide, Release 4.0(4)SV1(3).

General Restrictions for vCenter Server

When you are troubleshooting issues related to vCenter Server, make sure that you observe the following restrictions:

The name of a distributed virtual switch (DVS) name must be unique across Datacenters

You create a DVS in a network folder

A Datacenter cannot be removed unless the DVS folder or the underlying DVS is deleted.

A DVS can be deleted only with the help of VSM using the no vmware dvs command in config-svs-conn mode.

The no vmware dvs command can succeed only if there are no VMs using the DVS port-groups.

A port group on vCenter Server can be deleted only if there are no interfaces associated with it.

A sync operation performed in conjunction with the connect command helps VSM keep in sync with vCenter Server.

Each VSM uses a unique extension key to communicate with vCenter Server and perform operations on a DVS.

Extension Key

The VSM uses the extension key when communicating with the vCenter Server. Each VSM has its own unique extension key, such as Cisco_Nexus_1000V_32943215

Use the show vmware vc extension-key command to find the extension key of the VSM. It is also listed in the .xml file.

The extension key registered on the vCenter Server can be found through the MOB. For more information, see the "Finding the Extension Key Tied to a Specific DVS" procedure on page 3-8.

The same extension key cannot be used to create more than one DVS on the vCenter Server.

Recovering a DVS

You can use this procedure to recover a DVS if the VSM VM that was used to create it is lost or needs to be replaced. This section includes the following procedures:

Recovering a DVS With a Saved Copy of the VSM

Recovering a DVS Without a Saved Copy of the VSM

Recovering a DVS With a Saved Copy of the VSM

You can use this procedure to recover a DVS when you have previously saved a back up copy of the VSM configuration file.

BEFORE YOU BEGIN

Before starting this procedure, you must know or do the following:

Use this procedure if you have previously saved a back up copy of the VSM configuration file. If you have not previously saved a back up copy, the see the "Recovering a DVS Without a Saved Copy of the VSM" procedure.

Make sure that the VSM VM switchname is the same as the DVS switchname on the vCenter Server. This allows the VSM configuration to synchronize with the correct DVS on the vCenter Server.

To change the VSM switchname use the switchname newname command.


Step 1 From the MOB, find the DVS extension key.
For more information, see the "Finding the Extension Key Tied to a Specific DVS" procedure on page 3-8.

Step 2 On the VSM, add the DVS extension key found in Step 1.

The extension key allows the VSM to log in to the vCenter server.

Example:
n1000v# config t
n1000v(config)# vmware vc extension-key Cisco_Nexus_1000V_32943215

Step 3 From the MOB, unregister the extension key found in Step 1.

For more information, see the "Unregistering the Extension Key in the vCenter Server" procedure on page 3-12.

Step 4 From the VC client, register the extension (plug-in) for the VSM.

For more information see the following procedure in the Cisco Nexus 1000V Getting Started Guide, Release 4.0(4)SV1(3).

Creating a Cisco Nexus 1000V Plug-In on the vCenter Server

Step 5 On the VSM, restore the configuration using a previously saved copy of the VSM configuration file.

copy path/filename running-config

Example:
n1000v# copy sftp://user1@172.22.36.10/backup/hamilton_cfg running-config

Step 6 Do one of the following:

If the vCenter server connection is not part of the previously saved configuration, continue with the next step.

Otherwise, go to Step 8.

Step 7 On the VSM, restore the configuration for the vCenter server connection.

Example:
n1000v# config t
n1000v (config)# svs connection VC 
n1000v(config-svs-conn#) protocol vmware-vim
n1000v(config-svs-conn#) remote ip address 192.168.0.1 
n1000v(config-svs-conn#) vmware dvs datacenter-name Hamilton-DC 

Step 8 Connect to vCenter Server.

Example:
n1000v(config-svs-conn#) connect

You can now use the old DVS or remove it.


Recovering a DVS Without a Saved Copy of the VSM

You can use this procedure to recover a DVS when you have not previously saved a back up copy of the VSM configuration file.

BEFORE YOU BEGIN

Before starting this procedure, you must know or do the following:

The folder in which the VSM resides must be:

At the root-level of the Data Center in which it resides.
It cannot be embedded in another folder.

Of the same name as the VSM.

If the folder does not meet the above criteria, the connection to vCenter server fails with the error, the VSM already exists.

Use this procedure if you have not previously saved a back up copy of the VSM configuration file. If you have previously saved a back up copy, then see the "Recovering a DVS With a Saved Copy of the VSM" procedure.

If you have not previously saved a back up copy of the VSM configuration file, then you may try recreating the old port profiles before connecting to the VC. This procedure has a step for recreating port profiles. If you do not recreate these before connecting to VC, then all the port groups present on the VC are removed and all ports in use are moved to the quarantine port groups.

Make sure that the VSM VM switchname is the same as the DVS switchname on the vCenter Server. This allows the VSM configuration to synchronize with the correct DVS on the vCenter Server.

To change the VSM switchname use the switchname newname command.


Step 1 From the MOB, find the DVS extension key.
For more information, see the "Finding the Extension Key Tied to a Specific DVS" procedure on page 3-8.

Step 2 On the VSM, add the DVS extension key found in Step 1.

The extension key allows the VSM to log in to the vCenter server.

Example:
n1000v# config t
n1000v(config)# vmware vc extension-key Cisco_Nexus_1000V_32943215

Step 3 From the MOB, unregister the extension key found in Step 1.

For more information, see the "Unregistering the Extension Key in the vCenter Server" procedure on page 3-12.

Step 4 From the VC client, register the extension (plug-in) for the VSM.

For more information see the following procedure in the Cisco Nexus 1000V Getting Started Guide, Release 4.0(4)SV1(3).

Creating a Cisco Nexus 1000V Plug-In on the vCenter Server

Step 5 Manually recreate the old port profiles from your previous configuration.

For more information, see the following procedures in the Cisco Nexus 1000V Getting Started Guide, Release 4.0(4)SV1(3).

Configuring the System Port Profile for VSM-VEM Communication

Configuring the Uplink Port Profile for VM Traffic

Configuring the Data Port Profile for VM Traffic


Note If you do not manually recreate the port profiles, then all port groups on the vCenter Server are removed when the VSM connects.


Step 6 On the VSM, restore the configuration for the vCenter server connection.

Example:
n1000v# config t
n1000v (config)# svs connection VC 
n1000v(config-svs-conn#) protocol vmware-vim
n1000v(config-svs-conn#) remote ip address 192.168.0.1 
n1000v(config-svs-conn#) vmware dvs datacenter-name Hamilton-DC 

Step 7 Connect to vCenter Server.

Example:
n1000v(config-svs-conn#) connect

You can now use the old DVS or remove it.


Problems Related to VSM and vCenter Server Connectivity

Symptom
Solution

The vCenter Server connection seems to succeed, but does not.

Make sure that the domain ID is configured correctly.

The svs connection command fails.

1. Make sure you have configured all parameters for the svs connection command.

2. Make sure you can ping the vCenter Server IP address.

3. Make sure that the proxy.xml file is correct for both the IP address and length.

4. Restart the vCenter Server.

The connection fails after an ESX reboot

"Connection Failure After ESX Reboot" procedure

The host does not show up in the Add host to DVS screen.

Make sure that the Host is installed with VMware Enterprise plus

license containing the Distributed Virtual Switch feature.

Add host to DVS returns an error.

Confirm that the VEM software is installed on the ESX server,

The server name column of the show module command output shows the IP address.

The server name shows the host-name or IP address, whichever was used to add the host to the DVS on the vCenter Server.


Connection Failure After ESX Reboot

If you lose connectivity between the VSM and VEM following a reboot of the ESX server host, it may be caused by a reset of the MTU to its default on the host.

This section includes the following topics:

Information About Changing the MTU

Verifying that an MTU Mismatch Caused a Connection Failure

Restoring Network Connectivity to the VEM

Information About Changing the MTU

If you use an MTU other than 1500 (the default) for a physical NIC attached to the Cisco Nexus 1000V, then reboots of the ESX can revert this setting to the default setting of 1500. This results in a mismatch with the MTU setting on one of the following and a potential connection failure of the VSM and VEM.

VMware kernel NIC

Upstream switchport of the VEM uplink

For example, in networks where you use jumbo frames, you may manually configure an MTU of other than 1500. If the ESX reboots during a power cycle, the MTU of the physical NIC reverts to the default of 1500 but the MTU setting on the VMware kernel NIC or upstream switchport does not.

To verify that an MTU setting mismatch has caused a connection failure, see the "Verifying that an MTU Mismatch Caused a Connection Failure" procedure.

To restore connectivity that was lost due to an MTU mismatch, see the "Restoring Network Connectivity to the VEM" procedure.

To prevent a loss of connectivity between the VSM and VEM, and preserve a a physical NIC MTU setting of other than 1500 (the default) across reboots of the ESX, you must configure a system MTU in the system uplink port profile.

To configure the system mtu on the system uplink port profile, see the following procedure in the Cisco Nexus 1000V Port Profile Configuration Guide, Release 4.0(4)SV1(3)

Creating a System Port Profile

Verifying that an MTU Mismatch Caused a Connection Failure

You can use this procedure to verify that the cause of a VEM connection failure is an MTU mismatch.

BEFORE YOU BEGIN

Before beginning this procedure, you should know or do the following:

You are logged in to the VEM.

You are logged in to the CLI of the upstream switch.


Step 1 On the upstream switch, use the following command to verify the MTU on the interface connected to the VEM uplink.

show interface

show port

Example:
Console> show port jumbo
Jumbo frames MTU size is 9216 bytes.
Jumbo frames enabled on port(s) 6/1-2,7/1-8.
Console> 

Example:
switch# show interface vlan 10 
Vlan10 is up, line protocol is up
  Hardware is EtherSVI, address is  0005.9b78.6e7c
  MTU 8000 bytes, BW 1000000 Kbit, DLY 10 usec,
switch#

Step 2 On the ESX, verify the MTU setting for the physical NIC attached to the Cisco Nexus 1000V.

vem status

~ # vem status
VEM modules are loaded
 
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks   
vSwitch0       64          3           64                1500    vmnic0    
DVS Name       Num Ports   Used Ports  Configured Ports  Uplinks   
n1000v         256         9           256               vmnic1 VEM Agent is running

Step 3 On the ESX, identify the VMware kernel NICs that are attached to the Cisco Nexus 1000V.

vemcmd show port


Step 4 On the ESX, verify the MTU settings for the VMware kernel NICs identified in Step 3,

esxcfg-vmknic -l

Step 5 Do one of the following:

If the MTU settings do not match, go to the "Restoring Network Connectivity to the VEM" procedure.

If the MTU settings match, then this is not the cause of the connection failure.


Restoring Network Connectivity to the VEM

You can use this procedure to restore network connectivity with a VEM that was lost due to an MTU mismatch between the physical NIC and the VMware kernel NIC.

BEFORE YOU BEGIN

Before beginning this procedure, you should know or do the following:

You are logged in to the ESX host.


Step 1 Use the following commands to reset the MTU and restore connectivitythat was lost due to an MTU mismatch with the physical NIC attached to the Cisco Nexus 1000V:

vemcmd show port port-LTL-number

vemcmd set mtu size ltl port-LTL-number

The MTU is reset on the ESX and connectivity with the VEM is restored.

Example:
~ # vemcmd show port 48
LTL    IfIndex   Vlan    Bndl  SG_ID Pinned_SGID  Type  Admin State  CBL Mode   
Name  . . .   
17   1a030100      1 T   304      1          32  PHYS     UP    UP    1  Trunk vmnic1
~# vemcmd set mtu 9000 ltl 17

Step 2 On the VSM, update the MTU setting using one of the following documents:

To update the system port profile for system uplinks:
Cisco Nexus 1000V Port Profile Configuration Guide, Release 4.0(4)SV1(3)

To update the interface configuration for non-system uplinks:
Cisco Nexus 1000V Interface Configuration Guide, Release 4.0(4)SV1(3)


Displaying VC Event History

Example 18-1 shows the show vms internal event-history errors command that is useful for examining VC errors in detail. It shows whether an error is caused by a VSM (client) or the server.

Example 18-1 show vms internal event-history error Command

n1000v# show vms internal event-history errors

Event:E_DEBUG, length:239, at 758116 usecs after Tue Feb  3 18:21:58 2009
    [102] convert_soap_fault_to_err(1179): SOAP 1.1 fault: "":ServerFaultCode [VMWARE-VIM] 
A DVS n1000v with spec.name as n1000v already exists, cannot create DVS n1000v. A 
specified parameter was not correct.spec.name

Event:E_DEBUG, length:142, at 824006 usecs after Tue Feb  3 18:18:30 2009
    [102] convert_soap_fault_to_err(1179): SOAP 1.1 fault: SOAP-ENV:Client [VMWARE-VIM] 
Operation could not be completed due to connection failure.

Event:E_DEBUG, length:134, at 468208 usecs after Tue Feb  3 18:15:37 2009
    [102] convert_soap_fault_to_err(1179): SOAP 1.1 fault: "":ServerFaultCode [VMWARE-VIM] 
Extension key was not registered before its use.

VSM Creation

Symptom
Possible Causes
Solution

The VSM VM is stuck at the boot prompt.

Make sure that you have three e1000 NICs.

The VSM VM cannot ping itself.

Configure the mgmt0 interface.

The VSM VM can ping itself, but not the gateway.

Make sure the NIC order is correct: control, mgmt, inband.

The VSM VM can ping the gateway, but not the outside subnet.

Configure vrf context management.


Port Profiles

When creating a port profile, use the following commands to create the corresponding port groups on the vCenter Server:

vmware port-group

state enabled

Profiles that have the system VLAN configuration allow the VEM to communicate with the VSM.

Make sure that the system port-profile is defined with the right system VLANS.

Use the show port-profile and show port-profile usage commands to collect basic required information.

Problems with Port Profiles

Symptom
Possible Causes
Solution

You receive an error message "Possible failure in communication with vCenter Server."

The VSM is not connected to the vCenter Server.

Issue the svs connection vc command to connect to the vCenter Server.

The port group name is not unique.

Port group names must be unique within a vCenter Server Datacenter.

Port profile or port groups do not appear on the vCenter Server.

Make sure you have issued the vmware port-group command and state enable command.


Problems with Hosts

Symptom
Solution

You receive an error message, DVS Operation failed for one or more members."

Issue the vem status -v command to verify if the VEM is running on the host.

Issue the vem unload command to unload the VEM.

In the vSphere Client, remove the stale DVS:

1. Go to the Host tab Networking->Configuration->
Distributed Virtual Switch

2. Click Remove.

The host is visible on the vCenter Server, but not the VSM.

Issue the vemcmd show trunk command to verify that there is an uplink carrying the control VLAN. The profile applied to the uplink must be a system profile with a control VLAN as a system VLAN.

Verify the control VLAN in the upstream switch port and the path to the VSM VM. Make sure that one uplink at most carries the control VLAN, or that all uplinks and upstream ports carrying the control VLAN are in port channels.

A module flap occurs.

The VSM may be overloaded. Make sure that you have 1 GB of memory and CPU shares for the VSM VM on the vCenter Server.


Problems with VM Traffic

When troubleshooting problems with intra-host VM traffic, follow these guidelines:

Make sure that at least one of the VMware virtual NICs is on the correct DVS port group and is connected.

If the VMware virtual NIC is down, determine if there is a conflict between the MAC address configured in the OS and the MAC address assigned by VMware. You can see the assigned MAC addresses in the vmx file.

When troubleshooting problems with inter-host VM traffic, follow these guidelines:

Determine if there is exactly one uplink sharing a VLAN with the VMware virtual NIC. If there is more than one, they must be in a port channel.

Ping a SVI on the upstream switch using the show intX counters command.

VEM Troubleshooting Commands

Use the following commands to display VEM information:

vemlog - displays and controls VEM kernel logs

vemcmd - displays configuration and status information

vem-support all - collects support information

vem status- collects status information

vem version- collects version information

vemlog show last number-of-entries - displays the circular buffer

Example 18-2 vemlog show last Command

[root@esx-cos1 ~]# vemlog show last 5
Timestamp                  Entry CPU  Mod Lv         Message
Oct 13 13:15:52.615416      1095   1    1  4 Warning vssnet_port_pg_data_ ...
Oct 13 13:15:52.620028      1096   1    1  4 Warning vssnet_port_pg_data_ ...
Oct 13 13:15:52.630377      1097   1    1  4 Warning svs_switch_state ...
Oct 13 13:15:52.633201      1098   1    1  8    Info vssnet new switch ...
Oct 13 13:16:24.990236      1099   1    0  0         Suspending log

vemlog show info - displays information about entries in the log

Example 18-3 vemcmd show info Command

[root@esx-cos1 ~]# vemlog show info
          Enabled: Yes
    Total Entries: 1092
  Wrapped Entries: 0
     Lost Entries: 0
  Skipped Entries: 0
Available Entries: 6898
 Stop After Entry: Not Specified

vemcmd help - displays the type of information you can display

Example 18-4 vemcmd help Command

[root@esx-cos1 ~]# vemcmd help
show card              Show the card's global info
show vlan [vlan]       Show the VLAN/BD table
show bd [bd]           Show the VLAN/BD table
show l2 <bd-number>    Show the L2 table for a given BD/VLAN
show l2 all            Show the L2 table
show port [priv|vsm]   Show the port table
show pc                Show the port channel table
show portmac           Show the port table MAC entries
show trunk [priv|vsm]  Show the trunk ports in the port table
show stats             Show port stats

VEM Log Commands

Use the following commands to control the vemlog:

vemlog stop - stops the log

vemlog clear - clear s the log

vemlog start number-of-entries - starts the log and stops it after the specified number of entries

vemlog stop number-of-entries - stops the log after the next specified number of entries

vemlog resume - starts the log, but does not clear the stop value

Error Messages

On the vSphere Client, you can see error messages under the recent tasks tab. You can find detailed description of the error under the Tasks and Events tab. The same messages are also propagated to the VSM.

Table 18-1 lists error messages that you might see on the VSM.

Table 18-1 Error Messages on the VSM

Error
Description

ERROR: [VMWARE-VIM] Extension key was not registered before its use

This error indicates that VSM extension key is not registered.

ERROR: [VMWARE-VIM] A DVS n1000v with spec.name as n1000v already exists, cannot create DVS n1000v. A specified parameter was not correct. spec.name.

This error is displayed after you enter the first connect command, and indicates that a DVS already exists with the same name.

ERROR: [VMWARE-VIM] A DVS n1000v with spec.extensionKey as Cisco_Nexus_1000V_2055343757 already exists, cannot create DVS new-n1000v. A specified parameter was not correct. spec.extensionKey

This error is displayed when the VSM tries to create a different DVS after changing the switch name.

ERROR: [VMWARE-VIM] A DVS n1000v with name as n1000v already exists, cannot reconfigure DVS test. A specified parameter was not correct. Spec.name

This error indicates that a DVS with the same name already exists.

Warning: Operation succeeded locally but update failed on vCenter server.[VMWARE-VIM] DVPortgroup test port 0 is in use. The resource vim.dvs.DistributedVirtualPort 0 is in use.

This warning is displayed when the VSM tries to delete the port profile if the VSM is not aware of the nics attached to the port groups.