Cisco Nexus 1000V Troubleshooting Guide, Release 5.2(1)SV3(4.1)

Guidelines for Troubleshooting Modules

Follow these guidelines when troubleshooting a module controlled by the VSM.

• You must have a VSM VM and a VEM up and running.
• Make sure that you are running compatible versions of vCenter Server and VSM.

For more information, see the Cisco Nexus 1000V Compatibility Information.

• To verify network connectivity between the VSM and vCenter Server, ping the IP address of vCenter Server. If you are using a domain name service (DNS) name, use the DNS name in the ping. If a ping to vCenter Server fails, check to see if you can ping the gateway. Otherwise, check the mgmt0 interface configuration settings.
• In the Cisco Nexus1000V Distributed Virtual Switch (DVS), we support only one vmknic with "capability l3control". If a second vmknic is added with the same capability then the host connected as VEM modules on VSM in L3 mode will go offline. To recover from this scenario, we need to remove both the vmknics from Cisco Nexus1000V DVS or migrate them back to vSwitch/VMware DVS. Once you migrate or removed, you can recreate one vmknic on Cisco Nexus1000V DVS or migrate one of the vmknic from vswitch/VMware DVS back to Cisco Nexus1000V DVS.
• Make sure that the firewall settings are OFF on the vCenter Server. If you want the firewall settings, and check to see if these ports are open:

– Port 80

– Port 443

• If you see the following error, verify that the VSM extension was created from vCenter Server:

– ERROR: [VMware vCenter Server 4.0.0 build-150489]
Extension key was not registered before its use

To verity that the extension or plugin was created, see the “Finding the Extension Key on the Cisco Nexus 1000V” section.

For more information about extension keys or plugins, see the “Managing Extension Keys” section.

• If you see the following error, see the “Checking the vCenter Server Configuration” section.

– ERROR: Datacenter not found

• For a list of terms used with the Cisco Nexus 1000V, see the Cisco Nexus 1000V Getting Started Guide.

Flowchart for Troubleshooting Modules

Use the following flowchart to troubleshoot modules.

Flowchart: Troubleshooting Modules

Verifying the VSM Is Connected to vCenter Server

You can use the following procedure to verify that the VSM is connected to vCenter Server.

Step 1 Verify the connection between the VSM and vCenter Server.

show svs connections

The output should indicate that the operational status is Connected.

Step 2 Do one of the following:

• If the status is Connected, return to the “Flowchart: Troubleshooting Modules” section.
• If not, continue with the next step.

Step 3 Connect to vCenter Server.

config t

svs connection connection_name

connect

Step 4 Do one of the following:

• If you see an error message about the Extension key, continue with the next step.
• If not, go to Step 6.

Step 5 Do the following and then go to Step 6.

• Unregister the extension key using the “Unregistering the Extension Key in the vCenter Server” section.
• Install a new extension key using the “Creating a Cisco Nexus 1000V Plug-In on the vCenter Server” procedure in the Cisco Nexus 1000V Getting Started Guide.

Step 6 Verify the connection between the VSM and vCenter Server.

show svs connections

The output should indicate that the operational status is Connected.

Step 7 Do one of the following:

• If the status is Connected, you have completed this procedure.
• If not, return to the “Flowchart: Troubleshooting Modules” section.

Verifying the VSM Is Configured Correctly

This section includes the following topics:

• Verifying the Domain Configuration
• Verifying the System Port Profile Configuration
• Verifying the Control and Packet VLAN Configuration

Verifying the Domain Configuration

You can verify the domain configuration.

BEFORE YOU BEGIN

• Log in to the CLI in EXEC mode.
• Verify that the output of the show svs domain command indicates the following:

– The presence of a control VLAN and a packet VLAN.

– The domain configuration was successfully pushed to VC.

Step 1 On the VSM, verify the domain configuration.

show svs domain

Verifying the System Port Profile Configuration

You can verify the port profile configuration.

BEFORE YOU BEGIN

• Log in to the CLI in EXEC mode.
• Verify that the output of the show port-profile name command indicates the following:

– The control and packet VLANs are assigned.

– The port profile is enabled.

– If you have configured a non-default system MTU setting, check that it is the correct size.

Step 1 On the VSM, verify the system port profile configuration.

show port-profile name system-port-profile-name

Verifying the Control and Packet VLAN Configuration

You can verify that the control and packet VLANs are configured on the VSM.

Note The procedure documented is for troubleshooting VSM and VEM connectivity with Layer 2 mode.

BEFORE YOU BEGIN

• Log in to the CLI in EXEC mode.
• Check that the output of the show running-config command shows control and packet VLAN ID numbers among the VLANs configured,

Step 1 On the VSM, verify that the control and packet VLANs are present.

Step 2 Find the AIPC MAC address of the VSM on the VSM.

Step 3 Find the DPA MAC address of the VEM on the ESX host.

Step 4 Check the upstream switches for these MAC addresses in the correct VLANs.

Checking the vCenter Server Configuration

You can verify the configuration on vCenter Server.

Step 1 Confirm that the host is added to the data center and the Cisco Nexus 1000V DVS in that data center.

Step 2 Confirm that at least one pnic of the host is added to the DVS, and that pnic is assigned to the system-uplink profile.

Step 3 Confirm that the three VSM vnics are assigned to the port groups that contain the control VLAN, packet VLAN, and management network.

Checking Network Connectivity Between the VSM and the VEM

You can verify Layer 2 network connectivity between the VSM and the VEM.

Step 1 On the VSM, find its MAC address.

show svs neighbors

The VSM MAC address displays as the AIPC Interface MAC.

The user VEM Agent MAC address of the host displays as the Src MAC.

Step 2 Do one of the following:

• If the output of the show svs neighbors command in Step 1 does not display the VEM MAC address, there is a problem with connectivity between the server hosting the VSM and the upstream switch. Recheck the VSM configuration and vCenter Server configuration.
• Otherwise, continue with the next step.

Step 3 On the VEM, run the vem-health script using the VSM MAC address that you found in Step 1.

Note If the vem-health script is not in the PATH, you can find it under /usr/lib/ext/cisco/nexus/vem*/sbin/.

vem-health check vsm_ mac_address

The vem-health script output shows the cause of the connectivity problem and recommends the next steps for troubleshooting the problem.

Step 4 Do one of the following:

• If the VEM health check in Step 3 indicates a problem with connectivity to the upstream switch. continue with the next step.
• Otherwise, go to Step 7.

Step 5 On the upstream switch, display the MAC address table to verify the network configuration.

Step 6 Do one of the following:

• If the output from Step 5 does not display the MAC address of the VSM, then there is a problem with connectivity between the server hosting the VSM and the upstream switch. Recheck the VSM configuration and vCenter Server configuration.
• Otherwise, continue with the next step.

Step 7 On the VSM, enter the following commands to verify that the VSM MAC appears in the control and packet VLANs.

a. config t

b. module vem module_number execute vemcmd show l2 control_vlan_id

c. module vem module_number execute vemcmd show l2 packet_vlan_id

Step 8 Do one of the following:

• If the MAC address of the VSM does not appear in the output of Step 7, check the VEM configuration as explained in “Checking the VEM Configuration” section.
• Otherwise, you have completed this procedure.

Recovering Management and Control Connectivity of a Host When a VSM is Running on a VEM

When the VSM is running on a VEM that it manages, but the VSM ports are not configured with system port profiles, the control and management connectivity of the VSM can be lost after a host reboot or similar event. To recover from the loss, you can run the VEM connect script locally in the ESX host where the VEM is running, and then go to the VSM and configure the system VLANs in the port profile used for management.

Using the VEM Connect Script

The VEM connect script sets a given VLAN as a system VLAN on the VTEP that has the given IP address and also sets the VLAN on all the required uplinks.

If no uplink is carrying this VLAN, you also need to specify the uplink (vmnicN) on which this VLAN needs to be applied. The uplink can be a single port or a port-channel member. If it is the latter, then the script applies the VLANs as a system VLAN to all member uplinks of that port channel.

vem-connect -i ip_address -v vlan [ -p vmnicN ]

The -p parameter to the script is optional. If you run the script without the -p parameter, it tries to locate an uplink that carries this VLAN. If no such uplink exists, it reports this as an error. You need to specify the -p parameter and rerun the script.

You can recover management and control connectivity of a host when a VSM is running on a VEM.

SUMMARY

Step 1 Display the VEM ports:

vemcmd show port

Note The output *F/B The port is blocked on some of the VLANs means that the trunk is not forwarding all VLANs. This situation might be normal depending on the port profile allowed VLAN list. Compare the output of the vemcmd show port vlans command against the list of allowed VLANs in the trunk port profile. If the lists match, all of the expected VLANs are forwarding and the Cisco Nexus 1000V is blocking nonallowed VLANs.

Step 2 Display details about the system VLANs.

vemcmd show port vlans system

Example :

Step 3 Recover connectivity:

vem-connect -i ip_address -v vlan [ -p vmnicN ]

Step 4 Confirm management connectivity:

vemcmd show port vlans system

Example:

Checking the VEM Configuration

You can verify that the ESX host received the VEM configuration and setup.

Step 1 On the ESX host, confirm that the VEM Agent is running and that the correct host uplinks are added to the DVS.

vem status

Step 2 Restore connectivity that is lost due to an incorrect MTU value on an uplink.

a. vemcmd show port port-LTL-number

b. vemcmd set mtu value ltl port-LTL-number

Note Use these vemcmds only as a recovery measure and then update the MTU value in the port profile configuration for system uplinks or in the interface configuration for nonsystem uplinks.

Step 3 Verify that the domain ID, control VLANs, and packet VLANs are configured correctly on the host.

vemcmd show card

Step 4 Verify that the ports of the host added to the DVS are listed and that the ports are correctly configured as access or trunk on the host.

The last line of output indicates that vmnic1 should be in Trunk mode, with the CBL value of 1. The CBL value of the native VLAN does not have to be 1. It will be 0 if it is not allowed, or 1 if it is VLAN 1 and not allowed. This issue is not a problem unless the native VLAN is the Control VLAN.The Admin state and Port state should be UP.

Step 5 Verify that the vmnic port that is supposed to carry the control VLAN and packet VLAN is present.

Step 6 Verify the following:

• The control and packet VLANs are shown in the command output, indicating that the DV port groups are successfully pushed from vCenter Server to the host.
• The correct physical trunk port vmnic is used.

vemcmd show trunk

At least one physical uplink must be carrying the control and packet VLANs. If more than one uplink is carrying the control and packet VLANs, the uplinks must be in a port channel profile. The port channel itself would not be visible because the VEM is not yet added to the VSM.

Step 7 Restore connectivity that is lost due to incorrect port and system VLAN settings.

vemcmd show port port-ltl-number

vemcmd set system-vlan vlan_id ltl port-ltl-number

Note Use these vemcmds only as a recovery measure and then update the port profile configuration with correct system VLANs.

Collecting Logs

After you have verified network connectivity between the VEM and the VSM, you can use the following procedure to collect log files to help identify the problem.

Step 1 On the VEM, verify its UUID.

Step 2 On the VSM, verify the module number to which the corresponding UUID entry is mapped.

Step 3 Using the module number from Step 2, collect the output of the following commands:

• show system internal vem_mgr event-history module 13
• show module internal event-history module 13
• show system internal im event-history module 13
• show system internal vmm event-history module 13
• show system internal ethpm event-history module 13

Note If you need to contact Cisco TAC for assistance in resolving an issue, you will need the output of the commands listed in Step 3.