Using vemlog to debug Nexus 1000v problems

Introduction

The Virtual Ethernet Module (VEM) portion of the Nexus 1000v has built-in debug commands that you can use to assist in troubleshooting. These vemlog commands allow you insight to what certain processes of the VEM are doing, what commands they are sending out, and what sort of response they are receiving. For example, you might use them in these situations:

• A LACP port-channel is not coming up between the Nexus 1000v and the upstream switch. You would use a combination of “vemlog” on the host and “debug” on the upstream switch to narrow down the problem.

• QoS is not working correctly and you want to verify that the VEM is setting the values correctly.

• You want to see the changes to the VSM-VEM communication.

The most common use case for running these commands is debugging LACP. Using the debug module "sflacp" and "debug lacp" on the upstream switch you can detect which device is malfunctioning, or if the link between the two devices is dropping the LACP BPDUs.

The default levels usually have enough information so the log can be collected and analyzed without turning on additional debugs.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software and hardware versions.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Running vemlog commands

If you have any questions about the syntax of these commands you can type “vemlog” in the ESXi CLI and receive a help output. There is no tab complete or ‘?’ for these commands. You must fully type them out or they will not work correctly. In order to run vemlog commands you need one of the following:

• SSH/local console access to the ESXi host in question. SSH access is preferable because you can easily scroll back up.

• SSH/local console access to the VSM and the module has to be active in the VSM. Once again SSH access is preferable. If you are using this method you will need to preface every command with module vem # execute, where # is the module number of the VEM.

Vemlog debugging levels

Vemlog on the VEM is similar to the syslog on a normal switch. Like a normal switch, there are severity levels:

e – Error 
n – Notification
w – Warning
i – Information
d – Debug
p – Print
t – Temporary

This is ranked from most severe to least (with the less severe messages outputting far more information). By default the vemlog is set to ENW (Error, Notification, Warning).

Step 1: Clearing Current Capture settings

For troubleshooting a particular issue it is valuable to disable the default captures.

vemlog stop
vemlog clear
vemlog debug all none

These commands will stop any current vemlog commands that are happening, and will reset the vemlog to blank capture settings. You can verify this with the following command:

As you see here, the vemlog show debug command will display the current debug settings. The default for all modules for “Printing” is 0, or not printing.

Step 2: Setting the vemlog capture parameters

Now that you have a clean slate to start with, it is time to set the cookies to capture. For this example, the general port logs are captured:

The second command there has this syntax:

vemlog debug [module|all] [-][all|none|default|e|w|n|i|d|p|t]

Where “debug_module” is the vemlog package you want to capture. Usually “debug” is enough for troubleshooting.

If you add the [-] to the command that debug level will be removed for that module. This screenshot has examples of this:

You can view all the available debug modules with a vemlog show debug command.

Once you have specified which packages to monitor, then start the vemlog with a vemlog start command:

Step 3: Viewing the output

Once you are ready to view the log, issue a vemlog show all command. This prints all the log entries to the console (or SSH session). If you are directly connected to the host you can also send this to a file so that you can copy it off and analyze it later:

If you move this file to the /vmfs/volumes/[datastore_name] directory you can move it off the host using the datastore browser in ESXi.

The vemlog is also stored in the vmkernel log.

Step 4: Restoring vemlog to normal logging levels

~ # vemlog stop
Suspended log
~ # vemlog clear
Cleared log
~ # vemlog debug all default
~ # vemlog start
Started log

These commands will restore the vemlog to the standard configuration. This is important to ensure that debugging information is captured properly for future problems.

Because the vemlog is stored both in the vmkernel logs and the vemlog's buffer, a particularly chatty debug can flood both of these with unnecessary information, especially if they are left enabled.

Related Information

• Technical Support & Documentation - Cisco Systems

Revision History