Cisco Nexus 1000V Troubleshooting Guide, Release 4.2(1) SV1(4)
VSM and VEM Modules
Downloads: This chapterpdf (PDF - 260.0KB) The complete bookPDF (PDF - 5.9MB) | Feedback

VSM and VEM Modules

Table Of Contents

VSM and VEM Modules

Information About Modules

Troubleshooting a Module Not Coming Up on the VSM

Guidelines for Troubleshooting Modules

Flow Chart for Troubleshooting Modules

Problems with the VSM

Verifying the VSM Is Connected to the vCenter Server

Verifying the VSM Is Configured Correctly

Verifying the Domain Configuration

Verifying the System Port Profile Configuration

Verifying the Control and Packet VLAN Configuration

Checking the vCenter Server Configuration

Checking Network Connectivity Between the VSM and the VEM

Checking the VEM Configuration

Collecting Logs

VSM and VEM Troubleshooting Commands


VSM and VEM Modules


This chapter describes how to identify and resolve problems that relate to modules and includes the following sections:

Information About Modules

Troubleshooting a Module Not Coming Up on the VSM

Problems with the VSM

VSM and VEM Troubleshooting Commands

Information About Modules

Cisco Nexus 1000V manages a data center defined by a virtualCenter. Each server in the data center is represented as a module in Cisco Nexus 1000V and can be managed as if it were a module in a physical Cisco switch.

The Cisco Nexus 1000V implementation has 2 parts:

Virtual supervisor module (VSM) - This is the control software of the Cisco Nexus 1000V distributed virtual switch. It runs on a virtual machine (VM) and is based on NX-OS software.

Virtual Ethernet module (VEM) - This is the part of Cisco Nexus 1000V that actually switches data traffic. It runs on a VMware ESX 4.0 host. Several VEMs are controlled by one VSM. All the VEMs that form a switch domain should be in the same virtual Data Center as defined by VMware VirtualCenter.

Troubleshooting a Module Not Coming Up on the VSM

This section describes the process you can use when a module does not come up on the VSM. This section includes the following topics:

Guidelines for Troubleshooting Modules

Flow Chart for Troubleshooting Modules

Verifying the VSM Is Connected to the vCenter Server

Verifying the VSM Is Configured Correctly

Checking the vCenter Server Configuration

Checking Network Connectivity Between the VSM and the VEM

Checking the VEM Configuration

Collecting Logs

Guidelines for Troubleshooting Modules

Follow these guidelines when troubleshooting a module controlled by the VSM.

You must have a VSM VM and a VEM up and running.

Make sure you are running compatible versions of vCenter Server and VSM.

For more information, see the Cisco Nexus 1000V Compatibility Information, Release 4.2(1)SV1(4).

To verify network connectivity between the VSM and vCenter Server, ping the IP address of the vCenter Server. If you are using a domain name service (DNS) name, use the DNS name in the ping. If a ping to the vCenter Server fails, check to see if you can ping the gateway. Otherwise, check the mgmt0 interface configuration settings.

Make sure the firewall settings are OFF on the vCenter Server. If you want the firewall settings, then check to see if these ports are open.

Port 80

Port 443

If you see the following error, verify that the VSM extension was created from vCenter Server.

ERROR: [VMware vCenter Server 4.0.0 build-150489]
Extension key was not registered before its use

To verity that the extension or plugin was created, see the "Finding the Extension Key Tied to a Specific DVS" procedure.

For more information about extension keys or plugins, see the "Managing Extension Keys" section.

If you see the following error, see the "Checking the vCenter Server Configuration" procedure.

ERROR: Datacenter not found

For a list of terms used with Cisco Nexus 1000V, see the Cisco Nexus 1000V Getting Started Guide, Release 4.2(1)SV1(4).

Flow Chart for Troubleshooting Modules

Use the following flowchart to troubleshoot modules.

Flowchart: Troubleshooting Modules

Problems with the VSM

The following are symptoms, possible causes, and solutions for problems with the VSM.

Table 5-1 Problems with the VSM 

Symptom
Possible Causes
Solution

You see the following error on the VSM:

ERROR: [VMware vCenter 
Server 4.0.0 
build-150489]  
Extension key was not 
registered before its use

An extension or plug-in was not created for the VSM.

1. Verify the extension or plug-in was created.

"Finding the Extension Key Tied to a Specific DVS" procedure

2. If the plug-in is not found, then create one using the following procedure in the Cisco Nexus 1000V Getting Started Guide, Release 4.2(1)SV1(4):

Creating a Cisco Nexus 1000V Plug-In on the vCenter Server

Following a reboot of the VSM, the system stops functioning in one of the following states and does not recover on its own. Attempts to debug fail.

After boot, VSM stopped at "Loader Loading."

 

Corrupt boot menu file.

1. Boot the VSM from the CD ROM.

2. From the CD Boot menu,
choose Option 3, Install Nexus1000v only if the disk unformatted and bringup new image.

3. Do one of the following:

If you have a saved backup copy of your configuration file, restore the configuration on the VSM.

copy source filesystem: filename system:running-config

If not, reconfigure the VSM using the following section in the Cisco Nexus 1000V Getting Started Guide, Release 4.2(1)SV1(4):

Setting Up the Software

If using VSM serial port, after boot, VSM in loader prompt.

Corrupt VSM kickstart image.

1. Boot the VSM from the CD ROM with the same software version currently installed.

2. From the CD Boot menu,
choose Option 3, Install Nexus1000v only if the disk unformatted and bringup new image.

Follow the VSM installation procedure.

3. Verify the boot variables are configured properly:

show boot

4. If not, set the boot variables used to boot the VSM:

boot system bootflash:system-boot-variable-name

boot kickstart bootflash:kickstart-boot-variable-name

5. Save the configuration.

copy running-config startup-config

6. Reload.

reload

If using VSM serial port, after boot, VSM in loader prompt. (continued)

Boot variables are not set.

1. Boot the VSM from the CD ROM.

2. From the CD Boot menu,
choose Option 3, Install Nexus1000v only if the disk unformatted and bringup new image.

3. Verify the boot variables are configured properly:

show boot

4. If not, set the boot variables used to boot the VSM:

boot system bootflash:system-boot-variable-name

boot kickstart bootflash:kickstart-boot-variable-name

5. Save the configuration.

copy running-config startup-config

After boot, VSM in boot prompt.

Corrupt VSM system image.

1. Boot the VSM from the CD ROM.

2. From the CD Boot menu,
choose Option 1, Install Nexus1000v and bring up new image.

3. Follow the VSM installation procedure.

After boot, VSM re-configured.

Startup configuration is deleted.

Do one of the following:

If you have a saved backup copy of your configuration file, restore the configuration on the VSM.

copy source filesystem: filename system:running-config

If not, reconfigure the VSM using the following section in the Cisco Nexus 1000V Getting Started Guide, Release 4.2(1)SV1(4):

Setting Up the Software

Standby VSM reloads unexpectedly and it does not come back up as HA Standby.

Control VLAN or control interface down

1. Verify the standby VSM has booted up.

show module

2. Verify active communication between VSMs.

show system internal redundancy trace

3. If the output indicates either of the following, then troubleshoot the communication over the control VLAN.

DEGRADED_MODE
PRI_HB_STOPPED

Check for the following:

Links down between VSMs.

Broadcast traffic not forwarded between VSMs.

Control VLAN not properly configured.

Port profiles not defined properly, or wrong port profiles chosen for VSM or host where they reside.

After boot, secondaryVSM reboots continuously.

Active and standby VSMs fail to synchronize, remote storage is used, and the active has re-mounted its file system as read only.

1. From active VSM, check for a system message indicating the file system was remounted.

show logging

2. If you see the following message in the output, then reload the active VSM.

%KERN-2-SYSTEM_MSG: Remounting filesystem read-only - 
kernel

Active and standby VSMs fail to synchronize, and a process crashes at the standby.

1. From active VSM, look for a process crash.

show cores

See Example 5-3.

If the output indicates a process crash, then see the "Before Contacting Technical Support" section to open a support case.


Verifying the VSM Is Connected to the vCenter Server

You can use the following procedure to verify that the VSM is connected to the vCenter Server.


Step 1 Verify the connection between the VSM and vCenter Server.

show svs connections

The output should indicate that the operational status is Connected.

Example:
n1000v# show svs connections
connection vc:
    ip address: 172.23.231.223
    protocol: vmware-vim https
    certificate: user-installed
    datacenter name: hamilton-dc
    DVS uuid: 92 7a 14 50 05 11 15 9c-1a b0 f2 d4 8a d7 6e 6c
    config status: Disabled
    operational status: Disconnected
 
   

Step 2 Do one of the following:

If the status is Connected, then return to the Flowchart: Troubleshooting Modules.

If not, then continue with the next step.

Step 3 Connect to the vCenter Server.

config t

svs connection connection_name

connect

Example:
n1000v# conf t
n1000v(config)# svs connection HamiltonDC
n1000v(config-svs-conn)# connect
 
   
Example:
n1000v# conf t
n1000v(config)# svs connection HamiltonDC
n1000v(config-svs-conn)# connect
ERROR: [VMWARE-VIM] Extension key was not registered before its use. 
 
   

Step 4 Do one of the following:

If you see an error message about the Extension key, continue with the next stepTable 5-1.

If not, go to Step 6.

Step 5 Do the following and then go to Step 6.

Unregister the extension key using the "Unregister the Extension Key in the vCenter Server" procedure.

Install a new extension key using the following procedure in the Cisco Nexus 1000V Getting Started Guide, Release 4.2(1)SV1(4).

Creating a Cisco Nexus 1000V Plug-In on the vCenter Server

Step 6 Verify the connection between the VSM and vCenter Server.

show svs connections

The output should indicate that the operational status is Connected.

Example:
n1000v# show svs connections
connection vc:
    ip address: 172.23.231.223
    protocol: vmware-vim https
    certificate: user-installed
    datacenter name: hamilton-dc
    DVS uuid: 92 7a 14 50 05 11 15 9c-1a b0 f2 d4 8a d7 6e 6c
    config status: Disabled
    operational status: Disconnected
 
   

Step 7 Do one of the following:

If the status is Connected, then you have completed this procedure.

If not, then return to the Flowchart: Troubleshooting Modules.


Verifying the VSM Is Configured Correctly

This section includes the following procedures to verify the VSM configuration.

Verifying the Domain Configuration

Verifying the System Port Profile Configuration

Verifying the Control and Packet VLAN Configuration

Verifying the Domain Configuration

You can use the following procedure to verify the domain configuration.

BEFORE YOU BEGIN

Before beginning this procedure, you should know or do the following:

You are logged in to the CLI in EXEC mode.

The output of the show svs domain command should indicate the following:

The presence of a control VLAN and a packet VLAN.

The domain configuration was successfully pushed to VC.


Step 1 On the VSM, verify the domain configuration.

show svs domain

Example:
n1000v# show svs domain
SVS domain config: 
  Domain id:    682 
  Control vlan: 3002
  Packet vlan:  3003
  L2/L3 Control VLAN mode: L2 
  L2/L3 Control VLAN interface: mgmt0
  Status: Config push to VC successful

Verifying the System Port Profile Configuration

You can use the following procedure to verify the port profile configuration.

BEFORE YOU BEGIN

Before beginning this procedure, you should know or do the following:

You are logged in to the CLI in EXEC mode.

The output of the show port-profile name command should indicate the following:

The control and packet VLANs are assigned.

The port profile is enabled.

If you have configured a non-default system mtu setting, then it is of the correct size.


Step 1 On the VSM, verify the system port profile configuration.

show port-profile name system-port-profile-name

Example:
n1000v# show port-profile name SystemUplink
port-profile SystemUplink
  description: 
  type: ethernet
  status: enabled
  capability l3control: no
  pinning control-vlan: -
  pinning packet-vlan: -
  system vlans: 114,115
  port-group: SystemUplink
  max ports: 32
  inherit: 
  config attributes:
    switchport mode trunk
    switchport trunk allowed vlan all
    system mtu 1500
    no shutdown
  evaluated config attributes:
    switchport mode trunk
    switchport trunk allowed vlan all
    no shutdown
  assigned interfaces:
 
   

Verifying the Control and Packet VLAN Configuration

You can use the following procedure to verify that the control and packet VLANs are configured on the VSM.


Note The procedure documented is for troubleshooting VSM and VEM connectivity with layer 2 mode.


BEFORE YOU BEGIN

Before beginning this procedure, you should know or do the following:

You are logged in to the CLI in EXEC mode.

The output of the show running-config command should show control and packet VLAN ID numbers among the VLANs configured,


Step 1 On the VSM, verify that the control and packet VLANs are present.

n1000v# show running-config vlan 260-261
version 4.0(4)SV1(3)
vlan 260
  name cp_control
vlan 261
  name cp_packet
 
   
n1000v# 
. . .

Step 2 Find the AIPC MAC address of the VSM by running show svs neighbors on the VSM.

switch(config-svs-domain)# show svs neighbors
 
   
Active Domain ID: 27
 
   
AIPC Interface MAC: 0050-56bc-74f1 <-------------
Inband Interface MAC: 0050-56bc-62bd
 
   
Src MAC           Type   Domain-id    Node-id     Last learnt (Sec. ago)
------------------------------------------------------------------------
 
   
0050-56bc-6a3d     VSM        27         0201     771332.97
0002-3d40-1b02     VEM        27         0302        51.60
0002-3d40-1b03     VEM        27         0402        51.60
 
   

Step 3 Find the DPA MAC address of the VEM by running vemcmd show card on the ESX host.

# vemcmd show card
Card UUID type  2: 24266920-d498-11e0-0000-00000000000f
Card name:
Switch name: Nexus1000v
Switch alias: DvsPortset-0
Switch uuid: ee 63 3c 50 04 b1 6d d6-58 61 ff ba 56 05 14 fd
Card domain: 27
Card slot: 3
VEM Tunnel Mode: L2 Mode
VEM Control (AIPC) MAC: 00:02:3d:10:1b:02
VEM Packet (Inband) MAC: 00:02:3d:20:1b:02
VEM Control Agent (DPA) MAC: 00:02:3d:40:1b:02 <-------------
VEM SPAN MAC: 00:02:3d:30:1b:02
Primary VSM MAC : 00:50:56:bc:74:f1
Primary VSM PKT MAC : 00:50:56:bc:62:bd
Primary VSM MGMT MAC : 00:50:56:bc:0b:d5
Standby VSM CTRL MAC : 00:50:56:bc:6a:3d
Management IPv4 address: 14.17.168.1
Management IPv6 address: 0000:0000:0000:0000:0000:0000:0000:0000
Primary L3 Control IPv4 address: 0.0.0.0
Secondary VSM MAC : 00:00:00:00:00:00
Secondary L3 Control IPv4 address: 0.0.0.0
Upgrade : Default
Max physical ports: 32
Max virtual ports: 216
Card control VLAN: 168
Card packet VLAN: 168
Control type multicast: No
Card Headless Mode : No
       Processors: 16
  Processor Cores: 8
Processor Sockets: 2
  Kernel Memory:   25102148
Port link-up delay: 5s
Global UUFB: DISABLED
Heartbeat Set: True
PC LB Algo: source-mac
Datapath portset event in progress : no
Licensed: Yes
 
   

Step 4 Check the upstream switches for these MAC addresses in the correct VLANs.

switch1 # show mac address-table | grep 1b02
* 168      0002.3d20.1b02    dynamic   20         F    F  Veth854
* 168      0002.3d40.1b02    dynamic   0          F    F  Veth854
* 1        0002.3d40.1b02    dynamic   1380       F    F  Veth854
 
   
switch2 # show mac address-table | grep 74f1
* 168      0050.56bc.74f1    dynamic   0          F    F  Eth1/1/3
 
   

Checking the vCenter Server Configuration

You can use the following procedure from vSphere client to verify the configuration on the vCenter Server.


Step 1 Confirm that the host is added to the data center and the Cisco Nexus 1000V DVS in that data center.

Step 2 Confirm that at least one pnic of the host is added to the DVS, and that pnic is assigned to the system-uplink profile.

Step 3 Confirm that the three VSM vnics are assigned to the port groups containing the control VLAN, packet VLAN, and management network.


Checking Network Connectivity Between the VSM and the VEM

You can use the following procedure to verify Layer 2 network connectivity between the VSM and VEM.


Step 1 On the VSM, find its MAC address.

show svs neighbors

The VSM MAC address displays as the AIPC Interface MAC.

The user VEM Agent MAC address of the host displays as the Src MAC.

 
   
Example:
n1000v# show svs neighbors
 
   
Active Domain ID: 1030
 
   
AIPC Interface MAC: 0050-568e-58b7
Inband Interface MAC: 0050-568e-2a39
 
   
Src MAC           Type   Domain-id    Node-id     Last learnt (Sec. ago)
------------------------------------------------------------------------
 
   
0002-3d44-0602     VEM      1030         0302     261058.59
 
   

Step 2 Do one of the following:

If the output of the show svs neighbors command in Step 1 does not display the VEM MAC address, then there is a problem with connectivity between the server hosting the VSM and the upstream switch. Recheck the VSM configuration and vCenter Server configuration.

Otherwise, continue with the next step.

Step 3 On the VEM, run the vem-health script using the VSM MAC address you found in Step 1.


Note If the vem-health script is not in the PATH, you can find it under /usr/lib/ext/cisco/nexus/vem*/sbin/.


vem-health check vsm_mac_address

The vem-health script output shows the cause of the connectivity problem and recommends next steps in troubleshooting.

Example:
~ # vem-health check  00:50:56:a3:36:90
VSM Control MAC address: 00:50:56:a3:36:90
Control VLAN: 90
DPA MAC: 00:02:3d:40:5a:03
 
   
VSM heartbeats are not reaching the VEM.
Your uplink configuration is correct.
Recommended action:
Check if the VEM's upstream switch has learned the VSM's Control MAC.
 
   

Step 4 Do one of the following:

If the VEM health check in Step 3 indicates a problem with connectivity to the upstream switch. continue with the next step.

Otherwise, go to Step 7.

Step 5 On the upstream switch, display the MAC address table to verify the network configuration.

 
   
Example:
switch# show mac address-table interface Gi3/1 vlan 3002
Legend: * - primary entry
        age - seconds since last seen
        n/a - not available
 
   
  vlan   mac address     type    learn     age              ports
------+----------------+--------+-----+----------+--------------------------
Active Supervisor:
* 3002  0050.56be.7ca7   dynamic  Yes          0   Gi3/1
 
   
switch# show mac address-table interface Gi3/2 vlan 3002
Legend: * - primary entry
        age - seconds since last seen
        n/a - not available
 
   
  vlan   mac address     type    learn     age              ports
------+----------------+--------+-----+----------+--------------------------
Active Supervisor:
* 3002  00:02:3d:40:0b:0c   dynamic  Yes          0   Gi3/2
 
   

Step 6 Do one of the following:

If the output from Step 5 does not display the MAC address of the VSM, then there is a problem with connectivity between the server hosting the VSM and the upstream switch. Recheck the VSM configuration and vCenter Server configuration.

Otherwise, continue with the next step.

Step 7 On the VEM, enter the following commands to verify that the VSM MAC appears in the control and packet VLANs.

config t

module vem module_number execute vemcmd show l2 control_vlan_id

module vem module_number execute vemcmd show l2 packet_vlan_id

The VSM eth0 and eth1 MAC addresses should display in the host control and packet VLANs.
 
   
Example:
n100v# config t
n1000v(config)# module vem 3 execute vemcmd show l2 3002
Bridge domain 3002 brtmax 100, brtcnt 3, timeout 120
    Dynamic MAC 00:50:56:be:7c:a7  LTL    16 pvlan     0 timeout   110
    Dynamic MAC 00:02:3d:40:0b:0c  LTL    10 pvlan     0 timeout   110
 
   
n1000v(config)# module vem 3 execute vemcmd show l2 3003
Bridge domain 3002 brtmax 100, brtcnt 3, timeout 120
    Dynamic MAC 00:50:56:be:7c:a7  LTL    16 pvlan     0 timeout   110
    Dynamic MAC 00:02:3d:20:0b:0c  LTL    10 pvlan     0 timeout   110
 
   

Step 8 Do one of the following:

If the MAC address of the VSM does not appear in the output of Step 7, then check the VEM configuration as explained in "Checking the VEM Configuration" section.

Otherwise, you have completed this procedure.


Checking the VEM Configuration

You can use the following procedure to verify that the ESX host received the VEM configuration and setup.


Step 1 On the ESX host, use the following command to confirm that the VEM Agent is running, and that the correct host uplinks are added to the DVS.

vem status

 
   
Example:
~ # vem status
VEM modules are loaded
 
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks   
vSwitch0       64          3           64                1500    vmnic0    
DVS Name       Num Ports   Used Ports  Configured Ports  Uplinks   
n1000v         256         9           256               vmnic1 VEM Agent is running
 
   

Step 2 Use the following commands to restore connectivity that is lost due to an incorrect MTU value on an uplink:

vemcmd show port port-LTL-number

vemcmd set mtu value ltl port-LTL-number

Example:
~ # vemcmd show port 48
LTL    IfIndex   Vlan    Bndl  SG_ID Pinned_SGID  Type  Admin State  CBL Mode Name 
. . .   
17   1a030100      1 T   304      1          32  PHYS     UP    UP    1  Trunk vmnic1
~# vemcmd set mtu 9000 ltl 17
 
   

Note Use these vemcmds only as a recovery measure and then update the MTU value in the port profile configuration for system uplinks or in the interface configuration for non-system uplinks.


Step 3 Use the following command to verify that the domain ID, control VLANs, and packet VLANs are configured correctly on the host.

vemcmd show card

Example:
~ # vemcmd show card
Card UUID type  2: 58f8afd7-e1e3-3c51-85e2-6e6f2819a7b8
Card name: sfish-srvr-1
Switch name: n1000v
Switch alias: DvsPortset-0
Switch uuid: 56 e0 36 50 91 1c 32 7a-e9 9f 31 59 88 0c 7f 76
Card domain: 1024
Card slot: 4
VEM Control (Control VLAN) MAC: 00:02:3d:14:00:03
VEM Packet (Inband) MAC: 00:02:3d:24:00:03
VEM Control Agent (DPA) MAC: 00:02:3d:44:00:03
VEM SPAN MAC: 00:02:3d:34:00:03
Management IP address: 172.23.232.102
Max physical ports: 32
Max virtual ports: 216
Card control VLAN: 3002
Card packet VLAN: 3003
       Processors: 4
  Processor Cores: 4
Processor Sockets: 2
  Physical Memory: 4290351104
 
   

Step 4 Use the following command to verify that the ports of the host added to the DVS are listed, and that the ports are correctly configured as access or trunk on the host.

vemcmd show port 
 
   
Example:
~ # vemcmd show port  
LTL    IfIndex   Vlan    Bndl  SG_ID Pinned_SGID  Type  Admin State  CBL Mode   Name    
8          0   3969       0      2           2  VIRT     UP    UP    1 Access l20     
9          0   3969       0      2           2  VIRT     UP    UP    1 Access l21    
10         0   3002       0      2           2  VIRT     UP    UP    1 Access l22   
11         0   3968       0      2           2  VIRT     UP    UP    1 Access l23    
12         0   3003       0      2           2  VIRT     UP    UP    1 Access l24    
13         0      1       0      2           2  VIRT     UP    UP    0 Access l25    
14         0   3967       0      2           2  VIRT     UP    UP    1 Access l26    
16  1a030100      1 T     0      2           2  PHYS     UP    UP    1 Trunk vmnic1
 
   

The last line of output indicates that vmnic1 should be in Trunk mode, with the CBL value of 1. The CBL value of the native VLAN does not have to be 1. It will be 0 if it is not allowed, or 1 if it is VLAN 1 and not allowed. This is not an issue unless the native VLAN is the Control VLAN.The Admin state and Port state should be UP.

Step 5 Use the following commands to verify that the vmnic port that is supposed to carry the control VLAN and packet VLAN is present.

vemcmd show bd control_vlan
vemcmd show bd packet_vlan 
 
   
Example:
~ # vemcmd show bd 3002
BD 3002, vdc 1, vlan 3002, 2 ports
Portlist:
     10  l22
     16  vmnic1
~ # vemcmd show bd 3003
BD 3003, vdc 1, vlan 3003, 2 ports
Portlist:
     12  l24
     16  vmnic1
 
   

Step 6 Use the vemcmd show trunk command to verify the following:

The control and packet VLANs are shown in the command output, indicating that the DV port groups are successfully pushed from the vCenter Server to the host.

The correct physical trunk port vmnic is used.

Example:
~ # vemcmd show trunk
Trunk port 16 native_vlan 1 CBL 1vlan(1) cbl 1, vlan(3002) cbl 1, vlan(3003) cbl 1,
 
   

At least one physical uplink must be carrying the control and packet VLANs. If more than one uplink is carrying the control and packet VLANs, the uplinks must be in a port channel profile. The port channel itself would not be visible because the VEM is not yet added to the VSM.

Step 7 Use the following commands to restore connectivity that is lost due to incorrect port and system VLAN settings:

vemcmd show port port-ltl-number

vemcmd set system-vlan vlan_id ltl port-ltl-number

Example:
~ # vemcmd show port 48
LTL    IfIndex   Vlan    Bndl  SG_ID Pinned_SGID  Type  Admin State  CBL Mode   Name 
. . .   
48     1b030000     1       0     32           1 VIRT     UP  DOWN     0 Access vmk1
~ # vemcmd set system-vlan 99 ltl 48
 
   

Note Use these vemcmds only as a recovery measure and then update the port profile configuration with correct system VLANs.



Collecting Logs

After you have verified network connectivity between the VEM and the VSM, you can use the following procedure to collect log files to help identify the problem.


Step 1 On the VEM, use the following command to verify its UUID.

vemcmd show card info
 
   
Example:
~ # module vem 3 vemcmd show card info
Card UUID type  0: 4908a717-7d86-d28b-7d69-001a64635d18
Card name: sfish-srvr-7
Switch name: N1000v
Switch uuid: 50 84 06 50 81 36 4c 22-9b 4e c5 3e 1f 67 e5 ff
Card domain: 11
Card slot: 12
Control VLAN MAC: 00:02:3d:10:0b:0c
Inband MAC: 00:02:3d:20:0b:0c
SPAN MAC: 00:02:3d:30:0b:0c
USER DPA MAC: 00:02:3d:40:0b:0c
Management IP address: 172.28.30.56
Max physical ports: 16
Max virtual ports: 32
Card control VLAN: 3002
Card packet VLAN: 3003
 
   

Step 2 On the VSM, use the following command to verify the module number to which the corresponding UUID entry is mapped.

show module vem mapping
 
   
Example:
n1000v# show module vem mapping 
Mod     Status          UUID                                    License Status
---     -----------     ------------------------------------    --------------
60      absent          33393935-3234-5553-4538-35314e355400    unlicensed
66      powered-up      33393935-3234-5553-4538-35314e35545a    licensed
n1000v#
 
   

Step 3 Using the module number from Step 2, collect the output of the following commands:

show platform internal event-history module 13

show module internal event-history module 13

show system internal im event-history module 13

show system internal vmm event-history module 13

show system internal ethpm event-history module 13



Note If you need to contact Cisco TAC for assistance in resolving an issue, you will need the output of the commands listed in Step 3.


VSM and VEM Troubleshooting Commands

You can use the commands in this section to troubleshoot problems related to VSM.

Command
Description

module vem module_number execute vemcmd show l2 [control_vlan_id | packet_vlan_id]

Displays the VLAN configuration on the VEM to verify that the VSM MAC appears in the control and packet VLANs.

See Example 5-1.

show cores

Displays recent core images.

See Example 5-2.

See Example 5-3.

show mac address-table interface

Displays the MAC address table on an upstream switch to verify the network configuration.

See Example 5-4.

show module internal event-history module module-number

Displays the event log for a module.

show module vem mapping

Displays information about the VEM a VSM maps to, including VEM module number, status, UUID, and license status

See Example 5-5.

show platform internal event-history module module-number

Displays platform FSM event information.

show port-profile name name

Displays the configuration for a named port profile.

See Example 5-6.

show running-config vlan vlanID

Displays the VLAN information in the running configuration.

See Example 5-7.

show svs connections

Displays the Cisco Nexus 1000V connections.

See Example 5-8.

show svs domain

Displays the domain configuration.

See Example 5-9.

show svs neighbors

Displays all svs neighbors.

See Example 5-10.

show system internal ethpm event-history module module-number

Displays the module Ethernet event logs for the system.

show system internal ethpm event-history int type slot

Displays the Ethernet interface logs for the system.

show system internal im event-history module module-number

Displays the module IM event logs for the system.

show system internal vmm event-history module module-number

Displays the module VMM event logs for the system.

vem-health check vsm_mac_address

Displays the cause of a connectivity problem and recommends next steps in troubleshooting.

See Example 5-11.

vem status

Displays the VEM status to confirm that the VEM Agent is running, and that the correct host uplinks are added to the DVS.

See Example 5-12.

vemcmd show bd [control_vlan_id | packet_vlan_id]

Displays configured information on the VEM to verify that the VM NIC port that is supposed to carry the control VLAN and packet VLAN is present.

See Example 5-13.

vemcmd show card

Displays information about cards on the VEM to verify that the domain ID, control VLANs, and packet VLANs are configured correctly on the host.

See Example 5-14.

vemcmd show port [ port-LTL-number]

Displays information about ports on the VEM to verify that the ports of the host added to the DVS are listed, and that the ports are correctly configured as access or trunk on the host.

See Example 5-15.

See Example 5-16.

vemcmd show trunk

Displays configured information on the VEM to verify that the DV port groups are successfully pushed from the vCenter Server to the host and that the correct physical trunk port VM NIC is used.

See Example 5-17.


Example 5-1 module vem execute vemcmd show l2

n100v# config t
n1000v(config)# module vem 3 execute vemcmd show l2 3002
Bridge domain 3002 brtmax 100, brtcnt 3, timeout 120
    Dynamic MAC 00:50:56:be:7c:a7  LTL    16 pvlan     0 timeout   110
    Dynamic MAC 00:02:3d:40:0b:0c  LTL    10 pvlan     0 timeout   110
 
   
n1000v(config)# module vem 3 execute vemcmd show l2 3003
Bridge domain 3002 brtmax 100, brtcnt 3, timeout 120
    Dynamic MAC 00:50:56:be:7c:a7  LTL    16 pvlan     0 timeout   110
    Dynamic MAC 00:02:3d:20:0b:0c  LTL    10 pvlan     0 timeout   110

Example 5-2 show cores

n1000V# show cores
VDC No Module-num       Process-name      PID     Core-create-time
------ ----------       ------------      ---     ----------------
1      1                private-vlan      3207    Apr 28 13:29	
n1000V#

Example 5-3 show cores

n1000v# show core
Module-num       Instance-num      Process-name     PID      Core-create-time
----------       ------------      ------------     ---      ----------------
2                1                aclmgr          2894     Apr 1 23:44
n1000V#

Example 5-4 show mac address-table interface

switch# show mac address-table interface Gi3/1 vlan 3002
Legend: * - primary entry
        age - seconds since last seen
        n/a - not available
 
   
  vlan   mac address     type    learn     age              ports
------+----------------+--------+-----+----------+--------------------------
Active Supervisor:
* 3002  0050.56be.7ca7   dynamic  Yes          0   Gi3/1

Example 5-5 show module vem mapping

n1000v# show module vem mapping 
Mod     Status          UUID                                    License Status
---     -----------     ------------------------------------    --------------
60      absent          33393935-3234-5553-4538-35314e355400    unlicensed
66      powered-up      33393935-3234-5553-4538-35314e35545a    licensed
n1000v#

Example 5-6 show port-profile

n1000v# show port-profile name SystemUplink
port-profile SystemUplink
  description: 
  type: ethernet
  status: enabled
  capability l3control: no
  pinning control-vlan: -
  pinning packet-vlan: -
  system vlans: 114,115
  port-group: SystemUplink
  max ports: 32
  inherit: 
  config attributes:
    switchport mode trunk
    switchport trunk allowed vlan all
    system mtu 1500
    no shutdown
  evaluated config attributes:
    switchport mode trunk
    switchport trunk allowed vlan all
    no shutdown
  assigned interfaces:

Example 5-7 show running-configuration vlan

n1000v# show running-config vlan 260-261
version 4.0(4)SV1(3)
vlan 260
  name cp_control
vlan 261
  name cp_packet
 
   
n1000v# 

Example 5-8 show svs connections

n1000v# show svs connections
connection vc:
    ip address: 172.23.231.223
    protocol: vmware-vim https
    certificate: user-installed
    datacenter name: hamilton-dc
    DVS uuid: 92 7a 14 50 05 11 15 9c-1a b0 f2 d4 8a d7 6e 6c
    config status: Disabled
    operational status: Disconnected

Example 5-9 show svs domain

n1000v# show svs domain
SVS domain config: 
  Domain id:    682 
  Control vlan: 3002
  Packet vlan:  3003
  L2/L3 Control VLAN mode: L2 
  L2/L3 Control VLAN interface: mgmt0
  Status: Config push to VC successful

Example 5-10 show svs neighbors

n1000v# show svs neighbors
 
   
Active Domain ID: 113
 
   
AIPC Interface MAC: 0050-56b6-2bd3
Inband Interface MAC: 0050-56b6-4f2d
 
   
Src MAC           Type   Domain-id    Node-id     Last learnt (Sec. ago)
------------------------------------------------------------------------
 
   
0002-3d40-7102     VEM       113         0302     71441.12
0002-3d40-7103     VEM       113         0402       390.77
 
   
n1000v#
 
   

Example 5-11 vem-health check

~ # vem-health check  00:50:56:a3:36:90
VSM Control MAC address: 00:50:56:a3:36:90
Control VLAN: 90
DPA MAC: 00:02:3d:40:5a:03
 
   
VSM heartbeats are not reaching the VEM.
Your uplink configuration is correct.
Recommended action:
Check if the VEM's upstream switch has learned the VSM's Control MAC.

Example 5-12 vem status

~ # vem status
VEM modules are loaded
 
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks   
vSwitch0       64          3           64                1500    vmnic0    
DVS Name       Num Ports   Used Ports  Configured Ports  Uplinks   
n1000v         256         9           256               vmnic1 VEM Agent is running

Example 5-13 vemcmd show bd

~ # vemcmd show bd 3002
BD 3002, vdc 1, vlan 3002, 2 ports
Portlist:
     10  l22
     16  vmnic1

Example 5-14 vemcmd show card

~ # vemcmd show card
Card UUID type  2: 58f8afd7-e1e3-3c51-85e2-6e6f2819a7b8
Card name: sfish-srvr-1
Switch name: n1000v
Switch alias: DvsPortset-0
Switch uuid: 56 e0 36 50 91 1c 32 7a-e9 9f 31 59 88 0c 7f 76
Card domain: 1024
Card slot: 4
VEM Control (Control VLAN) MAC: 00:02:3d:14:00:03
VEM Packet (Inband) MAC: 00:02:3d:24:00:03
VEM Control Agent (DPA) MAC: 00:02:3d:44:00:03
VEM SPAN MAC: 00:02:3d:34:00:03
Management IP address: 172.23.232.102
Max physical ports: 32
Max virtual ports: 216
Card control VLAN: 3002
Card packet VLAN: 3003
       Processors: 4
  Processor Cores: 4
Processor Sockets: 2
  Physical Memory: 4290351104

Example 5-15 vemcmd show port

~ # vemcmd show port  
LTL    IfIndex   Vlan    Bndl  SG_ID Pinned_SGID  Type  Admin State  CBL Mode   Name    
8          0   3969       0      2           2  VIRT     UP    UP    1 Access l20     
9          0   3969       0      2           2  VIRT     UP    UP    1 Access l21    
10         0   3002       0      2           2  VIRT     UP    UP    1 Access l22   
11         0   3968       0      2           2  VIRT     UP    UP    1 Access l23    
12         0   3003       0      2           2  VIRT     UP    UP    1 Access l24    
13         0      1       0      2           2  VIRT     UP    UP    0 Access l25    
14         0   3967       0      2           2  VIRT     UP    UP    1 Access l26    
16  1a030100      1 T     0      2           2  PHYS     UP    UP    1 Trunk vmnic1

Example 5-16 vemcmd show port

~ # vemcmd show port 48
LTL    IfIndex   Vlan    Bndl  SG_ID Pinned_SGID  Type  Admin State  CBL Mode   
Name  . . .   
17   1a030100      1 T   304      1          32  PHYS     UP    UP    1  Trunk vmnic1

Example 5-17 vemcmd show trunk

~ # vemcmd show trunk
Trunk port 16 native_vlan 1 CBL 1vlan(1) cbl 1, vlan(3002) cbl 1, vlan(3003) cbl 1,