Troubleshooting Scenarios for Nexus 1000v (N1kv) with VSUM

Introduction

This document describes the troubleshooting scenarios for Nexus (N1kv) with Virtual Switch Update Manager (VSUM).

Background Information

Cisco VSUM is a virtual appliance that is registered as a plug-in to the VMware vCenter Server. The Cisco VSUM GUI is an integral part of the VMware vSphere Web Client; it can only be accessed when you log into the VMware vSphere Web Client. Cisco VSUM simplifies the installation and configuration of the Cisco Nexus 1000V and the Cisco Application Virtual Switch (AVI).

Cisco VSUM let's you do this:

• Cisco Nexus 1000V for VMware vSphere
• Install the Cisco Nexus 1000V switch
• Migrate the VMware vSwitch and VMware vSphere Distributed Switch (VDS) to the Cisco Nexus 1000V
• Monitor the Cisco Nexus 1000V
• Upgrade the Cisco Nexus 1000V and add hosts from an earlier version to the latest version
• Install the Cisco Nexus 1000V license
• View the health of the virtual machines in your data center using the Dashboard - Cisco Nexus 1000V
• Upgrade from an earlier release to Cisco VSUM 2.0

Overview of Troubleshooting Process

Follow the steps here in order to troubleshoot your network:

1. Gather information that defines the specific symptoms
2. Identify all the potential problems that can cause the symptoms
3. Systematically eliminate each potential problem (from most likely to least likely) until the symptoms disappear

Overview of Best Practices

Best practices are the recommended steps you must take in order to ensure the proper operation of your network. These best practices are recommended for most networks:

• Maintain a consistent Cisco VSUM release across all network devices

• Refer to the release notes for your Cisco VSUM release for the latest features, limitations, and caveats

• Enable system message log in

• Verify and troubleshoot any new configuration changes after you implement the change

Troubleshooting Scenarios

These are some of the most common issues Cisco Technical Assistance Center (TAC) has come across at the time this article was written.

Scenario 1. VSUM Icon Missing in Web Client

After VSUM VM deployment through Open Virtual Appliance (OVA) and successful initialization, the VSUM icon doesn't show up in vCenter inventory as shown in the image.

Network Connectivity

• First, confirm that the network connectivity exists among components.
• Next, ensure that the initial information provided in the VSUM OVA is correct. This includes vCenter IP’s and port numbers. 
• Finally, allow port 8443 for internal communication.
• Install guide has been updated with this information, tracking bug: CSCux34597)

Common Installation Issues

1. OVA Corruption
   • Confirm the MD5 checksum of the OVA before it's deployment
2. The N1kv Manager/VSUM icon does not show in the vCenter web client
   • Most likely, the appliance is not reachable via IP
   • Confirm connectivity via Secure Shell (SSH) and ensure that the VM’s port-group and IP address are correct
   • Ping the default gateway from the applicance console
   • Log into the appliance and check the installation log (see for filepath) for the failure reason. You must see NoRouteToHostException

In order to change the IP Address, navigate to /etc/cisco/app_install
Run cp app.cfg.template app.cfg
Run vi app.cfg and update the information. Here is an example:

 IpV4Address="10.28.28.121"
                IpV4Netmask="255.255.255.0"
                IpV4Gateway="10.28.28.1"
                DnsServer1="10.28.28.115"
                DnsServer2=""
                VcenterIPV4Address="10.28.28.120”
                VcenterUsernameFormat="hex"
                VcenterUsername="726f6f74"
                VcenterPasswordFormat="hex"
                VcenterPassword="<Password Encoded As Stated>"
                VcenterHttpPort="80"
                VcenterHttpsPort="443"

Save this file and run:

./config_apps.sh –n in order to update network information

./config_apps.sh –r in order to register the VM to vCenter web client

• Confirm that you can navigate to vc mob [https://<vcenter-IP>/mob] > Content > ExtensionManager
• Get to the extension list for VSUM (example: extensionList["com.cisco.n1kv”])
• Navigate to client
• Here, you can see a URL (example: https://<vCenter-IP>:8443/n1kv/static/client/cisco-n1kv-mgr.zip)
• Confirm that you can download this file; this also ensures that port 8443 is open

3. vCenter IP/credentials are incorrect

Log into the appliance and check the installation log (see for filepath) for the failure reason. You must see InvalidLogin.
Perform the same steps as mentioned earlier (for no route to host) and input the correct vCenter username and password in the file.

4. .com.cisco.n1kv is already on vCenter

• Check if an old N1kv plugin exists in vCenter (Navigate to Web Client > Distributed Virtual Switches > [any N1kv switch] > Monitor)
• Browse to https://<vcenter IP>/mob
• Navigate to content >extension manager
• Find the extension key with n1kv
• Unregister the extension
• Perform as follows:
  • On vCA:
    • Run rm -rf /var/lib/vmware/vsphere-client/vc-packages/vsphere-client-serenity/com.cisco.n1kv-0.9.1
    • Restart the web client service with /etc/init.d/vsphere-client restart
  •  On Windows vCenter Server:
    • Delete cisco.n1kv-0.9.1 from C:\ProgramData\VMware\vSphere Web Client\vc-packages\vsphere-client-serenity
    • Restart the web client by navigating to: start > run > services.msc. Right-click: VMware vsphere web client > stop and start
• Re-deploy the VM in order to register it successfully

Common Installation Errors

• "IP ADDRESS IN USE"

If the IP is already in use, the installation will roll back.

• "NO SUITABLE HOST FOUND"

This error is seen when the ctl port-group from one host and mgmt. port-group from another host are used. Ensure the ctrl and mgmt. vlan port groups are available on the same host.

• "VSM POWER ON FAILED"

The host selected on the instal screen doesn’t have enough resources (CPU, Memory) for the VSM to power on.

• "DUPLICATE VM NAME"

The switch name entered already exists on the host. The deployment rolls back at this stage.

• "NO DISK SPACE"

There is not enough disk space on the host in order to deploy the VSM.

Other Installation Logs

Some other errors that can be seen between VSUM and vCenter are logged as ajax Java exceptions. They can be found under /etc/cisco/app_install/logs/n1kv-manager_install.log. Example: 

1784 [main] ERROR com.cisco.vcenter.extension.register.ExtensionRegister - An extension with this key is already registered. Will not attempt to register. It must be unregistered manually first, before attempting to register again.
Exception in thread "main" java.lang.IllegalArgumentException
       at com.cisco.vcenter.extension.register.ExtensionRegister.unregisterPrevExt(ExtensionRegister.java:590)
       at com.cisco.vcenter.extension.register.ExtensionRegister.register(ExtensionRegister.java:629)
       at com.cisco.vcenter.extension.register.ExtensionRegister.doWork(ExtensionRegister.java:679)
       at com.cisco.vcenter.extension.register.PluginUtil.dispatchWork(PluginUtil.java:72)
       at com.cisco.vcenter.extension.register.PluginUtil.main(PluginUtil.java:116)

A good practice is to review the access log, in order to see what the latest action was when the command failed. You can search for it in the /usr/local/tomcat/logs/ciscoExt log, in order to find detailed debugging information. You can look at other live logged information under /usr/local/tomcat/logs/.

Scenario 2. Only Current Firmware Version Listed

Unable to upgrade the VSMs from release x to release y, as you do not see any images listed other than the current one that runs as shown in the image.

Collect VSUM Logs:

Step 1	Use SSH in order to connect into the Cisco VSUM. The default username is root and the password is cisco.
Step 2	Navigate to /etc/cisco/app_install and run the ./bundleLogs.sh command.
Step 3	In the root directory, retrieve the ajaxLogs folder, compress the folder, and send it to Cisco TAC.

Action Taken:

TAC checked the VSUM logs (ajaxLogs\tomcatAllLogs\usr\local\tomcat\logs\ciscoExt.log).

Found a difference between the VSM and DVS bundle ID's.

<vsm-bundle-id>VEM500-201411171101-BG</vsm-bundle-id>
<dvs-bundle-id>VEM410-201301152101-BG</dvs-bundle-id>

This was corrected by the procedure mentioned here:

VSM-01# show module

1  5.2(1)SV3(1.2) 0.0
2  5.2(1)SV3(1.2) 0.0
3  5.2(1)SV3(1.2) VMware ESXi 5.5.0 Releasebuild-2456374 (3.2)
4  5.2(1)SV3(1.2) VMware ESXi 5.5.0 Releasebuild-2456374 (3.2)
5  5.2(1)SV3(1.2) VMware ESXi 5.5.0 Releasebuild-2456374 (3.2)

VSM-01# show system vem feature level

Current feature level: 4.2(1)SV2(1.1)

VSM-01# system update vem feature level

Feature Version
Level String
--------------------
1 4.2(1)SV2(2.1)
2 4.2(1)SV2(2.2)
3 4.2(1)SV2(2.3)
4 5.2(1)SV3(1.1)
5 5.2(1)SV3(1.2)

VSM-01 # system update vem feature level?

<CR>
<1-50> Version number index from the list above

VSM-01# system update vem feature level 5  <<< 5 only for this sceanrio as the version N1k is currently is SV3(1.2)

VSM-01 # show system vem feature level

Current feature level: 5.2(1)SV3(1.2)

Note: When you upgrade the feature level on the Virtual Ethernet Module (VEM), it does not require any downtime.

VSM-01# show vmware vem upgrade status

Upgrade VIBs: System VEM Image
Upgrade Status:
Upgrade Notification Sent Time:
Upgrade Status Time(vCenter):
Upgrade Start Time:
Upgrade End Time(vCenter):
Upgrade Error:
Upgrade Bundle ID:
VSM: VEM500-201411171101-BG
DVS: VEM410-201301152101-BG    <<< same info as we noted in VSUM logs

VSM-01# vmware vem upgrade notify, as shown in the images.

Co-ordinate with and notify the server administrator of the VEM upgrade process.

VSM-01# vmware vem upgrade proceed

Note: If VUM is enabled in the vCenter environment, disable it before you run the vmware vem upgrade proceed command in order to prevent the new VIBs from being pushed to all the hosts.

Note: Run the vmware vem upgrade proceed command so that the Cisco Nexus 1000V Bundle ID on the vCenter Server gets updated. If VUM is enabled and you do not update the Bundle ID, an incorrect VIB version is pushed to the VEM when you next add the ESXi to the VSM.

Note: If VUM is not installed, the “The object or item referred to could not be found” error appears in the vCenter Server task bar. You can ignore this error message.

VSM-01# vmware vem upgrade complete

You must now be able to view the other N1kv releases in vCenter Web Client, as shown in the image.  

Scenario 3. Hosts Show Up Under 'No Upgrade Needed Hosts'

All the hosts (VEM's) show up under the No Upgrade Needed Hosts section, as shown in the image.

Collect VSUM Logs:

Step 1	Use SSH in order to connect into Cisco VSUM. The default username is root and the password is cisco.
Step 2	Navigate to /etc/cisco/app_install and run the ./bundleLogs.sh command.
Step 3	In the root directory, retrieve the ajaxLogs folder, compress the folder, and send it to Cisco TAC.

Verify:

• Clear the browser history and cached memory
• Log out of the vCenter Web Client and then log in again
• Verify that the host appears in the Eligible Hosts drop-down list
• Select the host and proceed with the upgrade

These steps can be found in the guide:

When upgrading Cisco N1000v, the hosts being upgraded should appear in Eligible Hosts drop-down list

If these steps didn't help, you might have run into defect CSCuz11671 as shown in the image.

This behavior was seen with VSUM 1.5.3 and would be fixed in VSUM 2.1 (so any version below 2.1 could be impacted).

Scenario 4. Unable to Upgrade VSM

Unable to upgrade VSM from SV3 (1.10) to SV3 (1.15).

In the Pre-upgrade settings, you attempt to upgrade a VSM with VSUM as shown in the image.

After you enter in their configuration and credentials, this error is displayed as shown in the image.

Collect VSUM Logs:

Step 1	Use SSH in order to connect into VSUM. The default username is root and the password is cisco.
Step 2.	Navigate to /etc/cisco/app_install and run the ./bundleLogs.sh command.
Step 3	In the root directory, retrieve the ajaxLogs folder, compress the folder, and send it to Cisco TAC.

Prime Network Services Controller (PNSC) and Voice Source-Group (VSG) are a part of your setup.

This issue was because VSUM received incorrect information for the PSNC and VSG versions, which caused it to halt the upgrade.

You can confirm with these commands:

N1kv # show vmware vem upgrade status | xml

<show>

<vmware>
<vem>
<upgrade>
<status>
….
<vsm-bundle-id>VEM500-201512250101-BG</vsm-bundle-id>
<dvs-bundle-id>VEM500-201510210101-BG</dvs-bundle-id> <-- these two IDs do not match
…..
</status>
</upgrade>
</vem>
</vmware>
</show>

 

An internal error refers to an error that is an exception in VSUM code.

These two values must match in order to allow the upgrade to proceed.

Here is the procedure to correct this behavior:

Run this command to ensure that both bundle id’s are equal on the Nexus 1000v switch:

vmware vem upgrade proceed

The command above makes both the bundle id’s same, if this does not work, run these commands:

vmware vem upgrade notify

vmware vem upgrade proceed

The vCenter administrator needs to accept Apply Upgrade prompt in vCenter. Once you do that, run vmware vem upgrade proceed.

Documentation:

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/5_2_1_s_v_3_1_5a/install_upgrade/guide/b_Cisco_N1KV_VMware_Install_and_Upgrade_Guide_521SV315a/b_Cisco_N1KV_VMware_Install_and_Upgrade_Guide_521SV314a_chapter_010.html#d8382e2398a1635

In the section: VMware Update Manager from Release 4.2(1) SV2 (1.1x)

Defect was created to track this issue: 

Scenario 5. Error "fault.com.cisco.n1kv.internalerror.summary"

When you attempt to upgrade the VSM from version X to Y, you recieve an error message in vCenter tasks that returns a status of "fault.com.cisco.n1kv.internalerror.summary".

However, this fault only displays in the traditional GUI and does not display this error in the vCenter web client as shown in the image.

Collect VSUM Logs:

Step 1.	Use SSH to connect into Cisco VSUM. The default username is root and the password is cisco.
Step 2.	Navigate to /etc/cisco/app_install and run the ./bundleLogs.sh command.
Step 3.	In the root directory, retrieve the ajaxLogs folder, compress the folder, and send it to the Cisco TAC.

Action Taken:

Review usr/tomcat/logs/ciscoEXT.log. Search for scp

You can find entries similar to this:

257266658 DEBUG 2016-07-15 06:26:18,855 [pool-2-thread-5] com.cisco.n1kv.vsm.SSHAgent - Raw output is copy scp://scpuser@10.10.100.10///etc/cisco/data/n1kvbins/VSM-v-j3-n1-u15-l-b-v/upgrade/vsm/n1000v-dk9-kickstart.5.2.1.SV3.1.15.bin bootflash:

From the VSM execute: scp://scpuser@10.10.100.10///etc/cisco/data/n1kvbins/VSM-v-j3-n1-u15-l-b-v/upgrade/vsm/n1000v-dk9-kickstart.5.2.1.SV3.1.15.bin bootflash:

If the connection is successful, you will receive a prompt for credentials.

If the command fails, this indicates that there is a communication problem between VSUM and the VSM.

The likely cause is a firewall between VSUM and the VSM.

Verify that there is a bi-direction firewall rule in place for port 22 between VSUM and VSM.

If there is no rule in place, please create the rule and attempt the upgrade process again.