Table Of Contents

Upgrade

Information about Upgrades

Problems with Upgrade

Recovering a Secondary VSM with Active Primary

Stopping a VSM Upgrade

Changing Boot Variables

Powering On the VSM

Changing the HA Role

Recovering a Primary VSM with Active Secondary

Disconnecting the Port Groups

Powering Off the VSM

Connecting the Port Groups

Collecting and Evaluating Logs

Upgrade Troubleshooting Commands

Upgrade

---

This chapter describes how to identify and resolve problems related to upgrading the VSM software, and includes the following sections:

•Information about Upgrades

•Problems with Upgrade

•Collecting and Evaluating Logs

•Upgrade Troubleshooting Commands

Information about Upgrades

The upgrade for the Cisco Nexus 1000V involves upgrading software on both the VSM and the VEM.

For detailed information, see the following documents:

•Cisco Nexus 1000V Software Upgrade Guide, Release 4.2(1)SV1(4).

•Cisco Nexus 1000V Virtual Ethernet Module Software Installation Guide, Release 4.2(1)SV1(4).

Problems with Upgrade

The following are symptoms, possible causes, and solutions for problems with software upgrade.

Symptom	Possible Causes	Solution
The upgrade GUI stops and times out after 10 minutes and displays the following message: Error: Could not contact the upgraded VSM at n.n.n.n. Please check the connection.	During the upgrade, you configured an unreachable IP address for the mgmt0 interface. In this case, one VSM in the redundant pair has new software installed and is unreachable; while the other VSM has the original pre-upgrade original pre-upgrade software version installed and is reachable.	1. Use one of the following sets of procedures to return your VSM pair to the previous software version: –"Recovering a Secondary VSM with Active Primary" section –"Recovering a Primary VSM with Active Secondary" section 2. Restart the software upgrade using the detailed instructions in the Cisco Nexus 1000V Software Upgrade Guide, Release 4.2(1)SV1(4).
The upgrade GUI stops and times out after 10 minutes and displays the following message: Error: Could not contact the upgraded VSM at 10.104.244.150. Please check the connection. After timing out, one VSM comes up in switch(boot) mode.	You have selected incompatible or incorrect VSM software images for the upgrade. The software images you selected from the GUI selection list included a system image for one software version and a kickstart image for another software version. These images must be for the same software version. For an example of how software images are selected during the upgrade, see Example 4-1.	1. To continue the upgrade, first recover the VSM using one of the following: –"Recovering a Secondary VSM with Active Primary" section –"Recovering a Primary VSM with Active Secondary" section 2. Restart the software upgrade using the detailed instructions in the Cisco Nexus 1000V Software Upgrade Guide, Release 4.2(1)SV1(4).

Example 4-1 Upgrade: Enter Upgrade Information

This example shows how you specify system and kickstart images during the upgrade process. In this example, the images specified are from the same release, SV1.4. If you specify a kickstart image from one release, and a system image from another, then the upgrade cannot proceed.

Recovering a Secondary VSM with Active Primary

You can use the following process to recover a secondary VSM when the primary VSM is active.

---

Step 1 Stop the upgrade on the VSM, using the "Stopping a VSM Upgrade" procedure

Step 2 Change the boot variables back to the previous version using the "Changing Boot Variables" procedure

Step 3 From the vCenter Server left-hand panel, right-click the secondary VSM and then choose Delete from Disk.

The secondary VSM is deleted.

Step 4 Create a new VSM by reinstalling the software using the vSphere client Deploy OVF Template wizard, specifying the following:

•Nexus 1000V Secondary configuration method
(Configures the secondary VSM in an HA pair using a GUI setup dialog.)

•The host or cluster of the primary VSM

•The same domain ID and password as that of the primary VSM.

For a detailed procedure, see the Cisco Nexus 1000V Software Installation Guide, Release 4.2(1)SV1(4).

The VSM comes up and forms an HA pair with the newly-created standalone VSM. The VSMs have the previous version of the software installed.

---

Stopping a VSM Upgrade

You can use this procedure to stop a VSM upgrade that is in progress.

BEFORE YOU BEGIN

•You are logged in to the CLI in EXEC mode.

DETAILED STEPS

---

Step 1 Display upgrade status.

show svs upgrade status

Example: 

n1000v# show svs upgrade status 

Upgrade State: Start

Upgrade mgmt0 ipv4 addr: 1.1.1.1

Upgrade mgmt0 ipv6 addr:  

Upgrade control0 ipv4 addr:  

Step 2 Stop the upgrade.

configure terminal

no svs upgrade start

Example: 

n1000v# configure terminal

n1000v#(config)# no svs upgrade start 

WARNING! VSM upgrade process is aborted

n1000v#(config)#

Step 3 Display upgrade status.

show svs upgrade status

Example: 

n1000v#(config)# show svs upgrade status 

Upgrade State: Abort

Upgrade mgmt0 ipv4 addr: 

Upgrade mgmt0 ipv6 addr:  

Upgrade control0 ipv4 addr:  

Step 4 You have completed this procedure. Return to the process that pointed you here:

•"Recovering a Secondary VSM with Active Primary" section

•"Recovering a Primary VSM with Active Secondary" section

---

Changing Boot Variables

You can use this procedure to replace the software images used to boot the VSM.

BEFORE YOU BEGIN

•You are logged in to the CLI in EXEC mode.

•You know the filenames of the pre-upgrade system and kickstart image files to apply.

DETAILED STEPS

---

Step 1 Display the current boot variables.

show boot

Example: 

n1000v# show boot 

sup-1

kickstart variable = bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3a.bin

system variable = bootflash:/nexus-1000v-mz.4.0.4.SV1.3a.bin

sup-2

kickstart variable = bootflash:/nexus-1000v-kickstart-mzg.4.2.1.SV1.4.bin

system variable = bootflash:/nexus-1000v-mzg.4.2.1.SV1.4.bin

No module boot variable set

n1000v(config)#  

Step 2 Remove the current system and kickstart boot variables.

configure terminal

no boot system

no boot kickstart

Example: 

n1000v# configure terminal

n1000v(config)# no boot system 

n1000v(config)# no boot kickstart 

n1000v#(config)# 

Step 3 Restore the system and kickstart boot variables to the original pre-upgrade filenames.

boot system bootflash:system-boot-variable-name

boot kickstart bootflash:kickstart-boot-variable-name

Example: 

n1000v#(config)# boot system bootflash:nexus-1000v-mz.4.0.4.SV1.3a.bin

n1000v#(config)# boot kickstart bootflash:nexus-1000v-kickstart-mz.4.0.4.SV1.3a.bin

n1000v#(config)#

Step 4 Copy the running configuration to the startup configuration.

copy run start

Example: 

n1000v#(config)# copy run start

[########################################] 100%e

n1000v#(config)#

Step 5 Verify the change in the system and kickstart boot variables.

show boot

Example: 

n1000v#(config)# show boot 

sup-1

kickstart variable = bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3a.bin

system variable = bootflash:/nexus-1000v-mz.4.0.4.SV1.3a.bin

sup-2

kickstart variable = bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3a.bin

system variable = bootflash:/nexus-1000v-mz.4.0.4.SV1.3a.bin

No module boot variable set

n1000v#(config)#

Step 6 You have completed this procedure. Return to the process that pointed you here:

•"Recovering a Secondary VSM with Active Primary" section

•"Recovering a Primary VSM with Active Secondary" section

---

Powering On the VSM

Use this procedure to power on the newly-created VSM.

---

Step 1 From the vCenter Server left-hand panel, right-click the VSM and then choose Power > Power On.

The VSM starts.

Step 2 You have completed this procedure. Return to the "Recovering a Primary VSM with Active Secondary" section.

---

Changing the HA Role

You can use this procedure to change the HA role of the VSM.

BEFORE YOU BEGIN

•You are logged in to the CLI in EXEC mode.

•You know the domain ID of the existing VSM.

DETAILED STEPS

---

Step 1 Go to the domain of the existing VSM.

configure terminal

svs-domain

domain id domain-id

Example:

n1000v# config t 

n1000v(config)# svs-domain

n1000v(config-svs-domain)# domain id 1941

Warning: Config saved but not pushed to vCenter Server due to inactive connection!

Step 2 Change the HA role.

system redundancy role [primary | secondary | standalone]

Example: 

n1000v(config-svs-domain)# system redundancy role secondary

Setting will be activated on next reload.

n1000v(config-svs-domain)# 

Example: 

n1000v(config-svs-domain)# system redundancy role primary

Setting will be activated on next reload.

n1000v(config-svs-domain)# 

Step 3 Copy the running configuration to the startup configuration.

copy run start

Example: 

n1000v#(config-svs-domain)# copy run start

[########################################] 100%e

n1000v#(config-svs-domain)#

Step 4 You have completed this procedure. Return to the "Recovering a Primary VSM with Active Secondary" section.

---

Recovering a Primary VSM with Active Secondary

You can use the following process to recover a primary VSM when the secondary VSM is active.

---

Step 1 Stop the upgrade on the secondary VSM, using the "Stopping a VSM Upgrade" procedure

Step 2 Change the boot variables back to the previous version using the "Changing Boot Variables" procedure

Step 3 From the vCenter Server left-hand panel, right-click the primary VSM and then choose Delete from Disk.

The primary VSM is deleted.

Step 4 Create a new VSM by reinstalling the software from the OVA and specifying the following:

•Manual (CLI) configuration method instead of GUI.

•The host or cluster of the existing secondary VSM

For a detailed installation procedure, see the Cisco Nexus 1000V Software Installation Guide, Release 4.2(1)SV1(4).

Step 5 Make sure the port groups between the host server and VSM are not connected when the new VSM is powered on, using the "Disconnecting the Port Groups" procedure.

Step 6 Power on the newly-created VSM using the "Powering On the VSM" procedure.

The VSM comes up with the standalone HA role.

Step 7 Change the HA role of the newly-created standalone VSM to primary and save the configuration, using the "Changing the HA Role" procedure.

Step 8 Power off the newly-created VSM, using the "Powering Off the VSM" procedure.

Step 9 Make sure the port groups between the host server and VSM are connected when the new VSM is powered on, using the "Connecting the Port Groups" procedure.

Step 10 Power on the newly-created VSM using the "Powering On the VSM" procedure.

The VSM comes up, connects with the host server, and forms an HA pair with the existing primary VSM.

---

Disconnecting the Port Groups

Use this procedure to disconnect and prevent port groups to the VSM from connecting to the host server.

---

Step 1 In vCenter Server, select the VSM and then choose Edit > Settings.

The Virtual Machine Properties dialog box opens.

Step 2 Select the Control port group and uncheck the following Device Settings:

•Connected

•Connect at Power On

The connection from the VSM to the host server through the control port is dropped and is not restored when you power on the VSM.

Step 3 Select the Management port group and uncheck the following Device Settings:

•Connected

•Connect at Power On

The connection from the VSM to the host server through the management port is dropped and is not restored when you power on the VSM.

Step 4 You have completed this procedure. Return to the "Recovering a Primary VSM with Active Secondary" section.

---

Powering Off the VSM

Use this procedure to power off the newly-created VSM.

---

Step 1 From the vCenter Server left-hand panel, right-click the VSM and then choose Power > Power Off.

The VSM shuts down.

Step 2 You have completed this procedure. Return to the "Recovering a Primary VSM with Active Secondary" section.

---

Connecting the Port Groups

Use this procedure to make sure the port groups to the host connect when you power on the VSM.

---

Step 1 In vCenter Server, select the VSM and then choose Edit > Settings.

The Virtual Machine Properties dialog box opens.

Step 2 Select the Control port group and check the following Device Settings:

•Connect at Power On

When you power on the VSM, it will connect to the host server through the control port.

Step 3 Select the Management port group and check the following Device Setting:

•Connect at Power On

When you power on the VSM, it will connect to the host server through the management port.

Step 4 You have completed this procedure. Return to the "Recovering a Primary VSM with Active Secondary" section.

---

Collecting and Evaluating Logs

You can view details of upgrade errors in the following log file.

•N1KV_Upgrade*.log
(where * is a number, most likely 0)

This log file is automatically created and stored in the following directory of the machine running the upgrade application:

•C:\Documents and Settings\user_id\Local Settings\Temp

If an error occurs during the upgrade, an error message displays and the details are added to the logfile.

To search within the logfile, use the egrep -n 'start:|finish:' N1KV_Upgrade0.log command.

Upgrade Troubleshooting Commands

You can use the commands in this section to troubleshoot problems related to upgrade.

Command	Description
show boot	Displays boot variable definitions, showing the names of software images used to boot the VSM. See Example 4-2.
show svs upgrade status	Displays the upgrade status. See Example 4-3.
show svs connections	Displays the current connections between the VSM and the VMware host server. See Example 4-4.
show running-config | in boot	Displays the boot variables currently in the running configuration. See Example 4-8.
show startup-config | in boot	Displays the boot variables currently in the startup configuration. See Example 4-9.
show system redundancy status	Displays the current redundancy status for the VSM. See Example 4-5.
show module	Displays module status for active and standby VSMs. See Example 4-7.
show vmware vem upgrade status	Displays the upgrade status. See Example 4-6.

Example 4-2 show boot

n1000v# show boot 

sup-1

kickstart variable = bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3a.bin

system variable = bootflash:/nexus-1000v-mz.4.0.4.SV1.3a.bin

sup-2

kickstart variable = bootflash:/nexus-1000v-kickstart-mzg.4.2.1.SV1.4.bin

system variable = bootflash:/nexus-1000v-mzg.4.2.1.SV1.4.bin

No module boot variable set

n1000v#  

Example 4-3 show svs upgrade status

n1000v# show svs upgrade status 

Upgrade State: Start

Upgrade mgmt0 ipv4 addr: 1.1.1.1

Upgrade mgmt0 ipv6 addr:  

Upgrade control0 ipv4 addr:  

n1000v# 

Example 4-4 show svs connections

n1000v# show svs connections

connection vc:

    hostname: 172.23.232.139

    remote port: 80

    protocol: vmware-vim https

    certificate: default

    datacenter name: Hamilton-DC

    DVS uuid: 9b dd 36 50 2e 27 27 8b-07 ed 81 89 ef 43 31 17

    config status: Enabled

    operational status: Connected

    sync status: -

    version: -

n1000v#

Example 4-5 show system redundancy status

switch# show system redundancy status

Redundancy role

---------------

      administrative:   primary

         operational:   primary

Redundancy mode

---------------

      administrative:   HA

         operational:   HA

This supervisor (sup-1)

-----------------------

    Redundancy state:   Active

    Supervisor state:   Active

      Internal state:   Active with HA standby

Other supervisor (sup-2)

-----------------------

    Redundancy state:   Standby

    Supervisor state:   HA standby

      Internal state:   HA standby

switch# 

Example 4-6 show vmware vem upgrade status

n1000v# show vmware vem upgrade status

Upgrade VIBs: System VEM Image

Upgrade Status: 

Upgrade Notification Sent Time: 

Upgrade Status Time(vCenter):  

Upgrade Start Time: 

Upgrade End Time(vCenter): 

Upgrade Error: 

Upgrade Bundle ID:

    VSM: VEM400-201007101-BG

    DVS: VEM400-201007101-BG

n1000v# 

Example 4-7 show module

n1000v# show module 

Mod  Ports  Module-Type                      Model              Status

---  -----  -------------------------------- ------------------ ------------

1    0      Virtual Supervisor Module        Nexus1000V         ha-standby

2    0      Virtual Supervisor Module        Nexus1000V         active *

3    248    Virtual Ethernet Module          NA                 ok

Mod  Sw               Hw      

---  ---------------  ------  

1    4.0(4)SV1(3)     0.0    

2    4.0(4)SV1(3)     0.0    

3    4.2(1)SV1(4)     1.9    

Mod  MAC-Address(es)                         Serial-Num

---  --------------------------------------  ----------

1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA         

2    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA         

3    02-00-0c-00-03-00 to 02-00-0c-00-03-80  NA         

Mod  Server-IP        Server-UUID                           Server-Name

---  ---------------  ------------------------------------  --------------------

1    10.78.109.43     NA                                    NA

2    10.78.109.43     NA                                    NA

3    10.78.109.51     4220900d-76d3-89c5-17d7-b5a7d1a2487f  10.78.109.51

n1000v#

Example 4-8 show running-config | include boot

n1000v# show running-config | include boot 

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin sup-1

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin sup-2

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4.bin sup-2

n1000v#

Example 4-9 show startup-config | include boot

switch# show startup-config | include boot

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3b.bin sup-1

boot system bootflash:/nexus-1000v-mz.4.0.4.SV1.3b.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3b.bin sup-2

boot system bootflash:/nexus-1000v-mz.4.0.4.SV1.3b.bin sup-2

n1000v#