Upgrading a Nexus Dashboard 4.x Cluster to This Release


Note


The procedures in this chapter are applicable if you are upgrading from Nexus Dashboard release 4.x to Nexus Dashboard release 4.3.1.

If you are upgrading from Nexus Dashboard release 3.2.2 to Nexus Dashboard release 4.3.1, follow the upgrade procedures provided in Upgrading a Nexus Dashboard 3.2.2 Cluster to This Release.


Prerequisites and guidelines for upgrading an existing Nexus Dashboard cluster

Before you upgrade your existing Nexus Dashboard cluster:

  • Ensure that you have read the target release's Release Notes for any changes in behavior, guidelines, and issues that may affect your upgrade.

  • Before upgrading to Nexus Dashboard release 4.3.1:

    • Make sure your NTP and DNS services are configured. At least one NTP and DNS are required for the system to upgrade successfully.

    • For LAN deployments, verify that the management network and data network are in different subnets. The upgrade will fail if the management network and data network are not in different subnets.

      Note that this restriction does not apply for SAN deployments (management network and data network can be in same subnet for SAN deployments).

    • We highly recommend that you use the Nexus Dashboard Preupgrade Validation script before performing any Nexus Dashboard upgrades. See Nexus Dashboard pre-upgrade validation script for more information.

  • Verify that the acs health is healthy.

    1. Access the Nexus Dashboard using ssh -l rescue-user {management-ip-of-nd}.

    2. Issue the acs health command.

    The output from the acs health command should show that all components are healthy:

    rescue-user@node1:~$ acs health 
    ======
    Status
    ======
    All components are healthy
  • Ensure that your current Nexus Dashboard cluster is healthy.

    You can check the system status on the Overview page of the Nexus Dashboard's Admin Console or by logging in to one of the nodes as rescue-user and ensuring that the acs health command returns All components are healthy.

  • If you have a multi-cluster federated Nexus Dashboard deployment, you must upgrade the primary cluster of the multi-cluster group to Nexus Dashboard version 4.3.1 or later before upgrading the local cluster.

  • Review these important guidelines about how backup and restore processes affect your upgrade:

    • Ensure that you perform a backup of your Nexus Dashboard cluster before upgrading and you store the backup file in a safe place. To perform a backup, refer to Backing Up and Restoring Your Nexus Dashboard.

    • An upgrade will not proceed if the most recent backup had a failure. Make sure you have a successful backup before proceeding with the upgrade. If you are unable to perform a successful backup and cannot upgrade, contact Cisco Technical Assistance Center (TAC) for support.

    • You will not be able to perform an upgrade if a restore has failed. After the restore has completed, click View History to navigate to the History area in the Backup and Restore page, as described in the "Restore Nexus Dashboard configurations" section in the Unified Backup and Restore for Nexus Dashboard and Services.

      The page should display Success in the Status column for the restore process. If you see any value other than Success in the Status column, redeploy the cluster and get a Success value for the restore process before attempting the upgrade again.

  • If you are upgrading a physical Nexus Dashboard cluster, ensure that the nodes have the minimum supported CIMC version for the target Nexus Dashboard release.

    Supported CIMC versions are listed in the Nexus Dashboard Release Notes for the target release.

    The CIMC upgrade is described in detail in the "Troubleshooting" article in the Nexus Dashboard documentation library.

  • If you are upgrading a virtual Nexus Dashboard cluster, Nexus Dashboard will enforce these checks:

    • A check of the HDD latency to verify that it is <30ms. If the HDD has a higher latency, the upgrade will fail.

    • A check of the network latency to verify that it is <50ms within cluster nodes. If the network has a highter latency, the upgrade will fail.

  • If you are upgrading a virtual Nexus Dashboard cluster deployed in VMware ESX, ensure that the ESX version is still supported by the target release.

    This release supports VMware ESXi 7.0, 7.0.1, 7.0.2, 7.0.3, 8.0, 8.0.2, 8.0.3.


    Note


    If you need to upgrade the ESX server, you must do that before upgrading your Nexus Dashboard. ESX upgrades are outside the scope of this document, but in short:

    1. Upgrade one of the ESX hosts as you typically would with your existing Nexus Dashboard node VM running.

    2. After the host is upgraded, ensure that the Nexus Dashboard cluster is still operational and healthy.

    3. Repeat the upgrade on the other ESX hosts one at a time.

    4. After all ESX hosts are upgraded and the existing Nexus Dashboard cluster is healthy, proceed with upgrading your Nexus Dashboard to the target release as described in this document.


  • Nexus Dashboard does not support platform downgrades.

    If you want to downgrade to an earlier release, you will need to deploy a new cluster.

Nexus Dashboard pre-upgrade validation script

We highly recommend that you use the Nexus Dashboard pre-upgrade validation script before performing any Nexus Dashboard upgrades. The Nexus Dashboard pre-upgrade validation script performs various checks for known issues that have been identified to affect the success of a Nexus Dashboard upgrade. The script is continuously updated and maintained in an effort to mitigate any new upgrade-related issues that are detected in the field.

  • Prior to Nexus Dashboard release 4.3.x, the Nexus Dashboard pre-upgrade validation script was a standalone Python script that you would download and run from an external host over SSH to perform health checks across nodes.

  • Beginning with Nexus Dashboard release 4.3.x, the Nexus Dashboard pre-upgrade validation script is now a signed plugin process, where you download the Cisco-signed .ndp (Nexus Dashboard plugin) file, which Nexus Dashboard validates and runs locally with elevated privileges.

Guidelines and limitations: Pre-upgrade validation script commands

These are the guidelines and limitations for the pre-upgrade validation script commands:

  • Only one plugin can be installed at a time. If you want to download the Nexus Dashboard pre-upgrade validation script plugin file and you already have a plugin file installed, you must remove the installed plugin first, then you can download and run another Nexus Dashboard pre-upgrade validation script plugin file. See Pre-upgrade validation script commandsfor more information.

Pre-upgrade validation script commands

The pre-upgrade validation script commands are acs CLI commands, similar to the Nexus Dashboard acs CLI commands that are described in the Cisco Nexus Dashboard Troubleshooting article. Log into any of the nodes in the Nexus Dashboard cluster to run these pre-upgrade validation script commands acs CLI commands.

These are the commands that are available for the new Nexus Dashboard .ndp version of the pre-upgrade validation script:

  • To download the Nexus Dashboard pre-upgrade validation script plugin file, enter:
    acs plugin download <url>

    where <url> is the plugin download URL. This command downloads, verifies, and installs the plugin file.

    The supported methods are positional arguments, and the supported download formats are:

    • HTTP download: http://<server>/<path>. For example:

      acs plugin download http://192.168.1.100/my-plugin.ndp

    • HTTPS download: https://<server>/<path>. For example:

      acs plugin download https://server.example.com/plugins/my-plugin.ndp

    • SFTP from remote secure storage: sftp://<server>:/<path>. For example:

      acs plugin download sftp://sftp-server.example.com:/export/plugins/my-plugin.ndp

    • SCP from remote secure storage: scp://<server>:/<path>. For example:

      acs plugin download scp://scp-server.example.com:/export/plugins/my-plugin.ndp


    Note


    • In order to use the SCP and SFTP download formats, you must first configure the remote storage. Refer to Create a remote storage location for those procedures and choose the SFTP/SCP Server option in the Remote Storage Location Type field.

    • SCP (scp://) also supports SFTP for backwards compatibility.


    If you already have a plugin file installed, you will see the message plugin already installed, run 'acs plugin delete' first. Since you can only have one installed plugin file at a time, determine if you want to replace the existing plugin file:

    • Enter acs plugin list to determine the version for the plugin that's currently installed. You can choose not to install the new plugin file if you see that the version is the same.

    • If you want to replace the existing plugin file, remove the installed plugin file using the acs plugin delete command as described below, then attempt to download the new plugin file again.

  • To run the Nexus Dashboard pre-upgrade validation script plugin file, enter:

    acs plugin run

    If you do not have a plugin installed, the output returns a no plugin installed message.

  • To show plugin file information, enter:

    acs plugin list
    • If you have a plugin installed, the output returns the name, version, and signed_by information.

    • If you do not have a plugin installed, the output returns a no plugin installed message.

  • To remove an installed plugin file, enter:

    acs plugin delete

    Since you can only have one installed plugin file at a time, use this command to remove an already installed plugin file if you are attempting to download a new plugin file.

Supported upgrade paths

As described in Nexus Dashboard deployment overview, in earlier releases, Nexus Dashboard shipped with only the platform software and no services included, which you would then download, install, and enable separately after the initial platform deployment. In addition, Nexus Dashboard release 3.1.1 introduced a tighter coupling between the Nexus Dashboard and individual services with only a single version of each service compatible with each version of the platform. As a result, as long as you were on the minimum required version of the Nexus Dashboard software, you could upgrade both the platform and all currently enabled services directly to Nexus Dashboard release 3.1x and 3.2x.

Beginning with Nexus Dashboard release 4.1.1, the platform and the individual services have been unified into a single product, which means that you no longer deploy, configure, or upgrade the services separately.

The following table provides a few example scenarios for specific deployment combinations:

Table 1.

Current Nexus Dashboard Release

Compatible Services

(depending on form factor and cluster size, you may have one or more of these services currently enabled)

Upgrade Workflow

4.2.1

N/A

Upgrade directly to release 4.3.1 as described in the following section.

4.1.1

N/A

Upgrade directly to release 4.3.1 as described in the following section.

3.2.2

Fabric Controller: 12.2(3)

Orchestrator: 4.4(2)

Insights: 6.5(2)

Upgrade directly to release 4.3.1 as described in Upgrading a Nexus Dashboard 3.2.2 Cluster to This Release.

All services are unified under a single Nexus Dashboard product in release 4.3.1.

3.2.1

Fabric Controller: 12.2(2)

Orchestrator: 4.4(1)

Insights: 6.5(1)

  1. Upgrade the Nexus Dashboard platform to release 3.2.2 as described in Nexus Dashboard Deployment Guide, Release 3.2.x

    All services will be automatically upgraded along with the platform.

  2. Upgrade from release 3.2.2 to release 4.3.1 as described in Upgrading a Nexus Dashboard 3.2.2 Cluster to This Release.

    All services are unified under a single Nexus Dashboard product in release 4.3.1.

3.1.1

Fabric Controller: 12.2(1)

Orchestrator: 4.3(x)

Insights: 6.4(1)

  1. Upgrade the Nexus Dashboard platform to release 3.2.2 as described in Nexus Dashboard Deployment Guide, Release 3.2.x

    All services will be automatically upgraded along with the platform.

  2. Upgrade from release 3.2.2 to release 4.3.1 as described in Upgrading a Nexus Dashboard 3.2.2 Cluster to This Release.

    All services are unified under a single Nexus Dashboard product in release 4.3.1.

3.0.1

Fabric Controller: 12.1(3)

Orchestrator: 4.2(x)

Insights: 6.3(1)

  1. Upgrade the Nexus Dashboard platform to release 3.2.2 as described in Nexus Dashboard Deployment Guide, Release 3.2.x

    All services will be automatically upgraded along with the platform.

  2. Upgrade from release 3.2.2 to release 4.3.1 as described in Upgrading a Nexus Dashboard 3.2.2 Cluster to This Release.

    All services are unified under a single Nexus Dashboard product in release 4.3.1.

2.3.2 and earlier

Fabric Controller: 12.1(2) or earlier

Orchestrator: 4.1(x) or earlier

Insights: 6.2(x) or earlier

  1. Upgrade the Nexus Dashboard platform to release 3.1.1 as described in Nexus Dashboard Deployment Guide, Release 3.1.x

    All services will be automatically upgraded along with the platform.

  2. Upgrade from release 3.1.1 to release 3.2.2 as described in Nexus Dashboard Deployment Guide, Release 3.2.x.

    All services will be automatically upgraded along with the platform.

  3. Upgrade from release 3.2.2 to release 4.3.1 as described in Upgrading a Nexus Dashboard 3.2.2 Cluster to This Release.

    All services are unified under a single Nexus Dashboard product in release 4.3.1.

Upgrade Nexus Dashboard

This section describes how to upgrade an existing Nexus Dashboard 4.x cluster to the Nexus Dashboard 4.3.1 release.

Before you begin

Ensure that you have completed the prerequisites described in Prerequisites and guidelines for upgrading an existing Nexus Dashboard cluster

Procedure


Step 1

In your Nexus Dashboard release 4.x system, download the Nexus Dashboard 4.3.1 image.

  1. Browse to the Software Download page.

    https://software.cisco.com/download/home/286327743/type/286328258/release/4.3.1

  2. Choose the Nexus Dashboard 4.3.1 release to download.

  3. Download the Nexus Dashboard image for the 4.3.1 release.

    Note

     
    • The upgrade process is the same for all Nexus Dashboard form factors and uses the Nexus Dashboard ISO image (nd-dk9.<version>.iso). In other words, even if you used the virtual form factors (such as the ESX .ova) or a cloud provider's marketplace for initial cluster deployment, you must still use the .iso image for upgrades.

    • If the image fails to download completely, verify the network connectivity between the Nexus Dashboard and the image server. Check the proxy configuration under Admin > System Settings > General > Proxy configuration.

  4. (Optional) Host the image on a web server in your environment.

    Note

     

    We recommend hosting the image on a server in your environment. When you upload the image to your Nexus Dashboard cluster, you will have an option to provide a direct URL to the image, which can significantly speed up the process.

Step 2

Log in to your current Nexus Dashboard's Admin Console as an Administrator user.

Step 3

Delete any older, non-active upgrade images from your cluster.

If this is the first time you're upgrading your cluster, you can skip this step.

  1. Navigate to Manage > Software Management.

  2. Click the trash icon on an upgrade image's tile to delete any older, non-active upgrade Images.

  3. Repeat this step for all older, non-active upgrade images.

Step 4

Upload the new image to the cluster.

  1. Navigate to Manage > Software Management.

  2. Click Add Image.

  3. In the Add Software Image window, select whether the image is Remote on a web server or Local on your machine.

    In both cases, the image will be a file ending with .iso.

    • Remote: Provide the URL to the image you downloaded in the first step.

    • Local: Click Choose file and navigate to the local folder where you downloaded the image.

  4. Click Add to add the image.

    Nexus Dashboard then downloads the upgrade image and starts processing the image, and goes through a number of preparation and validation stages to ensure successful upgrade. This may take several minutes to complete.

    Note

     

    See Troubleshooting upgrades for more information on the validation checks that occur during this point of the upgrade and how to deal with upgrade issues that might arise.

  5. After the validation is complete, these buttons appear in the card in the Nexus Dashboard Releases area in the System Software page.

    • Factory install: The Factory install option wipes all data, including bootstrap configurations. You will have to re-bootstrap the cluster afterward.

    • Upgrade: Click the Upgrade option if you are upgrading from a previous release. You do not have to perform a Factory install before upgrading.

    Click the appropriate button to install or upgrade the software.

    The installation progress window is displayed. You can navigate away from this screen while the update is in progress.

    This step may take up to 60 minutes or more, depending on the number of nodes in the cluster, during which the nodes will reboot and the GUI will not be accessible. Nexus Dashboard goes through several stages:

    • Install Release Firmware

    • Disable Services

    • Shutdown Infrastructure services

    • Update Platform Services

    • Enable Infrastructure Services

    • Enable Services

    You can click on the Details link to see the progress and the various stages of the upgrade.

    Note

     

    If you see any issues during the upgrade process, such as a possible indexing issue, refer to Prerequisites and guidelines for upgrading an existing Nexus Dashboard cluster for more information and possible workarounds.

    After the process above is complete, you should be upgraded to Nexus Dashboard 4.3.1.

    Note

     

    Depending on the cluster format and the number of cluster nodes that you have deployed, certain features (such as controller, orchestrator, or telemetry) might not be available. Review the information in the Nexus Dashboard Capacity Planning tool to verify what features would be available for your cluster installation.

Step 5

After the node upgrade tasks are completed, verify that the nodes are healthy and you can log into the UI.

Once the upgrade process completes, you can view the Nexus Dashboard UI as you typically would.

You can check the Overview page for overall system health and the Admin > System Software page to see the current Running version.

Step 6

Review the cluster name changes in DNS.

Beginning with Nexus Dashboard release 4.3.1, for fresh installs and upgrades, the cluster DNS will be hardcoded to nd-cluster.case.local. When upgrading from previous releases to Nexus Dashboard release 4.3.1, the DNS domain name that you see based on the cluster name will change after the upgrade, and you will see the fixed cluster name (nd-cluster.case.local) rather than the cluster name within that DNS domain.


Troubleshooting upgrades

After all the nodes restart during new image activation stage described in the previous section, you may log in to the GUI to check the status of the upgrade workflows. Initially, you can see the bootstrap process similar to the initial cluster deployment and once the nodes come up, you can see additional information about service activation in the GUI's Overview page.

In case the upgrade fails for any reason, the GUI will display the error and additional workaround steps. For example, you might then see an error message, along with the remedy, similar to this:

Failed to activate 
Upgrade failed while shutting down the cluster: Operation Timedout, last status: Operation Timedout

Please login to one of the primary nodes as 'rescue-user' and follow the steps provided by the upgrade recovery 
helper by invoking following command: 'acs upgrade recover Cluster Shutdown'. If the issue persists, please contact Cisco TAC for assistance.

If an issue persists, click Admin to access Tech support. See Working with Cisco Tech Support for more information.