Cisco Nexus Dashboard Troubleshooting

This table provides an overview of the significant changes up to this current release. The table does not provide an exhaustive list of all changes or of the new features up to this release.

You can log in to any of the cluster nodes as rescue-user for a limited access to system data. You can use the following commands to perform various operations in Cisco Nexus Dashboard.

$ kubectl describe pod -n mon
  mond   mongodb

$ kubectl get app
  NAME                                    ADMIN  RUNNING  INSTALLSTATE  OPERSTATE
  apiserver-0.0.3v1.0.1046054b8           Enable  Enable  success       Enabled
  auditmgr-1.0.7v1.0.10e607c54            Enable  Enable  success       Enabled
  ceti-1.0.11v1.0.101ed08f3               Enable  Enable  success       Enabled
  cisco-intersightdc-2.7.0v1.0.1087eecfc  Enable  Enable  success       Enabled
  cisco-mso-5.1.0.140                     Enable  Enable  success       Enabled
  cisco-ndfc-12.4.1.190                   Enable  Enable  success       Enabled
  cisco-nir-6.7.0.166                     Enable  Enable  success       Enabled
  cisco-nir-flow-elasticsearch-6.7.0      Enable  Enable  success       Enabled
  cisco-nir-nae-rs-mongodb-6.7.0          Enable  Enable  success       Enabled
  gollum-6.7.0.166                        Enable  Enable  success       Enabled
  kafka-2.7.2v1.0.10b4a9d2b               Enable  Enable  success       Enabled
  mond-2.7.3v1.0.1081e937f                Enable  Enable  success       Enabled
  opensearch-2.14.0.0v1.0.10cc0ddce       Enable  Enable  success       Enabled
  sm-2.7.1v1.0.10716ad05                  Enable  Enable  success       Enabled
  statscollect-2.7.2v1.0.101ee723d        Enable  Enable  success       Enabled
  streaming-4.0.1v1.0.105ecd89b           Enable  Enable  success       Enabled

When you upgrade Nexus Dashboard software, you may also have to upgrade the Cisco UCS server firmware (which includes CIMC, BIOS, RAID controller, and disk and NIC adapter firmware) that is running in your Nexus Dashboard nodes.

Supported UCS server firmware versions for each Nexus Dashboard release are listed in the Release Notes specific to that release.

The following steps describe how to upgrade the Nexus Dashboard UCS server firmware using the Cisco Host Upgrade Utility (HUU). Additional details about the Host Upgrade Utility are available at Upgrading the Firmware on a Cisco UCS C-Series Server Using the HUU.

Upgrading the UCS server firmware version does not affect your production network as the Nexus Dashboard nodes are not in the data path of the traffic.

To upgrade the Nexus Dashboard UCS server firmware:

When you first receive the Nexus Dashboard physical hardware, it comes preloaded with the software image. If you simply want to configure the existing software, skip this section and proceed to Managing Secondary Nodes or Managing Standby Nodes.

This section describes how to redeploy the software stack on the Nexus Dashboard hardware. You may need to use the following steps in case of a catastrophic failure where you are no longer able to access the server’s operating system and GUI, or in case you want to deploy a different release that does not support direct upgrade or downgrade from your existing cluster.

If you are planning to re-install an existing Nexus Dashboard cluster, you must clean up the fabric and app information first. In this case, ensure that the fabrics are disabled in all applications and removed from the ND cluster before bringing it down.

Install Nexus Dashboard using bootable USB

You can also re-install Nexus Dashboard using Bootable USB drive. Note that you create a bootable USB drive from the Nexus Dashboard software image (.iso).

---

We recommend using a high speed USB 3.2 or later.

---

To create a bootable USB drive and reinstall the Nexus Dashboard software:

1. Download the Cisco Nexus Dashboard ISO image.

   1. Browse to the Nexus Dashboard page.

      https://www.cisco.com/c/en/us/support/data-center-analytics/nexus-dashboard/series.html

   2. Click the Downloads tab.

   3. Choose the Nexus Dashboard version you want to download and click Download Options.

   4. Download the Cisco Nexus Dashboard ISO image (nd-dk9.<version>.iso).

2. Plug in the USB to your computer.

3. To find the USB and the correspoding path:

   • Access the USB icon on Windows.

   • Enter fdisk -l on Linux.

   • Enter diskutil list on Mac.

4. Navigate to the directory where you downloaded the Nexus Dashboard image.

5. Create the bootable USB drive using the downloaded image.

   Use the following command:

   sudo dd if=nd-dk9.<version>.iso of=/dev/dsg iflag=direct oflag=direct
   status=progress bs=4M

   In the above command:

   • Replace nd-dk9.<version>.iso with the ISO image you downloaded in Step 1.

   • Replace /dev/dsg with the path to the USB drive.

     For example:

     sudo dd if=nd-dk9.4.1.1g.iso of=/dev/disk4 iflag=direct oflag=direct
     status=progress bs=4M

6. Unmount the USB drive safely from your system.

   For example, if you’re using USB on a Mac in the path /dev/disk4, run the following command.

   diskutil unmountDisk /dev/disk4

7. Install Nexus Dashboard software from the USB.

   This step requires you to connect to the server’s CIMC. Detailed information about CIMC configuration is available at https://www.cisco.com/c/en/us/support/servers-unified-computing/ucs-c-series-integrated-management-controller/products-installation-and-configuration-guides-list.html.

   1. Plug in the USB to pND (Physical Nexus Dashboard).

   2. SSH into the server’s CIMC and connect to the server’s console using the connect host command.

   3. Browse to the server’s CIMC address, boot it from the USB drive you created, and install the software.

      You can SSH into the CIMC and connect to the host’s console.

   4. Access the KVM console to monitor the installation by clicking Launch vKVM.

      C220-WZP23150D4C# connect host
      CISCO Serial Over LAN:
      Press Ctrl+x to Exit the session

8. Reboot the Nexus Dashboard server.

   1. Monitor the progress on the KVM console.

   2. As the server is starting up, press F6 to enter the boot menu and choose the USB drive you created in the previous step.

   3. Enter the password, if required.

   4. At the prompt to select the boot device, select the USB.

   5. Enter skip to install from the USB when you see the following prompt:

      + echo 'To speed up the install, enter iso url. Type '\"skip'\" to use local
      media:
      To speed up the install, enter iso url. Type '\"skip'\" to use local media:
      + echo ' Either http://server/path/to/file or nfs:server:/path/to/file are
      supported'
      Either http://server/path/to/file or nfs:server:/path/to/file are supported
      + read -r -p '?' url
      ? skip

      The Nexus Dashboard ISO image is copied to the server. Depending on the USB speed, the process can take up to an hour or more. Hence, we recommend using USB 3.2 or later.

      Once the installation is complete, the Nexus Dashboard server powers off.

9. Unmount the USB drive and power on the server.

   You can power the server off using the KVM or the poweroff command in the SSH console.

   After you finish installing Nexus Dashboard software, add the node(s) to your cluster.

   If you are building a brand new cluster, complete the deployment process as described in the "Deploying as Physical Appliance" chapter of the Nexus Dashboard Deployment Guide for your release.

   If you are adding an extra node to an existing cluster, see Managing Standby Nodes for additional info.

In some cases, you may need to re-build an existing cluster, for example if you want to change the data network’s subnet or the nodes' data IP addresses, which requires redeploying the cluster.

This section describes how to dynamically recover a primary cluster using a backup cluster, where one cluster is essentially the primary (active) cluster and the second cluster is the backup (standby) cluster. In this situation, the second cluster is available specifically as a backup to the first cluster, where the second cluster is always available to restore from the first cluster if that first cluster becomes unavailable. Refer to the Unified Backup and Restore for Nexus Dashboard and Services article for more information on the unified backup feature that is introduced in ND release 3.2.1.

The following sections provide the information necessary to set up the clusters and to perform a dynamic recovery should a cluster become unavailable:

Syslog events are not reaching the intended external events monitoring service.

You can reset the entire physical cluster by running the following command on each node:

# acs reboot factory-reset

Doing this will lose all cluster configuration and applications and you will need to rebuild the cluster.

If you have a virtual or cloud Nexus Dashboard cluster, we recommend simply deleting the existing VMs and re-deploying the entire cluster instead of reseting all the nodes, as described in the Cisco Nexus Dashboard Deployment Guide.

Changing the data network IP address is not supported. If you want to change the data IP address for the cluster nodes, you must re-create the cluster.

If you are running a single-node cluster, changing the management IP address is also not supported without re-creating the cluster.

If you are running a multi-node cluster, you can change the management IP addresses of one or more nodes as follows:

When you configure or change the proxy server in Nexus Dashboard, you may see a number of cisco-mso service: Replicaset() not in desired state errors in the Cluster Configuration page.

After the initial login using two-factor authentication, subsequent login attempts do not ask for username and password information and present a blank page instead.

You can view the installation logs by logging into your RHEL system and checking the /logs/ndlinux/ directory.

In order to run the common Nexus Dashboard troubleshooting commands described in the Troubleshooting sections, you must first access the Nexus Dashboard environment.

To access the Nexus Dashboard environment from your RHEL system:

When you onboard a Cisco APIC fabric to Nexus Dashboard, APIC configuration is updated to reflect the onboarding. If you subsequently import an earlier configuration in APIC, the fabric may show as unavailable in Nexus Dashboard or services.

To resolve the issue after it occurs, you can re-register the fabric in the Nexus Dashboard GUI:

This section describes how to re-add a primary node to a physical cluster. This scenario can happen if the node was accidentally or deliberately removed via configuration reset (such as acs reboot factory-reset) or vMedia re-install.

If you have a standby node in your cluster, simply convert the standby into a primary node as described in Replacing Single Primary Node with Standby Node and then add the old primary node as a new standby node as described in Adding Standby Nodes.

If you need to completely replace (RMA) a primary node due to hardware failure and do not have a standby node available, follow the procedure described in Replace single physical primary node without standby node instead.

To re-add the primary node to the same cluster:

This section describes how to recover from a primary node failure in a VMware ESX or Linux KVM virtual Nexus Dashboard cluster. The procedure involves deploying a brand new Nexus Dashboard node using the same form factor as the node which you are replacing and joining it as a primary node to the remaining cluster.

The following section describes how to recover from a single primary node failure in a physical Nexus Dashboard cluster without a standby node. This procedure is for hardware issues that require it to be physically replaced. If the node is simply in a bad software state, you can use the acs reboot clean commands instead and re-add the same node to the cluster as described in Re-add same primary node to physical cluster.

If your cluster has a standby node configured, we recommend using the steps described in Replacing Single Primary Node with Standby Node instead.

To replace a single failed primary node:

When replacing a failed secondary or standby node, you can simply delete the Inactive node from the GUI and then deploy a brand new secondary or standby node as you typically would.

To replace a failed secondary or standby node:

If a primary node is in an Inactive state, you have the option to suspend the node.

A node may be inactive due to temporary issues. The node will automatically rejoin the cluster once the underlying issues are resolved. You can also suspend an inactive node, if you know that the node will not recover. Suspending a node prevents it from interfering with other primary nodes. After you suspend the node, you must replace it through either a failover or a Return Merchandise Authorization (RMA) operation.

You can perform RMA operations on nodes in both Inactive and Suspended states.

The following sections list common issues with multi-cluster connectivity.

For additional information about connecting multiple clusters together, see Connecting Clusters.

This section provides an example of how to generate a private key, creat a certificate signing request (CSR), and obtain a certificate signed by a Certificate Authority (CA) for use in your Nexus Dashboard cluster.

If you want to generate both a key and a self-signed certificate, skip this section and follow the steps described in Generate private dey and self-signed certificate instead.

The configuration steps required to add the keys and certificates in the Nexus Dashboard GUI are described in the Configuring Users and Security.

This section provides an example of how to generate a private key and custom certificates should you want to use them in your Nexus Dashboard cluster.

If you want to use a CA-signed certificate, skip this section and follow the steps described in Generate private key, create CSR, and obtain CA-signed certificate.

The configuration steps required to add the keys and certificates in the Nexus Dashboard GUI are described in Configuring Users and Security.