When the cluster is in a bad state as part of the upgrade to releases earlier than Cisco vManage Release 20.6.1, you should manually restart the NMS processes. Restart the processes one at a time
in an orderly manner instead of using
request nms all restart
or a similar command. The following manual restart order might vary for your
cluster, depending on what services you are running on the Cisco vManage devices in the
cluster. The following order is based on a basic cluster with three Cisco vManage devices.
-
On each Cisco vManage
device, stop all the NMS services:
request nms all stop
-
Verify that all the services have stopped. It is normal for the
request nms all stop command to display a message
about failing to stop a service if it takes too long. So use the following
command to verify that everything is stopped before proceeding further:
request nms all status
-
Start the Statistics database on each device that is configured to run it. Wait for the service to start each time before
proceeding to the next Cisco vManage device.
request nms statistics-db start
-
Verify that the service is started before proceeding to start it on the next
vManage. After the service starts, perform step 3 to start the Statistics
database on the next Cisco vManage device.
After all the Cisco vManage devices
have the Statistics database running, proceed to the next step.
request nms statistics-db status
-
Start the Configuration database on each device that is configured to run it. Wait for the service to start each time before
proceeding to the next Cisco vManage device.
request nms configuration-db start
-
For releases earlier than Cisco vManage Release 20.3.1, verify that the service has started before proceeding to start it on the
next Cisco vManage
device. Go to vshell and tail a log file to look for a message that the database
is online. After confirming, go to step 5 to start the Configuration database on
the next Cisco vManage
device. After all the Cisco vManage devices
have the Configuration database running, proceed to the next step.
tail -f -n 100 /var/log/nms/vmanage-neo4j-out.log
-
Start the Coordination server on each device. Wait for the service to start each time before proceeding to the next Cisco vManage device.
request nms coordination-server start
-
Verify that the service is started before proceeding to start it on the next vManage device. After verifying, go to step 7
to start the Coordination server on the next Cisco vManage device. After the Coordination server runs on all the Cisco vManage devices, proceed to the next step.
request nms coordination-server status
-
Start the Messaging server on each device. Wait for the service to start each time before proceeding to the next Cisco vManage device.
request nms messaging-server start
-
Verify that the service has started before proceeding to start the service on the
next Cisco vManage
device. After verifying, go to step 9 to start the Messaging server on the next
Cisco vManage
device. After the Messaging server runs on all the Cisco vManage devices,
proceed to the next step.
request nms messaging-server status
-
Start the Application server on each device. Wait for the service to start each time before proceeding to the next Cisco vManage device.
request nms application-server start
-
For Cisco vManage Release 20.3.1 and later releases, start the server-proxy service on each Cisco vManage device:
request nms server-proxy start
To verify that the service is fully started, open the GUI of that Cisco vManage device.
After the GUI is fully loaded and you are able to log in, start the server-proxy
service on the next Cisco vManage
device.
-
Restart the NMS cloud services on each device. Wait for the services to start
each time before proceeding to the next Cisco vManage
device.
You can verify that the cloud services are running by entering the following
commands:
request nms cloud-agent status
request nms cloud-agent-v2 status
Verify that the service has started before proceeding to start it on the next Cisco vManage device.
After verifying, start the cloud services on the next Cisco vManage device.
After the cloud services run on all the Cisco vManage devices,
continue to the next step.
-
To verify that there are no errors and everything has loaded cleanly, tail the log files.