When you start Cisco vManage for the first time, the default IP address of the Cisco vManage server is shown as localhost. Before you can add a new Cisco vManage server to a cluster, you must change the localhost address of the primary Cisco vManage server to an out-of-band IP address. (From Cisco vManage Release 20.6.1, the primary Cisco vManage server has the Compute+Data persona.) Servers in the cluster use this out-of-band IP address to communicate with each other.

If you need to change the out-of-band IP address in the future, contact your Cisco support representative.

Cluster interconnection between Cisco vManage servers requires that each of the servers be assigned a static IP address. Do not use DHCP to assign IP addresses to Cisco vManage servers that are to be a part of a cluster. Configure the IP address on a nontunnel interface in VPN 0.

Before you configure the cluster IP address of a Cisco vManage server, ensure that out-of-band IP addresses have been configured on VPN0 for its server interfaces. This configuration typically is done when the server is provisioned. The port type for an out-of-band IP address must be service for the IP address to be available for assigning to a Cisco vManage server.

Configure the IP Address for Releases Before Cisco vManage Release 20.6.1

Configure the IP address of a Cisco vManage server before you add the server to the cluster. To do so for releases before Cisco vManage Release 20.6.1, follow these steps:

1. From the Cisco vManage menu, choose Administration > Cluster Management and click Service Configuration.

2. Click Add vManage.

   The Edit vManage dialog box opens.

3. From the vManage IP Address drop-down list, choose an IP address to assign to the Cisco vManage server.

4. Enter the user name and password for logging in to the Cisco vManage server.

5. Click Update.

The Cisco vManage server reboots and displays the Cluster Management window.

Configure the IP Address for Cisco vManage Release 20.6.1 and Later Releases

Configure the IP address of a Cisco vManage server before you add the server to the cluster. To do so from Cisco vManage Release 20.6.1, perform the following steps.

Perform this procedure on the primary Cisco vManage server (which has the Compute+Data persona).

1. From the Cisco vManage menu, choose Administration > Cluster Management.

   The Cluster Management window is displayed. The table on this window lists the Cisco vManage servers that are in the cluster.

2. Click ... adjacent to the Cisco vManage server to configure and click Edit.

   The Edit vManage dialog box is displayed.

3. In the Edit vManage dialog box, perform the following actions.

   Note	You cannot change the persona of a server. So the Node Persona options are disabled.

   1. From the vManage IP Address drop-down list, choose an out-of-band static IP address to assign to the server.

   2. In the Username field, enter the user name for logging in to the server.

   3. In the Password field, enter the password for logging in to the server.

   4. (Optional) Click Enable SD-AVC if you want Cisco Software-Defined Application Visibility and Control (SD-AVC) to run on the server.

      Cisco SD-AVC is a component of Cisco Application Visibility and Control (AVC). It can be enabled on only one Cisco vManage server. The server on which it is enabled must have the Compute+Data or the Compute persona. Cisco SD-AVC cannot be enabled on a server that has the Data persona.

   5. Click Update.

      The server reboots and displays the Cluster Management window.

Add a Cisco vManage Server to a Cluster for Releases Before Cisco vManage Release 20.6.1

To add a new Cisco vManage server to a cluster for releases before Cisco vManage Release 20.6.1, perform the following steps on the primary Cisco vManage server.

Before you begin, ensure that the default IP address of the Cisco vManage server has been changed to an out-of-band IP address as described in Configure the Cluster IP Address of a Cisco vManage Server.

1. From the Cisco vManage menu, choose Administration > Cluster Management and click Service Configuration.

2. Click Add vManage.

   The Edit vManage window opens.

3. In the vManage IP Address field, select an IP address to assign to the Cisco vManage server.

4. Enter the username and password for logging in to the Cisco vManage server.

5. Enter the IP address of the Cisco vManage server that you are adding to the cluster.

6. Specify the username and password for the new Cisco vManage server.

7. Select the services to be run on the Cisco vManage server. You can select from the services listed below. Note that the Application Server field is not editable. The Cisco vManage Application Server is the local Cisco vManage HTTP web server.

   • Statistics Database: Stores statistics from all the Cisco SD-WAN devices in the network.

   • Configuration Database: Stores all the device and feature templates and configurations for all the Cisco SD-WAN devices in the network.

   • Messaging Server: Distributes messages and shares state among all the Cisco vManage cluster members.

8. Click Add.

   The Cisco vManage server that you just added reboots before joining the cluster.

Note

In a cluster, we recommend that you run at least three instances of each service. When you add the first two compute or compute+data nodes to the cluster, the host node’s application-server is unavailable. The following message is displayed on the host node’s GUI, before the application-server shuts down in the host node: \Node added to the cluster. The operation may take up to 30 minutes and may cause application-server to restart in between. Once the application server is back online, the post cluster operation progress can be viewed under tasks pop-up\.

Add a Cisco vManage Server to a Cluster for Cisco vManage Release 20.6.1 and Later Releases

From Cisco vManage Release 20.6.1, a cluster supports any of the following deployments of nodes:

• Three Compute+Data nodes

• Three Compute+Data nodes and three Data nodes

• Three Compute nodes and three Data nodes (supported only in an upgrade from an existing deployment)

If you require a different combination of nodes, contact your Cisco representative.

To add a Cisco vManage server to a cluster from Cisco vManage Release 20.6.1, perform the following steps.

Perform this procedure on a Compute+Data node or a Compute node. Performing this procedure on a Data node is not supported because a Data node does not run all the services that are required for the addition.

Do not add a server that was a member of the cluster and then removed from the cluster. If you need to add that server to the cluster, bring up a new VM on that server to be used as the node to add.

Before you begin, ensure that the default IP address of the Cisco vManage server has been changed to an out-of-band IP address, as described in Configure the Cluster IP Address of a Cisco vManage Server.

1. From the Cisco vManage menu, choose Administration > Cluster Management.

   The Cluster Management page window appears. The table on this window shows the Cisco vManage servers that are in the cluster.

2. Click Add vManage.

   The Add vManage dialog box opens.

   Note	If the Edit vManage dialog box opens, configure an out-of-band IP address for the server, as described in Configure the Cluster IP Address of a Cisco vManage Server, and then repeat this procedure for adding a server.

3. In the Add vManage dialog box, perform the following actions:

   1. Click the Node Persona option (Compute+Data, Compute, or Data) that corresponds to the persona that has been configured for the server.

      You can determine the persona of a server by logging in to the server and looking at the persona display on the Administration > Cluster Management window. If you choose an incorrect persona, a message displays the persona that you should choose.

   2. From the vManage IP Address drop-down list, choose the IP address of the server to be added to the cluster.

   3. In the Username field, enter the user name for logging in to the server.

   4. In the Password field, enter the password for logging in to the server.

   5. (Optional) Click Enable SD-AVC if you want Cisco Software-Defined Application Visibility and Control (SD-AVC) to run on the server.

      Cisco SD-AVC is a component of Cisco Application Visibility and Control (AVC). It can be enabled on one Cisco vManage server. The server on which it is enabled must have the Compute+Data or the Compute persona. Cisco SD-AVC cannot be enabled on a server that has the Data persona.

      If you enabled Cisco SD-AVC for this server when you changed its IP address, the Enable SD-AVC check box is checked by default.

   6. Click Add.

   7. To confirm, click OK.

      The dialog box indicates that the services will restart, and that the existing metadata and other information that is not required when the server joins the cluster will be deleted from the server.

      When you click OK, the system starts the server add operation. The Cluster Management window displays the tasks that the system performs as it adds the server.

      As part of this operation, the system checks the compatibility of the server that you are adding. This check ensures that the server has sufficient disk space, and that the persona that you specified matches the persona of the node.

      After the server is added, the system performs a cluster sync operation, which rebalances the services in the cluster. Then the Cisco vManage servers in the cluster restart.

View Statistics Database Space Usage

To view the amount of space available for and utilized by the statistics database on the local Cisco vManage server, choose Administration > Settings > Statistics Database Configuration and click View. The top of the window shows the maximum space available for the database and the total amount of space currently being utilized. The table shows the disk space currently being utilized for each statistics type.

For information about disk size recommendations and requirements, see Server Recommendations for your release in Cisco SD-WAN Controller Compatibility Matrix and Server Recommendations.

Configure Statistics Database

To configure the statistics database that stores all the real-time statistics from the local Cisco SD-WAN Controller Compatibility Matrix and Server Recommendations:

1. From the Cisco vManage menu, choose Administration > Settings.

2. In the Statistics Database Configuration section, click Edit to view the maximum space available for the database.

3. For each field in the Statistics Type column, assign the amount of storage to be allocated, in gigabytes (GB). The total value of all the fields cannot exceed the maximum available space.

4. Click Save.

Cisco vManage updates the storage allocations that you have assigned once a day, at midnight.

View Detailed Information about Services

To view detailed information about the services running on a Cisco vManage server:

1. From the Cisco vManage menu, choose Administration > Cluster Management and click Service Configuration.

2. Click the hostname of the Cisco vManage server.

   The vManage Details window opens, displaying the process IDs of all the Cisco vManage services that are enabled on Cisco vManage.

3. Click Cluster Management in the breadcrumb in the title bar to return to the Cluster Management window.

View Devices Connected to Cisco vManage

To view the list of devices connected to Cisco vManage:

1. From the Cisco vManage menu, choose Administration > Cluster Management and click Service Configuration.

2. Click the hostname of the Cisco vManage server.

3. Click Managed Devices.

Alternatively:

1. From the Cisco vManage menu, choose Administration > Cluster Management and click Service Configuration.

2. Click ... adjacent to the Cisco vManage server and choose Device Connected.

3. If a device is connected to Cisco vManage from a cluster, ensure that you do not configure the data stream hostname to the Cisco vManage system IP address. However, you can configure the management IP address on VPN 512 or the internet public IP address on VPN 0. For information about data stream troubleshooting tools, see Data Stream Troubleshooting Tools FAQ.

You can edit various parameters for a Cisco vManage server that has been added to a cluster. To do so, follow these steps:

1. From the Cisco vManage menu, choose Administration > Cluster Management and click Service Configuration.

2. Click ... adjacent to the Cisco vManage server to edit, and click Edit.

   The Edit vManage window opens.

3. Select an IP address to edit.

4. Enter the username and password, and edit parameters for the selected Cisco vManage server.

   • For releases before Cisco vManage Release 20.6.1, you can edit the cluster services.

   • From Cisco vManage Release 20.6.1, you can change the IP address to another IP address that appears in the vManage IP Address drop-down list, change the Cisco SD-AVC setting, or change the username and password if the server credentials have changed.

5. Click Update.

The default username of the configuration database is neo4j and the default password is password. To update the default login credentials of the configuration database, access Cisco vManage using a terminal and run the following commands. Do not use the SSH terminal option in Cisco vManage to run these commands. Doing so causes you to lose access to Cisco vManage.

1. Use request nms application-server stop to stop application servers on all the Cisco vManage servers whether configuration-db is enabled or not.

   .

2. Use one of the following commands to reset the user name and password for the configuration database on all the Cisco vManage servers:

   • For Cisco SD-WAN Release 20.1.1 and earlier:

     request nms configuration-db update-admin-user username username password password newusername newadminuser newpassword newpassword

   • For releases from Cisco SD-WAN Release 20.1.2:

     request nms configuration-db update-admin-user

     When prompted, enter your current username and password, and your new username and password.

     When you run one of these commands, Cisco vManage restarts the application server

   Note

   If you do not know the default credentials of the configuration database, contact your Cisco support representative to retrieve the credentials. You cannot use a previous username. Passwords can include only a mix of characters A to Z ( upper or lowercase), digits 0 to 9, and special characters @, #, and *.

   Example

   • For Cisco SD-WAN Release 20.1.1 and earlier:

     request nms configuration-db update-admin-user username neo4j password ******** newusername myusername newpassword mypassword

   • For releases from Cisco SD-WAN Release 20.1.2:

     request nms configuration-db update-admin-user

     Enter current user name: neo4j

     Enter current user password: password

     Enter new user name: myusername

     Enter new user password: mypassword

Note	After a configuration database admin user update, if you are unable to view a specific Cisco vManage instance, use the request nms application-server restart command to restart the application server on that Cisco vManage instance again.

Note	Starting from Cisco SD-WAN Release 20.6.1, neo4j takes time to replicate the new user and password to all the Cisco vManage nodes in the cluster. When you use the request nms configuration-db update-admin-user command, we recommend that you do not push the CLI to all the nodes at the same time.

This section describes how to upgrade Cisco vManage in a cluster.

You can upgrade directly from Cisco vManage 20.3.1 or later releases to Cisco vManage Release 20.6.1. To upgrade from earlier releases, first upgrade to Cisco vManage 20.4.2 or Cisco vManage Release 20.5.1.

If you are upgrading a Cisco vManage cluster deployment from Cisco vManage Release 20.3.1 or later to Cisco vManage Release 20.5.1 or later, you must do it through the CLI.

Before You Begin

Before you upgrade Cisco vManage nodes to Cisco vManage Release 20.6.1 or later releases, verify the following:

• Ensure that the internal user account vmanage-admin is not locked for any server that you are upgrading.

  You can check the status of this admin account by pushing a template to the devices that are connected to the server. The push fails if the account is locked. In such a scenario, you can unlock the account by using the request aaa unlock-user vmanage-admin command.

• Ensure that PKI keys have been exchanged between the servers that you are upgrading.

  To do so, ensure that the control connections are in the UP state on the servers and restart the application server.

• Ensure that the out-of-band IP address of each server is reachable.

• Ensure that the Cisco vManage UI is accessible on all servers in the cluster.

• Ensure that DCA is running on all nodes in the cluster.

  To do so, use the request nms data-collection-agent status command and ensure that the status value shows running for each node.

  To start DCA, if needed, use the request nms data-collection-agent start command.

• If you are upgrading from a six-node Cisco vManage cluster, disable the messaging-server service on each Cisco vManage node that is running the Elasticsearch service.

  To identify these nodes, choose Administration > Cluster Management > Service Configuration and, for each node, look at the services that are listed in the vManage Details page for the node. Then disable the messaging-server service from the vManage Details page.

Note

If these prerequisites are not met or if another error occurs during the upgrade, the activation of the image fails and a file named upgrade-context.json is created in the /opt/data/extra-packages/image-version folder on each node in the cluster. You can provide this file to your Cisco representative for assistance with resolving the issue.

If you are upgrading to Cisco vManage Release 20.6.1 or later releases from a six-node Cisco vManage cluster deployment in which not all services are running on all nodes, contact your Cisco support representative before performing the upgrade.

1. Take snapshots of all the vManage servers. Take a backup of the configuration database and save it in a location outside of the Cisco vManage server using the following command:

   request nms configuration-db backup path path_and_filename

2. Ensure that Cisco vManage Release 18.3 or later is installed.

3. For upgrades from Cisco vManage Release 20.3.1 or later, copy the current image to each Cisco vManage server in the cluster and install the image on each Cisco vManage server by using the following command. Do not activate the image at this time.

   request software install path

4. Stop NMS services on all the Cisco vManage nodes in the cluster using the following command on each Cisco vManage node:

   request nms all stop

5. For upgrades from Cisco vManage Release 20.3.1 or later, activate the current image on each Cisco vManage server using the following command. All servers reboot simultaneously.

   request software activate version

6. You must upgrade the configuration database when upgrading from:

   • Cisco vManage Release 18.4.x or 19.2.x to Cisco vManage 20.3.x or 20.4.x

   • Cisco vManage Release 20.3.x or 20.4.x to Cisco vManage Release 20.5.x or 20.6.x

   Before you upgrade the configuration database, ensure that you have activated the current image on each Cisco vManage server in the cluster as described in the previous step. In addition, ensure that all services except the application server and configuration-db services are running on these servers by entering the request nms all status command on each server.

   To upgrade the configuration database, enter the following command on any node in the cluster that has the configuration database enabled. Enter this command on one node only. You can verify that the node that you selected has the configuration database enabled by entering the request nms configuration-db status command on the node and making sure that the configuration-db status shows as enabled, but not as running.

   request nms configuration-db upgrade

   Note	Do not enter this command if you are upgrading from Cisco vManage Release 20.5.x to Cisco vManage Release 20.6.1 or later. It can take some time for this service to come up. Use the request nms configuration-db status command to check the status of this service.

   Enter your login credentials, if prompted. Login credentials are prompted in releases earlier than Cisco vManage Release 20.3.1 if all the vManage servers establish control connection with each other. After a successful upgrade, all the configuration database services are UP across the cluster, and the application server is started.

You can check the database upgrade logs at the following location: vmanage-server:/var/log/nms/neo4j-upgrade.log.

For information about how to upgrade Cisco vManage clusters by using the Cisco vManage GUI, see the Upgrade the Software Image on a Device section in Cisco SD-WAN Monitor and Maintain Configuration Guide.

You can remove a Cisco vManage node from a cluster, if necessary.

In releases earlier than Cisco vManage Release 20.6.1, you can only remove n - 2 Cisco vManage nodes from a cluster of n nodes. You must retain at least two Cisco vManage nodes in a cluster.

From Cisco vManage Release 20.6.1, you must retain at least two Cisco vManage nodes that include the compute capability and at least one node that includes the data capability. That is, the cluster must retain any of the following:

• At least two Cisco vManage nodes that include the Compute+Data persona

• At least one Cisco vManage nodes that includes the Compute+Data persona and one Cisco vManage node that includes the Compute persona

• At least two Cisco vManage nodes that include the Compute persona and one Cisco vManage node that includes the Data persona

From Cisco vManage Release 20.6.1, if a Cisco vManage node is reachable when you remove it from a cluster, Cisco vManage automatically performs a factory reset operation on the removed node to ensure that the node does not join the cluster again. If a Cisco vManage node is unreachable when you remove it from a cluster, a factory reset operation is not performed on the node. In this situation, the node is added back to the cluster automatically when the node becomes reachable. To prevent the node from being added back to the cluster, enter the command request software reset from the CLI of the node after the node is removed from the cluster.

To remove a Cisco vManage node from a cluster, follow these steps:

1. From the Cisco vManage, choose Administration > Cluster Management and click Service Configuration.

2. Click … adjacent to the Cisco vManage instance that you want to remove and click Remove.

   The Remove vManage dialog box opens.

3. Enter the username and password to confirm the removal of the device from the network.

4. Click Remove.

The Cisco vManage instance is removed from the cluster, the certificates for that Cisco vManage are deleted, and Cisco vManage undergoes a factory reset.