When you start Cisco SD-WAN Manager for the first time, the default IP address of the Cisco SD-WAN Manager server is shown as localhost. Before you can add a new Cisco SD-WAN Manager server to a cluster, you must change the localhost address of the primary Cisco SD-WAN Manager server to an out-of-band IP address. (From Cisco vManage Release 20.6.1, the primary Cisco SD-WAN Manager server has the Compute+Data persona.) Servers in the cluster use this out-of-band IP address to communicate with each other.

If you need to change the out-of-band IP address in the future, contact your Cisco support representative.

Cluster interconnection between Cisco SD-WAN Manager servers requires that each of the servers be assigned a static IP address. We recommend that you do not use DHCP to assign IP addresses to Cisco SD-WAN Manager servers that are to be a part of a cluster. Configure the IP address on a nontunnel interface in VPN 0.

Before you configure the cluster IP address of a Cisco SD-WAN Manager server, ensure that out-of-band IP addresses have been configured on VPN0 for its server interfaces. This configuration typically is done when the server is provisioned. The port type for an out-of-band IP address must be service for the IP address to be available for assigning to a Cisco SD-WAN Manager server.

Note	From Cisco vManage Release 20.11.1, some alarms display the hostname as localhost during the cluster setup for the first time as the system-ip/hostname is not configured in Cisco SD-WAN Manager. When the system-ip/hostname is configured, the alarms display the correct hostname.

Configure the IP Address for Releases Before Cisco vManage Release 20.6.1

Configure the IP address of a Cisco SD-WAN Manager server before you add the server to the cluster. To do so for releases before Cisco vManage Release 20.6.1, follow these steps:

1. From the Cisco SD-WAN Manager menu, choose Administration > Cluster Management and click Service Configuration.

2. Click Add vManage.

   The Edit vManage dialog box opens.

3. From the vManage IP Address drop-down list, choose an IP address to assign to the Cisco SD-WAN Manager server.

4. Enter the user name and password for logging in to the Cisco SD-WAN Manager server.

5. Click Update.

The Cisco SD-WAN Manager server reboots and displays the Cluster Management window.

Configure the IP Address for Cisco vManage Release 20.6.1 and Later Releases

Configure the IP address of a Cisco SD-WAN Manager server before you add the server to the cluster. To do so from Cisco vManage Release 20.6.1, perform the following steps.

Perform this procedure on the primary Cisco SD-WAN Manager server (which has the Compute+Data persona).

1. From the Cisco SD-WAN Manager menu, choose Administration > Cluster Management.

   The Cluster Management window is displayed. The table on this window lists the Cisco SD-WAN Manager servers that are in the cluster.

2. Click ... adjacent to the Cisco SD-WAN Manager server to configure and click Edit.

   The Edit vManage dialog box is displayed.

3. In the Edit vManage dialog box, perform the following actions.

   Note	You cannot change the persona of a server. So the Node Persona options are disabled.

   1. From the vManage IP Address drop-down list, choose an out-of-band static IP address to assign to the server.

   2. In the Username field, enter the user name for logging in to the server.

   3. In the Password field, enter the password for logging in to the server.

   4. (Optional) Click Enable SD-AVC if you want Cisco Software-Defined Application Visibility and Control (SD-AVC) to run on the server.

      Cisco SD-AVC is a component of Cisco Application Visibility and Control (AVC). It can be enabled on only one Cisco SD-WAN Manager server. The server on which it is enabled must have the Compute+Data or the Compute persona. Cisco SD-AVC cannot be enabled on a server that has the Data persona.

      Note

      If Cisco SD-WAN Manager is set up as a cluster and the cluster crashes as a result of a reboot or upgrade, the connection to the edge device is reset and the custom app ceases to function. To resolve this and to resume operation, redefine the custom application name with a new, unique name. For more information to define custom applications, see the Define Custom Applications Using ​Cisco Catalyst SD-WAN Manager chapter of the ​Cisco Catalyst SD-WAN Policies Configuration Guide.

   5. Click Update.

      The server reboots and displays the Cluster Management window.

Add a Cisco SD-WAN Manager Server to a Cluster for Releases Before Cisco vManage Release 20.6.1

To add a new Cisco SD-WAN Manager server to a cluster for releases before Cisco vManage Release 20.6.1, perform the following steps on the primary Cisco SD-WAN Manager server.

Before you begin, ensure that the default IP address of the Cisco SD-WAN Manager server has been changed to an out-of-band IP address as described in Configure the Cluster IP Address of a Cisco Catalyst SD-WAN Manager Server.

1. From the Cisco SD-WAN Manager menu, choose Administration > Cluster Management and click Service Configuration.

2. Click Add vManage.

   The Edit vManage window opens.

3. In the vManage IP Address field, select an IP address to assign to the Cisco SD-WAN Manager server.

4. Enter the username and password for logging in to the Cisco SD-WAN Manager server.

5. Enter the IP address of the Cisco SD-WAN Manager server that you are adding to the cluster.

6. Specify the username and password for the new Cisco SD-WAN Manager server.

7. Select the services to be run on the Cisco SD-WAN Manager server. You can select from the services listed below. Note that the Application Server field is not editable. The Cisco SD-WAN Manager Application Server is the local Cisco SD-WAN Manager HTTP web server.

   • Statistics Database: Stores statistics from all the Cisco Catalyst SD-WAN devices in the network.

   • Configuration Database: Stores all the device and feature templates and configurations for all the Cisco Catalyst SD-WAN devices in the network.

   • Messaging Server: Distributes messages and shares state among all the Cisco SD-WAN Manager cluster members.

8. Click Add.

   The Cisco SD-WAN Manager server that you just added reboots before joining the cluster.

Note

In a cluster, we recommend that you run at least three instances of each service. When you add the first two compute or compute+data nodes to the cluster, the host node’s application-server is unavailable. The following message is displayed on the host node’s GUI, before the application-server shuts down in the host node: \Node added to the cluster. The operation may take up to 30 minutes and may cause application-server to restart in between. Once the application server is back online, the post cluster operation progress can be viewed under tasks pop-up\. Starting Cisco IOS XE Catalyst SD-WAN Release 17.12.1a, ensure that you disable the HTTP/HTTPs Proxy option in the Cisco SD-WAN Manager settings, before adding a node to the cluster.

Add a Cisco SD-WAN Manager Server to a Cluster for Cisco vManage Release 20.6.1 and Later Releases

From Cisco vManage Release 20.6.1, a cluster supports any of the following deployments of nodes:

• Three Compute+Data nodes

• Three Compute+Data nodes and three Data nodes

  Note	DATA nodes should be added only after 3 node cluster with CONFIG+DATA is added.

• Three Compute nodes and three Data nodes (supported only in an upgrade from an existing deployment)

If you require a different combination of nodes, contact your Cisco representative.

To add a Cisco SD-WAN Manager server to a cluster from Cisco vManage Release 20.6.1, perform the following steps.

Perform this procedure on a Compute+Data node or a Compute node. Performing this procedure on a Data node is not supported because a Data node does not run all the services that are required for the addition.

Do not add a server that was a member of the cluster and then removed from the cluster. If you need to add that server to the cluster, bring up a new VM on that server to be used as the node to add.

Before you begin, ensure that the default IP address of the Cisco SD-WAN Manager server has been changed to an out-of-band IP address, as described in Configure the Cluster IP Address of a Cisco Catalyst SD-WAN Manager Server.

1. From the Cisco SD-WAN Manager menu, choose Administration > Cluster Management.

   The Cluster Management page window appears. The table on this window shows the Cisco SD-WAN Manager servers that are in the cluster.

2. Click Add vManage.

   The Add vManage dialog box opens.

   Note	If the Edit vManage dialog box opens, configure an out-of-band IP address for the server, as described in Configure the Cluster IP Address of a Cisco Catalyst SD-WAN Manager Server, and then repeat this procedure for adding a server.

3. In the Add vManage dialog box, perform the following actions:

   1. Click the Node Persona option (Compute+Data, Compute, or Data) that corresponds to the persona that has been configured for the server.

      You can determine the persona of a server by logging in to the server and looking at the persona display on the Administration > Cluster Management window. If you choose an incorrect persona, a message displays the persona that you should choose.

   2. From the vManage IP Address drop-down list, choose the IP address of the server to be added to the cluster.

   3. In the Username field, enter the user name for logging in to the server.

   4. In the Password field, enter the password for logging in to the server.

   5. (Optional) Click Enable SD-AVC if you want Cisco Software-Defined Application Visibility and Control (SD-AVC) to run on the server.

      Cisco SD-AVC is a component of Cisco Application Visibility and Control (AVC). It can be enabled on one Cisco SD-WAN Manager server. The server on which it is enabled must have the Compute+Data or the Compute persona. Cisco SD-AVC cannot be enabled on a server that has the Data persona.

      If you enabled Cisco SD-AVC for this server when you changed its IP address, the Enable SD-AVC check box is checked by default.

   6. Click Add.

   7. To confirm, click OK.

      The dialog box indicates that the services will restart, and that the existing metadata and other information that is not required when the server joins the cluster will be deleted from the server.

      When you click OK, the system starts the server add operation. The Cluster Management window displays the tasks that the system performs as it adds the server.

      As part of this operation, the system checks the compatibility of the server that you are adding. This check ensures that the server has sufficient disk space, and that the persona that you specified matches the persona of the node.

      After the server is added, the system performs a cluster sync operation, which rebalances the services in the cluster. Then the Cisco SD-WAN Manager servers in the cluster restart.

View Statistics Database Space Usage

To view the amount of space available for and utilized by the statistics database on the local Cisco SD-WAN Manager server, choose Administration > Settings > Statistics Database Configuration and click View. The top of the window shows the maximum space available for the database and the total amount of space currently being utilized. The table shows the disk space currently being utilized for each statistics type.

For information about disk size recommendations and requirements, see Server Recommendations for your release in Cisco Catalyst SD-WAN Controller Compatibility Matrix and Server Recommendations.

Configure Statistics Database

To configure the statistics database that stores all the real-time statistics from the local Cisco SD-WAN Controller Compatibility Matrix and Server Recommendations:

1. From the Cisco SD-WAN Manager menu, choose Administration > Settings.

2. In the Statistics Database Configuration section, click Edit to view the maximum space available for the database.

3. For each field in the Statistics Type column, assign the amount of storage to be allocated, in gigabytes (GB). The total value of all the fields cannot exceed the maximum available space.

4. Click Save.

Cisco SD-WAN Manager updates the storage allocations that you have assigned once a day, at midnight.

View Detailed Information about Services

To view detailed information about the services running on a Cisco SD-WAN Manager server:

1. From the Cisco SD-WAN Manager menu, choose Administration > Cluster Management and click Service Configuration.

2. Click the hostname of the Cisco SD-WAN Manager server.

   The vManage Details window opens, displaying the process IDs of all the Cisco SD-WAN Manager services that are enabled on Cisco SD-WAN Manager.

3. Click Cluster Management in the breadcrumb in the title bar to return to the Cluster Management window.

View Devices Connected to Cisco SD-WAN Manager

To view the list of devices connected to Cisco SD-WAN Manager:

1. From the Cisco SD-WAN Manager menu, choose Administration > Cluster Management and click Service Configuration.

2. Click the hostname of the Cisco SD-WAN Manager server.

3. Click Managed Devices.

Alternatively:

1. From the Cisco SD-WAN Manager menu, choose Administration > Cluster Management and click Service Configuration.

2. Click ... adjacent to the Cisco SD-WAN Manager server and choose Device Connected.

3. If a device is connected to Cisco SD-WAN Manager from a cluster, ensure that you do not configure the data stream hostname to the Cisco SD-WAN Manager system IP address. However, you can configure the management IP address on VPN 512 or the internet public IP address on VPN 0. For information about data stream troubleshooting tools, see Data Stream Troubleshooting Tools FAQ.

You can edit various parameters for a Cisco SD-WAN Manager server that has been added to a cluster. To do so, follow these steps:

1. From the Cisco SD-WAN Manager menu, choose Administration > Cluster Management and click Service Configuration.

2. Click ... adjacent to the Cisco SD-WAN Manager server to edit, and click Edit.

   The Edit vManage window opens.

3. Select an IP address to edit.

4. Enter the username and password, and edit parameters for the selected Cisco SD-WAN Manager server.

   • For releases before Cisco vManage Release 20.6.1, you can edit the cluster services.

   • From Cisco vManage Release 20.6.1, you can change the IP address to another IP address that appears in the vManage IP Address drop-down list, change the Cisco SD-AVC setting, or change the username and password if the server credentials have changed.

5. Click Update.

The default username of the configuration database is neo4j and the default password is password. To update the default login credentials of the configuration database, access Cisco SD-WAN Manager using a terminal and run the following commands. Do not use the SSH terminal option in Cisco SD-WAN Manager to run these commands. Doing so causes you to lose access to Cisco SD-WAN Manager.

1. Use request nms application-server stop to stop application servers on all the Cisco SD-WAN Manager servers whether configuration-db is enabled or not.

   .

2. Use one of the following commands to reset the user name and password for the configuration database on all the Cisco SD-WAN Manager servers:

   • For Cisco SD-WAN Release 20.1.1 and earlier:

     request nms configuration-db update-admin-user username username password password newusername newadminuser newpassword newpassword

   • For releases from Cisco SD-WAN Release 20.1.2:

     request nms configuration-db update-admin-user

     When prompted, enter your current username and password, and your new username and password.

     When you run one of these commands, Cisco SD-WAN Manager restarts the application server

   Note

   If you do not know the default credentials of the configuration database, contact your Cisco support representative to retrieve the credentials. You cannot use a previous username. Passwords can include only a mix of characters A to Z ( upper or lowercase), digits 0 to 9, and special characters @, #, and *.

   Example

   • For Cisco SD-WAN Release 20.1.1 and earlier:

     request nms configuration-db update-admin-user username neo4j password ******** newusername myusername newpassword mypassword

   • For releases from Cisco SD-WAN Release 20.1.2:

     request nms configuration-db update-admin-user

     Enter current user name: neo4j

     Enter current user password: password

     Enter new user name: myusername

     Enter new user password: mypassword

Note	After a configuration database admin user update, if you are unable to view a specific Cisco SD-WAN Manager instance, use the request nms application-server restart command to restart the application server on that Cisco SD-WAN Manager instance again.

Note

Starting from Cisco SD-WAN Release 20.6.1, when using the request nms configuration-db update-admin-user command to update the admin user credentials, provide the same inputs (old username, password and the new username, password) across all the nodes in the Cisco SD-WAN Manager cluster. You must execute the request nms configuration-db update-admin-user command one node at a time. We recommend that you do not push the CLI to all the nodes at the same time because the NMS services will restart for the new configuration to take effect.

This section describes how to upgrade Cisco SD-WAN Manager in a cluster.

You can upgrade directly from Cisco vManage 20.3.1 or later releases to Cisco vManage Release 20.6.1. To upgrade from earlier releases, first upgrade to Cisco vManage 20.4.2 or Cisco vManage Release 20.5.1.

If you are upgrading a Cisco SD-WAN Manager cluster deployment from Cisco vManage Release 20.3.1 or later to Cisco vManage Release 20.5.1 or later, you must do it through the CLI.

Before You Begin

Before you upgrade Cisco SD-WAN Manager nodes to Cisco vManage Release 20.6.1 or later releases, verify the following:

• Ensure that the internal user account vmanage-admin is not locked for any server that you are upgrading.

  You can check the status of this admin account by pushing a template to the devices that are connected to the server. The push fails if the account is locked. In such a scenario, you can unlock the account by using the request aaa unlock-user vmanage-admin command.

  For information about the vmanage-admin account, see Cisco Catalyst SD-WAN Manager.

  For information about the vmanage-admin account, see vmanage-admin User Account.

• Ensure that PKI keys have been exchanged between the servers that you are upgrading.

  To do so, ensure that the control connections are in the UP state on the servers and restart the application server.

• Ensure that the out-of-band IP address of each server is reachable.

• Ensure that the Cisco SD-WAN Manager UI is accessible on all servers in the cluster.

• Ensure that DCA is running on all nodes in the cluster.

  To do so, use the request nms data-collection-agent status command and ensure that the status value shows running for each node.

  To start DCA, if needed, use the request nms data-collection-agent start command.

Note

If these prerequisites are not met or if another error occurs during the upgrade, the activation of the image fails and a file named upgrade-context.json is created in the /opt/data/extra-packages/image-version folder on each node in the cluster. You can provide this file to your Cisco representative for assistance with resolving the issue.

If you are upgrading to Cisco vManage Release 20.6.1 or later releases from a six-node Cisco SD-WAN Manager cluster deployment in which not all services are running on all nodes, contact your Cisco support representative before performing the upgrade.

1. Take snapshots of all the Cisco SD-WAN Manager servers. Take a backup of the configuration database and save it in a location outside of the Cisco SD-WAN Manager server using the following command:

   request nms configuration-db backup path path_and_filename

2. Ensure that Cisco vManage Release 18.3 or later is installed.

3. For upgrades from Cisco vManage Release 20.3.1 or later, copy the current image to each Cisco SD-WAN Manager server in the cluster and install the image on each Cisco SD-WAN Manager server by using the following command. Do not activate the image at this time.

   request software install path

4. For upgrades from Cisco vManage Release 20.3.1 or later, activate the current image on each Cisco SD-WAN Manager server using the following command. All servers reboot simultaneously.

   request software activate version

5. You must upgrade the configuration database when upgrading from one of the following:

   • Cisco vManage Release 18.4.x or 19.2.x to Cisco vManage 20.3.x or 20.4.x

   • Cisco vManage Release 20.3.x or 20.4.x to Cisco vManage Release 20.5.x or 20.6.x

   • Any Cisco SD-WAN Manager release to Cisco vManage Release 20.10.1 or later

   Note

   Starting from Cisco vManage Release 20.1.1, before upgrading the configuration database, ensure that you verify the database size. We recommend that the database size is less than or equal to 5 GB. To verify the database size, use the following diagnostic command: request nms configuration-db diagnostics When you upgrade the configuration database, ensure that you have activated the current image on each Cisco SD-WAN Manager server in the cluster as described in the previous step. In addition, ensure that all services except the application server and configuration-db services are running on these servers by entering the request nms all status command on each server.

   To upgrade the configuration database, do the following:

   1. To determine which node to upgrade, enter the request nms configuration-db status command on each node. In the output look for the following:

      Enabled: true
      Status: not running

      Note

      After activating a new image on a Cisco SD-WAN Manager host server, the server reboots. After the reboot, for approximately 30 minutes, the output of the request nms configuration-db status command shows Enabled: false even on a node that has the configuration database enabled, while NMS services are being migrated to a containerized form.

   2. On the node to upgrade, as determined in the previous step, enter the following:

      request nms configuration-db upgrade

      Note	Enter this command on one node only. Do not enter this command if you are upgrading from Cisco vManage Release 20.5.x to Cisco vManage Release 20.6.1 or later.

6. Enter your login credentials, if prompted. Login credentials are prompted in releases earlier than Cisco vManage Release 20.3.1 if all the Cisco SD-WAN Manager servers establish control connection with each other. After a successful upgrade, all the configuration database services are UP across the cluster, and the application server is started.

You can check the database upgrade logs at the following location: vmanage-server:/var/log/nms/neo4j-upgrade.log.

For information about how to upgrade Cisco SD-WAN Manager clusters by using the Cisco SD-WAN Manager GUI, see the Upgrade the Software Image on a Device section in ​Cisco Catalyst SD-WAN Monitor and Maintain Configuration Guide.

Starting from Cisco IOS XE Catalyst SD-WAN Release 17.10.1a, a device-data-collector service container is added. The following is a sample output for the command, request nms device-data-collector.

Device# request nms device-data-collector
Possible completions:
  diagnostics   Run diagnostics on NMS component
  jcmd          Run jcmd on NMS component
  restart       Restart NMS component
  start         Start NMS component
  status        Status of NMS component
  stop          Stop NMS component

You can remove a Cisco SD-WAN Manager node from a cluster, if necessary.

In releases earlier than Cisco vManage Release 20.6.1, you can only remove n - 2 Cisco SD-WAN Manager nodes from a cluster of n nodes. You must retain at least two Cisco SD-WAN Manager nodes in a cluster.

From Cisco vManage Release 20.6.1, you must retain at least two Cisco SD-WAN Manager nodes that include the compute capability and at least one node that includes the data capability. That is, the cluster must retain any of the following:

• At least two Cisco SD-WAN Manager nodes that include the Compute+Data persona

• At least one Cisco SD-WAN Manager nodes that includes the Compute+Data persona and one Cisco SD-WAN Manager node that includes the Compute persona

• At least two Cisco SD-WAN Manager nodes that include the Compute persona and one Cisco SD-WAN Manager node that includes the Data persona

From Cisco vManage Release 20.6.1, if a Cisco SD-WAN Manager node is reachable when you remove it from a cluster, Cisco SD-WAN Manager automatically performs a factory reset operation on the removed node to ensure that the node does not join the cluster again. If a Cisco SD-WAN Manager node is unreachable when you remove it from a cluster, a factory reset operation is not performed on the node. In this situation, the node is added back to the cluster automatically when the node becomes reachable. To prevent the node from being added back to the cluster, enter the command request software reset from the CLI of the node after the node is removed from the cluster.

To remove a Cisco SD-WAN Manager node from a cluster, follow these steps:

1. From the Cisco SD-WAN Manager, choose Administration > Cluster Management and click Service Configuration.

2. Click … adjacent to the Cisco SD-WAN Manager instance that you want to remove and click Remove.

   The Remove vManage dialog box opens.

3. Enter the username and password to confirm the removal of the device from the network.

4. Click Remove.

The Cisco SD-WAN Manager instance is removed from the cluster, the certificates for that Cisco SD-WAN Manager are deleted, and Cisco SD-WAN Manager undergoes a factory reset.