The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes how to rebuild a Cisco SD-WAN fabric, including backing up and restoring controller configurations for various deployments.
Cisco recommends that you have knowledge of these topics:
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
vManage Deployment
DR Options
Note: For more details of type of disaster recovery refer to this link
Combinations:
| # | vManage Setup | DR Option |
|---|---|---|
| 1 | Standalone (1 node) | No DR |
| 2 | Standalone (1 node) | Single Node DR |
| 3 | Cluster (3-node or 6-node) | No DR |
| 4 | Cluster (3-node or 6-node) | Standby DR Cluster |
These steps are common to all deployment combinations. They cover the process of bringing up VM instances and applying basic CLI configuration. Each combination section tells you how many instances to deploy and which additional steps to complete.
Note: Cisco has rebranded certain terms, so these terms are interchangeable. Cisco vManage = Cisco Catalyst Manager, Cisco vBond = Cisco Catalyst Validator, Cisco vSmart = Cisco Catalyst Controller
Download the OVA files for SD-WAN controllers from the Cisco Software Download page here:
Note: On the ESXi/cloud platforms, spin up vSmart, vBond and vManage Controllers using the OVA file. Refer to the linked document and make sure sufficient CPU, RAM and disks are allocated to all the controllers depending on the SD-WAN deployment type. Navigate here for additional information. Make sure to assign secondary disk to vManage node as mentioned in the column Storage Size* in the linked compute guide.
For a standalone vManage, choose the persona as COMPUTE_AND_DATA.
For a 3 node cluster, on 3 vManage nodes, the persona is set to COMPUTE_AND_DATA.
For a 6 node cluster, on 3 vManage nodes the persona is COMPUTE_AND_DATA and on rest 3 vManage nodes persona DATA.
Example:Choose 1 for COMPUTE_AND_DATA

Choose the secondary disk as shown:


Example

Note: You can refer to the configuration from the existing vManage and configure the same IP address scheme here.
Conf t
vpn 0
no interface eth0
vpn 512
interface eth0
ip address <IP-address/mask>
no shutdown
!
ip route 0.0.0.0/0 <default-gateway IP>
!
Example:

Note: You can refer to the configuration from the existing vBond and configure the same configurations here.
Conf t
vpn 0
no interface eth0
vpn 512
interface eth0
ip address <IP-address/mask>
no shutdown
!
ip route 0.0.0.0/0 <default-gateway IP>
!
commit
Once you have SSH access to all the controllers, configure these CLI configurations on each controller.
config t
system
host-name <hostname>
system-ip <unique system-IP>
site-id <site-id>
organization-name <organization name>
vbond <IP address/URL of vBond>
commit
Note: If we are using URL as vBond address, make sure to configure DNS server IP addresses in VPN 0 configuration or ensure they can be resolved.
These configurations are needed on all the controllers to enable the transport interface used to establish control connections with the routers and the rest of the controllers.
config t
vpn 0
dns <IP-address> primary
dns <IP-address> secondary
interface eth1
ip address <IP-address/mask>
tunnel-interface
allow-service all
allow-service dhcp
allow-service dns
allow-service icmp
no allow-service sshd
no allow-service netconf
no allow-service ntp
no allow-service stun
allow-service https
!
no shutdown
!
ip route 0.0.0.0/0 <default-gateway IP>
commit
Note: You can refer to the configurations of your existing controller and if the config is present then you can add this configuration to the new controllers.
Configure the control protocol as TLS only if there is a requirement for routers to establish secure control connections with the vManage nodes using TLS. By default, all the controllers and routers establish control connection using DTLS. This is an optional config required only on vSmart and vManage nodes depending on you requirement.
Conf t
security
control
protocol tls
Commit
Ensure that the number of the activeCisco SD-WAN Manager instances are identical to the number of the newly installedCisco SD-WAN Manager instances.
Ensure that all the active and new Cisco SD-WAN Manager instances run the same software version.
Ensure that all the active and new Cisco SD-WAN Manager instances are able to reach the management IP address of the Cisco SD-WAN Validator.
Ensure that certificates have been installed on the newly installed Cisco SD-WAN Manager instances.
Ensure that the clocks on all Cisco Catalyst SD-WAN devices, including the newly installed Cisco SD-WAN Manager instances, are synchronized.
Ensure that a new set of System IPs and Site IDs is configured on the newly installed Cisco SD-WAN Manager instances, along with the same basic configuration as the active cluster.




In case, if we are using our own CA, Enterprise certificate authority, choose Enterprise.


Navigate to Configuration > Devices > Control Components in case of 20.15/20.18 vManage nodes. In case of 20.9/20.12 versions, Configuration > Devices > Controllers
Note: We need to useadmin credentials ofvBondor a user part ofnetadmingroup. You can verify this in the CLI of thevBond. Choose Yes in the dropdown of“Generate CSR" if we need to install a new certificate forvBond.
Note: If the vBond is behind a NAT device/Firewall, check if the vBond VPN 0 interface IP is translated to a public IP. If VPN 0 interface IP is not reachable from vManage, use the public IP address of VPN 0 interface in this step.

Click on Add vSmart in case of 20.12 vManage or Add Controller in case of 20.15/20.18 vManage.
A pop up opens, enter the VPN 0 transport IP of vSmart which is reachable from the vManage.
Check the reachability using ping if allowed from CLI of vManage to vSmart IP.
Enter the user credentials of vSmart Note that we need to use admin credentials of vSmart or a user part of netadmin group.
You can verify this in the CLI of the vSmart.
Set the protocol to TLS, if we intend to use TLS for routers to establish control connections with vSmart. This config needs to be configured on CLI of vSmarts and vManage nodes as well.
Choose Yes in the dropdown of "Generate CSR" if we need to install a new certificate for vSmart.
Note: If the vSmart is behind NAT device/Firewall, check if the vSmart VPN 0 interface IP is translated to a public IP, and if VPN 0 interface IP is not reachable from vManage, use public IP address of VPN 0 interface IP in this step.

Once all the steps are completed, verify that all the control components are reachable in Monitor>Dashboard


Confirm the configuration-db is running on the vManage node.
You can verify the same using the command request nms configuration-db status onvManageCLI. The output is as shown
vmanage# request nms configuration-db status
NMS configuration database
Enabled: true
Status: running PID:32632 for 1066085s
Native metrics status: ENABLED
Server-load metrics status: ENABLED
vmanage#
Use this command to collect the configuration-db backup from the identified configuration-db leader vManage node.
request nms configuration-db backup path /opt/data/backup/<filename>
The expected output is as shown:
vmanage# request nms configuration-db backup path /opt/data/backup/june18th
Starting backup of configuration-db
config-db backup logs are available in /var/log/nms/neo4j-backup.log file
Successfully saved backup to /opt/data/backup/june18th.tar.gz
sha256sum: 8d0f5af8aee4e70f05e3858be6bdd5e6c136134ae47c383569ec883080f5d359
Removing the temp staging dir :/opt/data/backup/staging
vmanage#
Copy the configuration-db backup to /home/admin/ directory of vManage using SCP.
Sample scp command output:
XXXXXXXXX Downloads % scp june18th.tar.gz admin@10.66.62.27:/home/admin/
viptela 20.15.4.1
(admin@10.66.62.27) Password:
(admin@10.66.62.27) Password:
june18th.tar.gz 0% 255KB 47.2KB/s - stalled -
To restore configuration-db backup, first we need to configure the configuration-db credentials. If your configuration-db credentials are default(neo4j/password), we can skip this step.
To configure configuration-db credentials, use the command request nms configuration-db update-admin-user. Use the username and password of your choice.
Kindly note that the Application server of vManage is restarted. Due to which vManage UI becomes inaccessible for a short time.
vmanage# request nms configuration-db update-admin-user
configuration-db
Enter current user name:neo4j
Enter current user password:password
Enter new user name:ciscoadmin
Enter new user password:ciscoadmin
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Successfully updated configuration database admin user(this is service node, please repeat same operation on all service/data nodes)
Successfully restarted vManage Device Data Collector
Successfully restarted NMS application server
Successfully restarted NMS data collection agent
vmanage#
Post which we can proceed to retore the configuration-db backup:
We can use the command request nms configuration-db restore path /home/admin/< > to restore the configuration-db to the new vManage:
vmanage# request nms configuration-db restore path /home/admin/june18th.tar.gz
Starting backup of configuration-db
config-db backup logs are available in /var/log/nms/neo4j-backup.log file
Successfully saved database to /opt/data/backup/configdb-local-tmp-20230623-160954.tar.gz
Successfully backup database to /opt/data/backup/configdb-local-tmp-20230623-160954.tar.gz
Configuration database is running in a standalone mode
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Successfully saved cluster configuration for localhost
Successfully saved vManage root CA information for device: "53f95156-f56b-472f-b713-d164561b25b7"
Stopping NMS application server on localhost
Stopping NMS configuration database on localhost
Reseting NMS configuration database on localhost
Loading NMS configuration database on localhost
Starting NMS configuration database on localhost
Waiting for 180s or the instance to start...
NMS configuration database on localhost has started.
Updating DB with the saved cluster configuration data
Successfully reinserted cluster meta information
Successfully reinserted vmanage root ca information
Starting NMS application server on localhost
Waiting for 180s for the instance to start...
Successfully restored database
Once the configuration-db is restored, make sure the vManage UI is accessible. Wait for around 5 minutes and then attempt to access the UI.
Once logged into UI successfully, ensure the Edge routers list, template, policies and all the rest of the configurations that were present on your previous or existing vManage UI is reflected on the new vManage UI.
Once configuration-db is restored, we need to reauthenticate all the new controllers (vmanage/vsmart/vbond) in the fabric.
Note: In actual production if the interface IP used to re-authenticate is the tunnel interface IP, need to ensure NETCONF service is allowed on the tunnel interface of the vManage, vSmart and vBond and also on the firewalls along the path. The firewall port to open is TCP port 830 as bi-directional rule from DR cluster to all vBonds and vSmarts.
On vmanage UI, click on Configuration > Devices > Controllers


Once all the controllers are onboarded, complete this step:
On any Cisco SD-WAN Manager server in the newly active cluster, perform these actions:
Enter this command to synchronize the root certificate with all Cisco Catalyst SD-WAN devices in the newly active cluster:
https://vmanage-url/dataservice/system/device/sync/rootcertchain
Enter this command to synchronize the Cisco SD-WAN Manager UUID with the Cisco SD-WAN Validator:
https://vmanage-url/dataservice/certificate/syncvbond
Once the fabric is restored and the control and bfd sessions are up for all edges and controlllers in the fabric,we need to invalidate the old controllers (vmanage/vsmart/vbond) from the UI
Note: Continue with the Post Checks section shown here, which is common to all deployment combinations.
Ensure that the number of the active Cisco SD-WAN Managerinstances are identical to the number of the newly installed Cisco SD-WAN Managerinstances.
Ensure that all the active and new Cisco SD-WAN Manager instances run the same software version.
Ensure that all the active and new Cisco SD-WAN Manager instances are able to reach the management IP address of the Cisco SD-WAN Validator.
Ensure that certificates have been installed on the newly installed Cisco SD-WAN Manager instances.
Ensure that the clocks on all Cisco Catalyst SD-WAN devices, including the newly installed Cisco SD-WAN Manager instances, are synchronized.
Ensure that a new set of System IPs and Site IDs is configured on the newly installed Cisco SD-WAN Manager instances, along with the same basic configuration as the active cluster.




In case, if we are using our own CA, Enterprise certificate authority, choose Enterprise.


Navigate to Configuration > Devices > Control Components in case of 20.15/20.18 vManage nodes. In case of 20.9/20.12 versions, Configuration > Devices > Controllers
Note: We need to useadmin credentials ofvBondor a user part ofnetadmingroup. You can verify this in the CLI of thevBond. Choose Yes in the dropdown of“Generate CSR" if we need to install a new certificate forvBond
Note: If the vBond is behind a NAT device/Firewall, check if the vBond VPN 0 interface IP is translated to a public IP. If VPN 0 interface IP is not reachable from vManage, use the public IP address of VPN 0 interface in this step

Click on Add vSmart in case of 20.12 vManage or Add Controller in case of 20.15/20.18 vManage.
A pop up opens, enter the VPN 0 transport IP of vSmart which is reachable from the vManage.
Check the reachability using ping if allowed from CLI of vManage to vSmart IP.
Enter the user credentials of vSmart Note that we need to use admin credentials of vSmart or a user part of netadmin group.
You can verify this in the CLI of the vSmart.
Set the protocol to TLS, if we intend to use TLS for routers to establish control connections with vSmart. This config needs to be configured on CLI of vSmarts and vManage nodes as well.
Choose Yes in the dropdown of "Generate CSR" if we need to install a new certificate for vSmart.
Note: If the vSmart is behind NAT device/Firewall, check if the vSmart VPN 0 interface IP is translated to a public IP, and if VPN 0 interface IP is not reachable from vManage, use public IP address of VPN 0 interface IP in this step.

Once all the steps are completed, verify that all the control components are reachable in Monitor>Dashboard


Note: While collecting configuration-database backup from the existing vManage node which has Disaster recovery enabled, make sure it is collected after the Disaster recovery on that node is paused and deleted.
Confirm there is no ongoing Disaster recovery replication. Navigate to Administration > Disaster Recovery and make sure the status Success and not in a transient state such as Import Pending, Export Pending, or Download Pending. If the status is not success, reach out to Cisco TAC and make sure replication is successful before you proceed to pause the disaster recovery.
First Pause the disaster recovery and make sure the task is complete. And then Delete the Disaster recovery and confirm the task is completed.

Reach out to Cisco TAC to ensure the Disaster Recovery is successfully cleaned up.
Confirm the configuration-db is running on the vManage node.
You can verify the same using the commandrequest nms configuration-db statusonvManageCLI. The output is as shown
vmanage# request nms configuration-db status
NMS configuration database
Enabled: true
Status: running PID:32632 for 1066085s
Native metrics status: ENABLED
Server-load metrics status: ENABLED
vmanage#
Use this command to collect the configuration-db backup from the identified configuration-db leader vManage node.
request nms configuration-db backup path /opt/data/backup/<filename>
The expected output is as shown:
vmanage# request nms configuration-db backup path /opt/data/backup/june18th
Starting backup of configuration-db
config-db backup logs are available in /var/log/nms/neo4j-backup.log file
Successfully saved backup to /opt/data/backup/june18th.tar.gz
sha256sum: 8d0f5af8aee4e70f05e3858be6bdd5e6c136134ae47c383569ec883080f5d359
Removing the temp staging dir :/opt/data/backup/staging
vmanage#
Copy the configuration-db backup to /home/admin/ directory of vManage using SCP.
Sample scp command output:
XXXXXXXXX Downloads % scp june18th.tar.gz admin@10.66.62.27:/home/admin/
viptela 20.15.4.1
(admin@10.66.62.27) Password:
(admin@10.66.62.27) Password:
june18th.tar.gz 0% 255KB 47.2KB/s - stalled -
To restore configuration-db backup, first we need to configure the configuration-db credentials. If your configuration-db credentials are default(neo4j/password), we can skip this step.
To configure configuration-db credentials, use the command request nms configuration-db update-admin-user.Use the username and password of your choice.
Kindly note that the Application server of vManage is restarted. Due to which vManage UI becomes inaccessible for a short time.
vmanage# request nms configuration-db update-admin-user
configuration-db
Enter current user name:neo4j
Enter current user password:password
Enter new user name:ciscoadmin
Enter new user password:ciscoadmin
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Successfully updated configuration database admin user(this is service node, please repeat same operation on all service/data nodes)
Successfully restarted vManage Device Data Collector
Successfully restarted NMS application server
Successfully restarted NMS data collection agent
vmanage#
Post which we can proceed to retore the configuration-db backup:
We can use the command request nms configuration-db restore path /home/admin/< >to restore the configuration-db to the new vManage:
vmanage# request nms configuration-db restore path /home/admin/june18th.tar.gz
Starting backup of configuration-db
config-db backup logs are available in /var/log/nms/neo4j-backup.log file
Successfully saved database to /opt/data/backup/configdb-local-tmp-20230623-160954.tar.gz
Successfully backup database to /opt/data/backup/configdb-local-tmp-20230623-160954.tar.gz
Configuration database is running in a standalone mode
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Successfully saved cluster configuration for localhost
Successfully saved vManage root CA information for device: "53f95156-f56b-472f-b713-d164561b25b7"
Stopping NMS application server on localhost
Stopping NMS configuration database on localhost
Reseting NMS configuration database on localhost
Loading NMS configuration database on localhost
Starting NMS configuration database on localhost
Waiting for 180s or the instance to start...
NMS configuration database on localhost has started.
Updating DB with the saved cluster configuration data
Successfully reinserted cluster meta information
Successfully reinserted vmanage root ca information
Starting NMS application server on localhost
Waiting for 180s for the instance to start...
Successfully restored database
Once the configuration-db is restored, make sure the vManage UI is accessible. Wait for around 5 minutes and then attempt to access the UI.
Once logged into UI successfully, ensure the Edge routers list, template, policies and all the rest of the configurations that were present on your previous or existing vManage UI is reflected on the new vManage UI.
Refer to Step 2: Prechecks in Combination 2: Standalone vManage + Single Node DR and make sure we have completed all the requirements before we proceed to enable Disaster recovery.
Out-of-band cluster interface in VPN 0
The bare minimumconfiguration forvManageprior to the Disaster Recovery registration is as shown
config t
system
host-name <hostname>
system-ip <unique system-ip>
site-id <site-id>
organization-name <organization name>
vbond <IP address/URL of vBond>
commit
Note: If we are using URL as vBond address, make sure to configure DNS server IP addresses in VPN 0 configuration or ensure they can be resolved.
These configurations are needed to enable transport interface used to establish control connections with the routers and rest of the controllers
config t
vpn 0
dns <IP-address> primary
dns <IP-address> secondary
interface eth1
ip address <IP-address/mask>
tunnel-interface
allow-service all
allow-service dhcp
allow-service dns
allow-service icmp
no allow-service sshd
no allow-service netconf
no allow-service ntp
no allow-service stun
allow-service https
!
no shutdown
!
ip route 0.0.0.0/0 <default-gateway IP>
commit
Also configure VPN 512management interfaceto enable out of band management access to the controller.
Conf t
vpn 512
interface eth0
ip address <IP-address/mask>
no shutdown
!
ip route 0.0.0.0/0 <default-gateway IP>
!
commit
Configure service interface on the vManage node. This interface is used for DR communication,
conf t
interface eth2
ip address <IP-address/mask>
no shutdown
commit
Make sure the same IP subnet is used for service interface on the Primary vManage and DR vManage
Proceed with the steps given under section Combination 2: Standalone vManage + Single Node DR Step 3: Configure vManage UI, Certificates, and Onboard Controllers to install the certificate on the Disaster recovery vManage.

Once filled, click ‘Next’.
Fill the vBond controllers’ details.
The vBond controllers must be reachable in the specified IP address via Netconf.
The credentials must be those of a netadmin user (dradmin) and they must not be changed once the DR is configured.
For this it is recommended that vBond have this dradmin user locally configured or you can use the admin user to add the vBond.


In the Replication Schedule, set the ‘Replication Interval’.Every replication interval time, the data is replicated from primary vManageto secondary vManage. The minimum configurable value is 15 minutes.


Note that the vManage GUI is restarted during this process.
Once finished, a status of Success must be seen.

Navigate toAdministration → Disaster Recoveryto see the Disaster Recovery status and when the data was replicated last time.

Once configuration-db is restored ,we need to reauthenticate all the new controllers (vmanage/vsmart/vbond) in the fabric
Note: In actual production if the interface IP used to re-authenticate is the tunnel interface IP, need to ensure NETCONF service is allowed on the tunnel interface of the vManage, vSmart and vBond and also on the firewalls along the path. The firewall port to open is TCP port 830 as bi-directional rule from DR cluster to all vBonds and vSmarts .
On vmanage UI,vclick on Configuration > Devices > Controllers

Once all the controllers are onboarded, complete this step:
On any Cisco SD-WAN Manager server in the newly active cluster, perform these actions:
Enter this command to synchronize the root certificate with all Cisco Catalyst SD-WAN devices in the newly active cluster:
https://vmanage-url/dataservice/system/device/sync/rootcertchain
Enter this command to synchronize the Cisco SD-WAN Manager UUID with the Cisco SD-WAN Validator:
https://vmanage-url/dataservice/certificate/syncvbond
Once the fabric is restored and the control and bfd sessions are up for all edges and controlllers in the fabric,we need to invalidate the old controllers (vmanage/vsmart/vbond) from the UI
Note: Continue with the Post Checks section shown here, which is common to all deployment combinations.
Instances needed:
Steps:
Ensure that the number of the active Cisco SD-WAN Managerinstances are identical to the number of the newly installed Cisco SD-WAN Managerinstances.
Ensure that all the active and new Cisco SD-WAN Manager instances run the same software version.
Ensure that all the active and new Cisco SD-WAN Manager instances are able to reach the management IP address of the Cisco SD-WAN Validator.
Ensure that certificates have been installed on the newly installed Cisco SD-WAN Manager instances.
Ensure that the clocks on all Cisco Catalyst SD-WAN devices, including the newly installed Cisco SD-WAN Manager instances, are synchronized.
Ensure that a new set of System IPs and Site IDs is configured on the newly installed Cisco SD-WAN Manager instances, along with the same basic configuration as the active cluster.




In case, if we are using our own CA, Enterprise certificate authority, choose Enterprise.


Navigate to Configuration > Devices > Control Components in case of 20.15/20.18 vManage nodes. In case of 20.9/20.12 versions, Configuration > Devices > Controllers
Note: We need to useadmin credentials ofvBondor a user part ofnetadmingroup. You can verify this in the CLI of thevBond. Choose Yes in the dropdown of“Generate CSR" if we need to install a new certificate forvBond
Note: If the vBond is behind a NAT device/Firewall, check if the vBond VPN 0 interface IP is translated to a public IP. If VPN 0 interface IP is not reachable from vManage, use the public IP address of VPN 0 interface in this step

Click on Add vSmart in case of 20.12 vManage or Add Controller in case of 20.15/20.18 vManage.
A pop up opens, enter the VPN 0 transport IP of vSmart which is reachable from the vManage.
Check the reachability using ping if allowed from CLI of vManage to vSmart IP.
Enter the user credentials of vSmart Note that we need to use admin credentials of vSmart or a user part of netadmin group.
You can verify this in the CLI of the vSmart.
Set the protocol to TLS, if we intend to use TLS for routers to establish control connections with vSmart. This config needs to be configured on CLI of vSmarts and vManage nodes as well.
Choose Yes in the dropdown of "Generate CSR" if we need to install a new certificate for vSmart.
Note: If the vSmart is behind NAT device/Firewall, check if the vSmart VPN 0 interface IP is translated to a public IP, and if VPN 0 interface IP is not reachable from vManage, use public IP address of VPN 0 interface IP in this step.

Once all the steps are completed, verify that all the control components are reachable in Monitor>Dashboard


Note: vManage Cluster can be configured with 3 vManage nodes or 6 vManage nodes depending on the number of sites onboarded to SD-WAN fabric. Kindly refer to your existing vManage cluster and choose the number of nodes as per the same.
config t
system
host-name <hostname>
system-ip <unique system-ip>
site-id <site-id>
organization-name <organization name>
vbond <IP address/URL of vBond>
commit
Note: If we are using URL as vBond address, make sure to configure DNS server IP addresses in VPN 0 configuration or ensure they can be resolved.
These configurations are needed to enable the transport interface used to establish control connections with the routers and rest of the controllers.
config t
vpn 0
dns <IP-address> primary
dns <IP-address> secondary
interface eth1
ip address <IP-address/mask>
tunnel-interface
allow-service all
allow-service dhcp
allow-service dns
allow-service icmp
no allow-service sshd
no allow-service netconf
no allow-service ntp
no allow-service stun
allow-service https
!
no shutdown
!
ip route 0.0.0.0/0 <default-gateway IP>
commit
Also configure VPN 512management interfaceto enable out of band management access to the controller.
Conf t
vpn 512
interface eth0
ip address <IP-address/mask>
no shutdown
!
ip route 0.0.0.0/0 <default-gateway IP>
!
Commit
Optional Config:
Conf t
security
control
protocol tls
commit
Configure service interface on all thevManagenodes including vManage-1 which has been onboarded already. This interface is used for cluster communication, meaning communication between thevManagenodes in the cluster.
conf t
interface eth2
ip address <IP-address/mask>
no shutdown
commit
Make sure thesame IP subnet is used for service interface across all the nodesin thevManagecluster.
We can use the same admin credentials of thevManagenodes to configure thevManagecluster. Else we can configure a new user credential which is part ofnetadmingroup. The configurations to configure new user credential is as shown
conf t
system
aaa
user <username>
password <password>
group netadmin
commit
Make sure toconfigure the same user credentials across all the vManagenodeswhich is part of the cluster.If we decide to use admin credentials, it must be the same username/password across all the vManagenodes.
Navigate to Configuration > Certificates > Control Components in case of 20.15/20.18 vManage nodes. In case of 20.9/20.12 versions, Configuration > Devices > Controllers
Click on ... for Manager/vManage and click on Generate CSR.

Once the CSR is generated, you can download the CSR and get it signed based on the Certificate authority chosen for controllers. You can verify this configuration in Administration > Settings > Controller Certificate Authorization. If Cisco (Recommended) is chosen, then the CSR is automatically uploaded to the PNP portal by the vManage and once the certificate is signed, it is installed on the vManage automatically.
If Manual is chosen, manually sign the CSR using the Cisco PNP portal by navigating to smart account and virtual account of the respective SD-WAN overlay. Same procedure is applicable if we are using Digicert and Enterprise Root Certificate.
Once the certificate is available from PNP portal, click on install certificate on the same section of vManage and upload the certificate and install the certificate.
Complete this step across all the vManage nodes which is part of the cluster.
Note: For a 3-node cluster, all 3 vManage nodes are brought up with compute+data as the persona.

Note: Please refer to this configuration in your existing cluster to Enable SDAVC- Need to be checked only if it is required and is needed only on one vManage node of the cluster.
Click on Update.




These points need to be taken into consideration before adding the next node to the cluster:
Please verify these points on all the UIs of the vManage nodes that are added to cluster so far:
Once all the controllers are onboarded, complete this step:
Note: While collecting configuration-database backup from the existing vManage cluster which has Disaster recovery enabled, make sure it is collected after the Disaster recovery on that node is paused and deleted.
Confirm there is no ongoing Disaster recovery replication. Navigate to Administration > Disaster Recovery and make sure the status Success and not in a transient state such as Import Pending, Export Pending, or Download Pending. If the status is not success, reach out to Cisco TAC and make sure replication is successful before you proceed to pause the disaster recovery.
First Pause the disaster recovery and make sure the task is complete. And then Delete the Disaster recovery and confirm the task is completed.

Reach out to Cisco TAC to ensure the Disaster Recovery is successfully cleaned up.
You can verify the same using the command requestnmsconfiguration-dbstatus on vManageCLI. The output is as shown
vmanage# request nms configuration-db status
NMS configuration database
Enabled: true
Status: running PID:32632 for 1066085s
Native metrics status: ENABLED
Server-load metrics status: ENABLED
vmanage#
vManage-3# request nms configuration-db diagnostics
NMS configuration database
Checking cluster connectivity for ports 7687,7474 ...
Pinging vManage node 0 on 169.254.1.5:7687,7474...
Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2026-02-18 12:41 UTC
SENT (0.0013s) Starting TCP Handshake > 169.254.1.5:7474
RCVD (0.0022s) Handshake with 169.254.1.5:7474 completed
SENT (1.0024s) Starting TCP Handshake > 169.254.1.5:7687
RCVD (1.0028s) Handshake with 169.254.1.5:7687 completed
SENT (2.0044s) Starting TCP Handshake > 169.254.1.5:7474
RCVD (2.0050s) Handshake with 169.254.1.5:7474 completed
SENT (3.0064s) Starting TCP Handshake > 169.254.1.5:7687
RCVD (3.0072s) Handshake with 169.254.1.5:7687 completed
SENT (4.0083s) Starting TCP Handshake > 169.254.1.5:7474
RCVD (4.0091s) Handshake with 169.254.1.5:7474 completed
SENT (5.0106s) Starting TCP Handshake > 169.254.1.5:7687
RCVD (5.0115s) Handshake with 169.254.1.5:7687 completed
Max rtt: 0.906ms | Min rtt: 0.392ms | Avg rtt: 0.724ms
TCP connection attempts: 6 | Successful connections: 6 | Failed: 0 (0.00%)
Nping done: 1 IP address pinged in 5.01 seconds
Pinging vManage node 1 on 169.254.2.5:7687,7474...
========== SNIP ==========
Connecting to 10.10.10.3...
+------------------------------------------------------------------------------------+
| type | row | attributes[row]["value"] |
+------------------------------------------------------------------------------------+
| "StoreSizes" | "TotalStoreSize" | 85828934 |
| "PageCache" | "Flush" | 4268666 |
| "PageCache" | "EvictionExceptions" | 0 |
| "PageCache" | "UsageRatio" | 0.09724264705882353 |
| "PageCache" | "Eviction" | 2068 |
| "PageCache" | "HitRatio" | 1.0 |
| "ID Allocations" | "NumberOfRelationshipIdsInUse" | 2068 |
| "ID Allocations" | "NumberOfPropertyIdsInUse" | 56151 |
| "ID Allocations" | "NumberOfNodeIdsInUse" | 7561 |
| "ID Allocations" | "NumberOfRelationshipTypeIdsInUse" | 31 |
| "Transactions" | "LastCommittedTxId" | 214273 |
| "Transactions" | "NumberOfOpenTransactions" | 1 |
| "Transactions" | "NumberOfOpenedTransactions" | 441742 |
| "Transactions" | "PeakNumberOfConcurrentTransactions" | 11 |
| "Transactions" | "NumberOfCommittedTransactions" | 414568 |
| "Causal Cluster" | "IsLeader" | 1 >>>>>>>>> |
| "Causal Cluster" | "MsgProcessDelay" | 0 |
| "Causal Cluster" | "InFlightCacheTotalBytes" | 0 |
+------------------------------------------------------------------------------------+
18 rows
ready to start consuming query after 388 ms, results consumed after another 13 ms
Completed
Connecting to 10.10.10.3...
Displaying the Neo4j Cluster Status
+---------------------------------------------------------------------------------------------------------------------------------+
| name | aliases | access | address | role | requestedStatus | currentStatus | error | default | home |
+---------------------------------------------------------------------------------------------------------------------------------+
| "neo4j" | [] | "read-write" | "169.254.3.5:7687" | "leader" | "online" | "online" | "" | TRUE | TRUE |
| "neo4j" | [] | "read-write" | "169.254.2.5:7687" | "follower" | "online" | "online" | "" | TRUE | TRUE |
| "neo4j" | [] | "read-write" | "169.254.1.5:7687" | "follower" | "online" | "online" | "" | TRUE | TRUE |
| "system" | [] | "read-write" | "169.254.3.5:7687" | "follower" | "online" | "online" | "" | FALSE | FALSE |
| "system" | [] | "read-write" | "169.254.2.5:7687" | "follower" | "online" | "online" | "" | FALSE | FALSE |
| "system" | [] | "read-write" | "169.254.1.5:7687" | "leader" | "online" | "online" | "" | FALSE | FALSE |
+---------------------------------------------------------------------------------------------------------------------------------+
6 rows
ready to start consuming query after 256 ms, results consumed after another 3 ms
Completed
Total disk space used by configuration-db:
60M .
Use this command to collect the configuration-db backup from the identified configuration-db leader vManage node.
request nms configuration-db backup path /opt/data/backup/<filename>
The expected output is as shown:
vmanage# request nms configuration-db backup path /opt/data/backup/june18th
Starting backup of configuration-db
config-db backup logs are available in /var/log/nms/neo4j-backup.log file
Successfully saved backup to /opt/data/backup/june18th.tar.gz
sha256sum: 8d0f5af8aee4e70f05e3858be6bdd5e6c136134ae47c383569ec883080f5d359
Removing the temp staging dir :/opt/data/backup/staging
vmanage#
Copy the configuration-db backup to /home/admin/ directory of vManage using SCP.
Sample scp command output:
XXXXXXXXX Downloads % scp june18th.tar.gz admin@10.66.62.27:/home/admin/
viptela 20.15.4.1
(admin@10.66.62.27) Password:
(admin@10.66.62.27) Password:
june18th.tar.gz 0% 255KB 47.2KB/s - stalled -
To restore configuration-db backup, first we need to configure the configuration-db credentials. If your configuration-db credentials are default(neo4j/password), we can skip this step.
To configure configuration-db credentials, use the command request nms configuration-db update-admin-user.Use the username and password of your choice.
Kindly note that the Application server of vManage is restarted. Due to which vManage UI becomes inaccessible for a short time.
vmanage# request nms configuration-db update-admin-user
configuration-db
Enter current user name:neo4j
Enter current user password:password
Enter new user name:ciscoadmin
Enter new user password:ciscoadmin
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Successfully updated configuration database admin user(this is service node, please repeat same operation on all service/data nodes)
Successfully restarted vManage Device Data Collector
Successfully restarted NMS application server
Successfully restarted NMS data collection agent
vmanage#
Post which we can proceed to retore the configuration-db backup:
We can use the command request nms configuration-db restore path /home/admin/< >to restore the configuration-db to the new vManage:
vmanage# request nms configuration-db restore path /home/admin/june18th.tar.gz
Starting backup of configuration-db
config-db backup logs are available in /var/log/nms/neo4j-backup.log file
Successfully saved database to /opt/data/backup/configdb-local-tmp-20230623-160954.tar.gz
Successfully backup database to /opt/data/backup/configdb-local-tmp-20230623-160954.tar.gz
Configuration database is running in a standalone mode
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Successfully saved cluster configuration for localhost
Successfully saved vManage root CA information for device: "53f95156-f56b-472f-b713-d164561b25b7"
Stopping NMS application server on localhost
Stopping NMS configuration database on localhost
Reseting NMS configuration database on localhost
Loading NMS configuration database on localhost
Starting NMS configuration database on localhost
Waiting for 180s or the instance to start...
NMS configuration database on localhost has started.
Updating DB with the saved cluster configuration data
Successfully reinserted cluster meta information
Successfully reinserted vmanage root ca information
Starting NMS application server on localhost
Waiting for 180s for the instance to start...
Successfully restored database
Once the configuration-db is restored, make sure the vManage UI is accessible. Wait for around 5 minutes and then attempt to access the UI.
Once logged into UI successfully, ensure the Edge routers list, template, policies and all the rest of the configurations that were present on your previous or existing vManage UI is reflected on the new vManage UI.
Once configuration-db is restored ,we need to reauthenticate all the new controllers (vmanage/vsmart/vbond) in the fabric
Note: In actual production if the interface IP used to re-authenticate is the tunnel interface IP, need to ensure NETCONF service is allowed on the tunnel interface of the vManage, vSmart and vBond and also on the firewalls along the path. The firewall port to open is TCP port 830 as bi-directional rule from DR cluster to all vBonds and vSmarts .
On vmanage UI, click on Configuration > Devices > Controllers

Once all the controllers are onboarded, complete this step:
On any Cisco SD-WAN Manager server in the newly active cluster, perform these actions:
Enter this command to synchronize the root certificate with all Cisco Catalyst SD-WAN devices in the newly active cluster:
https://vmanage-url/dataservice/system/device/sync/rootcertchain
Enter this command to synchronize the Cisco SD-WAN Manager UUID with the Cisco SD-WAN Validator:
https://vmanage-url/dataservice/certificate/syncvbond
Once the fabric is restored and the control and bfd sessions are up for all edges and controlllers in the fabric,we need to invalidate the old controllers (vmanage/vsmart/vbond) from the UI
Note: Continue with the Post Checks section shown here, which is common to all deployment combinations.
What is Manual/Cold Standby DR ? The backup SD-WAN Manager server or SD-WAN Manager cluster is kept shutdown in cold standby state.
Regular backups of the active database are taken, and if the primary SD-WAN Manager or SD-WAN Manager cluster goes down, the standby SD-WAN Manager or SD-WAN Manager cluster is brought up manually and the backup database restored on it.
Instances needed:
Steps:
Ensure that the number of the activeCisco SD-WAN Managerinstances are identical to the number of the newly installedCisco SD-WAN Managerinstances.
Ensure that all the active and new Cisco SD-WAN Manager instances run the same software version.
Ensure that all the active and new Cisco SD-WAN Manager instances are able to reach the management IP address of the Cisco SD-WAN Validator.
Ensure that certificates have been installed on the newly installed Cisco SD-WAN Manager instances.
Ensure that the clocks on all Cisco Catalyst SD-WAN devices, including the newly installed Cisco SD-WAN Manager instances, are synchronized.
Ensure that a new set of System IPs and Site IDs is configured on the newly installed Cisco SD-WAN Manager instances, along with the same basic configuration as the active cluster.




In case, if we are using our own CA, Enterprise certificate authority, choose Enterprise.


Navigate to Configuration > Devices > Control Components in case of 20.15/20.18 vManage nodes. In case of 20.9/20.12 versions, Configuration > Devices > Controllers
Note: We need to useadmin credentials ofvBondor a user part ofnetadmingroup. You can verify this in the CLI of thevBond. Choose Yes in the dropdown of“Generate CSR" if we need to install a new certificate forvBond
Note: If the vBond is behind a NAT device/Firewall, check if the vBond VPN 0 interface IP is translated to a public IP. If VPN 0 interface IP is not reachable from vManage, use the public IP address of VPN 0 interface in this step

Click on Add vSmart in case of 20.12 vManage or Add Controller in case of 20.15/20.18 vManage.
A pop up opens, enter the VPN 0 transport IP of vSmart which is reachable from the vManage.
Check the reachability using ping if allowed from CLI of vManage to vSmart IP.
Enter the user credentials of vSmart Note that we need to use admin credentials of vSmart or a user part of netadmin group.
You can verify this in the CLI of the vSmart.
Set the protocol to TLS, if we intend to use TLS for routers to establish control connections with vSmart. This config needs to be configured on CLI of vSmarts and vManage nodes as well.
Choose Yes in the dropdown of "Generate CSR" if we need to install a new certificate for vSmart.
Note: If the vSmart is behind NAT device/Firewall, check if the vSmart VPN 0 interface IP is translated to a public IP, and if VPN 0 interface IP is not reachable from vManage, use public IP address of VPN 0 interface IP in this step.

Once all the steps are completed, verify that all the control components are reachable in Monitor>Dashboard


Note: vManage Cluster can be configured with 3 vManage nodes or 6 vManage nodes depending on the number of sites onboarded to SD-WAN fabric
Proceed with the steps shared in "Onboard SD-WAN controllers with a single node vManage in the SD-WAN overlay" to first bring up the SD-WAN fabric with one vManage node and onboard all the required Validators(vBond) and Controllers(vSmart).
config t
system
host-name <hostname>
system-ip <unique system-ip>
site-id <site-id>
organization-name <organization name>
vbond <IP address/URL of vBond>
commit
Note: If we are using URL as vBond address, make sure to configure DNS server IP addresses in VPN 0 configuration or ensure they can be resolved.
These configurations are needed to enable the transport interface used to establish control connections with the routers and rest of the controllers.
config t
vpn 0
dns <IP-address> primary
dns <IP-address> secondary
interface eth1
ip address <IP-address/mask>
tunnel-interface
allow-service all
allow-service dhcp
allow-service dns
allow-service icmp
no allow-service sshd
no allow-service netconf
no allow-service ntp
no allow-service stun
allow-service https
!
no shutdown
!
ip route 0.0.0.0/0 <default-gateway IP>
commit
Also configure VPN 512management interfaceto enable out of band management access to the controller.
Conf t
vpn 512
interface eth0
ip address <IP-address/mask>
no shutdown
!
ip route 0.0.0.0/0 <default-gateway IP>
!
Commit
Optional Config:
Conf t
security
control
protocol tls
commit
Configure service interface on all thevManagenodes including vManage-1 which has been onboarded already. This interface is used for cluster communication, meaning communication between thevManagenodes in the cluster.
conf t
interface eth2
ip address <IP-address/mask>
no shutdown
commit
Make sure thesame IP subnet is used for service interface across all the nodesin thevManagecluster.
We can use the same admin credentials of thevManagenodes to configure thevManagecluster. Else we can configure a new user credential which is part ofnetadmingroup. The configurations to configure new user credential is as shown
conf t
system
aaa
user <username>
password <password>
group netadmin
commit
Make sure toconfigure the same user credentials across all thevManagenodeswhich is part of the cluster.If we decide to use admin credentials, it must be same username/password across all thevManagenodes.
Navigate to Configuration > Certificates > Control Components in case of 20.15/20.18 vManage nodes. In case of 20.9/20.12 versions, Configuration > Devices > Controllers
Click on ... for Manager/vManage and click on Generate CSR.

Once the CSR is generated, you can download the CSR and get it signed based on the Certificate authority chosen for controllers. You can verify this configuration in Administration > Settings > Controller Certificate Authorization. If Cisco (Recommended) is chosen, then the CSR is automatically uploaded to the PNP portal by the vManage and once the certificate is signed, it is installed on the vManage automatically.
If Manual is chosen, manually sign the CSR using the Cisco PNP portal by navigating to smart account and virtual account of the respective SD-WAN overlay.
Once the certificate is available from PNP portal, click on install certificate on the same section of vManage and upload the certificate and install the certificate.
Same procedure is applicable if we are using Digicert and Enterprise Root Certificate.
Complete this step across all the vManage nodes which is part of the cluster.
Note: For a 3-node cluster, all 3 vManage nodes are brought up with compute+data as the persona.
Optional Config:
Please refer to this configuration in your existing cluster to Enable SDAVC - Need to be checked only if it is required and is needed only on one vManage node of the cluster.
Click on Update.



These points need to be taken into consideration before adding the next node to the cluster:
Please verify these points on all the UIs of the vManage nodes that are added to cluster so far:
You can bring up one more vManage cluster using steps described in Step 4: Build vManage Cluster. Post that complete the steps described in Step 6: Config-db Backup/Restore to restore the config-db backup in the Standby cluster.
You can verify the same using the command requestnmsconfiguration-dbstatus on vManageCLI. The output is as shown
vmanage# request nms configuration-db status
NMS configuration database
Enabled: true
Status: running PID:32632 for 1066085s
Native metrics status: ENABLED
Server-load metrics status: ENABLED
vmanage#
vManage-3# request nms configuration-db diagnostics
NMS configuration database
Checking cluster connectivity for ports 7687,7474 ...
Pinging vManage node 0 on 169.254.1.5:7687,7474...
Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2026-02-18 12:41 UTC
SENT (0.0013s) Starting TCP Handshake > 169.254.1.5:7474
RCVD (0.0022s) Handshake with 169.254.1.5:7474 completed
SENT (1.0024s) Starting TCP Handshake > 169.254.1.5:7687
RCVD (1.0028s) Handshake with 169.254.1.5:7687 completed
SENT (2.0044s) Starting TCP Handshake > 169.254.1.5:7474
RCVD (2.0050s) Handshake with 169.254.1.5:7474 completed
SENT (3.0064s) Starting TCP Handshake > 169.254.1.5:7687
RCVD (3.0072s) Handshake with 169.254.1.5:7687 completed
SENT (4.0083s) Starting TCP Handshake > 169.254.1.5:7474
RCVD (4.0091s) Handshake with 169.254.1.5:7474 completed
SENT (5.0106s) Starting TCP Handshake > 169.254.1.5:7687
RCVD (5.0115s) Handshake with 169.254.1.5:7687 completed
Max rtt: 0.906ms | Min rtt: 0.392ms | Avg rtt: 0.724ms
TCP connection attempts: 6 | Successful connections: 6 | Failed: 0 (0.00%)
Nping done: 1 IP address pinged in 5.01 seconds
Pinging vManage node 1 on 169.254.2.5:7687,7474...
========== SNIP ==========
Connecting to 10.10.10.3...
+------------------------------------------------------------------------------------+
| type | row | attributes[row]["value"] |
+------------------------------------------------------------------------------------+
| "StoreSizes" | "TotalStoreSize" | 85828934 |
| "PageCache" | "Flush" | 4268666 |
| "PageCache" | "EvictionExceptions" | 0 |
| "PageCache" | "UsageRatio" | 0.09724264705882353 |
| "PageCache" | "Eviction" | 2068 |
| "PageCache" | "HitRatio" | 1.0 |
| "ID Allocations" | "NumberOfRelationshipIdsInUse" | 2068 |
| "ID Allocations" | "NumberOfPropertyIdsInUse" | 56151 |
| "ID Allocations" | "NumberOfNodeIdsInUse" | 7561 |
| "ID Allocations" | "NumberOfRelationshipTypeIdsInUse" | 31 |
| "Transactions" | "LastCommittedTxId" | 214273 |
| "Transactions" | "NumberOfOpenTransactions" | 1 |
| "Transactions" | "NumberOfOpenedTransactions" | 441742 |
| "Transactions" | "PeakNumberOfConcurrentTransactions" | 11 |
| "Transactions" | "NumberOfCommittedTransactions" | 414568 |
| "Causal Cluster" | "IsLeader" | 1 >>>>>>>>> |
| "Causal Cluster" | "MsgProcessDelay" | 0 |
| "Causal Cluster" | "InFlightCacheTotalBytes" | 0 |
+------------------------------------------------------------------------------------+
18 rows
ready to start consuming query after 388 ms, results consumed after another 13 ms
Completed
Connecting to 10.10.10.3...
Displaying the Neo4j Cluster Status
+---------------------------------------------------------------------------------------------------------------------------------+
| name | aliases | access | address | role | requestedStatus | currentStatus | error | default | home |
+---------------------------------------------------------------------------------------------------------------------------------+
| "neo4j" | [] | "read-write" | "169.254.3.5:7687" | "leader" | "online" | "online" | "" | TRUE | TRUE |
| "neo4j" | [] | "read-write" | "169.254.2.5:7687" | "follower" | "online" | "online" | "" | TRUE | TRUE |
| "neo4j" | [] | "read-write" | "169.254.1.5:7687" | "follower" | "online" | "online" | "" | TRUE | TRUE |
| "system" | [] | "read-write" | "169.254.3.5:7687" | "follower" | "online" | "online" | "" | FALSE | FALSE |
| "system" | [] | "read-write" | "169.254.2.5:7687" | "follower" | "online" | "online" | "" | FALSE | FALSE |
| "system" | [] | "read-write" | "169.254.1.5:7687" | "leader" | "online" | "online" | "" | FALSE | FALSE |
+---------------------------------------------------------------------------------------------------------------------------------+
6 rows
ready to start consuming query after 256 ms, results consumed after another 3 ms
Completed
Total disk space used by configuration-db:
60M .
Use this command to collect the configuration-db backup from the identified configuration-db leader vManage node.
request nms configuration-db backup path /opt/data/backup/<filename>
The expected output is as shown:
vmanage# request nms configuration-db backup path /opt/data/backup/june18th
Starting backup of configuration-db
config-db backup logs are available in /var/log/nms/neo4j-backup.log file
Successfully saved backup to /opt/data/backup/june18th.tar.gz
sha256sum: 8d0f5af8aee4e70f05e3858be6bdd5e6c136134ae47c383569ec883080f5d359
Removing the temp staging dir :/opt/data/backup/staging
vmanage#
Copy the configuration-db backup to /home/admin/ directory of vManage using SCP.
Sample scp command output:
XXXXXXXXX Downloads % scp june18th.tar.gz admin@10.66.62.27:/home/admin/
viptela 20.15.4.1
(admin@10.66.62.27) Password:
(admin@10.66.62.27) Password:
june18th.tar.gz 0% 255KB 47.2KB/s - stalled -
To restore configuration-db backup, first we need to configure the configuration-db credentials. If your configuration-db credentials are default(neo4j/password), we can skip this step.
To configure configuration-db credentials, use the command request nms configuration-db update-admin-user.Use the username and password of your choice.
Kindly note that the Application server of vManage is restarted. Due to which vManage UI becomes inaccessible for a short time.
vmanage# request nms configuration-db update-admin-user
configuration-db
Enter current user name:neo4j
Enter current user password:password
Enter new user name:ciscoadmin
Enter new user password:ciscoadmin
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Successfully updated configuration database admin user(this is service node, please repeat same operation on all service/data nodes)
Successfully restarted vManage Device Data Collector
Successfully restarted NMS application server
Successfully restarted NMS data collection agent
vmanage#
Post which we can proceed to retore the configuration-db backup:
We can use the command request nms configuration-db restore path /home/admin/< >to restore the configuration-db to the new vManage:
vmanage# request nms configuration-db restore path /home/admin/june18th.tar.gz
Starting backup of configuration-db
config-db backup logs are available in /var/log/nms/neo4j-backup.log file
Successfully saved database to /opt/data/backup/configdb-local-tmp-20230623-160954.tar.gz
Successfully backup database to /opt/data/backup/configdb-local-tmp-20230623-160954.tar.gz
Configuration database is running in a standalone mode
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Successfully saved cluster configuration for localhost
Successfully saved vManage root CA information for device: "53f95156-f56b-472f-b713-d164561b25b7"
Stopping NMS application server on localhost
Stopping NMS configuration database on localhost
Reseting NMS configuration database on localhost
Loading NMS configuration database on localhost
Starting NMS configuration database on localhost
Waiting for 180s or the instance to start...
NMS configuration database on localhost has started.
Updating DB with the saved cluster configuration data
Successfully reinserted cluster meta information
Successfully reinserted vmanage root ca information
Starting NMS application server on localhost
Waiting for 180s for the instance to start...
Successfully restored database
Once the configuration-db is restored, make sure the vManage UI is accessible. Wait for around 5 minutes and then attempt to access the UI.
Once logged into UI successfully, ensure the Edge routers list, template, policies and all the rest of the configurations that were present on your previous or existing vManage UI is reflected on the new vManage UI.
Once configuration-db is restored ,we need to reauthenticate all the new controllers (vmanage/vsmart/vbond) in the fabric
Note: In actual production if the interface IP used to re-authenticate is the tunnel interface IP, need to ensure NETCONF service is allowed on the tunnel interface of the vManage, vSmart and vBond and also on the firewalls along the path. The firewall port to open is TCP port 830 as bi-directional rule from DR cluster to all vBonds and vSmarts .
On vmanage UI, click on Configuration > Devices > Controllers

Once all the controllers are onboarded, complete this step:
On any Cisco SD-WAN Manager server in the newly active cluster, perform these actions:
Enter this command to synchronize the root certificate with all Cisco Catalyst SD-WAN devices in the newly active cluster:
https://vmanage-url/dataservice/system/device/sync/rootcertchain
Enter this command to synchronize the Cisco SD-WAN Manager UUID with the Cisco SD-WAN Validator:
https://vmanage-url/dataservice/certificate/syncvbond
Once the fabric is restored and the control and bfd sessions are up for all edges and controlllers in the fabric,we need to invalidate the old controllers (vmanage/vsmart/vbond) from the UI
Note: Continue with the Post Checks section shown here, which is common to all deployment combinations.
Instances needed:
Steps:
Ensure that the number of the activeCisco SD-WAN Managerinstances are identical to the number of the newly installedCisco SD-WAN Managerinstances.
Ensure that all the active and new Cisco SD-WAN Manager instances run the same software version.
Ensure that all the active and new Cisco SD-WAN Manager instances are able to reach the management IP address of the Cisco SD-WAN Validator.
Ensure that certificates have been installed on the newly installed Cisco SD-WAN Manager instances.
Ensure that the clocks on all Cisco Catalyst SD-WAN devices, including the newly installed Cisco SD-WAN Manager instances, are synchronized.
Ensure that a new set of System IPs and Site IDs is configured on the newly installed Cisco SD-WAN Manager instances, along with the same basic configuration as the active cluster.




In case, if we are using our own CA, Enterprise certificate authority, choose Enterprise.


Navigate to Configuration > Devices > Control Components in case of 20.15/20.18 vManage nodes. In case of 20.9/20.12 versions, Configuration > Devices > Controllers
Note: We need to useadmin credentials ofvBondor a user part ofnetadmingroup. You can verify this in the CLI of thevBond. Choose Yes in the dropdown of“Generate CSR" if we need to install a new certificate forvBond
Note: If the vBond is behind a NAT device/Firewall, check if the vBond VPN 0 interface IP is translated to a public IP. If VPN 0 interface IP is not reachable from vManage, use the public IP address of VPN 0 interface in this step

Click on Add vSmart in case of 20.12 vManage or Add Controller in case of 20.15/20.18 vManage.
A pop up opens, enter the VPN 0 transport IP of vSmart which is reachable from the vManage.
Check the reachability using ping if allowed from CLI of vManage to vSmart IP.
Enter the user credentials of vSmart Note that we need to use admin credentials of vSmart or a user part of netadmin group.
You can verify this in the CLI of the vSmart.
Set the protocol to TLS, if we intend to use TLS for routers to establish control connections with vSmart. This config needs to be configured on CLI of vSmarts and vManage nodes as well.
Choose Yes in the dropdown of "Generate CSR" if we need to install a new certificate for vSmart.
Note: If the vSmart is behind NAT device/Firewall, check if the vSmart VPN 0 interface IP is translated to a public IP, and if VPN 0 interface IP is not reachable from vManage, use public IP address of VPN 0 interface IP in this step.

Once all the steps are completed, verify that all the control components are reachable in Monitor>Dashboard


Note: vManage Cluster can be configured with 3 vManage nodes or 6 vManage nodes depending on the number of sites onboarded to SD-WAN fabric
Proceed with the steps shared in "Onboard SD-WAN controllers with a single node vManage in the SD-WAN overlay" to first bring up the SD-WAN fabric with one vManage node and onboard all the required Validators(vBond) and Controllers(vSmart).
config t
system
host-name <hostname>
system-ip <unique system-ip>
site-id <site-id>
organization-name <organization name>
vbond <IP address/URL of vBond>
commit
Note: If we are using URL as vBond address, make sure to configure DNS server IP addresses in VPN 0 configuration or ensure they can be resolved.
These configurations are needed to enable the transport interface used to establish control connections with the routers and rest of the controllers.
config t
vpn 0
dns <IP-address> primary
dns <IP-address> secondary
interface eth1
ip address <IP-address/mask>
tunnel-interface
allow-service all
allow-service dhcp
allow-service dns
allow-service icmp
no allow-service sshd
no allow-service netconf
no allow-service ntp
no allow-service stun
allow-service https
!
no shutdown
!
ip route 0.0.0.0/0 <default-gateway IP>
commit
Also configure VPN 512management interfaceto enable out of band management access to the controller.
Conf t
vpn 512
interface eth0
ip address <IP-address/mask>
no shutdown
!
ip route 0.0.0.0/0 <default-gateway IP>
!
Commit
Optional Config:
Conf t
security
control
protocol tls
commit
Configure service interface on all thevManagenodes including vManage-1 which has been onboarded already. This interface is used for cluster communication, meaning communication between thevManagenodes in the cluster.
conf t
interface eth2
ip address <IP-address/mask>
no shutdown
commit
Make sure thesame IP subnet is used for service interface across all the nodesin thevManagecluster.
We can use the same admin credentials of thevManagenodes to configure thevManagecluster. Else we can configure a new user credential which is part ofnetadmingroup. The configurations to configure new user credential is as shown
conf t
system
aaa
user <username>
password <password>
group netadmin
commit
Make sure toconfigure the same user credentials across all thevManagenodeswhich is part of the cluster.If we decide to use admin credentials, it must be same username/password across all thevManagenodes.
Navigate to Configuration > Certificates > Control Components in case of 20.15/20.18 vManage nodes. In case of 20.9/20.12 versions, Configuration > Devices > Controllers
Click on ... for Manager/vManage and click on Generate CSR.

Once the CSR is generated, you can download the CSR and get it signed based on the Certificate authority chosen for controllers. You can verify this configuration in Administration > Settings > Controller Certificate Authorization. If Cisco (Recommended) is chosen, then the CSR is automatically uploaded to the PNP portal by the vManage and once the certificate is signed, it is installed on the vManage automatically.
If Manual is chosen, manually sign the CSR using the Cisco PNP portal by navigating to smart account and virtual account of the respective SD-WAN overlay.
Once the certificate is available from PNP portal, click on install certificate on the same section of vManage and upload the certificate and install the certificate.
Same procedure is applicable if we are using Digicert and Enterprise Root Certificate.
Complete this step across all the vManage nodes which is part of the cluster.
Note: For a 3-node cluster, all 3 vManage nodes are brought up with compute+data as the persona. For a 6-node cluster, 3 vManage nodes are brought up with compute+data as the persona and 3 vManage nodes are brought up with data as the persona.

Optional Config:
Please refer to this configuration in your existing cluster to Enable SDAVC - Need to be checked only if it is required and is needed only on one vManage node of the cluster.
Click on Update.



These points need to be taken into consideration before adding the next node to the cluster:
Please verify these points on all the UIs of the vManage nodes that are added to cluster so far:
Note: While collecting configuration-database backup from the existing vManage cluster which has Disaster recovery enabled, make sure it is collected after the Disaster recovery on that node is paused and deleted.
Confirm there is no ongoing Disaster recovery replication. Navigate to Administration > Disaster Recovery and make sure the status Success and not in a transient state such as Import Pending, Export Pending, or Download Pending. If the status is not success, reach out to Cisco TAC and make sure replication is successful before you proceed to pause the disaster recovery.
First Pause the disaster recovery and make sure the task is complete. And then Delete the Disaster recovery and confirm the task is completed.

Reach out to Cisco TAC to ensure the Disaster Recovery is successfully cleaned up.
You can verify the same using the command requestnmsconfiguration-dbstatus on vManageCLI. The output is as shown
vmanage# request nms configuration-db status
NMS configuration database
Enabled: true
Status: running PID:32632 for 1066085s
Native metrics status: ENABLED
Server-load metrics status: ENABLED
vmanage#
vManage-3# request nms configuration-db diagnostics
NMS configuration database
Checking cluster connectivity for ports 7687,7474 ...
Pinging vManage node 0 on 169.254.1.5:7687,7474...
Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2026-02-18 12:41 UTC
SENT (0.0013s) Starting TCP Handshake > 169.254.1.5:7474
RCVD (0.0022s) Handshake with 169.254.1.5:7474 completed
SENT (1.0024s) Starting TCP Handshake > 169.254.1.5:7687
RCVD (1.0028s) Handshake with 169.254.1.5:7687 completed
SENT (2.0044s) Starting TCP Handshake > 169.254.1.5:7474
RCVD (2.0050s) Handshake with 169.254.1.5:7474 completed
SENT (3.0064s) Starting TCP Handshake > 169.254.1.5:7687
RCVD (3.0072s) Handshake with 169.254.1.5:7687 completed
SENT (4.0083s) Starting TCP Handshake > 169.254.1.5:7474
RCVD (4.0091s) Handshake with 169.254.1.5:7474 completed
SENT (5.0106s) Starting TCP Handshake > 169.254.1.5:7687
RCVD (5.0115s) Handshake with 169.254.1.5:7687 completed
Max rtt: 0.906ms | Min rtt: 0.392ms | Avg rtt: 0.724ms
TCP connection attempts: 6 | Successful connections: 6 | Failed: 0 (0.00%)
Nping done: 1 IP address pinged in 5.01 seconds
Pinging vManage node 1 on 169.254.2.5:7687,7474...
========== SNIP ==========
Connecting to 10.10.10.3...
+------------------------------------------------------------------------------------+
| type | row | attributes[row]["value"] |
+------------------------------------------------------------------------------------+
| "StoreSizes" | "TotalStoreSize" | 85828934 |
| "PageCache" | "Flush" | 4268666 |
| "PageCache" | "EvictionExceptions" | 0 |
| "PageCache" | "UsageRatio" | 0.09724264705882353 |
| "PageCache" | "Eviction" | 2068 |
| "PageCache" | "HitRatio" | 1.0 |
| "ID Allocations" | "NumberOfRelationshipIdsInUse" | 2068 |
| "ID Allocations" | "NumberOfPropertyIdsInUse" | 56151 |
| "ID Allocations" | "NumberOfNodeIdsInUse" | 7561 |
| "ID Allocations" | "NumberOfRelationshipTypeIdsInUse" | 31 |
| "Transactions" | "LastCommittedTxId" | 214273 |
| "Transactions" | "NumberOfOpenTransactions" | 1 |
| "Transactions" | "NumberOfOpenedTransactions" | 441742 |
| "Transactions" | "PeakNumberOfConcurrentTransactions" | 11 |
| "Transactions" | "NumberOfCommittedTransactions" | 414568 |
| "Causal Cluster" | "IsLeader" | 1 >>>>>>>>> |
| "Causal Cluster" | "MsgProcessDelay" | 0 |
| "Causal Cluster" | "InFlightCacheTotalBytes" | 0 |
+------------------------------------------------------------------------------------+
18 rows
ready to start consuming query after 388 ms, results consumed after another 13 ms
Completed
Connecting to 10.10.10.3...
Displaying the Neo4j Cluster Status
+---------------------------------------------------------------------------------------------------------------------------------+
| name | aliases | access | address | role | requestedStatus | currentStatus | error | default | home |
+---------------------------------------------------------------------------------------------------------------------------------+
| "neo4j" | [] | "read-write" | "169.254.3.5:7687" | "leader" | "online" | "online" | "" | TRUE | TRUE |
| "neo4j" | [] | "read-write" | "169.254.2.5:7687" | "follower" | "online" | "online" | "" | TRUE | TRUE |
| "neo4j" | [] | "read-write" | "169.254.1.5:7687" | "follower" | "online" | "online" | "" | TRUE | TRUE |
| "system" | [] | "read-write" | "169.254.3.5:7687" | "follower" | "online" | "online" | "" | FALSE | FALSE |
| "system" | [] | "read-write" | "169.254.2.5:7687" | "follower" | "online" | "online" | "" | FALSE | FALSE |
| "system" | [] | "read-write" | "169.254.1.5:7687" | "leader" | "online" | "online" | "" | FALSE | FALSE |
+---------------------------------------------------------------------------------------------------------------------------------+
6 rows
ready to start consuming query after 256 ms, results consumed after another 3 ms
Completed
Total disk space used by configuration-db:
60M .
Use this command to collect the configuration-db backup from the identified configuration-db leader vManage node.
request nms configuration-db backup path /opt/data/backup/<filename>
The expected output is as shown:
vmanage# request nms configuration-db backup path /opt/data/backup/june18th
Starting backup of configuration-db
config-db backup logs are available in /var/log/nms/neo4j-backup.log file
Successfully saved backup to /opt/data/backup/june18th.tar.gz
sha256sum: 8d0f5af8aee4e70f05e3858be6bdd5e6c136134ae47c383569ec883080f5d359
Removing the temp staging dir :/opt/data/backup/staging
vmanage#
Copy the configuration-db backup to /home/admin/ directory of vManage using SCP.
Sample scp command output:
XXXXXXXXX Downloads % scp june18th.tar.gz admin@10.66.62.27:/home/admin/
viptela 20.15.4.1
(admin@10.66.62.27) Password:
(admin@10.66.62.27) Password:
june18th.tar.gz 0% 255KB 47.2KB/s - stalled -
To restore configuration-db backup, first we need to configure the configuration-db credentials. If your configuration-db credentials are default(neo4j/password), we can skip this step.
To configure configuration-db credentials, use the command request nms configuration-db update-admin-user.Use the username and password of your choice.
Kindly note that the Application server of vManage is restarted. Due to which vManage UI becomes inaccessible for a short time.
vmanage# request nms configuration-db update-admin-user
configuration-db
Enter current user name:neo4j
Enter current user password:password
Enter new user name:ciscoadmin
Enter new user password:ciscoadmin
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Successfully updated configuration database admin user(this is service node, please repeat same operation on all service/data nodes)
Successfully restarted vManage Device Data Collector
Successfully restarted NMS application server
Successfully restarted NMS data collection agent
vmanage#
Post which we can proceed to retore the configuration-db backup:
We can use the command request nms configuration-db restore path /home/admin/< >to restore the configuration-db to the new vManage:
vmanage# request nms configuration-db restore path /home/admin/june18th.tar.gz
Starting backup of configuration-db
config-db backup logs are available in /var/log/nms/neo4j-backup.log file
Successfully saved database to /opt/data/backup/configdb-local-tmp-20230623-160954.tar.gz
Successfully backup database to /opt/data/backup/configdb-local-tmp-20230623-160954.tar.gz
Configuration database is running in a standalone mode
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
Successfully saved cluster configuration for localhost
Successfully saved vManage root CA information for device: "53f95156-f56b-472f-b713-d164561b25b7"
Stopping NMS application server on localhost
Stopping NMS configuration database on localhost
Reseting NMS configuration database on localhost
Loading NMS configuration database on localhost
Starting NMS configuration database on localhost
Waiting for 180s or the instance to start...
NMS configuration database on localhost has started.
Updating DB with the saved cluster configuration data
Successfully reinserted cluster meta information
Successfully reinserted vmanage root ca information
Starting NMS application server on localhost
Waiting for 180s for the instance to start...
Successfully restored database
Once the configuration-db is restored, make sure the vManage UI is accessible. Wait for around 5 minutes and then attempt to access the UI.
Once logged into UI successfully, ensure the Edge routers list, template, policies and all the rest of the configurations that were present on your previous or existing vManage UI is reflected on the new vManage UI.
Important Prechecks
Two separate vManage 3-node clusters must be configured and operational in order to proceed with disaster recovery. On the active cluster you must have validators and controllers onboarded. In case you have validator and controllers on the DR site, they must also be onboarded on the active cluster and not on the DR vManage cluster.
Cisco recommends that before registering disaster recovery, these requirements must be met:
Configurations
For more information on vManage Disaster Recovery, refer to this link.
The two separate 3-node-clusters are already created, assuming each SD-WAN manager has bare minimum configuration and certification part is completed.
Navigate to Administration > Cluster Management on both clusters and verify all nodes are in ready state.
DC vManage

DR vmanage

Navigate to Administration>Disaster Recovery of Primary vManage Cluster. Click Manage Disaster Recovery.

In the pop-up window, fill the details for both primary and secondary vManage.
The IP addresses to be indicated are the out-of-band cluster interfaces IP addresses. Preferrably use the IP address of VPN 0 service interface of vManage-1 in each of the cluster.
The credentials must be those of a netadmin user and they must not be changed once the DR is configured, unless it is deleted. A separate vManage local user credential for Disaster recovery can be used. We need to make sure the vManage local user is part of netadmin group. Even admin credential can be used here.

Once filled, click Next.
The vBond controllers must be reachable in the specified IP address via Netconf.

Once filled, click Next.


Set the value and click Save.

Verification
Navigate to Administration>Disaster Recovery in order to see the Disaster Recovery status and when the data was replicated last time.

Note: Replication can take several hours depending on the database size. Additionally, it can require a few cycles to achieve successful replication.
Once configuration-db is restored ,we need to reauthenticate all the new controllers (vmanage/vsmart/vbond) in the fabric
Note: In actual production if the interface IP used to re-authenticate is the tunnel interface IP, need to ensure NETCONF service is allowed on the tunnel interface of the vManage, vSmart and vBond and also on the firewalls along the path. The firewall port to open is TCP port 830 as bi-directional rule from DR cluster to all vBonds and vSmarts .
On vmanage UI, click on Configuration > Devices > Controllers

Once all the controllers are onboarded, complete this step:
On any Cisco SD-WAN Manager server in the newly active cluster, perform these actions:
Enter this command to synchronize the root certificate with all Cisco Catalyst SD-WAN devices in the newly active cluster:
https://vmanage-url/dataservice/system/device/sync/rootcertchain
Enter this command to synchronize the Cisco SD-WAN Manager UUID with the Cisco SD-WAN Validator:
https://vmanage-url/dataservice/certificate/syncvbond
Once the fabric is restored and the control and bfd sessions are up for all edges and controlllers in the fabric,we need to invalidate the old controllers (vmanage/vsmart/vbond) from the UI
These post checks apply to all deployment combinations.
request platform software sdwan vedge_cloud activate chassis-number <chassis-number > token <token number>
Verify that the items appear as expected:
Templates
Policies
Device page (both tabs)WAN vEdge ListandControllers
Applicable for vManage nodes:
Configuration-DB(Neo4j) Checks:
Execute the command "request nms configuration-db diagnostics" on all the vManage nodes:
1. Check for Node online and Leadership status:(not available for all versions)
“Neo4j” must show 3 nodes online and 1 leader and 2 followers. “system” must also show 3 nodes online and 1 leader and 2 followers, however as this is not the default Db the default flag is false. If there are less than 3 entries in each neo4j and system means node fallen off from cluster. Please contact Cisco TAC to troubleshoot the same.
2. Check if any node is "quarantine".

None of the nodes must be in quarantine state.
3. Schema validation must be "successful".

4. Collect a configuration-db backup using the command "request nms configuration-db diagnostics" and make sure it is successful.

If there are any inconsistency or errors seen, reach out to Cisco TAC for troubleshooting.
Alternatively we can run these API calls to confirm the vmanage node status for a cluster ( for all COMPUTE+DATA nodes) - works on version 20.12 and later only
go to vshell of the vmanage node ( to be done on all vmanages)
===============================================================
curl -u <config-db username>:<config-db password> -H "Content-Type: application/json" -d '{"statements":[{"statement":"call dbms.cluster.overview"}]}' http://<cluster OOB ip>:7474/db/neo4j/tx/commit | jq -r
curl -u <config-db username>:<config-db password> -H "Content-Type: application/json" -d '{"statements":[{"statement":"show databases"}]}' http://<cluster OOB ip>:7474/db/neo4j/tx/commit | jq -rEnsure in a cluster has only one leader for both neo4j and system and rest as followers .Ensure all the nodes are online .If you have 3 node cluster ( all three are COMPUTE+DATA),there must be only one leader for both neo4j and system .Any deviation ,you must contact TAC
5. Check in /var/log/kern.log for any Disk, Mem , IO errors. This needs to be checked on all the vManage nodes.
6.Check the step when you form a fresh cluster for vmanage which dont have CC between each node
Perform ssh as vmanage-admin from node 1 to other nodes cluster ip and vise versa, to check if public key is exchanged and password less ssh is working [Consent token is required for shown here steps]
DR-vManage-1:~# ssh -i /etc/viptela/.ssh/id_dsa -p 830 vmanage-admin@<ClusterIP>
The authenticity of host '[192.168.50.5]:830 ([192.168.50.5]:830)' can't be established.
ECDSA key fingerprint is SHA256:rSpscoYCVC+yifUMHVTlxtjqmyrZSFg93msFdoSUieQ.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[192.168.50.5]:830' (ECDSA) to the list of known hosts.
viptela 20.9.3.0.31
Password:
In case the output is asking to enter the password ,Contact TAC
Applicable for all the SD-WAN controllers (vBond, vManage, vSmart):
Execute the commands on all the controllers in the overlay and confirm the vManage UUID and serial numbers seen are valid for the currently fabric:
vBond commands:
show orchestrator valid-vsmarts
show orchestrator valid-vmanage-id
vManage/vSmart commands:
show control valid-vsmarts
show control valid-vmanage-id
Please note the output of show control valid-vsmarts includes the serial numbers of both vSmarts and vManage nodes.
Compare it with the ones seen in vManage UI. Navigate to section Configuration > Certificates > Controllers.
If any additional entries are seen for the UUID/serial number which are currently not onboarded to the fabric, we must delete them. You can contact Cisco TAC for the same.
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
25-Feb-2026
|
Initial Release |
Feedback