Cisco Catalyst SD-WAN disaster recovery guidelines

About backups and snapshots

• The system takes configuration database backups and disk volume snapshots daily (typically around midnight where vManage is located), securely storing them in the cloud.

• Starting with vEdge release 20.3.x and later, you can disable the configuration database backup feature, make your own backups, and provide them to CloudOps when needed.

• Disk volume snapshots are taken nightly, on-demand for your requests, or at major change windows. Snapshots cover all volumes of the vManage instance.

• Completed snapshots are copied to a designated backup region (usually a different geographic area). For example, if Cisco SD-WAN Manager runs in US-East, backups may go to US-West. The backup region is an identically configured region where the second Cisco Catalyst SD-WAN Validator and Cisco Catalyst SD-WAN Controller are already running.

• Cisco Catalyst SD-WAN Validator and Cisco Catalyst SD-WAN Controller are stateless services; their configuration is managed through Cisco SD-WAN Manager or CLI, so backups are not taken for them.

• High availability for Cisco SD-WAN Manager is managed by a cluster (three or six nodes) within the same availability zone and region. The backup region does not include a standby or active Cisco SD-WAN Manager service.

• Cisco Catalyst SD-WAN Validator and Cisco Catalyst SD-WAN Controller services are deployed in both primary and backup regions; both operate actively, and device and policy information is pushed from Cisco SD-WAN Manager to both. When one region fails, Cisco Catalyst SD-WAN Controller and Cisco Catalyst SD-WAN Validator continue to function in the backup region.

• The configuration database-based recovery method allows the restoration of templates and policies only. In contrast, volume-based recovery includes collected statistics data.

Note

The data plane is designed to continue functioning even if all control components fail, via the Graceful Restart (GR) timer. The GR timer holds routes from vSmart for 12 hours by default. Choose the GR timer value carefully for backup and route learning.

Configuration database backup conditions

Prior to Cisco vManage Release 20.3.1, the configuration database is backed up only if all these conditions are met:

• Monitoring is enabled in the CloudInfra system. If the viptelatac user is unusable on the Cisco SD-WAN Manager for any reason, monitoring is disabled and you are notified with a request for correction.

• The viptelatac user is usable on the Cisco SD-WAN Manager.

• The configuration database size is less than 4 GB.

In Cisco vManage Release 20.3.1 and later, the configuration database is backed up only if all these conditions are met:

• Monitoring is enabled in the CloudInfra system.

  Note

  In Cisco SD-WAN Manager, if the cloud service is disabled for any reason, monitoring is disabled on the CloudInfra system and you are notified with a request for correction.

• The nms configuration-db daily-backup service is enabled in the Cisco SD-WAN Manager CLI.

• Cloud Services, vMonitoring, and OTP are enabled in Cisco SD-WAN Manager Settings.

• The configuration database size is less than 4 GB.

Volume snapshot-based recovery

After the CloudOps team determines that the Cisco SD-WAN Manager instance needs to be replaced with a backup, we can initiate the DR process.

For DR in the same region, we select the same region and datacenter as the current Cisco SD-WAN Manager instance location. We specify the snapshot date and time of the snapshot based on requirements and availability.

Once DR triggers, the system first shuts down the existing Cisco SD-WAN Manager instance.

The system then uses the volume snapshots to create a new cloud instance with the same set of disks, instance size specifications, private subnets, security access list, and isolated environment that the original Cisco SD-WAN Manager had. Once the instance is up, the system swaps the public IPs from the old Cisco SD-WAN Manager instance to the new Cisco SD-WAN Manager instance.

The new running Cisco SD-WAN Manager instance has new private IPs but the same public IPs, software version, configuration, and data as when the snapshot was taken.

Cisco SD-WAN Manager is configured with the information necessary to join the fabric. You can use the same FQDN or URL to log in to the Cisco SD-WAN Manager instance as before.

In the unlikely case where the primary region of Cisco SD-WAN Manager has failed and is unavailable, we use the exact same process for DR to the backup region, except that the backup cloud region is selected.

When the new Cisco SD-WAN Manager instance runs in the backup region, the system does not swap public IPs between regions. Cloud regions have a specific public IP pool per region and cannot be assigned to instances across regions.

Thus, the new DR Cisco SD-WAN Manager instance in the backup region has new public IPs. The system updates the FQDN or DNS with the new public IP of the Cisco SD-WAN Manager.

In this case, you may need to update the enterprise end firewall with the new public IP of the Cisco SD-WAN Manager.

Configuration database-based recovery

If we cannot take a volume snapshot, we use the configuration database recovery process. We create a new Cisco SD-WAN Manager instance and use the configuration database backup to restore the original configuration files. With this method, the statistics database of the original Cisco SD-WAN Manager instance is not restored. This method restores your templates and policies configuration. The new Cisco SD-WAN Manager instance in this case has both new public IPs and new private IPs.

We update the FQDN or DNS of the Cisco SD-WAN Manager to use the new public IP of the new instance.

In this case, you may need to update the enterprise end firewall with the new public IP of the Cisco SD-WAN Manager.

The process for using a configuration database backup for DR is identical for both same region and backup region recovery.

For process details, refer to the section Restore a Cisco SD-WAN Manager Instance from Backup in the Recover Cisco Catalyst SD-WAN Manager Troubleshooting TechNote.