Manage Backups

This section contains the following topics:

Manage Cisco Crosswork Backup and Restore

Cisco Crosswork's backup and restore features help prevent data loss and preserve your installed applications and settings.


Note
If you want to include Cisco NSO data in the Cisco Crosswork backup process, follow the instructions given in Backup Cisco Crosswork with Cisco NSO instead of the instructions here.

When you create backups for the Cisco Crosswork cluster, or restore the cluster from a backup, follow these guidelines:

  • During your first login, configure a destination SCP server to store backup files. This configuration is a one-time activity. You can't take a backup or initiate a restore operation until you complete this task.

  • We recommend that you perform backup or restore operations during a scheduled maintenance window only. Users shouldn’t attempt to access Cisco Crosswork while these operations are running. Backups will take the system offline for about 10 minutes, but restore operations can be lengthy. Both will pause other applications until they are complete. These pauses can affect data-collection jobs.

  • When performing a normal restore, Cisco Crosswork applications and data are restored to the same version as when you took the backup. When performing a disaster restore, you must use the same Cisco Crosswork software image that you used when creating the backup. You can’t perform a disaster restore using a backup created using a different version of the software.

  • Use the dashboard to monitor progress of the backup or restore process, until the process completes. If you attempt to use the Cisco Crosswork system during the process, you may see incorrect content or errors, since various services pause and restart frequently.

  • You can run only one backup or restore operation at a given time.

  • Both the Cisco Crosswork cluster and the SCP server must be in the same IP environment. For example: If Cisco Crosswork is communicating over IPv6, so must the backup server.

  • Don’t attempt to move or rename any of the backup tarballs Cisco Crosswork creates on the backup server. To save space on your backup server, you may delete older backups, but they will still appear in the job list in this version.

Before you begin

Before you begin, ensure that you have:

  • The hostname or IP address and the port number of a secure SCP server.

  • A file path on the SCP server, to use as the destination for your backup files.

  • User credentials for an account with file read and write permissions to the remote path on the destination SCP server.

Procedure

Step 1

Configure an SCP backup server:

  1. From the main menu, choose Administration > Backup and Restore.

  2. Click Destination to display the Edit Destination dialog box. Make the relevant entries in the fields provided.

  3. Click Save to confirm the backup server details.

Step 2

Create a backup:

  1. Click Backup to display the Backup dialog box with the destination server details prefilled.

  2. Provide a relevant name for the backup in the Job Name field.

  3. If you want Cisco Crosswork to take the backup despite application or microservice issues, check the Force check box.

  4. Be sure to uncheck the Backup NSO checkbox.

    If you want to include Cisco NSO data in the Cisco Crosswork backup process, follow the instructions given in Backup Cisco Crosswork with Cisco NSO instead of the instructions here

  5. (Optional) Click Verify Backup to verify that Cisco Crosswork has enough free resources to complete the backup. If the check is successful, Cisco Crosswork displays a warning about the time-consuming nature of the operation. Click OK.

  6. Click Start Backup to start the backup operation. Cisco Crosswork creates the corresponding backup job set and adds it to the job list.

  7. To view the progress of a backup job: Enter the job details (such as Status or Job Type) in the search fields in the Backup Restore Job Sets table. Then click on the job set you want.

    The Job Details table displays information about the selected job set, such as the job Status, Job Type, and Start Time. If there’s a failed job, hover the mouse pointer over the Details icon icon near the Status column to view the error details.

Step 3

To restore from a backup file:

  1. Select the required backup file from the Backup Restore Job Sets table. The page displays the job list on the left, with details for the selected job on the right side.

  2. Click Restore to display the Restore dialog box with the destination server details prefilled.

  3. Provide a relevant name in the Job Name field.

  4. If you want Cisco Crosswork to take the backup despite application or microservice issues, check the Force check box.

  5. (Optional) Click Verify Restore to verify that Cisco Crosswork has enough free resources to complete the restore. If the check is successful, Cisco Crosswork displays a warning about the time-consuming nature of the operation. Click OK.

  6. Click Start Restore to start the restore operation. Cisco Crosswork creates the corresponding restore job set and adds it to the job list.

    To view the progress of the restore operation, click the link to the progress dashboard.

Restore After a Disaster

A disaster recovery is a restore operation that you use after a natural or human-caused disaster has destroyed a Cisco Crosswork cluster. You will need to deploy a new cluster first, following the instructions in the Cisco Crosswork Platform and Applications Installation Guide.

If your cluster only has one malfunctioning hybrid node, or one or more worker nodes, don't perform a disaster recovery. Instead, use cluster management features to redeploy these nodes, or replace them with new nodes, as explained in Manage the Crosswork Cluster chapter.

If you have more than one malfunctioning hybrid node, the system will not be in a functional state. Even if you replace or reboot the failed hybrid nodes, there is no guarantee that the system will recover correctly even if you replace or reboot the failed hybrid nodes. In this case, you can deploy a new cluster, and then recover the entire system using a recent backup taken from the old cluster. For more information, see the Manage the Crosswork Cluster chapter.

When conducting a disaster recovery, note the following:

  • The new Cisco Crosswork cluster to which you restore the backup must use the same IP addresses as the one where you took the backup. This guideline is important, as internal certificates use the IP addresses of the original cluster.

  • The new cluster must have the same number and types of nodes as the cluster where you took the backup.

  • The new cluster must use the same Cisco Crosswork software image that you used when creating the backup. You can’t restore the cluster using a backup that was created using a different version of the software.

  • Keep your backups current, so that you can recover the true state of your system as it existed before the disaster. The restore operation restores all applications that are installed at the time the backup was made. If you have installed more applications or patches since your last backup, take another backup.

  • If the disaster recovery fails, contact Cisco Customer Experience.

To perform a disaster recovery:

Before you begin

Get from the SCP backup server the full name of the backup file you want to use in your disaster recovery. This file is normally the most recent backup file you have made. Cisco Crosswork backup filenames have the following format:

backup_JobName_CWVersion_TimeStamp.tar.gz

Where:

  • JobName is the user-entered name of the backup job.

  • CWVersion is the Cisco Crosswork platform version of the backed-up system.

  • TimeStamp is the date and time when Cisco Crosswork created the backup file.

For example: backup_Wednesday_4-0_2021-02-31-12-00.tar.gz.

Procedure

Step 1

From the main menu of the newly deployed cluster, choose Administration > Backup and Restore.

Step 2

Click Disaster Restore to display the Disaster Restore dialog box with the destination server details pre-filled.

Step 3

Enter the backup filename in the Backup File Name field.

Step 4

Click Start Restore to initiate the disaster recovery operation.

To view the progress of the operation, click the link to the progress dashboard.

Resolve Missing SR-TE Policies and RSVP-TE Tunnels

The information in this topic is applicable only when Cisco Crosswork Optimization Engine is installed.

The Configuration Database contains all SR-TE policies and RSVP-TE tunnels of which Cisco Crosswork is aware. Cisco Crosswork updates the Configuration Database whenever you provision, modify or delete an SR-TE policy or RSVP-TE tunnel. You can use the Configuration Database CLI tool to do the following:

  • Read and write CSV files to the Configuration Database.

  • Populate SR-TE policy and RSVP-TE tunnel information from the Configuration Database to create a CSV file.

The Configuration Database CLI tool is especially useful when trying to recover missing SR-TE policies and RSVP-TE tunnels after a restore operation. For example, the --dump-missing option produces a CSV file which lists the SR-TE policies and RSVP-TE tunnels that are missing. Use this CSV file to determine which SR-TE policies and RSVP-TE tunnels are missing. Then load them back into the topology using the --load option. See the CLI tool help for more information.

Procedure

Step 1

Enter the optima-pce-dispatcher container: 

kubectl exec -it optima-pce-dispatcher-XXXXXXX-XXXX bash
Step 2

You can run the following commands:

  1. Show CLI tool help text.

    python3 /opt/optima/pce_dispatcher/config_db/csv_util.py --help

  2. Save all SR-TE policies and RSVP-TE tunnels that are in the Configuration Database to a CSV file.

    python3 /opt/optima/pce_dispatcher/config_db/csv_util.py --dump /<PathToFile>/dump_file.csv

  3. Load the contents from the provided CSV file and write policies to the Configuration Database.

    python3 /opt/optima/pce_dispatcher/config_db/csv_util.py --load /<PathToFile>/load_file.csv
    Note 

    This command overwrites any duplicate SR-TE policies or RSVP-TE tunnels that it finds, and adds only valid TE tunnels to the Configuration Database. Duplicate SR-TE policies have the same combination of headend, endpoint, and color. Duplicate RSVP-TE tunnels have the same combination of headend and tunnel name.

  4. After the CSV load completes, synchronize the Cisco Crosswork Optimization Engine UI with the Configuration Database by restarting Optimization Engine, as follows:

    1. From the main menu, select Administration > > Crosswork Manager > Crosswork Health > Optimization Engine.

    2. Select optima-ui-service > > Action > Restart. Restart takes approximately five minutes.

  5. After the restart, compare SR-TE policies and RSVP-TE tunnels that are currently in the topology with the Configuration Database contents. Save the missing SR policies and RSVP-TE tunnels to a CSV file. You can then use this CSV file and the following command to load the missing policies into the Configuration Database: 

    python3 /opt/optima/pce_dispatcher/config_db/csv_util.py –dump-missing /<PathToFile>/dump_file.cs

Backup Cisco Crosswork with Cisco NSO

Restore from the NSO backup file is a manual process, currently.

Before you begin

Before you begin, be sure:

  • You have the hostname or IP address and the port number of a secure SCP server.

  • You have a file path on the SCP server, to use as the destination for your backup files.

  • You have the user credentials for an account with read and write permissions to the storage folder on the destination SCP server.

Also ensure that the NSO provider, the Cisco Crosswork credential profile that is associated with the NSO provider, and the NSO server meet the following prerequisites:

  • The NSO provider configuration includes an SSH connection. If you don't enable SSH on the provider, Cisco Crosswork displays a warning alarm. Cisco Crosswork creates a backup for its own data, but not for NSO.

  • The NSO provider's credential profile contains the user ID and password of a user with sudo privileges on the NSO server.

  • The NSO server has NCT (NSO Cluster Tools) installed, and the user in the credential profile for the NSO provider can execute nct commands.

  • The NSO server has Python version 3.x installed, and the user in the credential profile for the NSO provider can execute python3 commands.

  • The user in the NSO provider's credential profile has full access to the NSO server's backup folder and the files in it. This requirement usually means full read and write access to the NSO server's /var/opt/ncs/backups/ folder.

Failure to meet any of these requirements means that all or part of the backup job will fail.

Procedure

Step 1

Configure an SCP backup server:

  1. From the main menu, choose Administration > Backup and Restore.

  2. Click Destination to display the Edit Destination dialog box. Make the relevant entries in the fields provided.

  3. Click Save to confirm the backup server details.

Step 2

Create Cisco Crosswork and Cisco NSO backups:

  1. Click Backup to display the Backup dialog box with the destination server details prefilled.

  2. Provide a relevant name for the backup in the Job Name field.

  3. If you want Cisco Crosswork to take the backup despite application or microservice issues, check the Force check box.

  4. Be sure to leave the Backup NSO check box checked.

  5. (Optional) Click Verify Backup to verify that Cisco Crosswork has enough free resources to complete the backup. If the check is successful, Cisco Crosswork displays a warning about the time-consuming nature of the operation. Click OK.

  6. Click Start Backup to start the backup operation. Cisco Crosswork creates the corresponding backup job set and adds it to the job list.

  7. To view the progress of a backup job: Enter the job details (such as Status or Job Type) in the search fields in the Backup Restore Job Sets table. Then click the job set you want.

    The Job Details table displays information about the selected job set, such as the job Status, Job Type, and Start Time. If there’s a failed job, hover the mouse pointer over the Details icon icon near the Status column to view the error details.

Restore with Cisco NSO

When you restore a Cisco Crosswork cluster and its associated Cisco NSO cluster from a backup, follow these guidelines:

  • We recommend that you perform restore operations during a scheduled maintenance window only. Users shouldn’t attempt to access Cisco Crosswork or Cisco NSO while these operations are running. Cisco Crosswork restore operations are lengthy, and will pause other Cisco Crosswork applications until they are complete. Cisco NSO must be stopped completely during restores.

  • You can run both a Cisco Crosswork and a Cisco NSO restore operation at the same time.

Before you begin

Get from the SCP server the full name of the backup file you want to restore. This file will contain both the Cisco Crosswork and Cisco NSO backups. Backup filenames have the following format:

backup_JobName_CWVersion_TimeStamp.tar.gz

Where:

  • JobName is the user-entered name of the backup job.

  • CWVersion is the Cisco Crosswork platform version of the backed-up system.

  • TimeStamp is the date and time when Cisco Crosswork created the backup file.

For example: backup_Wed_4-0_2021-02-31-12-00.tar.gz.

Procedure

Step 1

Log in (if needed) to the remote SCP backup server. Using the Linux command line, access the backup destination directory and find the backup file containing Cisco NSO information that you want to restore. For example: 

[root@localhost~]# ls -ltr
-rw-rw-r--. 1 root root 8265938605 backup_Wed_4-0_2021-02-31-12-00.tar.gz
Step 2

Use tar -xzvf to extract the Cisco NSO backup from the Cisco Crosswork backup file in the destination folder. For example: 

[root@localhost~]# tar -xzvf backup_Wed_4-0_2021-02-31-12-00.tar.gz
...
[root@localhost~]# ls -ltr
-rw-rw-r--. 1 root root 8265938605 backup_Wed_4-0_2021-02-31-12-00.tar.gz
-rw-r--r--. 1 root root 8267798605 468c4715-ea09-4c2b-905e-98999d.tar.gz
Step 3

Un-tar the Cisco NSO backup file in the destination folder. You will see Cisco NSO files being extracted to a folder structure under /nso/ProviderName/, where /nso/ProviderName/ is the name of the Cisco NSO provider as configured in Cisco Crosswork. In the following example, the Cisco NSO provider is named nso121: 

tar -xvsf 468c4715-ea09-4c2b-905e-98999d.tar.gz
468c4715-ea09-4c2b-905e-98999d/nso/
468c4715-ea09-4c2b-905e-98999d/nso/nso121/
468c4715-ea09-4c2b-905e-98999d/nso/nso121/log/
468c4715-ea09-4c2b-905e-98999d/nso/nso121/log/nso_backup_result_nso121_Wed.log
468c4715-ea09-4c2b-905e-98999d/nso/nso121/NSO_RESTORE_PATH_nso121
468c4715-ea09-4c2b-905e-98999d/nso/nso121/ncs-5.4.2@backup_Wed_nso121.backup.gz
...
Step 4

Locate the file with a backup.gz extension in the /nso/ProviderName/folder. This is the generated Cisco NSO backup file. In the example in the previous step, the file name is highlighted.

Step 5

Log in to Cisco NSO as a user with root privileges and access the command line. Then copy or move the generated Cisco NSO backup file from the SCP server to the specified restore path location of the Cisco NSO cluster. For example: 

[root@localhost nsol21]# ls
log ncs-5.4.2@backup_Wed_nso121.backup.gz NS0_REST0RE_PATH_nso121
[root@localhost nso121]# more NS0_REST0RE_PATH_nso121
/var/opt/ncs/backups/
[root@localhost nso121]# 
...
Step 6

You can perform Cisco NSO restore operations only while NSO is not running. At the Cisco NSO cluster command line, run the following command to stop Cisco NSO:

$/etc/init.d/ncs stop
Step 7

Once NCS has stopped, start the restore operation using the following command and the name of the generated Cisco NSO backup file. For example:

#ncs-backup --restore ncs-5.4.2@backup_Wed_nso121.backup.gz

If you have trouble running this command, first give yourself sudo su permission.

Step 8

Once the restore completes, restart Cisco NSO using the following command. This command may take a few minutes to complete.

$/etc/init.d/ncs start
Step 9

Once you have restored both Cisco Crosswork and Cisco NSO clusters from backups, re-add the Cisco NSO provider to Cisco Crosswork.