Cisco Crosswork Network Controller 7.2 Administration Guide

Cluster system recovery

Updated: March 2, 2026

Want to summarize with AI?

Review the recovery options and requirements for restoring a Crosswork cluster after node, service, application, or disaster-related failures.

A cluster system recovery is a disaster recovery strategy that

restores critical cluster services and data after failures or disruptions,
addresses platform-specific considerations to ensure compatibility and resilience, and
minimizes overall downtime to maintain business continuity.

A robust cluster system recovery approach helps ensure that Cisco Crosswork clusters can be restored quickly and reliably after failures, disruptions, or disasters. Understanding your recovery options and platform-specific requirements is essential to maintaining service continuity and minimizing downtime.

System recovery options and requirements

Successful cluster recovery depends on understanding platform requirements, backup practices, and the nature of the failure. This reference summarizes prerequisites, platform limitations, and actions for common recovery scenarios.

Before you begin

For cluster recovery, it is essential to have a recent backup.
The cluster you are restoring should have the same operational architecture, including the same number of hybrid and worker nodes.

Recovery conditions and system behavior

At some time during normal operations of your Cisco Crosswork cluster, you may need to recover the entire system. This can result from malfunctioning nodes, services, applications, or a disaster destroying hosts for the cluster.
A functional cluster requires a minimum of three hybrid nodes. These nodes share processing and traffic loads for management, orchestration, and infrastructure services.
The hybrid nodes are highly available and can redistribute processing among themselves and to worker nodes automatically.
The cluster can tolerate one hybrid node reboot (graceful or ungraceful); the system remains functional but with degraded availability.
The system can tolerate any number of failed worker nodes (with degraded availability until restored).
If two or more hybrid nodes are lost ("double fault"), recovery cannot be guaranteed – in such cases, redeploy a new cluster and restore from a recent backup.

Alarms and troubleshooting

Cisco Crosswork generates alarms when nodes, applications, or services malfunction.
Examine alarms and check health of the affected component(s). Use Crosswork features to drill down and, for service faults, attempt to restart the problem service.
If alarms show a single hybrid node, or a hybrid plus worker node(s) failure, start by rebooting or replacing (erasing, then readding) failed nodes; if unsuccessful, attempt a clean system reboot.
If the system remains unstable or degraded (loss of two or more hybrid nodes), deploy a new cluster and recover using a backup.

Platform limitations

Unintentional VM shutdown is not supported on a 3 VM cluster running Crosswork Network Controller. If a VM fails, the remaining two VMs cannot support migrating all pods from the failed VM. Add worker nodes to enable VM shutdown.
A reboot of one VM is supported in a 3 VM cluster. Restore may take 5 minutes (if the orch pod is not on the rebooted VM) up to 25 minutes (if it is).

Perform a clean system reboot (VMware)

Perform a coordinated reboot of all cluster VMs to restore operations or after failure.

A clean system reboot is sometimes required to restore cluster health following multiple node or service issues, or after system maintenance. This process ensures all VMs are properly powered down and brought back online in a specific order, supporting the stability and recovery of both hybrid and worker nodes in VMware deployments.

Follow these steps to perform a clean system reboot:

Procedure

	1.	Place Crosswork Network Controller in Maintenance mode. See Enable or disable maintenance mode for details. (Optional) Shut down Crosswork Data Gateways and other non-essential components, such as NSO and SR-PCE, that communicate with Crosswork.
	2.	Power down all VMs: Log in to the VMware vSphere Web Client. In the Navigator pane, right-click the VM you want to shut down. Choose Power > Power Off. Wait for the VM’s status to change to Off. Repeat for each VM in the cluster.
	3.	Power up the VM hosting the first hybrid node: In the Navigator pane, right-click the VM to power up. Choose Power > Power On. Wait for the VM’s status to change to On, then wait 30 seconds before continuing.
	4.	Repeat the previous step for each remaining hybrid node, staggering reboots by 30 seconds. Continue with each worker node using the same staggered interval.
	5.	After all VMs are powered on, wait a few minutes and login to Crosswork Network Controller.
	6.	Move Crosswork Network Controller out of maintenance mode. See Enable or disable maintenance mode for details. If your cluster is not healthy, maintenance mode attempts may fail. Alarms may indicate failed services and reasons. If issues persist, follow the "redeploy and restore" method. For more details, see Redeploy and restore a Crosswork cluster from backup (VMware).
	7.	Restart Crosswork Data Gateways and any other components in your ecosystem that communicate with Crosswork Network Controller.

The Crosswork Network Controller cluster completes a clean system reboot. If cluster health does not return, proceed with the redeploy and restore procedure.

Redeploy and restore a Crosswork cluster from backup (VMware)

Rebuild and restore a failed Crosswork cluster using a previously taken backup.

Redeployment and restoration from backup is required when a cluster is severely degraded (such as after double faults or catastrophic failures), and cannot be recovered through standard node replacement or reboot procedures. The procedure involves powering down and deleting existing VMs, deploying a new cluster, and then restoring system state from a backup to recover services and data.

Before you begin

Ensure you have a recent and valid backup file.
This method assumes you have taken periodic backups before recovery is required. (For details on backup, see Back up data.

Follow these steps to redeploy and restore the cluster:

Procedure

	1.	Power down all VMs: Log in to the VMware vSphere Web Client. In the Navigator pane, right-click the VM you want to shut down. Choose Power > Power Off. Wait for the VM’s status to change to Off. Repeat for each VM in the cluster.
	2.	Delete all VMs: In the VMware vSphere Web Client Navigator pane, right-click the VM you want to delete. Choose Delete from Disk. Wait for the VM’s status to show Deleted. Repeat for each VM in the cluster.
	3.	Deploy a new Cisco Crosswork cluster as explained in the Cisco Crosswork Network Controller 7.2 Installation Guide.
	4.	Recover the system state to the newly deployed cluster. For more information, see Restore data after a disaster.

A new Crosswork Network Controller cluster is deployed, and system state is restored using the most recent backup.

Need help?

Open a support case

(Requires a Cisco Service Contract)

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.