Geo Redundancy Switchover

This chapter contains the following topics:

Perform Switchover

Switchover is the process of interchanging the roles of the active cluster and standby cluster in the event of a failure.

In case of a failure, the system performs many preliminary checks (heartbeat count, connectivity checks, HTTP and SSH login checks, etc.) and raises alarms if they fail. If you notice an alarm, you are expected to check both clusters to verify the authenticity of the alarms before they attempt a switchover.


Note

  • If a switchover operation is completed on a standby VM (before the sync operation), there are no rows or entries displayed on the Publish Details for tech-support jobs. This happens because the tech-support history is written to ETCD which is not synced across geo redundancy setups. This is an expected system behavior.

  • During switchover, all services provisioned until Periodic/On-demand sync automatically gets updated in AZ2. A granular resync will be triggered which fetches the timestamp of the last successful backup from Crosswork and requests NSO to republish updates since that time. The granular resync (during AZ switchover) is specific to services provisioned from NSO/Crosswork Active Topology only.

Before you begin

Before the switchover, it is important that both clusters have the same application versions and resource footprints used.

Procedure

Step 1

Log in to the standby cluster.

Step 2

From the main menu, choose Administration > Cross Cluster. The Cross Cluster window is displayed.

Step 3

Click Actions > Switch Cluster Role

The Switch Cluster Role dialog box is displayed with the initial state of the clusters. For the purpose of this topic, SJC cluster (cluster-sjc) is in ACTIVE mode and NYC cluster (cluster-nyc) is in STANDBY mode.

Figure 1. Switch Cluster Role

Step 4

Click on the NYC cluster to change it to ACTIVE state. Click Save to confirm change.

Figure 2. Switch standby cluster to active

Step 5

Update the DNS server records of Management FQDN and Data FQDN to point to the new active cluster.

Step 6

Now log in to the SJC cluster (already active). In the Cross Cluster window, click Actions > Switch Cluster Role.

Note

 

At this point, till the time you change the cluster state, both clusters will be in ACTIVE state.

Step 7

In the Switch Cluster Role dialog box, click on the cluster to change it to STANDBY state.

Figure 3. Switch active cluster to standby

Click Save to confirm the change.

Note

 

Wait for the device reachability to converge before moving to resume operations on the standby cluster.

Step 8

After few minutes, log in to the first cluster. The switchover will be completed.

Geo Redundant Cluster Disaster Recovery

At some time during normal operations of your Cisco Crosswork cluster, you may find that you need to recover the entire system. This can be the result of one or more malfunctioning nodes, one or more malfunctioning services or applications, or a disaster that destroys the hosts for the entire cluster.

This section explains the steps needed to perform to recover your geo redundant clusters.

Destruction of Standby Cluster

This topic explains the recovery steps that must be executed when the standby cluster is destroyed.

Procedure

Step 1

Deploy the new standby cluster with the same IP and Virtual IP (VIP) addresses.

Step 2

Install the same version of applications as the active cluster.

Step 3

Enable pairing mode in the active cluster. For more information, see Enable Geo Redundancy

Step 4

Enable geo redundancy on the standby cluster. For more information, see Enable Geo Redundancy

Step 5

Enter the Cross Cluster settings on the new standby cluster as per the settings on the active cluster. Alternatively, you can enable the Apply the same configuration to peer cluster checkbox for the Cross Cluster settings on the active cluster, and save the settings.

Step 6

On the Cross Cluster window, use the Initiate Sync option to ensure the standby cluster pulls data from the active cluster.

Destruction of Active Cluster

This topic explains the recovery steps that must be executed when the active cluster is destroyed.

Procedure

Step 1

Switch the standby cluster to become the new active cluster. For more information, see Geo Redundancy Switchover.

Step 2

Re-deploy the destroyed cluster (currently, the standby cluster) with the same IP and Virtual IP (VIP) addresses.

Step 3

Install the same version of applications as the active cluster.

Step 4

Enable pairing mode in the active cluster. For more information, see Enable Geo Redundancy

Step 5

Edit the Cross Cluster inventory file (.yaml) to reflect the current state of the Cross Cluster after the switchover in step 1.

Step 6

On the Geo Redundancy window of the active cluster, click Browse and upload the edited inventory file (.yaml file).

Step 7

Enable geo redundancy on the standby cluster. For more information, see Enable Geo Redundancy

Step 8

Enter the Cross Cluster settings on the new standby cluster as per the settings on the active cluster. Alternatively, you can enable the Apply the same configuration to peer cluster checkbox for the Cross Cluster settings on the active cluster, and save the settings.

Step 9

On the Cross Cluster window, synchronize data to ensure the standby cluster pulls data from the active cluster.

Step 10

(Optional) Switch the standby cluster to become the new active cluster.