Explore Cisco
How to Buy

Have an account?

  •   Personalized content
  •   Your products and support

Need an account?

Create an account

Cisco Managed Service Accelerator 3.8 Installation and Upgrade Guide

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Book Contents
Find Matches in This Book
Available Languages
Download Options

Book Title

Cisco Managed Service Accelerator 3.8 Installation and Upgrade Guide

Chapter Title

Disaster Recovery

Results

Updated:
June 18, 2020

Chapter: Disaster Recovery

Disaster Recovery

This chapter contains the following topics:

Back up and Restore to the Same Cluster

You can back up the current state of a MSX cluster node, and later use the backed-up files to restore the node to the same cluster state. Use the procedure below to restore data to the same cluster from an existing backup.

Procedure

Step 1

Log in to the installer container.
Step 2

Go to ansible directory under installer container.


cd /msx-3.8.0/ansible
Step 3

Move to the msx 3.8.0 folder available from the list of folders inside the container: 


ansible-playbook vms-backup.yml --extra-var backup_tag=<SPECIFIC_TAG>
Note 

<SPECIFIC_TAG> gives the backup a tag identifier. Specify a unique tag to distinguish a series of backups created. It can be of any value.

All backup files created are stored in the folder /msx-3.8.0/ansible/vms-backup/SPECIFIC_TAG

Step 4

Perform a backup of the CockroachDB. For details, see Backing Up the CockroachDB.

Step 5

Copy the vms-backup directory to a place outside the installer container.

In addition to backing up the cluster data by running the above playbook, it is also recommended to backup all files present in the installer container. If the installer container is lost, you can recreate the container and get access to the deployment from the backed up container files. Copy the entire data under /msx-3.8.0/ansible and /tmp/ansible.log to a location outside the installer container.

Step 6

Restore the system from an existing backup using the command:


ansible-playbook vms-restore.yml --extra-var backup_tag=<SPECIFIC_TAG>
Note 

Select the backup file that you want to restore. If you want to restore the backup with a <SPECIFIC_TAG>, pass with that value in extra vars as mentioned above.

Step 7

Perform a restore operation for the CockroachDB. For details, see Restoring the CockroachDB.

Restoring MSX to a New Cluster

Use this procedure in cases where an existing deployment is unusable and unrecoverable. The overall process will be to create a new installation and then to restore the most recent data from a backup.

Procedure

Step 1

Log in to the MSX installer container.
Step 2

Copy passwords.yml from the vms-backup directory in the installer container. 


cp /msx-3.8.0/ansible/vms-backup/passwords.yml /msx-3.8.0/ansible/group_vars/all/passwords.yml
Step 3

Edit main.yml to set the auto_generate_password variable to ‘false’. This is the path to the file:

/msx-3.8.0/ansible/group_vars/all/main.yml.
Step 4

Copy the certs from either the Inception node, the Kube master or, if necessary, from your backup. The cert files are ca.pem and ca-key.pem, and they are located at /etc/ssl/vms-certs on either node. The cert files should be copied onto your installer container into the mounted directory on the host (/tmp/repo).

Step 5

Create your infra node. This process will create the target server infrastructure:

  1. Move to the ansible folder.

    cd /msx-3.8.0/ansible

  2. Export the Ansible password file so that you can run Ansible playbooks.

    export ANSIBLE_VAULT_PASSWORD_FILE=<path to file>

  3. Build the target-server infrastructure:

    ansible-playbook create-infra.yml (OpenStack)
ansible-playbook create-infra-aws.yml (AWS)

  4. Verify that the infrastructure was created properly.

    ansible-playbook checks/check-createinfra.yml
Step 6

Copy your ca files (ca.pem and ca-key.pem) to /etc/ssl/vms-certs on both the Inception and the Kube-master nodes.

Step 7

Proceed with the remainder of the MSX installation. That process will install the necessary services and verify them. For details on the install process, refer to Steps 2 and later in the Installing the MSX Platform in Standalone Mode.

Services being installed:

  • Isolated (isolated.yml)

  • Infra Services (deploy-infra-services.yml)

  • Platform Microservices (deploy-vms-microservices.yml)

Lastly, verify the install:

  • check-vms (check-vms.yml)
Step 8

Install the Service Packs that were previously installed in the deployment. For details on the process, refer to Installing Service Packs.

Step 9

Perform the data restore, while making sure that you use the same tag that was specified previously during the backup operation.

ansible-playbook vms-restore.yml --extra-var backup_tag=msx-backup-tag
Step 10

Perform a restore of the CockroachDB. For details, see Restoring the CockroachDB.

Dual Data Center Disaster Recovery

This section contains the following topics:

Failing Over to the Passive Data Center

Use the following procedure to manually fail over to the passive data center in case the active data center is not reachable or has become unusable in case of a disaster or for other reasons.

Procedure

Step 1

Bring down the VPN Tunnel. Log in to the CSR VPN using SSH on the passive data center node and do the following: 


config t
int tun 0
shut
Step 2

Power off the CSR VPN and CSR HUB VMs on the Active site. Powering off both these VMs will automatically break the tunnel.

Step 3

Source the OpenStack RC file that was copied to the passive data center node: 


source vms-backup/infra/openrc-passive
Step 4

Change directory to the /msx-version/ansible folder inside the container:


cd /msx-3.8.0/ansible
Step 5

Switch to the passive data center: 


ansible-playbook dualdc-switch-dc.yml --extra-vars '{dc: passive}'
Step 6

Stop consul replication on the passive data center: 


ansible -m command -a "kubectl delete -f /etc/kube-manifests/consul-replicate-rc.yml" kube-master[0]
Step 7

Switch NSO to become the primary instance on the passive data center: 


ansible -m command -a "curl -X PUT -d 'Active' -k -H 'X-CONSUL-TOKEN:<consul_root_token>' https://consul.service.consul:8500/v1/kv/private/dc_status" kube-master[0]

The default consul_root_token: 73b72724-4570-4000-b720-de18c30fabe2

Step 8

Stop platform microservices on the passive data center: 


ansible -m command -a "kubectl delete -f /etc/kube-manifests/administrationservice-rc.yml -f /etc/kube-manifests/consumeservice-rc.yml -f /etc/kube-manifests/devicemanagerservice-rc.yml -f /etc/kube-manifests/manageservice-rc.yml -f /etc/kube-manifests/monitorservice-rc.yml -f /etc/kube-manifests/serviceextensionservice-rc.yml -f /etc/kube-manifests/usermanagementservice-rc.yml -f /etc/kube-manifests/orchestrationservice-rc.yml -f /etc/kube-manifests/billingservice-rc.yml -f /etc/kube-manifests/notificationservice-rc.yml" kube-master[0]
Step 9

Start user management on the passive data center and make sure it is up (for orchestration to read ncs streams); and verify if it is up and running: 


ansible -m command -a "kubectl create -f /etc/kube-manifests/usermanagementservice-rc.yml"  kube-master[0]
Step 10

Ensure that the user management service status is up: 


ansible -m command -a "curl -X GET http://routerservice.service.consul:8765/idm/admin/health" kube-master[0]
Step 11

Start the platform microservices on the passive data center: 


ansible -m command -a "kubectl create -f /etc/kube-manifests/administrationservice-rc.yml -f /etc/kube-manifests/consumeservice-rc.yml -f /etc/kube-manifests/devicemanagerservice-rc.yml -f /etc/kube-manifests/manageservice-rc.yml -f /etc/kube-manifests/monitorservice-rc.yml -f /etc/kube-manifests/serviceextensionservice-rc.yml -f /etc/kube-manifests/orchestrationservice-rc.yml -f /etc/kube-manifests/billingservice-rc.yml -f /etc/kube-manifests/notificationservice-rc.yml"  kube-master[0]
Step 12

Restore the A0 backup to passive side: 


ansible-playbook backup-restore-ao.yml --extra-vars "{ BR_mode: restore, backup_tag: (tag from vms-backup dir) }"
Note 

The backup tag `tag from vms-backup dir` will be determined by the admin and is located in the `/msx-3.8.0/ansible/vms-backup/` directory.

Step 13

Stop the data platform probes on the passive data center: 


ansible -m command -a "kubectl delete -f /etc/kube-manifests/heartbeat-cm.yml -f /etc/kube-manifests/sshbeat-cm.yml -f /etc/kube-manifests/snmpbeat-cm.yml -f /etc/kube-manifests/heartbeat-ps.yml -f /etc/kube-manifests/sshbeat-ps.yml -f /etc/kube-manifests/snmpbeat-ps.yml" kube-master[0]
Step 14

Start the data platform probes on the passive data center: 


ansible -m command -a "kubectl create -f /etc/kube-manifests/heartbeat-cm.yml -f /etc/kube-manifests/sshbeat-cm.yml -f /etc/kube-manifests/snmpbeat-cm.yml -f /etc/kube-manifests/heartbeat-ps.yml -f /etc/kube-manifests/sshbeat-ps.yml -f /etc/kube-manifests/snmpbeat-ps.yml" kube-master[0]
Step 15

As a superuser, get the IDM token on the passive data center: 


ansible -m command -a "curl -X POST http://routerservice.service.consul:8765/idm/api/v1/accesstoken -H 'accept: application/json' -H 'cache-control: no-cache' -H 'content-type: application/json' -d '{\"granttype\": \"password\", \"password\": \"<superuser_password>\", \"scope\": \"write\", \"username\": \"superuser\"}'" kube-master[0]
Step 16

Push the probe configuration on the passive data center: 


ansible -m command -a "curl -X POST -H 'authorization: Bearer <token from previous step>' http://devicemanagerservice.service.consul:9104/devicemanagerservice/v1/admin/devicemetrics/start" kube-master[0]
Step 17

For the Cisco MSX SD-Branch service pack, run the following playbook:


ansible -m command -a "kubectl delete -f /etc/kube-manifests/nso-vbranch-ps.yml" kube-master[0]
Step 18

For the Cisco MSX Managed Device service pack, run the following playbook:


ansible -m command -a "kubectl delete -f /etc/kube-manifests/nso-manageddevice-ps.yml" kube-master[0]
Step 19

For the SD-Branch service pack, run the following playbook 


ansible -m command -a "sed -i 's/replicas: 1/replicas: 2/g" kube-master
Step 20

For the Cisco MSX Managed Device service pack, run the following playbook:


ansible -m command -a "sed -i 's/replicas: 1/replicas: 2/g" kube-master
Step 21

In the following playbook specify the service packs that have been installed:


ansible-playbook deploy-nso-consul-cleanup.yml --extra-vars '{servicepack_list: ['vbranch', 'manageddevice', ‘cloudutd’]}'
Step 22

Deploy SD-Branch ncs if the SD-Branch service pack is installed: 


ansible -m command -a "kubectl create -f /etc/kube-manifests/nso-vbranch-ps.yml" kube-master[0]
ansible -m command -a "sed -i 's/replicas: 1/replicas: 2/g" kube-master
Step 23

Deploy Managed Device ncs if the Managed Device service pack is installed: 


ansible -m command -a "kubectl create -f /etc/kube-manifests/nso-manageddevice-ps.yml" kube-master[0]
Step 24

Stop the SD-Branch service pack microservice if SD-Branch is installed: 


ansible -m command -a "kubectl delete -f /etc/kube-manifests/statemachineservice-rc.yml -f /etc/kube-manifests/vbranchservice-rc.yml" kube-master[0]
Step 25

Stop the Managed Device service pack microservice if Managed Device is installed: 


ansible -m command -a "kubectl delete -f /etc/kube-manifests/manageddeviceservice-rc.yml" kube-master[0]
Step 26

Stop the SD-WAN service pack microservice if SD-WAN is installed: 


ansible -m command -a "kubectl delete -f /etc/kube-manifests/sdwanservice-rc.yml 
kube-master[0]
Step 27

Start the SD-Branch service pack microservice if SD-Branch is installed: 


ansible -m command -a "kubectl create -f /etc/kube-manifests/statemachineservice-rc.yml -f /etc/kube-manifests/vbranchservice-rc.yml" kube-master[0]
Step 28

Start the Managed Device service pack microservice if Managed Device is installed: 


ansible -m command -a "kubectl create -f /etc/kube-manifests/manageddeviceservice-rc.yml" kube-master[0]
Step 29

Start the SD-WAN service pack microservice if SD-WAN is installed. 


ansible -m command -a "kubectl create -f /etc/kube-manifests/sdwanservice-rc.yml" kube-master[0]
Step 30

Run the following playbook to update the DNS entries:


ansible-playbook create-infra.yml --tags route53,server

  • For SD-Branch:

Update the PNP hosts of all ENCS devices to point to the Edge Node IP address for the passive data center. Log in to the ENCS NFVIS system, and provide the PNP server IP address or FQDN for the passive data center in the Host Plug-n-Play settings.

Note 

This step is required only when NFVIS version used is lower than 3.8.1 and the IP address is provided for PNP.

If you are using NFVIS 3.8.1, which has FQDN support, the FQDN name can be provided directly (For example: orange-customer.com) and this does not need to be updated after the failover.

  • For SD-WAN:

After the failover, execute the following steps before proceeding with service operations:

  1. Update the PNP hosts of all ENCS devices to point to the Edge Node IP address for the passive data center.

Note 

This step is required only when the NFVIS version used is lower than 3.8.1 and the IP address is provided for PNP.

If you are using NFVIS 3.8.1, which has FQDN support, the FQDN name can be provided directly (For example: orange-customer.com) and this does not need to be updated after the failover.

  1. Verify that the new passive data center NAT IP addresses have been opened out for vOrchestrator connectivity before creating control planes.

  2. For Managed Device:

After the failover, perform the following step on NSO through ncs_cli:

  1. Verify if the devices are in synch:

    
request devices check-sync

  2. If the devices are out-of-sync, run this command:

    
request devices sync-from
Note 

After the last step is performed, ensure that the ncs for either SD-Branch, or Managed Device, or both, is ready. If ncs is ready, enter this command: 


ansible -m shell -a "kubectl -n vms get pod | grep nso" kube-master[0]

If ncs is ready, then the return output is as follows.


nso-manageddevice-0 3/3 Running
nso-vbranch-0 3/3 Running
Step 31

Source the OpenStack RC file that was copied to the passive data center node: 


vms-backup/infra/openrc-active
Step 32

Switch to active DC using the following command:


ansible-playbook dualdc-switch-dc.yml --extra-vars '{dc: active}' ----
Step 33

Stop SD-Branch SP microservice if SD-Branch is installed:


ansible -m command -a "kubectl delete -f /etc/kube-manifests/
statemachineservice-rc.yml -f /etc/kube-manifests/vbranchservice-rc.yml" 
kube-master[0]
Step 34

Stop Managed Device SP microservice if Managed Device is installed:

ansible -m command -a "kubectl delete -f /etc/
kube-manifests/manageddeviceservice-rc.yml" kube-master[0]
Step 35

Scale SD-Branch NSO count to 0 for inactive passive DC if SD-Branch service pack is installed.

ansible -m command -a "kubectl scale --replicas=0 -f 
/etc/kube-manifests/nso-vbranch-ps.yml" kube-master[0]
Step 36

Scale manageddevice NSO count to 0 for inactive passive DC if manageddevice ServicePack is installed.

ansible -m command -a "kubectl scale --replicas=0 -f 
/etc/kube-manifests/nso-manageddevice-ps.yml" kube-master[0]
Step 37

Log in to the MSX Portal and create a tenant. For more information, see ‘Logging in to the Portal’ and ‘Managing Tenants’ in the Cisco Managed Services Accelerator (MSX) Platform User Guide.

What to do next

Enabling the VPN

Enabling the VPN

After the failover, use the following procedure to enable the VPN and allow the passive data center to send any device configuration and database changes to active data center.
Procedure
Step 1

Source the OpenStack RC file that was copied to the active data center node:


source vms-backup/infra/openrc-active
Step 2

Switch to the active data center:


ansible-playbook dualdc-switch-dc.yml --extra-vars '{dc: active}'
Step 3

Switch NSO to become the secondary instance on the active data center:


ansible -m command -a "curl -X PUT -d 'Passive' -k -H 'X-CONSUL-TOKEN:<consul_root_token>' https://consul.service.consul:8500/v1/kv/private/dc_status" kube-master[0]
Step 4

Start consul replication on the passive data center:


ansible -m command -a "kubectl create -f /etc/kube-manifests/consul-replicate-rc.yml" kube-master[0]
Step 5

Source the OpenStack RC file that was copied to the passive data center node:


source vms-backup/infra/openrc-passive
Step 6

Switch to the passive data center:


ansible-playbook dualdc-switch-dc.yml --extra-vars '{dc: passive}'
Step 7

Bring up the VPN Tunnel. Log in to the CSR VPN using SSH on the active data center node and do the following:


Enter into config mode (config t)
int tun 0
start
Note 

You can automate the failover steps in AWS and OpenStack using automation scripts. A sample script, /failover-aws.sh has been provided under /msx-3.8.0/ ansible/.

Failing Back to the Active Data Center

Use the following procedure to manually fail back to the active data center after a fail over.

Procedure

Step 1

Bring down the VPN Tunnel. Log in to the CSR VPN using SSH on the active data center node and do the following:


config t
int tun 0
shut
Step 2

Power off CSR VPN and CSR HUB VMs on the passive site. Powering off both these VMs will automatically break the tunnel.
Step 3

Source the OpenStack RC file that was copied to the active data center node:


source vms-backup/infra/openrc-active
Step 4

Switch to active data center:


ansible-playbook dualdc-switch-dc.yml --extra-vars '{dc: active}'
Step 5

Stop consul replication on active data center:


ansible -m command -a "kubectl delete -f /etc/kube-manifests/consul-replicate-rc.yml" kube-master[0]
Step 6

Switch NSO to become the primary instance on the active data center:


ansible -m command -a "curl -X PUT -d 'Active' -k -H 'X-CONSUL-TOKEN:<consul_root_token>' https://consul.service.consul:8500/v1/kv
/private/dc_status" kube-master[0]

The default consul_root_token: 73b72724-4570-4000-b720-de18c30fabe2
Step 7

Stop platform microservices on active data center:


ansible -m command -a "kubectl delete -f /etc/kube-manifests/administrationservice-rc.yml -f /etc/kube-manifests/consumeservice-rc.yml -f /etc/kube-manifests/devicemanagerservice-rc.yml -f /etc/kube-manifests/manageservice-rc.yml -f /etc/kube-manifests/monitorservice-rc.yml -f /etc/kube-manifests/serviceextensionservice-rc.yml-f
/etc/kube-manifests/usermanagementservice-rc.yml -f /etc/kube-manifests/orchestrationservice-rc.yml -f /etc/kube-manifests/billingservice-rc.yml -f /etc/kube-manifests/notificationservice-rc.yml" kube-master[0]
Step 8

Start user management on active data center, ensure that it’s up (for orchestration to read ncs streams); and then verify if it’s up and running: 


ansible -m command -a "kubectl create -f /etc/kube-manifests/usermanagementservice-rc.yml" kube-master[0]
Step 9

Ensure that the user management service status is up:


ansible -m command -a "curl -X GET http://routerservice.service.consul:8765/idm/admin/health" kube-master[0]
Step 10

Start platform microservices on active data center:


ansible -m command -a "kubectl create -f /etc/kube-manifests/administrationservice-rc.yml -f /etc/kube-manifests/consumeservice-rc.yml -f /etc/kube-manifests/devicemanagerservice-rc.yml -f /etc/kube-manifests/manageservice-rc.yml -f /etc/kube-manifests/monitorservice-rc.yml -f /etc/kube-manifests/serviceextensionservice-rc.yml -f /etc/kube-manifests/orchestrationservice-rc.yml -f /etc/kube-manifests/billingservice-rc.yml -f /etc/kube-manifests/notificationservice-rc.yml" kube-master[0]
Step 11

Stop data platform probes on the active data center:


ansible -m command -a "kubectl delete -f /etc/kube-manifests/heartbeat-cm.yml -f /etc/kube-manifests/sshbeat-cm.yml -f /etc/kubemanifests/
snmpbeat-cm.yml -f /etc/kube-manifests/heartbeat-ps.yml -f /etc/kube-manifests/sshbeat-ps.yml -f /etc/kube-manifests
/snmpbeat-ps.yml" kube-master[0]
Step 12

Start data platform probes on the active data center:


ansible -m command -a "kubectl create -f /etc/kube-manifests/heartbeat-cm.yml -f /etc/kube-manifests/sshbeat-cm.yml -f /etc/kubemanifests/snmpbeat-cm.yml -f /etc/kube-manifests/heartbeat-ps.yml -f /etc/kube-manifests/sshbeat-ps.yml -f /etc/kube-manifests
/snmpbeat-ps.yml" kube-master[0]
Step 13

As a superuser, get IDM token on the a ctive data center:


ansible -m command -a "curl -X POST http://routerservice.service.consul:8765/idm/api/v1/accesstoken -H 'accept: application/json' -H'cache-control: no-cache' -H 'content-type: application/json' -d '{\"granttype\": \"password\", \"password\": \"<superuser_password>\", \"
scope\": \"write\", \"username\": \"superuser\"}'" kube-master[0]
Step 14

Push the probe configuration on the active data center:


ansible -m command -a "curl -X POST -H 'authorization: Bearer <token from previous step>' http://devicemanagerservice.service.consul:9104/devicemanagerservice/v1/admin/devicemetrics/start
" kube-master[0]
Step 15

For the SD-Branch service pack, run the following playbook:


ansible -m command -a "kubectl delete -f /etc/kube-manifests/nso-vbranch-ps.yml" kube-master[0]
Step 16

For the Managed Device service pack, run the following playbook:


ansible -m command -a "kubectl delete -f /etc/kube-manifests/nso-manageddevice-ps.yml" kube-master[0]
Step 17

Specify only one service pack name if single servicepack has been installed:


ansible-playbook 
deploy-nso-consul-cleanup.yml --extra-vars '{servicepack_list: ['vbranch', 'manageddevice']}'
Step 18

Deploy SD-Branch ncs if the SD-Branch service pack is installed:


ansible -m command -a "kubectl create -f /etc/kube-manifests/nso-vbranch-ps.yml" 
kube-master[0]
Step 19

Deploy Managed Device ncs if the Managed Device service pack is installed:


ansible -m command -a "kubectl create -f /etc/kube-manifests/nso-manageddevice-ps.yml" kube-master[0]
Step 20

For the SD-Branch service pack, run the following playbook:


ansible -m command -a "sed -i 's/replicas: 2/replicas: 1/g" kube-master
Step 21

For the Managed Device service pack, run the following playbook:


ansible -m command -a "sed -i 's/replicas: 2/replicas: 1/g" kube-master
Step 22

Stop the SD-Branch service pack microservice if SD-Branch is installed:


ansible -m command -a "kubectl delete -f /etc/kube-manifests/statemachineservice-rc.yml -f /etc/kube-manifests/vbranchservice-rc.yml"kube-master[0]
Step 23

Stop the Managed Device service pack microservice if the Managed Device is installed:


ansible -m command -a "kubectl delete -f /etc/kube-manifests/manageddeviceservice-rc.yml" kube-master[0]
Step 24

Stop the SD-WAN service pack microservice if SD-WAN is installed:


ansible -m command -a "kubectl delete -f /etc/kube-manifests/sdwanservice-rc.yml 
kube-master[0]
Step 25

Start the SD-Branch service pack microservice if SD-Branch is installed:


ansible -m command -a "kubectl create -f /etc/kube-manifests/statemachineservice-rc.yml -f /etc/kube-manifests/vbranchservice-rc.yml"
kube-master[0]
Step 26

Start the Managed Device service pack microservice if Managed Device is installed:


ansible -m command -a "kubectl create -f /etc/kube-manifests/manageddeviceservice-rc.yml" kube-master[0]
Step 27

Start the SD-WAN service pack microservice if SD-WAN is installed.


ansible -m command -a "kubectl create -f /etc/kube-manifests/sdwanservice-rc.yml" 
kube-master[0]
Step 28

Run the following playbook to update the DNS entries:


ansible-playbook create-infra.yml --tags route53,server

  • For SD-Branch:

Update the PNP hosts of all ENCS devices to point to the Edge Node IP address for the active data center. Log in to the ENCS NFVIS system, and provide the PNP server IP address or the FQDN for the active data center in the Host Plug-n-Play settings.

Note 

This step is required only when NFVIS version used is lower than 3.8.1 and IP address is provided for PNP.

If you are using NFVIS 3.8.1, which has FQDN support, the FQDN name can be provided directly (For example: orange-customer.com) and this does not need to be updated after the failover.

  • For SD-WAN:

    After the failover, execute the following steps before proceeding with service operations:

    a. Update the PNP hosts of all ENCS devices to point to the Edge Node IP address for the passive data center.

    Note 
    This step is required only when NFVIS version used is lower than 3.8.1 and IP address is provided for PNP.

    If you are using NFVIS 3.8.1, which has FQDN support, the FQDN name can be provided directly (For example: orange-customer.com) and this does not need to be updated after the failover.

    Verify the active data center NAT IP addresses have been opened out for vOrchestrator connectivity before creating control planes.

  • For Managed Device

    After the failover, perform the following step on NSO through ncs_cli:

    Verify if the devices are in synch:

    
request devices check-sync

    If the devices are out-of-sync, use this command:

    
request devices sync-from
    Note 

    After the last step is performed, ensure SD-Branch or managed device ncs, or both are ready. If ncs is ready, enter this command: 

    
ansible -m shell -a "kubectl -n vms get pod | grep nso" kube-master[0]

If ncs is ready, the return output is as follows.

Example:


nso-manageddevice-0 3/3 Running
nso-vbranch-0 3/3 Running
Step 29

Log in to the MSX Portal and create a tenant. For more information, see ‘Managing Tenants’ in the Cisco Managed Services Accelerator (MSX) Platform User Guide .

What to do next

Enabling the VPN

Enabling the VPN

After failback, use the following procedure to enable the VPN and allow active data center to send any device configuration and database changes to the passive data center.

Procedure
  Command or Action Purpose
Step 1

Source the OpenStack RC file that was copied to the passive data center node:

 

source vms-backup/infra/openrc-passive
Step 2

Switch to the passive data center:

 
ansible-playbook dualdc-switch-dc.yml --extra-vars '{dc: passive}'
Step 3

Switch NSO to become the secondary instance on the passive data center:

 
ansible -m command -a "curl -X PUT -d 'Passive' -k -H 'X-CONSUL-TOKEN:<consul_root_token>' https://consul.service.consul:8500/v1/kv/private/dc_status" kube-master[0]
Step 4

Start consul replication on the passive data center:

 
ansible -m command -a "kubectl create -f /etc/kube-manifests/consul-replicate-rc.yml" kube-master[0]
Step 5

Source the OpenStack RC file that was copied to the active data center node:

 
source vms-backup/infra/openrc-active
Step 6

Switch to active data center.

 
ansible-playbook dualdc-switch-dc.yml --extra-vars '{dc: active}'
Step 7

Bring up the VPN Tunnel. Log in to the CSR VPN using SSH on the active data center node and do the following

 
Enter into config mode (config t)
int tun 0
start
Note 
You can automate the failover steps in AWS and OpenStack using automation scripts. A sample script, /failover-aws.sh has been provided under /msx-3.8.0/ ansible/.

Was this Document Helpful?

FeedbackFeedback

Contact Cisco