Verify that the nodes and services in your existing deployment are healthy. This process takes approximately 5 minutes. If the platform health check indicates that any components are unhealthy, then you should recover those components prior to proceeding with the upgrade.

ansible-playbook checks/check-vms.yml 

Run the following script from the 4.1.0 container to perform a data backup of the MSX instance that is in service. The data backup process allows you to take a backup of all persistent MSX data, such as Cassandra, Elasticsearch, and NSO data. This process is useful in cases where you might need to recover lost or corrupted MSX data from a backup. This process should take approximately 30 minutes.

ansible-playbook vms-backup.yml --extra-vars backup_tag=backup-tag-name 

Copy the files that are essential for the upgrade process. These files should not have changed from your first MSX install and they will be used for all future MSX versions. The commands for copying the files are:

cp /msx-4.1.0/ansible/ssh.cfg /msx-4.1.0/ansible/vms-backup/infra/
cp /msx-4.1.0/ansible/inventory/inventory /msx-4.1.0/ansible/vms-backup/infra/
cp /msx-4.1.0/ansible/group_vars/all/passwords.yml /msx-4.1.0/ansible/vms-backup/infra/
cp /msx-4.1.0/ansible/keys/id_rsa.pub /msx-4.1.0/ansible/vms-backup/infra/
cp /msx-4.1.0/ansible/keys/id_rsa /msx-4.1.0/ansible/vms-backup/infra/
cp /msx-4.1.0/ansible/group_vars/all/external_addresses.yml /msx-4.1.0/ansible/vms-backup/infra/
cp /msx-4.1.0/ansible/vms-backup/infra/<openrc> /msx-4.1.0/ansible/

Download the following tar files:

Copy the msx-binaries-4.2.0-xxx.tar.gz and msx-container-4.2.0-xxx.tar.gz to the host where you plan on installing MSX.

Use SSH to connect to the host where you will install MSX.

Run this command on the host to load the MSX container image:

[root@CentOS localadmin] # docker load -i MSX-container-4.2.0-xxx.tar.gz

Extract msx-binaries-4.2.0-xxx.tar.gz to /vms-binaries-mount-source.

Get the <docker_image_ID> using the command:

[root@CentOS localadmin] # docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
<none>              <none> 	    e94ad25e65a9         5 days ago          383 MB

Spin up the 4.2.0 container with the vms-backup-mount-source folder attached as a volume:

docker run --name <4.2.0-Container_Name> -v </vms-binaries-mount-source>:/msx-4.2.0/ansible/repo -v <vms-backup-mount-source>:/msx-4.2.0/ansible/vms-backup <docker image id> -it /bin/bash

Copy the required files as shown:

cp /msx-4.2.0/ansible/vms-backup/infra/inventory /msx-4.2.0/ansible/inventory/inventory
cp /msx-4.2.0/ansible/vms-backup/infra/ssh.cfg /msx-4.2.0/ansible/ssh.cfg
cp /msx-4.2.0/ansible/vms-backup/infra/passwords.yml /msx-4.2.0/ansible/group_vars/all/passwords.yml
cp /msx-4.2.0/ansible/vms-backup/infra/external_addresses.yml /msx-4.2.0/ansible/group_vars/all/external_addresses.yml
cp /msx-4.2.0/ansible/vms-backup/infra/<openrc> /msx-4.2.0/ansible/<openrc>
 

Create a folder to copy the keys into and perform the copy operation:

mkdir /msx-4.2.0/ansible/keys
cp /msx-4.2.0/ansible/vms-backup/infra/id_rsa /msx-4.2.0/ansible/keys
cp /msx-4.2.0/ansible/vms-backup/infra/id_rsa.pub /msx-4.2.0/ansible/keys

(OpenStack Only) Source the V3 OpenStack configuration file to enable access to your OpenStack project, which will use the values within the openrc file:

source <path-to-openrc> 

Navigate to the msx-4.2.0/ansible folder. All playbooks must be run from this folder.

cd /msx-4.2.0/ansible

Upgrade ssh.cfg with the new version.

ansible-playbook utils/upgrade-sshcfg.yml

Make the required changes in group_vars/all/main.yml, using the backed up main.yml as a reference.

If needed, you can scale MerakiBeats at this time to support more than 22,000 devices. For more details, see Scaling MerakiBeats.

Create a vault_password file with the same password used in MSX 4.2.0 and export the path as specified:

export ANSIBLE_VAULT_PASSWORD_FILE=<path to vault_password file>

Generate any new passwords that were added in this release by running the following playbook.

ansible-playbook upgrade-passwords.yml --extra-vars @vms-backup/infra/passwords.yml

Execute the MSX prerequisites playbook to prepare the new MSX 4.2.0 container for an upgrade. The playbook will perform password, security group-related, and other updates as required. This process should take approximately 5 minutes.

ansible-playbook upgrade-prereq.yml --extra-vars @vms-backup/infra/passwords.yml --skip-tags vnfoutside,csr_route_upgrade 

Upload the MSX isolated binaries to your S3 (AWS) / minio (OpenStack) bucket. This process should take approximately 15 minutes.

ansible-playbook upload-isolated-binaries.yml 

(Optional) If you are installing the Datadog monitoring service to provide metrics for your infrastructure, make sure to follow the procedure in Prerequisites for All Datadog Deployment Scenarios. Use this command:

ansible-playbook upload-datadog-images.yml

Edit each Service Pack deployment variables file so that it matches your deployment requirements. When you perform an upgrade, the previous changes to the file will be lost. Use the backup deployment variables file for each Service Pack to reinstate those changes. The Service Pack variable files are located at: /msx-4.2.0/ansible/group_vars/all/{servicepack_name}_variables.yml, where {servicepack_name} can be manageddevice, vbranch, sda, or sdwan.

Upgrade the Infra Services to update the infrastructure components for the Platform Microservices and Service Packs. This process should take approximately 20–40 minutes.

ansible-playbook upgrade-infra-services.yml 

Run the Blue/Green Platform Microservice update, which will upgrade the Platform and Service Packs. Both 4.1.0 and 4.2.0 Microservices will be running in parallel, however all traffic will be routed to 4.1.0 at this stage. Additionally, all Microservices will be running in non-HA mode.

ansible-playbook blue-green-ms-upgrade.yml 

Perform an upgrade of the Network Service Orchestrator (NSO) component. At this point, any updates to existing services or the provisioning of new services will be queued (and delayed) until the NSO upgrade has completed.

ansible-playbook bg-nso-upgrade.yml 

Run the switch-ms-routing.yml playbook. This will redirect traffic from MSX 4.1.0 to 4.2.0, enable HA mode, and will delete all older microservice instances once complete.

ansible-playbook switch-ms-routing.yml 

Upgrade the MSX user interface for the Platform and Service Packs.

ansible-playbook upgrade-ui.yml 

Upgrade Action Orchestrator and the Workflow engine. The Workflow engine is placed in maintenance mode, so no new Workflow is created or updated during this time. This process should take approximately 40 minutes.

ansible-playbook upgrade-ao.yml

Upgrade the Kubernetes cluster.

ansible-playbook os-k8s-rolling-upgrade.yml

Invoke this playbook to replace the master nodes and upgrade the underlying Operating System kernel, security components, and any other required system updates. You can find the node names in the inventory file located at /msx-version/ansible/inventory/inventory.

ansible-playbook --skip-tags force_interactive replace-master.yml --extra-vars "replace_node=Name-of-Master-Node1-to-Replace"

ansible-playbook --skip-tags force_interactive replace-master.yml --extra-vars "replace_node=Name-of- Master-Node2-to-Replace"

ansible-playbook --skip-tags force_interactive replace-master.yml --extra-vars "replace_node=Name-of- Master-Node3-to-Replace"

Verify that all services are properly running.

ansible-playbook checks/check-vms.yml

Invoke this playbook to replace the first Edge node. You can find the node name in the inventory file.

ansible-playbook --skip-tags force_interactive replace-node.yml --extra-vars
"replace_node=Name-of-Edge-Node1-to-Replace"

Add new nodes.

ansible-playbook --tags add-nodes upgrade-kube-nodes.yml

./upgrade-nodes.sh

1. Delete the old nodes.

   ansible-playbook --tags delete-nodes --skip-tags add-nodes upgrade-kube-nodes.yml

   Or run the following script from the Ansible directory:

   ./delete-nodes.sh

   Note	You might find that all the nodes are not deleted for various reasons. If the delete-nodes playbook does not successfully complete, perform the following steps to continue the process until completion:

2. Edit in_progress_old_nodes.yml and the inventory file (located at /msx-version/ansible/inventory/inventory) to remove the successfully deleted nodes in each file.

3. Continue running the delete-notes playbook in step a and modifying the contents of the node and inventory files until the playbook completes without errors.

Run update-node-config to clean up the old node configurations and delete temporary files from the system.

ansible-playbook --tags update-node-config --skip-tags add-nodes,delete-nodes upgrade-kube-nodes.yml

./update-node-config.sh

Invoke this playbook to replace the second Edge node. You can find the node name in the inventory file.

ansible-playbook --skip-tags force_interactive replace-node.yml --extra-vars

"replace_node=Name-of-Edge-Node2-to-Replace"

(Optional) Replace TLS Gateway nodes only if the TLS Gateway is installed.

ansible-playbook upgrade-tlsgateway-nodes.yml --skip-tags force_interactive

Replace the Inception node.

ansible-playbook --skip-tags force_interactive replace-node.yml --extra-vars

"replace_node=Name-of-Inception-Node-to-Replace"

Run the cleanup-aws-dataplane.yml playbook to remove unused AWS infrastructure.

ansible-playbook cleanup-aws-dataplane.yml

Verify that all services are properly running.

ansible-playbook checks/check-vms.yml

If you have cron backup enabled in the system then you need to run a backup now.

ansible-playbook cron-backup.yml

Upload the MSX isolated binaries to your S3 (AWS)/MinIO (OpenStack) bucket. This process should take approximately 15 minutes.

ansible-playbook upload-isolated-binaries.yml

Upgrade the MSX prerequisites in the new MSX 4.2.0 container. This playbook prepares the new container so that it is ready for the upgrade process. As necessary, it will perform password, certificate, and security group-related updates. This process should take approximately 5 minutes.

ansible-playbook upgrade-prereq.yml --skip-tags vnfoutside,csr_route_upgrade

Upgrade the Infra Services to update the infrastructure components for the Platform Microservices and Service Packs. This process should take approximately 20 minutes.

ansible-playbook upgrade-infra-services.yml

Upgrade Platform Microservices

ansible-playbook upgrade-vms-microservices.yml 

Upgrade Action Orchestrator and the Workflow engine.

ansible-playbook upgrade-ao.yml

Edit each Service Pack deployment variables file so that it matches your deployment requirements. When you perform an upgrade, the previous changes to the file will be lost. Use the backup deployment variables file for each Service Pack to reinstate those changes. The Service Pack variable files are located at:

/msx-4.2.0/ansible/group_vars/all/{servicepack_name}_variables.yml

Upgrade Service Packs.

ansible-playbook deploy-service.yml --extra-vars '{"service_name":{{spname}}, mode: Upgrade, milestoneupgrade: true }'

Upgrade the MSX user interface components.

ansible-playbook upgrade-ui.yml

If upgrading from M-Final to 4.2.0-GA, use the following sequence of commands:

1. ansible-playbook deploy-service.yml --extra-vars '{"service_name":{{spname}}, milestoneupgrade: true }'

   Where servicepack_name can be manageddevice, vbranch, sda, or sdwan.

2. Upgrade the MSX user interface components.

   ansible-playbook upgrade-ui.yml

3. Upgrade the Kubernetes cluster.

   ansible-playbook os-k8s-rolling-upgrade.yml

4. Invoke this playbook to replace the master nodes and upgrade the underlying Operating System kernel, security components, and any other required system updates. You can find the node names in the inventory file located at /msx-version/ansible/inventory/inventory.

   To replace the master nodes:

   ansible-playbook --skip-tags force_interactive replace-master.yml --extra-vars "replace_node=Name-of-Master-Node1-to-Replace"
   
   ansible-playbook --skip-tags force_interactive replace-master.yml --extra-vars "replace_node=Name-of- Master-Node2-to-Replace"
   
   ansible-playbook --skip-tags force_interactive replace-master.yml --extra-vars "replace_node=Name-of- Master-Node3-to-Replace"

5. Verify that all services are properly running.

   ansible-playbook checks/check-vms.yml

6. Invoke this playbook to replace the first Edge node. You can find the node name in the inventory file.

   To replace edge node1:

   ansible-playbook --skip-tags force_interactive replace-node.yml --extra-vars
   "replace_node=Name-of-Edge-Node1-to-Replace"

7. Add new nodes.

   ansible-playbook --tags add-nodes upgrade-kube-nodes.yml

   Or run the following script from the Ansible directory:

   ./upgrade-nodes.sh

8. Delete the old nodes.

   ansible-playbook --tags delete-nodes --skip-tags add-nodes upgrade-kube-nodes.yml

   Or run the following script from the Ansible directory:

   ./delete-nodes.sh

   Note	You might find that all the nodes are not deleted for various reasons. If the delete-nodes playbook does not successfully complete, perform the following steps to continue the process until completion:

9. Edit in_progress_old_nodes.yml and the inventory file (located at /msx-version/ansible/inventory/inventory) to remove the successfully deleted nodes.

10. Continue running the delete-notes playbook in step 9 g and modifying the contents of the node and inventory files until the playbook completes without errors.

11. Run update-node-config to clean up the old node configurations and delete temporary files from the system.

    ansible-playbook --tags update-node-config --skip-tags add-nodes,delete-nodes upgrade-kube-nodes.yml

    Or run the following script from the Ansible directory:

    ./update-node-config.sh

12. Invoke this playbook to replace the second Edge node. You can find the node name in the inventory file.

    To replace edge node2:

    ansible-playbook --skip-tags force_interactive replace-node.yml --extra-vars "replace_node=Name-of-Edge-Node2-to-Replace"

13. (Optional) Replace TLS Gateway nodes only if the TLS Gateway is installed.

    ansible-playbook upgrade-tlsgateway-nodes.yml --skip-tags force_interactive

14. Replace the Inception node.

    ansible-playbook --skip-tags force_interactive replace-node.yml --extra-vars
    
    "replace_node=Name-of-Inception-Node-to-Replace"

Run the cleanup-aws-dataplane.yml playbook to remove unused AWS infrastructure.

ansible-playbook cleanup-aws-dataplane.yml

Verify that all services are properly running.

ansible-playbook checks/check-vms.yml

If you have cron backup enabled in the system then you need to run a backup now.

ansible-playbook cron-backup.yml

Import Service Provider metadata into MSX.

1. Save the metadata file to the following location in the Kubernetes node:

   /data/vms/heapdumps/usermanagementservice/vmanage_metadata.xml

   MSX Usermanagement services maps the above Kubernetes location to the following:

   /data/conf/vmanage_metadata.xml

   Specify the location (/data/conf/vmanage_metadata.xml) as the metadata file location while importing the file into MSX.

2. Log in to the Cisco MSX Portal.

3. In the main menu, go to Settings > SSO Configuration.

4. In the Add SSO Client’s section, select the SAML service provider client ID (vManage in this case) and click the Edit icon.

   In the Attributes section, specify the vManage metadata file location (from step a) in the Metadata Source field. The metadata source field takes a URL or a file path.

   Note	Metadata fields appear only if you have selected SAML service provider client ID.

[Applicable only for Cisco SD-WAN] Turn off the SAML security settings.

1. Log in to Inception VM and access the kubernetes master node.

   ssh -i id_rsa centos@_INCEPTION_FLOATING_IP_ADDRESS_ -t ssh _kubernetes-master-1_IP_ADDRESS

2. Use the following cURL commands:

   curl --request PUT -g -k -v -H "X-Consul-Token: {consul_token_value}" --data 'false' https://localhost:8500/v1/kv/userviceconfiguration/defaultapplication/security.auth.saml.want-authn-request-signed

   curl --request PUT -g -k -v -H "X-Consul-Token: {consul_token_value}" --data 'false' https://localhost:8500/v1/kv/userviceconfiguration/defaultapplication/security.auth.saml.encrypt-assertion

   Note	Replace {consul_token_value} with your actual consul token value from the passwords.yml file.

3. Save the above consul entries. To save, restart the UserManagement microservices from the Kubernetes master node using the following commands:

   kubectl -n vms delete -f /etc/kube-manifests/usermanagementservice-rc.yml
   kubectl -n vms create -f /etc/kube-manifests/usermanagementservice-rc.yml