Cisco Managed Service Accelerator 3.10 Installation and Upgrade Guide

Shutting Down and Restarting an MSX Deployment

During scheduled network outages such as system upgrades or hardware maintenance, you may have to close or disable all the services or instances. A power outage or randomly shutting down an instance may result in data loss. In such cases, you can use the procedure detailed in this section to perform a shutdown of the MSX server and restart the instances without any data corruption.

Procedure

Step 1

Change directory to the /msx-version/ansible folder inside the container:


cd /msx-3.10.0/ansible

Step 2

Power down the MSX server:


ansible-playbook shutdown-vms.yml

Step 3

Restart the MSX server:


ansible-playbook bringup-vms.yml

Note

After running the bringup-vms.yml playbook wait for 10 minutes, until all tunnels come up automatically in CSRHUB.

Step 4

Verify the status of the MSX 3.9 Platform.


ansible-playbook checks/check-vms.yml

General Installation Errors

For any installation issues, log into the installer container and check the ansible logs at /tmp/ansible.log. It provides logs for all the playbooks that you have run.

The table below lists the general errors encountered during the MSX deployment.

Table 1. Installation Errors
Problem/Error	Possible Reason	User Action
Initial building of the target server infrastructure using the create-infra playbook fails		Ensure that: You have sourced your OpenStack credentials and you can connect to OpenStack using the standard CLI tools. You have sufficient quota to support your desired deployment. You have sufficient privileges to create servers, networks, routers, and security groups.
Create-infra playbook returns the following error: fatal: [localhost]: FAILED! => {"changed": false, "extra_data": null, "failed": true, "msg": "Error in creating instance (Inner Exception: create() takes at least 4 arguments (10 given))"}		Verify that the value for the os_image variable in main.yml matches the name of the CentOS image shown in the output of "openstack image list.
Backend hosts remain unreachable		Ensure that: You can use SSH to connect into the jumphost manually. You can use SSH to connect into the backend hosts from the jumphost. The inventory file is populated properly. The ssh.cfg file is correctly populated. You can use SSH to connect into the jumphost and backend hosts using the -F ssh.cfg option. If all else fails, copy your keys to your default ssh config directory.
Unable to create Kibana index	Kibana auto changes the configuration to read-only mode, if log volumes are full.	Clear old logs to free up some space and update the config mentioned as follows: `PUT _settings { "index": { "blocks": { "read_only_allow_delete": "false" } } }`

Accessing Log Files via the Terminal

The microservices container logs are a good resource, while troubleshooting issues.

To access the log files via the terminal:

Procedure

Step 1

Log in to the inception VM. Use this command:


ssh -i keys/id_rsa centos@<master nodes IP address> -t ssh <master node IP address>

Step 2

List the microservice containers available in the master node. Run this command as a root user to view this list:


kubectl get pods --namespace=vms (this has to be run with sudo permission on kube-master)

Example:

[centos@kubernetes-master-vmsservices-1 ~]$ kubectl get pods --namespace=vms | grep cvpn

Step 3

To view the logs, do one of the following:


kubectl logs [container-name] --namespace=vms

or

Use SSH to connect to the container and view logs:


kubectl exec -it [container-name] /bin/bash --namespace=vms

Accessing Log Files via Kibana

Providing administrators the ability to output and access logs is critical to maintaining and diagnosing the health and performance of a well-running MSX deployment. To aid in that effort, Cisco MSX logs all container activity via standard output (Stdout), redirects that data into var/log/containers/, and then makes that information available to an Elasticsearch cluster that can be accessed by the Kibana user interface for analysis.

Note

To view the Kibana logs, you must be using a role that has the "Event Log (View)" permission assigned to it.

To view the Kibana logs:

Procedure

Step 1	Log in to the Cisco MSX Portal.
Step 2	From the left pane, click System Logs at the bottom left of the MSX home page. The Kibana user interface is displayed.
Step 3	Click Discover at the upper left of the page.
Step 4	Click Open at the upper right of the page. A list of the predefined searches is displayed. The existing searches (subject to change) include: usermanagementservice administrationservice orchestrationservice notificationservice monitorservice billingservice consumeservice manageservice routerservice manageddeviceservice vbranchservice sdwanservice nso-manageddevice nso-vbranch cassandra zookeeper nginx
Step 5	Click any of the predefined logs to see the log entries for the prior 15 minutes. Clicking usermanagementservice, for example, will show you the user management log entries for the preceding 15 minutes.

Filtering by Field Values

If you need to restrict visible log entries to a specific error or condition, then you can filter within Kibana by the traceid value in searches where it is available. This same approach can also be used on other fields with similar results. For example, in the nginx search you can filter by a return code.

To filter by traceid:

Procedure

Step 1	Use a predefined Kibana search query to view your required records. In this example, `usermanagement` query is used.
Step 2	Locate the record type with the error or condition that you would like to filter for.
Step 3	Hover your cursor over the record's traceid field and click "+" to "filter for value". Alternately, you can filter out the value from the resulting list by clicking "-".
Step 4	The log listing will be restricted to records that have the same traceid value as the one you have selected.

Fixing Action Orchestrator Database Sync Issues

If you encounter a scenario where Action Orchestrator’s database Arangodb and the Cassandra database go out of sync (during a shutdown/restart cycle), use the following playbook to recover Action Orchestrator:

cd /msx-3.10.0/ansible
export ANSIBLE_VAULT_PASSWORD_FILE=<path_to_ansible_password_file>
ansible-playbook recover-ao.yml

Replacing Worker or Inception Nodes

When either a Kubernetes worker node or the Inception node fails or needs replacement, you can use this procedure to restore the node. Node replacement might be necessary, for example, if you want to change the instance type.

To replace a worker node:

Procedure

Step 1

Invoke this playbook to replace the node. You can find the node name in the inventory file located at /msx-version/ansible/inventory/inventory.


ansible-playbook replace-node.yml --extra-vars "replace_node=<Name-of-Node-to-Replace>”

For Example:

ansible-playbook replace-node.yml --extra-vars "replace_node=kubernetes-node-name-1"

Step 2

The playbook verifies that the node exists and is powered on. If the node does not exist, then the playbook does not display a confirmation, but will continue to create a new instance (see the output below).


Thursday 14 May 2020  16:40:33 +0000 (0:00:00.080)       0:00:15.810 ********** 
    [node-actions/replace : Prompt to confirm replace node "kubernetes-node-name-1"]
    kubernetes-node-name-1: Currently exists
    kubernetes-node-name-1: Powered ON

Step 3

Type the node FQDN to confirm the replacement operation: kubernetes-node-name-1.msx-domain.com.

Step 4

If the node is still running, the following process will replace the node in approximately 30 minutes:

The playbook will drain the node. Pods that cannot be rescheduled onto another node will remain offline until the node is brought back into the cluster.
The node instance will be removed from Kubernetes.
The node instance will be deleted from your cloud platform.
A new node instance will be created.
The instance will be reconfigured.
The node will be brought back into the cluster.

Note

When you replace a node, the pods will not be automatically rebalanced onto the new replacement node by Kubernetes. You may, instead, manually rebalance the work onto that node. The process for rebalancing depends on many factors within your environment. Please consult with your operations team on the best approach for your circumstances.

Replacing Kube-Master Nodes

Replacing a master node is a service-impacting operation, so you should only perform this procedure during a maintenance window when no critical processes are running. The replace-master.yml playbook attempts to minimize service impacts, but minor disruptions may occur because IP addresses and certificates will change. This process causes minor service and network updates within a running system and will take approximately one to two hours to complete, depending on your deployment.

To replace a kube-master node:

Procedure

Step 1

Verify that the kube-master node is malfunctioning or not running.

Step 2

Run replace-master.yml.

ansible-playbook replace-master.yml --extra-vars "replace_node=<Name-of-Node-to-Replace>

Step 3

Ensure that the playbook has completed successfully.

MSX Node Clock Synchronization

When you deploy CockRoachDB and Cassandra within MSX, ensure that clocks are synchronized across all Kubernetes nodes. If clocks are not synchronized, those components will not start. This section describes the playbooks that are available to help in validating and synchronizing node clocks.

Verifying and Establishing Chrony Sync

Chrony is used on each MSX node to synchronize with a valid NTP clock source. To verify whether clocks (resident on all MSX nodes) are synchronized, run the check-chrony.yml playbook. If a node’s clock is not synchronized, then chrony is restarted in an attempt to establish sync. If chrony cannot be synchronized, then the playbook will fail and list the nodes that are not in sync. If the playbook completes without an error, then chrony is in sync on all MSX nodes.

To verify whether clocks are synchronized, run the following command:

ansible-playbook checks/check-chrony.yml

If the configured NTP source malfunctions, you should use the update-chrony-source.yml playbook to update the NTP source for all MSX nodes to another clock source. Before running this playbook, update the NTP entry in group_vars/all/main.yml to set the new source. When run, this playbook will update the NTP clock source for chrony on all nodes, restart chrony, and then check if the clocks are synchronized.

To update the NTP source for all MSX nodes, run the following command:

ansible-playbook checks/update-chrony-source.yml

Book Title

Chapter Title

Results

Chapter: Troubleshooting an MSX Installation

Troubleshooting an MSX Installation

Shutting Down and Restarting an MSX Deployment

Procedure

General Installation Errors

Related Troubleshooting Topics

Accessing Log Files via the Terminal

Procedure

Accessing Log Files via Kibana

Procedure

Filtering by Field Values

Procedure

Fixing Action Orchestrator Database Sync Issues

Replacing Worker or Inception Nodes

Procedure

Replacing Kube-Master Nodes

Procedure

MSX Node Clock Synchronization

Was this Document Helpful?

Contact Cisco

Related Cisco Community Discussions