Troubleshooting Installations

This chapter contains the following topics:

Shutting Down and Restarting

During scheduled network outages such as system upgrades or hardware maintenance, you may have to close or disable all the services or instances. A power outage or randomly shutting down an instance may result in data loss. In such cases, you can use the procedure detailed in this section to perform a shutdown of the MSX server and restart the instances without any data corruption.

Procedure

Step 1

Change directory to the /msx-version/ansible folder inside the container:


cd /msx-4.1.0/ansible
Step 2

Power down the MSX server:


ansible-playbook shutdown-vms.yml
Step 3

Restart the MSX server:


ansible-playbook bringup-vms.yml
Note 

After running the bringup-vms.yml playbook wait for 10 minutes, until all tunnels come up automatically in CSRHUB.

Step 4

Verify the status of the existing MSX Platform. 


ansible-playbook checks/check-vms.yml

Typical Installation Errors

For any installation issues, log into the installer container and check the ansible logs at /tmp/ansible.log. This file provides logs for all the playbooks that you have run.

The table below lists some typical errors that might be encountered during the MSX installation.

Table 1. Installation Errors

Problem/Error

User Action
Initial building of the target server infrastructure using the create-infra playbook fails

Ensure that:

  • You have sourced your OpenStack credentials and you can connect to OpenStack using the standard CLI tools.

  • You have sufficient quota to support your desired deployment.

  • You have sufficient privileges to create servers, networks, routers, and security groups.

Create-infra playbook returns the following error:

fatal: [localhost]: FAILED! => {"changed": false, "extra_data": null, "failed": true, "msg": "Error in creating instance (Inner Exception: create() takes at least 4 arguments (10 given))"}

Verify that the value for the os_image variable in main.yml matches the name of the CentOS image shown in the output of "openstack image list.

Backend hosts remain unreachable

Ensure that:

  • You can use SSH to connect into the jumphost manually.

  • You can use SSH to connect into the backend hosts from the jumphost.

  • The inventory file is populated properly.

  • The ssh.cfg file is correctly populated.

  • You can use SSH to connect into the jumphost and backend hosts using the -F ssh.cfg option.

  • If all else fails, copy your keys to your default ssh config directory.

Unable to create Kibana index

Kibana automatically changes the configuration to read-only mode if log volumes are full.

Clear old logs to free up some space and update the config mentioned as follows:


PUT _settings
    {
    "index": {
    "blocks": {
    "read_only_allow_delete": "false"
    }
    }
    }

Accessing Log Files

The microservices container logs are a good resource, while troubleshooting issues.

Accessing Log Files via the Terminal

To access the log files via the terminal:

Procedure

Step 1

Log in to the inception VM. Use this command:


ssh -i keys/id_rsa centos@<master nodes IP address> -t ssh <master node IP address>
Step 2

List the microservice containers available in the master node. Run this command as a root user to view this list: 


kubectl get pods --namespace=vms (this has to be run with sudo permission on kube-master)

Example: 

[centos@kubernetes-master-vmsservices-1 ~]$ kubectl get pods --namespace=vms | grep cvpn
Step 3

To view the logs, do one of the following:


kubectl logs [container-name] --namespace=vms

or

Use SSH to connect to the container and view logs:


kubectl exec -it [container-name] /bin/bash --namespace=vms

Accessing Log Files via Kibana

Providing administrators the ability to output and access logs is critical to maintaining and diagnosing the health and performance of a well-running MSX deployment. To aid in that effort, Cisco MSX logs all container activity via standard output (Stdout), redirects that data into var/log/containers/, and then makes that information available to an Elasticsearch cluster that can be accessed by the Kibana user interface for analysis.


Note

To view the Kibana logs, you must be using a role that has the "Event Log (View)" permission assigned to it.

To view the Kibana logs:

Procedure

Step 1

Log in to the Cisco MSX Portal.
Step 2

From the left pane, click System Logs at the bottom left of the MSX home page.

The Kibana user interface is displayed.
Step 3

Click Discover at the upper left of the page.

Step 4

Click Open at the upper right of the page.

A list of the predefined searches is displayed. The existing searches (subject to change) include:

  • usermanagementservice

  • administrationservice

  • orchestrationservice

  • notificationservice

  • monitorservice

  • billingservice

  • consumeservice

  • manageservice

  • routerservice

  • manageddeviceservice

  • vbranchservice

  • sdwanservice

  • nso-manageddevice

  • nso-vbranch

  • cassandra

  • zookeeper

  • nginx

Step 5

Click any of the predefined logs to see the log entries for the prior 15 minutes. Clicking usermanagementservice, for example, will show you the user management log entries for the preceding 15 minutes.

Filtering by Field Values

If you need to restrict visible log entries to a specific error or condition, then you can filter within Kibana by the traceid value in searches where it is available. This same approach can also be used on other fields with similar results. For example, in the nginx search you can filter by a return code.

To filter by traceid:

Procedure
Step 1

Use a predefined Kibana search query to view your required records. In this example, usermanagement query is used.

Step 2

Locate the record type with the error or condition that you would like to filter for.

Step 3

Hover your cursor over the record's traceid field and click "+" to "filter for value". Alternately, you can filter out the value from the resulting list by clicking "-".

Step 4

The log listing will be restricted to records that have the same traceid value as the one you have selected.

Fixing Action Orchestrator Database Sync Issues

If you encounter a scenario where Action Orchestrator’s database Arangodb and the Cassandra database go out of sync (during a shutdown/restart cycle), use the following playbook to recover Action Orchestrator: 

cd /msx-4.1.0/ansible
export ANSIBLE_VAULT_PASSWORD_FILE=<path_to_ansible_password_file>
ansible-playbook recover-ao.yml

Replacing Worker or Inception Nodes

When either a Kubernetes worker node or the Inception node fails or needs replacement, you can use this procedure to restore the node. Node replacement might be necessary, for example, if you want to change the instance type.

To replace a worker node:

Procedure

Step 1

Invoke this playbook to replace the node. You can find the node name in the inventory file located at /msx-version/ansible/inventory/inventory.

ansible-playbook replace-node.yml --extra-vars "replace_node=<Name-of-Node-to-Replace>”

For Example:

ansible-playbook replace-node.yml --extra-vars "replace_node=kubernetes-node-name-1"
Step 2

The playbook verifies that the node exists and is powered on. If the node does not exist, then the playbook does not display a confirmation, but will continue to create a new instance (see the output below). 

Thursday 14 May 2020  16:40:33 +0000 (0:00:00.080)       0:00:15.810 ********** 
    [node-actions/replace : Prompt to confirm replace node "kubernetes-node-name-1"]
    kubernetes-node-name-1: Currently exists
    kubernetes-node-name-1: Powered ON
Step 3

Type the node FQDN to confirm the replacement operation: kubernetes-node-name-1.msx-domain.com.

Step 4

If the node is still running, the following process will replace the node in approximately 30 minutes:

  1. The playbook will drain the node. Pods that cannot be rescheduled onto another node will remain offline until the node is brought back into the cluster.

  2. The node instance will be removed from Kubernetes.

  3. The node instance will be deleted from your cloud platform.

  4. A new node instance will be created.

  5. The instance will be reconfigured.

  6. The node will be brought back into the cluster.

Note 

When you replace a node, the pods will not be automatically rebalanced onto the new replacement node by Kubernetes. You may, instead, manually rebalance the work onto that node. The process for rebalancing depends on many factors within your environment. Please consult with your operations team on the best approach for your circumstances.

Replacing Kube-Master Nodes

Replacing a master node is a service-impacting operation, so you should only perform this procedure during a maintenance window when no critical processes are running. The replace-master.yml playbook attempts to minimize service impacts, but minor disruptions may occur because IP addresses and certificates will change. This process causes minor service and network updates within a running system and will take approximately one to two hours to complete, depending on your deployment.

To replace a kube-master node:

Procedure

Step 1

Verify that the kube-master node is malfunctioning or not running.
Step 2

Run replace-master.yml. 

ansible-playbook replace-master.yml --extra-vars "replace_node=<Name-of-Node-to-Replace>
Step 3

Restart the beat services.

library/ansible-playrole dataplatform/beats kube-master
Step 4

Ensure that the playbook has completed successfully.

Node Clock Synchronization

When you deploy CockroachDB and Cassandra within MSX, ensure that clocks are synchronized across all Kubernetes nodes. If clocks are not synchronized, those components will not start. This section describes the playbooks that are available to help in validating and synchronizing node clocks.

Verifying and Establishing Chrony Sync

Chrony is used on each MSX node to synchronize with a valid NTP clock source. To verify whether clocks (resident on all MSX nodes) are synchronized, run the check-chrony.yml playbook. If a node’s clock is not synchronized, then chrony is restarted in an attempt to establish sync. If chrony cannot be synchronized, then the playbook will fail and list the nodes that are not in sync. If the playbook completes without an error, then chrony is in sync on all MSX nodes.

To verify whether clocks are synchronized, run the following command:

ansible-playbook checks/check-chrony.yml

If the configured NTP source malfunctions, you should use the update-chrony-source.yml playbook to update the NTP source for all MSX nodes to another clock source. Before running this playbook, update the NTP entry in group_vars/all/main.yml to set the new source. When run, this playbook will update the NTP clock source for chrony on all nodes, restart chrony, and then check if the clocks are synchronized.

To update the NTP source for all MSX nodes, run the following command:

ansible-playbook checks/update-chrony-source.yml

Integrating Datadog

Datadog is a cloud-based third-party monitoring service that provides detailed metrics, insights, and alerts for your underlying infrastructure. To use Datadog, you must obtain a Datadog API key and container images for the Datadog agents and the kube-state-metric. The agent containers reside on each node and on each Cassandra pod. This section explains the different installation scenarios.


Note

The Datadog (and kube-state-metric) container images are not included with the deployment and must be made available to the installer container (in /msx-<version>/ansible/repo/) before installing the Datadog components. You can obtain the Datadog container images from the Datadog website and the kube-state-metric from this location.

Datadog can be installed in the following scenarios:

  • On a Running MSX System

  • During an Initial MSX Deployment

  • During an MSX Upgrade

This appendix contain the following sections:

Prerequisites for All Datadog Deployment Scenarios

Use this procedure to install Datadog in all scenarios.

Procedure

Step 1

Open group_vars/all/datadog.yml on the Installer container.

Step 2

Uncomment and edit the dd_api_key variable to insert a valid API key.

Step 3

By default, the install process looks for the Datadog container archive images in /msx-<version>/ansible/repo/ on the installer container. If you would like to use a different directory, then specify that location in the dd_images_dir variable.

Step 4

Update the dd_images dictionary variable with the appropriate image names and versions.

Note 

The dd_images.dd_agent variable is not currently used but is defined here for future reference.

Installing Datadog on a Running System


Note

If you install Datadog on a running MSX system, it will cause the Cassandra nodes to perform a rolling restart as a new sidecar with the Datadog agent added to them.

Use this procedure to install Datadog on a running system.

Procedure

Step 1

Run ansible-playbook upload-datadog-images.yml to upload the container images to the registry.

Step 2

Run ansible-playbook deploy-datadog.yml to perform the installation.

Installing Datadog During an Initial MSX Deployment

Use this procedure to install Datadog during the MSX installation process.

After running the isolated.yml playbook, use this command: 

ansible-playbook upload-datadog-images.yml

This action will upload the container images to the registry. You continue the install process with the deploy-infra-services.yml as usual. If a valid API key has been configured in datadog.yml, Datadog will automatically be installed during the MSX installation.

Installing Datadog During the MSX Upgrade

Use this procedure to install Datadog during the MSX upgrade process.

After running the upload-isolated-binaries.yml playbook, use this command: 

ansible-playbook upload-datadog-images.yml

This action will upload the Datadog container images to the registry. The rest of the Datadog installation process will automatically proceed during the MSX upgrade process with no user intervention required.