Generate and Extract the RCA File from Cisco DNA Center

Introduction

This document describes how to create and extract the Root Cause Analysis (RCA) file from the Cisco Digital Network Architecture (DNA) Center.

Prerequisites

Requirements

You must have CLI access to Cisco DNA Center. To log in to Cisco DNA Center using the CLI, you must connect via Secure Socket Shell (SSH) to your Cisco DNA Center's management IP address with maglev as the username on port 2222.

Generate the RCA File in a Single-Node Cluster

Step 1. Log in to the Cisco DNA Center CLI on port 2222. Use maglev as the username, unless the username was modified at the time of the initial setup. Then run the rca command.

[Tue Sep 11 15:08:48 UTC] maglev@x.x.x.x (maglev-master-1) ~
$ sudo rca
[sudo] password for maglev: 

===============================================================
Verifying ssh/sudo access
===============================================================
Done

===============================================================
Verifying administration access
===============================================================
[administration] password for 'admin': <type your admin password>
User 'admin' logged into 'kong-frontend.maglev-system.svc.cluster.local' successfully

===============================================================
RCA package created on Tue Sep 11 15:32:47 UTC 2018
===============================================================

2018-09-11 15:32:47 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
/etc/cron.d/clean-journal-files
<snip>
/data/rca/maglev-x.x.x.x-rca-2018-09-11_15-32-40_UTC/docker_inspect_k8s_platform-ui_platform-ui-2963217120-rxv5d_maglev-system_1a09eb87-9f00-11e8-9d42-005d73c0c790_0.log
/data/rca/maglev-x.x.x.x-rca-2018-09-11_15-32-40_UTC/sudo_ethtool_calife1d52fff20.log
2018-09-11 15:43:14 | INFO | Cleaning up RCA temp files...

Created RCA package: /data/rca/maglev-x.x.x.x-rca-2018-09-11_15-32-40_UTC.tar.gz

[Tue Sep 11 15:43:14 UTC] maglev@x.x.x.x (maglev-master-1) ~

Note: The RCA file is generated and stored in /data/rca. It usually takes around 20 minutes to create the file. The filename will have this format: maglev-<inter-cluster link IP address>-rca<date and time>.tar.gz.

Generate the RCA File in an N-Node Cluster

Tip: When you have a functional n-node cluster, services are distributed. When the services are distributed, the RCA from an individual node will not include logs from services that run on other nodes. For example, if you have service A that runs on node-1 and you get the RCA from node-2, the logs from service A will not be included. Therefore, it is recommended that you capture and include the RCA file of all nodes in the cluster when the TAC requests an RCA file.

When you have a 3-node cluster and you run the rca command on any device, Cisco DNA Center prompts you for a cluster IP address. At the prompt, enter the inter-cluster IP address of the node that you want to retrieve the RCA from.

In this example, the inter-cluster IP addresses are in the 10.1.1.0/29 range.

[Wed May 30 18:24:26 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$  rca
 
===============================================================
Verifying ssh/sudo access
===============================================================
Done
 
===============================================================
Verifying administration access
===============================================================
Cluster: 10.1.1.3
[administration] username for 'https://10.1.1.3:443': admin
[administration] password for 'admin': <type your admin password>
User 'admin' logged into '10.1.1.3' successfully
 
===============================================================
RCA package created on Wed May 30 18:24:44 UTC 2018
===============================================================
 
2018-05-30 18:24:44 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
/etc/cron.d/run-remedyctl

After you run the rca command, the inter-cluster IP addresses that you specified are cached in /home/maglev/.maglevconf. The next time you run the rca command, Cisco DNA Center will use the same node to get the RCA information.

[Wed May 30 18:23:37 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ rca
[sudo] password for maglev: 
 
===============================================================
Verifying ssh/sudo access
===============================================================
Done
 
===============================================================
Verifying administration access
===============================================================
[administration] password for 'admin': <type the admin password>
User 'admin' logged into '10.1.1.3' successfully <-- it automatically logged into the cluster previously defined as the inter-cluster IP address
 
===============================================================
RCA package created on Wed May 30 18:23:46 UTC 2018
===============================================================
 
2018-05-30 18:23:46 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
… rca continued…

If you need to run the rca command on a different node, you must delete the context that is configured in Cisco DNA Center, then Cisco DNA Center will ask you to select a new inter-cluster IP address and you can define the other node's IP address.

[Wed May 30 18:24:10 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ sudo maglev context delete maglev-1
Removed command line context 'maglev-1'
 
[Wed May 30 18:24:18 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ more /home/maglev/.maglevconf
;---------------------------------------------------------------------
; Modified by Maglev: Wed, 30 May 2018 18:24:18 UTC
; maglev 73529
;---------------------------------------------------------------------
 
[global]
 
 
[Wed May 30 18:24:26 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ rca
 
===============================================================
Verifying ssh/sudo access
===============================================================
Done
 
===============================================================
Verifying administration access
===============================================================
Cluster: 10.1.1.2 <-- now it asks for the new cluster IP address
[administration] username for 'https://10.1.1.2:443': admin
[administration] password for 'admin': <type your admin password>
User 'admin' logged into '10.1.1.2' successfully
 
===============================================================
RCA package created on Wed May 30 18:24:44 UTC 2018
===============================================================
 
2018-05-30 18:24:44 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
/etc/cron.d/run-remedyctl

Extract the RCA file on a Windows computer

Step 1. Download WinSCP or your favorite SCP client.

Step 2. Log in to Cisco DNA Center using your CLI credentials, choose SCP as the file protocol, and select port number 2222.

Step 3. Navigate to the /data/rca folder.

Step 4. Copy the RCA file to your local computer.

Extract the RCA file on a Mac or Linux computer

Note: In this example, the Cisco DNA Center IP address resolves to mxc-dnac4.cisco.com. Replace this hostname with your Cisco DNA Center appliance's fully qualified domain name (FQDN) or IP address.

Step 1. Open a terminal session, then follow these steps to copy the RCA file named maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz stored on the Cisco DNA Center appliance in the /data/rca directory to the present working directory on your computer.

ALECARRA-M-P1Z8:~ alecarra$ scp -P 2222 maglev@mxc-dnac4.cisco.com:/data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz ./
Welcome to the Maglev Appliance
maglev@mxc-dnac4.cisco.com's password: <type your maglev password>
maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz                                                                                                    100%  335MB   3.3MB/s   01:41    
ALECARRA-M-P1Z8:~ alecarra$ 

Push the RCA file to a Mac or Linux computer

From the CLI of the Cisco DNA Center appliance, use this syntax:

 $ scp /data/rca/<RCA file name> <Mac/Linux username>@<Mac/Linux IP address>:<path to save the file>

Here is an example of the command used in the lab:

$ scp /data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz alecarra@10.24.133.238:/Users/alecarra/Documents/DNA
The authenticity of host '10.24.133.238 (10.24.133.238)' can't be established.
ECDSA key fingerprint is SHA256:u660kUomvMParNkcPIm7oXrDp84rilP5CM9wCWCFOAE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.24.133.238' (ECDSA) to the list of known hosts.
Password: <type your Linux or Mac user password>
maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz                                                                                                    100%  335MB   3.7MB/s   01:32    

Upload the RCA file to an existing TAC Service Request from your computer

You can use the Case File Uploader tool to upload the RCA file to your TAC Service Request via a browser. Specify the case number where required. 

Push the RCA file directly from your Cisco DNA Center appliance to your TAC Service Request

There are two options to upload a file (such as the RCA) directly from a Cisco DNA Center appliance to a TAC Service Request (SR). In both options, the username is the SR number and the password is a token that is unique to every SR. The username/password is always present in a note at the start of your service request, and can also be retrieved from SCM. For more details on the token, refer to Customer File Uploads to Cisco Technical Assistance Center.

Sample output from a service request:

Subject: 688046089: CXD Upload Credentials

You can now upload files to the case using FTP/FTPS/SCP/SFTP/HTTPS protocols and the following details:
Hostname: cxd.cisco.com
Username: 688046089
Password: gX***********P7

Upload Option 1. Upload the file via HTTPS (fastest option and uses port 443)

Step 1. Test whether you have connectivity from your Cisco DNA Center appliance to cxd.cisco.com via port 443. Here is one way to perform the test:

$ nc -zv cxd.cisco.com 443
Connection to cxd.cisco.com 443 port [tcp/https] succeeded!
$

Note: If the test did not succeed, then you cannot use this method to upload your file.

Step 2. If the test succeeded, upload the file via HTTPS with the use of this command:

$ curl –T “<filename with path>” -u <SR number> https://cxd.cisco.com/home/ 

(If you want to see a more detailed view of the upload, then add the -v option. For example, “curl -vT …”)

For example:

$ curl -T "./test.txt" -u 688046089 https://cxd.cisco.com/home/
Enter host password for user '688046089': <Type your CXD Upload password, unique to a Service Request, here> 
[Tue Dec 10 13:35:47 UTC] maglev@192.168.5.11 (maglev-master-1) ~
$

Upload Option 2. Upload the file via SCP (uses port 22)

Step 1. Test whether you have connectivity from your Cisco DNA Center appliance to cxd.cisco.com via port 22. Here is one way to perform the test:

$ nc -zv cxd.cisco.com 22
Connection to cxd.cisco.com 22 port [tcp/ssh] succeeded!
$

Note: If the test did not succeed, then you cannot use this method to upload your file.

Step 2. If the test succeeded, upload the file via SCP with the use of this command:

$ scp <local filename with path> <SR number>@cxd.cisco.com:

For example:

$ scp ./test.txt 688046089@cxd.cisco.com:
The authenticity of host 'cxd.cisco.com (X.X.X.X)' can't be established.
RSA key fingerprint is SHA256:3c8Vi3Ms2AITZlNzkBccR1pvE5ie9oMs64Uh0uhRado.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cxd.cisco.com,X.X.X.X' (RSA) to the list of known hosts.
688046089@cxd.cisco.com's password: <Type your CXD Upload password, unique to a service request, here>
test.txt                                                                                                                                                                                                        100%   39     0.0KB/s   00:00    

[Tue Dec 10 13:44:27 UTC] maglev@192.168.5.11  (maglev-master-1) ~
$