Generate and Extract the RCA File from Cisco DNA Center

Introduction

This document describes the process to create and extract the Root Cause Analysis (RCA) file from Cisco Dital Network Architecture (DNA) Center.

Contributed by Alexandro Carrasquedo and Rahul Rammanohar, Cisco TAC Engineers

Prerequisites

Requirements

It is required that the user as Command Line Interface (CLI) access to the DNA Center. To log into DNA Center using CLI you must connect via Secure Socket Shell (SSH) to your DNA Center's IP address using maglev as the username on port 2222.

Generate the RCA file in a single node cluster.

Step 1. Log into DNA Center Command Line Interface (CLI) on port 2222. Use maglev as username, unless it was modified during the initial setup. Then run run the sudo RCA command.

[Tue Sep 11 15:08:48 UTC] maglev@10.1.1.4 (maglev-master-1) ~
$ sudo rca
[sudo] password for maglev: 

===============================================================
Verifying ssh/sudo access
===============================================================
Done

===============================================================
Verifying administration access
===============================================================
[administration] password for 'admin': <type your admin password>
User 'admin' logged into 'kong-frontend.maglev-system.svc.cluster.local' successfully

===============================================================
RCA package created on Tue Sep 11 15:32:47 UTC 2018
===============================================================

2018-09-11 15:32:47 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
/etc/cron.d/clean-journal-files
<snip>
/data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC/docker_inspect_k8s_platform-ui_platform-ui-2963217120-rxv5d_maglev-system_1a09eb87-9f00-11e8-9d42-005d73c0c790_0.log
/data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC/sudo_ethtool_calife1d52fff20.log
2018-09-11 15:43:14 | INFO | Cleaning up RCA temp files...

Created RCA package: /data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz

[Tue Sep 11 15:43:14 UTC] maglev@10.1.1.4 (maglev-master-1) ~

Note: The RCA file is generated and stored in /data/rca. The name of the file is in the following format: maglev-<inter-cluster link IP>-rca<date and time>.tar.gz it usually takes around 20 minutes to create the file.

Generate the RCA file in an n-node cluster. 

Tip: When you have a functional n-node cluster, services are distribuited this means that if you have service A running in node-1 and you get the RCA from node-2, the logs from service A will not be included. Therefore, we recommend that you include the RCA file of all nodes when Cisco TAC requests an RCA file.

When you have a 3-node cluster and you run the rca command on any device, DNA Center  asks you for a cluster. DNA Center is refering to the enter inter-cluster IP address of the node you want to extract the RCA from.

In this example, the inter-cluster IP addresses are in the 10.1.1.0/29 range.

[Wed May 30 18:24:26 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ sudo rca
 
===============================================================
Verifying ssh/sudo access
===============================================================
Done
 
===============================================================
Verifying administration access
===============================================================
Cluster: 10.1.1.3
[administration] username for 'https://10.1.1.3:443': admin
[administration] password for 'admin': <type your admin password>
User 'admin' logged into '10.1.1.3' successfully
 
===============================================================
RCA package created on Wed May 30 18:24:44 UTC 2018
===============================================================
 
2018-05-30 18:24:44 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
/etc/cron.d/run-remedyctl

After you run the rca command, the the inter-cluster IP addresses that you specified are cached in /home/maglev/.maglevconf. The next time you run the rca command, DNA Center will use the same node to get the RCA information.

[Wed May 30 18:23:37 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ sudo rca
[sudo] password for maglev: 
 
===============================================================
Verifying ssh/sudo access
===============================================================
Done
 
===============================================================
Verifying administration access
===============================================================
[administration] password for 'admin': <type your admin password>
User 'admin' logged into '10.1.1.3' successfully <-- it automatically logged into the cluster you previously defined inter-cluster IP
 
===============================================================
RCA package created on Wed May 30 18:23:46 UTC 2018
===============================================================
 
2018-05-30 18:23:46 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
… rca continued…

If you need to run the rca command on a different node, you need to delete the context that is configured in DNA Center. Then, DNA Center will ask you to select a new inter-cluster IP address and you can define the other node's IP address.

[Wed May 30 18:24:10 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ sudo maglev context delete maglev-1
Removed command line context 'maglev-1'
 
[Wed May 30 18:24:18 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ more /home/maglev/.maglevconf
;---------------------------------------------------------------------
; Modified by Maglev: Wed, 30 May 2018 18:24:18 UTC
; maglev 73529
;---------------------------------------------------------------------
 
[global]
 
 
[Wed May 30 18:24:26 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ sudo rca
 
===============================================================
Verifying ssh/sudo access
===============================================================
Done
 
===============================================================
Verifying administration access
===============================================================
Cluster: 10.1.1.2 <-- now it asks for the new cluster IP
[administration] username for 'https://10.1.1.2:443': admin
[administration] password for 'admin': <type your admin password>
User 'admin' logged into '10.1.1.2' successfully
 
===============================================================
RCA package created on Wed May 30 18:24:44 UTC 2018
===============================================================
 
2018-05-30 18:24:44 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
/etc/cron.d/run-remedyctl

Extract the RCA file on a Windows computer.

Step 1. Download winscp or your favorite SCP client.

Step 2. Log into DNA Center using your CLI credentials, choose SCP as the File protocol and select port number 2222.

Step 3. Go to the /data/rca folder

Step 4. Copy the RCA file to your local computer.

Extract the RCA file on a MAC/Linux computer.

Note: In this example my DNA Center IP address resolves to mxc-dnac4.cisco.com, please replace this hostname with your DNA Center FQDN or IP address

Step 1. Open your terminal and follow the next steps to copy the RCA file named maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz stored in your DNA Center in /data/rca to the present working directory in your computer.

ALECARRA-M-P1Z8:~ alecarra$ scp -P 2222 maglev@mxc-dnac4.cisco.com:/data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz .
Welcome to the Maglev Appliance
maglev@mxc-dnac4.cisco.com's password: <type your maglev password>
maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz                                                                                                    100%  335MB   3.3MB/s   01:41    
ALECARRA-M-P1Z8:~ alecarra$ 

Push the RCA file to a MAC/Linux computer with the use of the following syntax.

$ scp /data/rca/<RCA file name> <MAC/Linux username>@<MAC/Linux IP address>:<path to save the file>

The following is an example of the command used in the lab:

$ scp /data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz alecarra@10.24.133.238:/Users/alecarra/Documents/DNA
The authenticity of host '10.24.133.238 (10.24.133.238)' can't be established.
ECDSA key fingerprint is SHA256:u660kUomvMParNkcPIm7oXrDp84rilP5CM9wCWCFOAE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.24.133.238' (ECDSA) to the list of known hosts.
Password: <type your linux/MAC password>
maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz                                                                                                    100%  335MB   3.7MB/s   01:32    

Upload the RCA file to the TAC Case from your MAC/Linux/Windows

You can use the Case File Uploader tool to upload the RCA file using a browser to your Cisco TAC Service Request and specify the case number where required. 

Push the RCA file (or any other file) directly from your Cisco DNA Center to your Cisco Service Request

There are two options to upload a file (like the RCA) directly from the Cisco DNA Center to a Cisco Service Request. In both options, the username is the SR number and the password is a token that is unique to every SR. The username/password is always present in a note at the beginning of your Service Request and can also be retrieved from SCM, for more details on the token check out link.

Sample output from a Service Request:

Subject: 688046089: CXD Upload Credentials

You can now upload files to the case using FTP/FTPS/SCP/SFTP/HTTPS protocols and the following details:
Hostname: cxd.cisco.com
Username: 688046089
Password: gX***********P7

Option 1: Upload the file via HTTPS (fastest option and uses 443)

Step 1 :: Test whether you have connectivity from your Cisco DNA Center to cxd.cisco.com via port 443. There are multiple ways to do this, here is one of the ways.

$ nc -zv cxd.cisco.com 443
Connection to cxd.cisco.com 443 port [tcp/https] succeeded!
$

Step 2 :: If the above step is unsuccesful, then you cannot use this option to upload your file. Once the above test is successful, then upload the file via HTTPS.

Command:

$ curl –T “<filename with path>” -u <SRnumber> https://cxd.cisco.com/home/ 

(if you want to see a more detailed view of the upload, then add the -v option. For example, “curl -vT …”)

For example:

$ curl -T "./test.txt" -u 688046089 https://cxd.cisco.com/home/
Enter host password for user '688046089': <Type your CXD Upload password, unique to a Service Request, here> 
[Tue Dec 10 13:35:47 UTC] maglev@192.168.5.11 (maglev-master-1) ~
$

Option 2: Upload the file via SCP (uses port 22)

Step 1:: Test whether you have connectivity from your Cisco DNA Center to cxd.cisco.com via port 22. There are multiple ways to do this, here is one of the ways.

$ nc -zv cxd.cisco.com 22
Connection to cxd.cisco.com 22 port [tcp/ssh] succeeded!
$

Step 2 :: If the above step is unsuccesful, then you cannot use this method to upload your file. Once the above test is successful, then upload the file via scp.

Command:

$ scp <local filename with path> <SRnumber>@cxd.cisco.com:

For example:

$ scp ./test.txt 688046089@cxd.cisco.com:
The authenticity of host 'cxd.cisco.com (72.163.14.108)' can't be established.
RSA key fingerprint is SHA256:3c8Vi3Ms2AITZlNzkBccR1pvE5ie9oMs64Uh0uhRado.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cxd.cisco.com,72.163.14.108' (RSA) to the list of known hosts.
688046089@cxd.cisco.com's password: <Type your  CXD Upload password, unique to a Service Request, here>
test.txt                                                                                                                                                                                                        100%   39     0.0KB/s   00:00    

[Tue Dec 10 13:44:27 UTC] maglev@192.168.5.11  (maglev-master-1) ~
$