Generate and Extract the RCA File from Cisco DNA Center

Introduction

This document describes how to create and extract the Root Cause Analysis (RCA) file from the Cisco Digital Network Architecture (DNA) Center.

Prerequisites

Requirements

You must have Command Line Interface (CLI) access to the DNA Center. To log into DNA Center using CLI, you must connect via Secure Socket Shell (SSH) to your DNA Center's IP address with maglev as the username on port 2222.

Generate the RCA file in a single-node cluster

Step 1. Log into DNA Center Command Line Interface (CLI) on port 2222. Use maglev as the username, unless the username was modified during the initial setup. Then run the rca command.

[Tue Sep 11 15:08:48 UTC] maglev@10.1.1.4 (maglev-master-1) ~
$ rca
[sudo] password for maglev: 

===============================================================
Verifying ssh/sudo access
===============================================================
Done

===============================================================
Verifying administration access
===============================================================
[administration] password for 'admin': <type your admin password>
User 'admin' logged into 'kong-frontend.maglev-system.svc.cluster.local' successfully

===============================================================
RCA package created on Tue Sep 11 15:32:47 UTC 2018
===============================================================

2018-09-11 15:32:47 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
/etc/cron.d/clean-journal-files
<snip>
/data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC/docker_inspect_k8s_platform-ui_platform-ui-2963217120-rxv5d_maglev-system_1a09eb87-9f00-11e8-9d42-005d73c0c790_0.log
/data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC/sudo_ethtool_calife1d52fff20.log
2018-09-11 15:43:14 | INFO | Cleaning up RCA temp files...

Created RCA package: /data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz

[Tue Sep 11 15:43:14 UTC] maglev@10.1.1.4 (maglev-master-1) ~

Note: The RCA file is generated and stored in /data/rca. It usually takes around 20 minutes to create the file. The filename will have this format: maglev-<inter-cluster link IP>-rca<date and time>.tar.gz

Generate the RCA file in an n-node cluster

Tip: When you have a functional n-node cluster, services are distributed. When services are distributed, the RCA from an individual node will not include logs from services that run in other nodes. For example, if you have service A running in node-1 and you get the RCA from node-2, the logs from service A will not be included. Therefore, we recommend that you include the RCA file of all nodes when Cisco TAC requests an RCA file.

When you have a 3-node cluster and you run the rca command on any device, DNA Center prompts you for a cluster. At the prompt, enter the inter-cluster IP address of the node that you want to extract the RCA from.

In this example, the inter-cluster IP addresses are in the 10.1.1.0/29 range.

[Wed May 30 18:24:26 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$  rca
 
===============================================================
Verifying ssh/sudo access
===============================================================
Done
 
===============================================================
Verifying administration access
===============================================================
Cluster: 10.1.1.3
[administration] username for 'https://10.1.1.3:443': admin
[administration] password for 'admin': <type your admin password>
User 'admin' logged into '10.1.1.3' successfully
 
===============================================================
RCA package created on Wed May 30 18:24:44 UTC 2018
===============================================================
 
2018-05-30 18:24:44 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
/etc/cron.d/run-remedyctl

After you run the rca command, the inter-cluster IP addresses that you specified are cached in /home/maglev/.maglevconf. The next time you run the rca command, DNA Center will use the same node to get the RCA information.

[Wed May 30 18:23:37 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ rca
[sudo] password for maglev: 
 
===============================================================
Verifying ssh/sudo access
===============================================================
Done
 
===============================================================
Verifying administration access
===============================================================
[administration] password for 'admin': <type your admin password>
User 'admin' logged into '10.1.1.3' successfully <-- it automatically logged into the cluster you previously defined inter-cluster IP
 
===============================================================
RCA package created on Wed May 30 18:23:46 UTC 2018
===============================================================
 
2018-05-30 18:23:46 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
… rca continued…

If you need to run the rca command on a different node, you must delete the context that is configured in DNA Center. Then DNA Center will ask you to select a new inter-cluster IP address and you can define the other node's IP address.

[Wed May 30 18:24:10 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ sudo maglev context delete maglev-1
Removed command line context 'maglev-1'
 
[Wed May 30 18:24:18 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ more /home/maglev/.maglevconf
;---------------------------------------------------------------------
; Modified by Maglev: Wed, 30 May 2018 18:24:18 UTC
; maglev 73529
;---------------------------------------------------------------------
 
[global]
 
 
[Wed May 30 18:24:26 UTC] maglev@10.1.1.2 (maglev-master-10) ~
$ rca
 
===============================================================
Verifying ssh/sudo access
===============================================================
Done
 
===============================================================
Verifying administration access
===============================================================
Cluster: 10.1.1.2 <-- now it asks for the new cluster IP
[administration] username for 'https://10.1.1.2:443': admin
[administration] password for 'admin': <type your admin password>
User 'admin' logged into '10.1.1.2' successfully
 
===============================================================
RCA package created on Wed May 30 18:24:44 UTC 2018
===============================================================
 
2018-05-30 18:24:44 | INFO | Generating log for 'date'...
tar: Removing leading `/' from member names
/etc/cron.d/
/etc/cron.d/run-remedyctl

Extract the RCA file on a Windows computer

Step 1. Download winscp or your favorite SCP client.

Step 2. Log into DNA Center using your CLI credentials, choose SCP as the file protocol, and select port number 2222.

Step 3. Go to the /data/rca folder.

Step 4. Copy the RCA file to your local computer.

Extract the RCA file on a MAC/Linux computer

Note: In this example, the DNA Center IP address resolves to mxc-dnac4.cisco.com. Replace this hostname with your DNA Center fully qualified domain name (FQDN) or IP address.

Step 1. Open your terminal. Then follow the next steps to copy the RCA file named maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz stored in your DNA Center in /data/rca to the present working directory in your computer.

ALECARRA-M-P1Z8:~ alecarra$ scp -P 2222 maglev@mxc-dnac4.cisco.com:/data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz .
Welcome to the Maglev Appliance
maglev@mxc-dnac4.cisco.com's password: <type your maglev password>
maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz                                                                                                    100%  335MB   3.3MB/s   01:41    
ALECARRA-M-P1Z8:~ alecarra$ 

Push the RCA file to a MAC/Linux computer

Use this syntax:

 $ scp /data/rca/<RCA file name> <MAC/Linux username>@<MAC/Linux IP address>:<path to save the file>

Here is an example of the command used in the lab:

$ scp /data/rca/maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz alecarra@10.24.133.238:/Users/alecarra/Documents/DNA
The authenticity of host '10.24.133.238 (10.24.133.238)' can't be established.
ECDSA key fingerprint is SHA256:u660kUomvMParNkcPIm7oXrDp84rilP5CM9wCWCFOAE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.24.133.238' (ECDSA) to the list of known hosts.
Password: <type your linux/MAC password>
maglev-10.1.1.4-rca-2018-09-11_15-32-40_UTC.tar.gz                                                                                                    100%  335MB   3.7MB/s   01:32    

Upload the RCA file to the TAC Case from your MAC/Linux/Windows computer

You can use the Case File Uploader tool to upload the RCA file to your Cisco TAC Service Request via a browser. Specify the case number where required. 

Push the RCA file directly from your Cisco DNA Center to your Cisco Service Request

There are two options to upload a file (such as the RCA) directly from the Cisco DNA Center to a Cisco Service Request (SR). In both options, the username is the SR number and the password is a token that is unique to every SR. The username/password is always present in a note at the beginning of your Service Request and can also be retrieved from SCM. For more details on the token, refer to Customer File Uploads to Cisco Technical Assistance Center.

Sample output from a Service Request:

Subject: 688046089: CXD Upload Credentials

You can now upload files to the case using FTP/FTPS/SCP/SFTP/HTTPS protocols and the following details:
Hostname: cxd.cisco.com
Username: 688046089
Password: gX***********P7

Upload Option 1. Upload the file via HTTPS (fastest option and uses 443)

Step 1. Test whether you have connectivity from your Cisco DNA Center to cxd.cisco.com via port 443. Here is one way to perform the test:

$ nc -zv cxd.cisco.com 443
Connection to cxd.cisco.com 443 port [tcp/https] succeeded!
$

Note: If the test did not succeed, then you cannot use this method to upload your file.

Step 2. If the test succeeded, then upload the file via HTTPS.

Command:

$ curl –T “<filename with path>” -u <SRnumber> https://cxd.cisco.com/home/ 

(if you want to see a more detailed view of the upload, then add the -v option. For example, “curl -vT …”)

For example:

$ curl -T "./test.txt" -u 688046089 https://cxd.cisco.com/home/
Enter host password for user '688046089': <Type your CXD Upload password, unique to a Service Request, here> 
[Tue Dec 10 13:35:47 UTC] maglev@192.168.5.11 (maglev-master-1) ~
$

Upload Option 2. Upload the file via SCP (uses port 22)

Step 1. Test whether you have connectivity from your Cisco DNA Center to cxd.cisco.com via port 22. Here is one way to perform the test:

$ nc -zv cxd.cisco.com 22
Connection to cxd.cisco.com 22 port [tcp/ssh] succeeded!
$

Note: If the test did not succeed, then you cannot use this method to upload your file.

Step 2. If the test succeeded, then upload the file via scp.

Command:

$ scp <local filename with path> <SRnumber>@cxd.cisco.com:

For example:

$ scp ./test.txt 688046089@cxd.cisco.com:
The authenticity of host 'cxd.cisco.com (72.163.14.108)' can't be established.
RSA key fingerprint is SHA256:3c8Vi3Ms2AITZlNzkBccR1pvE5ie9oMs64Uh0uhRado.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cxd.cisco.com,72.163.14.108' (RSA) to the list of known hosts.
688046089@cxd.cisco.com's password: <Type your  CXD Upload password, unique to a Service Request, here>
test.txt                                                                                                                                                                                                        100%   39     0.0KB/s   00:00    

[Tue Dec 10 13:44:27 UTC] maglev@192.168.5.11  (maglev-master-1) ~
$