- About This Guide
- Ultra M Overview
- Hardware Specifications
- Software Specifications
- Networking Overview
- Deploying the Ultra M Solution
- Event and Syslog Management Within the Ultra M Solution
- Network Definitions (Layer 2 and 3)
- Example ultram_cfg.yaml File
- Ultra M MIB
- Ultra M Component Event Severity and Fault Code Mappings
- Ultra M Troubleshooting
- Using the UCS Utilities Within the Ultra M Manager
- ultram_ucs_utils.py Help
Ultra M Solutions Guide, Release 5.7
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
- Updated:
- October 26, 2017
Chapter: Ultra M Troubleshooting
Ultra M Troubleshooting
Ultra M Component Reference Documentation
The following sections provide links to troubleshooting information for the various components that comprise the Ultra M solution.
UCS C-Series Server
Nexus 9000 Series Switch
Catalyst 2960 Switch
Red Hat
OpenStack
UAS
Refer to the USP Deployment Automation Guide.
UGP
Refer to the Ultra Gateway Platform System Administration Guide.
Collecting Support Information
From UCS:
-
Collect support information:
chassis show tech support show tech support (if applicable)
-
Check which UCS MIBS are being polled (if applicable). Refer to https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/mib/c-series/b_UCS_Standalone_C-Series_MIBRef/b_UCS_Standalone_C-Series_MIBRef_chapter_0100.html
From Host/Server/Compute/Controler/Linux:
-
Identify if Passthrought/SR-IOV is enabled.
-
Run sosreport:
Note
This functionality is enabled by default on Red Hat, but not on Ubuntu. It is recommended that you enable sysstat and sosreport on Ubuntu (run apt-get install sysstat and apt-get install sosreport). It is also recommended that you install sysstat on Red Hat (run yum install sysstat). -
Get and run the os_ssd_pac script from Cisco:
-
Compute (all):
./os_ssd_pac.sh -a ./os_ssd_pac.sh -k -s
Note
For initial collection, it is always recommended to include the -s option (sosreport). Run ./os_ssd_pac.sh -h for more information. -
Controller (all):
./os_ssd_pac.sh -f ./os_ssd_pac.sh -c -s
Note
For initial collection it is always recommended to include the -s option (sosreport). Run ./os_ssd_pac.sh -h for more information.
-
-
For monitoring purposes, from crontab use option: -m ( for example run every 5 or 10 minutes)
From Switches
From all switches connected to the Host/Servers. (This also includes other switches which have same vlans terminated on the Host/Servers.)
show tech-support syslogs snmp traps
Note | It is recommended that mac-move notifications are enabled on all switches in network by running mac address-table notification mac-move. |
From ESC (Active and Standby)
Note | It is recommended that you take a backup of the software and data before performing any of the following operations. Backups can be taken by executing opt/cisco/esc/esc-scripts/esc_dbtool.py backup. (Refer to https://www.cisco.com/c/en/us/td/docs/net_mgmt/elastic_services_controller/2-3/user/guide/Cisco-Elastic-Services-Controller-User-Guide-2-3/Cisco-Elastic-Services-Controller-User-Guide-2-2_chapter_010010.html#id_18936 for more information.) |
/opt/cisco/esc/esc-scripts/health.sh /usr/bin/collect_esc_log.sh ./os_ssd_pac -a
From UAS
-
Monitor ConfD:
confd -status confd --debug-dump /tmp/confd_debug-dump confd --printlog /tmp/confd_debug-dump
Note
Once the file /tmp/confd_debug-dump> is collected, it can be removed (rm /tmp/confd_debug-dump). -
Monitor UAS Components:
source /opt/cisco/usp/uas/confd-6.1/confdrc confd_cli -u admin -C show uas show uas ha-vip show uas state show confd-state show running-config show transactions date-and-time show logs | display xml show errors displaylevel 64 show notification stream uas_notify last 1000 show autovnf-oper:vnfm show autovnf-oper:vnf-em show autovnf-oper:vdu-catalog show autovnf-oper:transactions show autovnf-oper:network-catalog show autovnf-oper:errors show usp show confd-state internal callpoints show confd-state webui listen show netconf-state
-
Monitor Zookeeper:
/opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh ls /config/control-function /opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh ls /config/element-manager /opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh ls /config/session-function /opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh ls / /opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh ls /stat /opt/cisco/usp/packages/zookeeper/current/bin/zkCli.sh ls /log
-
Collect Zookeeper data:
cd /tmp tar zcfv zookeeper_data.tgz /var/lib/zookeeper/data/version-2/ ls -las /tmp/zookeeper_data.tgz
-
Get support details
./os_ssd_pac -a
From UEM (Active and Standby)
-
Collect logs
/opt/cisco/em-scripts/collect-em-logs.sh -
Monitor NCS:
ncs -status ncs --debug-dump /tmp/ncs_debug-dump ncs --printlog /tmp/ncs_debug-dump
Note
Once the file /tmp/ncs_debug-dump is collected, it can be removed (rm /tmp/ncs_debug-dump). -
Collect support details:
./os_ssd_pac -a
From UGP (Through StarOS)
-
Collect the multiple outputs of the show support details.
Note
It is recommended to collect at least two samples, 60 minutes apart if possible. -
Collect raw bulkstats before and after events.
-
Collect syslogs and snmp traps before and after events.
-
Collect PCAP or sniffer traces of all relevant interfaces if possible.
Note
Familiarize yourself with how running SPAN/RSPAN on Nexus and Catalyst switches. This is important for resolving Passthrough/SR-IOV issues. -
Collect console outputs from all nodes.
-
Export CDRs and EDRs.
-
Collect the outputs of monitor subscriber next-call or monitor protocol depending on the activity
-
Refer to https://supportforums.cisco.com/sites/default/files/cisco_asr5000_asr5500_troubleshooting_guide.pdf for more information.
About Ultra M Manager Log Files
All Ultra M Manager log files are created under “/var/log/cisco/ultram-health”.
cd /var/log/cisco/ultram-health ls -alrt
Example output:
total 116 drwxr-xr-x. 3 root root 4096 Sep 10 17:41 .. -rw-r--r--. 1 root root 0 Sep 12 15:15 ultram_health_snmp.log -rw-r--r--. 1 root root 448 Sep 12 15:16 ultram_health_uas.report -rw-r--r--. 1 root root 188 Sep 12 15:16 ultram_health_uas.error -rw-r--r--. 1 root root 580 Sep 12 15:16 ultram_health_uas.log -rw-r--r--. 1 root root 24093 Sep 12 15:16 ultram_health_ucs.log -rw-r--r--. 1 root root 8302 Sep 12 15:16 ultram_health_os.error drwxr-xr-x. 2 root root 4096 Sep 12 15:16 . -rw-r--r--. 1 root root 51077 Sep 12 15:16 ultram_health_os.report -rw-r--r--. 1 root root 6677 Sep 12 15:16 ultram_health_os.log
NOTES:
-
The files are named according to the following conventions:
-
ultram_health_os: Contain information related to OpenStack
-
ultram_health_ucs: Contain information related to UCS
-
ultram_health_uas: Contain information related to UAS
-
-
Files with the “*.log” extension contain debug/error outputs from different components. These files get added to over time and contain useful data for debugging in case of issues.
-
Files with the “.report” extension contain the current report. These files get created on every tun.
-
Files with the “.error” extension contain actual data received from the nodes as part of health monitoring. These are the events that causes the Ultra M health monitor to send traps out. These files are updated every time a component generates an event.
Feedback