FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)
Information About Tracing
The tracing functionality logs internal events. Trace files are automatically created and saved to the tracelogs subdirectory under crashinfo.
The contents of trace files are useful for the following purposes:
Troubleshooting—If a switch has an issue, the trace file output may provide information that can be used for locating and solving the issue.
Debugging—The trace file outputs helps users get a more detailed view of system actions and operations.
To view the most recent trace information for a specific module, use the show platform software trace message command.
To modify the trace level to increase or decrease the amount of trace message output, you can set a new trace level using the set platform software trace command. Trace levels can be set for each process using the all-modules keyword in the set platform software trace command, or per module within a process.
Each process uses btrace infrastructure to log its trace messages. When a process is active, the corresponding in-memory tracelog is found in the directory /tmp/<FRU>/trace/, where <FRU> refers to the location where the process is running (rp, fp, or cc).
When a tracelog file has reached the maximum file size limit allowed for the process, or if the process ends, it gets rotated into the following directory:
/crashinfo/tracelogs, if the crashinfo: partition is available on the switch
/harddisk/tracelogs, if the crashinfo: partition is not available on the switch
The tracelog files are compressed before being stored in the directory.
All the tracelogs that are created using btrace have the following naming convention:
<process_name>_<FRU><SLOT>-<BAY>.<pid>_<counter>.<creation_timestamp>.bin
Here, counter is a free-running 64-bit counter that gets incremented for each new file created for the process. For example, wcm_R0-0.1362_0.20151006171744.bin. When compressed, the files will have the gz extension appended to their names
The maximum size limit for a tracelog file is 1MB for each process, and the maximum number of tracelog files that are maintained for a process is 25.
Initially, all the tracelog files are moved from the initial /tmp/<FRU>/trace directory to the /tmp/<FRU>/trace/stage staging directory. The btrace_rotate script then moves these tracelogs from the staging directory to the /crashinfo/tracelogs directory. When the number of files stored in the /crashinfo/tracelogs directory per process reaches the maximum limit, the oldest files for the process are deleted, while the newer files are maintained. This is repeated at every 60 minutes under worst-case situations.
There are two other sets of files that are purged from the /crashinfo/tracelogs directory:
Files that do not have the standard naming convention (other than a few exceptions such as fed_python.log)
Files older than two weeks
The throttling policy has been introduced so that a process with errors does not affect the functioning of the switch. Whenever a process starts logging at a very high rate, for example, if there are more than 16 files in a 4-second interval for the process in the staging directory, the process is throttled. The files do not rotate for the process from /tmp/<FRU>/trace into /tmp/<FRU>/trace/stage, however the files are deleted when they reach the maximum size. Throttling is re-enabled, when the count goes below 8.
Tracing levels determine how much information should be stored about a module in the trace buffer or file.
The following table shows all of the tracing levels that are available, and provides descriptions of the message that are displayed with each tracing level.
|
Tracing Level |
Description |
|---|---|
|
Emergency |
The message is regarding an issue that makes the system unusable. |
|
Error |
The message is regarding a system error. |
|
Warning |
The message is regarding a system warning. |
|
Notice |
The message is regarding a significant issue, but the switch is still working normally. |
|
Informational |
The message is useful for informational purposes only. |
|
Debug |
The message provides debug-level output. |
|
Verbose |
All possible trace messages are sent. |
|
Noise |
All possible trace messages for the module are logged. The noise level is always equal to the highest possible tracing level. Even if a future enhancement to tracing introduces a higher tracing level, the noise level will become equal to the level of that new enhancement. |
To set the trace level for a specific module within a process, use the set platform software trace command in privileged EXEC or user EXEC mode.
set platform software trace process slot module trace-level
The default tracing level for all modules is notice.
User EXEC (>)
Privileged EXEC (#)
|
Release |
Modification |
|---|---|
|
Cisco IOS XE Denali 16.1.1 |
This command was introduced. |
The module options vary by process and by hardware-module. Use the ? option when entering this command to see which module options are available with each keyword sequence.
Use the show platform software trace message command to view trace messages.
Trace files are stored in the tracelogs directory in the harddisk: file system. These files can be deleted without doing any harm to your switch operation.
Trace file output is used for debugging. The trace level is a setting that determines how much information should be stored in trace files about a module.
This example shows how to set the trace level for all the modules in dbm process:
Device# set platform software trace dbm R0 all-modules debug
To display the most recent trace information for a specific module, use the show platform software trace filter-binary command in privileged EXEC or user EXEC mode.
show platform software trace filter-binarymodules [ context mac-address]
| contextmac-address |
Represents the context used to filter. Additionally, you can filter based on module names and trace levels. The context keyword accepts either a MAC address or any other argument based on which a trace is tagged. |
User EXEC (>)
Privileged EXEC (#)
|
Release |
Modification |
|---|---|
|
Cisco IOS XE Denali 16.1.1 |
This command was introduced. |
This command collates and sorts all the logs present in the /tmp/.../ across all the processes relevant to the module. The trace logs of all the processes relevant to the specified module are printed to the console. This command also generates a file named collated_log_{system time} with the same content, in the /crashinfo/tracelogs directory.
This example shows how to display the trace information for a wireless module:
Device# show platform software trace filter-binary wireless
To display the trace messages for a process, use the set platform software trace command in privileged EXEC or user EXEC mode.
show platform software trace message process slot
User EXEC (>)
Privileged EXEC (#)
|
Release |
Modification |
|---|---|
|
Cisco IOS XE Denali 16.1.1 |
This command was introduced. |
This example shows how to display the trace messages for the Stack Manager and the Forwarding Engine Driver processes:
Device# show platform software trace message stack-mgr switch active R0 10/30 09:42:48.767 [btrace] [8974]: (note): Successfully registered module [97] [uiutil] 10/30 09:42:48.762 [btrace] [8974]: (note): Successfully registered module [98] [tdl_cdlcore_message] 10/29 13:28:19.023 [stack_mgr] [8974]: (note): Examining peer state 10/29 13:28:19.023 [stack_mgr] [8974]: (note): no switch eligible for standby election presently 10/29 13:28:19.022 [stack_mgr] [8974]: (note): Posting event stack_fsm_event_wait_standby_elect_timer_expired, curstate stack_fsm_state_active_ready 10/29 13:28:19.022 [stack_mgr] [8974]: (note): Timer HDL - STACK_WAIT_STANDBY_ELECT_TIMER expired 10/29 13:26:46.584 [btrace] [8974]: (note): Successfully registered module [99] [tdl_ui_message] 10/29 13:26:46.582 [bipc] [8974]: (note): Pending connection to server 10.129.1.0 10/29 13:26:36.582 [evutil] [8974]: (ERR): Connection attempt for sman-ui-serv (uipeer uplink to slot 1) failed, invoking disconnect 10/29 13:26:36.582 [evutil] [8974]: (ERR): Asynchronous connect failed for [uipeer uplink to slot 1] (fd == -1) 10/29 13:26:36.581 [bipc] [8974]: (note): Pending connection to server 10.129.1.0 10/29 13:26:26.581 [evutil] [8974]: (ERR): Connection attempt for sman-ui-serv (uipeer uplink to slot 1) failed, invoking disconnect Device# show platform software trace message fed switch active 11/02 10:55:01.832 [btrace]: [11310]: UUID: 0, ra: 0 (note): Successfully registered module [86] [uiutil] 11/02 10:55:01.848 [btrace]: [11310]: UUID: 0, ra: 0 (note): Single message size is greater than 1024 11/02 10:55:01.822 [btrace]: [11310]: UUID: 0, ra: 0 (note): Successfully registered module [87] [tdl_cdlcore_message] 11/01 09:54:41.474 [btrace]: [12312]: UUID: 0, ra: 0 (note): Successfully registered module [88] [tdl_ngwc_gold_message] 11/01 09:54:11.228 [btrace]: [12312]: UUID: 0, ra: 0 (note): Successfully registered module [89] [tdl_doppler_iosd_matm_type] 11/01 09:53:37.454 [btrace]: [11310]: UUID: 0, ra: 0 (note): Successfully registered module [90] [tdl_ui_message] 11/01 09:53:37.382 [bipc]: [11310]: UUID: 0, ra: 0 (note): Pending connection to server 10.129.1.0 11/01 09:53:34.227 [xcvr]: [18846]: UUID: 0, ra: 0 (ERR): FRU hardware authentication Fail, result = 1. 11/01 09:53:33.775 [ng3k_scc]: [18846]: UUID: 0, ra: 0 (ERR): SMART COOKIE: SCC I2C receive failed: rc=10 11/01 09:53:33.775 [ng3k_scc]: [18846]: UUID: 0, ra: 0 (ERR): SMART COOKIE receive failed, try again 11/01 09:53:33.585 [ng3k_scc]: [18846]: UUID: 0, ra: 0 (ERR):
To view the trace levels for all the modules under a specific process, use the show platform software trace level command in privileged EXEC or user EXEC mode.
show platform software trace level process slot
User EXEC (>)
Privileged EXEC (#)
|
Release |
Modification |
|---|---|
|
Cisco IOS XE Denali 16.1.1 |
This command was introduced. |
This example shows how to view the trace level:
Device# show platform software trace level dbm switch active R0
Module Name Trace Level
-------------------------------------------------
binos Notice
binos/brand Notice
bipc Notice
btrace Notice
bump_ptr_alloc Notice
cdllib Notice
chasfs Notice
dbal Informational
dbm Debug
evlib Notice
evutil Notice
file_alloc Notice
green-be Notice
ios-avl Notice
klib Debug
services Notice
sw_wdog Notice
syshw Notice
tdl_cdlcore_message Notice
tdl_dbal_root_message Notice
tdl_dbal_root_type Notice
To archive all the trace logs relevant to all the processes running on a system since the last reload on the switches and to save this in the specified location, use the request platform software trace archive command in privileged EXEC or user EXEC mode.
request platform software trace archive [ last number-of-days [ days [ target location]] | target location]
| last number-of-days |
Specifies the number of days for which the trace files have to be archived. |
| target location |
Specifies the location and name of the archive file. |
User EXEC (>)
Privileged EXEC (#)
|
Release |
Modification |
|---|---|
|
Cisco IOS XE Denali 16.1.1 |
This command was introduced. |
This archive file can be copied from the system, using the tftp or scp commands.
This example shows how to archive all the trace logs of the processes running on the switch since the last 5 days:
Device# request platform software trace archive last 5 days target flash:test_archive
To rotate all the current in-memory trace logs into the crashinfo partition and start a new in-memory trace log for each process, use the request platform software trace rotate all command in privileged EXEC or user EXEC mode.
request platform software trace rotate all
User EXEC (>)
Privileged EXEC (#)
|
Release |
Modification |
|---|---|
|
Cisco IOS XE Denali 16.1.1 |
This command was introduced. |
The trace log files are for read-only purpose. Do not edit the contents of the file. If there is a requirement to delete the contents of the file to view certain set of logs, use this command to start a new trace log file.
This example shows how to rotate all the in-memory trace logs of the processes running on the switch since the last one day:
Device# request platform software trace slot switch active R0 archive last 1 days target flash:test
To collate and sort all the archived logs present in the tracelogs subdirectory, use the request platform software trace filter-binary command in privileged EXEC or user EXEC mode.
request platform software trace filter-binary modules [ context mac-address]
|
context mac-address |
Represents the context used to filter. Additionally, you can filter based on module names and trace levels. The context keyword accepts either a MAC address or any other argument based on which a trace is tagged. |
User EXEC (>)
Privileged EXEC (#)
|
Release |
Modification |
|---|---|
|
Cisco IOS XE Denali 16.1.1 |
This command was introduced. |
This command collates and sorts all the archived logs present in the tracelogs subdirectory, across all the processes relevant to the module. This command also generates a file named collated_log_{system time} with the same content, in the /crashinfo/tracelogs directory.
This example shows how to display the trace information for a wireless module:
Device# request platform software trace filter-binary wireless
To trace the Cisco Hyperlocation related messages, use the set platform software trace wireless switch active R0 hyperlocation command.
set platform software trace wireless switch active R0 hyperlocation { debug | emergency | error | info | noise | notice | verbose | warning}
| debug |
Debug messages |
| emergency |
Emergency possible message |
| error |
Error messages |
| info |
Informational messages |
| noise |
Maximum possible message |
| notice |
Notice messages |
| verbose |
Verbose debug messages |
| warning |
Warning messages |
Privileged EXEC
| Release | Modification |
|---|---|
|
Cisco IOS XE Denali 16.2.1 |
This command was introduced. |