Trace Management

Information About Trace Management

The tracing functionality logs internal events. Trace files are automatically created and saved on the persistent storage device of specific platforms.

If the device has issues, the contents of the trace files are useful to troubleshoot the issue. The trace file outputs provide logs that are used to locate and solve the issue, and helps to get a detailed view of system actions and operations.

To view the recent trace information for a specific process, use the show logging [process | Profile | process-helper] command. The process keyword uses the first few letters of the name of a process and provides trace logs of the process that starts or matches with the entered string, the profile keyword lists the predefined set of process names, and the profile-helper keyword displays the available names.

To change the verbosity in a trace message output, you can adjust the trace level of processes using the set platform software trace level command. You can choose the all keyword to adjust the trace level for all the processes listed or you can select a specific process. When you select a specific process, there's also the option to adjust the trace level for a specific module, or you can use the all-modules keyword to adjust all the modules of processes.

Introduction to Binary Tracing

Binary tracing is helpful in gathering trace information with a minimal impact on performance. In binary tracing, the tracing is always on for the system components and a basic level of trace is collected on all the time; thus, the data necessary for troubleshooting a problem has been captured the first time it occurs.

Introduction to Conditional Debugging and Radioactive Tracing

The Conditional Debugging feature allows you to enable debugging and logging for specific features based on the set of conditions you define. This feature is useful in systems where many features are supported.

The Conditional debug allows granular debugging in a network that is operating at a large scale with a large number of features. It allows you to observe detailed debugs for granular instances within the system. This type of debugging is useful when we need to debug only a particular session among thousands of sessions. It's also possible to specify multiple conditions.

A condition refers to a feature or identity, where an identity could be an interface, IP Address, or a MAC address and so on.

Conditional debugging is in contrast to the general debug command, that produces its output without discriminating on the feature objects that are being processed. General debug command consumes numerous system resources and impacts the system performance.

Radioactive tracing provides the ability to form a chain of execution for operations of interest across the system, at an increased verbosity level. This provides a way to print conditionally debug information (up to DEBUG Level or a specified level) across threads, processes, and function calls.

Radioactive Tracing when coupled with Conditional Debugging, provides a single debug command to debug all execution contexts related to the condition. You can execute this command without being aware of the various control flow processes of the feature within the box and without having to issue debugs at these processes individually.

Tracing Levels

Trace level determines the types of traces outputted. Each trace message is assigned a trace level. If the trace level of a process or its module it set as greater than or equal to the level as the trace message, the trace message is displayed otherwise, it's skipped. For example, the default trace level is Notice level, so all traces with the Notice level and below the notice level are included while the traces above the Notice level are excluded.

The following table shows the available tracing levels, and provides descriptions of the message that are displayed with each tracing level. The tracing levels listed in the table are from the lowest to the highest order. The default trace level is Notice.

Table 1. Tracing Levels and Descriptions

Tracing Level

Description

Fatal

The message stating the process is aborted.

Emergency

The message is regarding an issue that makes the system unusable.

Alert

The message indicating that an action must be taken immediately.

Critical

The message is regarding a critical event causing loss of important functions.

Error

The message is regarding a system error.

Warning

The message is regarding a system warning.

Notice

The message is regarding a significant event.

Informational

The message is useful for informational purposes only.

Debug

The message provides debug-level output.

Verbose

All possible trace messages are sent.

Noise

All possible trace messages for the module are logged.

The noise level is always equal to the highest possible tracing level. Even if a future enhancement to tracing introduces a higher tracing level, the noise level will become equal to the level of that new enhancement.

How to Configure Conditional Debugging

Conditional Debugging and Radioactive Tracing

Radioactive Tracing when coupled with Conditional Debugging, enable us to have a single debug CLI to debug all execution contexts related to the condition. This can be done without being aware of the various control flow processes of the feature within the box and without having to issue debugs at these processes individually.

Configuring Conditional Debugging

To configure conditional debugging, follow the steps given below:

Procedure

  Command or Action Purpose

Step 1

enable

Example:


Device> enable

Enables privileged EXEC mode.

  • Enter your password if prompted.

Step 2

debug platform condition mac {mac-address }

Example:

Device# debug platform condition mac bc16.6509.3314

Configures conditional debugging for the MAC Address specified.

Step 3

debug platform condition start

Example:

Device# debug platform condition start

Starts conditional debugging (this will start radioactive tracing if there is a match on one of the conditions above).

Step 4

show platform condition OR show debug

Example:

Device# show platform condition
Device# show debug

Displays the current conditions set.

Step 5

debug platform condition stop

Example:

Device# debug platform condition stop

Stops conditional debugging (this will stop radioactive tracing).

Step 6

request platform software trace archive [last {number } days ] [target {crashinfo: | flashinfo: }]

Example:

# request platform software trace archive last 2 days

(Optional) Displays historical logs of merged tracefiles on the system. Filter on any combination of number of days or location.

Step 7

show platform software trace [filter-binary | level | message ]

Example:

Device# show platform software trace message

(Optional) Displays logs merged from the latest tracefile. Filter on any combination of application condition, trace module name, and trace level.

  • filter-binary - Filter the modules to be collated

  • level - Show trace levels

  • message - Show trace message ring contents

Note

 

On the device:

  • Available from IOS console in addition to linux shell.

  • Generates a file with merged logs.

  • Displays merged logs only from staging area

Step 8

clear platform condition all

Example:

Device# clear platform condition all

Clears all conditions.

What to do next


Note
The commands request platform software trace filter-binary and show platform software trace filter-binary work in a similar way. The only difference is:

  • request platform software trace filter-binary - Sources the data from historical logs.

  • show platform software trace filter-binary – Sources the data from the flash Temp directory.

Of these, mac_log <..date..> is the most important file, as it gives the messages for the MAC we are debugging. The command show platform software trace filter-binary also generates the same flash files, and also prints the mac_log on the screen.

Collecting Trace Files

To collect trace files from a device, follow these steps:

  1. To request the tracelogs for a specific time period (For example: Five days), use the command:

    Device# request platform software trace archive last 5 day

  2. The system generates a tar ball (.gz file) of the tracelogs in the location /flash :

Copying tracefiles off the box

An example of the tracefile is shown below: 

Device# dir crashinfo:/tracelogs
Directory of crashinfo:/tracelogs/

50664 -rwx 760 Sep 22 2015 11:12:21 +00:00 plogd_F0-0.bin_0.gz
50603 -rwx 991 Sep 22 2015 11:12:08 +00:00 fed_pmanlog_F0-0.bin_0.9558.20150922111208.gz
50610 -rw- 11 Nov 2 2015 00:15:59 +00:00 timestamp
50611 -rwx 1443 Sep 22 2015 11:11:31 +00:00 auto_upgrade_client_sh_pmanlog_R0-.bin_0.3817.20150922111130.gz
50669 -rwx 589 Sep 30 2015 03:59:04 +00:00 cfgwr-8021_R0-0.bin_0.gz
50612 -rwx 1136 Sep 22 2015 11:11:46 +00:00 reflector_803_R0-0.bin_0.1312.20150922111116.gz
50794 -rwx 4239 Nov 2 2015 00:04:32 +00:00 IOSRP_R0-0.bin_0.14239.20151101234827.gz
50615 -rwx 131072 Nov 2 2015 00:19:59 +00:00 linux_iosd_image_pmanlog_R0-0.bin_0
--More—
The trace files can be copied using one of the various options shown below: 

Device# copy crashinfo:/tracelogs ?
  crashinfo: Copy to crashinfo: file system
  flash: Copy to flash: file system
  ftp: Copy to ftp: file system
  http: Copy to http: file system
  https: Copy to https: file system
  null: Copy to null: file system
  nvram: Copy to nvram: file system
  rcp: Copy to rcp: file system
  running-config Update (merge with) current system configuration
  scp: Copy to scp: file system
  startup-config Copy to startup configuration
  syslog: Copy to syslog: file system
  system: Copy to system: file system
  tftp: Copy to tftp: file system
  tmpsys: Copy to tmpsys: file system
The general syntax for copying onto a TFTP server is as follows: 

Device# copy source: tftp:
Device# copy crashinfo:/tracelogs/IOSRP_R0-0.bin_0.14239.20151101234827.gz tftp:
Address or name of remote host []? 2.2.2.2
Destination filename [IOSRP_R0-0.bin_0.14239.20151101234827.gz]?

Note
It is important to clear the generated report or archive files off the switch in order to have flash space available for tracelog and other purposes.

Configuration Examples for Trace Management

The following is an output example of the show platform condition command.

The following is a sample of the debug platform condition stop command. 

Device# debug platform condition stop
Conditional Debug Global State: Stop

The following is an example of the show logging command for the ios process. 

Device# show logging process ios         
Logging display requested on 2022/10/27 09:32:06 (PDT) for Hostname: [vwlc_1_9222], Model: [C9800-CL-K9], Version: [17.11.01], SN: [9ZY0U03YBM0], MD_SN: [9ZY0U03YBM0]

Displaying logs from the last 0 days, 0 hours, 10 minutes, 0 seconds
executing cmd on chassis 1 ...
Unified Decoder Library Init .. DONE
Found 1 UTF Streams

2022/10/27 09:31:52.835197577 {iosrp_R0-0}{1}: [parser_cmd] [26471]: (note): id= console@console:user= cmd: 'show logging process ios' SUCCESS 2022/10/27 08:31:48.762 PST
2022/10/27 09:31:59.651965736 {iosrp_R0-0}{1}: [parser_cmd] [26471]: (note): id= console@console:user= cmd: 'show logging process ios internal' SUCCESS 2022/10/27 08:31:56.485 PST
===========================================================
======= Unified Trace Decoder Information/Statistics ======
===========================================================
----------------- Decoder Input Information ---------------
===========================================================
 Num of Unique Streams .. 1
 Total UTF To Process ... 1
 Total UTM To Process ... 75403
 UTM Process Filter ..... ios
 MRST Filter Rules ...... 4
===========================================================
----------------- Decoder Output Information --------------
===========================================================
 First UTM TimeStamp ............... 2022/10/27 02:21:47.048461994
 Last UTM TimeStamp ................ 2022/10/27 09:32:04.919540850
 UTM [Skipped / Rendered / Total] .. 75401 / 2 / 75403
 UTM [ENCODED] ..................... 75266
 UTM [PLAIN TEXT] .................. 94
 UTM [DYN LIB] ..................... 0
 UTM [MODULE ID] ................... 0
 UTM [TDL TAN] ..................... 43
 UTM [APP CONTEXT] ................. 0
 UTM [MARKER] ...................... 0
 UTM [PCAP] ........................ 0
 UTM [LUID NOT FOUND] .............. 0
===========================================================

The following is an example of the show logging profile wireless command. 

Device# show logging profile wireless          
Logging display requested on 2023/03/13 09:07:09 (UTC) for Hostname: [FABRIEK], Model: [C8300-1N1S-4T2X], Version: [17.12.01], SN: [FDO24190V85], MD_SN: [FDO2451M13G]

Displaying logs from the last 0 days, 0 hours, 10 minutes, 0 seconds
executing cmd on chassis local ...
Unified Decoder Library Init .. DONE
Found 1 UTF Streams

2023/03/13 08:57:34.084609935 {iosrp_R0-0}{255}: [parser_cmd] [3793]: (note): id= 10.68.219.145@vty0:user= cmd: 'show logging profile wireless level info' SUCCESS 2023/03/13 08:57:31.376 UTC
2023/03/13 09:07:03.562290152 {iosrp_R0-0}{255}: [parser_cmd] [3793]: (note): id= 10.68.219.145@vty0:user= cmd: 'show logging profile wireless internal ' SUCCESS 2023/03/13 08:58:51.922 UTC
===========================================================
======= Unified Trace Decoder Information/Statistics ======
===========================================================
----------------- Decoder Input Information ---------------
===========================================================
 Num of Unique Streams .. 1
 Total UTF To Process ... 1
 Total UTM To Process ... 55410
 UTM Process Filter ..... cpp_cp,fman_fp,fman_rp,fman-rp,fman-fp,fed,mobilityd,nmspd,odm_proxy,rogued,rrm,repm,wncd,wncmgrd,wncd_x,IOSRP,smd,odm,wstatsd,linux-iosd-image,wncloudm,locationd
 MRST Filter Rules ...... 24
===========================================================
----------------- Decoder Output Information --------------
===========================================================
 First UTM TimeStamp ............... 2023/03/13 08:13:19.321653302
 Last UTM TimeStamp ................ 2023/03/13 09:07:08.462269864
 UTM [Skipped / Rendered / Total] .. 55408 / 2 / 55410
 UTM [ENCODED] ..................... 2
 UTM [PLAIN TEXT] .................. 0
 UTM [DYN LIB] ..................... 0
 UTM [MODULE ID] ................... 0
 UTM [TDL TAN] ..................... 0
 UTM [APP CONTEXT] ................. 0
 UTM [MARKER] ...................... 0
 UTM [PCAP] ........................ 0
 UTM [LUID NOT FOUND] .............. 0
 UTM Level [EMERGENCY / ALERT / CRITICAL / ERROR] .. 0 / 0 / 0 / 0
 UTM Level [WARNING / NOTICE / INFO / DEBUG] ....... 0 / 2 / 0 / 0
 UTM Level [VERBOSE / NOISE / INVALID] ............. 0 / 0 / 0
===========================================================

The following is an example of the show logging process-helper command. 

Device# show logging process-helper ios
Logging display requested on 2023/03/13 10:30:29 (UTC) for Hostname: [FABRIEK], Model: [C8300-1N1S-4T2X], Version: [17.12.01], SN: [FDO24190V85], MD_SN: [FDO2451M13G]

Displaying logs from the last 0 days, 0 hours, 10 minutes, 0 seconds
executing cmd on chassis local ...
Unified Decoder Library Init .. DONE
Found 1 UTF Streams

2023/03/13 10:30:16.884663022 {iosrp_R0-0}{255}: [parser_cmd] [3793]: (note): id= 10.68.219.145@vty0:user= cmd: 'enable' SUCCESS 2023/03/13 10:30:10.721 UTC
===========================================================
======= Unified Trace Decoder Information/Statistics ======
===========================================================
----------------- Decoder Input Information ---------------
===========================================================
 Num of Unique Streams .. 1
 Total UTF To Process ... 1
 Total UTM To Process ... 88985
 UTM Process Filter ..... IOSRP
 MRST Filter Rules ...... 1
===========================================================
----------------- Decoder Output Information --------------
===========================================================
 First UTM TimeStamp ............... 2023/03/13 08:13:19.321653302
 Last UTM TimeStamp ................ 2023/03/13 10:30:27.267645695
 UTM [Skipped / Rendered / Total] .. 88984 / 1 / 88985
 UTM [ENCODED] ..................... 1
 UTM [PLAIN TEXT] .................. 0
 UTM [DYN LIB] ..................... 0
 UTM [MODULE ID] ................... 0
 UTM [TDL TAN] ..................... 0
 UTM [APP CONTEXT] ................. 0
 UTM [MARKER] ...................... 0
 UTM [PCAP] ........................ 0
 UTM [LUID NOT FOUND] .............. 0
 UTM Level [EMERGENCY / ALERT / CRITICAL / ERROR] .. 0 / 0 / 0 / 0
 UTM Level [WARNING / NOTICE / INFO / DEBUG] ....... 0 / 1 / 0 / 0
 UTM Level [VERBOSE / NOISE / INVALID] ............. 0 / 0 / 0
===========================================================

Additional References for Trace Management

Related Documents

Related Topic Document Title

For complete syntax and usage information for the commands used in this chapter.

Command Reference Guide for catalyst 9K platforms.

Feature History for Trace Management

This table provides release and related information for features explained in this module.

These features are available on all releases subsequent to the one they were introduced in, unless noted otherwise.

Release

Feature

Feature Information
Cisco IOS XE Fuji 16.9.2

Conditional Debugging and Radioactive Tracing

The Conditional Debugging feature allows you to selectively enable debugging and logging for specific features based on the set of conditions you define.

Cisco IOS XE Cupertino 17.7.x

Binary Tracing

Binary tracing helps in gathering of trace information with a minimal impact on performance.

Use Cisco Feature Navigator to find information about platform and software image support. To access Cisco Feature Navigator, go to http://www.cisco.com/go/cfn.