Conditional Debug, Radioactive Tracing, and Packet Tracing

Feature history for conditional debug and radioactive tracing

This table provides release and related information about the feature explained in this section.

This feature is also available in all the releases subsequent to the one in which they are introduced in, unless noted otherwise.

Table 1. Feature history for conditional debug and radioactive tracing

Feature Name

Release Information

Feature Description

Conditional debug and radioactive tracing

Cisco IOS XE 17.11.1

Conditional debugging is a diagnostic feature that selectively activates debugging and logging based on defined conditions to optimize resource usage and performance in complex networks, while radioactive tracing is a diagnostic mechanism that stitches together execution chains with increased verbosity, enabling conditional detailed debug information across system components for troubleshooting specific features.

Conditional debugging

A conditional debugging is a diagnostic feature that

  • enables selective activation of debugging and logging for specific features based on conditions you define

  • allows you to define custom conditions such as interface, IP address, or MAC address to narrow debug scope, and

  • helps reduce system resource usage and performance impact in large, complex networks.

Use this feature in systems with many supported features.

Conditional debug allows granular debugging in networks operating at large scale with many features. It allows you to observe detailed debug output for specific instances within the system.

In contrast, the general debug command produces output without filtering feature objects being processed. The general debug command uses significant system resources and affects performance.

Radioactive tracing

A radioactive tracing is a diagnostic tracing mechanism that

  • stitches together a chain of execution for operations of interest across the system, at an increased verbosity level

  • enables conditional printing of detailed debug information across threads, processes and function calls, and

  • troubleshoot system features, especially First-Hop Security (FHS) and mesh networking functions.


Note

  • For more information on First Hop Security features, see System Management > Wireless Multicast > Information About Wireless Multicast > Information About IPv6 Snooping.

  • The radioactive tracing filter does not work, if the certificate is not valid.

  • To debug mesh features effectively, add both Ethernet and Radio MAC addresses as conditional MACs for RA tracing when you collect logs.

  • To enable debugging for wireless IPs, use the debug platform condition feature wireless ip ip-address command.

Table 2. Components Supporting Radio Active Tracing

Components

Details

SISF or FHS

The first-hop security features, includes IPv6 Address Glean and IPv6 Device Tracking. For more information, see Information About IPv6 Snooping.

LISP

Locator or ID Separation Protocol.

Conditional debugging and radioactive tracing

A conditional debugging and radioactive tracing feature is a diagnostic mechanism that

  • combines the precision of conditional debugging to target specific execution contexts

  • leverages radioactive tracing to follow code paths across multiple control flows, and

  • enables use of a single debug CLI to debug all execution contexts related to the condition.


Note

Use the clear platform condition all command to remove the debug conditions applied to the platform.

Location of tracefiles

Tracefiles location.

Tracefiles are generated for each process and saved in either the /tmp/rp/trace or /tmp/fp/trace directory. Each trace log is written to a file of 1 MB. The directory can hold up to a maximum of 25 such files for a given process. When a tracefile in the /tmp directory reaches its 1MB limit or the configured size, it is rotated out to an archive location in the /crashinfo partition under the tracelogs directory.

  • Process-name_Process-ID_running-counter.timestamp.gz

    Example: IOSRP_R0-0.bin_0.14239.20151101234827.gz

  • Process-name_pmanlog_Process-ID_running-counter.timestamp.bin.gz

    Example: wncmgrd_R0-0.27958_1.20180902081532.bin.gz

The /tmp directory holds only a single tracefile for a given process. Once a tracefile reaches its size limit, the system moves it to /crashinfo/tracelogs. The archive directory stores up to 25 files and replaces the oldest file with each new file from /tmp. File size varies by process. Some processes use larger files that can reach 10 MB. The number of files stored in the tracelogs directory also depends on the process. For example, the WNCD process may use a limit of up to 400 files per instance, depending on the platform.

  • You can verify these logs (per-process) using the show platform software trace message process_name chassis active R0 command.

Configure conditional debugging (GUI)

Enable targeted troubleshooting by configuring conditional debugging through the GUI, allowing you to capture and analyze logs for specific MAC or IP addresses.

Procedure

Step 1

Choose Troubleshooting > Radioactive Trace. Click Add.

Step 2

Enter the MAC/IP Address.

Step 3

Click Apply to Device.

Step 4

Click Start to start or Stop to stop the conditional debug.

Step 5

Click Generate to create a radioactive trace log.

Step 6

Click the radio button to set the time interval.

Step 7

Click the Download Logs icon that is displayed next to the trace file name, to download the logs to your local folder.

Step 8

Click the View Logs icon that is displayed next to the trace file name, to view the log files on the GUI page.

Step 9

Click Load More to view more lines of the log file. Click Apply to Device.

Configure conditional debugging (CLI)

Enable targeted troubleshooting for wireless features by filtering debug output to specific APs, clients, or IP addresses using commands.

Procedure

Step 1

Configure conditional debugging for a feature using the specified MAC address.

Example:

Device# debug platform condition feature wireless mac b838.61a1.5433

Note

 

This is supported with AP or client MAC/IP and also on CMX IP address and mobility peer IP.

Step 2

Start conditional debugging.

Example:

Device# debug platform condition start

This starts radioactive tracing if there is a match on one of the conditions above.

Note

 

This is supported with AP or client MAC/IP and also on CMX IP address and mobility peer IP.

Step 3

Display the current conditions set.

Example:

Device# show platform condition
Device# show debug

Step 4

Stop conditional debugging.

Example:

Device# debug platform condition stop

This will stop radioactive tracing.

Note

 

This is supported with AP or client MAC/IP and also on CMX IP address and mobility peer IP.

Step 5

Display the logs from the latest wireless profile.

Example:

Device# show logging profile wireless [ counter start last {20 minutes} | filter mac {mac-address}  to-file bootflash:logs.txt]

Note

 

You can use either the show logging profile wireless command or show logging process command to collect the logs.

Step 6

Display the logs collection specific to the process.

Example:

Device# show logging process wncd to-file flash:wncd.txt

Step 7

Clear all conditions.

Example:

Device# clear platform condition all

What to do next


Note
The command request platform software trace filter-binary wirelessmac-address generates 3 flash files:

  • collated_log_<.date..>

  • mac_log <..date..>

  • mac_database .. file

The mac_log <..date..> file is the most important because it contains messages related to the MAC address under investigation. The show platform software trace filter-binary command generates the same flash files and displays the mac_log on the screen.

Radioactive Tracing for L2 Multicast

To identify a specific multicast receiver, specify the MAC address of the joiner or the receiver client, Group Multicast IP address and Snooping VLAN. Additionally, enable the trace level for the debug. The debug level will provide detailed traces and better visibility into the system.

debug platform condition feature multicast controlplane mac client-mac-addr ip group-ip-addr vlan id level debug level

Recommended workflow for trace files

The recommended workflow for trace files is listed below:

  • To request the tracelogs for a specific time period, such as 1 day, use the command: Device# show logging process wncd to-file flash:wncd.txt .

  • The system generates a text file of the tracelogs in the location /flash:

  • Copy the file off the device so the tracelogs can be used to work offline. For more details on copying files, see section below.

  • Delete the tracelog file (.txt) from the /flash: location to ensure enough space on the device for other operations.

Copy tracefiles off the box

This is an example of the tracefile:


Device# dir crashinfo:/tracelogs
Directory of crashinfo:/tracelogs/
50664 -rwx 760 Sep 22 2015 11:12:21 +00:00 plogd_F0-0.bin_0.gz
50603 -rwx 991 Sep 22 2015 11:12:08 +00:00 fed_pmanlog_F0-0.bin_0.9558.20150922111208.gz
50610 -rw- 11 Nov 2 2015 00:15:59 +00:00 timestamp
50611 -rwx 1443 Sep 22 2015 11:11:31 +00:00 auto_upgrade_client_sh_pmanlog_R0-.bin_0.3817.20150922111130.gz
50669 -rwx 589 Sep 30 2015 03:59:04 +00:00 cfgwr-8021_R0-0.bin_0.gz
50612 -rwx 1136 Sep 22 2015 11:11:46 +00:00 reflector_803_R0-0.bin_0.1312.20150922111116.gz
                50794 -rwx 4239 Nov 2 2015 00:04:32 +00:00 IOSRP_R0-0.bin_0.14239.20151101234827.gz
50615 -rwx 131072 Nov 2 2015 00:19:59 +00:00 linux_iosd_image_pmanlog_R0-0.bin_0

The trace files can be copied using one of the various options shown below:


Device# copy crashinfo:/tracelogs ?
  crashinfo: Copy to crashinfo: file system
  flash: Copy to flash: file system
  ftp: Copy to ftp: file system
  http: Copy to http: file system
  https: Copy to https: file system
  null: Copy to null: file system
  nvram: Copy to nvram: file system
  rcp: Copy to rcp: file system
  running-config Update (merge with) current system configuration
  scp: Copy to scp: file system
  startup-config Copy to startup configuration
  syslog: Copy to syslog: file system
  system: Copy to system: file system
  tftp: Copy to tftp: file system
  tmpsys: Copy to tmpsys: file system

The general syntax for copying onto a TFTP server is as follows:


Device# copy source: tftp:
Device# copy crashinfo:/tracelogs/IOSRP_R0-0.bin_0.14239.20151101234827.gz tftp:
Address or name of remote host []? 2.2.2.2
Destination filename [IOSRP_R0-0.bin_0.14239.20151101234827.gz]?

Note
Clear the generated report and archive files from the switch to ensure that sufficient flash space remains available for tracelog and other uses.

Configuration examples for conditional debugging

This is an output example of the show platform condition command. 

Device# show platform condition
Conditional Debug Global State: Stop
Conditions Direction
----------------------------------------------------------------------------------------------|---------
MAC Address 0024.D7C7.0054 N/A
Feature Condition Type Value
-----------------------|-----------------------|--------------------------------

This is an output example of the show debug command. 

Device# show debug
IOSXE Conditional Debug Configs:
Conditional Debug Global State: Start
Conditions Direction
----------------------------------------------------------------------------------------------|---------
MAC Address 0024.D7C7.0054 N/A
Feature Condition Type Value
-----------------------|-----------------------|--------------------------------
Packet Infra debugs:
Ip Address Port
------------------------------------------------------|----------

Verify conditional debugging

The table shown below lists the various commands that can be used to verify conditional debugging:

Command

Purpose

show platform condition

Displays the current conditions set.

show debug

Displays the current debug conditions set.

show platform software trace filter-binary

Displays logs merged from the latest tracefile.

request platform software trace filter-binary

Displays historical logs of merged tracefiles on the system.

Example: verify radioactive tracing log for SISF

This is an output example of the show platform software trace message ios chassis active R0 | inc sisf command. 

Device# show platform software trace message ios chassis active R0 | inc sisf
2017/10/26 13:46:22.104 {IOSRP_R0-0}{1}: [parser]: [5437]:  UUID: 0, ra: 0 (note):  CMD: 'show platform software trace message ios switch active R0 | inc sisf' 13:46:22 UTC Thu Oct 26 2017
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):   FF8E802918 semaphore system unlocked
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):  Unlocking, count is now 0
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):   FF8E802918 semaphore system unlocked
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):  Unlocking, count is now 1
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):  Gi1/0/5 vlan 10 aaaa.bbbb.cccc Setting State to 2
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):  Gi1/0/5 vlan 10 aaaa.bbbb.cccc Start timer 0
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):  Gi1/0/5 vlan 10 aaaa.bbbb.cccc  Timer value/granularity for 0 :299998/1000
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):  Gi1/0/5 vlan 10 aaaa.bbbb.cccc Updated Mac Timer : 299998 
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):  Gi1/0/5 vlan 10 aaaa.bbbb.cccc Before Timer :  350000 
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):  Gi1/0/5 vlan 10 aaaa.bbbb.cccc Timer 0, default value is 350000
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):   Allocating timer wheel for 0
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):  Gi1/0/5 vlan 10 aaaa.bbbb.cccc No timer running
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):  Granularity for timer MAC_T1  is 1000
2017/10/26 13:46:10.667 {IOSRP_R0-0}{1}: [sisf]: [5437]:  UUID: 4800000000060, ra: 7 (debug):  Gi1/0/5 vlan 10 aaaa.bbbb.cccc Current State :MAC-STALE, Req Timer : MAC_T1  Current Timer MAC_T1

Packet tracing

A packet tracing is a network diagnostic tool that

  • tracks the flow and processing of individual data packets through a network device

  • provides detailed per-packet information based on user-defined conditions for troubleshooting purposes, and

  • helps identify network issues such as misconfiguration, capacity overload, and software bugs.

The packet tracing feature explains how to perform data plane packet tracing for Cisco Catalyst 9800 Series Wireless Controller for Cloud software.

To trace packets on the controller:

  1. Enable conditional debugging on selected packets or traffic to trace.

  2. Enable packet tracing (per-AP or per-Client).


Note

Use per-AP conditional debugging with a MAC address filter when the AP and controllers are in the same VLAN. If they are in different VLANs, per-AP packet tracing with a MAC address does not capture packets because the MAC address varies.

Limitation of Conditional Debugging Packet Tracing

The MAC or IP filter applies only to the outer Ethernet or IP header. If a packet is CAPWAP encapsulated, the MAC or IP filter does not affect the inner 802.11 MAC or IP header.

Configure conditional debugging packet tracing (CLI)

Enable and manage in-depth packet tracing for troubleshooting and diagnostics while minimizing resource usage using commands.

Procedure

Step 1

Enable the privileged EXEC mode.

Example:

Device# enable

Enter your password, if prompted.

Step 2

Configure packet tracing to capture the last set of packets.

Example:

Device# debug platform packet-trace packet 8192 circular fia-trace data-size 2048

packet-count: Here, the valid range is from 16 to 8192.

data-size: Here, the valid range is from 2048 to 16384 bytes.

Step 3

Configure packet tracing for a copy of packet data.

Example:

Device# debug platform packet-trace copy packet both size 2048

packet-size: Here, the valid range is from 16 to 2048 bytes.

Step 4

Enable conditional debugging for an interface, MAC, or IP filter.

Example:

Enable conditional debugging for TenGigabitEthernet 0/0/0 and match packets whose source and destination MAC is 192.0.2.1:

Device# debug platform condition interface {interface-name TenGigabitEthernet | cpp} 0/0/0 {mac | ipv4 | match} 192.0.2.1 {both | ingress | egress}

An interface refers to any physical port, port channel, internal vlan, SVI, or wireless client.

Step 5

Start conditional debugging packet tracing.

Example:

Device# debug platform condition start

Step 6

Stop conditional debugging packet tracing.

Example:

Device# debug platform condition stop

Step 7

Redirect all traced packets to bootflash.

Example:

Device# show platform hardware chassis active qfp feature packet-trace packet all | redirect bootflash:packet_trace.txt

Convert the packet_trace.txt to pcap and download the pcap files. You can do so using the following link:

http://wwwin-dharton-dev.cisco.com/pactrac2pcap.html

Configure conditional debugging packet tracing per AP (CLI)

Enable conditional debugging to trace CAPWAP packets for a specific AP by filtering with MAC or IP address using commands.

Procedure

Step 1

Enable privileged EXEC mode.

Example:

Device# enable

Enter your password, if prompted.

Step 2

Enable conditional debugging with MAC filter.

Example:

Device# debug platform condition interface {interface-name TenGigabitEthernet | cpp} 0/0/0 {mac 192.0.2.1 | access-list acl-name | ipv4 | match} {both | ingress | egress}

Here, the CLI matches the packets whose source or destination MAC address is 192.0.2.1

Step 3

Enable conditional debugging with inline MAC ACL.

Example:

Device# debug platform condition interface TenGigabitEthernet 0/0/0 match mac { H.H.H. | any | host} 192.0.2.1 {both | ingress | egress}

Step 4

Enable conditional debugging with IP filter.

Example:

Device# debug platform condition interface TenGigabitEthernet 0/0/0 ipv4 {192.0.2.1 | access-list ip-acl-name| both | egress | ingress} {both | egress | ingress}

0/0/0: is the GigabitEthernet interface number. The valid range is from 1 to 32.

Configure conditional debugging packet tracing per client (GUI)

Enable targeted packet tracing for individual clients to facilitate in-depth troubleshooting and analysis using GUI.

Procedure

Step 1

Choose Troubleshooting > Radioactive Trace.

Step 2

Click Add.

Step 3

In the Add MAC/IP Address window, enter the MAC/IP Address.

Step 4

Click Apply to Device.

Configure conditional debugging packet tracing per client (CLI)

Enable conditional debugging to trace packets for a specific wireless client, helping to diagnose network issues more efficiently using commands.

Procedure

Step 1

Enable privileged EXEC mode.

Example:

Device# enable

Enter your password, if prompted.

Step 2

Enable conditional debugging for a wireless client interface.

Example:

Device# debug platform condition interface {intf-name | cpp 0xa0000001} { mac | ipv4 | match [ipv4 | ipv6 | mac]} protocol icmp host 192.0.2.1 host 192.0.2.1 {both | ingress | egress}

cpp-handle-index: The valid range is from 1 to 4294967295.

Verify conditional debugging packet tracing configuration

To view the summary of the traced packet, use this command:

Device# show platform packet-trace summary

To view a specific traced packet, use this command:

Device# show platform packet-trace packet packet-number

To view the wireless client interface handle, use this command:

Device# show platform hardware chassis active qfp feature wireless wlclient cpp-client mac-address client-mac details
Device# show platform hardware chassis active qfp feature wireless wlclient cpp-client mac-address 8825.93b0.b51f details 
Client Details for client cpp_if_handle: 0x34
Name : WLCLIENT-IF-0x00a0000001
Mac Addr : 8825.93b0.b51f
pal_if_handle : 0xa0000001
Mobility State : LOCAL
Multicast Action : FORWARD
Auth State : RUN