Catalyst 6500 Series Switches Netdr Tool for CPU-Bound Packet Captures

Introduction

This document describes an available tool, Netdr, on Cisco Catalyst 6500 Series switches that run Supervisor Engines 720 or 32 that allows you to capture packets on the internal inband path to the Route Processor CPU (RP) or Switch Processor CPU (SP).

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on the Cisco Catalyst 6500 Series switches that run Supervisor Engine 720.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information

The RP CPU is typically used in order to handle Layer 3 (L3) control traffic as well as L3 data traffic that cannot be hardware-switched. Some examples of L3 control traffic are Open Shortest Path First (OSPF), Enhanced Interior Gateway Routing Protocol (EIGRP), Border Gateway Protocol (BGP), and Protocol Independent Multicast (PIM) packets. Some examples of L3 data traffic that cannot be hardware-switched are packets with IP options set, packets with Time To Live (TTL) values of 1, and packets that require fragmentation.

The SP CPU is typically used in order to handle Layer 2 (L2) control traffic. Some examples of this are Spanning Tree Protocol (STP), Cisco Discovery Protocol (CDP), and VLAN Trunking Protocol (VTP) packets.

The Netdr tool is used in order to capture both transmit (Tx) and receive (Rx) packets on the internal inband CPU software switching path. This tool cannot be used to capture traffic that is hardware-switched.

Netdr is helpful in attempts to troubleshoot high-CPU usage scenarios. In order to check how busy the RP CPU is, issue the show process cpu command or show process cpu history command. In order to check how busy the SP CPU is, issue the remote command switch show process cpu command or remote command switch show process cpu history command.

Netdr is useful only to troubleshoot interrupt-driven, high-CPU utilization. Interrupt-driven CPU utilization is the result of processing incoming packets sent to the CPU.

Cat6500#show process cpu

CPU utilization for five seconds: 90%/81%; one minute: 89%; five minutes: 80%

In the previous example:

• 90% is the total CPU utilization.
• 81% is the CPU utilization due to interrupts, which constitutes traffic processed by the CPU.
• 9% (90 - 81) is CPU utilization due to Cisco IOS^? software processes.

Use the Netdr Tool

This section describes how to use the Netdr tool.

Note: Netdr is safe for use in high-CPU usage conditions on newer Cisco IOS software versions, such as Version 12.2(33)SXH, and later. On a few old software releases, Netdr might utilize more CPU, and might be unsafe to run on a switch that already sees high-CPU utilization. If the switch runs an older software version, it is recommended to use this feature under the supervision of the Cisco Technical Assistance Center (TAC).

In order to capture packets on the RP inband CPU path, use this syntax:

Cat6500#debug netdr capture ?

acl                     (11) Capture packets matching an acl
and-filter               (3) Apply filters in an and function: all must match
continuous               (1) Capture packets continuously: cyclic overwrite
destination-ip-address  (10) Capture all packets matching ip dst address
dstindex                 (7) Capture all packets matching destination index
ethertype                (8) Capture all packets matching ethertype
interface                (4) Capture packets related to this interface
or-filter                (3) Apply filters in an or function: only one must match
rx                       (2) Capture incoming packets only
source-ip-address        (9) Capture all packets matching ip src address
srcindex                 (6) Capture all packets matching source index
tx                       (2) Capture outgoing packets only
vlan                     (5) Capture packets matching this vlan number

Note: Several options are available, and the numbers in parentheses to the right of each option indicate the order in which the options must be specified.

In order to capture packets on the SP inband CPU path, you must run all of the commands from the SP console.

Cat6500#remote login switch
Trying Switch ...
Entering CONSOLE for Switch
Type "^C^C^C" to end this session

Cat6500-sp#debug netdr capture ?

Note: Enter exit in order to return to the regular RP CPU command prompt.

Once the packets are captured, they are displayed with the show netdr capture command.

Options

Here are some of the available options for Netdr:

• When you use the continuous option, the switch has packets on the inband CPU path continuously fill the entire capture buffer (4096 packets), and begin to overwrite the buffer in a first-in, first-out (FIFO) fashion.
• The tx and rx options capture packets that come from the CPU and go to the CPU, respectively.
• The interface option is used in order to capture packets to or from the specified interface. The interface is either a switch virtual interface (SVI) or an L3 interface on the switch.
• The vlan option is used in order to capture all packets in the specified VLAN. The VLAN specified can be one of the internal VLANs associated with an L3 interface. The show vlan internal usage command is used in order to see the internal VLAN to L3 interface-mapping.
• LTL (local target logic) is an internal software representation of an interface. The src_indx (source index) and dst_indx (destination index) options are used in order to capture all packets that match the source LTL and destination LTL indices, respectively. Note that the interface option only allows the capture of packets to or from an L3 interface (SVI or physical). Use of the src_indx or dst_indx options allows the capture of Tx or Rx packets on an L2 interface. The src_indx and dst_indx options work with either L2 or L3 interface indices.

Troubleshoot

Note: Netdr is safe for use in high-CPU usage conditions on newer Cisco IOS software versions, such as Version 12.2(33)SXH, and later. On a few old software releases, Netdr might utilize more CPU and might be unsafe to run on a switch that already sees high-CPU utilization. If the switch runs an older software version, it is recommended to use this feature under the supervision of the Cisco TAC.

Complete these steps in order to troubleshoot with Netdr:

1. Start a Netdr capture for traffic that comes in the RP CPU:

   Cat6500#debug netdr capture rx

2. Display the captured packets:

   Cat6500#show netdr capture
   A total of 4096 packets have been captured
   The capture buffer wrapped 0 times
   Total capture capacity: 4096 packets
     ------- dump of incoming inband packet -------
   interface NULL, routine mistral_process_rx_packet_inlin, timestamp 06:35:39.498
   dbus info: src_vlan 0x3F1(1009), src_indx 0x102(258), len 0x40(64)
     bpdu 0, index_dir 1, flood 0, dont_lrn 1, dest_indx 0x387(903)
     05000018 03F16000 01020000 40000000 00117F00 00157F00 00100000 03870000 
   mistral hdr: req_token 0x0(0), src_index 0x102(258), rx_offset 0x76(118)
     requeue 0, obl_pkt 0, vlan 0x3F1(1009)
   destmac 00.1A.A2.2D.B3.A4, srcmac 00.00.00.00.AA.AA, protocol 0800
   protocol ip: version 0x04, hlen 0x05, tos 0x00, totlen 46, identifier 8207
     df 0, mf 0, fo 0, ttl 32, >src 127.0.0.16, dst 127.0.0.21
       udp src 68, dst 67 len 26 checksum 0xB8BC

3. Review the packets in order to identify the top talkers and trends. You can use the "| include" option in order to search based on fields such as Source MAC (srcmac) address, Destination MAC (destmac) address, Source and Destination (src & dst) IP addresses, and Source Index (src_indx). 

   Cat6500#show netdr capture | include srcmac
   destmac 00.1A.A2.2D.B3.A4, srcmac 00.00.00.00.AA.AA, protocol 0800
   destmac 00.1A.A2.2D.B3.A4, srcmac 00.00.00.00.AA.AA, protocol 0800
   destmac 00.1A.A2.2D.B3.A4, srcmac 00.00.00.00.AA.AA, protocol 0800
   destmac 00.1A.A2.2D.B3.A4, srcmac 00.00.00.00.AA.AA, protocol 0800
   destmac 00.1A.A2.2D.B3.A4, srcmac 00.00.00.00.AA.AA, protocol 86DD
   destmac 00.1A.A2.2D.B3.A4, srcmac 00.00.00.00.AA.AA, protocol 86DD
   destmac 00.1A.A2.2D.B3.A4, srcmac 00.00.00.00.AA.AA, protocol 86DD
   
   Cat6500#show netdr capture | inc src_indx 
   dbus info: src_vlan 0x3F1(1009), src_indx 0x102(258), len 0x40(64)
   dbus info: src_vlan 0x3F1(1009), src_indx 0x102(258), len 0x40(64)
   dbus info: src_vlan 0x3F1(1009), src_indx 0x102(258), len 0x40(64)
   dbus info: src_vlan 0x3F1(1009), src_indx 0x102(258), len 0x40(64)
   dbus info: src_vlan 0x3F1(1009), src_indx 0x102(258), len 0x54(84)
   dbus info: src_vlan 0x3F1(1009), src_indx 0x102(258), len 0x54(84)
   dbus info: src_vlan 0x3F1(1009), src_indx 0x102(258), len 0x54(84)

4. Decode the src_indx and dest_indx in order to discover the source and destination interfaces of the packet.   
   

   Cat6500#remote command switch test mcast ltl-info index 102
   
   index 0x102 contain ports 5/3
   ! This is the physical interface sourcing the packet going to the CPU.
   
   Cat6500#remote command switch test mcast ltl-info index 387
   
   index 0x387 contain ports 5/R 
   !5/R refers to RP CPU on the supervisor engine in slot 5