Use the Ethernet Packet Capture Feature to Troubleshoot High CPU Utilization

Available Languages

Updated:June 14, 2013
Document ID:116139

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

This document describes the use the Ethernet Packet Capture (EPC) feature in order to capture packets that are process-switched, generated locally, or Cisco Express Forwarding (CEF)-punted. The CPU inband Switch Port Analyzer (SPAN) capture is not supported on Supervisor Engine 2T (Sup2T).

Note: The EPC feature on Sup2T cannot capture traffic that is hardware switched. In order to capture hardware switched packets, the Mini Protocol Analyzer feature should be used. Refer to the Mini Protocol Analyzer section of the Catalyst 6500 Release 12.2SX Software Configuration Guide for more information.

Prerequisites

Requirements

Cisco recommends that you have knowledge of the EPC feature and high CPU utilization due to interrupts on Catalyst 6500 Series switches.

Components Used

The information in this document is based on the Cisco Catalyst 6500 Series switch run on a Sup2T.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Initial Configuration

Here is the initial configuration.

6500#monitor capture buffer CAP_BUFFER
! Create a capture buffer

6500#monitor capture point ip cef CEF_PUNT punt
! Create capture point for cef punted traffic

6500#monitor capture point ip process-switched PROCESS_SW both
! Create capture point for process switched traffic

6500#monitor capture point ip process-switched LOCAL_TRAFFIC from-us
! Create capture point for locally generated traffic

6500#monitor capture point associate PROCESS_SW CAP_BUFFER
6500#monitor capture point associate LOCAL_TRAFFIC CAP_BUFFER
6500#monitor capture point associate CEF_PUNT CAP_BUFFER
! Associate capture points to capture buffer

6500#monitor cap buffer CAP_BUFFER  size 128 
! Set packet dump buffer size (in Kbytes)

6500#monitor cap buffer CAP_BUFFER  max-size 512  
! Set element size in bytes : 1024 bytes or less (default is 68 bytes)

Configuration

Here is the configuration:

6500#show monitor capture buffer CAP_BUFFER parameters 

Capture buffer CAP_BUFFER (linear buffer)
Buffer Size : 131072 bytes, Max Element Size : 512 bytes, Packets : 0
Allow-nth-pak : 0, Duration : 0 (seconds), Max packets : 0, pps : 0
Associated Capture Points:
Name : PROCESS_SW, Status : Inactive
Name : LOCAL_TRAFFIC, Status : Inactive
Name : CEF_PUNT, Status : Inactive
Configuration:
monitor capture buffer CAP_BUFFER size 128 max-size 512 
monitor capture point associate PROCESS_SW CAP_BUFFER
monitor capture point associate LOCAL_TRAFFIC CAP_BUFFER
monitor capture point associate CEF_PUNT CAP_BUFFER

Captures of Process-Switched Data

Use this procedure in order to capture process-switched data:

  1. Start the capture point PROCESS_SW.
    6500#monitor capture point start PROCESS_SW
    *Jun  1 06:26:51.237: %BUFCAP-6-ENABLE: Capture Point PROCESS_SW enabled.
  2. Verify how fast the packet count increases.
    6500#show monitor capture buffer CAP_BUFFER parameters
    Capture buffer CAP_BUFFER (linear buffer)
    Buffer Size : 131072 bytes, Max Element Size : 512 bytes, Packets : 20
    Allow-nth-pak : 0, Duration : 0 (seconds), Max packets : 0, pps : 0
    Associated Capture Points:
    Name : PROCESS_SW, Status : Active
    Name : LOCAL_TRAFFIC, Status : Inactive
    Name : CEF_PUNT, Status : Inactive
    Configuration:
    monitor capture buffer CAP_BUFFER size 128 max-size 512 
    monitor capture point associate PROCESS_SW CAP_BUFFER
    monitor capture point associate LOCAL_TRAFFIC CAP_BUFFER
    monitor capture point associate CEF_PUNT CAP_BUFFER
  3. Inspect the captured packets in order to verify that they are legitimate packets for process-switching.
    6500#show monitor capture buffer CAP_BUFFER dump

    06:26:52.121 UTC Jun 1 2000 : IPv4 Process    : Gi1/3 None

    0F6FE920:          01005E00 00020000 0C07AC02      ..^.......,.
    0F6FE930: 080045C0 00300000 00000111 CCF70A02  ..E@.0......Lw..
    0F6FE940: 0202E000 000207C1 07C1001C 95F60000  ..`....A.A...v..
    0F6FE950: 10030A64 02006369 73636F00 00000A02  ...d..cisco.....
    0F6FE960: 020100                               ...             

    06:26:52.769 UTC Jun 1 2000 : IPv4 Process    : Gi1/3 None

    0F6FE920:          01005E00 000A0019 AAC0B84B      ..^.....*@8K
    0F6FE930: 080045C0 00420000 00000158 83E8AC10  ..E@.B.....X.h,.
    0F6FE940: A8A1E000 000A0205 EDEB0000 00000000  (!`.....mk......
    0F6FE950: 00000000 00000000 00CA0001 000C0100  .........J......
    0F6FE960: 01000000 000F0004 00080C02 01020006  ................
    0F6FE970: 0006000D 00                          .....        
    <snip>
  4. Stop the capture point and clear the buffer when you are finished with the capture.
    6500#monitor capture point stop PROCESS_SW
    *Jun  1 06:28:37.017: %BUFCAP-6-DISABLE: Capture Point PROCESS_SW disabled.
    6500#monitor capture buffer CAP_BUFFER clear

Captures of Locally-Generated Traffic

Use this procedure in order to capture locally-generated traffic:

  1. Start the capture point LOCAL_TRAFFIC.
    6500#monitor capture point start LOCAL_TRAFFIC  
    *Jun  1 06:29:17.597: %BUFCAP-6-ENABLE: Capture Point LOCAL_TRAFFIC enabled.
  2. Verify how fast the packet count increases.
    6500#show monitor capture buffer CAP_BUFFER parameters 
    Capture buffer CAP_BUFFER (linear buffer)
    Buffer Size : 131072 bytes, Max Element Size : 512 bytes, Packets : 5
    Allow-nth-pak : 0, Duration : 0 (seconds), Max packets : 0, pps : 0
    Associated Capture Points:
    Name : PROCESS_SW, Status : Inactive
    Name : LOCAL_TRAFFIC, Status : Active
    Name : CEF_PUNT, Status : Inactive
    Configuration:
    monitor capture buffer CAP_BUFFER size 128 max-size 512 
    monitor capture point associate PROCESS_SW CAP_BUFFER
    monitor capture point associate LOCAL_TRAFFIC CAP_BUFFER
    monitor capture point associate CEF_PUNT CAP_BUFFER
  3. Inspect the captured packets.

    The traffic found here is locally-generated by the switch. Some examples of traffic are control protocols, Internet Control Message Protocol (ICMP), and data from the switch.
    6500#show monitor capture buffer CAP_BUFFER dump 

    06:31:40.001 UTC Jun 1 2000 : IPv4 Process    : None Gi1/3

    5616A9A0: 00020000 03F42800 03800000 76000000  .....t(.....v...
    5616A9B0: 00000000 00000000 00000000 00000000  ................
    5616A9C0: 001D4571 AC412894 0FFDE940 08004500  ..Eq,A(..}i@..E.
    5616A9D0: 0064000A 0000FF01 29A8AC10 9215AC10  .d......)(,...,.
    5616A9E0: A7B00800 2F230002 00000000 00000239  '0../#.........9
    5616A9F0: 4CECABCD ABCDABCD ABCDABCD ABCDABCD  Ll+M+M+M+M+M+M+M
    5616AA00: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
    5616AA10: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
    5616AA20: ABCDABCD ABCDABCD ABCDABCD ABCDABCD  +M+M+M+M+M+M+M+M
    5616AA30: ABCD00                               +M.             
    <snip>
  4. Stop the capture point and clear the buffer when finished with the capture.
    6500#monitor capture point stop LOCAL_TRAFFIC
    *Jun  1 06:33:08.353: %BUFCAP-6-DISABLE: Capture Point LOCAL_TRAFFIC disabled.

    6500#monitor capture buffer CAP_BUFFER clear

Captures of CEF-Punted Traffic

Use this procedure in order to capture CEF-punted traffic:

  1. Start the capture point CEF_PUNT.
    6500#monitor capture point start CEF_PUNT
    *Jun  1 06:33:42.657: %BUFCAP-6-ENABLE: Capture Point CEF_PUNT enabled.
  2. Verify how fast the packet count increases.
    6500#show monitor capture buffer CAP_BUFFER parameters 

    Capture buffer CAP_BUFFER (linear buffer)
    Buffer Size : 131072 bytes, Max Element Size : 512 bytes, Packets : 8
    Allow-nth-pak : 0, Duration : 0 (seconds), Max packets : 0, pps : 0
    Associated Capture Points:
    Name : PROCESS_SW, Status : Inactive
    Name : LOCAL_TRAFFIC, Status : Inactive
    Name : CEF_PUNT, Status : Active
    Configuration:
    monitor capture buffer CAP_BUFFER size 128 max-size 512 
    monitor capture point associate PROCESS_SW CAP_BUFFER
    monitor capture point associate LOCAL_TRAFFIC CAP_BUFFER
    monitor capture point associate CEF_PUNT CAP_BUFFER
  3. Inspect the captured packets.

    Packets found here would be punted to the CPU because of punt adjacency programmed for the flow. Check the CEF adjacency and troubleshoot for the root cause.
    6504-E#show monitor capture buffer CAP_BUFFER dump    

    06:47:21.417 UTC Jun 1 2000 : IPv4 CEF Punt   : Gi1/1 None

    5616B090: 01005E00 000A0019 AAC0B846 080045C0  ..^.....*@8F..E@
    5616B0A0: 00420000 00000158 84E8AC10 A7A1E000  .B.....X.h,.'!`.
    5616B0B0: 000A0205 EDEB0000 00000000 00000000  ....mk..........
    5616B0C0: 00000000 00CA0001 000C0100 01000000  .....J..........
    5616B0D0: 000F0004 00080C02 01020006 0006000D  ................
    5616B0E0: 00                                   .               
    <snip>
  4. Filter the captured packets as needed.
    6500#show monitor capture buffer CAP_BUFFER dump filter input-interface gi1/3

    06:47:21.725 UTC Jun 1 2000 : IPv4 CEF Punt   : Gi1/3 None
    5607DCF0:          01005E00 0005001F 6C067102      ..^.....l.q.
    5607DD00: 080045C0 004CD399 00000159 F8F60A02  ..E@.LS....Yxv..
    5607DD10: 0202E000 00050201 002C0A02 02020000  ..`......,......
    5607DD20: 0001D495 00000000 00000000 0000FFFF  ..T.............
    5607DD30: FF00000A 12010000 00280A02 02020000  .........(......
    5607DD40: 0000FFF6 00030001 00040000 000100    ...v........... 

    06:47:22.837 UTC Jun 1 2000 : IPv4 CEF Punt   : Gi1/3 None
    5607DCF0:          01005E00 00020000 0C07AC02      ..^.......,.
    5607DD00: 080045C0 00300000 00000111 CCF70A02  ..E@.0......Lw..
    5607DD10: 0202E000 000207C1 07C1001C 95F60000  ..`....A.A...v..
    5607DD20: 10030A64 02006369 73636F00 00000A02  ...d..cisco.....
    5607DD30: 020100                               ...             
    <snip>
  5. Stop the capture point and clear the buffer when finished with the capture.
    6500#monitor capture point stop CEF_PUNT
    *Jun  1 06:36:01.285: %BUFCAP-6-DISABLE: Capture Point CEF_PUNT disabled.
    6500#monitor capture buffer CAP_BUFFER clear

Verify

Refer to the verification steps listed in the configuration processes in order to confirm that your configuration works properly.

Troubleshoot

There is currently no specific troubleshooting information available for this configuration.

Revision History

Revision Publish Date Comments
1.0
14-Jun-2013
Initial Release
TAC Authored

Contributed by Cisco Engineers

  • Shashank Singh
    Cisco TAC Engineer.

Was this Document Helpful?

FeedbackFeedback

Contact Cisco