Guest

Cisco Catalyst 2900 XL Series Switches

High CPU Utilization on Catalyst 2900XL/3500XL Switches

Document ID: 10557

Updated: Aug 30, 2005

   Print

Introduction

After issuing a show processes cpu command, the Catalyst fixed configuration switches 2900XL and 3500XL report a high value for CPU utilization at idle. With other Cisco devices, high CPU utilization at idle is cause for concern; however, it is normal for the 2900XL and 3500XL.

This document explains the reason for the high CPU utilization value and gives some suggestions for monitoring and troubleshooting excessively high CPU utilization.

Before You Begin

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Prerequisites

There are no specific prerequisites for this document.

Components Used

This document is not restricted to specific software and hardware versions.

Background

The 2900XL/3500XL switches have two separate hardware subsystems, the switching subsystem and the internal CPU engine. Each is described below:

  • The switching subsystem controls the receiving and forwarding of data traffic.

  • The internal CPU engine implements higher-level protocols to control and monitor switch behavior.

As a result, 2900XL/3500XL CPU utilization does not reflect:

  • The number of packets being switched.

  • The total load on the switch.

CPU utilization does reflect the switch CPU processing, which includes the following:

  • spanning-tree processing

  • port status

  • LED indicators

In comparison, Cisco routers such as the Cisco 3600 process and route packets using software; therefore, the output of the show processes cpu command provides insight into the overall load on the router. The CPU utilization is linear for the amount of packet processing and routing being performed on the Cisco 3600 router.

The common observation is that the CPU utilization output for the show processes cpu command on the 2900XL/3500XL switch shows a fairly high value when the switch is idle and has no active ports. When viewing the output of the show processes cpu command, the highest percentage processes are Port Status Proc (Port Status Process) and LED Control Proc (LED Control Process). It is normal behavior for these processes to be a high percentage of the total CPU utilization. These processes run constantly to maintain and monitor the status of the ports and the different LEDs available on the panel of the switch.

Baseline CPU Utilization

The table below summarizes typical CPU utilization seen by 3500XL switches at idle and under load. It is noteworthy that CPU utilization does not increase linearly with increased load.

Switch CPU Utilization at IDLE CPU Utilization at 50 Percent Write Speed Using Unicast Traffic
3512 XL 5 seconds (32 percent/2 percent) 1 minute (32 percent) 5 minutes (32 percent) 5 seconds (32 percent/2 percent) 1 minute (33 percent) 5 minutes (32 percent)
3548 XL 5 seconds (44 percent/6 percent) 1 minute (44 percent) 5 minutes (32 percent) 5 seconds (46 percent/7 percent) 1 minute (33 percent) 5 minutes (33 percent)

Therefore, it is normal for a 2900XL or 3500XL switch to indicate a CPU utilization value of 30 percent to 50 percent, even under minimal load.

Troubleshoot High CPU Utilization

The CPU is responsible for the following:

  • processing broadcasts

  • spanning-tree Bridge Protocol Data Units (BPDUs)

  • address learning

  • port status

  • LED operations

The processes listed above may contribute to excessive CPU utilization. If the CPU utilization is extremely high (around 80 percent to 99 percent), or if the CPU utilization is higher than normal, follow these recommendations.

  1. Create a baseline CPU utilization at no load.

  2. Note which processes are generating the most load.

  3. Check for consistent MAC address clearing or spanning-tree instability. Use the show spanning-tree vlan vlan-id command to gather this information.

  4. Check for network broadcast storm.

  5. Create a Service Request with Cisco Technical Support, if necessary.

Check Utilization of 2900XL and 3500XL Switches

The show controller switch command provides indicative information regarding the total switch utilization. This is an example:

Switch#show controller switch
Switch registers:

Device Type : 0x00040273
Congestion Threshold : 0x00000E95
Peak Total Allocation : 0x0000001A
Total Allocation : 0x00000000
Peak Total Bandwidth : 0x00000020
Total Bandwidth : 0x00000000
Total Bandwidth Limit : 0x000003DE
Lower Bandwidth Limit : 0x000003DE
Switch Mode : 0x00040000

Switch#

The Total Bandwidth Limit varies between different 2900XL and 3500XL models. When the Total Bandwidth reaches the Total Bandwidth Limit value, the switch has reached its full bandwidth capacity and begins to drop packets. The Peak Total Bandwidth is the highest value attained by the Total Bandwidth since the last time the show controller switch command was executed. Note, the values for the above parameters are in hexadecimal.

The Congestion Threshold value is used as conservative value for the maximum global buffer utilization. When the buffer utilization noted by Total Allocation reaches this value, the switch may drop frames. The Peak Total Allocation value shows the highest value attained by the Total Allocation since the last time the show controller switch command was executed. It is possible for the Peak Total Allocation and/or the Total Allocation to be greater than Congestion Threshold. If the Total Allocation reaches or is over the Congestion Threshold amount, the switch is experiencing considerable network activity near its full capacity.

The global buffer utilization may be adversely affected by several configuration issues, described here:

  1. Speed mismatch between an ingress and egress port; for example, several 100 Mb clients transferring files to a server connected to the switch at 10 Mb, half-duplex.

  2. Multiple input ports feeding a single output port.

  3. Duplex mismatch on multiple ports.

  4. Numerous ports that are experiencing collisions and/or output errors due to half-duplex configuration or oversubscription of a slow link.

Sample show processes cpu Command Output

Below is sample output from the show processes cpu command.

Switch#show processes cpu
CPU utilization for five seconds 35%/4%; one minute 35%; five minutes 35%
 PID  Runtime(ms)  Invoked  uSecs    5Sec   1Min   5Min TTY Process
   1          15     44349      0   0.00%  0.00%  0.00%   0 Load Meter
   2          18        31    580   0.24%  0.02%  0.00%   0 Exec
   3       90945     22514   4039   0.00%  0.01%  0.00%   0 Check heaps
   4           3         1   3000   0.00%  0.00%  0.00%   0 Chunk Manager
   5           2         3    666   0.00%  0.00%  0.00%   0 Pool Manager
   6           0         2      0   0.00%  0.00%  0.00%   0 Timers
   7           0         1      0   0.00%  0.00%  0.00%   0 Entity MIB API
   8         451      3742    120   0.00%  0.00%  0.00%   0 ARP Input
   9           0         1      0   0.00%  0.00%  0.00%   0 RAM Access (dm 0
  10           0         1      0   0.00%  0.00%  0.00%   0 Critical Bkgnd
  11        1202     89758     13   0.00%  0.00%  0.00%   0 Net Background
  12           0        53      0   0.00%  0.00%  0.00%   0 Logger
  13        2410    221081     10   0.00%  0.00%  0.00%   0 TTY Background
  14      352967    767348    459   0.04%  0.09%  0.08%   0 Per-Second Jobs
  15         708      9223     76   0.00%  0.00%  0.00%   0 Net Input
  16          70     44350      1   0.00%  0.00%  0.00%   0 Compute load avg
  17       37697      3697  10196   0.00%  0.01%  0.00%   0 Per-minute Jobs
  18      877166  17911145     48   0.32%  0.21%  0.20%   0 LED Control Proc
  19       11915    221085     53   0.00%  0.00%  0.00%   0 Frank Aging
  20    23499061  26508015    886   5.73%  5.54%  5.55%   0 Port Status Proc
  21           0         1      0   0.00%  0.00%  0.00%   0 VM Prune Events
  23     1554627    741189   2097   0.32%  0.41%  0.43%   0 GDS Frame Ager
  24           0         1      0   0.00%  0.00%  0.00%   0 RAM Access (gi0/
  25           0         1      0   0.00%  0.00%  0.00%   0 IP NAT Ager
  28           0         1      0   0.00%  0.00%  0.00%   0 RAM Access (gi0/
  30     2568178   5890745    435   0.65%  0.73%  0.74%   0 Broadcast Storm
  31           0         1      0   0.00%  0.00%  0.00%   0 Port Group Chang
  32        2265      5952    380   0.00%  0.00%  0.00%   0 IP Input
  33         627       392   1599   0.00%  0.00%  0.00%   0 Address Learning
  34           0         1      0   0.00%  0.00%  0.00%   0 RAM Access (dm 1
  35           0         1      0   0.00%  0.00%  0.00%   0 RAM Access (dm 2
  36      369323    663847    556   0.00%  0.00%  0.00%   0 Enet Aging
  37        3073     26588    115   0.00%  0.00%  0.00%   0 CDP Protocol
  38         805        70  11500   0.00%  0.00%  0.00%   0 Address Deletion
  39          71        35   2028   0.00%  0.00%  0.00%   0 Switch CGMP Prot
  40           0         1      0   0.00%  0.00%  0.00%   0 TCP Timer
  41           0         1      0   0.00%  0.00%  0.00%   0 TCP Protocols
  42           0         1      0   0.00%  0.00%  0.00%   0 Socket Timers
  43           0         1      0   0.00%  0.00%  0.00%   0 HTTP Timer
  44           0         1      0   0.00%  0.00%  0.00%   0 CGMP Forwarding
  45         121     22176      5   0.00%  0.00%  0.00%   0 Cluster RARP
  46         184     27803      6   0.00%  0.00%  0.00%   0 Cluster Base
  47         105       113    929   0.00%  0.00%  0.00%   0 Spanning Tree
  48        6438     36032    178   0.00%  0.00%  0.00%   0 STP Hello
  49      283088    128597   2201   0.00%  0.00%  0.00%   0 STP Queue Handle
  50         172        93   1849   0.00%  0.00%  0.00%   0 Malibu STP Adjus
  51           0         1      0   0.00%  0.00%  0.00%   0 Time Range Proce
  52           0         1      0   0.00%  0.00%  0.00%   0 SNMP ConfCopyPro
  53           0         2      0   0.00%  0.00%  0.00%   0 Bridge MIB traps

Known Issues

High CPU Utilization Due to Autoconf Process

If you are experiencing high CPU utilization (up to 99 percent) and the show processes cpu command output indicates a large portion of this as the Autoconf process, you are probably hitting a known bug. This problem is filed against Cisco IOS Software Release 12.0(5.4)WC1 and earlier.

The problem occurs when you boot a switch which is configured with VLAN 1 interface in the administratively down state without an IP address, and another VLAN interface exists with an IP address that is administratively up. With this configuration, when the switch boots up, it finds that there is no IP address for VLAN 1 and starts the DHCP-based auto-configuration. This calls a function in the software which checks for interface VLAN 1 to be up before starting the DHCP, and hence comes up in an infinite loop. This results in the high CPU utilization.

The workaround is to configure the following commands, save the configuration, and reload the switch:

  • interface vlan 1

  • no ip address

Note: If you have any other interface VLANs without IP addresses, you will have to follow the same procedure for all those VLANs.

The following command shows the sample output if the switch is hitting this bug:

Switch#show processes cpu
CPU utilization for five seconds: 99%/7%; one minute: 99%; five minutes: 99%
PID  Runtime(ms)  Invoked  uSecs    5Sec   1Min   5Min TTY Process
   1        4034    538437      7   0.00%  0.00%  0.00%   0 Load Meter
   2           0         1      0   0.00%  0.00%  0.00%   0 IP NAT Ager
!--- Output suppressed.

  62  1888112123 117660129  16047  24.77% 49.89% 51.43%   0 Router Autoconf  
  63       12583      1254  10034   0.00%  0.00%  0.00%   0 SNMP ConfCopyPro
  64          20         9   2222   0.00%  0.00%  0.00%   0 Bridge MIB traps
  66      592259    538607   1099   0.00%  0.00%  0.00%   0 Runtime diags
  67           0         1      0   0.00%  0.00%  0.00%   0 SNMP Timers
!--- Output suppressed.

Switch#

The fix for this Autoconf process high CPU issue has been integrated in code Cisco IOS Software Release 12.0(5)WC2 and later. Refer to Cisco bug ID CSCdv21552 (registered customers only) in the Bug Toolkit (registered customers only) .

High CPU Utilization Due to Excess STP Instances

You create a Spanning Tree Protocol (STP) instance when you assign an interface to a VLAN. The STP instance is removed when the last interface is moved to another VLAN. Each VLAN is a separate STP instance. If you have already used up all available STP instances on a switch, adding another VLAN anywhere in the VLAN Trunking Protocol (VTP) domain creates a VLAN that does not run STP on that switch. For example, if 250 VLANs are defined in the VTP domain, you can enable STP on 64 of those VLANs. The rest of the VLANs must operate with STP disabled. There is a limitation on the number of instances of STP that a switch can handle.

The Catalyst 2912 XL, Catalyst 2924 XL, and Catalyst 2924C XL support only 64 STP instances and 64 VLANs. All other Catalyst 2900 XL switches and all Catalyst 3500 XL switches support 64 STP instances and 250 VLANs. Ensure that the total number of logical ports across all instances of STP for different VLANs does not exceed the maximum number supported. If you exceed these STP instance recommendations, the switch exhibits high CPU utilization. The Spanning Tree Algorithm (STA) is generally not processor intensive and has priority over other processes. You can see the CPU usage for the Spanning Tree process from the show processes cpu command output.

Switch#show processes cpu
CPU utilization for five seconds: 98%/10%; one minute: 96%; five minutes: 96%
PID  Runtime(ms)  Invoked  uSecs    5Sec   1Min   5Min TTY Process
  1         111    411944      0   0.00%  0.00%  0.00%   0 Load Meter
  2        8251      2638   3127   0.90%  0.07%  0.01%   0 Exec
  3      753782    209157   3603   0.00%  0.01%  0.00%   0 Check heaps
  4           0         1      0   0.00%  0.00%  0.00%   0 Chunk Manager
  
!--- Output suppressed

 54        9862    282046     34   0.00%  0.00%  0.00%   0 Cluster Base
 55        2050    959972   3791  32.63% 30.03% 17.35%   0 Spanning Tree
 56      144660   6673996     21   0.00%  0.00%  0.00%   0 STP Hello
 57           0        14      0   0.00%  0.00%  0.00%   0 STP Queue Handle
 58         167       193    865   0.00%  0.00%  0.00%   0 Malibu STP Adjus
 
!--- Output suppressed

Issue the show spanning-tree summary command to verify the number of logical ports or interfaces per VLAN in the STP Active column.

Switch#show spanning-tree summary

UplinkFast is disabled

Name                 Blocking Listening Learning Forwarding STP Active
-------------------- -------- --------- -------- ---------- ----------
VLAN1                19       0         0        7          26        
VLAN3                19       0         0        5          24        
VLAN4                19       0         0        5          24        
VLAN5                19       0         0        5          24        
VLAN6                19       0         0        5          24        
VLAN7                19       0         0        5          24        
VLAN9                19       0         0        5          24        
VLAN10               19       0         0        5          24        
VLAN11               19       0         0        5          24        
VLAN12               19       0         0        5          24        
VLAN20               19       0         0        7          26        
VLAN21               19       0         0        5          24

!--- Output suppressed

Name                 Blocking Listening Learning Forwarding STP Active
-------------------- -------- --------- -------- ---------- ----------
VLAN111              19       0         0        5          24        
VLAN200              19       0         0        5          24        
VLAN222              19       0         0        5          24        
VLAN900              19       0         0        5          24        
-------------------- -------- --------- -------- ---------- ----------
            64 VLANs 1217     0         0        329        1546

Related Information

Updated: Aug 30, 2005
Document ID: 10557