Guest

IP Application Services

IP Simple Network Management Protocol (SNMP) Causes High CPU Utilization

Document ID: 7270

Updated: Jun 28, 2006

   Print

Introduction

This document explains how to troubleshoot high CPU utilization in a router due to the SNMP ENGINE process running in the router, especially in low end routers.

Prerequisites

Requirements

Readers of this document should have knowledge of these topics:

  • SNMP

  • Cisco IOS

Components Used

The information in this document is based on routers running Cisco IOS® Software.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

High CPU Utilization Caused by the SNMP ENGINE Process

Sometimes, messages like this might appear in the router console:

%SNMP-3-CPUHOG: Processing [chars] of [chars]

They mean that the SNMP agent on the device has taken too much time to process a request.

You can determine the cause of high CPU use in a router by using the output of the show process cpu command.

The following example shows the output of the show process cpu command.

cacuk#show process cpu 
CPU utilization for five seconds: 0%/0%; 
one minute: 0%;
five minutes: 0% 
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 68 258816 0 0.00% 0.00% 0.00% 0 Load Meter
2 0 1 0 0.00% 0.00% 0.00% 0 OSPF Hello
3 788132 131480 5994 0.00% 0.00% 0.00% 0 Check heaps
4 0 1 0 0.00% 0.00% 0.00% 0 Chunk Manager
5 56 131 427 0.00% 0.00% 0.00% 0 Pool Manager
69 202700 421730 480 0.00% 0.01% 0.00% 0 IP SNMP
71 1193648 211250 5650 0.00% 0.19% 0.15% 0 SNMP ENGINE

The debug snmp command checks which Object IDs (OIDs) or Management Information Base (MIBs) are being queried at the time.

Note: Running a debug over a production network may overwhelm the router.

There are two probable causes for high CPU utilization occuring in relation to Simple Network Management Protocol (SNMP):

  1. Large route and/or ARP tables being polled by the NMS station

  2. Polling certain MIBs

Large Route and/or ARP Tables Polled by the NMS Station

The Network Management station queries routers for their entire route table to learn about other networks. It uses this information to find other routers and query them about their knowledge of networks around them. In this fashion, the management station can learn the topology of the entire network.

The router stores the route table in a hashed format, more conducive to quick route searches. However, SNMP responses for the route are required to be returned in lexicographical order per RFC1213. Therefore, for each SNMP request the router receives, the hash table must be sorted lexicographically before a SNMP response PDU can be built. The larger the route table, the more CPU intesive the sort.

SNMP is a low priority process as far as the CPU scheduler is concerned, so another process requiring CPU resources takes priority. Therefore, while CPU spikes occur in this scenario, they should not affect performance.

Recommendation 1: Use snmp-server view Commands

To avoid performance issues, force the router to prematurely end queries for the route table from the network management system server. Configure the router to respond with a complete message as soon as it receives the start of a request for the route table, as follows:

snmp-server view cutdown iso included

!--- ISO is used as a starting point as it is the snmp 'root'.

 
snmp-server view cutdown 1.3.6.1.2.1.4.21 excluded 
snmp-server view cutdown 1.3.6.1.2.1.4.22 excluded 
snmp-server view cutdown 1.3.6.1.2.1.3 excluded
snmp-server view cutdown 1.3.6.1.6.3.15 excluded

!--- The additional object snmpUsmMIB is excluded.

snmp-server view cutdown 1.3.6.1.6.3.16 excluded

!--- The additional object snmpVacmMIB is excluded.

snmp-server view cutdown 1.3.6.1.6.3.18 excluded

!--- The additional object snmpCommunityMIB is excluded.
 
snmp-server community public view cutdown RO
snmp-server community private view cutdown RW

caution Caution: In the above example, snmpUsmMIB is excluded, snmpVacmMIB is excluded, and snmpCommunityMIB is excluded. Together these objects can be used to get information about configured users and community strings to gain administrative access to the device. It is recommended that these objects be exlcluded from view on any device which can be accessed by public users.

This configuration blocks requests to retrieve the route table (ipRouteTable) and the Address Resolution Protocol (ARP) table (ipNetToMediaTable), but allows all other requests through. If you have an older versions of Cisco IOS® Software, it won't recognize the MIB object ipRouteTable, so use the following configuration instead:

snmp-server view cutdown iso included

!--- ISO is used as a starting point as it is the snmp 'root'.

snmp-server view cutdown 1.3.6.1.2.1.4.21 excluded 
snmp-server view cutdown 1.3.6.1.2.1.4.22 excluded 
snmp-server view cutdown at excluded
snmp-server view cutdown 1.3.6.1.6.3.15 excluded

!--- The additional object snmpUsmMIB is excluded.

snmp-server view cutdown 1.3.6.1.6.3.16 excluded

!--- The additional object snmpVacmMIB is excluded.

snmp-server view cutdown 1.3.6.1.6.3.18 excluded

!--- The additional object snmpCommunityMIB is excluded.
 
snmp-server community public view cutdown RO 
snmp-server community private view cutdown RW

caution Caution: In the above example, snmpUsmMIB is excluded, snmpVacmMIB is excluded, and snmpCommunityMIB is excluded. Together these objects can be used to get information about configured users and community strings to gain administrative access to the device. It is recommended that these objects be exlcluded from view on any device which can be accessed by public users.

In both of these examples, you can substitute your own community strings.

The outcome of these configurations is that the router no longer returns either the ARP table or the IP route table when queried. This prevents SNMP network discoveries from creating CPU spikes on the router in question, but it also removes a degree of manageability from the router.

Routers that previously were not exhibiting CPU spikes may begin to do so if there is a change in their routing table. The amount of cycles required to reply to the IP route table requests is a function of the number of routes in the routing table. If the number of routes increases, CPU utilization increases as well.

Recommendation 2: Turn on CEF

A change was made in the Cisco IOS code to allow SNMP to query the Cisco Express Forwarding (CEF) table for routing entries if CEF switching is used. This significantly improves the situation. With CEF enabled, the SNMP agent responds to a get-next/get-bulk operation for the routing or ARP tables with information from the Forwarding Information Base (FIB). The FIB is stored in lexicographical order and no sorting is needed. Without CEF enabled, the SNMP agent responds with information from the Routing Information Base (RIB), which must be sorted into lexicographical order causing high-CPU.

Follow the bug ID link below and see detailed bug information.

CSCdk54265 (registered customers only) - CPU HOG gets generated when polling routing table via SNMP from Network Management Station.

Polling Certain MIBs

Network Management Applications frequently retrieve information from devices using SNMP. One example would be Resource Manager Essentials (RME), an applications within the CiscoWorks 2000 suite of products. RME can retrieve flash information defined in the CISCO-FLASH-MIB. If the device has an ATA flash disk, it can cause CPU utilization to jump.

Follow the bug ID link below and see detailed bug information.

CSCdt97325 (registered customers only)

To verify and apply the workaround, follow these steps:

  1. Verify if the device has an ATA Flash Card in one of its slots:

    Router#show disk1: 
    ******** ATA Flash Card Geometry/Format Info ******** 
    
    ATA CARD GEOMETRY 
        Number of Heads:        12 
        Number of Cylinders     906 
        Sectors per Cylinder    63 
        Sector Size             512 
        Total Sectors           684936 
    
    ATA CARD FORMAT 
        Number of FAT Sectors  84 
        Sectors Per Cluster    32 
        Number of Clusters     21372 
        Number of Data Sectors 684117 
        Base Root Sector       169 
        Base FAT Sector        1 
        Base Data Sector       201 
    
    Router# 
    
    

    If you're not sure or if you do not get output for the show diskX command, run the show tech command and do a search for disk.

  2. Once you know you have an ATA Flash Disk, you can apply the following workaround to prevent the SNMP queries on the FLASH-MIB:

    Router(config)#snmp-server view <any_word> iso included
    
    !--- ISO is used as a starting point as it is the snmp 'root'.
     
    Router(config)#snmp-server view <any_word> ciscoFlashMIB excluded 
    
    !--- The additional object snmpUsmMIB is excluded.
    
    Router(config)#snmp-server view <any_word> 1.3.6.1.6.3.15 excluded
    
    !--- The additional object snmpVacmMIB is excluded.
    
    Router(config)#snmp-server view <any_word> 1.3.6.1.6.3.16 excluded
    
    !--- The additional object snmpCommunityMIB is excluded.
    
    Router(config)#snmp-server view <any_word> 1.3.6.1.6.3.18 excluded
    Router(config)#snmp-server community <read_community_string> view <any_word> ro 
    Router(config)#snmp-server community <write_community_string> view <any_word> rw 
    Router(config)#exit
    Router#

    Note: Choose the same word on every line for the <any_word> in the above configuration.

    caution Caution: In the above example, snmpUsmMIB is excluded, snmpVacmMIB is excluded, and snmpCommunityMIB is excluded. Together these objects can be used to get information about configured users and community strings to gain administrative access to the device. It is recommended that these objects be exlcluded from view on any device which can be accessed by public users.

  3. Verify that it is correctly implemented:

    Router#show run 
    ... 
    snmp-server view <any_word> iso included
    
    !--- ISO is used as a starting point as it is the snmp 'root'.
    
    snmp-server view <any_word> ciscoFlashMIB excluded
    snmp-server view <any_word> internet.6.3.15 excluded
    
    !--- The additional object snmpUsmMIB is excluded.
    
    snmp-server view <any_word> internet.6.3.16 excluded
    
    !--- The additional object snmpVacmMIB is excluded.
    
    snmp-server view <any_word> internet.6.3.18 excluded 
    
    !--- The additional object snmpCommunityMIB is excluded.
    
    snmp-server community <write_community_string> view <any_word> RW 
    snmp-server community <read_community_string> view <any_word> RO
    

    Note: <any_word>, <write_community_string>, and <read_community_string> are those that are in the configuration.

    caution Caution: In the above example, snmpUsmMIB is excluded, snmpVacmMIB is excluded, and snmpCommunityMIB is excluded. Together these objects can be used to get information about configured users and community strings to gain administrative access to the device. It is recommended that these objects be exlcluded from view on any device which can be accessed by public users.

  4. Issue the show proc cpu command to verify if the CPU-Util for SNMP goes down and look for the IP_SNMP process.

Note: The bug CSCdt97325 has also been fixed in some later Cisco IOS releases, so check the bug for any details.

Other bugs related to polling MIBs:

Follow the Cisco bug ID links below and see detailed bug information.

Related Information

Updated: Jun 28, 2006
Document ID: 7270