Contents

Introduction

Prerequisites

Requirements

Components Used

Troubleshoot Most Common Errors

Related Information

Introduction

This document describes the Simple Network Management Protocol (SNMP) and how to test its functionality on a device.

Prerequisites

Requirements

Cisco recommends that you have knowledge of SNMP protocol and its communications with the Network Management System (NMS) server.

Components Used

The information in this document is based on these software and hardware versions:

- SNMP

- Cisco WS-C3650-12X48UZ

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Troubleshoot Most Common Errors

1. Error Message: "%SNMP-3-RESPONSE_DELAYED: processing GetNext of "Any OID"."

GetNext of ciscoMgmt.810.1.2.1.1 (24004 msecs)

*May 24 01:30:48.463: %SNMP-3-RESPONSE_DELAYED: processing GetNext of ciscoMgmt.810.1.2.1.1 (24008 msecs)

---> In this scenario ciscoMgmt.810.1.2.1.1 is the OID causes the issue.

*May 24 01:31:12.477: %SNMP-3-RESPONSE_DELAYED: processing GetNext of ciscoMgmt.810.1.2.1.1 (24012 msecs)

*May 24 01:31:36.486: %SNMP-3-RESPONSE_DELAYED: processing GetNext of ciscoMgmt.810.1.2.1.1 (24008 msecs)

*May 24 01:32:00.503: %SNMP-3-RESPONSE_DELAYED: processing GetNext of ciscoMgmt.810.1.3.1.1 (24016 msecs)

*May 24 01:32:24.515: %SNMP-3-RESPONSE_DELAYED: processing GetNext of ciscoMgmt.810.1.3.1.1 (24012 msecs)

*May 24 01:32:48.528: %SNMP-3-RESPONSE_DELAYED: processing GetNext of ciscoMgmt.810.1.3.1.1 (24012 msecs)

*May 24 01:33:12.537: %SNMP-3-RESPONSE_DELAYED: processing GetNext of ciscoMgmt.810.1.3.1.1 (24008 msecs)

To troubleshoot:

Check SNMP configuration on the device. For SNMPv2, it needs to look like this:

snmp-server community TAC1 RO

snmp-server community TAC2 RO  --> If multiple communities are added to device.

For SNMPv3:

snmp-server view TESTV3 iso include

#snmp-server group TestGroupV3  v3 auth read TESTV3

#snmp-server user cisco TestGroupV3 v3 auth md5 ciscorules priv des56 cisco123

Enter the configuration mode of the device and add a view to the SNMP configuration to change it.

For SNMPv2:

snmp-server community  TAC1 RO view cutdown RO

snmp-server community TAC2 RO view cutdown RO

Some lines from the configuration mode:

snmp-server view cutdown iso included

snmp-server view cutdown ciscoMgmt.810 excluded -->>>

The Idea is to exclude the OID causes the issue, however,

please read out what is the function of the OID that that is excluded.

For SNMPv3:

#snmp-server view TESTV3 internet included

#snmp-server view TESTV3 ciscoMgmt.810  excluded

#snmp-server group  TestGroupV3   v3 priv write TESTV3

2. Error Message "High CPU Utilization due to SNMP Flash Cache".

#show processes cpu sorted

CPU utilization for five seconds: 99%/0%; one minute: 22%; five minutes: 18%

PID    Runtime(ms)     Invoked      uSecs      5Sec          1Min     5Min     TTY    Process

447    561399          143012       3925       0.00%         1.58%    1.83%     0     Snmp Flash Cache

SNMP Logs:

%SYS-2-SIGPENDING: Multiple Signals are sent to a process 91 -Process= "Snmp Flash Cache", ipl= 0, pid= 91.

888888888888888888888888888888888888888888888898878889

625424254283314655456532533533772205363424335694492379

100                                              *      *

90 * *    *  *    *** ***  *  *  **  * *      ***    **

80 ******************************************************

70 ******************************************************

60 ******************************************************

50 ******************************************************

40 ######################################################

30 ######################################################

20 ######################################################

10 ######################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

To work around:

The Flash MIB data collection process is disabled by default. If it is enabled with the use of snmp mib flash cache command (possibly after a reload), it can cause a high CPU in some cases.

Instead, use the  #no snmp mib flash cache  command in the configuration mode.

Or install this EEM script:

event manager applet SNMP authorization bypass

event syslog pattern "SYS-5-RESTART"

action 11 cli command "enable"

action 12 cli command "conf t"

action 13 cli command "no snmp mib flash cache"

action 14 cli command "end"

3. Error Message: "%SNMP-3-INPUT_QFULL_ERR:Packet dropped due to input queue full"

A possible reason for a queue full error can be heavy polling on the device or a specific OID that causes the issue. To mitigate this, first, check if the device is heavily polled.

In order to do so, execute this command:

B02#show snmp stats oid

time-stamp                                #of times requested            OID

15:40:19 BKK Dec 27 2019        11180008                           ifAlias

15:40:19 BKK Dec 27 2019        44018183                        dot1dBasePortEntry.4

15:40:19 BKK Dec 27 2019        44018212                        dot1dBasePortEntry.3

15:40:19 BKK Dec 27 2019        45216156                        ipNetToPhysicalEntry.4

15:40:19 BKK Dec 27 2019        44018059                        dot1dBasePortEntry.5

15:40:19 BKK Dec 27 2019        44578303                        dot1dBasePortEntry.1

15:40:19 BKK Dec 27 2019        6011756                        dot3StatsEntry.19

15:40:19 BKK Dec 27 2019        11095925                        ifSpeed

15:40:19 BKK Dec 27 2019        12879927                        dot1dTpFdbEntry.3

15:40:19 BKK Dec 27 2019        84535                              vmMembershipSummaryEntry.2

15:40:19 BKK Dec 27 2019        3241107                          vmMembershipSummaryEntry.3

15:40:19 BKK Dec 27 2019        45208908                        ipNetToMediaEntry.2

15:40:19 BKK Dec 27 2019        45223410                        ipNetToPhysicalEntry.6

15:40:19 BKK Dec 27 2019        44018324                        dot1dBasePortEntry.2

To troubleshoot:

You need to change the settings on your NMS and reduce the polling intervals for the device. Once the polling interval is reduced, queue full error must be mitigated. If not, then you need to check for the OID that causes the issue. To find the OID that causes the issue and to troubleshoot on the same, please refer to the previously mentioned Error Message 1.

4. Error Message: "High CPU utilization due to SNMP ENGINE".

Identify the issue:

The router suffers a high CPU at the time it is polled by a client, and this can be checked with the  #show process cpu <sorted>  command at the time of the high CPU. You can see that the SNMP Engine process takes all the CPU resources:

#show processes cpu sorted

CPU utilization for five seconds: 99%/0%; one minute: 22%; five minutes: 18%

PID    Runtime(ms)      Invoked      uSecs   5Sec          1Min              5Min    TTY    Process

189     1535478456      697105815    2202    88.15%        13.40%            8.74%    0     SNMP ENGINE

The problematic OID causes the high CPU to be slower than the others, which could also cause some timeout when the client requests this OID. Most of the methods attempt to find the OID that provides a slower answer. That is because they are the most likely to cause the high CPU. Once the OID is identified, you can lock that respective OID in order to mitigate the errors.

Note: If none of the methods listed here help to identify an OID that causes the issue, please open a case with TAC.

Method 1. Use show snmp stats oid command.

The show snmp stats oid command displays the last OID that was polled. It displays the timestamp in order, the goal is to identify the OID that did respond slowly. This command is also helpful if you want to find what MIBs are polled more often by the client.

#show snmp stats oid

time-stamp #of times     requested   OI

14:34:38 CET Oct 25 2020 24       atEntry.2

14:34:29 CET Oct 25 2020 40       atEntry.1

14:34:11 CET Oct 25 2020  11       ifOutErrors

14:34:07 CET Oct 25 2020 10       ifOutDiscards

14:34:06 CET Oct 25 2020 10       ifOutUcastPkts

14:34:06 CET Oct 25 2020 10      ifOutOctets

14:34:05 CET Oct 25 2020 10      ifInUnknownProtos

You can see that Entry.1 took 18 seconds to get computed. This suggests that the CPU was busy in order to compute this data.

Method 2. Observe the SNMP client.

In order to find the OID that is responsible for the high CPU usage on the device, you can initiate a  snmpwalk to a device from an NMS server and observe the output. The OIDs that respond slower than the other OIDs can be the ones responsible for high CPU utilization.

To troubleshoot:

Check SNMP configuration on the device. For SNMPv2, it needs to look like:

snmp-server community TAC1 RO

snmp-server community TAC2 RO --> If multiple communities are added to snmp.

- For SNMPv3

snmp-server view TESTV3 iso include

#snmp-server group TestGroupV3  v3 auth read TESTV3

#snmp-server user cisco TestGroupV3 v3 auth md5 ciscorules priv des56 cisco123

Enter the configuration mode of the device and add a view to the SNMP configuration to change it.

- For SNMPv2

snmp-server community  TAC1 RO view cutdown RO

snmp-server community TAC2 RO view cutdown RO

Add these lines in the configuration mode:

snmp-server view cutdown iso included

snmp-server view cutdown OID _causes_the issue_is _to_excluded  excluded

-->>> The Idea is to exclude the OID causes the issue, however,

please read out what is the function of the OID that  we are about to exclude.

Related Information

- Configuration Guide: Chapter: SNMP

- Cisco Technical Support & Downloads