Guest

Cisco Catalyst 6500 Series Switches

Use of EEM Scripts to Monitor High CPU Utilization on Cisco Catalyst Series Switches

Document ID: 116141

Updated: May 09, 2013

Contributed by Shashank Singh and Saurav Lahiri, Cisco TAC Engineers.

   Print

Introduction

This document describes how to use the Cisco IOS? Embedded Event Manager (EEM) subsystem in order to monitor high CPU utilization on Cisco Catalyst series switches.

Prerequisites

Requirements

This document assumes that user has knowledge of Cisco IOS Embedded Event Manager (EEM).

Components Used

The information in this document is based on the Cisco Catalyst series switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for information on document conventions.

Background Information

EEM is a very useful tool to troubleshoot occasional, sporadic CPU spikes that are short-lived and difficult to troubleshoot manually with the command line interface. This is an example of CPU spikes:

Switch#show process cpu history
<snip>
    11111822511   11 111277711111 124111  11 1211111112161116
    143342171209994090111878458239607111981270283361362429475
100
 90
 80      *               ***
 70      *               ***                                *
 60      *               ***                            *   *
 50      *  *            ***        *                   *   *
 40      *  *            ***        *                   *   *
 30      * **            ***        *                   *   *
 20      ****           **** **   ***         **  *  ** ** **
 10 *********************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

Troubleshoot

This section includes several examples of the use of EEM scripts to monitor CPU utilization. Catalyst 2960 and 3750 switches allow EEM to use non-volatile RAM (NVRAM); Catalyst 4500 switches allow EEM to write to bootflash; and Catalyst 6500 switches allow EEM to use disk0 and sup-bootdisk.

Note: Use the Command Lookup Tool (registered customers only) in order to obtain more information on the commands used in this section.

Email Alerts

This script emails an alert when CPU utilization goes above 50 percent. The body of the email is the output of the show process cpu sorted command.

event manager applet highcpu
  event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op ge
entry-val 50 poll-interval 0.5
action 1.0 cli command "enable"
  action 2.0 cli command "show proc cpu sorted"
  action 3.0 mail server "192.168.1.1" to "user-to@domain.com" from "user-from@domain.com"
subject "High CPU Alert" body "$_cli_result"

The definitions of italicized variables are:

  • highcpu - name of the event manager applet/script
  • 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 - Object identifier (OID) for polling the total CPU utilization of the route processor (RP)
  • 50 - CPU utilization that triggers the script
  • poll-interval 0.5 - Frequency (every 0.5 seconds) the script monitors the CPU
  • 192.169.1.1 - IP of the mail server

Append Output to Local File

This script appends required outputs to a file in the local file system. Replace file system with the appropriate file system on the switch.

event manager scheduler script thread class default number 1 
event manager applet High_CPU
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op ge
entry-val 50 poll-interval 0.5
action 0.0 syslog msg "High CPU DETECTED. Please wait - logging Information
to file system:high_cpu.txt"
action 0.1 cli command "enable"
  action 0.2 cli command "show clock | append file system:high_cpu.txt"
  action 1.2 cli command "term length 0"
  action 1.3 cli command "show process cpu sorted | append file system:high_cpu.txt"
  action 1.4 cli command "show log | append file system:high_cpu.txt"
  action 1.5 cli command "show interfaces | append file system:high_cpu.txt"
  action 1.6 cli command "term length 24"

Append Output to Local File and Remove Script

This script appends the output of the show process cpu sorted command to a file in the local file system, then removes itself once completed. Replace file system with the appropriate file system on the switch.

event manager scheduler script thread class default number 1 
event manager applet High_CPU
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op gt
entry-val 50 poll-interval 0.5
action 0.0 syslog msg "High CPU DETECTED. Please wait - logging Information
to flash:high_cpu.txt"
  action 0.1 cli command "enable"
  action 0.2 cli command "term exec prompt timestamp"
  action 1.3 cli command "show process cpu sorted | append file system:high_cpu.txt"
  action 1.4 cli command "show process cpu sorted | append file system:high_cpu.txt"
  action 1.4 cli command "show process cpu sorted | append file system:high_cpu.txt"
  action 5.1 syslog msg "Finished logging information to file system:high_cpu.txt..."
  action 5.1 syslog msg "Self-removing applet from configuration..."
  action 5.2 cli command "term no exec prompt timestamp"
  action 9.1 cli command "configure terminal"
  action 9.2 cli command "no event manager applet High_CPU"
  action 9.3 cli command "end"

Collect Output and Write to Local File

This script uses a syslog-based trigger in order to run and collect required outputs and write those outputs to the local file system. Replace file system with the appropriate file system on the switch.

process cpu threshold type total rising 70 interval 15 
event manager applet DETECT_CPU
event syslog pattern ".*SYS-1-CPURISINGTHRESHOLD.*"
action 1 cli command "en"
action 2 cli command "show clock | append file system:cpuinfo"
action 3 cli command "show proc cpu sort | append file system:cpuinfo"
action 4 cli command "show line | append file system:cpuinfo"

Monitor CPU Utilization on Modular IOS

The Cisco EEM can also be used to monitor CPU utilization on modular IOS. Because of the differences in how the CPU is monitored on modular IOS, you can use the Simple Network Management Protocol (SNMP) OID (1.3.6.1.4.1.9.9.109.1.1.1.1.3.1) in order to check CPU utilization by the IOS base process.

This script uses the OID as a trigger and writes required outputs to the local file system. Replace file system with the appropriate file system on the switch.

event manager scheduler script thread class default number 1
event manager applet High_CPU
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.10.1  get-type exact entry-op ge
entry-val 50 poll-interval 0.5
  action 0.0 syslog msg "High CPU DETECTED. Please wait - logging Information
to file system:high_cpu.txt"
  action 0.1 cli command "enable"
  action 0.2 cli command "show clock | append file system:high_cpu.txt"
  action 1.2 cli command "term length 0"
  action 1.3 cli command "show process cpu sorted | append file system:high_cpu.txt"
  action 1.4 cli command "show log | append file system:high_cpu.txt"
  action 1.5 cli command "show interfaces | append file system:high_cpu.txt"
  action 1.6 cli command "term length 24"

Remove Script

Enter this command in order to remove an EEM script:

Switch(config)#no event manager applet applet name

Related Information

Updated: May 09, 2013
Document ID: 116141