Use of EEM Scripts to Monitor High CPU Utilization on Cisco Catalyst Series Switches

Introduction

This document describes how to use the Cisco IOS^® Embedded Event Manager (EEM) subsystem in order to monitor high CPU utilization on Cisco Catalyst series switches.

Prerequisites

Requirements

This document assumes that user has knowledge of Cisco IOS Embedded Event Manager (EEM).

Components Used

The information in this document is based on the Cisco Catalyst series switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for information on document conventions.

Background Information

EEM is a very useful tool to troubleshoot occasional, sporadic CPU spikes that are short-lived and difficult to troubleshoot manually with the command line interface. This is an example of CPU spikes:

Switch#show process cpu history
<snip>
    11111822511   11 111277711111 124111  11 1211111112161116
    143342171209994090111878458239607111981270283361362429475
100
 90
 80      *               ***
 70      *               ***                                *
 60      *               ***                            *   *
 50      *  *            ***        *                   *   *
 40      *  *            ***        *                   *   *
 30      * **            ***        *                   *   *
 20      ****           **** **   ***         **  *  ** ** **
 10 *********************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
             0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

Troubleshoot

This section includes several examples of the use of EEM scripts to monitor CPU utilization. Catalyst 2960 and 3750 switches allow EEM to use non-volatile RAM (NVRAM); Catalyst 4500 switches allow EEM to write to bootflash; and Catalyst 6500 switches allow EEM to use disk0 and sup-bootdisk.

Note: Use the Command Lookup Tool (registered customers only) in order to obtain more information on the commands used in this section.

EEM to capture data in event of CPU spike on 2960X.

Catalyst 2960X does not support appending to flash. EEM below would redirect captured data in respective files in flash:/TAC from last run of the EEM.

logging buffered 8192 informational
process cpu threshold type total rising 80 interval 5
!
event manager applet High_CPU authorization bypass
event syslog pattern "CPURISINGTHRESHOLD"
action 001 syslog msg "EEM: HIGH CPU detected. Writing info to flash:"
action 002 cli command "enable"
action 003 cli command "term exec prompt timestamp"
action 004 cli command "term len 0"
action 005 cli command "mkdir TAC" pattern "Create.*"
action 006 cli command "TAC" pattern "Create.*"
action 007 cli command " "
action 008 cli command "show process cpu sorted | redirect flash:TAC/TAC_CPU.txt"
action 009 cli command "show buffers | redirect flash:TAC/TAC_Buffers.txt"
action 010 cli command "show interfaces summary | redirect flash:TAC/TAC_IntSumm.txt"
action 011 cli command "show interfaces stats | redirect flash:TAC/TAC_IntStat.txt"
action 012 cli command "show ip traffic | redirect flash:TAC/TAC_IPTraffic.txt"
action 013 cli command "show ip cef switching statistics | redirect flash:TAC/TAC_IPCef.txt"
action 014 cli command "show controllers cpu-interface | redirect flash:TAC/TAC_Controllers.txt"
action 015 cli command "show platform port-asic stats drop | redirect flash:TAC/TAC_Port-Asic.txt"
action 016 cli command "show platform ip unicast statistics | redirect flash:TAC/TAC_UnicastStats.txt"
action 017 cli command "show platform ip unicast counts | redirect flash:TAC/TAC_UnicastCounts.txt"
action 018 cli command "show platform tcam utilization | redirect flash:TAC/TAC_TCAM.txt"
action 019 syslog msg "EEM: Self-removing applet from configuration..."
action 020 cli command "configure terminal"
action 021 cli command "no event manager applet High_CPU"
action 022 syslog msg "EEM: Finished removing from configuration!"
action 023 cli command "end"

Email Alerts

This script emails an alert when CPU utilization goes above 50 percent. The body of the email is the output of the show process cpu sorted command.

event manager applet highcpu
  event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op ge entry-val 50 poll-interval 0.5
  action 1.0 cli command "enable"
  action 2.0 cli command "show proc cpu sorted"
  action 3.0 mail server "192.168.1.1" to "user-to@domain.com" from "user-from@domain.com" subject "High CPU Alert" body "$_cli_result"

The definitions of italicized variables are:

• highcpu - name of the event manager app
• let/script
• 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 - Object identifier (OID) for polling the total CPU utilization of the route processor (RP)
• 50 - CPU utilization that triggers the script
• poll-interval 0.5 - Frequency (every 0.5 seconds) the script monitors the CPU
• 192.169.1.1 - IP of the mail server

Append Output to Local File

This script appends required outputs to a file in the local file system. Replace file system with the appropriate file system on the switch.

event manager scheduler script thread class default number 1 
event manager applet High_CPU
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op ge entry-val 50 poll-interval 0.5
  action 0.0 syslog msg "High CPU DETECTED. Please wait - logging Information to file system:high_cpu.txt"
  action 0.1 cli command "enable"
  action 0.2 cli command "show clock | append file system:high_cpu.txt"
  action 1.2 cli command "term length 0"
  action 1.3 cli command "show process cpu sorted | append file system:high_cpu.txt"
  action 1.4 cli command "show log | append file system:high_cpu.txt"
  action 1.5 cli command "show interfaces | append file system:high_cpu.txt"
  action 1.6 cli command "term length 24"

Append Output to Local File and Remove Script

This script appends the output of the show process cpu sorted command to a file in the local file system, then removes itself once completed. Replace file system with the appropriate file system on the switch.

event manager scheduler script thread class default number 1 
event manager applet High_CPU
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op gt entry-val 50 poll-interval 0.5
  action 0.0 syslog msg "High CPU DETECTED. Please wait - logging Information to flash:high_cpu.txt"
  action 0.1 cli command "enable"
  action 0.2 cli command "term exec prompt timestamp"
  action 1.3 cli command "show process cpu sorted | append file system:high_cpu.txt"
  action 1.4 cli command "show process cpu sorted | append file system:high_cpu.txt"
  action 1.4 cli command "show process cpu sorted | append file system:high_cpu.txt"
  action 5.1 syslog msg "Finished logging information to file system:high_cpu.txt..."
  action 5.1 syslog msg "Self-removing applet from configuration..."
  action 5.2 cli command "term no exec prompt timestamp"
  action 9.1 cli command "configure terminal"
  action 9.2 cli command "no event manager applet High_CPU"
  action 9.3 cli command "end"

Collect Output and Write to Local File

This script uses a syslog-based trigger in order to run and collect required outputs and write those outputs to the local file system. Replace file system with the appropriate file system on the switch.

process cpu threshold type total rising 70 interval 15 
event manager applet DETECT_CPU
event syslog pattern ".*SYS-1-CPURISINGTHRESHOLD.*" 
  action 1 cli command "en" 
  action 2 cli command "show clock | append file system:cpuinfo" 
  action 3 cli command "show proc cpu sort | append file system:cpuinfo" 
  action 4 cli command "show line | append file system:cpuinfo"

Monitor CPU Utilization on Modular IOS

The Cisco EEM can also be used to monitor CPU utilization on modular IOS. Because of the differences in how the CPU is monitored on modular IOS, you can use the Simple Network Management Protocol (SNMP) OID (1.3.6.1.4.1.9.9.109.1.1.1.1.3.1) in order to check CPU utilization by the IOS base process.

This script uses the OID as a trigger and writes required outputs to the local file system. Replace file system with the appropriate file system on the switch.

event manager scheduler script thread class default number 1
event manager applet High_CPU
event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.10.1  get-type exact entry-op ge entry-val 50 poll-interval 0.5 
  action 0.0 syslog msg "High CPU DETECTED. Please wait - logging Information to file system:high_cpu.txt"
  action 0.1 cli command "enable"
  action 0.2 cli command "show clock | append file system:high_cpu.txt"
  action 1.2 cli command "term length 0"
  action 1.3 cli command "show process cpu sorted | append file system:high_cpu.txt"
  action 1.4 cli command "show log | append file system:high_cpu.txt"
  action 1.5 cli command "show interfaces | append file system:high_cpu.txt"
  action 1.6 cli command "term length 24"

Remove Script

Enter this command in order to remove an EEM script:

Switch(config)#no event manager applet applet name

Related Information

• Technical Support & Documentation - Cisco Systems