Introduction
Event Detectors
      Actions
      Applets and Scripts
Security Relevance of EEM
      EEM Examples
References

Introduction

The Embedded Event Manager (EEM) functionality initially added in Cisco IOS Software versions 12.0(26)S and 12.3(4)T adds reactive capabilities that can aid the security posture of a network. At a very simple level, EEM detects events and performs actions. At a more advanced level, EEM can leverage Tcl scripts to perform advanced, dynamic actions based on events. These capabilities are especially useful in situations in which logging and MIB data are insufficient or do not exist, or when actions must be performed without the assumption of network access.

EEM is a comprehensive system with several components. At the center of EEM are event detectors (EDs), actions, and policies. There are two types of policies: applets and scripts.

Event Detectors

EDs serve as the first line of the EEM system. Configured EDs are constantly monitoring the system for an event to take place. Every applet or script-based policy registers with exactly one ED. For script-based policies, this registration must be the first non-comment line in the policy.

The available EDs are detailed in the following table:

Event Detector	Description
Application specific	Detects events published by other EEM policies.
Command-line interface	Receives commands as users enter them. This ED can operate in one of two modes: In synchronous mode, the applet or script runs to completion and must permit the command before it is executed. In asynchronous mode, commands are sent to the ED and executed in parallel.
Counter	Uses named counters that provide configurable thresholds. These counters can be updated by policies and can be used to trigger other policies when thresholds are crossed.
Enhanced object tracking	Can be used to execute policies when a tracked object, such as Hot Standby Router Protocol (HSRP), changes state.
Interface counter	Can establish absolute or relative thresholds based on a particular interface statistic. Policies are triggered when a threshold has been crossed.
None	The none ED is used when an applet or script is only to be executed manually through the event manager run privileged EXEC command.
Online insertion and removal	Detects online insertion and removal events.
Resource	Used in applets to detect the crossing of an Embedded Resource Manager (ERM) policy threshold.
Redundancy framework	Detects events related to redundant route processor switchover.
SNMP	Can monitor a local MIB object and trigger registered policies when the value crosses a configurable threshold.
Syslog	Uses regular expressions to match syslog messages created by the system. Optionally, rate-based thresholds can be applied before a policy is executed.
Timer	Triggers events when a timer has expired. There are four types of timers available: Absolute time of day Countdown timer (executed only once) Watchdog timer (executed at regular intervals) Timer using a standard cron format
Watchdog system monitor	Monitors the CPU or memory used by a process and creates an event when a threshold of one or both characteristics is crossed.

Refer to the Cisco IOS documentation for the Embedded Event Manager at http://www.cisco.com/univercd/cc/td/doc/product/software/ios124/124newft/124t/124t2/ht_eem.htm for more information regarding EDs.

Actions

After being triggered by an ED, a policy can perform one or many actions. The following actions are detailed in the Cisco IOS EEM documentation.

Executing one or more commands using the Cisco IOS command-line interface (CLI)
Generating a Cisco Networking Services event
Setting or altering a named counter
Forcing switchover to a redundant processor
Sending a short e-mail using a Cisco-provided Tcl library
Executing a predefined EEM policy
Publishing an event to the application-specific ED
Reloading the device
Generating an SNMP trap
Generating a syslog message
Reading or setting the state of a tracked object

Applets and Scripts

Policies that exist entirely in the configuration of the router and use a limited command set are known as applets; the much more advanced and flexible Tcl-based policies are called scripts.

Applets provide an easy method for implementing reactive capabilities in a network device. However, their functionality is severely limited when compared to scripts. For example, it is not possible to parse CLI output within an applet. With applets, the selection of an event and one or more actions can be plainly viewed using the show running-config privileged EXEC command.

Scripts are Tcl-based and exist in flat-text files on the device. After an administrator copies scripts to an IOS device, it is possible to view the contents of a script using the more device:tcl-script EXEC command. These policies are registered using the event manager policy policy-filename.tcl global configuration command. These Tcl files are afforded nearly all the functionality of standard Tcl. Cisco-specific functionality is also available in Cisco IOS versions that support EEM. Specific limitations of Tcl on IOS are available at http://www.cisco.com/en/US/products/ps6441/products_configuration_guide_chapter09186a00807c6784.html#wp1014310.

See Writing Embedded Event Manager Policies Using Tcl at http://www.cisco.com/en/US/products/ps6441/products_configuration_guide_chapter09186a00807c6784.html for more information regarding the creation of Tcl-based policies.

Security Relevance of EEM

With EEM, it is possible to alert the security operations staff if certain, previously uninstrumented, events take place. For example, a router can send a syslog message when its AAA configuration has been removed or send an e-mail if the number of packets dropped by Unicast RPF crosses a threshold. The possibilities range from simple applet-based configurations to elaborate Tcl scripts.

It is a common scenario that identification or classification ACLs are placed at the edge of the network. However, ACL logging is rarely enabled due to the high CPU load impact. With EEM, it is possible to generate a syslog message when a configurable threshold has been crossed without encountering the ACL logging–induced CPU load. A sample EEM script for generating this message is in the “Examples” section of this document.

A similar scenario would be the creation of a script to generate syslog messages when Unicast RPF failures are seen on a router. This capability demonstrates the how EEM provides insight into functionality for which syslogs and MIBS do not exist or where they have serious performance and availability implications.

Another example of the usefulness of EEM includes the ability to block CLI commands from being executed. For example, users entering the tclsh command may be attempting to exploit the vulnerability disclosed by Cisco in Cisco Security Notice: Response to AAA Command Authorization Bypass. With EEM, administrators can detect users executing this command, then send a syslog message and block execution of the command.

EEM Examples

The examples in this section demonstrate the security value of EEM; however, they are by no means an exhaustive catalog of how EEM can increase the security of an IOS device. These examples range from the very simple (blocking access to an unused and potentially vulnerable command) to the more complex (generating syslog messages based on statistics and user-configurable thresholds).

Many of the lines in these examples are too long to be correctly displayed. As a result, lines have been wrapped using the backslash (\) convention. This will allow Tcl scripts to be directly cut and pasted into standalone files. However, IOS configuration lines wrapped with the backslash must be manually reconstructed before being placed on any device.

Blocking Commands and Sending Syslogs

The example below demonstrates the ability of an EEM policy, and specifically the CLI ED, to block command execution. This is accomplished using the CLI ED in synchronous mode and using the _exit_status variable. The command is denied if _exit_status is 0 ; it is permitted if _exit_status is not 0.

This policy denies access by any user to the tclsh command. Access to this command in environments that do not use Tcl may be an attempt to exploit the AAA bypass vulnerability detailed at http://www.cisco.com/warp/public/707/cisco-response-20060125-aaatcl.shtml.

!
event manager applet EEM-APPLET-BLOCK-TCLSH 
 event cli pattern "tclsh" sync yes
 action 1.0 syslog msg "Attempted tclsh command by user\
                        at $_event_pub_time"
 set 2.0 _exit_status 0
!

With the preceding policy configured, attempts to execute tclsh are blocked and logged.

router#tclsh
router#
*Feb 19 22:34:17.467: %HA_EM-6-LOG: EEM-APPLET-BLOCK-TCLSH: Attempted tclsh command by user at Feb 21 21:28:57.451
router#

Threshold-Based Logging for ACL-Dropped Packets

For environments in which IP Options are not used, it may be desirable to filter packets that contain IP options at the network edge. Administrators can implement infrastructure ACLs to perform this filtering. However, without ACL logging, it may be difficult and extremely tedious to detect when these ACLs are filtering packets. Using EEM, administrators can check the ACL counters at regular intervals and send a syslog message if drops are seen.

This example details a script-based policy that interacts with the CLI and is configured using router configuration-based environment variables. The use of environment variables makes it possible to configure the rate-based thresholds without editing the script itself. In this example, a syslog is generated when 10 packets (EEM_ACL_COUNTERS_THRESHOLD) are dropped by a particular ACL (EEM_ACL_COUNTERS_ACL_NAME) in 20 seconds (EEM_ACL_COUNTERS_INTERVAL).

These environment variables are configured using the following global configuration commands.

!
event manager environment EEM_ACL_COUNTERS_INTERVAL 20
event manager environment EEM_ACL_COUNTERS_ACL_NAME\
                          ACL-DROP-IP-OPTIONS
event manager environment EEM_ACL_COUNTERS_THRESHOLD 10
!

Because scripts exist in files on the flash file system of the router, their configuration is not present in the router configuration. However, the script must be registered with EEM via the event manager policy script-name global configuration command. The following command registers this script:

!
event manager policy EEM-ACL-COUNTERS.tcl
!

The following example shows the EEM script that performs the desired actions. The comments within the script describe its operation.

#
# Register with the timer ED. We use a watchdog timer to
# execute this policy every EEM_ACL_COUNTERS_INTERVAL
# seconds.
#
::cisco::eem::event_register_timer watchdog name timer\
              time $EEM_ACL_COUNTERS_INTERVAL

#
# Import the ::cisco::eem namespace so that we do not need
# to qualify EEM-specific functions.
#
namespace import ::cisco::eem::*

#
# Check that the variables we need have been added to the 
# router configuration using the ‘event manager environment’
# global configuration command.
#
if {![info exists EEM_ACL_COUNTERS_INTERVAL]} {
 set result "Policy cannot be run:\
            variable EEM_ACL_COUNTERS_INTERVAL has not been set"
 error $result $errorInfo
}
if {![info exists EEM_ACL_COUNTERS_ACL_NAME]} {
 set result "Policy cannot be run:\
            variable EEM_ACL_COUNTERS_ACL_NAME has not been set"
 error $result $errorInfo
}
if {![info exists EEM_ACL_COUNTERS_THRESHOLD]} {
 set result "Policy cannot be run:\
            variable EEM_ACL_COUNTERS_THRESHOLD has not been set"
 error $result $errorInfo
}

#
# Retrieve the count variable from the context
# EEM_ACL_COUNTERS. If this generates an error this is the
# first time the script has run, so initialize to zero. If
# no error, place the value into ‘globalcounter’.
#
if [catch {context_retrieve "EEM_ACL_COUNTERS" "count"} result] {
 set globalcounter 0
} else {
 set globalcounter $result
}

#
# Open the CLI; the cli_* functions are part of the
# ::cisco::eem namespace.
#
if [catch {cli_open} result] {
 error $result $errorInfo
} else {
 array set cli $result
}

#
# Execute the ‘show access-list’ command for the ACL
# named in EEM_ACL_COUNTERS_ACL_NAME and place the results
# in the variable cmd_output.
#
if [catch {cli_exec $cli(fd) "show access-list\
                    $EEM_ACL_COUNTERS_ACL_NAME"} result] {
 error $result $errorInfo
} else {
 set cmd_output $result
}

#
# Close the CLI.
#
if [catch {cli_close $cli(fd) $cli(tty_id)} result] {
 error $result $errorInfo
}

#
# Initialize the list variable to catch each occurrence of 
# deny ... (XX matches) using the regexp function.
#
set list ""
catch {regexp -all -inline {deny[^\n]+?\((\d+) matches\)}\
      $cmd_output} list

#
# Initialize i for stepping through the list returned above;
# and count to sum the packet matches.  We step through the list
# in twos so that we only get the (\d+) component of the regex.
#
set i 1
set count 0
while {$i <= [llength $list]} {
 set count [expr $count + [lindex $list $i]]
 set i [expr $i + 2]
}

#
# Determine the difference between the previous counter
# (in globalcounter) and the present counter (in count).
#
set diff [expr $count - $globalcounter]

#
# This is useful for troubleshooting, but should be removed
# before a production deployment.
#
# puts "New count = $count, old count = $globalcounter,\
#       difference = $diff"

#
# Use action_syslog to send a syslog if the packet count
# delta (in diff) is beyond our threshold (in
# EEM_ACL_COUNTERS_THRESHOLD).
#
if {$diff >= $EEM_ACL_COUNTERS_THRESHOLD} {
 action_syslog priority notice msg "Threshold\
               ($EEM_ACL_COUNTERS_THRESHOLD) exceeded for\
               $EEM_ACL_COUNTERS_ACL_NAME, $diff\
               packets in the last $EEM_ACL_COUNTERS_INTERVAL\
               seconds."
}

#
# Save our new counter in the context EEM_ACL_COUNTERS so
# that we can access it next time this script is executed.
#
context_save EEM_ACL_COUNTERS count

If the threshold is crossed in the interval specified—10 packets in 20 seconds in the preceding example—a syslog message is generated. This message will be sent to all logging destinations that have been configured on the router. The following is an example of the syslog generated:

*Feb 20 22:42:52.011: %HA_EM-5-LOG: tmpsys:/eem_policy/EEM-ACL-COUNTERS.tcl: 
Threshold (10) exceeded for ACL-DROP-IP-OPTIONS, 18 packets in the last 20 seconds.

Sending a Syslog Message When Unicast RPF Failures Occur

This example details a script-based policy that interacts with the CLI and is configured using router configuration-based environment variables. The use of environment variables makes it possible to configure the rate-based thresholds without editing the script itself. In this example, a syslog message is generated when 10 packets (EEM_URPF_COUNTER_THRESHOLD) are dropped by Unicast RPF in 20 seconds (EEM_URPF_COUNTER_INTERVAL).

These environment variables are configured using the following global configuration commands:

!
event manager environment EEM_URPF_COUNTER_INTERVAL 20
event manager environment EEM_URPF_COUNTER_THRESHOLD 10
!

The command needed to register this script is shown below.

!
event manager policy EEM-URPF-COUNTER.tcl
!

The following script-based policy performs the actions described above. Because scripts exist in files on the file system of the router, their configuration is not present in the router configuration. The following example shows the EEM script that performs the desired actions.

#
# Register with the timer ED.  We use a watchdog timer to
# execute this policy every EEM_URFP_COUNTER_INTERVAL
# seconds.
#
::cisco::eem::event_register_timer watchdog name timer\
              time $EEM_URPF_COUNTER_INTERVAL

#
# Import the ::cisco::eem namespace so that we do not need
# to qualify EEM-specific functions.
#
namespace import ::cisco::eem::*

#
# Check that the variables we need have been added to the 
# router configuration using the ‘event manager environment’
# global configuration command.
#
if {![info exists EEM_URPF_COUNTER_INTERVAL]} {
 set result "Policy cannot be run:\
            variable EEM_URPF_COUNTER_INTERVAL has not been set"
 error $result $errorInfo
}
if {![info exists EEM_URPF_COUNTER_THRESHOLD]} {
 set result "Policy cannot be run:\
            variable EEM_URPF_COUNTER_THRESHOLD has not been set"
 error $result $errorInfo
}

#
# Retrieve the count variable from the context
# EEM_URPF_COUNTER. If this generates an error this is the
# first time the script has run, so initialize to zero. If
# no error, place the value into ‘globalcounter’.
#
if [catch {context_retrieve "EEM_URPF_COUNTER" "count"}\
    result] {
 set globalcounter 0
} else {
 set globalcounter $result
}

#
# Open the CLI; the cli_* functions are part of the
# ::cisco::eem namespace.
#
if [catch {cli_open} result] {
 error $result $errorInfo
} else {
 array set cli $result
}

#
# Execute the ‘show ip traffic | in unicast RPF’ command to 
# determine the number of unicast RPF drops the system has 
# experienced since startup.
#
if [catch {cli_exec $cli(fd) "show ip traffic | in unicast RPF"}\
    result] {
 error $result $errorInfo
} else {
 set cmd_output $result
}

#
# Close the CLI.
#
if [catch {cli_close $cli(fd) $cli(tty_id)} result] {
 error $result $errorInfo
}

#
# Initialize the null variable to receive data from regexp
# that we do not want. The count variable will receive the 
# digits in front of ‘ unicast RPF’.
#
set null ""
set count 0
regexp {, (\d+) unicast RPF,} $cmd_output null count

#
# Determine the difference between the previous counter
# (in globalcounter) and the present counter (in count).
#
set diff [expr $count - $globalcounter]

#
# This is useful for troubleshooting, but should be removed
# before a production deployment.
#
# puts "New count = $count, old count = $globalcounter,\
#      difference = $diff"

#
# Use action_syslog to send a syslog if the packet count
# delta (in diff) is beyond our threshold (in
# EEM_URPF_COUNTER_THRESHOLD).
#
if {$diff >= $EEM_URPF_COUNTER_THRESHOLD} {
 action_syslog priority notice msg "Threshold\
               ($EEM_URPF_COUNTER_THRESHOLD) exceeded, $diff in\
               last $EEM_URPF_COUNTER_INTERVAL seconds."
}

#
# Save our new counter in the context EEM_URPF_COUNTER so
# that we can access it next time this script is executed.
#
context_save EEM_URPF_COUNTER count

If the threshold is crossed in the interval specified—10 packets in 20 seconds in the preceding example—a syslog message is generated. This will be sent to all logging destinations that have been configured on the router. The following is an example of the syslog message generated.

*Feb 22 05:57:53.759: %HA_EM-5-LOG: tmpsys:/eem_policy/EEM-URPF-COUNTER.tcl: 
Threshold (10) exceeded, 16 in last 20 seconds.

Building a Control Plane Policing Baseline Using EEM

The policies earlier in this document provide a reactive functionality that overcomes an existing limitation. This next policy is different in that it provides a somewhat proactive capability. This script parses the configuration of Control Plane Policing on a device and records the number of packets and bytes matching each applied class map. The script output is saved into a CSV file for offline processing. Together, this script and the offline processing will allow an organization to tune a CoPP configuration based on real-world packet rates.

This script uses the EEM timer ED to execute periodically as well as the EEM-native Tcl to interact with the CLI and parse its output. The configuration of this script is controlled using two environment variables. The first, EEM_COPP_BASELINE_DIR, is used to determine the output location for the created CSV files. The second, EEM_COPP_BASELINE_INTERVAL is used to configure the interval in seconds between two executions of the baselining script.

!
event manager environment EEM_COPP_BASELINE_DIR disk0:w/
event manager environment EEM_COPP_BASELINE_INTERVAL 300
!

The following command registers this script:

!
event manager policy EEM-COPP-BASELINE.tcl
!

The contents of the TCL policy follow. The comments within the code describe the operation of the script.

# 
# Use the timer ED to execute every 
# EEM_COPP_BASELINE_INTERVAL seconds.
#
::cisco::eem::event_register_timer watchdog name timer\
 time $EEM_COPP_BASELINE_INTERVAL

#
# Import the cisco namespaces to gain access to the
# cli_* keywords.
#
namespace import ::cisco::eem::*

#
# Check to make sure our variables exist. If not, exit 
# with an error.
#
if {![info exists EEM_COPP_BASELINE_INTERVAL]} {
 set result "Policy cannot be run:\
 variable EEM_COPP_BASELINE_INTERVAL has not been set"
 error $result $errorInfo
}
if {![info exists EEM_COPP_BASELINE_DIR]} {
 set result "Policy cannot be run:\
 variable EEM_COPP_BASELINE_DIR has not been set"
 error $result $errorInfo
}

#
# Retrieve the statistics gathered the last time the script was
# executed. This is stored in the line variable in the 
# EEM_COPP_BASELINE context.
#
if [catch {context_retrieve "EEM_COPP_BASELINE" "line"} result] {
 set line ""
 set linesplit ""
} else {
 set line $result
 catch {regexp -all -inline {(\d+)\D} $line} linesplit
}

#
# Use the Cisco provided CLI library to open a CLI, execute
# the 'show policy-map control-plane' command, place the 
# output in the cmd_output variable, then close the CLI.
#
if [catch {cli_open} result] {
 error $result $errorInfo
} else {
 array set cli $result
}

if [catch {cli_exec $cli(fd) "show policy-map control-plane"}\
 result] {
 error $result $errorInfo
} else {
 set cmd_output $result
}
if [catch {cli_close $cli(fd) $cli(tty_id)} result] {
 error $result $errorInfo
}

#
# Create the filename to use, place in outputfilename. This 
# creates a new CSV file each day. Default the header variable
# to 0.
#
set outputfilename "$EEM_COPP_BASELINE_DIR[clock format\
 [clock seconds] -format "%Y-%m-%d"]_EEM_COPP_BASE.csv"
set header 0

#
# If the output file does not exist, set header to 1 so that
# we print a header row in the CSV file.
#
if {![file exists $outputfilename]} {
 set header 1
} 

#
# Open the output file for appending, place the handle in the 
# outputfile variable.
#
set outputfile [open $outputfilename a+]

#
# Match each class map name, followed by the next occurrence 
# of (\d+) packets, (\d+) bytes. Place all matches in the 
# ‘list’ variable list.
#
catch {regexp -all –inline\
 {Class-map: (\S+)[^\n]+\s+(\d+) packets, (\d+) bytes}\
 $cmd_output} list

#
# If we previously set header to 1 (based on the nonexistent 
# output file), print a header in the file.
#
if {$header == 1} {
 set ii 1
 puts -nonewline $outputfile "time,"

 while {$ii <= [llength $list]} {
puts -nonewline $outputfile "[lindex $list $ii]\
 packets,[lindex $list $ii] bytes,"
  incr ii 4
 }
 puts $outputfile ""
} 

#
# Print the first column of data, the current date and time in 
# 2007-03-06 13:00:00 format.
#
puts -nonewline $outputfile "[clock format\
 [clock seconds] -format "%Y-%m-%d %H:%M:%S"],"
set line ","

#
# Iterate over the list, printing the packet and byte count
# for each class map. Add values to the line variable to 
# access on the next script execution.
#
set i 2
set iii 1
while {$i <= [llength $list]} {
 if {[llength $linesplit] == 0} {
  set var1 [lindex $list $i]
  set var2 [lindex $list [expr $i + 1]]

 } else {
  set var1 [ expr [lindex $list $i] - [lindex $linesplit $iii] ]
set var2 [ expr [lindex $list [expr $i + 1]] - [lindex\
 $linesplit [expr $iii + 2]] ]
 }

 puts -nonewline $outputfile "$var1,$var2,"

 set line "$line[lindex $list $i],[lindex $list [expr $i + 1]],"

 incr i 4
 incr iii 4
}
#
# Wrap the line.
#
puts $outputfile ""

#
# Save line to the EEM_COPP_BASELINE context so that we\
# can retrieve it on the next script execution.
#
context_save EEM_COPP_BASELINE "line"

# Close output file.
#
close $outputfile

This policy creates a new output CSV file for each day. The dir command can be used on the directory configured in EEM_COPP_BASELINE_DIR to display the files currently present. The following is an example using the directory disk0:e/ :

router#dir disk0:e/
Directory of disk0:/e/

    8  -rw-        2622   Mar 6 2007 14:44:44 +00:00  2007-03-06_EEM_COPP_BASE.csv
	
47951872 bytes total (14061568 bytes free)

The contents of the file can be examined on the router using the more command.

router#more disk0:/e/2007-03-06_EEM_COPP_BASE.csv
time,CM-COPP-MGMT packets,CM-COPP-MGMT bytes,CM-COPP-BGP packets,CM-COPP-BGP bytes,class-default packets,class-default bytes,
2007-03-06 13:36:45,0,0,0,0,127,32052,
2007-03-06 13:37:45,0,0,0,0,5,1739,
2007-03-06 13:38:45,0,0,0,0,3,1055,
2007-03-06 13:39:45,0,0,0,0,5,1739,
2007-03-06 13:40:45,0,0,0,0,5,1739,
2007-03-06 13:41:45,0,0,0,0,3,1055,
2007-03-06 13:42:45,0,0,0,0,5,1739,
2007-03-06 13:43:45,0,0,0,0,5,1739,
2007-03-06 13:44:45,0,0,0,0,3,1055,
...

References

Cisco Beyond
http://www.cisco.com/go/ciscobeyond

Embedded Event Manager
http://www.cisco.com/univercd/cc/td/doc/product/software/ios124/124newft/124t/124t2/ht_eem.htm

Embedded Resource Manager
http://www.cisco.com/en/US/products/sw/iosswrel/ps5207/products_feature_guide09186a00803790a7.html

IOS Network Management Command Reference:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios124/124cr/hnm_r/nmg_01h.htm

Writing Embedded Event Manager Policies Using Tcl:
http://www.cisco.com/en/US/products/ps6441/products_configuration_guide_chapter09186a00807c6784.html

This document is part of the Cisco Security Center.

This document is provided on an "as is" basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information on the document or materials linked from the document is at your own risk. Cisco reserves the right to change or update this document at any time.

Cisco Security Center

Embedded Event Manager in a Security Context

Contents