This document describes possible issues caused by event action overrides on the Cisco Intrusion Prevention System (IPS) and offers recommendations to tune and troubleshoot your installation.
Note: Event action overrides are global actions taken on signatures based upon a risk rating. As with any global configuration, take great care with configuration changes and additions.
Event Action Override Problems
Event action overrides add additional actions to a signature event when that event falls within a specified risk rating range. Use event action overrides carefully. Iif you create an override with a wide risk rating range for an event that is triggered frequently (especially specific, expensive actions, such as IP logging actions), you might cause problems.
Excessive writes to the event store are typically associated with high CPU utilization and general unresponsiveness of the sensor to management access tools such as the command-line interface (CLI) and the Cisco IPS Device Manager (IDM).
IP Logging Actions and File Descriptors
A file descriptor is a data structure used by a program in order to get a handle on a file; well-known descriptors are 0,1,2 for standard in, standard out, and standard error. A file descriptor is created when a process opens a new file or socket.
If you create an event action override for an IP logging action such as log-attacker-packets, log-pair-packets, or log-victim-packets, this might exhaust the pool of file descriptors; overall sensor performance might be negatively affected and the sensor may not function properly.
SNMP Trap Actions and Event Action Overrides
A signature that has only the single action of request-snmp-trap also generates an alert event that is written to the event store. So, excessive firing of the Simple Network Management Protocol (SNMP) trap action might also trigger the same problems seen with excessive produce alert actions.
Actions for Normalizer Engine Signatures
Do not add any action that causes event store writes (such as produce alert, request-snmp-trap, or log-actions) to Normalizer signatures. This applies to all 1200-1330 range signature IDs.
Except for brief troubleshooting scenarios, you should not use event action overrides for the Normalizer engine signatures. This can be particularly problematic in:
highly fragmented IP scenarios (due to the 1200-range signatures)
For example, an event action override that causes a write to the event store for every ooo TCP packet can cause resource and utilization issues.
Event Action Overrides with Risk Rating of 0-100
In general, avoid event action overrides with a risk rating of 0-100 because the low rating can put your sensor at risk of failure in certain circumstances.
Meta component signatures often fire for seemingly benign (and common) types of traffic. Meta signatures look for a combination of one or more Meta component signatures to trigger before the parent Meta signature fires an alert. Meta component signatures, by default, have no actions associated with them; this is intentional because they frequently match on common traffic. Meta component signatures have a default base risk rating of 15. In order to exclude capture of these signature matches in an event action override, Cisco recommends that you do not use a risk rating lower than 25 when you create an event action override; that is, the risk rating should not be below 25-100.
In IPS Versions 7.0(8)E4 and 7.1(6)E4, the show inspection-load command has been added:
sensor# show inspection-load history sensor 10:17:57 UTC Mon Apr 05 2013
This is example output from that command:
A very high load percentage (90% or higher) might indicate that there are excessive events triggered by event action overrides. Refer to the logs in order to further confirm this possibility.
The main indicator of excessive event action overrides is rapid event store wrapping, as seen in this example main.log file:
25Jan2010 05:13:08.326 19.897 sensorApp IdsEventStore/W errWarning - the event store wrapped around [IdsEventStore::writeEvent(), index = 19530] 25Jan2010 05:32:05.751 85.031 sensorApp IdsEventStore/W errWarning - the event store wrapped around [IdsEventStore::writeEvent(), index = 19529] 25Jan2010 05:50:45.442 4.989 sensorApp IdsEventStore/W errWarning - the event store wrapped around [IdsEventStore::writeEvent(), index = 19530] 25Jan2010 06:08:59.281 70.143 sensorApp IdsEventStore/W errWarning - the event store wrapped around [IdsEventStore::writeEvent(), index = 19529] 25Jan2010 06:25:40.923 34.562 sensorApp IdsEventStore/W errWarning - the event store wrapped around [IdsEventStore::writeEvent(), index = 19531]
In general, event store wrapping that occurs more often than once an hour may indicate a problem. In some scenarios, the wrapping is so excessive that it may occur many times within a minute. There are many variables, such as the overall performance capability of the platform, to consider.
Determine what type of event, traffic, or action is causing the event action override problem. Is it a produce alert, IP logging, Normalizer signature, or Meta component signature?
If it is a 'chatty' signature and you determine the signature creates false positives for events, write an event action filter (EAF).
For IP logging, Cisco recommends you avoid EAFs or use EAFs with caution and with a complete understanding of the risks.
Normalizer signatures and Meta component signatures should not have an alert action except for temporary troubleshooting scenarios.