Guest

IDS/IPS - Reaction & Response

Event Action Overrides Troubleshooting

Event Action Overrides Troubleshooting

Document ID: 116100

Updated: May 29, 2013

Contributed by Aastha Chaudhary, Sid Chandrachud, and David Houck, Cisco TAC Engineers.

   Print

Introduction

This document describes possible issues caused by event action overrides on the Cisco Intrusion Prevention System (IPS) and offers recommendations to tune and troubleshoot your installation.

Note: Event action overrides are global actions taken on signatures based upon a risk rating. As with any global configuration, take great care with configuration changes and additions.

Event Action Override Problems

Description

Event action overrides add additional actions to a signature event when that event falls within a specified risk rating range. Use event action overrides carefully. Iif you create an override with a wide risk rating range for an event that is triggered frequently (especially specific, expensive actions, such as IP logging actions), you might cause problems.

Impact

Excessive writes to the event store are typically associated with high CPU utilization and general unresponsiveness of the sensor to management access tools such as the command-line interface (CLI) and the Cisco IPS Device Manager (IDM).

IP Logging Actions and File Descriptors

A file descriptor is a data structure used by a program in order to get a handle on a file; well-known descriptors are 0,1,2 for standard in, standard out, and standard error. A file descriptor is created when a process opens a new file or socket.

If you create an event action override for an IP logging action such as log-attacker-packets, log-pair-packets, or log-victim-packets, this might exhaust the pool of file descriptors; overall sensor performance might be negatively affected and the sensor may not function properly.

SNMP Trap Actions and Event Action Overrides

A signature that has only the single action of request-snmp-trap also generates an alert event that is written to the event store. So, excessive firing of the Simple Network Management Protocol (SNMP) trap action might also trigger the same problems seen with excessive produce alert actions.

Actions for Normalizer Engine Signatures

Do not add any action that causes event store writes (such as produce alert, request-snmp-trap, or log-actions) to Normalizer signatures. This applies to all 1200-1330 range signature IDs.

Except for brief troubleshooting scenarios, you should not use event action overrides for the Normalizer engine signatures. This can be particularly problematic in:

  • highly fragmented IP scenarios (due to the 1200-range signatures)
  • heavily out-of-order (ooo) TCP scenarios (1300-range signatures)

For example, an event action override that causes a write to the event store for every ooo TCP packet can cause resource and utilization issues.

Event Action Overrides with Risk Rating of 0-100

In general, avoid event action overrides with a risk rating of 0-100 because the low rating can put your sensor at risk of failure in certain circumstances.

Meta component signatures often fire for seemingly benign (and common) types of traffic. Meta signatures look for a combination of one or more Meta component signatures to trigger before the parent Meta signature fires an alert. Meta component signatures, by default, have no actions associated with them; this is intentional because they frequently match on common traffic. Meta component signatures have a default base risk rating of 15. In order to exclude capture of these signature matches in an event action override, Cisco recommends that you do not use a risk rating lower than 25 when you create an event action override; that is, the risk rating should not be below 25-100.

Verify IPS Utilization

Commands

Note: Use the Command Lookup Tool (registered customers only) in order to obtain more information on the commands used in this section

Enter the show statistics virtual-sensor command on the CLI in order to look for the inspection load percentage:

sensor# show statistics virtual-sensor | inc Load
Processing Load Percentage = 100

In IPS Versions 7.0(8)E4 and 7.1(6)E4, the show inspection-load command has been added:

sensor#  show inspection-load history
sensor 10:17:57 UTC Mon Apr 05 2013

This is example output from that command:

116100-trouble-event-action-01.png

A very high load percentage (90% or higher) might indicate that there are excessive events triggered by event action overrides. Refer to the logs in order to further confirm this possibility.

Logs

The main indicator of excessive event action overrides is rapid event store wrapping, as seen in this example main.log file:

25Jan2010 05:13:08.326 19.897 sensorApp[18316] IdsEventStore/W errWarning - 
the event store wrapped around [IdsEventStore::writeEvent(), index = 19530]
25Jan2010 05:32:05.751 85.031 sensorApp[18316] IdsEventStore/W errWarning -
the event store wrapped around [IdsEventStore::writeEvent(), index = 19529]
25Jan2010 05:50:45.442 4.989 sensorApp[18316] IdsEventStore/W errWarning -
the event store wrapped around [IdsEventStore::writeEvent(), index = 19530]
25Jan2010 06:08:59.281 70.143 sensorApp[18316] IdsEventStore/W errWarning -
the event store wrapped around [IdsEventStore::writeEvent(), index = 19529]
25Jan2010 06:25:40.923 34.562 sensorApp[18316] IdsEventStore/W errWarning -
the event store wrapped around [IdsEventStore::writeEvent(), index = 19531]

In general, event store wrapping that occurs more often than once an hour may indicate a problem. In some scenarios, the wrapping is so excessive that it may occur many times within a minute. There are many variables, such as the overall performance capability of the platform, to consider.

Troubleshoot

Determine what type of event, traffic, or action is causing the event action override problem. Is it a produce alert, IP logging, Normalizer signature, or Meta component signature?

  • If it is a 'chatty' signature and you determine the signature creates false positives for events, write an event action filter (EAF).
  • For IP logging, Cisco recommends you avoid EAFs or use EAFs with caution and with a complete understanding of the risks.
  • Normalizer signatures and Meta component signatures should not have an alert action except for temporary troubleshooting scenarios.

Related Information

Updated: May 29, 2013
Document ID: 116100