Splunk On-Call High Alert Lag and Auto-Pause, what's this?

Applicability

Description

Splunk On-Call High Alert Lag and Auto-Pause:

What is Auto-Pause?

Auto-Pause is a security event that occurs when your On-Call instance has reached a total of 5,000 or more active incidents, defined as incidents that are in a Triggered or Acknowledged state. When Auto-Pause is triggered, all alert processing is halted and begin to queue all incoming alerts to the instance. Alert Lag may soon follow due to this backlog. If your instance exceeds the 5,000 active incident threshold and Auto-Pause is triggered, communication with On-Call Support will be required to re-enable your account.

What is a High Alert Lag?

High Alert Lag occurs when alerts are sent from a monitoring tool to your On-Call instance at a rate faster than our system can process them. This can happen without an organization being paused and can be caused by alerts that don’t create incidents, like a mass of INFO or RESOLVED alerts. Our alert processing limit is 150 alerts per minute (2 alerts per second).

What Causes Auto-Pause and Alert Lag?

Both Auto-Pause and Alert Lag are often a symptom of a repeating loop sending alerts from the monitoring tool upstream of Splunk On-Call and ask to investigate upstream monitoring tools as a potential cause of the issue. Auto-pause specifically can also be triggered because too many alerts have piled up without being resolved, independent of an alert storm. Following best practices for incident management can help you avoid reaching these states.

Related Links

Auto Pause and High Alert Lag