Failing throttle on Search Head Cluster

Applicability

Splunk Cloud Splunk Enterprise Search Head Cluster

Description

The issue pertains exclusively to a highly specific scenario, contingent upon two specific conditions being met:

1. The environment includes a Search Head Cluster (SHC).
2. An alert is configured to run at one-minute intervals.

Within the SHC an alert configured to operate at one-minute intervals. Despite the attempt to suppress the alert by implementing a throttle and saving the throttling setting, the alert continues to trigger at least twice. 

The observed behaviour appears inconsistent, manifesting solely right after the application and preservation of the updated throttling setting.
 

Investigation

It is important to refresh how a SHC operates, specifically emphasising its mechanism of replicating information across all members.
This replication extends to known components such as the KV store and the Search bundle.
Notably, this principle also applies to the replication of suppression settings within the SHC.
All suppressions are saved into a .csv file $SPLUNK_HOME/var/run/splunk/scheduler/suppression/<file_name>.csv

The greater the number of suppressions set, the larger the file becomes, consequently resulting in an extended duration for replication across all members of the SHC.
The sequence of operations unfolds as follows:
The captain delegates the search, SH executes the search, triggers the alert, adds the suppression, and subsequently initiates the replication of the suppression. Use the following search to check a specific alert and throttling:

index=_internal sourcetype=scheduler savedsearch_name=nameOfAlert

The results should be arranged to show the start of the alert and throttling for better clarity and understanding.

Note: Timestamps for the following logs are an example. 

01-08-2024 11:46:19.744 +0000 INFO SavedSplunker - savedsearch_id="nobody; nameOfAlert", search_type="", search_streaming=0, user="userName", app="search", savedsearch_name="nameOfAlert", priority-default, status-success, digest_mode=0, durable_cursor=0, scheduled_time=1704714360, window_time=600, dispatch_time=1704714377, run_time-1.753, result_count=2, alert_actions=*, sid="scheduler_RANDOMNpZS10ZWxIbwV0cnktYWx1cnRpbmcsearch_RMD57406f60_85217_RANDOM", suppressed=1, fired=1, skipped=1, action_time_ms=5, thread_id="AlertNotifierWorker-0", message="", workload_pool="standard_perf"
host = sh-i-001.stackname.splunkcloud .com | source = /opt/splunk/var/log/splunk/scheduler.log sourcetype = scheduler

01-08-2024 11:47:19.854 +0000 INFO SavedSplunker - savedsearch_id= "nobody; search; nameOfAlert", search_type="", search_streaming=0, user="userName", app="search", savedsearch_name="nameOfAlert", priority-default, status-success, digest_mode=0, durable_cursor=0, scheduled_time=1704714420, window_time=600, dispatch_time=1704714437, run_time-1.715, result_count=2, alert_actions=", sid="scheduler_RANDOMcnktYWx1cnRpbmc_search__20_76385_RANDOM", suppressed=1, fired=1, skipped=1, action_time_ms=3, thread_id="AlertNotifierWorker-0", message="", workload_pool="standard_perf"
host = sh-i-002.stackname.splunkcloud .com / source = /opt/splunk/var/log/splunk/scheduler.log sourcetype = scheduler

From the logs above, the alert was triggered twice result_count=2 one was fired and one suppressed suppressed=1, fired=1 
Both of the logs above each come from a different search head and there is exactly 1 minute difference because as mentioned before the alert is set to run every minute. The reason for the alert firing is that the replication of the suppression has not yet reached these two Search Heads (SHs) within the cluster.

The 3rd log again just one minute apart will have similar results result_count=2 however suppressed=2, fired=2 This indicates both results of the alert were successfully suppressed. 

01-08-2024 11:48:19.223 +0000 INFO SavedSplunker - savedsearch_id= "nobody; search; nameOfAlert", search_type"*, search_streaming=0, user= "userName", appe"search", savedsearch_name="nameOfAlert", priority default, status-success, digest_mode=0, durable_cursor=®, scheduled_time=1704714480, window_time=600, dispatch_time=1704714497, run_time=1.836, result_count=2, alert_actions"", sid="scheduler_RANDOMW5mLWNpZS10ZWx1bWV@cnktYWxIcnRpbmc_search__RMD57406f cbb5655a518_at_1704714480_58986_71C4975-0443-RANDOM", suppressed=2, fired=0, skipped=2, action_time_ms=2, thread_id="AlertNotifierWorker-0", message="", workload_pool-"standard_perf"
host = sh-1-003.stackname.splunkcloud .com source = /opt/splunk/var/log/splunk/scheduler.log sourcetype = scheduler

Currently, the suppression setting has successfully replicated to all members of the SHC, and the throttling functionality aligns with the intended specifications.

Resolution

Due to the inherent time required for the replication of suppression settings among all members of the SHC, it took 2 minutes in the provided example. Notably, as the number of suppression settings increases, the duration for replication also extends, on Splunk Enterprise (on-prem) custom configuration of replication may also play a role in how long it takes for replication to complete.
To address this situation, the recommended resolution is to modify the alert setting to run every 2 minutes instead of the initial one-minute frequency.  If it becomes evident that the replication exceeds the 2-minute example in this scenario, make appropriate adjustments to the alert settings accordingly to fit the specific scenario.

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release