Require alert is not triggered in Splunk.

Applicability

Description

The logs showed the alert/saved search was triggered successfully at the scheduled time, but no emails were received as per the alert actions set, even when the alert was triggered.

02-16-2022 21:31:45.988 +0000 INFO  SavedSplunker - savedsearch_id="nobody;xxxx_custom_SAP;Splunk SAP Alert Process P1", search_type="scheduled", user="t983360", app="xxxx_custom_SAP", savedsearch_name="Splunk SAP Alert Process P1", priority=default, status=success, digest_mode=0, scheduled_time=1645046100, window_time=0, dispatch_time=1645046738, run_time=362.251, result_count=0, alert_actions="", sid="scheduler__t983360_c3luZ2VudGFfY3VzdG9tX1NBUA__RMD5f33dcaeaadb81d04_at_1645046100_35521", suppressed=0, fired=0, skipped=0, action_time_ms=2, thread_id="AlertNotifierWorker-41", message="", workload_pool=""

Investigation

Run the below search from the SH where the alert was configured:

index=_internal source=*scheduler.log savedsearch_name="Name of your saved search"

Running this query at the precise time when the alert is triggered would give us the information on the saved search..
Fields and values to consider:

* result_count
* alert_actions
* status 

To reproduce the issue:

• Create a simple alert for a search having 4-6 events, which would trigger alert-action(send email).
• Also create a simple alert for a search with no events, with the same alert-action and trigger conditions previously.
• Observe the alert trigger in the scheduled time for both sample alerts:

Case 1: result_count=4, alert_actions="email" which implies the trigger conditions were met and alert-actions(send email) is success.
Case 2: result_count=0, alert_actions="" which implies the trigger conditions were not met and no alert-actions were done, since the alert triggering condition set were "greater than 0" for the number of results. Here, result_count=0 contradicted the alert trigger conditions set.

Conclusion:

• Based on the results from the scheduled search and the alert trigger condition selected a better idea of why the alert is triggered but no alert actions made can be cleared.
• If the alert condition that was selected is "When the number of results is greater than 0" and don't see any events when the search is executed, it could mean that there is an issue with the search the alert runs, that the events the alert should look for are not being recorded. 
• Hence, if the alert conditions: "send mail when the result count is greater than 0"; implies when the alert is triggered it will look for the relevant events and send mail as per the trigger condition set (ex. when result_count>0)
• On the contrary with the same given conditions, when a search does not find any events, the results_count would be 0 and therefore, no alert(email) will be triggered, because the conditions are not met. (ex. when result_count<=0)

Resolution

In this case "result_count=0", implies no events were recorded for the given search, the alert got triggered as per schedule, but no "alert_actions"(send email) could be made, as the result count is 0, and contradicts the alert triggering conditions set for the saved search, here for "Splunk SAP Alert Process P1", "Trigger alert when number of results is greater than 0". 
Hence, this also concludes the search used did not find any events in first place.
 

Related Links

Alert Trigger Conditions:
Community reference: Alert not triggering