Detector fails to trigger alerts due to data delay exceeding max delay

Applicability

Description

A detector fails to trigger incidents or send notifications to integrations such as Slack or Opsgenie despite conditions appearing to be met in the chart.

This issue typically occurs when the incoming data is sparse or late, resulting in a data lag that exceeds the analytics evaluation job's default max delay.

Consequently, the detector does not process the data points for the evaluation window.

Investigation

1.Identify the detector ID and the current job ID: 

<realm>-sfc appsdb detector get HGA9-h2A0BA | ts -i

2. Search for the job active during the specific failure window: 

<realm>-sfc flow search 'originId:HGA9-h2A0BA' -l 10 -f originalStartTime -f lastTimestamp | ts -i

3. Check for dropped datapoints using the identified job ID: 

<realm>-sfc flow debug <jobID>

4. Verify if the value for Others Dropped is greater than 0.

5. Compare the Max Delay value against the actual data lag observed in the stream.

6. Verify no muting rules were active during the failure window: 

<realm>-sfc -O <orgId> mb s 'sf_type:AlertMuting AND sf_startTime:[<start_ms> TO <end_ms>]' -f sf_description -f sf_filters -f sf_startTime -f sf_stopTime | ts -i

Resolution

1. Open the Splunk Observability Cloud user interface.
2. Navigate to Alerts & Detectors and select Detectors.
3. Search for and select the affected detector.
4. Click Edit Detector.
5. Navigate to the Alert Settings or the SignalFlow editor.
6. Locate the settings for Max Delay.
7. Set a manual Max Delay to a higher value (e.g., 1 or 2 minutes) to accommodate the observed data latency.