Health Report shows stale warning for "Searches Delayed": The percentage of non high priority searches delayed (A%) over the last 24 hours is very high

Applicability

Description

Users found no searches were delayed over last 24 hours, but the health warning in the UI continues to turn red for many days: 

The percentage of non high priority searches delayed (A%) over the last 24 hours is very high and exceeded the red thresholds (B%) on this Splunk instance. 
Total Searches that were part of this percentage=X. Total delayed Searches=Y

Investigation

Step 1. Verify search delay or skipped:

Run the SPL search for the last 24 hours:

index=_internal host=sh* sourcetype=scheduler* OR source=*scheduler* status=deferred OR status="skipped" OR status="continued"

If there is no result coming from the above search, then it's impacted by this issue.  Otherwise, further investigation is needed for verification.

Step 2: Check Health.log 

index=_internal source= */health.log* component=PeriodicHealthReporter feature="Searches Delayed" color=red OR color = yellow

From the above search, the log result will look similar like the following example: 

• due_to_threshold_value= 20 >> threshold % to trigger this alert. 
• measured_value=3  >> The number of delayed searches. 

02-14-2025 15:57:45.547 +0900 INFO  PeriodicHealthReporter - feature="Searches Delayed" color=red indicator="percent_searches_delayed_non_high_priority_last_24h" due_to_threshold_value=20 measured_value=3 reason="The percentage of non high priority searches delayed (25%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=12. Total delayed Searches=3" node_type=indicator node_path=splunkd.search_scheduler.searches_delayed.percent_searches_delayed_non_high_priority_last_24h

According to the search result from  health.log, the same host has consistently reported measured_value=3 and due_to_threshold_value=20 for several days, with a status of color=red.

Given there was no delayed search from Step 1 above, this indicate the health warning is stale and incorrect. 



## Additional Information about "Searches Delayed" Warning: 

• Non-High priority searches: Any splunk saved search with priority set to "default". 
• Window of measure: 24 hours
• Description: Represents the percentage of the delayed non-high priority searches to the total non-high priority searches that ran over the last 24 hour period.
  Default thresholds:
      * Green - percentage of the delayed non high priority searches to the total non high priority searches that ran over the last 24 hour period <= 10%
      * Yellow - percentage of the delayed non high priority searches to the total non high priority searches that ran over the last 24 hour period > 10% and <= 20%
      * Red - percentage of the delayed non high priority searches to the total non high priority searches that ran over the last 24 hour period > 20%

Resolution

If the health warning is confirmed to be stale, follow these workarounds:

Workaround Option 1:
Manually restart splunkd for the search head which reported the stale warning. 

Workaround Option 2:

Manually update the health indicator in Web UI and re-saved it. 

OR refresh the health indicator using the following REST commands (example assumes the default threshold of red=20):

curl -k -u : https://:/services/server/health-config/feature:searches_delayed -d indicator:percent_searches_delayed_non_high_priority_last_24h:red=100curl -k -u : https://:/services/server/health-config/feature:searches_delayed -d indicator:percent_searches_delayed_non_high_priority_last_24h:red=20

Long term solution is being investigated under Splunk Internal Jira: SPL-271620. 

Related Links

Savedsearchesconf
#Set_thresholds_using_Splunk_Web