How to find if there are delays in start time for scheduled searches

Applicability

Description

​​​​

In  this example there is a saved search called "2steps Alert" and the following is required. 

• What time the searched was scheduled to run, for exampe: the scheduled_time.
• What time the search actually runs, for example: the dispatch_time.
• The difference between the scheduled_time and the dispatch_time.
• How long the search ran, for example: the run_time.
• Status of the search.
• How many events did the search find.

Resolution

To be able to determine all of the above we need to review the Splunk scheduler log with the following search:

index=_internal sourcetype=scheduler savedsearch_name="2steps Alert"
| eval time=strftime(_time,"%Y-%m-%d %H:%M:%S") | eval delay_in_start = 
(dispatch_time - scheduled_time)
| eval scheduled_time=strftime(scheduled_time,"%Y-%m-%d %H:%M:%S")
| eval dispatch_time=strftime(dispatch_time,"%Y-%m-%d %H:%M:%S")
| table savedsearch_name,, scheduled_time, dispatch_time, delay_in_start, 
run_time, status, result_count

Let us break down the search to help understand what it is doing.  To do that let us put a line number in each part of the search. Please note do not run this search with the line numbers , this is just for educational purposes.

Line 1: index=_internal sourcetype=scheduler savedsearch_name="2steps Alert"
Line 2:  | eval time=strftime(_time,"%Y-%m-%d %H:%M:%S") | eval delay_in_start = (dispatch_time - scheduled_time)
Line 3:  | eval scheduled_time=strftime(scheduled_time,"%Y-%m-%d %H:%M:%S")
Live 4:  | eval dispatch_time=strftime(dispatch_time,"%Y-%m-%d %H:%M:%S")
Live 5:  | table savedsearch_name, scheduled_time, dispatch_time, delay_in_start, run_time, status, result_count

Line 1
Searching the Splunk scheduler. Specify the _internal index , the sourcetype of scheduler and lastly the savedsearch_name. 

Line 2
This is where a new field called "time" is created in the search results. The strftime value of the _time value is used and the date and time are in a more easier human format for reading.  The second part is where "delay_in_start" is created and here it is subtracted  the scheduled_time from the dispatch_time to provide the delay_in_start. 

Line 3 and 4
Update the date and time into a more easier human format for reading.

Line 5
Takes all results and tables the fields savedsearch_name, scheduled_time, dispatch_time, delay_in_start, run_time, status, result_count

This screenshot shows the results.
 

With this information we have a better understanding of what is is happening with our saved  searches.