SVC Utilization details required for scheduled searches

Applicability

Description

In CMC --> workload--> "SVC usage per hour by top 10 searches", SVC utilization for only top 10 searches. Post drilldown of that panel, the svc utilization only for 115 searches but, have a more than 800 schedule searches.

Investigation

1.) The Panel (SVC usage per hour by top 10 searches), Noticed that the svc utilization data for the scheduled searches has been pulling from the "index=summary" and source="splunk-svc-consumer". Need information for why 115 searches svc information has been ingested to summary index instead of 800+ searches.

2.) The below query can help to monitor the resource usage of your scheduled searches.

SPL Search 1:

| rest /services/saved/searches | search is_scheduled=1 | table title, eai:acl.app, eai:acl.owner, cron_schedule, dispatch.earliest_time, dispatch.latest_time, dispatch.time, dispatch.total_run_time, dispatch.total_run_time_secs SPL

SPL Search 2:

| rest /services/saved/searches | where is_scheduled=1 | join type=left savedsearch_name [ search index=_internal sourcetype=scheduler | stats sum(run_time) as total_run_time by savedsearch_name | eval svc_usage=total_run_time/60 ] | table title, svc_usage

To get a history of scheduled searches, you can query the internal logs: SPL Search:

index=_internal sourcetype=scheduler | table _time user savedsearch_name status scheduled_time run_time result_count

4.) Ensuring that the time range for your search is set correctly. If the time range is too narrow, it might not capture all relevant events.

5.) Verify that the data is being ingested into the "summary" index correctly. If there are issues with data ingestion, the summary index might not have all the expected data.

6.) If scheduled searches are being used to populate the summary index, checking if these searches are running as expected and not failing or being skipped.

7.) Also, ensure that you have the necessary permissions to access the "summary" index and the "splunk-svc-consumer" source.

Resolution

-This is an expected behaviour.

- In a non-UF instance, the introspection is run every 10 seconds, so a search would have to take longer than that to generate a "pct_cpu". The value is configurable, though making it smaller will inevitably hit performance overall, so should be viewed with great caution.

- Existing searches power the `SVC Usage by Search Type` for accuracy.

- Check if the attribute searches spanning < 10s.

- Consider if require changes in the "_instrospection" collection machinery in Splunkd or if need to augment existing audit logs to have searches report search execution on completion.

- One solution to this issue is using search seconds to attribute “work” to every search process regardless of how long that search took to complete. It is not currently possible to report that in terms of SVC but it is arguably a much better measurement as it takes other components into account such as I/O wait.

- It is not currently possible to report CPU utilization at resolutions higher than 10s without changes to the splunkd reporting components.

- Also, The existing searches powering the `SVC Usage by Top 10` for accuracy and Understand that correct attribute searches without the ‘search_app’, ‘search_label’ and ‘search_user’ fields.

Note: Again, this will only flag searches according to data which is known to be inaccurate with respect to the actual amount of CPU required. The existing Top 10 X by search seconds is an objectively better measurement because it does not rely on sampled CPU data but each search reports the real time required to complete.