Data model searches with tstats get stuck on "Parsing job"

Applicability

Description

It was reported that ad hoc tstats searches against the Endpoint.Processes data model remained in Parsing job for approximately two minutes before returning results.

Example affected search:
| tstats summariesonly=true count from datamodel=Endpoint.Processes where earliest=-1h

The issue affected short time ranges as well, including 15-minute searches. The actual tstats command execution was not the bottleneck; the delay occurred before the search execution phase.
The customer initially observed acceleration-related skipped jobs.

After increasing Data Model maximum concurrent summarization searches and rebuilding the data model, skipped acceleration jobs stopped.

However, tstats searches still spent 90–140 seconds in Parsing job.

Investigation

1.. Confirmed that Data Model acceleration contention was present initially:

Endpoint Data Model had been building with no summary data.
Scheduler logs showed skipped acceleration searches due to the maximum concurrent auto-summarization-search limit.
Increasing the Maximum Concurrent Summarization Searches setting and rebuilding the model eliminated skipped summarization jobs.

2.. Confirmed that acceleration tuning was not the complete resolution:
Despite no further skipped acceleration searches, the following search still took approximately 110–140 seconds:
  | tstats summariesonly=true count from datamodel=Endpoint.Processes where earliest=-15min

3.. Reviewed Job Inspector and search.log:
Job Inspector execution-cost components accounted for only approximately one second, leaving most of the elapsed time unexplained.
search.log showed extended lookup-expansion activity and repeated KV Store retries.

4.. Identified KV Store HTTP 429 errors:
  KVServiceClient ... Received HTTP 429. Retrying #1

5.. Identified Splunk Security Essentials as the source of the affected KV Store activity:
  KVServiceClient ... _appId=Splunk_Security_Essentials ...

6.. Confirmed prolonged lookup expansion:
  SearchEvaluatorBasedExpander ... Lookup expansion took 65600 ms

7.. A separate source of KV Store rate-limit errors was also identified from the Splunk Add-on for Datadog checkpointer. Increasing its polling interval stopped those errors, but did not resolve the tstats parsing delay.

This confirmed that the remaining parsing delay was associated with Splunk Security Essentials lookup expansion.

Resolution

1.. Collect the complete search.log from a delayed search through Job > Inspect Job.

2.. Search the log for:
KVServiceClient
HTTP 429
SearchEvaluatorBasedExpander
_appId=Splunk_Security_Essentials

3.. If Splunk Security Essentials is not required, remove/uninstall the app through the normal change-control process.

4.. If Splunk Security Essentials is required, review its enabled automatic lookups and KV Store-dependent features. Reduce, disable, or scope the affected lookup functionality as appropriate.

5.. Retest the original tstats search.
Validated result from this case:
| tstats summariesonly=true count from datamodel=Endpoint.Processes where earliest=-1h

Before remediation: approximately 140 seconds
After removal of unused Splunk Security Essentials: under one second

Related Links

Manage data models
Accelerate data models
View search job properties with the Job Inspector