search_telemetry is missing fields when exceeding 'max_count' for saved searches

Available Languages

Updated:September 29, 2026
Document ID:SPLK046630

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Applicability

Splunk Enterprise 9.0.3 Splunk Enterprise 9.1.2

Description

In Splunk Enterprise version 9.0.3 and 9.1.2, when referring to "index=_introspection sourcetype=search_telemetry" where events are more than "max_count", phase_0 and sourcetype information is missing:

phases: { [-]
phase_1: { [+]
}
phases_count: 2
}sourcetypes: { [-]
}

 


Investigation

Backtracking of events for missing search_telemetry

Supporting artifacts from dispatch folder

1

Background - Search Telemetry for distributed search: At the end of search execution, the search head fetches the remote telemetry from all the associated indexers/search-peers and aggregates into search_telemetry.json on SH, which is later ingested into _introspection index. Also, the individual remote search telemetry fetched from indexers is available in SH's dispatch folder.

-

2

The SH’s dispatch folder indicates that, the remote search telemetry is not fetched from indexers and as a result the search_telemetry.json is incomplete (missing Phase0, sourcetype, etc)

Dispatch folder is missing remote_search_telemetry folder and no traces of the following log message indicating the downloading of remote telemetry ReducePhaseExecutor [612022 phase_1] - Downloading all remote search_telemetry.json files took 0.004 seconds

3

Remote Telemetry is not fetched as “remote_event_providers.csv“ file is not written to dispatch folder(this file has the list of peers from which the remote artifacts has to be fetched).

Missing the following log message indicating that “remote_event_providers.csv“ is written to dispatch folder. INFO ResultsCollationProcessor [612022 phase_1] - Writing remote_event_providers.csv to disk

4

“remote_event_providers.csv“ file is written to dispatch folder after collecting the results from all the peers. But before this could happen, the query execution is short circuited for the following limit:

05-08-2024 00:12:09.832 INFO TimelineCreator [837464 phase_1] - Job truncated due to max_count=500000 reached

Missing the following log message indicating that query has executed completely and the query_finished flag is set to true

DEBUG DispatchExecutor [612022 phase_1] - Source Processor=resultsCollationProcessor set qf=1

Summary:

The search telemetry is incomplete because remote telemetry from indexers is not being fetched, as the search has hit the “max_count” limit(present in limits.conf) and the query execution is short circuited.

 


Resolution

  • Currently, from Splunk Enterprise version 9.2, WARN is raised  whenever the remote_event_providers.csv file is missing from the dispatch folder.
  • There is no current fix to this issue, however fix request has been raised under SPL-256670

Workaround:

  • Currently identified workaround is to limit the scheduled search to return results within max_count range.

WARN alert when max_count is reached: SPL-249789

fix for this issue: SPL-256670

Revision History

Revision Publish Date Comments
1.0
29-Sep-2026
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products

Related Articles