Search Heads unable to search Indexer Cluster due to Splunk Enterprise Version Compatibility

Applicability

Description

Splunk documentation states that the Search Heads and Cluster Manager must be on the same or higher version of Splunk in relation to the indexers. Normally, having disparity in versions will produce messages in the Splunk UI or in Splunkd.log for the affected search head indicating the version issue. However, if differences between versions is disparate enough, the Search Heads may fail to communicate with the indexers entirely, depending upon what has changed between Splunk versions.

The simple solution is to upgrade the Search Heads. However, that requires validating all apps on that Search Head will support the newer version of Splunk (which may support varying versions of python). This article offers additional validation steps in the event upgrading search heads is a significant change which requires further investigation.

Investigation

Search head splunkd.log messages below show they are failing to authenticate with the indexers:

02-20-2025 17:47:21.080 +0000 ERROR DistributedPeerManagerHeartbeat [8948 DistributedPeerMonitorThread] - Status 502 while sending public key to cluster search peer https://XXX.XXX.XXX.XXX:8089:
02-20-2025 17:47:21.080 +0000 ERROR DistributedPeerManagerHeartbeat [8948 DistributedPeerMonitorThread] - Send failure while pushing public key to search peer = https://XXX.XXX.XXX.XXX:8089 , Connect Timeout

The following event may be present on search heads:

WARN SearchMessages - orig_component="DistributedSearchResultCollectionManager" app="system" sid="SummaryDirector_<epoch.time>" message_key="DISTRIBUTED:SEARCH_UNSUPPORTED_PEERS" message=Search Head '<hostName>' running Splunk version 'N.N.N' does not support distributing searches to the following peers: {peer:<host.fqdn> version:N.N.N} {peer:<host.fqdn> version:N.N.N} {peer:<host.fqdn> version:N.N.N} {peer:hksplunk version:N.N.N}.

If this message is seen, the search head is most-likely on an older version of Splunk than the indexers or cluster manager it is reaching out to. Please skip to the resolution section for further instruction. If the above message is not seen and Splunk versioning is the same on search heads, indexers, and cluster managers, use the following instructions to enable debug logging and collect a diag. Be sure the indexers are capable of receiving internal logs from the host in question without an issue:

02-20-2025 17:47:30.397 +0000 INFO AutoLoadBalancedConnectionStrategy [15231 TcpOutEloop] - Connected to idx=XXX.XXX.XXX.XXX:9997:0, pset=0, reuse=0. autoBatch=1

The demonstrates that the issue is affecting the management port (8089) but not the indexers' data receiving port (9997). 

The indexers show no references to the search heads at all in their logs (not by IP, hostname or GUID), and the Cluster Manager (CM) only shows the record where each search head was added to the cluster.

Debug logging can verify whether the discrepancy in Splunk versions is causing this issue. Debug logging settings cannot be pushed from the deployer, and the debug logging will affect only the one search head where it is set. Splunk Documentation offers WebUI, CLI and REST API options for enabling debug logging:

From the Web UI (for clustered search heads, this will require a restart of splunkd for the affected member)

1. Using Splunk Web, go to Settings
2. If the search head is clustered, select "Show All Settings" (to avoid a restart, instead use the command line option in step 4 below)
3. Select Server Settings  and choose Server Logging from the list.
4. In the search field, type "DistributedPeer" and change the following components from WARN to DEBUG by clicking on the processor name and selecting DEBUG from the dropdown, then clicking Save 
   

   DistributedPeerManagerHeartbeat
   DistributedPeer
   DistributedPeerManager

   Or, set these via the CLI, as shown below. Once the needed data is captured change these elements back to INFO in the same manner. This is great for a clustered search head as it does not require a restart of Splunkd to revert the changes:
   

   ./splunk set log-level DistributedPeer -level DEBUG
   ./splunk set log-level DistributedPeerManager -level DEBUG
   ./splunk set log-level DistributedPeerManagerHeartbeat -level DEBUG

5. The following search will confirm these elements have been set to DEBUG mode (you may not get entries for all 3 components):
   

   index=_internal host=<search_head_whose_log_levels_were_updated> log_level=DEBUG component=DistributedPeer*

6. Allow Splunk run on for a 2-5 minutes. Perhaps run a simple search on this search head for good measure. 
7. Then, collect a diag from that search head for support. 
8. Once the diag is collected:
   1. If this is NOT a clustered search head, go back to Settings > Server Settings and change the processors back to "WARN".
   2. If this IS a clustered search head:
      1. If the CLI was used to change the logging level, use it again to revert those elements back to their default values:
         

         ./splunk set log-level DistributedPeer -level WARN
         ./splunk set log-level DistributedPeerManager -level WARN
         ./splunk set log-level DistributedPeerManagerHeartbeat -level WARN

      2. If the UI was used to change logging level, restart Splunk on that member (from Settings > Server Controls on the UI or via ./splunk restart on the CLI) . The "Show All Settings" button from step 2 in the UI can be used to configure specific settings on the current search head only (debug settings are not replicated across the cluster). Restarting splunk will revert the components in step 4 above back to their default values and will revert the Settings menu back to hiding items that will not get replicated across the cluster. As a best practice, do not leave these items visible.

The steps above should capture DEBUG messages which state why it is failing to push the public key to the indexers. The debug logs are saved in the diag for support and can also be seen by searching for them in the _internal index.

Resolution

Per Splunk documentation, the Search Heads (SH's) must be on the same or higher level of Splunk than the search peers.

If the indexers are clustered:

• The manager node must run the same or a higher version from the peer nodes and search heads.
• The search heads must run the same or a higher version from the peer nodes.
• The peer nodes must all run exactly the same version, down to the maintenance level.

Related Links

• Splunk documentation regarding version compatibility between clustered components
• Splunk Documentation regarding how to set Splunk component logging levels via UI, CLI or REST API endpoints
• Splunk Enterprise Version Compatibility - Clustered indexers