Cluster Manager's Indexer Clustering console shows internal indexes only, with SF/RF being met, while custom indexes are not listed. Btool output on indexers also does not pick up custom indexes, even though these exist in the indexers' peer-apps directory. Searches produce 0 events, but indexers do appear to be indexing data.
In this scenario, a Search Head Cluster was searching two separate indexer clusters. One of the clusters was not returning search results, and its Cluster Manager showed only two internal indexes on its main Indexer Clustering dashboard, with Search Factor and Replication factor being met.
A btool output from the impacted indexers' CLI also captured only internal indexes.
$SPLUNK_HOME/bin/splunk btool indexes list --debug
Additionally, on startup, indexers from the affected cluster showed the following messages:
05-03-2024 12:30:35.136 -0500 INFO ServerRoles [3444708 MainThread] - Declared role=cluster_slave.
05-03-2024 12:30:35.177 -0500 INFO CMSlave [3444708 MainThread] - The cluster peer will continue startup. Bundle has been moved to slave-apps.
05-03-2024 12:30:35.177 -0500 INFO loader [3444708 MainThread] - Bundle is already available in the slave-apps. Continuing with startup.
05-03-2024 12:30:37.544 -0500 WARN ApplicationManager [3444768 HTTPDispatch] - Found duplicate app(s): _cluster. This is expected if you push an app from the cluster manager that the peer needs to update. Because apps under $SPLUNK_HOME/etc/slave-apps are read-only, the peer creates a second copy of the app under $SPLUNK_HOME/etc/apps and updates it there. For further information, see the Splunk documentation.
Yet:
All indexers had the latest common bundle from the cluster Manager. Removing this bundle and pushing the Cluster Manager's bundle did not change the behavior.
It was also noticed that the impacted cluster had been upgraded from an earlier, pre-9.x version of Splunk to 9.1.1 while the functioning cluster was relatively new, with 9.1.1 being its first version. The two clusters' server.conf [clusteirng] stanzas were configured differently:
search heads' $SPLUNK_HOME/etc/system/local/server.conf:
[clustering]
manager_uri = https://managerhostname:8089
mode = searchhead
Cluster managers' (both clusters) $SPLUNK_HOME/etc/system/local/server.conf:
[clustering]
mode = manager
Non-functioning Indexers' $SPLUNK_HOME/etc/system/local/server.conf:
NOTE: with clustered indexers, any settings applied via CM apps will override the same attributes in $SPLUNK_HOME/etc/system/local -- for all other instances, $SPLUNK_HOME/etc/system/local configurations take precedence)
[clustering]
master_uri = https://managerhostname:8089
mode = slave
In this example, it was determined this was likely caused by a corrupted bundle, but it was not known how the bundle became corrupted.
The issue was resolved by deleting/renaming the bundle on the Cluster Manager's $SPLUNK_HOME/var/run/splunk/cluster/peer-bundles directory, then restarting Splunk so the CM would generate a new bundle. It is possible simply running this command from the CM's CLI to push a new bundle may have been a better choice:
$SPLUNK_HOME/bin/splunk apply cluster-bundle
It is also suggested to homogenize the clustering nomenclature across all configurations to eliminate mixed use of "master/manager" and "slave/peer" across the deployment as a possible contributing factor.
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
29-Sep-2026
|
Initial Release |