On two indexers network connectivity issues were found.
As shown on the errors below on splunkd logs:
At Indexer 1:
01-19-2022 02:29:15.523 +0000 WARN BucketReplicator - Connection failed
01-19-2022 02:29:15.523 +0000 WARN BucketReplicator - Failed to replicate warm bucket bid=main~9293~520D89E5-E69A-40F5-8CE4-3E5B0B835887 to guid=7BB1759C-48A8-439D-A661-AD04695829DD host=x.x.x.x s2sport=9995. Connection failed
01-19-2022 02:29:18.933 +0000 WARN BucketReplicator - Connection failed
At Indexer 2:
01-20-2022 02:24:28.677 +0000 WARN BucketReplicator - Connection failed
01-20-2022 02:24:28.680 +0000 WARN BucketReplicator - Failed to replicate Streaming bucket bid=linux~2166~E2B743CE-1547-4860-A4B8-67C241833465 to guid=520D89E5-E69A-40F5-8CE4-3E5B0B835887 host=x.x.x.x s2sport=9995. Connection failed
01-20-2022 02:24:28.680 +0000 INFO BucketReplicator - Forcing hot bid=linux~2166~E2B743CE-1547-4860-A4B8-67C241833465 to roll for target node=guid=520D89E5-E69A-40F5-8CE4-3E5B0B835887 host=x.x.x.x s2sport=9995 reason=bucket_replication_failed Connection failed
1. Correct the network connectivity issues preventing stable connectivity.
2. Perform the changes below on the Cluster Manager's server.conf:
[clustering]
manager_uri = https://splunk-manager.splunk.svc.cluster.local:8089
mode = peerAfter changing server.conf file validate and push the the changes on all indexers.
On the Manager node, in Splunk Web, click Settings > Indexer Clustering.
On the Indexer Clustering page, click Edit > Configuration Bundle Actions.
If the changes are not replicating to all indexers, please perform a rolling-restart from the Cluster Manager:
1. Enable maintenance mode on Cluster Manager
To enable maintenance mode navigate to the Splunk bin directory:
cd $SPLUNK_HOME/splunk/bin
Then run the following command to enable maintenance mode:
splunk enable maintenance-mode
2. After enabling maintenance-mode on Cluster Manager, please restart splunk service on indexers.
3. Then, go to Cluster Manager and disable maintenance-mode from the same command line.
splunk disable maintenance-mode
After maintenance mode is disabled, wait for 5 minutes and check whether the app distributed or not in that particular indexer.
This should fix the issue, however if you still facing issues please do below steps
1. On Cluster Manager, Settings > Indexer Clustering
2. On the Indexer Clustering page, click Edit > Rolling Restart
3. Click Begin rolling restart
After all indexers restarted, check the changes distributed on all indexers.
After restart verify SF and RF is met on Cluster Manager.
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
29-Sep-2026
|
Initial Release |