Cluster Bundle Push Is Getting Failed and Causing the Peers to Disconnect.

Applicability

Description

Environment:   -Cluster Master - Splunk Enterprise - 7.2.6/9.2.1
                        - 2 Indexers - Splunk Enterprise - 7.2.6/9.2.1

Platform - Both indexers and cluster master are running in Linux.

Trying to push the bundle from cluster master with new index detail, but it is getting failed with the below error:
Trying to add new peero to the cluster, but it is unable to join and splunkd shutdown.
 

ERROR loader [472966 MainThread] - Failed to download bundle from the cluster manager, err App='system' with replicated index='_introspection' is neither in the bundle downloaded from 
master nor managed by local deployment client. Either define this index at the master or specify repFactor=0 on peer to 
skip replication.

Investigation

Verified the clustering page Settings > Indexer Clustering of cluster master and found that the indexer was disconnected. From the CM logs, found the above error as well as some connection refused with the indexers after the cluster bundle push.

Performed a rollback of cluster bundle and then restarted the CM/indexers individually which worked and cluster came back online.

Also, verify the _introspection index or others may apply in the CM/indexers which was not explicitly mentioned either for replication across the cluster or individually causing the cluster bundle push failed, If an index is created through the GUI, the index definition is saved in $SPLUNK_HOME/etc/system/local/indexes.conf. Clustering of the Indexers becomes a problem because of these local indexes.

Moreover, few indexes' directories are owned by root user which also causes the permission issue for a bundle.

 

Resolution

Change the permissions of indexes' directories to Splunk user and add the below stanza in indexes.conf along with new index details while pushing the cluster bundle.

[_introspection]
repFactor = auto

NOTE: If you are adding an old Indexer to the indexer cluster, make sure to get rid of indexers.conf files from paths like: 

• $SPLUNK_HOME/etc/system/local
• $SPLUNK_HOME/etc/apps/search/local

Clustering of the Indexers becomes a problem because of these local indexes. Indexes must be managed by the cluster master using master-apps.

Related Links

Splunk Answer:
Instrospection-replication