Indexers Rolling Restart Without Admin Interactions When Cluster Master Restarts

Applicability

Description

It started when cluster master replacement was performed and a new cluster manger was booting. Immediately after the new CM was up running, all indexers in the cluster started restarting, generating messages that indicate they are rolling restarting.  This behavior spread across entire cluster until the very last member of the cluster finished restarting. During this event, search slowness was observed because some indexers were not available until the RR finished. Disk usage of cold path went up by 5% in the end of the RR since all hot buckets rolled to warm state, and warm buckets reached its size limit and rolled to cold.

Investigation

We investigated the logs from cluster manager and indexers. We were also informed that there was cluster manager switching from active to standby. This issue occurred when standby was just up running. 

From the Standby CM logs provided, we can see that the CM started, and found an active bundle REDACTED2E6E and when the indexers start joining, from the very first indexer "Bundle Mismatch" is detected:

REDACTED_DATE 16:43:25.239 +0900 INFO  ServerConfig [0 MainThread] - My GUID is REDACTED_GUID....
....
REDACTED_DATE 16:43:35.995 +0900 INFO  BundlesUtil [52782 MainThread] - Found the '_active' file - REDACTED_ID.bundle_active
REDACTED_DATE 16:43:35.995 +0900 INFO  BundlesUtil [52782 MainThread] - Start scan to find the _active file on the cluster manager.
REDACTED_DATE 16:43:35.995 +0900 WARN  BundlesUtil [52782 MainThread] - Populated manager-apps location not found. Master-apps is deprecated, but falling back to: /opt/splunk/etc/master-apps
REDACTED_DATE 16:43:35.995 +0900 INFO  BundlesUtil [52782 MainThread] - Bundle found: /opt/splunk/etc/master-apps/SA-IndexCreation/default/transforms.conf
REDACTED_DATE 16:43:35.995 +0900 INFO  CMBundleMgr [52782 MainThread] - apply bundle status transitioning from='None', to='None'
REDACTED_DATE 16:43:36.023 +0900 INFO  BundlesUtil [52782 MainThread] - Found the '_previousActive' file - REDACTED_ID.bundle_previousActive
REDACTED_DATE 16:43:36.023 +0900 INFO  BundlesUtil [52782 MainThread] - Start scan to find the _previousActive file on the cluster manager.
REDACTED_DATE 16:43:36.023 +0900 INFO  CMBundleMgr [52782 MainThread] - reencryptBundleIfRequired: Nothing to do. Encryption already done with current pass4SymmKey for bundle="/opt/splunk/var/run/splunk/cluster/remote-bundle/REDACTED_ID.bundle"
REDACTED_DATE 16:43:36.051 +0900 INFO  CMMaster [52782 MainThread] - Cluster manager initialized active_bundle_id=REDACTED2E6E latest_bundle_id=REDACTED2E6E
REDACTED_DATE 16:43:36.081 +0900 WARN  DistributedBundleReplicationManager [52782 MainThread] - replicationBlacklist in distsearch.conf is deprecated, use replicationDenylist instead.
REDACTED_DATE 16:43:39.725 +0900 INFO  CMMaster [54368 CMMasterServiceThread] - event=addPeer msg='Bundle mismatch; restarting peer=REDUCTEDGUID peer_name=REDUCTEDNAME02. Potential cluster manager restart while applying bundle. Assuming rollingRestart is true.'
REDACTED_DATE 16:43:39.725 +0900 INFO  CMBundleMgr [54368 CMMasterServiceThread] - apply bundle status transitioning from='None', to='Rolling restart of the peers is in progress.'
REDACTED_DATE 16:43:40.201 +0900 INFO  CMMaster [54368 CMMasterServiceThread] - event=addPeer msg='Bundle mismatch; restarting peer=REDUCTEDGUID peer_name=REDUCTEDNAME03. Potential cluster manager restart while applying bundle. Assuming rollingRestart is true.'
REDACTED_DATE 16:43:40.756 +0900 INFO  CMMaster [54368 CMMasterServiceThread] - event=addPeer msg='Bundle mismatch; restarting peer=REDUCTEDGUID peer_name=REDUCTEDNAME04. Potential cluster manager restart while applying bundle. Assuming rollingRestart is true.'........

And all the peers got the bundle mismatch clearly telling us that the ex-standby CM was having a different bundle than what all the cluster peers had

It was confirmed by the engineering team, this rolling restart was triggered by a bundle mismatch between indexers and newly started ex - standby cluster master. 

Resolution

It was found that the ex standy cluster master had discrepancy in bundle from active cluster master. This bundle mismatch caused the issue. 

You can prevent it by implementing Splunk supported cluster manager redundancy solution.  https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/CMredundancy

If that is not possible, make sure the integrity between active CM and standby CM before switching.