Indexer fails to start up with message "The downloaded bundle checksum doesn't match the activeBundleChecksum."

Applicability

Description

When indexer fails to start when joining a cluster with the message  "The downloaded bundle checksum doesn't match the activeBundleChecksum.", Won't start splunkd", it is most likely the version of the indexer joining the cluster is below 9.0.0.

Investigation

During the investigation, it was observed that an indexer cluster bundle file will be encrypted using server.conf:[clustering]:pass4SymmKey which is shared amongst peers and will allow peers to decrypt the received bundle.

When creating a new bundle, cluster master encrypts the bundle if all the peers joining the cluster have a version greater or equal to 9.0.0 and pass4SymmKey is set. Otherwise it creates an unencrypted bundle. If the cluster is using an encrypted bundle and a peer with a version older than 9.0.0 tries to join the cluster, CM will reject it and the peer will not be able to download the bundle and start. A message will be added to splunkd.log to highlight that:

CM logs:
04-27-2025 01:23:45.769 -0700 ERROR CMBundleStreamHandler [2468522 TcpChannelThread] - event=streamingbundle status=failure path=/home/hdjoumi/investigate/SPL-162257/repro/cluster/cm/var/run/splunk/cluster/remote-bundle/123fc4fd56789e3ac40d23e1b91cc1e1-1651063940.bundle err='Peer ip=127.0.0.1 does not handle encrypted bundle.'

Peer logs:
04-27-2025 01:23:45.786 -0700 ERROR loader [2593780 MainThread] - Failed to download bundle from the cluster manager, err="The downloaded bundle checksum doesn't match the activeBundleChecksum.", Won't start splunkd.

Resolution

To prevent the issue, you should upgrade your indexer to align with the cluster master's version. The upgraded indexer will join the cluster without any issue. 

Please also note that the peer nodes must all run exactly the same version, down to the maintenance level.