Search head cluster members keep restarting by themselves

Applicability

Description

A rolling restart was triggered manually on a 3-member search head cluster (SHC). A few seconds after the members came back from the initial restart, they unexpectedly restarted again and continued in a restart loop until their splunkd processes were terminated manually.

Investigation

The restart sequence can be traced from the members' splunkd.log. For example:

1. Initial restart:

12-10-2024 16:44:29.277 +0000 INFO  Shutdown [1895601 Shutdown] - Shutting down splunkd
12-10-2024 16:45:10.307 +0000 INFO  Shutdown [1895601 Shutdown] - Shutdown complete in 41.03 seconds
12-10-2024 16:45:17.842 +0000 INFO  ServerConfig [0 MainThread] - My GUID is DD1F406B-F031-4A66-9486-BD7775E21199
12-10-2024 16:45:17.895 +0000 INFO  loader [219549 MainThread] - Splunkd starting (build 78803f08aabb).
12-10-2024 16:45:17.842 +0000 INFO  ServerConfig [0 MainThread] - My server name is "SHCMEMBER1".

2. When the SHC members start, by default, they will contact the deployer and ask for any new app configurations that need to be applied. To determine whether an app has a new configuration, the members compare their apps' checksum with the one provided by the deployer. If the checksums differ, the members will download and apply a new baseline configuration, followed by another restart.

12-10-2024 16:45:29.658 +0000 INFO  loader [219549 MainThread] - Downloaded new baseline configuration; restarting ...

3. After the second restart, another checksum comparison was initiated. In this example, the members' checksum differed from the deployer's. This resulted in another download and restart, and so on.

Root cause:
After applying a new app configuration, the SHC members will update the app's install_source_checksum value in the <app_name>/default/app.conf file. 

In this example, an admin inadvertently put a copy of the app's app.conf file in the <app_name>/local/ directory and the file contained the checksum:

install_source_checksum = edc11e818856f250e794ef5f782a56b95a5a5e8f

Since the local directory takes precedence over the default directly, the above checksum was always used to compare with the deployer's checksum.

The <app_name>/default/app.conf contained the updated checksum, however, it was ignored. This resulted in a persistent checksum mismatch, which resulted in the restart loop.

Resolution

1. Locate all <app_name>/local/app.conf files that contain an install_source_checksum line.

The below command can be used on a Linux platform: 

find /opt/splunk/etc/apps -type f | grep local | grep app.conf | xargs grep install_source_checksum

2. Comment out all the install_source_checksum lines returned from step 1 and save the file.

3. Allow the SHC members to restart again. The restart loop will stop since the checksums are now in sync.

Tips:

Adding the following setting in the SHC members' server.conf can be used as a workaround to stop the restart loop:

[shclustering]
conf_deploy_fetch_mode = none

This workaround should be only used as a last resort since any new baseline configuration will need to be pushed manually from the deployer.

Going forward, please refrain from directly adding or editing an app's configuration file on search head cluster members. Any changes performed this way will not be replicated and any mistake may bring the cluster down. Refer to the documentation for the recommended methods.