Search Head cluster member is not pulling configurations from the search head cluster captain

Applicability

Description

Search head cluster member is having problems pulling configurations from the search head cluster

Investigation

After reviewing the diag I found the following log: 

Error pulling configurations from captain=Search_Head_member:8089, consecutiveErrors=46582 msg="Error in fetchLookupTableContents, asset_id=<id>: Failed to create /opt/splunk/var/run/splunk/lookup_tmp/ Permission denied": Search head cluster member (Search_Head_member:8089) is having problems pulling configurations from the search head cluster captain (Captain:8089). Changes from the other members are not replicating to this member, and changes on this member are not replicating to other members. Consider performing a destructive configuration resync on this search head cluster member

This message indicates that the SHC member is failing to pull updated configurations (including lookup tables) from the SHC captain. The error occurred over 46,000 times, showing it’s a persistent issue. The failure is due to a file system-level permission error and we can see it in the following part of the log error.

Failed to create /opt/splunk/var/run/splunk/lookup_tmp/ Permission denied"

This prevents SHC member from writing temporary files required to sync lookup table assets, breaking its ability to participate in SHC configuration replication. The Splunk process on prod4 is currently unable to write to the directory /opt/splunk/var/run/splunk/lookup_tmp/, which is likely causing configuration sync failures within the search head cluster.
This issue may be due to incorrect file or directory ownership like not owned by the splunk user, a file system remount or corruption, recent permission changes made by external processes or scripts, or the disk being full or mounted in a read-only state.

To mitigate this you can run one of the following commands to make sure splunk user can read/write to avoid the "permission denied"
sudo chown -R splunk:splunk /opt/splunk/var/run/splunk/lookup_tmp or sudo chown -R splunk:splunk /opt/splunk/var/run/splunk

After running either of the command you will need to restart this Search Head, to do that you can run the following command:
$SPLUNK_HOME/bin/splunk restart
You can monitor after the restart by running the command:
splunk show shcluster-status

Resolution

To mitigate this run the following command to make sure splunk user can read/write to avoid the "permission denied"

sudo chown -R splunk:splunk /opt/splunk/var/run/splunk/lookup_tmp 

After running the command restart the Search Head, running the following command:

$SPLUNK_HOME/bin/splunk restart

Monitor after the restart by running the command:

splunk show shcluster-status