Error - Failed to verify HMAC signature on Cluster Master

Applicability

Description

Cluster Manager continuously logs the following messages in $SPLUNK_HOME/var/log/splunk/splunkd.log

ERROR DigestProcessor [10038 TcpChannelThread] - Failed signature match
ERROR HTTPAuthManager [18246 TcpChannelThread] - Failed to verify HMAC signature, uri: /services/indexer_discovery

Investigation

1. On the cluster manager in $SPLUNK_HOME/var/log/splunk/splunkd.log the following entries are reported:

   ERROR DigestProcessor [10038 TcpChannelThread] - Failed signature match
   ERROR HTTPAuthManager [18246 TcpChannelThread] - Failed to verify HMAC signature, uri: /services/indexer_discovery

   This indicates an incorrect pass4SymmKey for indexer discovery.

2. Use this search identify "401 Unauthorized" messages in splunkd_access.log in order to identify which instances are failing to connect.

   index=_internal source=*splunkd_access.log* /services/indexer_discovery | stats count by clientip status | sort - count

   You should see the clientip of the instances in question.

3. Use the following command to identify the pass4SymmKey value currently in use on the affected instance(s), and in which file that value originates:
   splunk btool <configfile> list --debug indexer_discovery
   replace <pass4SymmKey> with the hashed value from the .conf files below (still wrapped in single quotes)
   
   Example for Cluster Manager's server.conf file:

   splunker@myclustermanager:~$ splunk btool server list --debug indexer_discovery
   /opt/splunk/etc/system/local/server.conf [indexer_discovery]
   /opt/splunk/etc/system/local/server.conf pass4SymmKey = $7$eNoUVUMRVSDWfyCV3xSHQM0EIqe7rgYZixLuUWwnWZjm/IL115zC7dn7B2Us

   Example of an affected instance's outputs.conf file (could be any instance that is not an indexer -- forwarder, search head, license manager, cluster manager, etc. -- any instance that forwards logs to the indexer cluster)

   splunker@myinstance:~$ splunk btool outputs list --debug indexer_discovery
   /opt/splunk/etc/apps/_nonclustered-se/local/outputs.conf [indexer_discovery:default-autolb-group]
   /opt/splunk/etc/apps/_nonclustered-se/local/outputs.conf manager_uri = https://myclustermanager:8089
   /opt/splunk/etc/apps/_nonclustered-se/local/outputs.conf pass4SymmKey = $7$3bsmU0jPGGnCrxjSKzHesujN9a6XoZJNdYBFhHKGme1T9evf3hpi1vTuCGRt

4. Use the following command to confirm the hashed values for pass4SymmKey from the outputs above. All should be identical.
   splunk show-decrypted --value '<pass4SymmKey>'

Resolution

To correct the pass4SymmKey for any instance where the decrypted value does not resolve to the correct passcode, replace the hashed value in the affected file with the correct plain text value and restart Splunk.

If at any time the configuration file or pass4SymmKey value copied in its hashed format to another instance, the decrypt will produce nonsense results. This is because initially, the value was provided in the configuration file in plain text, and each instance, Cluster Manager included, then hashed the value based on its own 'secret'. The pass4SymmKey can only be decrypted by that Splunk instance. Each fresh install of Splunk has a different secret and will hash the value differently, so copying a file to a different instance, or wiping Splunk from an instance then reinstalling it and re-applying the config files will produce the same nonsense result.

Please note also, while this article focuses on indexer discovery, the same principle applies to any pass4SymmKey value. For example, all indexers and search heads connecting to the cluster do so through the pass4SymmKey value in the [clustering] stanza of server.conf. If certain instances are failing to communicate with the cluster manager and are showing the 'Failed to verify HMAC signature' message, use these same commands to check the pass4SymmKey value in the clustering stanza for all impacted instances, including the cluster manager. Similarly, clustered search heads communicate with one another via the pass4SymmKey in their [shcluster] stanza.

Related Links

https://docs.splunk.com/Documentation/Splunk/latest/Indexer/indexerdiscovery#Configure_indexer_discovery