Splunk Add-on for AWS - Decryption failed

Applicability

Splunk Apps Add ons

Description

Environment:
Using a Splunk Deployment Server (DS) to push Splunk-Add-on for AWS to the Splunk Heavy Forwarder (HF), which is configured for collecting AWS log files.

Issue:
AWS logs still were got getting ingested.

 

Investigation

The following error message was found in the log files at the HF instance.

splunkd.log:

07-03-2019 13:28:35.327 -0600 ERROR Crypto - Decryption operation failed: AES-GCM Decryption failed! 
07-03-2019 13:28:35.327 -0600 WARN ConfigEncryptor - Decryption operation failed: AES-GCM Decryption failed!

Based on the error message, this is very similar to the issue addressed in the following Splunk Answers article:

Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.
https://answers.splunk.com/answers/723728/installed-splunk-enterprise-721-and-promoted-it-as.html
 

Resolution

On the deployment server: 

1. Made a backup copy of splunk.secret in $SPLUNK_HOME/etc/auth directory.
2. Removed splunk.secret
3. Copied splunk.secret key from heavy forwarder (HF) to the deployment server (DS).
4. Made a backup copy of /opt/splunk/etc/system/local/server.conf file. 
5. Commented out in pass4SymmKey in server.conf file 
6. Reentered sslPassword under [sslConfig] stanza in server.conf file.
7. Restarted Splunk 
8. Pushed new Splunk_TA_aws/local/passwords.conf file to the HF.

 

Future recommendations:
1. Current approach of copying splunk.secret key from HW to the DS instance is a workable solution, but the following may needed 
to be considered in the future. 
In order for the DS to work with other forwarders/deployment clients, the same splunk.secret key must be distributed to all forwarders/clients. This will avoid from getting failed to decrypt errors.

2. An alternative approach is to to set passwords in clear text format in passwords.conf file on DS instance.
Then let the deployment client (HF) to encrypt the passwords once the files are pushed from DS. HW will decrypt the passwords
immediately after it has been restarted.

Related Links

Splunk Answers

Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading. https://answers.splunk.com/answers/723728/installed-splunk-enterprise-721-and-promoted-it-as.html

Revision History

Revision	Publish Date	Comments
1.0	28-Sep-2026	Initial Release