ERROR: Decryption operation failed: AES-GCM Decryption failed!

Applicability

Splunk Enterprise

Description

After migrated (cluster master) instance to a new server (also license master). Noted AES-GCM decryption errors in many of the instances across the environment. This was fixed in most cases by updating the pass4SymmKey to that of the new license master. However, the issue persists on the deployment server:
 

[splunk@splunk-sandbox bin]$ ./splunk reload deploy-server
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!An unforeseen error occurred:Exception: <class 'httplib2.RelativeURIError'>, Value: Only absolute URIs are allowed. uri = error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
https://127.0.0.1:8092/services/deployment/server/config/_reloadTraceback (most recent call last):
File "/data/splunk-deployment/splunk/lib/python3.7/site-packages/splunk/clilib/cli.py", line 1206, in main
parseAndRun(argsList)
File "/data/splunk-deployment/splunk/lib/python3.7/site-packages/splunk/clilib/cli.py", line 977, in parseAndRun
timeout=timeout, token=usingToken)
File "/data/splunk-deployment/splunk/lib/python3.7/site-packages/splunk/rcUtils.py", line 649, in makeRestCall
serverResponse, serverContent = simpleRequest(uri, sessionKey=sessionKey, getargs=getargs, postargs=postargs, method=method, timeout=timeout, token=token)
File "/data/splunk-deployment/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 515, in simpleRequest
serverResponse, serverContent = h.request(uri, method, headers=headers, body=payload)
File "/data/splunk-deployment/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 1767, in request
(scheme, authority, request_uri, defrag_uri) = urlnorm(uri)
File "/data/splunk-deployment/splunk/lib/python3.7/site-packages/httplib2/__init__.py", line 236, in urlnorm
raise RelativeURIError("Only absolute URIs are allowed. uri = %s" % uri)
httplib2.RelativeURIError: Only absolute URIs are allowed. uri = error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
https://127.0.0.1:8092/services/deployment/server/config/_reload
Please file a case online at http://www.splunk.com/page/submit_issue

Investigation

Delete the passwords.conf files from the local folder within the affected apps and reconfigure those apps. 

I checked on the /etc/system/default/serverclass.conf file in the [global] stanza, the "endpoint" value is:

"$deploymentServerUri$/services/streams/deployment?name=$tenantName$:$serverClassName$:$appName$"

The default value is:

"$deploymentServerUri$/services/streams/deployment?name=$serverClassName$:$appName$"

Check the statement mainly the "serverClassName" and the "appName".

As example:

"$deploymentServerUri$/services/streams/deployment?name=$tenantName$:$appName$"

It seemed this error is due to when Splunk tries to decrypt something encrypted by another copy of Splunk. This can happen with any content that has encrypted credentials - the credentials get encrypted on the first run, so copying to another instance will have it try to decrypt using an incorrect key.

Resolution

Remove the pass4SymmKey value from the server.conf and restarting Splunk.

Related Links

Installed-Splunk-Enterprise-7-2-1 and promoted it as cluster

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release