Unable to Add the AWS Account in Splunk Add-on for AWS.

Applicability

Description

When trying to add the AWS account details from the Splunk GUI, the account details were not getting saved, getting the below SSL errors -

In Splunk GUI - 

SSL validation failed for https://sts.amazonaws.com/ [SSL: CERTIFICATE_VERIFY_FAILED] 
certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)

In splunkd.log:

ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call 
last):\nFile/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py
line 662, in urlopen\n self._prepare_proxy(conn)\n File 
"/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 948
in _prepare_proxy\n conn.connect()\n File "/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/
python3/urllib3/connection.py", line 360, in connect\n ssl_context=context,\n File 
"/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/util/ssl_.py", line 370, 
in ssl_wrap_socket\n return context.wrap_socket(sock, server_hostname=server_hostname)\n 
File "/opt/splunk/lib/python3.7/ssl.py", line 423, in wrap_socket\n session=session\n File 
"/opt/splunk/lib/python3.7/ssl.py", line 870, in _create\n self.do_handshake()\n File 
"/opt/splunk/lib/python3.7/ssl.py", line 1139, in do_handshake\n self._sslobj.do_handshak
e()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify 
failed: unable to get local issuer certificate (_ssl.c:1091)\n\nDuring handling of the 
above exception, another exception occurred:\n\nTraceback (most recent call last):\n File 
"/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/botocore/httpsession.py", line 263, 
in send\n chunked=self._chunked(request.headers),\n File 
"/opt/splunk/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/urllib3/connectionpool.py", line 720
 in urlopen\n method, url, error=e, _pool=self, _stacktrace=sys.exc_

Investigation

Verify the AWS account details and permissions to add. Didn't find much info related to the issue from the splunkd logs apart from the above-mentioned SSL errors.

Check the proxy/organisational certs in the location: $SPLUNK_HOME/etc/apps/SplunkTAaws/bin/3rdparty/botocore/vendored/requests/
or
.../etc/apps/Splunk_TA_aws/bin/3rdparty/python3/botocore/vendored/requests
If it is not available, add the cert file and restart the splunk service. Even after following the above process, still get the same error.

Tried adding the splunk ca certs to the above location but it still didn't work.

Tried disabling the global parameters - verify_certificates, is_secure in the add-on configuration file $SPLUNK_HOME/etc/apps/Splunk_TA_aws/default/aws_global_settings.conf but it didn't work.

 

Resolution

After adding the proxy/organizational certs to the location:

$SPLUNK_HOME/etc/apps/Splunk_TA_aws/bin/3rdparty/python3/certifi/

and restarted the splunk service, able to add the AWS account from the Splunk GUI.

 

Related Links

Splunk Documentation:

AWS-Add-on-unable-to-connect-to-AWS-due-to-SSL-inspection