Error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:741)

Applicability

Description

"ServiceNow " is installed on a clustered Search Head (SH) and it's trying to add connections to another instance, however,  the following error message is received:

"Unable to reach server at https://something-now.com/. Check configurations and 
network settings.
2020-06-12 17:01:21,909 ERROR pid=6520 tid=MainThread file=splunk_ta_snow_account_
validation.py:validate:66 | Unable to reach ServiceNow instance at https://something-now.com/.
The reason for failure is=Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/splunk_ta_snow_account_validation.py", line 62,
 in validate
resp, content = http.request(url)
File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/httplib2/__init__.py", line 1593, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, 
redirections, cachekey)
File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/httplib2/__init__.py", line 1335, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/httplib2/__init__.py", line 1257, in 
_conn_request
conn.connect()
File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/httplib2/__init__.py", line 1044, in connect
raise SSLHandshakeError(e)
SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:741)"

HFs and UFs send data to the indexers. The HFs is used for syslog and apps with inputs not required in the search heads.
 

Investigation

In regards to the code (_ssl.c:741) ; Splunk contains its own Python environment, which is different from the host’s Python environment (for both Windows and Linux install). ServiceNow's current Splunk Add-on, ServiceNow SecOps Add-on (3921) - leverages a Python script to handle the connect command, to talk to ServiceNow. This Python script leverages the Splunk Python instance that explicitly trusts certain SSL CA Certs (local to Splunk’s Python environment)

ServiceNow users leveraging internally issued SSL certificates (non-public) need Splunk to trust these certificates (or the Certificate Authority that issued the certificate) so that they can successfully use the SN SecOps Add-on (3921).

However, the next troubleshooting steps were not the solution:
To update the Splunk Install Directory:
Splunk Install Directory:

/opt/splunk/etc/apps/TA-ServiceNow-Secops/bin

This is the file that needs to be modified:

/opt/splunk/etc/apps/TA-ServiceNow-Secops/bin/sn_connect.py

Adjust Line 40 and 42 of "sn_connect.py" python script to either
1- Ignore SSL verification
2- Create a new directory, Store CA certs in Directory, code path to the directory.
 

Resolution

It was determined that in one SNOW instance used a different server certificate - one that was generated by an internal CA.
Adding the root CA's certificate to the cacerts.txt file got the connections working.
Testing was slowed somewhat by the fact that the 4.0.0 TA has two different cacerts.txt files.

Related Links