Splunk Enterprise Security and Splunk SOAR pairing fails with CERTIFICATE_VERIFY_FAILED for a self-signed certificate

Applicability

Description

Pairing Splunk Enterprise Security with Splunk SOAR On-premises through Configure > All configurations > Splunk SOAR can fail with Unable to pair when the SOAR certificate is self-signed or signed by a private CA.

The SOAR port can be reachable, but ES can still reject certificate validation when it calls the SOAR /rest/version endpoint.

The following error can appear:

Unable to pairSOAR REST Exception: HTTPSConnectionPool(host='xxx-testsoar01', port=8443): Max retries exceeded with url: /rest/version (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1162)')))

This issue can occur when the SOAR certificate is copied under $SPLUNK_HOME/etc/apps/missioncontrol/certificates/ with a filename other than soar_cert_bundle.pem, or when the file does not contain the PEM-formatted SOAR certificate or CA chain.

Settings in essoar.conf, such as verify_ssl = false and sslVerifyServerCert = false, do not replace the required Mission Control certificate bundle.

Investigation

1. Confirm that ES can reach the SOAR host and port.

openssl s_client -connect xxx-testsoar01:8443 -servername xxx-testsoar01 -showcerts

2. Verify that the SOAR certificate bundle exists with the exact filename required by Mission Control.

ls -l $SPLUNK_HOME/etc/apps/missioncontrol/certificates/soar_cert_bundle.pem

3. Verify that the file contains a certificate and not a public key.

head -1 $SPLUNK_HOME/etc/apps/missioncontrol/certificates/soar_cert_bundle.pem

The output must start with:

-----BEGIN CERTIFICATE-----

4. Verify that ES can validate the SOAR certificate with the bundle.

openssl s_client -connect xxx-testsoar01:8443 -servername xxx-testsoar01 -CAfile $SPLUNK_HOME/etc/apps/missioncontrol/certificates/soar_cert_bundle.pem

The command must end with:

Verify return code: 0 (ok)

5. Confirm that the hostname used in the SOAR URL matches the certificate Subject Alternative Name returned by SOAR.

openssl s_client -connect xxx-testsoar01:8443 -servername xxx-testsoar01 </dev/null 2>/dev/null | openssl x509 -noout -text | grep -A2 "Subject Alternative Name"

Resolution

1. Obtain the PEM-formatted SOAR server certificate or the CA/intermediate chain that signs the SOAR certificate.
2. On the ES search head, create the Mission Control certificates directory if it does not exist.

mkdir -p $SPLUNK_HOME/etc/apps/missioncontrol/certificates

3. Save the SOAR certificate bundle with the exact filename soar_cert_bundle.pem. Back up the existing file before replacing or editing it.

cp /tmp/soar_cert.pem $SPLUNK_HOME/etc/apps/missioncontrol/certificates/soar_cert_bundle.pem

If the certificate already exists in the directory with another filename, copy or rename it to soar_cert_bundle.pem.

4. Set ownership and permissions for the certificate bundle.

chown splunk:splunk $SPLUNK_HOME/etc/apps/missioncontrol/certificates/soar_cert_bundle.pem
chmod 644 $SPLUNK_HOME/etc/apps/missioncontrol/certificates/soar_cert_bundle.pem

5. Confirm that the bundle starts with -----BEGIN CERTIFICATE-----.

head -1 $SPLUNK_HOME/etc/apps/missioncontrol/certificates/soar_cert_bundle.pem

6. Confirm that the SOAR certificate validates successfully from ES.

openssl s_client -connect xxx-testsoar01:8443 -servername xxx-testsoar01 -CAfile $SPLUNK_HOME/etc/apps/missioncontrol/certificates/soar_cert_bundle.pem

7. If ES is deployed as a search head cluster, place the same bundle on the deployer and distribute it to all search head members.

mkdir -p $SPLUNK_HOME/etc/shcluster/apps/missioncontrol/certificates
cp /tmp/soar_cert.pem $SPLUNK_HOME/etc/shcluster/apps/missioncontrol/certificates/soar_cert_bundle.pem

8. Restart Splunk on the ES search head or search head members.

$SPLUNK_HOME/bin/splunk restart

9. Retry pairing in Configure > All configurations > Splunk SOAR.

Related Links

Pair Splunk Enterprise Security with Splunk SOAR

Pair Splunk SOAR On-premises with Splunk Enterprise Security