KVStore fails to start after upgrading to 9.4.1

Applicability

Description

After upgrading Splunk to 9.4.1 KVStore fails to start, and the following error is seen in splunkd.log indicating that KVStore fails to connect to mongod on local server and KVStore upgrade to version 7 did not take place:

ERROR KVStorageProvider [5669 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '127.0.0.1:8191']

Investigation

Splunkd.log indicates that KVStore fails to connect to mongod:

ERROR KVStorageProvider [5669 KVStoreUpgradeStartupThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '127.0.0.1:8191']

mongod.log indicates there is an issue with the certificate chain:

NETWORK  [conn200] Error receiving request from client: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate in certificate chain. Ending connection from 127.0.0.1:49500 (connection id: 200)

mongod.log also shows mongo is reading the server certificate is located at /opt/splunk/etc/auth/server.pem and that it reads trusted certificates from  /etc/pki/tls/certs/ca-bundle.crt: 

CONTROL  [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, tls: { CAFile: "/etc/pki/tls/certs/ca-bundle.crt", allowConnectionsWithoutCertificates: true, allowInvalidHostnames: true, certificateKeyFile: "/opt/splunk/etc/auth/server.pem", certificateKeyFilePassword: "<password>", disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireTLS", tlsCipherConfig: "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256" }

Resolution

Since the server is configured to trust only certificates stored in  /etc/pki/tls/certs/ca-bundle.crt the server certificate (/opt/splunk/etc/auth/cacert.pem) needs to be added to the trust store (/etc/pki/tls/certs/ca-bundle.crt ).