HTTP 503 Service Unavailable -- KV Store initialization failed

Applicability

Description

The data collection stops from different sources of any third party app that might try to use KV Store.

Investigation

Verify Mongo DB logs. The following message indicates a problem with the certificate in use.

NETWORK [main] The provided SSL certificate is expired or not yet valid.

Verify the affected add-on logs. In the example below this is "Mimecast", and its log file ta_mimecast_for_splunk_mimecast_audit.log

ERROR pid=9999 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events.
Traceback (most recent call last):
File "$SPLUNK_HOME\etc\apps\TA-mimecast-for-splunk\bin\ta_mimecast_for_splunk\aob_py3\solnlib\modular_input\checkpointer.py", line 218, in get
record = self._collection_data.query_by_id(key)
File "$SPLUNK_HOME\etc\apps\TA-mimecast-for-splunk\bin\ta_mimecast_for_splunk\aob_py3\solnlib\packages\splunklib\client.py", line 3648, in query_by_id
return json.loads(self._get(UrlEncoded(str(id))).body.read().decode('utf-8'))
solnlib.packages.splunklib.binding.HTTPError: HTTP 503 Service Unavailable -- KV Store initialization failed. Please contact your system administrator.

Verify Splunk internal logs under $SPLUNK_HOME/var/log/splunk.

WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'.
WARN HttpListener - Socket error from xxx.xxx.xxx.xxx:54886 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'.
WARN HttpListener - Socket error from xxx.xxx.xxx.xxx:54887 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown

Verify the KVStore status in the instance with the affected add-on installed.

$SPLUNK_HOME/bin/splunk show kvstore-statusThis member:
backupRestoreStatus : Ready
disabled : 0
guid : <GUID>
port : 8191
standalone : 1
status : failed 

Run the following command to confirm the certificate expiration date:

$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem

Resolution

Renew the Splunk SSL certificates.

1. Go to $SPLUNK_HOME/etc/auth/
2. Rename server.pem to server.pembackup
3. Restart Splunk service using the command "splunk restart" under $SPLUNK_HOM/bin directory.

After restart a new server.pem file is available under $SPLUNK_HOME/etc/auth/

Check the new certificate expiration date using the command below: 

$SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in $SPLUNK_HOME/etc/auth/server.pem

Verify the KVStore status with the command below:

$SPLUNK_HOME/bin/splunk show kvstore-status

The expected status is ready.

Related Links