ERROR: "HTTP 503 Service Unavailable -- KV Store initialization failed"

Applicability

Splunk Enterprise

Description

Mimecast app was installed and configured correctly, but we were not able to see data from it, according to the logs kvstore was not able to start in the Heavy Forwarder due to a HTTP error 503.

 

Investigation

The kvstatus was set as failed 
 

c:\>"C:\Program Files\Splunk\bin\splunk" show kvstore-status
Splunk username: splunkMVdata
Password:This member:
backupRestoreStatus : Ready
disabled : 0
guid : D678B02B-D8FF-4327-BF4D-32087D412C20
port : 8191
standalone : 1
status : failed

Splunkd.log from the HF:

ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\TA-mimecast
-for-splunk\bin\mimecast_ttp_url.py"" HTTPError: HTTP 503 Service Unavailable -- KV Store 
initialization failed. Please contact your system administrator.

Mongod.log

 I CONTROL  [DataFileSync] FlushViewOfFile for C:/Program Files/Splunk/var/lib/splunk
/kvstore/mongo/local.ns failed with error 1117 after 1 attempts taking 670648ms F -        [DataFileSync] Fatal Assertion 16387 at src\mongo\db\storage\mmap_v1
\mmap_windows.cpp 453 F -        [DataFileSync]  ***aborting after fassert() failure I CONTROL  [DataFileSync] writing minidump diagnostic file C:\Program Files\Splunk
\bin\mongod.2020-02-24T18-39-54.mdmpI JOURNAL  [journal writer] error exception in dur::journal error appending to file 
C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo\journal\j._6 8192 8192 The 
request could not be performed because of an I/O device error.F JOURNAL  [journal writer] dbexception in journalWriterThread causing immediate 
shutdown: Location13517: error appending to file C:\Program Files\Splunk\var\lib
\splunk\kvstore\mongo\journal\j._6 8192 8192 The request could not be performed 
because of an I/O device error.F -        [journal writer] Invariant failure false src\mongo\db\storage\mmap_v1
\dur_journal_writer.cpp 262
 2020-02-24T18:40:43.606Z F -        [journal writer] 
 ***aborting after invariant() failure

 

Resolution

When the kvstore status is set as failed  is becasue "Failed to bootstrap and join the search head cluster." , this is quite common, therefore the solution is the following :

a. stop splunk

b. run the following command "splunk clean kvstore --local"

c. start splunk

d. check kvstore-status

 

Revision History

Revision	Publish Date	Comments
1.0	28-Sep-2026	Initial Release