In Splunk Enterprise environments, KV Store may become unavailable intermittently. When checking the status via CLI, Splunk reports a failure to connect to the local embedded MongoDB listener on port 8191.
splunk show kvstore-status --verboseOutput:
This member:
backupRestoreStatus : Ready
disabled : 0
featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053') : No suitable servers found: `serverSelectionTimeoutMS' expired: [Failed to connect to target host: 127.0.0.1:8191]
guid : ****-****-****-****-***********
port : 8191
standalone : 1
status : failed
storageEngine : wiredTiger
This failure is triggered by an abnormal termination of the embedded mongod process. This issue is not caused by individual user collection records exceeding 16 MB; rather, it occurs when MongoDB's internal FTDC mechanism assembles system metrics in environments with numerous collections or WiredTiger tables.
Follow these steps to confirm if an instance is affected by this specific FTDC issue:
Check $SPLUNK_HOME/var/log/splunk/splunkd.log for process termination events:
ERROR MongodRunner - mongod exited abnormally (exit code 6, status: PID 1193253 killed by signal 6: Aborted) - look at mongod.log to investigate.
ERROR KVStoreBulletinBoardManager - KV Store process terminated abnormally (exit code 6, status PID 1193253 killed by signal 6: Aborted). See mongod.log and splunkd.log for details.
ERROR KVStoreBulletinBoardManager - KV Store changed status to failed. KVStore process terminated.
Inspect $SPLUNK_HOME/var/log/splunk/mongod.log for the FTDC assertion:
{"t":{"$date":"2026-03-24T00:03:17.020Z"},"s":"E","c":"ASSERT","id":23077,"ctx":"ftdc","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 23823936 (0x16B8640) is invalid. Size must be between 0 and 16793600(16MB) First element: start: new Date(0000000000000)","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2026-03-24T00:03:17.035Z"},"s":"F","c":"CONTROL","id":6384300,"ctx":"ftdc","msg":"Writing fatal message","attr":{"message":"terminate() called. An exception is active; attempting to gather more information\n"}}
{"t":{"$date":"2026-03-24T00:03:17.035Z"},"s":"F","c":"CONTROL","id":6384300,"ctx":"ftdc","msg":"Writing fatal message","attr":{"message":"DBException::toString(): BSONObjectTooLarge: BSONObj size: 23823936 (0x16B8640) is invalid. Size must be between 0 and 16793600(16MB) First element: start: new Date(0000000000000)\nActual exception type: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)10334, mongo::AssertionException>\n\n"}}
Run Search Processing Language (SPL) queries to verify crash signatures across the cluster:
Identify FTDC crashes:
index=_internal sourcetype=splunkd ("mongod exited abnormally" OR "KV Store changed status to failed")
| rex field=_raw "exit code (?<exit_code>\d+)"
| rex field=_raw "signal (?<signal>\d+): (?<signal_name>\w+)"
| eval crash_type=case(exit_code=="6" AND signal=="6", "SIGABRT - FTDC pattern", exit_code=="9" OR signal=="9", "SIGKILL - OS OOM", true(), "Other: exit=".exit_code." sig=".signal)
| stats count as crashes, earliest(_time) as first_crash, latest(_time) as last_crash by host, crash_type
| eval first_crash=strftime(first_crash, "%Y-%m-%d %H:%M:%S"), last_crash=strftime(last_crash, "%Y-%m-%d %H:%M:%S")
| sort - crashes
Confirm BSONObjectTooLarge occurrences in MongoDB logs:
index=_internal sourcetype=mongod ("BSONObjectTooLarge" OR ("ftdc" AND "Assertion") OR ("ftdc" AND "terminate"))
| rex field=_raw "BSONObj size: (?<bson_size>\d+)"
| eval bson_size_MB=round(bson_size/1024/1024, 2)
| table _time host bson_size bson_size_MB _raw
| sort - _time
Count WiredTiger tables (collections + indexes):
| rest splunk_server=local /services/server/introspection/kvstore/collectionstats
| mvexpand data
| spath input=data
| rename ns as namespace, nindexes as index_count
| stats dc(namespace) as collections, sum(index_count) as total_indexes by splunk_server
| eval total_wt_tables=collections + total_indexes
| eval risk=case(total_wt_tables>=500, "HIGH", total_wt_tables>=200, "MEDIUM", true(), "LOW")
| table splunk_server collections total_indexes total_wt_tables riskThis is a known issue originating from an upstream MongoDB defect (SERVER-115585). The permanent fix is included in bundled MongoDB binary updates (MongoDB 8.0.26+), which introduce the "diagnosticDataCollectionDiscardLargeSamples" setting to drop oversized diagnostic samples instead of aborting the process.
Upgrade Splunk Enterprise to one of the following fixed maintenance releases or later:
9.4.x: 9.4.14
10.0.x: 10.0.9
10.2.x: 10.2.6
10.4.x: 10.4.2
10.6.x: 10.6.0
Workaround / Temporary Mitigation:
If an immediate upgrade is not possible, perform the following steps to restore service and reduce crash frequency:
$SPLUNK_HOME/bin/splunk restart
2. Clean up accumulated FTDC diagnostic files:
3. Stop Splunk and remove older diagnostic files from the MongoDB path:
$SPLUNK_HOME/bin/splunk stop
rm -f $SPLUNK_DB/kvstore/mongo/diagnostic.data/metrics.*
$SPLUNK_HOME/bin/splunk start| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
01-Oct-2026
|
Initial Release |