KV Store fails to start after Splunk 9.4.4 upgrade on RHEL 9 due to MongoDB library conflicts and NUMA warnings

Applicability

Description

On RHEL 9, these conflicts are especially common because the system no longer provides the OpenSSL 1.0 libraries (libssl.so.10 and libcrypto.so.10) required by MongoDB’s older linkage model. As a result, the mongod process may fail to start if incompatible or missing libraries are encountered during initialization. This behavior occurs because the system libraries take precedence over Splunk’s bundled libraries. 

Investigation

Verify KV Store health:

$SPLUNK_HOME/bin/splunk show kvstore-status --verbose

If KV Store is down or the command fails, open mongod log file:

cat $SPLUNK_HOME/var/log/splunk/mongod.log

In the logs, it should display the following:

{"t":{"$date":"2025-10-22T16:43:32.328+00:00"},"s":"I","c":"NETWORK","id":4915701,"ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":21},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":21},"outgoing":{"minWireVersion":6,"maxWireVersion":21},"isInternalClient":true}}}
{"t":{"$date":"2025-10-22T16:43:32.330+00:00"},"s":"I","c":"CONTROL","id":23285,"ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2025-10-22T16:43:32.331+00:00"},"s":"I","c":"NETWORK","id":4648601,"ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2025-10-22T16:43:32.332+00:00"},"s":"I","c":"CONTROL","id":23403,"ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"7.0.14","openSSLVersion":"OpenSSL 1.0.2zl-fips  11 Feb 2025"}}}
{"t":{"$date":"2025-10-22T16:43:32.332+00:00"},"s":"I","c":"CONTROL","id":51765,"ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Red Hat Enterprise Linux release 9.6 (Plow)","version":"Kernel 5.14.0-570.32.1.el9_6.x86_64"}}}
{"t":{"$date":"2025-10-22T16:43:32.333+00:00"},"s":"E","c":"CONTROL","id":20557,"ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"NonExistentPath: Data directory /data/db not found. Create the missing directory or specify another path using (1) the --dbpath command line option, or (2) by adding the 'storage.dbPath' option in the configuration file."}}
{"t":{"$date":"2025-10-22T16:43:32.333+00:00"},"s":"I","c":"CONTROL","id":23138,"ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":100}}

NUMA startup warning:

{"t":{"$date":"2025-10-20T17:28:18.214Z"},"s":"W","c":"CONTROL","id":22167,"ctx":"initandlisten","msg":"You are running on a NUMA machine. We suggest launching mongod like this to avoid performance problems: numactl --interleave=all mongod [other options]","tags":["startupWarnings"]}

Reproduce the behavior to confirm the path issue.

Run a quick diagnostic invocation don’t keep mongod running this way (numa may require a installation):

numactl --interleave=all $SPLUNK_HOME/bin/mongod

It should display the same default-path failure:

{"t":{"$date":"2025-10-31T18:04:08.331-04:00"},"s":"E","c":"CONTROL","id":20557,"ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"NonExistentPath: Data directory /data/db not found. Create the missing directory or specify another path using (1) the --dbpath command line option, or (2) by adding the 'storage.dbPath' option in the configuration file."}}
{"t":{"$date":"2025-10-31T18:04:08.332-04:00"},"s":"I","c":"CONTROL","id":23138,"ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":100}}

Quickly isolate just errors & warnings from the logs:

grep -E "ERROR|WARN|DBException|NonExistentPath|replSet|wiredTiger|mongod\.lock" $SPLUNK_HOME/var/log/splunk/mongod.log
grep -E "ERROR|WARN|kvstore|DbInputCheckpointManager|error loading checkpoint" $SPLUNK_HOME/var/log/splunk/splunkd.log

Note:  Do not leave mongod running manually outside of Splunk control. Use only for verification. 

Resolution

Link Splunk’s internal libraries using LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=/opt/splunk/lib:$LD_LIBRARY_PATH 

Verify NUMA is enabled:

grep NUMA=y /boot/config-`uname -r`

Then ensure system LDAP libraries take precedence by removing or renaming Splunk’s conflicting LDAP libs (because they are unsupported by RHEL 9):

mv /opt/splunk/lib/libldap.so.2 /opt/splunk/lib/libldap.so.2.bak
mv /opt/splunk/lib/liblber.so.2 /opt/splunk/lib/liblber.so.2.bak

1) Create a backup if KV Store is up, if it does not let you to backup, use step 8:

$SPLUNK_HOME/bin/splunk backup kvstore -pointInTime true

2) Stop Splunk and clear stale lock; fix ownership

$SPLUNK_HOME/bin/splunk stop
rm -f /opt/splunk/var/lib/splunk/kvstore/mongo/mongod.lock && chown -R splunk:splunk /opt/splunk/var/lib/splunk/kvstore

3) Repair the KV Store database

$SPLUNK_HOME/bin/splunk cmd mongod --repair

4) Retrieve replSet GUID from mongod.log

cat /opt/splunk/var/log/splunk/mongod.log | grep -i ',"replSet":"'

# Example:

"replSet":"33F38D52-0958-43F7-9BE5-582AB0AC2EC7"

5) Start mongod once with explicit KV Store paths and GUID:

splunk cmd mongod --dbpath "$SPLUNK_HOME/var/lib/splunk/kvstore/mongo" --logpath "$SPLUNK_HOME/var/lib/splunk/kvstore/mongo/mongod.log" --port 8191 --replSet "<GUID>" --keyFile "$SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key" --tlsMode requireTLS --tlsCAFile "$SPLUNK_HOME/etc/auth/cacert.pem" --tlsCertificateKeyFile "$SPLUNK_HOME/etc/auth/server.pem" --setParameter enableLocalhostAuthBypass=0 --setParameter minSnapshotHistoryWindowInSeconds=5 --storageEngine wiredTiger

Note: Password is "password" or "changeme" are/is the default from Splunk.

6) Start Splunk and verify KV Store

$SPLUNK_HOME/bin/splunk start$SPLUNK_HOME/bin/splunk show kvstore-status

7) Restore from backup

$SPLUNK_HOME/bin/splunk stop

$SPLUNK_HOME/bin/splunk restore kvstore --file /opt/splunk/var/lib/splunk/kvstorebackup/<DATE>

$SPLUNK_HOME/bin/splunk start

8) Alternative reset path rename on-disk Mongo dir and let Splunk recreate (data loss risk):

mv /opt/splunk/var/lib/splunk/kvstore/mongo /opt/splunk/var/lib/splunk/kvstore/mongo.old

Note: DB Connect checkpoints once KV Store is healthy, rising-mode inputs will recreate checkpoint keys automatically; timing depends on each input’s configured interval and the next few input cycles.

What success looks like after using the correct Splunk KV path

After letting Splunk manage KV Store the process list should show the right arguments:

$ ps aux | grep mongod
splunk  23980  ... mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --storageEngine=wiredTiger --port=8191 --replSet=22632523-05BD-4906-B28C-6B827DA93BD4 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --tlsMode=requireTLS --tlsCAFile=/opt/splunk/etc/auth/cacert.pem --tlsCertificateKeyFile=/opt/splunk/etc/auth/server.pem --setParameter=enableLocalhostAuthBypass=0 --setParameter=minSnapshotHistoryWindowInSeconds=5 --bind_ip=0.0.0.0 --nounixsocket --noscripting

Related Links

Back up and restore KV store

Mongo Database Commands

9.4.0 Upgrade Broke Splunk DB Connect - Checkpoint not found

Common issues for Splunk DB Connect