Kvstore migration failing post upgrade to Splunk Enterprise 9.4.0

Applicability

Description

After upgrading to Splunk 9.4.0, mongo should also upgrade to 7.0, but mongo migration failed with the below error in messages tab,

“KVSTore pre-upgrade checks failed! Reason=Cannot get the size of the KVStore folder at=C:\Splunk\var\lib\splunk\kvstore\mongo, due to reason=”
messages and 41 files with File Integrity check issues:

Investigation

1. Check the status of kvstore and serverVersion. Use the below command to validate.

splunk show kvstore-status --verbose

2. Check in splunkd.log or messages tab, if there are any kvstore migration error like the below,

“KVSTore pre-upgrade checks failed! Reason=Cannot get the size of the KVStore folder at=C:\Splunk\var\lib\splunk\kvstore\mongo, due to reason=”
messages and 41 files with File Integrity check issues:

3. Validate if the pre-requisites for mongo migration to 7.0 are met. Below are the pre-requisites
A. Minimum of 50%disk space on partition where kvstore is present.
B. ipV6 cannot be used during the process of migration.
C. Custom SSL certificates cannot be used during the process of migration, Splunk should reply on Splunk default certs for sslConfig and kvstore stanza server.conf.

4. If custom SSL certificates are being used under [sslConfig] or [kvstore] stanza then comment them out and configure Splunk to use Splunk default certificates.

./splunk/etc/system/local/server.conf
[kvstore]
serverCert = /opt/splunk/etc/auth/server.pem
sslPassword = password
sslRootCAPath = /opt/splunk/etc/auth/cacert.pem[sslConfig]
serverCert = /opt/splunk/etc/auth/server.pem
sslPassword = password
sslRootCAPath = /opt/splunk/etc/auth/cacert.pemRestart Splunk service.

5. Check if mongo is now automatically migrated to 7.0 as "kvstoreUpgradeOnStartupEnabled" is by default set to true.

Also validate if the error mentioned above, still exists. If the error still exists, validate the partitions under which Splunk is installed and where kvstore is present.

6. Validate if Splunk is installed on a different partition and kvstore is present on a different path. KVstore path can be validated by checking the dbPath in splunk-launch.conf under

 ./splunk/etc/splunk-launch.conf.

7. Check if the dbPath partition has the kvstore directory with mongo directory in it.

Create a dbPath in [kvstore] stanza pointing to the current partition where kvstore is present and restart Splunk.

./splunk/etc/system/local/server.conf
[kvstore]
dbPath = <current kvstore directory location>Restart Splunk

8. Check if there are file integrity errors mentioning about missing files under splunk/bin, as this could be causing Splunk not able to read the mongo location.
messages and 41 files with File Integrity check issues:

C:\Splunk\bin\api-ms-win-core-console-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-datetime-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-debug-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-errorhandling-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-file-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-file-l1-2-0.dll,missing
C:\Splunk\bin\api-ms-win-core-file-l2-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-handle-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-heap-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-interlocked-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-libraryloader-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-localization-l1-2-0.dll,missing
C:\Splunk\bin\api-ms-win-core-memory-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-namedpipe-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-processenvironment-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-processthreads-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-processthreads-l1-1-1.dll,missing
C:\Splunk\bin\api-ms-win-core-profile-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-rtlsupport-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-string-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-synch-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-synch-l1-2-0.dll,missing
C:\Splunk\bin\api-ms-win-core-sysinfo-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-timezone-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-core-util-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-conio-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-convert-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-environment-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-filesystem-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-heap-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-locale-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-math-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-multibyte-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-private-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-process-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-runtime-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-stdio-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-string-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-time-l1-1-0.dll,missing
C:\Splunk\bin\api-ms-win-crt-utility-l1-1-0.dll,missing
C:\Splunk\bin\ucrtbase.dll,missing

Copy the files from a working installation of Splunk 9.4.0.

Resolution

As Splunk is installed on a different partition and kvstore/dbPath is pointed to a different partition, Splunk should be looking for mongo file under the partition where kvstore is present. But its looking for mongo in the Splunk installation directory, looking into the default dbpath = ./splunk/var/lib/splunk/kvstore/mongo.

Setting only the kvstore dbPath to default path will help fix the issue where as keeping other indexes dbPath to the other directory.

 Create a kvstore directory under $SPLUNK_HOME/var/lib/splunk/ and configure the below.

./splunk/etc/system/local/server.conf
[kvstore]
dbPath = $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/
Restart Splunk and validate the status of kvstore and serverVersion.

Mongo should be migrated to 7.0.