KV Store Backup Fails

Applicability

Description

KV store backup file kvdump_<epoch>.tar.gz is supposed to be created in $SPLUNK_DB/kvstorebackup directory when backing up KV store. But, the backup file either is not created, or an empty directory in the kvdump<epoch> format is created instead of the KVstore backup gunzip tarball file.

Investigation

The backup file is not created because KV store backup process crashes. Follow the steps below to confirm:

1- Run the SPL below

index="_internal" sourcetype=splunkd "Got invalid process token when starting mongo tool" 

and confirm if this error is logged when KV store backup command is executed:

ERROR MongodRunner [1547843 KVStoreBackupThread] - Got invalid process token when starting mongo tool

2- Log into the Search Head host and navigate to $SPLUNK_HOME/var/log/splunk and locate the file(s) that start with the word "crash" (crash-<date>-<time>.log). Open the crash log file, and confirm that the crashing thread is "KVStoreBackupThread". Example:

If the symptoms explained above are present, confirm that the KVstore backup process crashes when the KVstore backup is executed.

Resolution

The KV store backup process crashes because it is blocked by app control agents, EDR (Endpoint Detection and Response), antivirus, or other security processes.

This issue is resolved to ensure that those processes and agents allow the execution of the KV store backup process.