KV Store Mongo-to-PostgreSQL migration failure blocks restart in Splunk Enterprise 10.6.x

Applicability

Description

Users are unable to stop or restart Splunk Enterprise after a failed KV Store migration from MongoDB to PostgreSQL.

The observed behaviour of this issue may include :

• KV Store reports status : ready and continues to use MongoDB.
• The PostgreSQL migration status remains Migration_Failed at 0.00%.
• splunk stop and splunk restart are blocked.
• Setting postgresMigrateOnStartup = false does not remove the restart restriction.

The issue was observed on a standalone Splunk Enterprise deployment with a single member KV Store. This issue can occur when the KV Store migration runs while Splunk Enterprise is running as the root user. 

The following error is displayed when restarting Splunk :

KVStore Mongo to Postgres migration is in progress... Splunk stop / restart is not allowed!

The migration status may show :

KV Store migration to Postgres status:
                migration status: Migration_Failed
                message: KV Store migration to Postgres failed, continuing with Mongo.
                last attempted version: 10.6.0.5
                percentage completed: 0.00%

Investigation

1. Check the effective KV Store configuration :

$ splunk btool server list --debug kvstore

The following configuration may be present :

[kvstore]
defaultKVStoreType = local
postgresMigrateOnStartup = false

2. Check the current KV Store state :

$ splunk show kvstore-status --verbose

The observed output can include :

migrationStatus : NotStarted
replicationStatus : KV store captain
standalone : 1
status : ready
storageEngine : wiredTiger

This confirms that KV Store is available and still using the MongoDB backend.

3. Check the PostgreSQL migration status :

$ splunk show kvstore-postgres-migration-status

The observed output can include :

migration status: Migration_Failed
message: KV Store migration to Postgres failed, continuing with Mongo.
last attempted version: 10.6.0.5
percentage completed: 0.00%

4. Check for the migration runtime marker :

$ ls -la /opt/splunk/var/run/splunk/kvstore_mongo_to_postgres_migration_inprogress

The observed output was:

-rw------- 1 root root 0 Oct 15 13:44 /opt/splunk/var/run/splunk/kvstore_mongo_to_postgres_migration_inprogress

The presence of this file can cause Splunk to continue treating the migration as active.

5. Review splunkd.log and diagnostic logs for PostgreSQL initialisation errors. The following error may be found :

pg_ctl: cannot be run as root

This indicates that PostgreSQL initialisation failed because Splunk was running under the root user.

Resolution

The confirmed cause is a failed PostgreSQL migration combined with a stale migration runtime marker. PostgreSQL initialisation failed because Splunk Enterprise was running as root. PostgreSQL does not allow pg_ctl to run as the root user.

Splunk Enterprise 10.6.x uses PostgreSQL as an internally managed sidecar service for features including the PostgreSQL-backed KV Store. PostgreSQL prevents its server processes and pg_ctl from running as root. As a result, Splunk Enterprise must run as a dedicated non-root service account for PostgreSQL dependent workflows.

Do NOT use the following command as a workaround :

$ splunk start --run-as-root

The "--run-as-root" option is deprecated and planned for removal in a future Splunk Enterprise release. It also does not allow PostgreSQL to run as root, so it does not resolve a failed KV Store migration to PostgreSQL. 

To resolve the failed migration :

1. Stop Splunk Enterprise completely using the approved local operational procedure.

2. Correct mixed ownership below the Splunk installation directory so that the intended dedicated Splunk service account owns the required Splunk files and directories. For example :

   chown -R splunk:splunk /opt/splunk

3. Preserve the stale runtime marker by moving it to a timestamped backup file :

$ mv $SPLUNK_HOME/var/run/splunk/kvstore_mongo_to_postgres_migration_inprogress $SPLUNK_HOME/var/run/splunk/kvstore_mongo_to_postgres_migration_inprogress.bak.$(date +%Y%m%d%H%M%S)

Do not permanently delete the marker before preserving a copy.

4. Start Splunk Enterprise using the dedicated non-root Splunk service account.

5. Clean up the previous failed migration state :

$ curl -sku "admin:<password>" -X POST "https://127.0.0.1:8089/services/kvstore/migrateToPostgres/cleanup?output_mode=json"

6. Enable the PostgreSQL migration in server.conf :

[kvstore]
postgresMigrateOnStartup = true

7. Restart Splunk Enterprise and validate the migration :

$ splunk restart 
$ splunk show kvstore-postgres-migration-status 
$ splunk show kvstore-status --verbose

A successful migration reports 100.00% completion. A subsequent restart can clear residual MongoDB related status messages after PostgreSQL becomes the active KV Store backend.

Related Links

Splunk Docs - Upgrade to a cohosted KV Store

Splunk Docs - Run Splunk Enterprise as a different or non-root user