Edge Processor Control Plane Sidecar Not Running and 404 Error on Edge Processor API Endpoint

Applicability

Description

After upgrading Splunk from 10.2.2 to 10.2.4 (or similar version upgrades) and migrating from RHEL 8.10 to RHEL 9.7 users may encounter the following issue:

A Service Health alert reports the Edge Processor Control Plane sidecar is not running.

The Edge Processor Control Plane sidecar is not running or is not available.

A 404 error is returned when accessing the Edge Processor API endpoint:

404 GET /en-US/splunkd/__raw/servicesNS/admin/splunk_pipeline_builders/edge/v1alpha3/processors?output_mode=json

The following errors appear in $SPLUNK_HOME/var/log/splunk/splunkd.log:

ERROR DataManagementHealthReportManager [8897 DMHealthReportManagerThread] - Requested health status='' does not map to any StatusType
WARN DataManagementHealthReportManager [8897 DMHealthReportManagerThread] - edge-processor-config sidecar is not running, last seen running 2026-06-05T14:51:34.041582065+02:00

Investigation

1. Check the PostgreSQL sidecar log for authentication failures:

grep -E 'Deployment Id to be used|password authentication failed|Failed PgBouncer auth bootstrap|connection established|Database bootstrapping completed|Readiness status updated' $SPLUNK_HOME/var/log/splunk/sup-pkg-postgres-stdout.log | tail -50

If password authentication failed or Failed PgBouncer auth bootstrap messages appear, the deployment identity file is out of sync with PostgreSQL.

2. Check the Edge Processor config sidecar log for database initialization failures:

grep -E 'BeanCreationException|DataAccessResourceFailureException|initDatabaseMigration' $SPLUNK_HOME/var/log/splunk/sup-pkg-edge-processor-config-stdout.log | tail -20

If BeanCreationException or DataAccessResourceFailureException messages appear, the Edge Processor configuration service cannot connect to the database.

3. Verify the current deployment identity file exists and is not empty:

cat $SPLUNK_HOME/var/packages/data/postgres/.deployment_id

4.Compare the value in .deployment_id with the identity stored in the Secret Store:

curl --fail --silent --show-error --insecure \
  -H "Authorization: Bearer <session_token>" \
  "https://127.0.0.1:8089/servicesNS/nobody/system/storage/passwords/postgres:deployment_id?output_mode=json"

The identity is in entry[0].content.clear_password. A mismatch confirms the root cause.

Root cause

The Edge Processor Control Plane sidecar failure is caused by a PostgreSQL credential desynchronization. During the upgrade, the internal PostgreSQL sidecar loses alignment between the deployment identity file ($SPLUNK_HOME/var/packages/data/postgres/.deployment_id) and the password used to authenticate to the bundled PostgreSQL instance. This causes the sidecar bootstrap to fail, which prevents the Edge Processor configuration service from completing its database migrations (initDatabaseMigration), resulting in a cascading failure across Edge Processor, Data Management, and Data Orchestrator services.

Resolution

Two resolution paths are available. Option A preserves existing data and is recommended. Option B is faster but removes all data stored for SPL2, Data Orchestrator, and Data Management.

Option A: Staged Recovery (Preserves Existing Data)

Note: Proceed sequentially. Stage 2 should only be attempted after Stage 1 succeeds.

Stage 1: Restore Automatic postgres_admin Authentication

The sidecar derives the postgres_admin password from the deployment identity stored at:

$SPLUNK_HOME/var/packages/data/postgres/.deployment_id

Step 1 — Back up the current identity file:

cp -p $SPLUNK_HOME/var/packages/data/postgres/.deployment_id \
  $SPLUNK_HOME/var/packages/data/postgres/.deployment_id.backup

Step 2 — Test candidate deployment identities.

Replace the contents of .deployment_id with each candidate value below, restart Splunk, then inspect the PostgreSQL sidecar log:

grep -E \
  'Deployment Id to be used|password authentication failed|Failed PgBouncer auth bootstrap|connection established|Database bootstrapping completed|Readiness status updated' \
  $SPLUNK_HOME/var/log/splunk/sup-pkg-postgres-stdout.log | tail -50

Candidate identities to test:

standalone-7C1CD90C-9F95-4424-86E1-58D6F383FDEF
standalone-CD6DC756-6BA0-4DA3-9058-752ECA580B1F
standalone-30279EF3-D32B-4196-AA11-9414C5332900

Note: A failed identity typically takes approximately 6–7 minutes to exhaust its retry cycle. If repeated authentication failures appear in the log, proceed to the next candidate without waiting for the full cycle.

Success indicators:

PgBouncer auth bootstrap: connection established
Database bootstrapping completed successfully
Readiness status updated: Ready

Step 3 — If the candidates above do not work, retrieve stored identities from the Secret Store.

Set the environment variables:

export SPLUNKD_URL='https://127.0.0.1:8089'
export SPLUNK_TOKEN='<your_session_token>'

Retrieve the primary identity:

curl --fail --silent --show-error --insecure \
  -H "Authorization: Bearer ${SPLUNK_TOKEN}" \
  "${SPLUNKD_URL}/servicesNS/nobody/system/storage/passwords/postgres:deployment_id?output_mode=json"

Retrieve the historical identity:

curl --fail --silent --show-error --insecure \
  -H "Authorization: Bearer ${SPLUNK_TOKEN}" \
  "${SPLUNKD_URL}/servicesNS/nobody/system/storage/passwords/postgres:deployment_guid?output_mode=json"

The identity value is in entry[0].content.clear_password. If it differs from the current .deployment_id file, replace the file contents with the retrieved value and restart Splunk.

If none of the identities restore automatic authentication, stop here and contact Splunk Support with the log output. A separate recovery procedure will be provided.

Stage 2: Synchronize Existing PostgreSQL Users

Proceed only after Stage 1 succeeds.

Step 1 — Verify superuser connectivity:

sudo -u splunk env \
  LD_LIBRARY_PATH=$SPLUNK_HOME/lib \
  PGPASSFILE=$SPLUNK_HOME/var/packages/data/postgres/.pgpass \
  $SPLUNK_HOME/bin/psql \
  -h 127.0.0.1 -p 5432 \
  -U postgres_admin -d postgres -w \
  -c 'SELECT current_user, rolsuper FROM pg_roles WHERE rolname = current_user;'

Expected output:

current_user   | rolsuper
----------------+---------
 postgres_admin | t

If .pgpass authentication fails, retrieve the postgres_admin password from the Secret Store:

curl --fail --silent --show-error --insecure \
  -H "Authorization: Bearer ${SPLUNK_TOKEN}" \
  "${SPLUNKD_URL}/servicesNS/nobody/system/storage/passwords/postgres:postgres_admin?output_mode=json" |
  python3 -c 'import json,sys; print(json.load(sys.stdin)["entry"][0]["content"]["clear_password"])'

Security reminder: Do not share this password in Slack, email, or the support case.

Connect interactively using the retrieved password:

sudo -u splunk env \
  LD_LIBRARY_PATH=$SPLUNK_HOME/lib \
  $SPLUNK_HOME/bin/psql \
  -h 127.0.0.1 -p 5432 \
  -U postgres_admin -d postgres -W

Step 2 — List existing roles:

\du

Step 3 — Resynchronize application role passwords.

For each role that appears in the \du output, retrieve its password from the Secret Store and reset it in PostgreSQL. Example for acc.dbuser:

Retrieve the password:

curl --fail --silent --show-error --insecure \
  -H "Authorization: Bearer ${SPLUNK_TOKEN}" \
  "${SPLUNKD_URL}/servicesNS/nobody/system/storage/passwords/postgres:acc.dbuser?output_mode=json" |
  python3 -c 'import json,sys; print(json.load(sys.stdin)["entry"][0]["content"]["clear_password"])'

Reset the password in the open psql session:

\password "acc.dbuser"

Paste the retrieved password when prompted and confirm.

Repeat for each of the following roles, only if they appear in \du output:

do.root
do.dbuser
kvstore.pdl
acc.dbuser
oas.root
oas.dbuser
ep_preview.root
ep_preview.dbuser

Skip any role that does not appear in \du or does not have a corresponding Secret Store entry. Do not manually create missing roles — the sidecar creates them automatically once Stage 1 succeeds.

Option B: Full PostgreSQL Data Directory Reset (Does Not Preserve Data)

Use this option only if existing data stored for SPL2, Data Orchestrator, or Data Management is not required.

Reset:

$SPLUNK_HOME/bin/splunk stop
sudo mv $SPLUNK_HOME/var/packages/data/postgres \
  $SPLUNK_HOME/var/packages/data/postgres.new_backup
$SPLUNK_HOME/bin/splunk start

Rollback (if needed):

$SPLUNK_HOME/bin/splunk stop
sudo mv $SPLUNK_HOME/var/packages/data/postgres \
  $SPLUNK_HOME/var/packages/data/postgres.new_backup
sudo mv $SPLUNK_HOME/var/packages/data/postgres.backup \
  $SPLUNK_HOME/var/packages/data/postgres
$SPLUNK_HOME/bin/splunk start

KV Store and core Splunk functions continue operating normally regardless of which option is chosen.

Expected Outcome:
- Data Management Services Service Health is healthy.
- The Edge Processors page is accessible.
- The Pipelines page is accessible and the existing pipelines are visible.

Related Links

Postgres not initializing due to 'password authentication failed for user "postgres_admin"' error