Troubleshooting Splunk Enterprise Search Head PostgreSQL Core Dump Crash Loop and Disk Space Exhaustion (invalid record length: expected at least 24, got 0)

Applicability

Description

After initiating a restart of Splunk Enterprise on a Search Head Cluster member, the node experiences rapid disk space exhaustion on /opt/splunk. Inspection of $SPLUNK_HOME/var/run/splunk/ reveals multiple core dump files named postgres-6.core.<pid>  being generated every 30 seconds.

Because disk capacity becomes completely exhausted, related sidecar processes (such as etcd and splunk-supervisor) begin throwing I/O errors (no space left on device and connection refused), leading service degradation on the affected Search Head.

Investigation

1. Verify Core Dump Signal:
   Run: file $SPLUNK_HOME/var/run/splunk/postgres-6.core.*
   Finding: Confirms process terminated via Signal 6 (SIGABRT).

2. Inspect Database Cluster State:
Run: /opt/splunk/bin/pg_controldata "$PGDATA" | egrep 'Database cluster state|Latest checkpoint'
Finding: Reports Database cluster state: shut down in recovery.

3. Check for Torn WAL Records:
Run: /opt/splunk/bin/pg_waldump -s <Latest_Checkpoint_Location> "$PGDATA/pg_wal/<REDO_WAL_file>"
Finding: error in WAL record: invalid record length at <LSN>: expected at least 24, got 0
Root Cause: An earlier abrupt termination left a truncated 0-byte WAL record header. During startup recovery, PostgreSQL panics (SIGABRT) to prevent data corruption, resulting in an automated restart crash loop that generates core dumps until disk space is exhausted.

4. Check Patroni Cluster Status:
Run: /opt/splunk/bin/splunk cmd python3 /opt/splunk/bin/patronictl -c /opt/splunk/var/packages/data/postgres/patroni/patroni.yml list
Finding: The affected node shows Role: Replica and State: start failed. Healthy cluster members show Role: Replica or Leader in State: streaming or running on an advanced timeline.

We Can Check Internal logs as well using the

SPL Query index=_internal sourcetype=sup-pkg-postgres-stdout 

Resolution