Splunk 10.x upgrade: splunkd fails to start due to legacy 1024-bit distServerKeys.

Applicability

Description

Problem:
After upgrading Splunk Enterprise from 9.x to 10.x, splunkd may fail to start (or repeatedly crash/abort). The crash commonly shows signal 6 (Aborted) and the crashing thread DistributedPeerMonitorThread. In splunkd.log, you may also see OpenSSL signing errors such as “invalid key length” 

Cause:
Splunk 10.x uses OpenSSL 3 (and may run with the FIPS provider enabled). If the distributed search keypair under:
$SPLUNK_HOME/etc/auth/distServerKeys/
was generated by an older Splunk release, it may be 1024-bit RSA. OpenSSL 3/FIPS requires RSA keys ≥ 2048 bits, so request signing fails during distributed peer monitoring/heartbeat activity, which can trigger splunkd abort on startup.

Resolution:
Regenerate/replace the distributed search keys in:
$SPLUNK_HOME/etc/auth/distServerKeys/
with 2048-bit (or higher) RSA keys, set proper ownership/permissions, and restart Splunk. After updating the keys, Splunk should start normally under the Enterprise license.

Verification:
Confirm the key size of private.pem (and corresponding public key) is 2048-bit+, then restart Splunk and verify splunkd remains up and no longer logs “invalid key length” signing errors.

Investigation

• key-length/ signing error:
  

  02-21-2026 21:09:54.843 -0500 WARN UserManagerPro [64446 DistributedPeerMonitorThread] - Error signing request for remote peer="https://10.200.8.184:8089": 'Failed to init EVP_PKEY_sign, error=SSL error code=478150761 message="error:1C800069:Provider routines::invalid key length"

• Crash log info:
  

  [build e2d18b4767e9] 2026-02-21 21:09:54
  Received fatal signal 6 (Aborted) on PID 63773.
  Cause:
  Signal sent by PID 63773 running under UID 0.
  Crashing thread: DistributedPeerMonitorThread
  Registers:
  RIP: [0x00007F8C8294C6AF] gsignal + 271 (libc.so.6 + 0x4E6AF)
  ...
  Backtrace (PIC build):
  [0x00005591F417B935] _ZN32HeartbeatTransactionSynchronizer13ScopedUpdater5applyENS_6StatusE + 389 (splunkd + 0x3DB6935)
  [0x00005591F414F3D3] _ZN33GetServerInfoHeartbeatTransaction12dataFinishedEv + 1379 (splunkd + 0x3D8A3D3)
  [0x00005591F4A081DA] _ZN20HttpClientConnection10parseReplyEPKcS1_ + 1802 (splunkd + 0x46431DA)
  ....
  [0x00005591F4A6C863] _ZN15TcpOutboundLoop3runEv + 19 (splunkd + 0x46A7863)
  [0x00005591F226B401] _ZN28DistributedPeerMonitorThread4mainEv + 49 (splunkd + 0x1EA6401)
  [0x00005591F4B2469E] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 46 (splunkd + 0x475F69E)
  [0x00005591F4B247C3] _ZN6Thread8callMainEPv + 163 (splunkd + 0x475F7C3)
  [0x00007F8C8326ACFA] ? (libpthread.so.0 + 0x7CFA)
  Linux / aza0prd0lspl54 / 5.15.0-315.196.5.2.el8uek.x86_64 / #2 SMP Sat Dec 20 05:32:36 PST 2025 / x86_64
  Last few lines of stderr (may contain info on assertion failure, but also could be old):
  FIPS provider enabled. name: OpenSSL FIPS Provider, version: 3.0.9, buildinfo: 3.0.9, status: Success
  2026-02-21 20:32:08.511 -0500 splunkd started (build e2d18b4767e9) pid=50044
  WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
  2026-02-21 20:32:53.099 -0500 splunkd started (build e2d18b4767e9) pid=51166
  2026-02-21 20:51:00.551 -0500 splunkd started (build e2d18b4767e9) pid=56971
  2026-02-21 21:01:03.624 -0500 splunkd started (build e2d18b4767e9) pid=60557
  2026-02-21 21:09:06.532 -0500 Interrupt signal received sent by PID 63431, command="./splunk restart" (UID 0, same as my group)
  2026-02-21 21:09:23.702 -0500 splunkd started (build e2d18b4767e9) pid=63773

Crash characterization (observed behaviour):
From the crash artifacts:

• splunkd terminates with fatal signal 6 (Aborted).
• The crashing thread is DistributedPeerMonitorThread.
• The stack trace repeatedly involves:
  • HeartbeatTransactionSynchronizer::ScopedUpdater::apply
  • GetServerInfoHeartbeatTransaction::dataFinished
  • HttpClientConnection::parseReply

Suspected root cause:
Based on behavior consistent with known issues, a likely cause is the presence of legacy distributed search RSA keys located in:
/opt/splunk/etc/auth/distServerKeys

In older deployments these keys may be 1024-bit RSA (generated by very old Splunk releases). With Splunk 10 + OpenSSL 3, RSA keys must be 2048-bit or larger, and smaller keys can cause signing/authentication failures.

How to validate the hypothesis:
Check the RSA key length for the key in distServerKeys:

• cd /opt/splunk/etc/auth/distServerKeys
• openssl pkey -in private.pem -text -noout | head -n 5

If the output indicates 1024-bit (example: Private-Key: (1024 bit, 2 primes)), that strongly supports this hypothesis and is not supported for Splunk 10.
If the key is 2048-bit+, the investigation needs to be continued.

Resolution

1. Stop Splunk

   • $SPLUNK_HOME/bin/splunk stop

2. Back up existing distributed search keys

   • cd $SPLUNK_HOME/etc/auth/distServerKeys
   • cp -p private.pem private.pem.bak
   • cp -p trusted.pem trusted.pem.bak

3. Regenerate RSA keys at 2048-bit (or higher)

   • openssl genrsa -out private.pem 2048
   • openssl rsa -in private.pem -pubout -out trusted.pem

4. Set correct permissions/ownership

   • chown splunk:splunk private.pem trusted.pem
   • chmod 600 private.pem
   • chmod 644 trusted.pem

5. Start Splunk and validate

   • $SPLUNK_HOME/bin/splunk start (or restart)
   • Confirm splunkd stays running and the OpenSSL error “invalid key length” no longer appears in splunkd.log.

Result: Splunk 10.x starts normally 

Related Links

The pair of distributed search keys need to be regenerated if it is not in valid length

Splunk keeps crashing on SHs after Splunk 10 upgrade