Applicability

Description

Default Splunk Enterprise installations utilize an out-of-the-box root Certificate Authority ($SPLUNK_HOME/etc/auth/cacert.pem) and an instance certificate ($SPLUNK_HOME/etc/auth/server.pem) for splunkd management communication (8089) and local KV Store engine authentication (8191).

When default certificates near expiration (e.g., standard default CA expiration in Jan 2027), they must be replaced. In standard Splunk Enterprise deployments where peer certificate verification (sslVerifyServerCert = false, requireClientCert = false) is not enforced across nodes, a transitional multi-CA bundle is not required. Individual nodes only enforce certificate validity locally (matching private key, valid date range, and valid chain between local server.pem and local cacert.pem).

This procedure outlines generating a compliant CA on the Cluster Manager (or a designated instance), distributing public CA trust, signing compliant 2048-bit server certificates, and safely rolling the instances without downtime.

Investigation

1. Confirm Peer and Client Certificate Verification Settings

Verify that inter-node certificate verification is not active. Run the following checks across representative instances:

$SPLUNK_HOME/bin/splunk btool server list --debug | grep -i -E "sslVerifyServerCert|requireClientCert|cliVerifyServerName"
$SPLUNK_HOME/bin/splunk btool inputs list --debug | grep -i -E "requireClientCert|serverCert"
$SPLUNK_HOME/bin/splunk btool outputs list --debug | grep -i -E "sslVerifyServerCert|clientCert"

• Expected Result for Standard Deployments:

  • sslVerifyServerCert = false

  • requireClientCert = false

  • cliVerifyServerName = false

• Note on Ingestion Ports (9997): If forwarders send data via a separate custom certificate specified in inputs.conf / outputs.conf (e.g., in /etc/auth/mycerts/), replacing files under $SPLUNK_HOME/etc/auth/ will not impact data forwarding.

2. Inspect Existing Expiration and Baseline Compliance

Check the expiration dates and baseline strict verification status on an instance:

cd $SPLUNK_HOME/etc/auth
$SPLUNK_HOME/bin/splunk cmd openssl x509 -in cacert.pem -noout -subject -issuer -enddate
$SPLUNK_HOME/bin/splunk cmd openssl x509 -in server.pem -noout -subject -issuer -enddate
$SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile cacert.pem server.pem

• Note: On stock Splunk 10.4.x instances using the original shipped certificates, strict verification will report error 85 (Missing Authority Key Identifier) and error 79 (invalid CA certificate) because the shipped 2017 CA was X.509 v1. The procedure below resolves both errors.

3. Pre-Flight Deployment Health Check

Before initiating changes:

• Indexer Cluster: Run $SPLUNK_HOME/bin/splunk show cluster-status --verbose on the CM. Ensure Replication factor met = YES, Search factor met = YES, All data is searchable = YES, and no fixup tasks are active.

• Search Heads / KV Store: Run $SPLUNK_HOME/bin/splunk show kvstore-status --verbose. Ensure status : ready.

Resolution

Phase 0: Backup Existing Auth Directories

On every instance participating in the renewal, take a backup of the $SPLUNK_HOME/etc/auth directory:

tar -czvf /tmp/splunk_auth_backup_$(hostname)_$(date +%Y%m%d).tgz -C $SPLUNK_HOME/etc auth

Phase 1: Generate the Compliant Root CA (on Cluster Manager or Designated Admin Host)

Because OpenSSL 3 ignores the -extensions flag unless -extfile is supplied (SPL-295526), generate the new CA files using the following corrected sequence:

# 1. Create a staging directory 
mkdir -p $SPLUNK_HOME/etc/auth/newca 
chmod 700 $SPLUNK_HOME/etc/auth/newca 
cd $SPLUNK_HOME/etc/auth/newca # 2. Run genRootCA.sh to generate initial key and cert structure 
# When prompted for a password, enter 'password' (or your organizational default) 
$SPLUNK_HOME/bin/genRootCA.sh -d $SPLUNK_HOME/etc/auth/newca -l 2048 # 3. Create an OpenSSL extension file ensuring X.509 v3 CA constraints 
cat << 'EOF' > ca_ext.cnf 
[v3_ca] 
basicConstraints = critical, CA:TRUE 
keyUsage = critical, digitalSignature, cRLSign, keyCertSign 
subjectKeyIdentifier = hash 
authorityKeyIdentifier = keyid:always,issuer 
EOF # 4. Re-sign cacert.pem with the extension file to inject v3 Basic Constraints 
$SPLUNK_HOME/bin/splunk cmd openssl x509 -in cacert.pem -signkey cakey.pem -passin pass:password -days 3650 -out cacert.pem -extfile ca_ext.cnf -extensions v3_ca # 5. Recombine cacert.pem and cakey.pem into ca.pem 
cat cacert.pem cakey.pem > ca.pem 
chmod 600 ca.pem cakey.pem cacert.pem # 6. Validate that the CA is now a recognized, valid v3 CA 
$SPLUNK_HOME/bin/splunk cmd openssl x509 -in cacert.pem -noout -text | grep -A2 "Basic Constraints"

Passphrase Best Practice Note for Step 2 above:

• CA Passphrase: any secure passphrase for the root CA can be used. Splunk runtime does not need this password; it is only used during administrative certificate generation and signing (in Step4 above and during createssl server-cert).

• Server Certificate Passphrase: If a passphrase other than password is chosen when generating server.pem in Phase 3 below, update sslPassword = <chosen_password> in $SPLUNK_HOME/etc/system/local/server.conf under [sslConfig] on every instance so splunkd can load the certificate.

Expected Output:

X509v3 Basic Constraints: critical 
     CA:TRUE

Generate a checksum file for distribution tracking:

sha256sum cacert.pem ca.pem > newca.sha256

Phase 2: Distribute CA Files to All Target Instances

Transfer the new CA artifacts to the staging directory on each target instance (Indexers, Search Heads, Heavy Forwarders, License Managers):

# Example from CM/Admin host via SSH/SCP: scp $SPLUNK_HOME/etc/auth/newca/cacert.pem $SPLUNK_HOME/etc/auth/newca/ca.pem $SPLUNK_HOME/etc/auth/newca/newca.sha256 <admin_user>@<target_node>:/tmp/

On each target instance, move files to the secure staging location:

sudo mkdir -p /opt/splunk/etc/auth/newca 
sudo chmod 700 /opt/splunk/etc/auth/newca 
sudo cp /tmp/cacert.pem /tmp/ca.pem /tmp/newca.sha256 /opt/splunk/etc/auth/newca/ 
sudo chown -R splunk:splunk /opt/splunk/etc/auth/newca 
sudo chmod 600 /opt/splunk/etc/auth/newca/* 
rm -f /tmp/cacert.pem /tmp/ca.pem /tmp/newca.sha256 # Verify checksum integrity matches the source CA 
cd /opt/splunk/etc/auth/newca 
sha256sum -c newca.sha256

Important Security Notice: ca.pem contains the root CA's private signing key. Ensure transit occurs exclusively over secure channels (SSH/SCP) into restricted directories (chmod 700). Enforce strict file permissions (chmod 600, owned by splunk:splunk) on all instances. Once server.pem is generated, consider removing ca.pem from distributed nodes and archiving it securely on the administrative host, as Splunk runtime operations only require cacert.pem

Phase 3: Generate and Validate Host Server Certificates (server.pem)

Execute this step on each individual Splunk instance (including the CM/Admin host itself):

cd $SPLUNK_HOME/etc/auth/newca # Generate server.pem signed by the local staged CA 
# Passphrase prompts: 
# - "Enter a secret passphrase when requested" -> enter 'password' 
# - "Enter the root CA passphrase" -> enter 'password' 
$SPLUNK_HOME/bin/splunk createssl server-cert -d $SPLUNK_HOME/etc/auth/newca -n server -c SplunkServerDefaultCert -l 2048 chmod 600 server.pem

Pre-Deployment Strict Verification:

Perform strict OpenSSL verification against the staged files on each host. Do not proceed if any errors are returned.

$SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile cacert.pem server.pem

Expected Result:

server.pem: OK

Verify key structure and passphrase readability:

# Check RSA key opens cleanly with the standard password 
$SPLUNK_HOME/bin/splunk cmd openssl rsa -in server.pem -passin pass:password -noout -check # Verify structure shows CERTIFICATE, ENCRYPTED PRIVATE KEY, CERTIFICATE 
grep -- "-----BEGIN" server.pem

Phase 4: Controlled Rolling Switchover

Apply the staged certificates host-by-host using the following strict sequencing.

Step 4.1: Cluster Manager (CM)

1. Overwrite live auth files from the staging directory:

   cp /opt/splunk/etc/auth/newca/cacert.pem /opt/splunk/etc/auth/cacert.pem 
   cp /opt/splunk/etc/auth/newca/server.pem /opt/splunk/etc/auth/server.pem 
   cp /opt/splunk/etc/auth/newca/ca.pem /opt/splunk/etc/auth/ca.pem

2. Restart Splunk:

   $SPLUNK_HOME/bin/splunk restart

3. Validation:

   $SPLUNK_HOME/bin/splunk show cluster-status --verbose 
   $SPLUNK_HOME/bin/splunk cmd openssl s_client -connect localhost:8089 -showcerts </dev/null 2>/dev/null | $SPLUNK_HOME/bin/splunk cmd openssl x509 -noout -subject -issuer -enddate

   Confirm the active issuer is CN=SplunkCA, O=SplunkInc and the indexer cluster remains searchable and all peers report Up.

Step 4.2: Indexer Cluster Peers (One Peer at a Time)

Perform the switchover sequentially across each peer (e.g., idx01, then idx02).

1. Take the peer offline gracefully to protect bucket searchability:

   $SPLUNK_HOME/bin/splunk offline

2. Copy new certificate files into place:

   cp /opt/splunk/etc/auth/newca/cacert.pem /opt/splunk/etc/auth/cacert.pem 
   cp /opt/splunk/etc/auth/newca/server.pem /opt/splunk/etc/auth/server.pem 
   cp /opt/splunk/etc/auth/newca/ca.pem /opt/splunk/etc/auth/ca.pem

3. Start the peer:

   $SPLUNK_HOME/bin/splunk start

4. Validation Gate on Cluster Manager:
   Before touching the next peer, check on the CM:

   $SPLUNK_HOME/bin/splunk show cluster-status --verbose

   Wait until the peer shows Status: Up, Searchable: YES, and any transient fixup tasks complete (No fixup tasks in progress = YES).

Step 4.3: Search Head(s)

1. Copy new certificate files into place:

   cp /opt/splunk/etc/auth/newca/cacert.pem /opt/splunk/etc/auth/cacert.pem 
   cp /opt/splunk/etc/auth/newca/server.pem /opt/splunk/etc/auth/server.pem 
   cp /opt/splunk/etc/auth/newca/ca.pem /opt/splunk/etc/auth/ca.pem

2. Restart Splunk:

   $SPLUNK_HOME/bin/splunk restart

3. Validate KV Store initialization and distributed search:

   $SPLUNK_HOME/bin/splunk show kvstore-status --verbose

   Verify status : ready. Run a test search across all indexers via Splunk Web or CLI (splunk search "index=_internal | head 5").

Step 4.4: Heavy Forwarders / Standalone Roles

1. Apply certificates:

   cp /opt/splunk/etc/auth/newca/cacert.pem /opt/splunk/etc/auth/cacert.pem 
   cp /opt/splunk/etc/auth/newca/server.pem /opt/splunk/etc/auth/server.pem 
   cp /opt/splunk/etc/auth/newca/ca.pem /opt/splunk/etc/auth/ca.pem 
   $SPLUNK_HOME/bin/splunk restart

2. Check splunkd.log to ensure successful connection to indexers and deployment servers:

   tail -n 50 $SPLUNK_HOME/var/log/splunk/splunkd.log | grep -i -E "TcpOutputProc|SSL|connection"

Phase 5: Post-Implementation Cleanup

Once all instances are running on the renewed certificates and the deployment is fully validated:

1. Secure or remove the staging directory across all hosts:

   rm -rf $SPLUNK_HOME/etc/auth/newca

2. Retain the Phase 0 backup archives (/tmp/splunk_auth_backup_*.tgz) in a secure administrative location until the next maintenance cycle.

Rollback

If a host fails to start or reports persistent TLS errors during the change:

1. Restore the original files from the Phase 0 backup:

   cd $SPLUNK_HOME/etc/auth 
   tar -xzvf /tmp/splunk_auth_backup_$(hostname)_*.tgz auth/

2. Ensure correct ownership and permissions:

   chown splunk:splunk cacert.pem server.pem ca.pem 
   chmod 600 cacert.pem server.pem ca.pem

3. Restart Splunk:

   $SPLUNK_HOME/bin/splunk restart

4. Verify cluster and service health.

Related Links

Renew existing TLS certificates

Prepare TLS certificates for the Splunk platform

Perform an indexer cluster rolling restart

Script genRootCA.sh Produces Invalid CA Certificate on Splunk 10.x (OpenSSL v3)