Script genRootCA.sh Produces Invalid CA Certificate on Splunk 10.x (OpenSSL v3)

Applicability

Description

The genRootCA.sh script uses the following OpenSSL command to sign the CA certificate:

openssl x509 -req -in careq.pem -sha256 -extensions v3_ca -signkey cakey.pem -out cacert.pem -days 3650

In OpenSSL v3, the -extensions flag is silently ignored unless accompanied by -extfile pointing to a configuration file containing the referenced section. The following warning is emitted during execution:

Warning: ignoring -extensions option without -extfile

As a result, the generated cacert.pem is a Version 3 X.509 certificate with no extensions, meaning it has no basicConstraints and is not recognized as a CA certificate. Any certificate signed by it will fail strict path validation.

Investigation

Step 1: Check the CA certificate version and extensions

$SPLUNK_HOME/bin/splunk cmd openssl x509 -in $SPLUNK_HOME/etc/auth/cacert.pem -noout -text | grep "Version:"

If the output shows Version: 3 (0x2), proceed to check for extensions:

$SPLUNK_HOME/bin/splunk cmd openssl x509 -in $SPLUNK_HOME/etc/auth/cacert.pem -noout -text | grep -A2 "Basic Constraints"

If there is no output, the CA certificate is missing the required basicConstraints extension.

Step 2: Verify certificate purpose

$SPLUNK_HOME/bin/splunk cmd openssl x509 -in $SPLUNK_HOME/etc/auth/cacert.pem -noout -purpose

A broken certificate will show:

SSL server CA : No 
SSL client CA : No 
Any Purpose CA : No

A healthy CA certificate will show:

SSL server CA : Yes
SSL client CA : Yes
Any Purpose CA : Yes

Step 3: Verify server certificate purpose

$SPLUNK_HOME/bin/splunk cmd openssl x509 -in $SPLUNK_HOME/etc/auth/server.pem -noout -purpose

A healthy server certificate will show:

SSL server : Yes
SSL client : Yes
SSL server CA : No

If SSL server or SSL client shows No, the server certificate is not usable for TLS connections.

Step 4: Verify the certificate chain

$SPLUNK_HOME/bin/splunk cmd openssl verify -CAfile $SPLUNK_HOME/etc/auth/cacert.pem $SPLUNK_HOME/etc/auth/server.pem

A broken CA will produce:

error 79 at 1 depth lookup: invalid CA certificate

Step 5: Check for symptoms in Splunk logs

Common error patterns in splunkd.log:

SSL peer certificate validation failed: certificate signature failure

KVStore stuck in "starting" state

TcpOutputFd - Connection to indexer failed due to SSL validation error

SSL peer certificate validation failed: unsuitable certificate purpose

Resolution

This issue is resolved in Splunk Enterprise 10.4.2. The updated certificate-generation scripts pass the OpenSSL configuration file using -extfile when applying certificate extensions. This causes OpenSSL v3 to include the required CA extensions(such as basicConstraints = CA:TRUE)in the generated CA certificate.

After generating certificates, validate them with:

$SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile <path>/cacert.pem <path>/server.pem

A successful validation returns:

<path>/server.pem: OK

For releases earlier than Splunk Enterprise 10.4.2, use the below workaround or generate the certificates with an OpenSSL configuration that explicitly supplies the required CA and server-certificate extensions.

Workaround: Regenerate the CA certificate with correct extensions

1. Back up existing certificates:

cd $SPLUNK_HOME/etc/auth 
mkdir -p backup 
cp ca.pem cacert.pem ca.srl server.pem dp_ca.pem dp_ca.srl server_dp.pem server_dp_ca.pem server_dp_ca.srl backup/

2. Run the existing genRootCA.sh as-is.

/opt/splunk/bin/genRootCA.sh -d /opt/splunk/etc/auth

3. Re-sign the CSR with the correct extensions:

Create an OpenSSL extensions configuration file:

cd $SPLUNK_HOME/etc/auth 
cat > ca_ext.cnf <<EOF 
[v3_ca] 
basicConstraints = critical, CA:TRUE 
keyUsage = critical, keyCertSign, cRLSign 
subjectKeyIdentifier = hash 
EOF

4. Re-generate cacert.pem from the existing careq.pem + cakey.pem with the extensions file:

$SPLUNK_HOME/bin/splunk cmd openssl x509 -req -in careq.pem -passin pass:password -sha256 -extensions v3_ca -extfile ca_ext.cnf -signkey cakey.pem -out cacert.pem -days 3650

5. Assemble ca.pem and clean up:

cat cacert.pem cakey.pem > ca.pem 
rm ca_ext.cnf

6. Verify the fix for the CA cert:

$SPLUNK_HOME/bin/splunk cmd openssl x509 -in cacert.pem -noout -text | grep -A2 "Basic Constraints"

Expected output: 

            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical

7. Regenerate server.pem and restart:

mv $SPLUNK_HOME/etc/auth/server.pem $SPLUNK_HOME/etc/auth/backup/server.pem
$SPLUNK_HOME/bin/splunk restart

Splunk will auto-generate a new server.pem signed by the corrected CA on startup.

8. Validate the new server certificate:

$SPLUNK_HOME/bin/splunk cmd openssl verify -verbose -x509_strict -CAfile $SPLUNK_HOME/etc/auth/cacert.pem $SPLUNK_HOME/etc/auth/server.pem

Expected output: server.pem: OK

Distributed Environments

This article covers CA/server certificate regeneration on a single instance only. For search head clusters, indexer clusters, or any deployment where certificates must be rotated across multiple nodes without service disruption, follow article: Replacing TLS CA Certificates in Distributed Splunk Enterprise Deployments instead of applying these steps node-by-node.

Rollback

If issues arise, restore from backup and restart:

cp $SPLUNK_HOME/etc/auth/backup/* $SPLUNK_HOME/etc/auth/ $SPLUNK_HOME/bin/splunk restart

Related Links

Replacing TLS CA Certificates in Distributed Splunk Enterprise Deployments