SAML idp cert are being deleted, this causes users not to be able to log in

Applicability

Splunk Enterprise 7.3.3 Splunk Enterprise 8.2.1

Description

After upgrading search heads from 7.3.3 to 8.2.1, the SAML idp certificates were being deleted randomly from each search head, this occurred several times a week.

Investigation

From the logs the following can be seen

09:47 the certs being deleted and at 10:24 the Splunk Admin is putting them back

host     time                                   action path
xxxxxx 2021-08-20 10:24:40.498 add /opt/splunk/etc/auth/idpCerts/idpCertChain_1/cert_3.pem
xxxxxx 2021-08-20 10:24:40.496 add /opt/splunk/etc/auth/idpCerts/idpCertChain_1/cert_2.pem
xxxxxx 2021-08-20 10:24:40.493 add /opt/splunk/etc/auth/idpCerts/idpCertChain_1/cert_1.pem
xxxxxx 2021-08-20 10:24:40.390 add /opt/splunk/etc/auth/idpCerts/idpCertChain_1
xxxxxx 2021-08-20 10:24:40.387 add /opt/splunk/etc/auth/idpCerts/idpCert.pem
xxxxxx 2021-08-20 10:24:40.283 update /opt/splunk/etc/auth/idpCertsxxxxxx 2021-08-20 10:11:58.389 delete-parent /opt/splunk/etc/auth/idpCerts/idpCertChain_1/cert_3.pem
xxxxxx 2021-08-20 10:11:58.387 delete-parent /opt/splunk/etc/auth/idpCerts/idpCertChain_1/cert_2.pem
xxxxxx 2021-08-20 10:11:58.384 delete-parent /opt/splunk/etc/auth/idpCerts/idpCertChain_1/cert_1.pem
xxxxxx 2021-08-20 10:11:58.381 delete /opt/splunk/etc/auth/idpCerts/idpCertChain_1
xxxxxx 2021-08-20 10:11:58.379 delete /opt/splunk/etc/auth/idpCerts/idpCert.pem
xxxxxx 2021-08-20 10:11:58.275 update /opt/splunk/etc/auth/idpCerts
xxxxxx 2021-08-20 09:47:58.605 update /opt/splunk/etc/auth/idpCerts/idpCert.pem

 

Resolution

From the logs  the certs are being deleted due to an authentication config reload: 

08-26-2021 10:28:15.915 +0200 INFO ApplicationUpdater 
[27908 ConfReplicationThread] - Reloading via GET on /servicesNS/nobody/system/authentication/providers/services/_reload
08-26-2021 10:28:16.232 +0200 WARN AuthorizationManager 
[27908 ConfReplicationThread] - Capability 'extra_x509_validation' is not recognized by Splunk. Ignoring...
.
.
08-26-2021 10:28:16.324 +0200 WARN SAMLConfig
[27908 ConfReplicationThread] - Use RSA-SHA256, RSA-SHA384, or RSA-SHA512 for 'signatureAlgorithm' rather than 'RSA-SHA1'
08-26-2021 10:28:16.324 +0200 INFO SAMLConfig 
[27908 ConfReplicationThread] - No values for 'inboundSignatureAlgorithm' specified, using default values
08-26-2021 10:28:16.324 +0200 WARN SAMLConfig 
[27908 ConfReplicationThread] - Use RSA-SHA256, RSA-SHA384, or RSA-SHA512 for 'inboundSignatureAlgorithm' rather than 'RSA-SHA1'
08-26-2021 10:28:16.324 +0200 INFO SAMLConfig 
[27908 ConfReplicationThread] - No values for 'inboundDigestMethod' specified, using default values
08-26-2021 10:28:16.324 +0200 WARN SAMLConfig 
[27908 ConfReplicationThread] - Use SHA256, SHA384, or SHA512 for 'inboundDigestMethod' rather than 'SHA1'
08-26-2021 10:28:16.324 +0200 INFO SAMLConfig 
[27908 ConfReplicationThread] - Skipping :idpCert.pem because it does not begin with idpCertChain_ when populating idpCertChains
08-26-2021 10:28:16.324 +0200 WARN UserManager 
[27908 ConfReplicationThread] - Could not find role mapping 'roleMap_SAML' in authentication.conf
08-26-2021 10:28:16.324 +0200 WARN UserManager 
[27908 ConfReplicationThread] - Could not find role mapping 'rolemap_SAML' in authentication.conf
08-26-2021 10:28:16.326 +0200 INFO AuthenticationProviderSAML
[27908 ConfReplicationThread] - Removed cert chain dir:idpCertChain_1
08-26-2021 10:28:16.331 +0200 INFO CertStorageProvider 
[27908 ConfReplicationThread] - Synced all certificates from KVStore to local filesystem

It was found that the IDP certs were being deleted due to kvstore not being in sync with the IDP cert on disk, The kvstore held no idp certs, so deleted them from the disk

1. First fix: To fix this the user needs to delete and re-configure sam idp certs, and make sure the certs get added to the KVStore,

An example call to the endpoint would be constructed like this

curl -k -XDELETE -u <local_splunk_user> https://<host>:<mPort>/services/authentication/providers/SAML/<saml_stanza_name>

Where local_splunk_user is a Splunk user, and saml_stanza_name is the name of their SAML stanza (default is "saml") is the correct endpoint to remove the SAML configuration.

The IdP certs stored in the KVStore will be removed by this action, they will still have to be manually removed from the filesystem if they don't want copies in the idpCerts directory anymore though. Upon re-configuring SAML these removed certs will not be restored.

Just to be clear as well, by removing the SAML configuration, the SHC will revert to Splunk authentication until SAML is reconfigured and the specified saml stanza will be deleted from authentication.conf.

Please note that once the saml config is deleted the SHC will revert to Splunk authentication until it is reconfigured. Please be aware of this so you can login to the instance.

2. Second workaround: You can disable kvstore certs replication.

• In $SPLUNK_HOME/etc/system/local/authentication.conf under the saml stanza

[saml]
replicateCertificates=false

• Remove old certs, and add new ones in the SHC
• $SPLUNK_HOME/bin/splunk reload auth
• $SPLUNK_HOME//splunk rolling-restart shcluster-members

See authorization.conf

Related Links

REST Access Authentication Providers SAML stanza_name

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release