Unable to generate or use tokens

Applicability

Description

There is the need to generate new Tokens but when going to:
Settings > Tokens > there is a pop-up error stating:

KVStore is not ready. Token auth system will not work.

Investigation

It was verified that the issue is not related to certificate expiration with the openssl command:

splunk@XXXX~/bin$ ./openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem
notAfter=Sep 26 23:59:59 2024 GMT

The status status of the KV Store is “failed”.

 splunk@XXXX:~/bin$ ./splunk show kvstore-status

This member:
                           backupRestoreStatus : Ready
                                      disabled : 0
                                          guid : XXXXXX-XXXX-XXX-XXX-XXXXXXXXX
                                          port : 8191
                                    standalone : 1
                                        status : failed
                                 storageEngine : wiredTiger External KVStore Information:
                                        status : ready
        Service Info
                                           env : XXXXXXXX.splunk.com
                                     namespace : kvstore
                                        tenant : XXXXXXX
                                          type : scs

splunk@XXXX ~/bin$ grep -i error /opt/splunk/var/log/splunk/splunkd.log | tail -30 | grep "KV"

01-02-2024 14:38:46.813 +0000 ERROR KVServiceClient [3202862 TcpChannelThread] - KVServiceClient transaction failed after 0 retries.
01-02-2024 14:39:04.984 +0000 ERROR KVStorageProvider [3222763 TcpChannelThread] - An error occurred during the last operation ('replSetGetStatus', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '127.0.0.1:8191']

Resolution

• It was found the issue presents any time a user (no matter the role or capabilities) goes to Settings > Users and Authentication > Tokens
  
  As per investigation:
  On Splunk Enterprise Environments:
   Verify KV Store status from all members as they need to be in a running state otherwise continue with the steps.
  

  SPLUNK_HOME/bin/ ./splunk show kvstore-status

   

  Verify SSL certificate validation using the below open SSL command while in the SPLUNK_HOME/bin/ directory
   

  ./openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

   

  In the case that all SH's SSL certificates are expired.
  Next step , renewed the SSL cert in all SHs by following the below steps
  Rename the server.pem
  Restart the Splunk service
  Check the SSL certificate validity and in all SHs it got renewed
  Check the status of the KV store and it changed from Starting state to ready state
   
  Splunk Cloud Environment:
  A support case should be open due to the lack of CLI back-end access:
  The support team should be able to Restart the SH and get the status of the KV Store up without issues.