FIPS Forwarder not Connecting to Splunk Cloud - FIPS_DIGESTINIT:disabled for fips.

Applicability

Description

When connecting to Splunk Cloud the following error appears:
 

ERROR SSLCommon - Can't read key file 
/opt/splunk/etc/apps/100_<stackname>_splunkcloud/default/<stackname>_server.pem errno=101351587
error:060A80A3:digital envelope routines:FIPS_DIGESTINIT:disabled for fips.

Investigation

This occurs due to the private key encryption found in 100_<stackname>_splunkcloud/default/<stackname>_server.pem, it's not FIPS compliant.

NOTE: Only FedRAMP Subscriptions has FIPS enabled in Splunk Cloud, connecting Forwarders with FIPS enabled to Splunk Cloud with no FedRAMP subscription will not work.

Splunk Cloud Platform Service Details | Splunk Docs

Resolution

Follow these steps:
///put the decryption up front first
1. Copy  $SPLUNK_HOME/etc/apps/100_<stackname>_splunkcloud/ to another location as a backup.

2. Go to  $SPLUNK_HOME/etc/apps/100_<stackname>_splunkcloud/default/<stackname>_server.pem and use the text editor of your preference, it will see something like this (This also applies to RSA private keys):

-----BEGIN CERTIFICATE-----
MIICET...
-----END CERTIFICATE-----
-----BEGIN EC PRIVATE KEY----- 
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-256-CBC,EC584E340374F03B50B6AEBE8B4A6CF4
260c5n6WcHc+jpyd2EgO...
-----END EC PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIICFTCCAb...
-----END CERTIFICATE-----

3. Copy the private key section in a new file

-----BEGIN EC PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-256-CBC,EC584E340374F03B50B6AEBE8B4A6CF4
260c5n6WcHc+jpyd2EgO...
-----END EC PRIVATE KEY-----

4. Execute the following command: 

$SPLUNK_HOME/bin/splunk cmd openssl ec -in /<path>/<to>/newfile.key -out /<path>/<to>/decryp.key

the command will prompt a passphrase, take it from

$SPLUNK_HOME/etc/apps/100_<stack_name>_splunkcloud/local/outputs.conf
sslPassword = <PASSWORD>

Note: Unencrypt this password before using it due to is Splunk Encrypted, run this:

splunk show-decrypted --value 'sslpassword_here'

*This command has to be run in the same instance where is the 100_<stackname>_splunkcloud


5. This will generate the decrypted output of newfile.key in decryp.key, then use the following command to re-encrypt decryp.key using the same password in:  $SPLUNK_HOME/etc/apps/100_<stackname>_splunkcloud/local/outputs.conf

$SPLUNK_HOME/bin/splunk cmd openssl pkcs8 -topk8 -v2 aes-256-cbc 
-in /<path>/<to>/decryp.key -out /<path>/<to>/re_encryp.key

*The previous command it's a one-line command.
* Use the decrypted password from step 4 to unencrypt the previous key.

6. Once encrypted go to $SPLUNK_HOME/etc/apps/100_<stackname>_splunkcloud/default/<stackname>_server.pem and replace the private key string with the new one, then save the changes

7. Restart splunkd running  $SPLUNK_HOME/bin/splunk restart.