Splunk 10 OpenSSL 3.0 Upgrade

Applicability

Description

The upgrade of OpenSSL from version 1.0.2 to 3.0 in Splunk 10 is a major change that is not generally backward compatible. This means applications designed for OpenSSL 1.x may require significant modifications to function correctly with OpenSSL 3.0.

The primary breaking changes are:

1. OpenSSL Version Upgrade (1.0.2 to 3.0): The version of OpenSSL shipped with the Splunk platform has been updated from 1.0.2 to 3.0. This upgrade enhances security and efficiency. Direct usage of deprecated APIs within OpenSSL 1.0.2 will cause failures in Splunk 10. This is especially critical for users operating Splunk in FIPS mode, as low-level deprecated APIs are directly disabled.

2. TLS Protocol Deprecation: Fields like *sslVersions* no longer support SSL3. TLS1.0 and TLS1.1 are deprecated.

   • Users must update all configurations to use TLS1.2 only.

   • This aligns with industry best practices and compliance mandates (e.g., NIST 800-52, PCI DSS 3.1) and lays the groundwork for future TLS 1.3 adoption.

3. X509v3 CA Certificates: Any intermediate CA certificate used with Splunk must now include the X509v3 Basic Constraints extension with CA:TRUE.

   • Certificates without this extension will not be considered valid CA certificates and will be ignored.

   • Self-signed root CAs are exempt from this specific requirement.

Investigation

Before upgrading to Splunk 10, or if issues are encountered post-upgrade, perform the following investigations:

Pre-Upgrade Configuration Validations

1. TLS Version Inspection:

   • Inspect all sslVersions and sslVersionsForClient configurations across the Splunk environment.

   • SPL Search to detect deprecated TLS versions:

     | rest /services/configs/conf-*
     | search title=*ssl*Versions*
     | eval invalid_tls = if(match(value, "(?i)tls1.0|tls1.1|ssl3"), "Yes", "No")
     | where invalid_tls="Yes"
     | table title, stanza, value, eai:acl.app, eai:acl.owner
     | rename title AS "Configuration File", stanza AS "Stanza", value AS "Configured Value", eai:acl.app AS "App Context", eai:acl.owner AS "Owner"
     | sort "Configuration File", "App Context", "Stanza"

   • If Splunk is running in FIPS mode, sslVersions and sslVersionsForClient must be set to tls1.2.

2. X509 Basic Constraints Validation:

   • Ensure that any certificate used as a CA certificate in Splunk contains the X509v3 Basic Constraints extension with CA:TRUE.

   • You can check this from the command line using openssl:

     openssl x509 -in <path_to_cacert_file> -text -noout | grep 'CA:TRUE'

3. App Compatibility:

   • Use the Splunk Enterprise beta to determine if your apps are compatible with OpenSSL 3. This is especially important for apps that directly interact with low-level OpenSSL APIs.
   • App developers should have received communications about these changes in CY25.

Post-Upgrade Error Messages and Investigation

Monitor _internal index for the following error messages:

• Received fatal SSL3 alert. ssl_state='SSL negotiation finished successfully', alert_description='certificate unknown'.

  • Cause: A server (TLS) or client (for mTLS) certificate is not valid (e.g., expired, unreadable).

  • SPL Search:

    index=_internal "Received fatal SSL3 alert" "alert_description='certificate unknown'" OR "alert_description='certificate expired'" OR "alert_description='bad certificate'"
    | rex "ssl_state='(?[^']+)'"
    | table _time, host, source, ssl_state, alert_description, log_level
    | sort - _time

• Received fatal SSL3 alert. ssl_state='<...>', alert_description='unknown CA'.

  • Cause: The verifying endpoint (client or server) cannot find the CA certificate(s) in its trust store to verify the receiving certificate.

  • SPL Search:

    index=_internal "Received fatal SSL3 alert" "alert_description='unknown CA'"
    | rex "ssl_state='(?[^']+)'"
    | table _time, host, source, ssl_state, alert_description, log_level
    | sort - _time

• X509 certificate (CN=<...>) common name (<...>) did not match any allowed names (<...>)

  • Cause: The common name (CN) of the presented certificate does not match the expected value configured via sslServerNameToVerify.

  • SPL Search:

    index=_internal "X509 certificate" "common name" "did not match any allowed names"
    | rex "CN=(?[^)]+).*allowed names \((?[^)]+)\)"
    | table _time, host, source, certificate_cn, allowed_names, log_level
    | sort - _time

• X509 certificate (<...>) is not a valid Certificate Authority (CA) certificate under Common Criteria: required=X509_CA_V3_basicConstaints; actual=<...>

  • Cause: Splunk is attempting to use a CA certificate that does not include CA:TRUE under the X509v3 Basic Constraints extension.

  • SPL Search:

    index=_internal "X509 certificate" "not a valid Certificate Authority" 
    | rex "certificate \((?[^)]+)\).*required=(?[^;]+); actual=(?[^)]+)"
    | table _time, host, source, certificate_name, required_value, actual_value, log_level
    | sort - _time

• Received fatal SSL3 alert. ssl_state='SSL negotiation finished successfully', alert_description='protocol version'. or 140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

  • Cause: Client and server do not have a common TLS version during protocol message exchange.

  • SPL Search:

    index=_internal "Received fatal SSL3 alert" "alert_description='protocol version'" OR "unknown protocol"
    | rex "ssl_state='(?[^']+)'.*alert_description='(?[^']+)'"
    | table _time, host, source, ssl_state, alert_description, log_level
    | sort - _time

• error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher or error:140740B5:SSL routines:SSL23_CLIENT_HELLO:no ciphers available

  • Cause: Client and server TLS configurations do not contain any mutually shared ciphers compatible with the TLS version being used, or the client has no ciphers available.

  • SPL Search:

    index=_internal "no shared cipher" OR "no ciphers available"
    | table _time, host, source, log_level, _raw
    | sort - _time

Resolution

For App Developers

• Update App Configurations: Update all app configurations to use TLS1.2 only.

• OpenSSL API Usage: If your app directly uses low-level OpenSSL APIs, refer to the OpenSSL Migration Guide (external link) for detailed information on updating your code. Deprecated APIs will cause failures.

• Beta Testing: Use the Splunk Enterprise beta to determine if your app is compatible with OpenSSL 3 and if its configurations are correct.

• Update App Versions: Ensure your apps are updated to the latest versions, especially those that have been updated by their developers for OpenSSL 3.0 compatibility.

For Users

1. Update TLS Configurations:

   • In all relevant .conf files (e.g., server.conf, web.conf, inputs.conf, outputs.conf), replace any sslVersions or sslVersionsForClient values other than tls1.2 with tls1.2. The highest precedence configuration should never contain a TLS version less than TLS1.2.
   • Ensure all components of a Splunk deployment (Search Heads, Indexers, Forwarders, etc.) are configured to use TLS1.2.

2. Update/Verify CA Certificates:

   • Update or replace any CA certificate that does not include CA:TRUE in the X509v3 Basic Constraints extension.
   • Note: Self-signed root CAs are exempt from this CA:TRUE requirement.

3. Monitor with CMC/MC:

   • Use the Centralized Management Console (CMC) or Monitoring Console (MC) (with updated Splunk Health Assistant Add-on - SHAA) to check for issues related to TLS protocol versions and unknown CAs.

4. General Recovery Actions for Error Messages:

   • Certificate Unknown/Expired/Bad:

     • Check that the certificate is readable (e.g., with splunk cmd openssl) and valid. It should not be expired.
     • Verify that the clocks on both server and client ends are synchronized.

   • Unknown CA:

     • Verify that sslRootCAPath is correctly set on the Splunk instances for both client and server.
     • Add a valid CA certificate into the the trust store.
     • Verify that the CA certificate is readable (e.g., with splunk cmd openssl).

   • Common Name Mismatch:

     • Inspect sslServerNameToVerify in the configuration stanza for the problematic channel.
     • Check if sslServerNameToVerify and the certificate common name match. If not:
       • Set sslServerNameToVerify to match the certificate's CN.
       • Alternatively, replace the certificate with one that has a CN matching sslServerNameToVerify.
       • Discouraged: Set sslVerifyServerName to false in the configuration stanza (this reduces security).

   • Not a Valid CA Certificate:

     • Inspect sslRootCAPath in the configuration stanza for the problematic channel.
     • Check if all certificates at the sslRootCAPath contain CA:TRUE under the X509v3 Basic Constraints extension. If any do not, replace or remove them.

   • Protocol Version Mismatch / No Shared Cipher / No Ciphers Available:

     • Verify that secure communication with TLS is enabled on both client and server.
     • Verify that the TLS settings on the client and server have common attributes such as TLS protocol versions (must be TLS1.2) and cipher suites.
     • Inspect sslVersions (or sslVersionsForClient) and cipherSuite settings in the relevant configuration stanzas on both client and server.
     • Determine the highest mutually shared TLS version (should be TLS1.2) and check for mutually shared ciphers compatible with this version. If none, add compatible ciphers to both server and client configurations.

5. Use Splunk Enterprise Beta: Utilize the Splunk Enterprise beta to test your key apps and workflows for impact before a full production upgrade.

Related Links

Preparing to upgrade from 9.x to Splunk Enterprise and Cloud Platform 10.0

Splunk 10: Release Notes