Data ingestion stops after upgrading heavy forwarder to version 10.4.x

Applicability

Description

To understand why this behaviour changed between versions, take a look at how Splunk handles backend architecture updates and configuration file (specifically outputs.conf). When a major version upgrade occurs (especially to 10.4), the underlying data routing engine and communication protocols are optimised, which can change how legacy configuration parameters are interpreted.
- Protocol and Engine Updates: In version 10.2, explicitly setting compression to disabled (or false) was a fully supported parameter that the Splunk-to-Splunk (S2S) forwarding protocol understood and accepted.
- Deprecation/Incompatibility: By version 10.4, Splunk likely updated its forwarding architecture to optimize network traffic and pipeline performance. This update introduced a requirement for the forwarder and indexer to dynamically negotiate compression (hence the auto setting). Because the underlying code handling the forwarding pipeline changed in 10.4, the legacy hardcoded disabled setting became incompatible, causing the pipeline to fail and data ingestion to stop.

Resolution

Splunk Cloud 10.4.2604 TLS Updates:
The Splunk platform now supports TLS 1.3 (alongside TLS 1.2) for all public-facing connections, enhancing security with stronger encryption, eliminating outdated cipher suites, and delivering better performance and efficiency. Due to the deactivation of TLS 1.0 and TLS 1.1, environments with the SSL/TLS versions older than TLS 1.2 will experience connectivity impacts. Recommendations are to validate the followings:
- Forwarder Versions & Configs: Verify that HFs are using a supported forwarder version and ensure that TLS 1.2 is not explicitly excluded in the sslVersions setting within your outputs.conf.
- Network Paths: Verify that TLS 1.2 or higher can be used on your network paths to the cloud. The same verification is required for network paths used for non-cloud Federated Search, REST API access, and HEC access.
- App/Add-on Overrides: Check if there are any overrides for sslVersions or [tcp-ssl:<port>] configured by Apps or Add-ons at the Search Head (SH) layer.

To support TLS 1.3, it is recommended to proactively configure the compression setting to compressed = true in the outputs.conf as early as possible.

In Cribl, set the compression setting to automatic instead of disabled.

Related Links

What's new version 10.4.2604

Splunk 10 OpenSSL 3.0 Upgrade