Edge Processor nodes stuck in Pending after upgrade due to TLS key mismatch errors

Applicability

Splunk Enterprise 10.4.4, Splunk Enterprise 10.6.0.5 Edge Processor 1.12.4 Data Management Plane for Splunk Enterprise S2S destinations configured with TLS/mTLS

Description

After the Splunk is upgraded from 10.4.3 to 10.4.4 or 10.6.0.5, Edge Processor nodes never reach a healthy state.
In Data Management > Edge Processors, the nodes switch between Pending and Offline and show no instance version. Uninstalling and reinstalling the Edge Processor service does not help, and all data routed through the Edge Processor stops reaching the indexers, including HTTP Event Collector (HEC) traffic.

The Edge Processor log shows the OpenTelemetry collector exiting while it builds the S2S exporter for a TLS-enabled destination:

otel-collector failed to apply new settings. The collector will remain in idle state: failed to setup configuration components: failed to build pipelines: failed to create "S2S/envs.splunk.local.shunits.local.apps.splunk_pipeline_builders.<destination_name>" exporter for data type "logs": tls: failed to find any PEM data in key input

When the certificate of the default destination is edited, a different error appears:

failed to setup configuration components: failed to build pipelines: failed to create "S2S/envs.splunk.local.shunits.local.ingest.edge_processor.default_telemetry_indexer" exporter for data type "metrics": tls: private key does not match public key

The certificates themselves are valid. The same client certificate, private key, and CA chain work on a newly created destination, while the existing destinations keep failing.
Turning off TLS on the destination brings the nodes back to healthy. However, indexers that require TLS on the receiving port then reject the connection:

SSL routines::wrong version number

The problem lies in the TLS settings saved on destinations that existed before the upgrade.
When one of these destinations is edited, the Edge Processor receives a key that is empty or does not belong to the certificate. A new destination starts with no saved values, so it is not affected. The default destination cannot be deleted or recreated, so it needs a full TLS reset instead.

Investigation

Confirm the collector error on the Edge Processor nodes

On each Edge Processor node, search the Edge Processor log for the TLS errors. The default install directory is /opt/splunk-edge:

grep -E "failed to find any PEM data|private key does not match public key" /opt/splunk-edge/var/log/edge.log

The exporter name in the error tells which destination is failing. A name ending in splunk_pipeline_builders.<destination_name> points to a user-created destination. A name ending in edge_processor.default_telemetry_indexer points to the default destination.

Rule out a bad certificate or key

The Edge Processor accepts only an unencrypted private key in PEM format. Check that the key file has no passphrase:

grep -E "ENCRYPTED|Proc-Type" client_key.pem

This command should return nothing. The file should start directly with -----BEGIN PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY-----, with no text above the header.

Confirm the private key belongs to the client certificate. The two hashes must be identical:

openssl x509 -noout -pubkey -in client_cert.pem | openssl sha256
openssl pkey -pubout -in client_key.pem | openssl sha256

Confirm the three files complete an mTLS handshake with an indexer receiving port:

openssl s_client -connect <indexer_host>:9997 -cert client_cert.pem -key client_key.pem -CAfile ca_cert.pem

Confirm the problem is the saved destination settings

Create a temporary test destination in Data Management > Destinations using the same three files. Point a single low-impact pipeline at it.

If the test destination sends data over TLS while the original destination keeps failing, the files are good. The problem is the TLS values saved on the original destination, and the resolution below applies.

Resolution

Prepare the certificate files

Use three separate PEM files, each pasted into its own field. Do not use a combined forwarder file such as server.pem that holds the certificate, key, and CA together.

Client certificate: the leaf certificate only
Client private key: the unencrypted private key only
CA certificates: the CA chain only

If only have a combined or passphrase-protected file, extract the pieces with OpenSSL. The second command prompts for the key passphrase:

# Extract the leaf certificate
openssl x509 -in combined.pem -out client_cert.pem# Extract and decrypt the private key
openssl pkey -in combined.pem -out client_key.pem

For user-created destinations

Create a new destination in Data Management > Destinations using the three files, with TLS enabled. Move every pipeline that uses the old destination to the new one.

Once no pipeline references the old destination, delete it so it cannot be selected again by mistake.

For the default destination

The default destination cannot be recreated, so reset its TLS settings while the Edge Processor service is stopped. Plan a maintenance window, because ingestion through the Edge Processor stops during this procedure.

Stop the Edge Processor service on every node:

sudo systemctl stop splunk-edge

In Data Management > Destinations, open default_destination, select Stop authenticating with TLS, and save.

Restart Splunk on the Data Management Plane instance:

$SPLUNK_HOME/bin/splunk restart

Log back in and open default_destination again. Turn TLS back on and paste each of the three files into its own field. Point the destination back at the indexers and save.

Start the Edge Processor service on every node:

sudo systemctl start splunk-edge

Verify

Confirm the nodes report healthy in Data Management > Edge Processors. Then check that the TLS errors no longer appear in the log:

tail -f /opt/splunk-edge/var/log/edge.log | grep -E "failed to find any PEM data|private key does not match public key"

If either error comes back, turn TLS off on the destination right away to restore data flow. 

Related Links

Send data from Edge Processors to Splunk platform deployments using S2S
Obtain TLS certificates for data sources and Edge Processors

Revision History

Revision	Publish Date	Comments
1.0	14-Oct-2026	Initial Release