ERROR curl: (60) SSL certificate problem: self-signed certificates in certificate chain Splunk

Applicability

Description

When sending an event to HEC the following error appears:

Investigation

The error above is reproduced by running the following command:

curl "https://http-inputs.mysplunkserver.com:8088/services/collector" \
    -H "Authorization: Splunk CF179AE4-3C99-45F5-A7CC-3284AA91CF67" \
    -d '{"event": "Hello, world!", "sourcetype": "manual"}'

Resolution

The error message received ('curl: (60) SSL certificate problem: self-signed certificates in certificate chain Splunk') indicates that HEC is requesting a valid official CA-signed certs, not self-signed. If the curl command is run with -k to skip insecure verify works, it will work.

By default, when enabling HEC and choose to enable SSL (enableSSL=1), Splunk uses the same self-signed certs as for port 8089.

To secure HEC, it is recommended to change the configuration in inputs.conf. 

To resolve this issue, follow the next steps:

1. Generate a new certification chain (new pem with server cert + key + CA cert), if possible, with CA certification valid. Please refer to the following guides: 

1. 1. How to get third-party certificates
   2. How to prepare your signed certificate for Splunk

2. Then, edit the HEC stanza in inputs.conf:

[http]
disabled = 0
index = main
enableSSL = 1
serverCert = <full path to the certificate chain pem file>
sslPassword = <password for server key used in chain

3. Then, restart splunkd and run the curl command again, it should work as expected now.

If a new error message  'curl (28) SSL connection timeout' is displayed, double-check that the combined certificate file created is right by reviewing it using a text editor. Its contents must contain, in the following order:
-The server certificate
-The private key
-The certificate authority certificate

Then, run a test on this certificate as indicated below:

1. Open a terminal (Terminal 1), and run this command

openssl s_server -accept 10000 -cert <fullpath of server.pem>

2. Then, open a duplicate terminal (Terminal 2). In the Terminal 2, run this command

openssl s_client -connect 127.0.0.1:10000 -CAfile <Fullpath of rootcertificate.pem>

If the certificate is right, the result received should be something like:  'Verify return code: 0 (ok)'

Related Links

 Troubleshoot HTTP Event Collector (HEC)

HTTP Event Collector (HEC) inputs.conf

Use cURL to manage HTTP Event Collector tokens, events, and services