AWS Elastic Load Balancer (ELB) Health check to Splunk ports fail

Applicability

Description

The Splunk Search Heads sit behind an AWS Elastic Load Balancer (ELB), the users are experiencing intermittent disconnects to the Splunk Web UI and also see API requests fail. The ELB logs show that it is responding to the client with HTTP response code 502.

This can occur when the keep-alive timeout of Splunk is shorter than the idle timeout of the Load Balancer as the Load Balancer will try to reuse a connection that has already been closed on the Splunk side.

While this article refers to the AWS ELB, the same applies to any load balancer that is used in front of Splunk Search Heads or if you are using HTTP Event Collector (HEC) behind a Load Balancer you may also experience a similar issue. 

Investigation

1. Check the AWS ELB logs to confirm the response received. In the example below we can see that the elb_status_code is "502" and the target_status_code is "-",  the request_processing_time and the target_processing_time both have values of 0.000 but the response_processing_time is -1.

https 2024-05-02T06:01:56.160457Z app/splunk/f3a5d86aaaaaaaa 10.0.1.4:60912 10.0.0.19:8089 0.000 0.000 -1 502 - 2798 688 "POST https://splunk.example.com:8089//services/search/jobs HTTP/1.1" "Go-http-client/1.1" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:ap-southeast-2:2121212:targetgroup/splunk-alb-8089/79dsagfadgs "Root=1-65635412125463" "splunk.example.com" "arn:aws:acm:ap-southeast-2:56342315458:certificate/ea4623546354XXXXXXX" 0 2024-05-02T06:01:56.159000Z "forward" "-" "-" "10.0.0.19:8089" "-" "-" "-"

If the request_processing_time, target_processing_time, and response_processing_time fields in the access logs are each set to value -1. The issue is not related to the timeout settings and refer to the AWS ELB ALB troubleshoot 502 errors
2. Confirm the value of the AWS ELB Connection Idle Timeout in the AWS Management Console, the default is 60 seconds. Further information on configuring the Idle timeout can be found here: AWS Config Idle Timeout
3. Confirm the value of the Splunk keep-alive timeout. 
For the Splunk Management port (8089 by default) this can be found with the following btool command run in the CLI from $SPLUNK_HOME/bin

./splunk btool server list httpServer | grep busyKeepAliveIdleTimeout

For the Splunk Web UI (port 8000 by default) this can be found with the following btool command run in the CLI from $SPLUNK_HOME/bin

./splunk btool web list settings | grep busyKeepAliveIdleTimeout

In the below example both keep-alive timeouts are the default of 12 seconds which is shorter than the AWS default of 60 seconds. 

splunker@so1:/opt/splunk/bin$ ./splunk btool server list httpServer | grep busyKeepAliveIdleTimeout
busyKeepAliveIdleTimeout = 12splunker@so1:/opt/splunk/bin$ ./splunk btool web list settings | grep busyKeepAliveIdleTimeout
busyKeepAliveIdleTimeout = 12

Resolution

If it is found that the Splunk keep-alive timeout is shorter than the AWS load balancer Connection Idle Timeout either the Splunk keep-alive timeout will need to be increased or the load balancer Idle Timeout will need to be reduced.

For example when using the default AWS Connection Idle Timeout of 60 seconds we can increase the Splunk keep-alive timeout to 65 seconds. This can be done by adding or updating the settings in the following conf files

Splunk Management port:

$SPLUNK_HOME/etc/system/local/server.conf

[httpServer]
busyKeepAliveIdleTimeout = 65

Splunk Web UI:

$SPLUNK_HOME/etc/system/local/web.conf

[settings]
busyKeepAliveIdleTimeout = 65

If you are using HTTP Event Collector (HEC) behind a Load Balancer this can be configured in:
$SPLUNK_HOME/etc/system/local/inputs.conf

[http]
busyKeepAliveIdleTimeout = 65

Unless advised by Splunk the recommended maximum busyKeepAliveIdleTimeout value is 120 seconds. Configuring this setting to a value that is too large can result in file descriptor exhaustion due to idling connections.

Related Links

AWS - load balancer troubleshooting

AWS - ELB ALB troubleshoot 502 errors

Server.conf.spec

web.conf.spec

inputs.conf.spec