Splunk Add-on for AWS : Resolving OSError - 407 Proxy Authentication Required on Tunnel Connection Failure

Applicability

Description

An issue may occur in the Splunk Add-on for AWS where logs for CloudWatch Logs Input fail to ingest into Splunk after configuring the Input. This problem occurs when the environment uses a proxy, but the Splunk Add-on for AWS is not configured with the necessary proxy settings, causing log collection attempts to receive the following error response:

"OSError: Tunnel connection failed: 407 Proxy Authentication Required"

Investigation

Below SPL query can used to review the internal logs of the Input: 

index=_internal sourcetype=aws:cloudwatchlogs:log ERROR

Following errors can be observed in the internal logs : 

2025-05-27 14:25:31,259 level=ERROR pid=1796617 tid=Thread-4149 logger=splunk_ta_aws.modinputs.cloudwatch_logs.aws_cloudwatch_logs_data_loader pos=aws_cloudwatch_logs_data_loader.py:index_data:158 | datainput="Test_input" region="eu-west-1" log_group="/aws/lambda/test-log-group" | message="Failed to collect cloudwatch logs, error=Traceback (most recent call last):
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/urllib3/connectionpool.py", line 713, in urlopen
self._prepare_proxy(conn)
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/urllib3/connectionpool.py", line 1015, in _prepare_proxy
conn.connect()
File "/opt/splunk/etc/apps/Splunk_TA_aws/lib/urllib3/connection.py", line 374, in connect
self._tunnel()
File "/opt/splunk/lib/python3.7/http/client.py", line 931, in _tunnel
message.strip()))
OSError: Tunnel connection failed: 407 Proxy Authentication Required

This error indicates that the add-on failed to establish a tunnel through the proxy because it lacked valid authentication credentials, preventing it from collecting CloudWatch logs.

Resolution

To resolve the issue, configure the proxy setting in Splunk Add-on for AWS using the below steps : 

1. On the Splunk Web home page, click Splunk Add-on for AWS in the left navigation bar.
2. Click Configuration in the app navigation bar.
3. Click the Proxy tab.
4. Select the Enable box to enable the proxy connection and fill in the fields required for proxy.
5. Click Save.

Related Links

Splunk Add-on for AWS : Configure a proxy connection