Splunk Add-on for Office 365 : Throwing 503 Tunnel Proxy ERRORs

Applicability

Description

Splunk Add-on for Office 365 data ingestion stopped due to Proxy problems.

Investigation

While doing troubleshooting using below SPL:

SPL:

index=_internal *O365* log_level=ERROR

ERRORs:

2024-08-08 18:07:13,415 level=ERROR pid=81361 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=utils.py:wrapper:72 | datainput=b'Management_Activity_Input_Azure' start_time=1723108028 | message="Data input was interrupted by an unhandled exception." 
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/connectionpool.py", line 712, in urlopen
    self._prepare_proxy(conn)
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/connectionpool.py", line 1012, in _prepare_proxy
    conn.connect()
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/connection.py", line 374, in connect
    self._tunnel()
  File "/opt/splunk/lib/python3.9/http/client.py", line 930, in _tunnel
    raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 503 Service UnavailableDuring handling of the above exception, another exception occurred:Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='manage.office.com', port=443): Max retries exceeded with url: /api/v1.0/5ccfd302-9b3d-4ad3-b2e2-1742c9befc0d/activity/feed/subscriptions/list?PublisherIdentifier=5ccfd302-9b3d-4ad3-b2e2-1742c9befc0d (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 503 Service Unavailable')))

The error logs indicate a failure in establishing a tunnel connection through the proxy, resulting in OSError: Tunnel connection failed: 503 Service Unavailable. The traceback shows that the issue originates from the proxy configuration, leading to repeated MaxRetryError in the requests library due to the inability to connect to the proxy. This suggests potential issues with the proxy configuration at either the OS or network level.

Resolution

To address the 503 Service Unavailable errors: Need to reach out to the network or OS team in order to review and resolve proxy-related issues. Once the request is able to go properly from the proxy, then it will allow the connection to go through.