Splunk Add-on for Microsoft O365 : Throwing 429 ERRORs from Graph API input.

Applicability

Description

The Splunk Add-on for Microsoft O365 may encounter 429 "TenantThresholdExceeded" errors from the Microsoft Graph API when request limits are exceeded.

Investigation

While checking internal logs using SPL, found repeating ERROR logs:

SPL:

index=_internal *O365* log_level=ERROR

ERROR Logs:

2024-08-06 12:13:41,896 level=DEBUG pid=1657759 tid=MainThread logger=splunk_ta_o365.common.portal pos=portal.py:get:838 | datainput=b'MailboxUsageMailboxCounts_TEST_v451' start_time=1722946419 | message="429:{"error":{"code":"TenantThrottleThresholdExceeded","message":"Please retry later"}}" 

While further checking at the code level, found that the add-on has already doing exponential retries to recover from this ERROR Code.

Hence, it doesn’t affect data ingestion.

Add-on's code portion which handles this ERROR is as below:

           # if API throttle limit is reached
            if response.status_code == 429:
                logger.info("Reached API throttle limit")
                message = str(response.status_code) + ":" + str(response.text)
                logger.debug(message)
                raise CustomThrottleException(response)

           # if API throttle limit is reached
            if response.status_code == 429:
                logger.info("Reached API throttle limit")
                message = str(response.status_code) + ":" + str(response.text)
                logger.debug(message)
                raise CustomThrottleException(response)

Resolution

1) It is being concluded that the ERROR "Code 429" doesn’t affect data ingestion.
2) In order to avoid these errors, use  Microsoft Document and make the required changes.