Splunk Add-on for Microsoft O365 : Throwing 504 Unknown ERROR for AuditSign.In Logs

Applicability

Description

Splunk Add-on for Office 365 throwing 504 ERRORs from AuditSign.In logs, which results in data ingestion corruption with an add-on.

Investigation

While checking internal logs, found the below error message:

SPL:

index=_internal *O365* log_level=ERROR

ERROR:

2024-07-16 14:54:31,075 level=ERROR pid=4809 tid=MainThread logger=splunk_ta_o365.modinputs.graph_api.GraphApiConsumer pos=GraphApiConsumer.py:run:481 | datainput=b'graph_audit_log_signins' start_time=1721136750 | message="Error retrieving Graph API Messages." exception=504:{"error":{"code":"UnknownError","message":"","innerError":{"date":"2024-07-16T14:54:31","request-id":"ee34bdd7-1251-4bb4-8a80-8a2073c5534a","client-request-id":"ee34bdd7-1251-4bb4-8a80-8a2073c5534a"}}}

From the above error logs, it seems to be an issue from Microsoft's side itself, as per the below documentation: 

Microsoft Document 1

Microsoft Document 2

To verify further details, an internal ticket was raised for the Add-on engineering team to determine if the issue was originating from the Add-on side.

The Add-on engineering team consulted with the Microsoft team regarding recurring 504 errors impacting the overall data collection of the modular input (Case ID: 2407260030005856). The concerns were discussed in a call with the Microsoft team. While they understood the problem, they required access to the affected tenant for further investigation. It was concluded that the issue was specific to the tenant and not related to the Add-on.

Resolution

A support ticket needs to be raised on the Microsoft side, as this issue is specific to the tenant and not related to the Add-on.