Message Trace Logs Stopped Ingesting After Splunk Add-on for Microsoft O365 Upgrade to v6.x

Applicability

Description

Message Trace log ingestion for GovCloud and FedRAMP stacks will stop after upgrading the Splunk Add-on for Microsoft Office 365 from 5.1.0 to 6.x.  Microsoft ended their legacy Message Trace support on Reporting Webservice on March 18, 2026.  Afterwards, Message Trace onto Graph API will be implemented. 

Tenants configured with the Worldwide (commercial) endpoint will not be affected and those stacks should be able to follow the Update Message Trace Inputs instructions in the add-on documentation if this issue is seen.

But GovCloud or FedRAMP stacks using message trace inputs will break and the logs will stop ingesting. 

Investigation

Run this search on the stack IDM:

index=_internal sourcetype="splunk:ta:o365:log" host="idm*" "*error*" datainput=message_trace

The following error might be available:

level=ERROR pid=180403 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=provisioning.py:_provision_service_principal:155 | datainput="message_trace" start_time=1775174379 | message="Provisioning Message Trace service principal failed." stack_info=True
Traceback (most recent call last):
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/provisioning.py", line 152, in _provision_service_principal
response.raise_for_status()
File "/opt/splunk/etc/apps/splunk_ta_o365/lib/requests/models.py", line 1026, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://graph.microsoft.com/v1.0/servicePrincipals...url, response=response, _pool=self) File "/opt/splunk/lib/python3.9/site-packages/urllib3/util/retry.py", line 594, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='graph.microsoft.com', port=443): Max retries exceeded with url:/v1.0/admin/exchange/tracing/messageTraces?$filter=receivedDateTime%20ge%202026-01-28T19:37:43.000Z%20and%20receivedDateTime%20le%202026-01-29T19:47:42.999Z (Caused by ResponseError('too many 500 error responses'))

The root cause of this failure is the migration from the legacy Exchange Online Reporting Web Service (RWS) API to the Microsoft Graph API in version 6.x.x. Because the Microsoft Graph Message Trace endpoint is currently only available for commercial (worldwide) cloud environments and lacks support for government clouds, the add-on fails to ingest Message Trace data for these specific tenants, resulting in logged errors and a total loss of data collection for this source.

Microsoft has not provided a timeline for when GCC, GCC-High, DoD, and other sovereign clouds will be compatible. 

Splunk is working on a resolution for future versions of the add-on.

Resolution

Related Links

Announcing General Availability (GA) of the New Message Trace in Exchange Online

Follow the official migration guide to transition from the deprecated Message Trace V1 input to the supported Message Trace V2 input:
- Update Message Trace Inputs

Configure the new Message Trace V2 input parameters as described in the updated add-on documentation:
- Configure Message Trace Input