Message trace Input stopped ingesting new logs

Applicability

Description

The Message Trace input configured in the Splunk Add-on for Microsoft Office 365 stopped collecting logs unexpectedly. The required API permissions were confirmed to be configured correctly in Azure.

Investigation

Reviewed the Splunk internal logs for errors related to the Office 365 add-on by searching the "_internal" index for the "splunk:ta:o365:log"sourcetype.

Example SPL:

index=_internal sourcetype="splunk:ta:o365:log"

The logs showed repeated "500 Server Error" messages each time the Message Trace input attempted to run.

2023-05-12 02:45:01,718 level=ERROR pid=1835639 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:run:376 | datainput=b'dhhstasmania_Exchange_MessageTrace' start_time=1683859500 | message="An error occurred while collecting data" stack_info=True
Traceback (most recent call last):
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 371, in run
self._collect_events(app)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 145, in _collect_events
self._get_events_continuous(app)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 216, in _get_events_continuous
self._process_messages(start_date, end_date)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 283, in _process_messages
message_response = self._get_messages(microsoft_trace_url)
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 270, in _get_messages
raise e
File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 262, in _get_messages
response.raise_for_status()
File "/opt/splunk/etc/apps/splunk_ta_o365/lib/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: for url: https://reports.office365.com/ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2023-04-11T21:56:04Z'%20and%20EndDate%20eq%20datetime'2023-04-11T22:56:04Z'

The reporting web service API throws 500 server error when the specified startdate in the request is more than 10 days in the past.

This was the API behaviour during the feature development and that is why the restriction was added during the feature development and is reflected in the StartDate field of the addon, which prevents users from not getting the logs from more than 10 days in the past.

Resolution

To resolve the issue, create a new Message Trace input with the following settings:

• Use a unique input name.
• Configure a valid start date.
• Ensure the start date is not more than 10 days in the past.

After creating the new input, confirm that data collection resumes and monitor _internal logs for any additional errors.