Splunk Add-on for Microsoft O365 : Message Trace Inputs Stopping Event Ingestion from Source with Errors

Applicability

Description

All of a sudden, the configured Message Trace inputs stopped ingesting events from the source, resulting in the following errors from the respective inputs:

Error Log:

AttributeError: 'Response' object has no attribute 'encode'

Attempts to recreate the input and select a start date within the last 5 days have not resolved the issue, and no data is being ingested.

Investigation

Upon further investigation of the internal logs, it was found error logs related to the stopped input, but they did not provide clear insight into the issue. However, when cross-checking with the internal logs of other inputs as mentioned below, it was discovered that the client's secret key had expired for the configured tenant.

SPL : 

index=_internal sourcetype="splunk:ta:o365:log" datainput=*messagetrace* level=ERROR

Error log :

1. 

2024-06-24 13:25:41,622 level=ERROR pid=3744733 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:run:586 | datainput=b'o365_exchange_messagetrace' start_time=1719235540 | message="An error occurred while collecting data" stack_info=True
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 581, in run
    self._collect_events(app)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 148, in _collect_events
    self._get_events_continuous(app)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 240, in _get_events_continuous
    self._process_messages(start_date, end_date, input_mode="cont_mon")
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 432, in _process_messages
    message_response = self._get_messages(microsoft_trace_url)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 321, in _get_messages
    raise e
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 307, in _get_messages
    session = self.get_session()
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 284, in get_session
    session = self._token.auth(self._session)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/token.py", line 202, in auth
    self._token = self._policy(self._resource, session)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/token.py", line 181, in __call__
    self._client_id, self._client_secret, resource, session
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 255, in get_messagetrace_token_by_psk
    raise O365PortalError(response.encode("utf-8"))
AttributeError: 'Response' object has no attribute 'encode'

2.

2024-06-24 13:25:41,621 level=ERROR pid=3744733 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:_get_messages:320 | datainput=b'o365_exchange_messagetrace' start_time=1719235540 | message="HTTP Request error: 'Response' object has no attribute 'encode'" stack_info=True
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 307, in _get_messages
    session = self.get_session()
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 284, in get_session
    session = self._token.auth(self._session)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/token.py", line 202, in auth
    self._token = self._policy(self._resource, session)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/token.py", line 181, in __call__
    self._client_id, self._client_secret, resource, session
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 255, in get_messagetrace_token_by_psk
    raise O365PortalError(response.encode("utf-8"))
AttributeError: 'Response' object has no attribute 'encode'

Checking with the other data input with the below SPL : 

index=_internal sourcetype="splunk:ta:o365:log" level=ERROR  datainput="b'o365_AZServiceHealth'"

Error log : 

2024-06-24 18:11:02,781 level=ERROR pid=283114 tid=MainThread logger=splunk_ta_o365.modinputs.graph_api pos=utils.py:wrapper:72 | datainput=b'o365_AZServiceHealth' start_time=1719252661 | message="Data input was interrupted by an unhandled exception." 
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/splunksdc/utils.py", line 70, in wrapper
    return func(*args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/graph_api/__init__.py", line 123, in run
    consumer.run(content_type)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/graph_api/GraphApiConsumer.py", line 444, in run
    self._session = self._token.auth(self._session)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/token.py", line 202, in auth
    self._token = self._policy(self._resource, session)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/token.py", line 139, in __call__
    self._client_id, self._client_secret, resource, session
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/portal.py", line 208, in get_v2_token_by_psk
    raise O365PortalError(response)
splunk_ta_o365.common.portal.O365PortalError: 401:{"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app 'd046d62f-46c9-4ac3-b9e8-fbb5af17a030'. Trace ID: ff782611-6340-42fc-8940-2e20ecc3b400 Correlation ID: 3170169d-4c51-48d9-bd33-8d41d1b6b4d3 Timestamp: 2024-06-24 18:11:02Z","error_codes":[7000215],"timestamp":"2024-06-24 18:11:02Z","trace_id":"ff782611-6340-42fc-8940-2e20ecc3b400","correlation_id":"3170169d-4c51-48d9-bd33-8d41d1b6b4d3","error_uri":"https://login.microsoftonline.com/error?code=7000215"}

Resolution

The issue was resolved when the client secret key was updated for the affected tenant from the Microsoft side.

Related Links

Refer to the Splunk Documentation for the mentioned error: here