Splunk Add-on for Microsoft O365: Message Trace Input Error Due to KVStore Initialization Failure

Applicability

Description

The Splunk Add-on for Microsoft Office 365 fails to process "Message Trace Inputs" because the KVStore service has not been initialized.

The error log reveals a service unavailability issue, causing the input to stop working.

Investigation

While troubleshooting the issue with the below SPL query:
SPL:

index=_internal *O365* *ERROR*

Below error logs were observed,

ERROR Logs:

2025-02-03 14:28:50,064 level=ERROR pid=696520 tid=MainThread logger=splunk_ta_o365.modinputs.message_trace pos=__init__.py:run:585 | datainput="ms_azure_message_trace" start_time=1738618112 | message="An error occurred while collecting data" stack_info=True
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 583, in run
    self._collect_events(app)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 147, in _collect_events
    self._get_events_continuous(app)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 224, in _get_events_continuous
    start_date = self.get_start_date(checkpoint_key)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/modinputs/message_trace/__init__.py", line 163, in get_start_date
    date = self._ckpt.get(checkpoint_key)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/checkpoint.py", line 69, in wrapper
    raise last_ex
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/checkpoint.py", line 52, in wrapper
    return func(*args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/bin/splunk_ta_o365/common/checkpoint.py", line 189, in get
    data = self._collection.query_by_id(id=checkpoint_key)
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/splunklib/client.py", line 3912, in query_by_id
    return json.loads(self._get(UrlEncoded(str(id), encode_slash=True)).body.read().decode('utf-8'))
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/splunklib/client.py", line 3877, in _get
    return self.service.get(self.path + url, owner=self.owner, app=self.app, sharing=self.sharing, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/splunklib/binding.py", line 321, in wrapper
    return request_fun(self, *args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/splunklib/binding.py", line 76, in new_f
    val = f(*args, **kwargs)
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/splunklib/binding.py", line 737, in get
    response = self.http.get(path, all_headers, **query)
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/splunklib/binding.py", line 1272, in get
    return self.request(url, {'method': "GET", 'headers': headers})
  File "/opt/splunk/etc/apps/splunk_ta_o365/lib/splunklib/binding.py", line 1344, in request
    raise HTTPError(response)
splunklib.binding.HTTPError: HTTP 503 Service Unavailable -- KV Store initialization failed. Please contact your system administrator.

• The error log indicates an "HTTP 503 Service Unavailable" issue.
• The cause is identified as the KVStore initialization failure, which is crucial for managing and tracking inputs in the Splunk Add-on for Microsoft Office 365.
• The error message specifically points to the KVStore service not being enabled or initialized properly.

Resolution

To resolve the issue, ensure that the KVStore service is enabled and running on the Splunk instance.

Kindly refer to the Splunk KVStore Documentation for detailed instructions on enabling and managing the KVStore service.

Related Links

Configure Message Trace inputs for the Splunk Add-on for Microsoft Office 365