UI for Splunk Add-on for Microsoft Cloud Services Fails to Load After Upgrading Splunk from Version 9.2 to 9.3

Applicability

Description

After upgrading Splunk from 9.2.2 to 9.3.0, a message appeared in the "Messages" drop-down saying:

Unable to initialize modular input "mscs_storage_table" defined in the app "Splunk_TA_microsoft-cloudservices": Introspecting scheme=mscs_storage_table: script running failed (exited with code 1)..

Below error is getting on the UI on the input and configuration page of Splunk Add-on for Microsoft Cloud Services.

"Unable to xml-parse the following data: %s"

Below is the reference image.

Investigation

While investigating the splunkd.log, the following error was observed:

08-20-2024 15:23:38.384 -0500 WARN PersistentScript [44264 PersistentScriptIo] - Process {"C:\Program Files\Splunk\bin\Python3.9.exe" "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_storageaccount.py" persistent}: exited with code 1
08-20-2024 15:23:38.887 -0500 ERROR AdminManagerExternal [88948 TcpChannelThread] - Received malformed XML from external handler:
08-20-2024 15:23:38.887 -0500 ERROR AdminManagerExternal [88948 TcpChannelThread] - Unable to xml-parse the following data: %s
08-20-2024 15:23:39.867 -0500 INFO TailReader [29388 tailreader0] - Batch input finished reading file='C:\Program Files\Splunk\var\spool\splunk\tracker.log'
08-20-2024 15:23:39.960 -0500 ERROR PersistentScript [44264 PersistentScriptIo] - From {"C:\Program Files\Splunk\bin\Python3.9.exe" "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_storageaccount.py" persistent}: Traceback (most recent call last):
08-20-2024 15:23:39.960 -0500 ERROR PersistentScript [44264 PersistentScriptIo] - From {"C:\Program Files\Splunk\bin\Python3.9.exe" "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_storageaccount.py" persistent}: File "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_storageaccount.py", line 19, in <module>
08-20-2024 15:23:39.960 -0500 ERROR PersistentScript [44264 PersistentScriptIo] - From {"C:\Program Files\Splunk\bin\Python3.9.exe" "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_storageaccount.py" persistent}: from mscs_storage_service import ( # isort:skip # pylint: disable=import-error
08-20-2024 15:23:39.960 -0500 ERROR PersistentScript [44264 PersistentScriptIo] - From {"C:\Program Files\Splunk\bin\Python3.9.exe" "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_storageaccount.py" persistent}: File "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\mscs_storage_service.py", line 20, in <module>
08-20-2024 15:23:39.960 -0500 ERROR PersistentScript [44264 PersistentScriptIo] - From {"C:\Program Files\Splunk\bin\Python3.9.exe" "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_storageaccount.py" persistent}: from azure.storage.blob import ( # isort: skip # pylint: disable=import-error
08-20-2024 15:23:39.960 -0500 ERROR PersistentScript [44264 PersistentScriptIo] - From {"C:\Program Files\Splunk\bin\Python3.9.exe" "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_storageaccount.py" persistent}: File "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\azure\storage\blob\__init__.py", line 10, in <module>
08-20-2024 15:23:39.960 -0500 ERROR PersistentScript [44264 PersistentScriptIo] - From {"C:\Program Files\Splunk\bin\Python3.9.exe" "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_storageaccount.py" persistent}: from ._blob_client import BlobClient
08-20-2024 15:23:39.960 -0500 ERROR PersistentScript [44264 PersistentScriptIo] - From {"C:\Program Files\Splunk\bin\Python3.9.exe" "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_storageaccount.py" persistent}: File "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\lib\azure\storage\blob\_blob_client.py", line 28, in <module>
08-20-2024 15:23:39.960 -0500 ERROR PersistentScript [44264 PersistentScriptIo] - From {"C:\Program Files\Splunk\bin\Python3.9.exe" "C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-cloudservices\bin\splunk_ta_mscs_rh_storageaccount.py" persistent}: from ._shared.encryption import generate_blob_encryption_data

For troubleshooting and validation, the following workarounds were attempted but did not resolve the issue:

• Restarted the Splunk service
• Reinstalled the add-on
• Removed Python cache files

The issue was identified as follows: Splunk Enterprise 9.2 ships with Python 3.7, while Splunk Enterprise 9.3 ships with Python 3.9. The Splunk Add-on for Microsoft Cloud Services versions 5.3.0 or older are compatible only with Python 3.7 or earlier. This incompatibility causes issues and results in the documented error.

Resolution

According to the development team, the issue will be resolved in the upcoming release, version 5.4.0.

The available workarounds are to either downgrade Splunk to version 9.2 or wait for the new release of the Splunk Add-on for Microsoft Cloud Services.