Splunk IT Service Intelligence (ITSI) 4.5.0 - ERROR: ”Failed to load the glass table. An internal error has occurred."

Applicability

Description

After ITSI upgrade to 4.7.x, users may see the next pop-up error and the glass table would not open:

Failed to load the glass table. An internal error has occurred.

GUI Navigation:  Open Splunk® IT Service Intelligence > Glass Tables >  click to open any of Glass Tables, which used to be "Beta":
 

Only non-Classic (called Beta before 4.7.x) glass tables have this issue, whilst Classic glass table can be opened without any issue after the upgrade.
 

Investigation

HTTP traffic check using HAR dump on the GUI navigation showed the following ERROR:

Request URL: 
https://<host name>/en-US/splunkd/__raw/servicesNS/-/SA-ITOA/gt_service_check?id=<Glass Table ID>
Request Method: GET
Status Code: 500 Error starting: No module named supervisor

Using the following search

index=_internal sourcetype=splunkd component=PersistentScript OR component=HttpListener ERROR

11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}: 
ImportError: No module named supervisor11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
from migration.supervisor import MigrationSupervisor11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
File "/opt/splunk/etc/apps/SA-ITOA/lib/ITOA/rest_interface_provider_base.py", line 21, in <module>11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
from ITOA.rest_interface_provider_base import SplunkdRestInterfaceBase11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
File "/opt/splunk/etc/apps/SA-ITOA/bin/gt_service_check.py", line 19, in <module>11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
m = imp.load_module(munged_name, filehandle, filename, data)11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
File "/opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py", line 57, in _load_file11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
rv = PersistentServerConnectionApplicationServer._load_file(filename)11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
File "/opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py", line 42, in 
_cached_load_file11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
meths = self._cached_load_file(filename)11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
File "/opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py", line 31, in 
_get_class_and_methods11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
class_and_meths = self._get_class_and_methods(command[0], stream_allowed)11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
File "/opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py", line 22, in load11-24-2020 11:08:21.371 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
self._current_handler = self.load(in_packet.command, in_packet.command_arg, in_packet.allow_stream())11-24-2020 11:08:21.370 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
File "/opt/splunk/lib/python2.7/site-packages/splunk/persistconn/handle_loop.py", line 36, in 
handle_packet11-24-2020 11:08:21.370 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
self.handle_packet(in_packet)11-24-2020 11:08:21.370 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
File "/opt/splunk/lib/python2.7/site-packages/splunk/persistconn/packet.py", line 191, in run11-24-2020 11:08:21.370 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
h.run()11-24-2020 11:08:21.370 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
File "/opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py", line 114, in <module>11-24-2020 11:08:21.370 +0000 ERROR PersistentScript [54524 PersistentScriptIo] - From 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
Traceback (most recent call last):11-24-2020 11:08:21.370 +0000 WARN  PersistentScript [54524 PersistentScriptIo] - Process 
{/opt/splunk/bin/python2.7 /opt/splunk/lib/python2.7/site-packages/splunk/persistconn/appserver.py}:
exited with code 111-24-2020 11:08:21.204 +0000 ERROR HttpListener [48821 TcpChannelThread] - Exception while processing
request from 127.0.0.1:51388 for /en-US/splunkd/__raw/servicesNS/-/SA-ITOA/gt_service_check?id=
<Glass Table ID>: Error starting: No module named supervisor

Note that persistconn/appserver.py is using Python 2.7 whilst this Splunk Cloud version is Python 3 capable.

Check persistconn/appserver.py on this environment using Python 3.7 for any running persistconn/appserver.py process:

splunk 71406 26531 0 12:09 ? 00:00:00 /opt/splunk/bin/python3.7 
                                      /opt/splunk/lib/python3.7/site-packages/splunk/persistconn/appserver.py

 
Check  time stamps of the below Python code files and compiled code:
~/etc/apps/SA-ITOA/bin/gt_service_check.py
~/etc/apps/SA-ITOA/lib/ITOA/rest_interface_provider_base.py
~/etc/apps/SA-ITOA/lib/migration/supervisor.py

Note: Many nearly 2 years old .pyo (python optimized compiled code) files exist in the environment. The main difference within the lib folder is old .pyo files (do not exist in the new version) and __pycache__/xxxxx.cpython-37.pyc files (will be re-generated).
 

Resolution

Extract and/or swap /opt/splunk/etc/apps/SA-ITOA/lib/ITOA or /opt/splunk/etc/apps/SA-ITOA/lib from the ITSI installer of the same version of the problematic environment to resolve the issue.

Notes:
Back up SA-ITOA app (directory) before touching any files.
Confirm file permission and owner is correct on the extraction or swap
ITSI search head restart is not an absolute requirement after the above is done.
 

Related Links

External Documentation:
PEP 488 - Elimination of PYO Files

What do the python file extensions .pyc .pyd .pyo stand for