Analyst Queue REST handler fails with persistent "bad character (49) in reply size" errors

Applicability

Description

Accessing the Analyst Queue UI, one can observe the queues fail to load and every REST handler such as v1/metadata, v2/queues, /customfields, search/aq etc will fail with exception

bad character (49) in reply size

Corresponding splunkd logs will raise an exception

ERROR HttpListener [TcpChannelThread] - Exception while processing request for /en-US/splunkd/__raw/<REST_HANDLER_PATH>: bad character (49) in reply size

Investigation

When the Analyst Queue is loaded, the The subsequent python REST handlers behind import modules during startup. Check for module errors from the python.log if the import is failing. 

An example error is illustrated below if the 'cim_actions' module is missing.

ModuleNotFoundError: No module named 'cim_actions'
[Errno 2] No such file or directory:
    /opt/SplunkSH/splunk/etc/apps/Splunk_SA_CIM/lib/cim_models.py

Check if the relevant app is present under the etc/apps . As per the above example, Splunk_SA_CIM is missing from the apps folder.

Resolution

Make sure to install all the apps and add-ons that are shipped with ES on the search head.

If it is a search head cluster managed by a Deployer, ensure the app bundles are always pushed through the deployment server.

To ensure that the Splunk Enterprise Security app works correctly, install and turn on the following add-ons. 

• DA-ESS-AccessProtection
• DA-ESS-EndpointProtection
• DA-ESS-IdentityManagement
• DA-ESS-NetworkProtection
• DA-ESS-ThreatIntelligence
• SA-AccessProtection
• SA-AuditAndDataProtection
• SA-EndpointProtection
• SA-IdentityManagement
• SA-NetworkProtection
• SA-ThreatIntelligence
• Splunk_SA_CIM
• Splunk_SA_Scientific_Python_linux_x86_64
• SplunkEnterpriseSecuritySuite
• Splunk_ML_Toolkit

Related Links

https://help.splunk.com/en/splunk-enterprise-security-8/release-notes-and-resources/8.6/splunk-enterprise-security-release-notes/release-notes-for-splunk-enterprise-security#ariaid-title8