Health Check: A script exited abnormally with exit status: 1

Applicability

Description

Onprem Environment:

• Indexer clustering with multi-site(3): 3 indexers in each site
• ES SHC - 3 members
• Splunk 8.1.6
• ES 6.6.2

Description:
After upgrading ES from version 6.4.1 to 6.6.2 the below warning messages popped up on the health check alongside with errors in threatmatch.log

Health Check: msg="A script exited abnormally with exit status: 1" input="input/location/path.py" stanza="stanza://name"

threatmatch.log:

ERROR pid=7458 tid=MainThread file=threatmatch.py:handle_threatgen_search:341 | status="Error occurred while processing search spec, exiting" stanza="threatmatch://user" error="[HTTP 500] Splunkd internal error; []"
ERROR pid=2726 tid=MainThread file=threatmatch.py:handle_threatgen_search:341 | status="Error occurred while processing search spec, exiting" stanza="threatmatch://url" error="[HTTP 500] Splunkd internal error; []"

Investigation

• Since the mentioned errors are coming only for few threatmatch stanza's. Disabled and enabled the respective threatmatch from UI.
• Tried rolling restart of the SHC.
• Validated the datamodel acceleration details from UI.

When digging further the below errors related to the Web datamodel were found:

ERROR pid=6695 tid=MainThread file=rest_handler.py:handle:766 | InvalidSearchPart: Error loading datamodel Web
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/SA-ThreatIntelligence/bin/custom_search_builder/base.py", line 152, in getDataModelJson
    return DataModels.getDatamodelJson(modelName, sessionKey)
  File "/opt/splunk/etc/apps/Splunk_SA_CIM/lib/cim_models.py", line 84, in getDatamodelJson
    return json.loads(json.loads(c)['entry'][0]['content']['eai:data'])
  File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/opt/splunk/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/splunk/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1001 column 5 (char 41876)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/SA-Utils/lib/SolnCommon/rest_handler.py", line 741, in handle
    response = handler(self, args, params)
  File "/opt/splunk/etc/apps/SA-ThreatIntelligence/bin/customsearchbuilder_rest_handler.py", line 105, in csb_build
    return_tuple = method(spec, session_key, self.logger, preview_mode=preview_mode)
  File "/opt/splunk/etc/apps/DA-ESS-ThreatIntelligence/bin/custom_search_builder/make_threat_generating_search.py", line 404, in makeThreatGeneratingSearch
    datamodel, session_key)
  File "/opt/splunk/etc/apps/SA-ThreatIntelligence/bin/custom_search_builder/base.py", line 156, in getDataModelJson
    raise InvalidSearchPart(e)
csb_exceptions.InvalidSearchPart: Error loading datamodel Web

For this scenario the Web.json file was different compared to the one from the CIM addon

Resolution

Once the Web,json file was identified the following process was done:

a. Took the backup of the current file - Web.json in the location: SPLUNK_HOME/etc/apps/Splunk_SA_CIM/default/data/model
b. Downloaded the same version - 4.20.2 of CIM app from splunkbase and untar the package file.
c. Copied the Web.json file from the downloaded package to all the SH's in the location: SPLUNK_HOME/etc/apps/Splunk_SA_CIM/default/data/model.
d. Did a restart of SHC and validated the issue.

After replacing the Web.json from the newly downloaded CIM addon and restarting the Splunk service on SHC, the issue is resolved i.e the errors logs related to the datamodel as well as threatmatch errors/warnings disappeared.