Health Check warning for ws_client.py exiting abnormally with exit status 1

Applicability

Description

Administrators may observe a persistent warning message on the Splunk Enterprise Security Search Head Health Check page. The warning indicates that a script has failed. The exact message displayed is:

Health Check: msg="A script exited abnormally with exit status: 1" input=".$SPLUNK_HOME/etc/apps/dlx-app/bin/tasks/ws_client.py" stanza="default"

This occurs because during a Splunk restart or initial app provisioning, the ws_client.py script attempts to connect to its cloud service before the service is fully ready, resulting in an initial exit status of 1. The script subsequently restarts and runs continuously in the background. Because the script does not exit again, it never logs a successful exit status of 0. The Health Check relies on a saved search that only evaluates the last recorded exit code, causing the warning to persist in the UI even though the script is functioning correctly.

Investigation

To confirm if the environment is experiencing this specific known issue, verify that there are no active errors being generated by the script.

Run the following Splunk Processing Language (SPL) search over the last 7 days:

index=_internal (source="*splunkd.log" OR source="*python.log") "ws_client.py" ERROR OR WARN

If the search returns no recent results, the script is currently running successfully, and the Health Check warning is a false positive based on the historical startup event.

Resolution

To resolve the persistent warning, update the Enterprise Security search macro to ignore this expected startup behavior for the ws_client.py script.

1. Navigate to Settings > Advanced Search > Search macros.
2. Change the App context to SplunkEnterpriseSecuritySuite or All.
3. Search for the macro named script_error_msg_ignore.
4. Click on the macro to edit its definition.
5. Append the following logic to the end of the existing string in the Definition box:
   

   OR (script LIKE "%ws_client.py" AND exit_status=1)

   For example, if the current definition contains exclusions for other scripts, the updated definition will look similar to this:
   

   (match(script, "(streamfwd|splunk-(wmi\.path|MonitorNoHandle\.exe|winevtlog\.exe|netmon\.exe|perfmon\.exe|regmon\.exe|winprintmon\.exe|admon\.exe|powershell\.exe))") AND exit_status=1) OR (script LIKE "%instrumentation.py" AND exit_status=114) OR (script LIKE "%save_image_and_icon_on_install.py" AND exit_status=1) OR (script LIKE "%ws_client.py" AND exit_status=1)

    

6. Click Save.
7. Navigate back to the Health Check UI and clear the current warning messages. The warning will not reappear on subsequent Health Check executions.

Related Links

N/A