Excessive Internal Logging Errors Causing Drive Space Issues.

Applicability

Description

Encountering excessive internal logging errors that are filling up the drives. The specific error message is:

081 ERROR [assist::supervisor_modular_input.py] [runner] [log_stream] [1938104] package=assist-certificate message="Failed to decode certificate PEM file"

Investigation

The internal logs are recorded in splunk_assist_supervisor_modular_input.log. The error appears as follows:

-0400 ERROR ExecProcessor [491591 ExecProcessor] - message from "/opt/splunk/bin/python3.9 /opt/splunk/etc/apps/splunk_assist/bin/supervisor_modular_input.py" logging.FileHandler.emit(self, record)

Issue arose after enabling Splunk Assist. Once Splunk Assist was disabled, the problem was resolved.

Note:
Splunk Assist will reach End of Life (EOL) on January 30, 2025. Since Splunk Assist is not used, disabling it did not cause any issues.

Resolution

The problem is caused by the supervisor_modular_input.py script, which is part of Splunk Assist. Disabling Splunk Assist resolved the issues with this script.

Follow one of the next options to disable Splunk Assist:

1- Open the $SPLUNK_HOME/etc/apps/splunk_assist/local/app.conf file in a text editor
2- Edit the file to include [install] allows_disable = true
3- Save the file and close the editor
4- Restart Splunk Enterprise

Or  use a single command followed by Splunk restart

$SPLUNK_HOME/bin ./splunk disable app splunk_assist -auth <username>:<password>

Related Links

About Splunk Assist