In Universal forwarder 9.3.x: Errors such as "No such file or directory" and "segmentation fault" occur when splunk command is executed

Applicability

Description

In universal forwarder 9.3.x, the errors shown below appear when running variations of the splunk command.

A.

/opt/splunkforwarder/bin/splunk status
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"
Couldn't change ownership for /opt/splunkforwarder/lib: No such file or directory

B.

/opt/splunkforwarder/bin/splunk list
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunk:splunk /opt/splunkforwarder"
Couldn't change ownership for /opt/splunkforwarder/lib: No such file or directory
Segmentation fault

Error B occurs when running commands without an argument required for a valid use such as "splunk list" or "splunk show". 

Investigation

Error A occurs due to the unused, uncleaned symlinks not in use under /$SPLUNK_HOME/lib, which were used in previous versions but are no longer needed in 9.3.x.
Error B occurs due to a regression introduced in this version of universal forwarder. 

Resolution

You can prevent these messages by manually delete the unused symlinks following the steps below

#FIND THE UNUSED SYMLINKS
/opt/splunkforwarder# find /opt/splunkforwarder/lib/ -xtype l
/opt/splunkforwarder/lib/libdlstub.so
/opt/splunkforwarder/lib/libdlwrapper.so
/opt/splunkforwarder/lib/libdlstub.so.1
/opt/splunkforwarder/lib/libdlwrapper.so.1#DELETE THEM
/opt/splunkforwarder# rm -f <symlinks you found in the previous step>

This issue will be fixed in the next minor release and backported. As of 2024/Feb, it is expected to be available in 9.3.4.
The information above should be updated once more details become available.