Splunk forwarder with Dynatrace agent not starting after upgrade

Applicability

Splunk Enterprise Splunk Universal Forwarder

Description

Splunk Forwarders 8.1.2 with Dynatrace agent is installed. When the user upgraded to Splunk 8.2.0, the Splunk service didn't start. 
 

Investigation

By looking at the logs when trying to start Splunk, the entries below confirm that Dynatrace agent is the problem.

Jul 13 13:31:09 lh989.server.raiffeisen.ch rch-splunkfw: Starting Splunk Forwarding ...
Jul 13 13:31:09 lh989.server.raiffeisen.ch su: (to splunkfw) root on none
Jul 13 13:31:09 lh989.server.raiffeisen.ch rch-splunkfw: Splunk> Be an IT superhero. Go home early.
Jul 13 13:31:09 lh989.server.raiffeisen.ch rch-splunkfw: Checking prerequisites ...
Jul 13 13:31:09 lh989.server.raiffeisen.ch rch-splunkfw: Management port has been set disabled; cli support for this configuration is currently incomplete.
Jul 13 13:31:10 lh989.server.raiffeisen.ch rch-splunkfw: Checking conf files for problems ...
Jul 13 13:31:10 lh989.server.raiffeisen.ch rch-splunkfw: Done
Jul 13 13:31:10 lh989.server.raiffeisen.ch rch-splunkfw: Checking default conf files for edits ...
Jul 13 13:31:10 lh989.server.raiffeisen.ch rch-splunkfw: Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-8.2.0-e053ef3c985f-linux-2.6-x86_64-manifest'
Jul 13 13:31:10 lh989.server.raiffeisen.ch kernel: traps: splunkd [5914] trap invalid opcode ip: 7ff758d7b3ab sp: 7ffe2e18f640 error: 0 in liboneagentproc.so [7ff758d66000 + 86000]
Jul 13 13:31:10 lh989.server.raiffeisen.ch rch-splunkfw: ERROR: pid 5914 terminated with signal 4 (core dumped)
Jul 13 13:31:10 lh989.server.raiffeisen.ch rch-splunkfw: [OK]
Jul 13 13:31:10 lh989.server.raiffeisen.ch rch-splunkfw: All installed files intact.
Jul 13 13:31:10 lh989.server.raiffeisen.ch rch-splunkfw: Done
Jul 13 13:31:10 lh989.server.raiffeisen.ch rch-splunkfw: All preliminary checks passed.
Jul 13 13:31:10 lh989.server.raiffeisen.ch rch-splunkfw: Starting splunk server daemon (splunkd) ...
Jul 13 13:31:10 lh989.server.raiffeisen.ch rch-splunkfw: Done
Jul 13 13:31:10 lh989.server.raiffeisen.ch systemd: Started SYSV: Splunk indexer service.

Resolution

There is an issue between Dynatrace agent and Splunk (8.2.0, 8.2.1 & 8.2.2) both using the same hooks on a lib file.

This error was fixed in version 8.2.2109, To correct this upgrade Splunk.

If an upgrade cannot be performed, a workaround to fix this is to add the following attribute in server.conf:

[watchdog]
usePreloadedPstacks = false

 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release