Universal Forwarder Splunk Service Restarting Multiple Times a Day

Applicability

Splunk Enterprise

Description

When using multiple Deployment Servers behind a load balancer it is possible that the checksum calculated for each App does not match across all Deployment Servers. This is because when the checksum is calculated it includes the file's timestamp and permissions, which can differ between the Deployment Servers.

In the Universal Forwarder $SPLUNK_HOME/var/log/splunk/splunkd.log you will see the Checksum mismatch INFO event flapping between multiple checksums:

Checksum mismatch 2411343616691216195 <> 14186209848050062373 for app=splunk_nix_inputs. Will reload from='splunkds.example.com:8089/services/streams/deployment?name=default:EMEA_NIX:emea_nix_inputs'
Checksum mismatch 14186209848050062373 <> 2411343616691216195 for app=splunk_nix_inputs. Will reload from='splunkds.example.com:8089/services/streams/deployment?name=default:EMEA_NIX:emea_nix_inputs'
Checksum mismatch 2411343616691216195 <> 14186209848050062373 for app=splunk_nix_inputs. Will reload from='splunkds.example.com:8089/services/streams/deployment?name=default:EMEA_NIX:emea_nix_inputs'

Investigation

Checksum Mismatch flapping on the Universal Forwarder
From the Splunk Monitoring Console Server run the following search changing <splunk_uf> to the hostname of the Universal Forwarder experiencing the issue

index=_* host=<splunk_uf> sourcetype=splunkd component=DeployedApplication "Checksum mismatch" | stats count by event_message

there will be multiple counts of the Checksum mismatch event however the checksums will flap between at least two different values for the same App, the number of checksum values will depend on the number of Deployment Servers behind the load balancer.

In the example search results below the checksum is flapping between 14186209848050062373 and 2411343616691216195 for the same App emea_nix_inputs

event_message	count
Checksum mismatch 14186209848050062373 <> 2411343616691216195 for app=emea_nix_inputs. Will reload from='splunkds.example.com:8089/services/streams/deployment?name=default:EMEA_NIX:emea_nix_inputs'	24
Checksum mismatch 2411343616691216195 <> 14186209848050062373 for app=emea_nix_inputs. Will reload from='splunkds.example.com:8089/services/streams/deployment?name=default:EMEA_NIX:emea_nix_inputs'	26

Confirm the Checksum for each App on the Deployment Servers
From the Splunk Monitoring Console Server run the following search to confirm the Checksum of each App across all Deployment Servers:

| rest splunk_server_group=dmc_group_deployment_server /services/deployment/server/applications | table splunk_server, title, archive, size, updated, checksum

Apps with the same title should have a checksum that matches across all Deployment Servers. If you find one server has a checksum for an app that does not match the others this would be your suspect server.

Resolution

Validate Cross Server Checksum is set to true on all Deployment Servers
When using multiple Deployment Servers crossServerChecksum in $SPLUNK_HOME/etc/system/local/serverclass.conf should be set to true to prevent the timestamp and files permissions from being included when creating the checksum.

From the Splunk Monitoring Console Server run the following search to confirm crossServerChecksum = true on all Deployment Servers.

| rest splunk_server_group=dmc_group_deployment_server /services/configs/conf-serverclass | table splunk_server, title, crossServerChecksum

To configure crossServerChecksum = true edit the $SPLUNK_HOME/etc/system/local/serverclass.conf file and add the following to the [global] stanza.

[global]
crossServerChecksum = true

Once the change has been made restart Splunk to apply the change.

Is the Deployment Apps folder in sync on all Deployment Servers
The syncing of the $SPLUNK_HOME/etc/deployment-apps folder across multiple Deployment Servers is not part of the Splunk services and is out of scope for Splunk support. Confirm the method used to sync the folders between the servers is healthy and all servers are in sync with the System Administrator.

Check for hidden files or directories within the Deployment Apps folder on all Deployment Servers
Check within  $SPLUNK_HOME/etc/deployment-apps folder to confirm there are no hidden files or directories as this will also cause the checksum to be different.
In linux you can list all hidden files and directories with the following command:

cd $SPLUNK_HOME/etc/deployment-apps
find . -name '.*'

Related Links

serverclass.conf.spec
 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release