Search Head bundle replication fails

Applicability

Splunk Enterprise Search head clustering

Description

A search head cluster can experience replication issues when search bundles grow too large. This can happen due to large lookups and lookups that are frequently updated.

Investigation

Any changes made by the user to their dashboard don’t show up in their SHC.

• In SH captain, there is a warning message in splunkd.log that snapshot creation has failed frequently.  

ERROR ConfReplication - Error creating snapshot: 
/opt/splunk/var/run/splunk/snapshot/XXXXXX-XXXXXXXXXXXX.bundle
.XXXXXXXXX.tmp; Configurations changed while generating snapshot, original_latest
_change=XXXXXXXXXX, new_latest_change=XXXXXXXXX
XXXXXXXXX; consecutiveRejectionFromNewChanges=XX; Check conf.log to see if any app or 
client is making frequent configuration changes; Continuous snapshot creation failures can 
lead to configuration replication issues if this member becomes the captain.

• Check if there are any lookups being updated frequently.

index=_internal source=*conf.log* component=ConfOp data.task=acceptPush data.optype_desc=NOTIFY_UPDATE_LOOKUP
| spath input=_raw output=objname path=data.asset_uri{3} 
| timechart count by objname limit=15 useother=0 usenull=0

If CSV files are updated frequently, they can cause errors with snapshots and increase disk usage on search heads and indexers considerably. 

Resolution

There are multiple options for resolving this issue:

Disable the saved search updating the lookup and add to the denyList in distsearch.conf.

1. Disable any frequently running saved search updating lookup. Savedsearches.conf
 

The setting for disabling a search in savedsearches.conf file. Please apply these settings using the search head cluster deployer whenever possible:

[<stanza name>]
* Create a unique stanza name for each saved search.
* Follow the stanza name with any number of the following settings.
* If you do not specify a setting, Splunk software uses the default.disabled = <boolean>
* Disable your search by setting 'disabled=true'.
* You cannot run a disabled search.
* This setting is typically used to prevent a scheduled search from running
  on its schedule, without deleting the stanza for the search in the
  savedsearches.conf file.
* Default: false 

2. If the lookup file remains on the search head after disabling the saved search, and the lookup is not required by any searches running on the indexer tier, add it to the replication deny list in distsearch.conf to prevent it from being included in bundle replication.

REPLICATION_DENY_LIST_OPTIONS

[replicationDenylist]
<name> = <string>
* All comments from the replication allow list notes above also apply here.
* Replication deny list takes precedence over the allow list, meaning that a
  file that matches both the allow list and the deny list is NOT replicated.
* Use this setting to prevent unwanted bundle replication in two common
  scenarios:
    * Very large files which part of an application might not want to be
      replicated, especially if they are not needed on search nodes.
    * Frequently updated files (for example, some lookups) will trigger
      retransmission of all search head data.
* These lists are applied globally across all configuration data. Especially
  for deny listing, be sure to constrain your deny list to match only data
  that your application does not need.

Convert the frequently updating lookups to kvstore. 

Directions on this procedure can be found here migrate an app from using csv and are summarized below:

1. Create a kvstore collection

Create a collections.conf file on the search head. It is probably best to use a cluster deployer to complete this task in clustered environments. 

collections.conf

[myMigratedCSVFile]

2. Update the lookup definition

Find the transforms.conf file that is associated with the lookup that is frequently updated. change the external_type to kvstore and change the collection setting to the name of the kvstore collection

[CSVFile]
external_type = kvstore
collection = myMigratedCSVFile
field_list = all, the, fields, in, the, csv

WARNING: Do not modify the name of the lookup in this configuration file. Changing the stanza name will break existing searches. 

3. Deploy the new configuration and make sure the settings reloaded. If the settings aren't updated, try browsing to https://<yourSHCMember>/debug/refresh and click the refresh button or perform a rolling restart of the search head cluster. 

Related Links

REPLICATION_DENY_LIST_OPTIONS
Savedsearches.conf
migrate an app from using csv 
 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release