Distributed Bundle Replication Manager: The current bundle directory contains a large lookup file that might cause bundle replication fail

Applicability

Description

Distributed Bundle Replication Manager: The current bundle directory contains a large lookup file that might cause bundle replication fail. 
The path to the directory is /opt/splunk/var/run/<cloudstack>.splunkcloud.com-1573504421-1573504499.delta.

Investigation

Find files bigger than 500MB using the following command:  

find /opt/splunk/etc -type f -size +500M -exec ls -lh {} \;

If files exist over 500 MB, it is recommended to add them to the replicationDenyList if they are not required on the indexer to satisfy the search. Take the lookup name (excluding the full path and .csv) and use it in this btool command to look for instances of the lookup existing in props.conf:

splunk btool props list --debug | grep -i <lookup>

If the lookup is called in props.conf, it may be needed to satisfy searches. Look into alternatives like Migrating lookups to KVStore

Resolution

How to denylist

The denylist shoud be applied on the search head in distsearch.conf

The [replicationDenylist] stanza in the system default version of distsearch.conf will deny  all the .csv files that are specified 

This is most useful for limiting the size of the knowledge bundle, particularly in the case of very large files that do not need to be replicated to the search peers. The deny list takes precedence over any allow list.

Below is an example: 

[replicationDenylist]
my_lookup = apps/search/lookups/my_lookup.csv

How to change the warning:

Increasing the default number for the warning (Not Recommended)

This option is only to get rid of the warning message. If the file continues growing, replication to the indexers can begin to fail. The end result is Splunk using outdated bundles potentially leading to application not found (exit code=111) errors and many more. 

How to do it
Before changing something, please be sure to make a backup of Splunk files with the best practices here: Backup configurations

1. Gain access to the CLI
2. Move to /opt/splunk/etc/system/local/
3. Edit the file distsearch.conf
4. Add the following

[replicationSettings]
concerningReplicatedFileSize = 2000 
maxBundleSize = 4096

5. Save and restart to apply the changes. 

Related Links

Splunk Documentation
Modify the knowledge bundle

Limit the knowledge bundlesize