Cascading Knowledge Bundle Replication Issues in Splunk Search Heads

Applicability

Description

In Splunk, Search Heads replicate knowledge bundles (configurations, searches, etc.) to Search Peers (indexers) to execute distributed searches. The knowledge bundle replication uses cascading replication, where bundles are sent incrementally to optimize performance. However, problems can arise if a Search Peer becomes unreachable or is restarted during the replication process. This can cause the knowledge bundle to remain in an enqueued state, leading to cascading replication failing to progress.

• Some Search Heads successfully replicate bundles, while others fail intermittently.
• Switching to "classic" replication mode allows the affected Search Head to resume bundle replication.

Investigation

Verify Search Peer (Indexer) Status:

Use the below command to check the status of Search Peers and ensure all indexers are online.
Investigate the status of each indexer to confirm whether any indexers were restarted or became unavailable during the bundle replication process.

splunk show cluster-status

Search Head Logs:

Review Search Head logs ($SPLUNK_HOME/var/log/splunk/splunkd.log) for errors related to bundle replication. Look for messages such as:

ERROR DistributedPeerManager - Failed to replicate knowledge bundle to peer=<indexer_name> due to timeout. Retrying...
WARN DistributedPeerManager - Bundle replication plan stuck in queue, unable to complete due to unreachable peer <indexer_name>.

These messages indicate that the replication process was interrupted because the Search Peer became unavailable. Use the Splunk btool command to inspect configuration related to replication settings:

./splunk btool distsearch list replicationSettings

[replicationSettings]
cascade_plan_replication_retry_fast = true
cascade_plan_replication_threshold_failures = 0

Make sure the value cascade_plan_replication_retry_fast is set to true for improved fault tolerance.

Resolution

Make sure that all Search Peers are online and reachable. Use splunk show cluster-status to verify the status of all indexers.
If the issue persists, switch the Search Head captain to a different node to regenerate a new knowledge bundle replication plan. Use the following command from the $SPLUNK_HOME/bin directory:

./splunk transfer shcluster-captain -mgmt_uri <new_captain_mgmt_uri>

This command will assign a new captain, restart the replication plan, and resolve any stuck or failed replication loops.

Example:

./splunk transfer shcluster-captain -mgmt_uri https://new-sh-mgmt-uri:8089

Workaround:

If a Search Peer becomes unreachable, try switching captaincy between Search Heads to generate a new replication plan and restart cascading replication and ensure all Search Peers are online to prevent replication issues. To avoid known issues related to bundle replication, consider upgrading to Splunk version 9.0.10+, 9.1.5+, or 9.2.2+, which contain improvements for handling peer availability changes during the replication process and set cascade_plan_replication_retry_fast = true in distsearch.conf to improve fault tolerance and confirm this setting by running:

./splunk btool distsearch list replicationSettings

The above setting in config setting was used to provide better resilience when Search Peers become temporarily unavailable. This setting was made default in Splunk 9.0.2+.

Related Links

Cascading Knowledge Bundle replication

Distsearch.conf