Search head cluster history replication

Applicability

Description

Starting with Splunk Enterprise 9.1, search history can be replicated across all Search Head Cluster members using kvstore-based replication, replacing the legacy file-based storage method. However, enabling this feature requires configuration of TWO distinct parameters that serve different purposes:

1. kvstore Storage Mode - Enables kvstore as the backend for storing search history data
2. Configuration Replication - Ensures search history settings are consistently replicated across all cluster members

Configuring  only the first parameter may cause search history not replicating or disappearing entirely. This occurs because the cluster members are not receiving consistent configuration instructions for how to handle the kvstore-based history.

Legacy file-based history replication is not supported via traditional configuration bundle replication.

Investigation

Below investigation steps can be followed to troubleshoot this issue:

1. The parameter search_history_storage_mode=kvstore was introduced in Splunk 9.1 to enable search history replication across Search Head Clusters.  (Limits.conf):

search_history_storage_mode = <string>
* The storage mode by which a search head cluster saves search history
* Valid storage modes include "csv" and "kvstore"
* This setting is valid only when the 'enable_history' setting has a value of "true"
* A value of "kvstore" means that the cluster can replicate search history across 
  all its members using the App Key Value Store service
* A value of "csv" means that search history is saved to CSV files only on the 
  local search head
* When you initially give this setting a value of "kvstore", the search head 
  migrates existing search history records (if present in CSV files) into the 
  App Key Value Store service
* This migration can only happen once. If you later change the storage mode to 
  "csv", then back to "kvstore", subsequent migrations do not occur
* In "kvstore" storage mode, 'max_history_length' is the maximum number of 
  searches the SHC can store for each user (maximum acceptable value: 1000)
* Default: csv

2. Before implementing kvstore-based replication, the following parameters must be configured in limits.conf:

Parameter A: enable_history

enable_history = <boolean>
* Whether Splunk records search history
* This is MANDATORY - must be set to "true"
* If this is not enabled, search history will not be recorded at all
* Default: true (in most configurations)

Parameter B: max_history_length

max_history_length = <integer>
* Maximum number of searches to store in history for each user and application
* When 'search_history_storage_mode' has a value of "kvstore", this value is 
  applicable per user only (not per user and application combination)
* Default: 500

Parameter C: max_history_storage_retention_time

max_history_storage_retention_time = <integer>[s|m|h|d]
* The maximum time to store search history records for each user and application
* This setting and 'max_history_length' together determine how many search 
  history records appear in persistent storage
* Search stops storing records when either the retention time or the number of 
  records exceeds the configured values
* A value of 0 means search only uses 'max_history_length' to retain history
* Time units: s, m, h, d (seconds, minutes, hours, days)
* This setting is only applicable when 'search_history_storage_mode' has a 
  value of "kvstore"
* Default: 90d

3. The Critical Missing Configuration: conf_replication_include 

Initial configuration often includes only the limits.conf changes shown above. However, this is insufficient for SHC replication. An additional parameter in Server.conf is required:

Parameter D: conf_replication_include.history (server.conf)

conf_replication_include.<conf_file_name> = <boolean>
* Whether Splunk replicates changes to a particular type of *.conf file, 
  along with any associated permissions in *.meta files
* When set to "true", the Cluster Manager/Deployer replicates these 
  configuration changes to all SHC members
* This ensures consistent configuration across the cluster
* Default: false

Critical : When search_history_storage_mode=kvstore is set, the parameter conf_replication_include.history=true must ALSO be set in server.conf. Without this parameter:

• Individual SHC members may have inconsistent search history configuration
• The cluster cannot guarantee uniform behavior across all members
• New search history may not be saved consistently across the cluster
• Users may see different history when accessing different cluster members
• Before setting the search_history_storage_mode=kvstore, set  conf_replication_include.history = true in SHC, it will save the old history. If failed to enable, the  conf_replication_include.history = true, the old history will be permanently deleted, only new history will be available for use in SHC.

To verify that both parameters are properly configured, use the btool command on each SHC member:

Check Parameter 1 (limits.conf):

$SPLUNK_HOME/bin/splunk btool limits list --debug | grep -A 10 "search_history"search_history_storage_mode = kvstore
enable_history = true
max_history_length = 500
max_history_storage_retention_time = 90d

Check Parameter 2 (server.conf):

$SPLUNK_HOME/bin/splunk btool server list --debug | grep "conf_replication_include.history"conf_replication_include.history = true

kvstore Health Verification
Search history replication depends entirely on kvstore functioning properly. Verify kvstore status:

$SPLUNK_HOME/bin/splunk show kvstore-status
Expected output should show kvstore as "running" and healthy. If kvstore is not running, search history replication will fail.

Search for kvstore-related errors in logs:

index=_internal source=*kvstore* ERROR OR WARN

Check for migration logs:

index=_internal component="SearchHistoryMigrate" INFO OR ERROR

If this component logs appear on cluster members during and after configuration, the migration is active. However, if no logs appear on ANY cluster member (including the captain), this may indicate:

• Configuration changes were not properly deployed
• The parameter conf_replication_include.history=true was not set
• Configuration changes were not replicated to cluster members
• splunkd was not restarted to apply changes

Resolution

1. Verify KVstore is running: Use below from the command line or check the Splunk UI.
   

   splunk show kvstore-status

2. Check for KVstore replication errors in splunkd.log on all SHC members.
   
   
3. Verify both configuration parameters (search_history_storage_mode=kvstore in limits.conf and conf_replication_include.history=true in server.conf) are set correctly on ALL SHC members.
   
   
4. Confirm a full SHC restart was performed after applying changes.

 History on only one member or inconsistent history
Verify conf_replication_include.history=true is set in server.conf on all SHC members.
Check SHC configuration bundle push status to ensure all members have received the updated configurations.
Verify KVstore replication health across the cluster. In some cases, if multiple search heads are restarted, the 'last SH to restart wins' scenario might occur, leading to temporary inconsistencies.

Following the completion of all investigative procedures, it was determined that an additional parameter exists in the server.conf file as outlined below:

conf_replication_include.<conf_file_name> = <boolean>
* Whether Splunk replicates changes to a particular type of *.conf
  file, along with any associated permissions in *.meta files.
* Default: false

After setting the parameter conf_replication_include.history=true, the issue was resolved, and new history began to replicate throughout the entire cluster. However, it is important to note that the old history will not be available because this parameter was configured after the search_history_storage_mode was set to kvstore. Therefore, if it was not set prior to this, the old history will not be retrievable, although you will observe the new history being replicated across the cluster.

Related Links

Server.conf

Limits.conf

Configuration updates that the cluster replicates