Unable to use preserve /unified search history feature across search heads, some users' search history is missing

Applicability

Splunk Enterprise 9.1.1

Description

 After updating to 9.1.1, the users were expecting the preserve/unified search history across the search heads. This feature solves the issue with lost search history when users switch between various nodes in a search head cluster. This feature utilizes KV store to keep search history replicated across nodes. However, this feature is not working as expected, and some user's search history is missing.

Investigation

To demonstrate this issue, ask the client to follow these steps to check the user search history and related settings:

1. Please ask the affected user to run this search :

| history | search NOT search="| history*" AND NOT search="*metadata*" AND NOT search="*loadjob*" AND NOT savedsearch_name="*" AND NOT search="search" AND NOT search="*"

This SPL is used for displaying results for Search History of the user. 

Another way to check this is by manually listing the user's search history files, which are located in the folder $SPLUNK_HOME/etc/users/<username>/search/history/

If either option displays empty results, it demonstrates that the user's search history is indeed missing.


2. Check if history search is turned off. This is controlled via the limits.conf under the stanza [search] by the parameter enable_history Please note that it will be disabled if the parameter enable_history is false, otherwise, it is enabled since the default value is true.

./splunk btool limits search list --debug | grep local

If the parameter is set to enable_history = false, then this is the cause of the issue. 


3. Verify whether this missing search history issue is happening to users with the same role. If so, check the role capabilities and restrictions. 

If the missing search history issue is not explained by the role capabilities/restriction or history search disabled, then this problem is related to the preserve/unified search history feature configuration. 

Resolution

To resolve this issue, follow the next steps:

1. Check the KV store status. The preserve/unified search history feature utilizes KV store to keep search history replicated across nodes. Therefore, run the following test to check that KV store is working as expected:

./splunk show kvstore-status

The output should show that the process is running, with status 'ready'.

2. Switch from csv history storage to KV Store. In order to do this, modify the $SPLUNK_HOME/system/local/limits.conf file in the Deployer by changing the parameter search_history_storage_mode to kvstore under the stanza [search] as shown below:
 

[search]
search_history_storage_mode = kvstore

A value of "kvstore" means that the cluster can replicate search history across all its members using the App Key Value Store service.

3. Save the configuration file, restart the Deployer.

4. Once the limits.conf file is updated, push the change through the deployer 

5. Check that the configuration file was updated in the search head peers and restart.

In order to verify that the issue is solved, check the affected user search history again in  $SPLUNK_HOME/etc/users/username/search/history/  folder or run the SPL detailed above, the search history should be available now.

NOTE: when this setting value of "kvstore" is initially given, the search head migrates the existing search history records, if they are present in existing CSV files, into the App Key Value Store service. This migration of search history can only happen once.



 

Related Links

Preserved/unified search history feature
search_history_storage_mode
Use the deployer to distribute apps and configuration updates
 

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release