When to disable the KV Store in Splunk Enterprise instances

Applicability

Description

In any server, it is a best practice to disable processes or services which are not expressly needed. This helps with resource management, ensures unrelated issues are avoided during upgrades and restarts, and can help reduce a system's attack surface. This article discusses in which Splunk Enterprise instances need the KV Store to remain active, and which do not.

Investigation

Splunk Enterprise will run the KV Store (MongoDB database) by default unless it is specifically disabled. Because the KV Store is only used for searches, it can be disabled on every Splunk Enterprise instance that does not need it to function. 

In most use cases, the KV Store can (and probably should) be disabled on the following instances:

• Indexers, whether clustered or not (in most cases, the Web UI is disabled entirely on indexers -- KV Store should be also)
• Cluster Manager (if indexers are clustered)
• Search Head Cluster Deployer (if a search head cluster is configured)
• License Manager
• Deployment Server
• Heavy Forwarders which are not used as search heads and do not run DB Connect (or other app(s) which rely on the KV Store)

The only Splunk Enterprise instances that expressly use the KV Store are:

• Standalone Splunk instances
• Search Heads
• Monitoring Console (which uses the KV Store only for KV Store related dashboards)
• Deployment Server (uses KV Store collections in its forwarder management searches)
• Heavy Forwarders (if running the Splunk DB Connect app or a custom app that relies on the KV Store).
• Any instance which uses token-based authentication for logging into the Splunk Web UI. For example, if administrators at your organization use token-based authentication even for logging in to the UI on the Cluster Manager or Deployment Server (or other non-Search Head instance where the Web UI is enabled), in many cases those tokens rely on the KV Store, so the KV Store should not be disabled.

If your Monitoring Console (MC) is co-located with another server role such as the License Manager or Search Head Cluster Deployer, and you want to ensure the MC's KV Store dashboards will function properly, do not disable the KV Store. (See also Splunk Doc on Which instance should host the console)

Resolution

Disable the KV Store

To disable the KV Store on a Splunk Enterprise instance:

1. add the following configuration to server.conf:
   

   [kvstore]
   disabled = true

2. Then, restart Splunkd on the affected instance:

   $SPLUNK_HOME/bin/splunk restart

   OR

   sudo systemctl restart Splunkd.service

3. For verification:
   1. Check KV Store status 
      

      $SPLUNK_HOME/bin/splunk show kvstore-status

      If KV Store is disabled, result should show:
      

      Encountered some errors while trying to obtain kvstore status.
      KVStore is unavailable.

       

   2. Verify on-disk settings for server.conf's [kvstore] stanza for a specific instance:
      

      $SPLUNK_HOME/bin/splunk btool server list kvstore --debug

      If KV Store is still running but this command shows disabled=true, restart Splunkd on this instance for the settings to take effect.

Considerations

Clustered Indexers

For clustered indexers, this setting should be added to the Cluster Manager in a custom app or the Splunk-provided _cluster app located at management_apps/_cluster/local/server.conf.

• For clustered indexers only, apps coming from the Cluster Manager take precedence over configurations located in $SPLUNK_HOME/etc/system/local.
• Push the bundle from the Cluster Manager in order to apply the configuration (this will likely initiate a rolling restart of the indexer cluster)

Deployment Clients

For configurations deployed via a Deployment Server, place the server.conf configuration into a custom app in the Deployment Server's $SPLUNK_HOME/etc/deployment-apps directory.

• For Deployment Clients, the $SPLUNK_HOME/etc/system/local path will take precedence over any configurations pushed by the Deployment Server. Use the btool command to validate current on-disk configurations and from where a Splunk Enterprise instance's configurations originate.
• Check "enable Splunkd restart" to ensure Splunkd is restarted when the new configurations are received.

All Splunk Enterprise Instances that are not Clustered Indexers

On all Splunk Enterprise instances that are NOT clustered indexers, $SPLUNK_HOME/etc/system/local settings take the highest precedence. So, just like with Deployment Clients, any configuration coming from this folder will override all other settings on the server.

Related Links

• Splunk Documentation:
  • Which instance should host the console
  • token-based authentication
  • Disabling the KV Store