Search Head Clustering Replications Problems

Applicability

Description

Typical Root Causes of Replication Problems

• Configuration Changes

Changes to search and UI configurations are driven by user activity in the UI/CLI/REST.

Changes to system configurations made by administrators.

• Saved Searches

• Other Configuration Changes

• Large/Frequent CSV Changes

Investigation

Configuration Changes
Under search head clustering, there are two broad categories of configuration changes:

Changes to search and UI configurations are driven by user activity in the UI/CLI/REST

Save report

Add a panel to the dashboard

Create field extraction

Changes to system configurations made by administrators

Configure forwarding

Deploy centralized authentication (e.g. LDAP)

Install an entirely new app or hand-edited configuration file

Changes in the first category are replicated across a search head cluster automatically.

Changes in the second category must be validated outside of the search head cluster entirely and then pushed from a central instance – the deployer

Notable Behaviors
By default, configuration changes are replicated approximately every 5s. Under good conditions, we expect a change made on one member to be reflected SHcluster-wide in 5-10s.

The UI under search head clustering automatically disables access to system configurations. In other words, the "Settings" menu hides configurations that are meant to be deployed instead of edited "live" on a search head cluster member – this includes indexes, inputs, authentication, users, roles, etc.

If an administrator wants to make "one-off" changes to individual search head cluster members, it is possible to re-enable the full Settings menu via the UI itself. If a user does this, we log a WARN to splunkd.log to make this easy to spot.

Changes to log.cfg, log-local.cfg, etc. are not replicated.

Notable REST APIs

/replication/configuration

/apps/local

/apps/deploy

If you think there's a problem with configurations:

Before anything else, sanity-check the search head cluster:

search head clustering is configured on every node of interest
bootstrap succeeded
there's a stable captain in the shcluster – splunk show shcluster-status
the captain has a reasonable configuration – splunk list shcluster-captain-info
the member of interest has a reasonable configuration – splunk list shcluster-member-info 
**The member of interest has a stable connection to the captain**

Check splunkd.log. In particular, look at the following channels:

ConfReplication
ConfReplicationThread
ConfReplicationHandler
loader (during startup)

If perf issues are a potential concern, check metrics.log and look for group=conf. These metrics messages include summaries of how often various conf-related actions were performed during the last 30s interval, how much cumulative time was spent on each action "type", and the worst/longest execution time for a single invocation of any action. A rough, high-level description of what each "action" means:

• accept_push: on the captain, accept replicated changes from a member (related: push_to)
• acquire_mutex: acquire a mutex that "protects" the configuration system (related: release_and_reacquire_mutex)
• add_commit: on a member, record a change
  • base_initialize: initialize a configuration "root", e.g. $SPLUNK_HOME/etc (related: repo_initialize)
  • check_range: compare two ranges of configuration changes (related: compute_common)
  • compute_common: find the latest common change between a member and the captain (related: check_range)
  • pull_from: on a member, pull changes from the captain (related: reply_pull)
  • purge_eligible: on a member, purge sufficiently old changes from the repo
  • push_to: on a member, push changes to the captain (related: accept_push)
  • release_and_reacquire_mutex: release, then re-acquire a mutex that "protects" the configuration system (related: acquire_mutex)
  • reply_pull: on the captain, reply to a member's pull_from request (related: pull_from)
  • repo_initialize: initialize a configuration repo from disk (related: base_initialize)

If network activity between search head cluster members is of particular concern, check splunkd_access.log on the various members. Look for GETs and POSTs against the /replication/configuration REST API.

If network activity between search head cluster members and the deployer is of concern, look for GETs and POSTs against the /apps/deploy and /apps/local REST APIs.
 

Resolution

Worst Case Problems Caused by Replication Failure

If the captain becomes so overloaded that replications are delayed significantly, configurations may not complete across the cluster. This leads to individual SHC members falling out of sync with the captain, termed "out of baseline". If the number of operations out of sync remains below 10 or so temporarily, usually the SHC member will recover. However, should it grow beyond the ~10 threshold and remain there for 24 hours or more, it will not recover due to ops.json being purged of older changes.

Should captaincy re-election happen while configuration or CSV changes are in flight, the configuration changes may be lost. "May" is the key word here as it all depends upon how badly and how many of the SHC members were behind in receiving changes. If the captain happens to land on one that hasn't received all the changes, then the changes will be lost because that member will not have a record of the in-flight changes.

Finally, in the absolute worst case, if the load on the captain becomes so high that it is not able to respond to the cloud service provider's automated health checks, the captain instance will be re-provisioned.

Similar Impact: Large/Frequent KVstore Changes

Even though the KVstore replication process is separate from internal Splunk SHC replication if the KVstore is updated too rapidly or by fully replacing collections (think database tables) and/or should disk I/O also be constrained, this can impact overall SHC operations including replication. It can also cause the KVStore to become "stale," e.g. out of sync, just as SHC can fall out of the baseline and be unable to "catch up".

The only solution in this case is to resync the KVStore, which necessarily means a loss of any local changes on the SHC member where the KVStore is stale.

If the KVStore has continuous high rates of change, the best course of action is to increase the MongoDB oplog size to increase the window of time for replication catch-up.

Related Links

• Configuration updates that the cluster replicates
• How-to-troubleshoot-Search-Head-Clustering-initial-bootstrap
• Troubleshoot knowledge bundle replication