Search factor not met error on Cluster master

Applicability

Description

Search factor not met error on Cluster master.

Error message:

One or more replicated indexes might not be fully searchable. Some search results might be incomplete
or duplicated during bucket fix up. For more information, check the cluster manager page on the master.

• Indexes with Excess Bucket = 1
• Remove function does nothing.
• Master node is showing: 106 searchable 1 not searchable
• This index is one of the largest and most important.

Investigation

1. As per splunkd.log:

splunkd.log.2:11-17-2020 01:55:01.806 +0000 WARN CMBucket - primalities did not match! bf: 
41, internal:61

Also:

$ grep "WARN CMBucket - primalities did not match" splunkd.log* | wc -l
3217

2. Error:

11-17-2020 08:13:16.892 +0000 INFO CMMaster - event=commitGenerationFailure pendingGen=47360 
requesterReason=changeBucketMasks failureReason='event=checkDirtyBuckets first unmet 
bid=oneid_audit~1294~4AFA-8D7E-41DF7437D9F9'

Errors:

$ grep commitGenerationFailure splunkd.log | grep -v "pendingGen=47360" | wc -l
14
$ grep commitGenerationFailure splunkd.log | grep "pendingGen=47360" | wc -l
1831

11-17-2020 08:14:33.981 +0000 INFO CMMaster - event=commitGenerationFailure pendingGen=47360 
requesterReason=changeBucketMasks failureReason='event=checkDirtyBuckets first unmet 
bid=oneid_audit~1294~4AFA-8D7E-41DF7437D9F9'

Rolling restart did not solve the issue

./splunk rolling-restart cluster-peers

Resolution

Master node is now showing
106 searchable 1 not searchable

2. Troubleshooting steps: 
- Remove excess buckets.
- Found the bucket id from logs.
- Ran:

<host>:<port>/services/cluster/master/buckets/oneid_audit~1294~7C701695-43D2-4AFA-8D7E-41DF7437D9F9

- Bucket was searchable and no issues with checksum 3. Upon further checking, found a known issue: SPL-90331.

-Multi-site indexer cluster doesn't meet replication factor/search head factor due to bucket issue.

Tried fixing the bucket:
From the endpoint, add the buckets missing RF/SF to the to_fix list.
endpoint: https://[host]:[port]/services/cluster/master/buckets/{bucket_id}/fix (https://[host][port])

Even after applying the above fix, the Search Factor (SF) was not met.


3. On master:
- ran: 

ps auxww | grep splunk

It was found that Splunk was started on a previous date, and also another Splunk process running since a more recent date,  meaning that Splunk did not restart properly on Cluster  Master (CM).

4. To fix:
- Stop Splunk on  the Cluster Master and make sure no stale process is running.
- Start Splunk service

- both SF and RF met.
 

Related Links

See Endpoints reference list for examples and more information about how to use the endpoints.