Search Head Cluster members report error pulling configurations due to outdated baseline

Applicability

Description

Search Head Cluster deployment with 20+ Nodes.
All  Search head cluster is stable and also Kvstore is in synch.
These members started to report out of synch  and reporting errors    Cannot accept push with outdated_baseline_op_id  and the configuration changes are not being replicated between them 

Error pushing configurations to captain=https://XXXXX-XX-478:8089, consecutiveErrors=62 msg="Error in acceptPush: Non-200 status_code=400: ConfReplicationException: Cannot accept push with outdated_baseline_op_id=d225a6ec256c69b28b959d376a93d2a24ac6d8ff;

Search to check missing baseline 

index=_* source=*conf.log* STOP_ON_MISSING_LOCAL_BASELINE host IN ( SHC1,SHC2,SHC3...SHC21)
| timechart count by host

So based on this screenshot the issue seems to have started on May29:11:20AM, it helps find that start time of the issue.

Investigation

1) Checking  the changes are for which knowledge Object Type

index=_* source=*conf.log* "data.task"=pullFrom OR data.task=addCommit  "data.optype_desc"=WRITE_STANZA
| eval user= if( isnull('data.asset_uri{}'), mvindex('data.dst{}',0), mvindex('data.asset_uri{}',0))
| eval app=if( isnull('data.asset_uri{}'), mvindex('data.dst{}',1), mvindex('data.asset_uri{}',1))
| eval KO_type=if( isnull('data.asset_uri{}'), mvindex('data.dst{}',2), mvindex('data.asset_uri{}',2))
| eval KO=if( isnull('data.asset_uri{}'), mvindex('data.dst{}',3), mvindex('data.asset_uri{}',3))
| where isnotnull(KO)
| timechart span=1h count by KO_type

So it's the authentication

2) Lets check changes are being made for which stanza

SPL

index=_* source=*conf.log* component=ConfOp WRITE_STANZA host IN (XXXXXX*, XXXXX*)  
|rex field=_raw "(\"asset_uri\"\:\[\"(?<user>\w+)\"\,\"(?<app>\w+)\",\"(?<conf_type>\w+)\",\"(?<stanza>\w+)\"") 
|search stanza=lockedRoleToFullDNMap_SAML
|timechart count by stanza

The issue seems to be with the replication of the stanza  lockedRoleToFullDNMap_SAML  from authetication.conf    as too many changes are being made for it.

3)   From these error messages review the configuration replication between Search Head Cluster Member

Check the object that   for one  of the op_id in screenshot of point (3) that is reporting - Cannot accept push with outdated_baseline



Notice that Search Head Cluster Member has not received changes for KO_type : authentication
 

Resolution

The review  of the authetication.conf shows the stanza  lockedRoleToFullDNMap_SAML  has 660 lines as shown below 

This is SAML group mapping with roles of Splunk. 
SMAL IDP is returning long list of groups which caused authentication.conf stanza  lockedRoleToFullDNMap_SAML to have 600 lines of SAML group and Splunk role mapping.
To resolve the issue  IDP was fixed to limit the groups being returned, in addition the authentication on captain was manually fixed.
Other Search Head cluster member were  resynched with Captain using CLI command.

splunk resync shcluster-replicated-config