Data Model Acceleration has gaps after restart of the Cluster Master and Search Head Cluster

Applicability

Description

The Data Model Acceleration is reporting gaps, as shown below.
The environment is simple: one Standalone Search Head and two Peer Cluster Managers. 
The Cluster Manager has ( Replication Factor = Search Factor = 2) 

Investigation

Here are the various checks that were done:

1) The DataModel  "Authentication"  is fully accelerated  and it was verified using the REST call shown below

What is the stats of all the DMAs?This rest search can be used to give you a single pane view of all the information surrounding all the DMAs
| rest /services/admin/summarization by_tstats=t splunk_server=local count=0
| eval datamodel=replace('summary.id',(("DM_" . 'eai:acl.app') . "_"),"")
| join type=left datamodel
    [| rest /services/data/models splunk_server=local count=0
    | table title, "acceleration.cron_schedule", "eai:digest"
    | rename title as datamodel
    | rename "acceleration.cron_schedule" as cron]
| table datamodel, "eai:acl.app", "summary.access_time", "summary.is_inprogress", "summary.size", "summary.latest_time", "summary.complete", "summary.buckets_size", "summary.buckets", cron, "summary.last_error", "summary.time_range", "summary.id", "summary.mod_time", "eai:digest", "summary.earliest_time", "summary.last_sid", "summary.access_count"
| rename "summary.id" as summary_id, "summary.time_range" as retention, "summary.earliest_time" as earliest, "summary.latest_time" as latest, "eai:digest" as digest
| rename "summary.*" as "*", "eai:acl.*" as "*"
| convert ctime(mod_time), ctime(earliest), ctime(latest), ctime(access_time)
| eval retention_days=retention/60/60/24
| sort datamodel |table datamodel ,app,access_time,is_progress,size,complete,bucket_size,cron,retention_days,mod_time,earliest,latest

2) Next, using the search below, verify that the Authentication DMA is based only on the needed indexes. 

There is an endpoint for macros that can be filtered to display the indexes white list macros. The following search will display all the macros for the Data models in the CIM app. The 'title' filter can be updated to include other index whitelists | rest /servicesNS/-/-/admin/macros splunk_server=local | search title=cim_*_indexes | table title definition

In this case, for the DMA: Authentication, the critical index is pan_logs, which has a retention of frozenTimePeriodInSecs=15552000 (180 days).
Based on this retention, we don't expect any of the summary index data to hit retention, so we eliminated that as the cause of this issue.

3) Next check: What is the acceleration status of a few buckets during the gap for some buckets?

To get the acceleration status at the bucket level, two searches need to be run.

The first search hits the summarization endpoint and constructs another search with the results. The values in the 'final_search_cmd' field are probing summarize searches that will yield the needed results.
This particular search will generate a search for the Authentication Datamodel. Update the "| search summary_id =" section with criteria that match the appropriate data model. 
  

| rest splunk_server=local by_tstats=t /services/admin/summarization
 | rename summary.id as summary_id
 | table summary_id, search| search summary_id ="*Authentication*"
 | eval final_search_cmd = "| summarize tstats=t action=probe id=".summary_id." normid=[".search."]"
 | table final_search_cmd

Get the search string starting from (index=* OR index=_*)…….  and it was run for the time when the data was missing.

The screenshot below shows one base bucket from the gap period.

Here is an analysis of one bucket from the gap time period, where DMA is expected to show data.

4) Using |dbisnpect, check the bucket that exists on which indexers.

5) Check the Datamodel acceleration for this bucket on each of the indexers listed above

5.1) On indexer1, DMA bucket's  exists as shown below

DMA:DM_SPLUNK_SA_CIM_Intrusion_detection 
DMA:DM_SPLUNK_SA_CIM_Web
DMA: DM_SPLUNK_SA_CIM_Authentication 
DMA: DM_SPLUNK_SA_CIM_NetworkTraffice

5.2) On indexer2 the  DMA buckets listed are less as compared to indexer1.

DMA:DM_SPLUNK_SA_CIM_Intrusion_detection 
DMA:DM_SPLUNK_SA_CIM_Web

So, the Data model acceleration between the two Indexers is different. Depending on which indexer is primary, the Authentication Data Model may or may not display data.

Resolution

To resolve this issue set  summary_replication = true  in server.conf clustering]

Refer :  Clustersandsummaryreplication
 

Related Links

 
DMA Troubleshooting