Applicability

Description

Looking for Splunk Search that would provide help with the review of the Data Model Acceleration.

Investigation

Datamodel Acceleration Explained

For Splunk, DM acceleration is extremely resource intensive. Think of it as an investment:

• Data models on their own are really just a collection of searches meant to allow users with no knowledge of SPL to look through data on their own.
• CIM uses them differently though. Users don't pivot through CIM data models. Instead these exist for apps like ES, PCI, etc.
• When data is accelerated, it is getting re-indexed – this means additional disk space is used and the searches are CPU/Memory heavy. By default, Splunk will accelerate the data every 5 minutes with up to 3 concurrent searches PER data model.
• The point of accelerating CIM data models is that all extracted CIM fields are stored as indexed fields, making it possible to run tstats searches against those fields, which is extremely fast, but if not executing a fair number of tstats searches, this is a net negative overall. Again, this is an investment.
• The two important aspects of a Data Model
  • Definition: The indexes inside this definition are what the acceleration searches are run against. This needs to be constrained as much as possible. Having a blank value "()" means they are searching index=*!!! They should ONLY include indexes which contain CIM'd data tagged for that data model, and even then, ONLY if that data source is actually needed while running searches.
  • Searches_scheduled_using_tstats & dashboards_using_tstat
    • If both numbers are 0, there is literally no reason for acceleration and it is wasting a ton of resources.
    • If Searches_scheduled_using_tstats is 0 and dashboards_using_tstats has hits, this doesn't mean people are necessarily looking at those panels just that they opened some dashboard that had the panel. It would be still best to disable acceleration and wait to see if anyone has issues with their dashboards.

The search below will provide a high-level detail of current Datamodels in. your environment.

Sample:

Scope: Standalone SH or all members of a SHC

SPL (Search)

| rest /servicesNS/-/-/admin/macros splunk_server=local 
| search title=Cim_*_indexes | table title definition 
| rex field=title "cim_(?<datamodel>\w+)_indexes" 
| rename title AS macro 
| join type=outer datamodel[| rest /servicesNS/nobody/-/datamodel/model splunk_server=local | table title acceleration | rex field=acceleration "\"enabled\":(?<acceleration_enabled>[^,\"]+)" | rex field=acceleration "\"earliest_time\":\"(?<acceleration_earliest>[^\"]+)" | fillnull acceleration_earliest value="N/A" | rename title AS datamodel | fields - acceleration] 
| join type=outer datamodel [| `datamodel("Splunk_Audit", "Datamodel_Acceleration")` | `drop_dm_object_name("Datamodel_Acceleration")` | eval "size(MB)"=round(size/1048576,1), "retention(days)"=if(retention==0,"unlimited",round(retention/86400,1)), "complete(%)"=round(complete*100,1), "runDuration(s)"=round(runDuration,1) | sort 100 + datamodel | table datamodel,complete(%),size(MB),access_time | eval datamodel=if(datamodel="Endpoint.Filesystem","Endpoint",datamodel)] 
| join type=outer datamodel [| rest splunk_server=local /servicesNS/-/-/configs/conf-savedsearches | eval rule_name=title | fields + title,rule_name,dispatch.earliest_time,dispatch.latest_time, cron_schedule | join type=outer title [| rest splunk_server=local /servicesNS/-/-/configs/conf-savedsearches | fields + title,search,disabled, cron_schedule] 
| rex field=search "datamodel\W{1,2}(?<datamodel>\w+)" 
| rex field=search "tstats.*?from datamodel=(?<datamodel>\w+)" 
| eval datamodel2=case(match(search, "src_dest_tstats"), mvappend("Network_Traffic", "Intrusion_Detection", "Web"), match(search, "(access_tracker|inactive_account_usage)"), "Authentication", match(search, "malware_operations_tracker"), "Malware", match(search, "(primary_functions|listeningports|localprocesses|services)_tracker"), "Application_State", match(search, "useraccounts_tracker"), "Compute_Inventory") 
| eval datamodel=mvappend(datamodel, datamodel2) 
| search datamodel=* 
| mvexpand datamodel 
| eval uses_tstats=if(match(search, ".*tstats.*"), "1", "0") 
| eval is_scheduled=if(match(cron_schedule,".+"), "1", "0") 
| eval is_scheduled_with_tstats=uses_tstats+is_scheduled 
| eval is_scheduled_with_tstats=if(is_scheduled_with_tstats==2, "1", "0") 
| eval enabled=if(disabled==0, "Yes", "No") 
| search enabled=yes 
| stats sum(is_scheduled_with_tstats) as searches_scheduled_using_tstats, sum(is_scheduled) as searches_scheduled, sum(uses_tstats) as searches_using_tstats, count(rule_name) as searches_enabled by datamodel 
| fillnull searches_enabled,searches_using_tstats,searches_scheduled,searches_scheduled_using_tstats value="0"] 
| join type=outer datamodel [| search index=_audit provenance=UI* action=search info=completed datamodel earliest=-7d@d host=SH_HERE | dedup app,savedsearch_name | where len(trim(savedsearch_name)) != 0 | rex field=search "datamodel\W{1,2}(?<datamodel>\w+)" | rex field=search "tstats.*?from datamodel=(?<datamodel>\w+)" | eval uses_tstats=if(match(search, ".*tstats.*"), "1", "0") | stats sum(uses_tstats) as dashboards_using_tstats by datamodel] 
| fieldformat access_time=strftime(access_time, "%m/%d/%Y %H:%M:%S") 
| table datamodel acceleration_enabled, acceleration_earliest, macro, definition, size(MB), searches_scheduled,searches_scheduled_using_tstats, dashboards_using_tstats 
| sort -acceleration_enabled definition -size(MB)

Datamodel Acceleration Analysis

To determine the appropriate summary ranges for data models, review the search below. For example, if no searches are looking more than 24 hours into the past (earliest), yet the data model is accelerated for 90 days (acceleration_earliest), this is an opportunity to trim the data model resource utilization and make rebuilds less intensive by reducing the length of acceleration to match the associated search which looks back the farthest.

The search below can also be used to identify which searches are using which data models. Again, this pairs with the previous search because all Datamodels should be reviewed (especially scheduled tstats searches) to evaluate whether they're actually valuable. If a datamodel is not being used effectively or is not used at all, a lot of resources are being spent accelerating them with no real benefit.

Scope: Standalone SH or all members of a SHC

SPL (Search):

| rest splunk_server=local /servicesNS/-/-/configs/conf-savedsearches
| eval rule_name=title
| fields + title,rule_name,dispatch.earliest_time,dispatch.latest_time, cron_schedule
| join type=outer title
    [| rest splunk_server=local /servicesNS/-/-/configs/conf-savedsearches
    | fields + title,search,disabled, cron_schedule]
| rex field=search "datamodel\W{1,2}(?<datamodel>\w+)"
| rex field=search "tstats.*?from datamodel=(?<datamodel>\w+)"
| eval datamodel2=case(match(search, "src_dest_tstats"), mvappend("Network_Traffic", "Intrusion_Detection", "Web"), match(search, "(access_tracker|inactive_account_usage)"), "Authentication", match(search, "malware_operations_tracker"), "Malware", match(search, "(primary_functions|listeningports|localprocesses|services)_tracker"), "Application_State", match(search, "useraccounts_tracker"), "Compute_Inventory")
| eval datamodel=mvappend(datamodel, datamodel2)
| search datamodel=*
| mvexpand datamodel
| join type=outer datamodel
    [| rest /servicesNS/nobody/-/datamodel/model splunk_server=local
    | table title acceleration
    | rex field=acceleration "\"enabled\":(?<acceleration_enabled>[^,\"]+)"
    | rex field=acceleration "\"earliest_time\":\"(?<acceleration_earliest>[^\"]+)"
    | fillnull acceleration_earliest value="N/A"
    | rename title AS datamodel
    | fields - acceleration]
| eval uses_tstats=if(match(search, ".*tstats.*"), "yes", "no")
| eval enabled=if(disabled==0, "Yes", "No")
| search enabled=yes
| rename dispatch.earliest_time AS earliest, dispatch.latest_time AS latest
| table rule_name enabled uses_tstats earliest latest datamodel acceleration_enabled, acceleration_earliest, cron_schedule
| sort - acceleration_enabled datamodel

Shows all enabled searches using data models.

Resolution

The shared Splunk searches can be used to review Data Model definitions and their utilization.

Related Links

data-model-acceleration-enforcement