How to Export Large Amounts of Data Using CLI Search.

Applicability

Description

Step-by-step guide

To export large amounts of data you can use a search along the lines of the following from the CLI prompt on a search head:

splunk search "index=pan_log sourcetype=pan:* src_ip!=<Internal_IP_Scheme> dest_ip!=<Internal_IP_Scheme> earliest=06/01/2019:00:00:00 latest=06/01/2019:23:59:00 
| fillnull value=null 
| dump basefilename=<insert_your_chosenfilename> compress=5 fields=_time,app,app:subcategory,bytes_in,bytes_out,client_ip,dest_host,
dest_ip,dest_port,dest_translated_ip,dest_user,dest_zone,direction,duration,filename,http_referrer,iv_geo_city,iv_geo_country,misc,packets_in,
packets_out,sig,signature,sourcetype,src_ip,src_translated_ip,src_user,target_id,threat:category,threat:cve,threat:name,threat:severity,threat_category,url,url_length

The breakdown of the above search is as follows:

Searching the 'pan_log' index for pan:* sourcetypes and ruling out an IP range, able to fill nulls, and pipe to a dump command, which then proceeds to do the following:

This command runs a specified search query and oneshot export search result to local disk at "$SPLUNK_HOME/var/run/splunk/dispatch/<sid>/dump". It recognizes a special field in the input events, _dstpath, which if set will be used as a path to be appended to dst to compute final destination path.

The one major downside to this process is that the results have a TTL of 2 minutes, so you MUST create a save token file to make sure these results are not reaped before you can grab the *.gz files that are created by the process. See "RELATED" link below.

RELATED:

How to create a file with no extension in Windows

Investigation

While investigating this issue, it was found, not getting full data set.

Determined that it was due to search reaping grabbing this data and culling it faster than expected. Found this to be normal behavior after research. Summary index searches and others can run with a 2 minute search artifact TTL. Due to this, create a save token file in the <sid> directory. This is covered for Windows machines in the link in the description section of this article. With *nix you navigate to $SPLUNK_HOME/var/run/splunk/dispatch/<sid> and then issue the command 'touch save'

Once that was achieved were able to bypass the artifact reaping issue [***DONT FORGET TO DELETE THE SAVE FILES!***]

After this, press through a bit more troubleshooting to get a workable timeframe that would generate a defined full set of data. 24 hour runs of the search to complete it on a day by day export were necessary in the end due to the extreme amount of events in the data set.
 

Resolution

Once narrowed in on the dataset needing to be run against a 24 hour time period, able to generate full dumps of 24 hour stretches of PAN logging. Another TSE has been able to utilize this same method recently to go back historically and grab specific.

Related Links