How to clean up srtemp directory

Applicability

Splunk Enterprise Splunk Cloud

Description

What is the srtemp directory in "/opt/splunk/var/run/splunk" used for ?

srtemp is the temp directory for search post-process actions, which are used in Splunk dashboards to provide additional search pipeline processing of searches.

This is useful for dashboards when you want one base search to gather information, and then portions of that information presented in multiple ways, and permits creation of dashboards with complex displays efficiently.

Additionally, srtemp is used by search actions that are categorized as dataset processing commands .  Notable examples of commands that use srtemp are sort, trail, and fillnull.


Occasionally the srtemp directory slowly fills up the space in "/opt/splunk/" 

A real world example of this issue occurred when a user ran a search using the table command with over 75 fields being called and over a time span of 30 days that was appended with the fillnull command written this way:

|fillnull value=-EMPTY-

The above command tells Splunk post process to identify all empty fields from the completed search and then fill those empty fields with the string "-EMPTY-." This search spiked the disk usage of Splunk to 96%

The process was killed and afterwards, the srtemp directory was manually cleared of the offending 400GB file.

Of note is that the search itself was not excessively large, and the Jobs Activity dashboard only showed a search size of 15MB at the time the search was killed.  The search process was not the cause of the problem; the post search process was. 

Investigation

This is an example of how much space the srtemp directory can consume:

$ du -hax /opt/splunk/|sort -rh | grep srtemp
384G    /opt/splunk/var/run/splunk/srtemp
224G    /opt/splunk/var/run/splunk/srtemp/1365559618_7777_at_1623978042.1
155G    /opt/splunk/var/run/splunk/srtemp/1856343123_30528_at_1623996545.1
5.8G    /opt/splunk/var/run/splunk/srtemp/1724076486_30642_at_1623996558.1

Understanding what the numbers mean:

For example let us take 1724076486_30642_at_1623996558.1

• The first number 1724076486 is a random number.
• The second number 30642 is the PID.
• The third number 1623996558.1 is a timestamp.

So, to find the search ID, you need to take the PID and run the next command:

$ ps -ef | grep 30642
splunk   30642 23472  2 Jun18 ?        02:14:56 [splunkd pid=23393] search --id=1623996541.320580 
--maxbuckets=300 --ttl=600 --maxout=500000 --maxtime=8640000 --lookups=1 --reduce_freq=10 --rf=* 
--user=username@splunk --pro 
--roles=ess_admin:ess_analyst:ess_user:power:sc_admin:tokens_auth:user

Resolution

To clean up the srtemp directory the following can be run:

$SPLUNK_HOME/bin/splunk clean-srtemp

While srtemp does have a reaper to maintain the folder, with srtemp_dir_ttl by default set for one day in limits.conf, manually cleaning the folder will still be necessary in circumstances where the post process creating the large file was halted abnormally. In those cases, the reaper will not clean the file despite the TTL setting.



 

Related Links

Commands By Type

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release