tstat search with large lookup as sub-search failing with error "exited with code 255"

Applicability

Description

When using a sub search in the where clause of a tstats command that is too large (looks like more than 5,000 rows), the search fails and throws the error:
Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.

 

| tstats count from datamodel=Web where NOT
[| inputlookup lookup.csv
| fields dest
| rename dest as Web.dest
| format ] by Web.src Web.dest

In this example lookup_name.csv had ~ 110k rows.

Investigation

When the search fails with error below. 

[<INDEXER_NAME>] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.

The error message means that the search process on indexer INDEXER_NAME had a problem - in most cases, it crashed (or was killed).
If there is no crash.log, it means that the splunk was killed externally or that we had a problem saving the crash.log.
Looking at the search string, though (number of OR-s) - it's most likely stack overflow due to recursion.

 

Resolution

It's a known problem that recursion may cause crashes (i.e., inefficient search string with a lot of ORs/ANDs) 
Tracked by   SPL-144975:Redesign parts of Splunk and remove recursion

One way to work around the problem is to increase the ulimit for the stack, and the default is 8064, and enhanced values like 128k or 256k could address the issue.

Related Links