Unable to clear old threat intel from large kvstore via 'outputlookup' command

Applicability

Description

Trying to remove old threat intel from kvstore via outputlookup command, however the searches seems to hang and there is no timeout message showing.

 The following searches below over the weekends (in background), however the search does not update the respective kvstore:

| inputlookup ip_intel
| search threat_key!=rf*
| outputlookup ip_intel

| inputlookup http_intel
| search threat_key!=rf*
| outputlookup http_intel

| inputlookup file_intel
| search threat_key!=rf*
| outputlookup file_intel

Investigation

Steps for investigation

1. Verify the size of kvstore, by checking the amount of entries/data
2. Checked search log to determine what is happening during the search job
3. Check documentation on known issues and JIRAs for similiar issue

Findings from investigation

1. All 3 lookup files were large (more than 1 million entries), with the ip_intel lookup file having more than 2.5 Million entries.
2. Search log shows ERRORs when executing 'inputlookup' command, this indicates that the command is being executed. However, it does not show information on 'outputlookup' command, this indicates that the command is not being executed.
3. Looking through JIRAs, there is a known issue for kvstore client socket timeout while modifying large kvstore collection that aligns with the issue.
4. kvstore clientSocketTimeout is set to be 300 (default value)

Resolution

Increase clientSocketTimeout to 1800 in /opt/splunk/etc/system/local/server.conf

server.conf
[kvstore]
clientSocketTimeout = 1800

NOTE: 'outputlookup' command to a kvstore has a default of 50,000 limit, do use 'max' optional argument to change the default value for the search job. If outputlookup has to push more than 50 000 entries into a kvstore collection.

Related Links

Outputlookup