Missing Notable Events from Incident Review Dashboard in Splunk Enterprise Security

Applicability

Splunk Enterprise Security Splunk Enterprise Splunk Cloud

Description

When viewing Splunk Enterprise Security Incident Review dashboard new notable events are no longer be displayed and stream search error messages are displayed in the web UI dashboards. 

Investigation

Extract and run the dashboard search to verify if the error below is returned: 

Streamed search execute failed because: Error in 'lookup' command: Failed to re-open lookup file: '/export/gcs1/data/splunk/var/run/searchpeers/F6E1A2AD-150D-4BB5-9E8F-62274D0C12E6-1690218873/kvstore_s_SA-IdeRjww0FotymhlCIaS1cqkc05a_assetsXy0Y9f6F5lMW4rOy8KLC@P22'

Or 

Run any related look up generating search on the afflicting search head: 
 

index=my_index
| lookup oneshot.csv output abc, abcd
| search abc=*
| table
[search index=master_list | fields xyv, cde | fields - _raw,_time
| outputlookup oneshot.csv | eval results = " " | return $results]

Resolution

To resolve this issue do the following

1. Update the [lookup] stanza as follows on all search peers"

$SPLUNK_HOME/etc/system/local/limits.conf
 

 [lookup] 
enable_splunkd_kv_lookup_indexing = true

2. Remove "mongod.log" file on affected Search heads: 

$SPLUNK_HOME/var/lib/splunk/kvstore/mongo/mongod.log
 

rm -f $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/mongod.log

3. Rolling restart or restart of the Search head / Search head cluster members: 
 

Stand alone Search head:
$SPLUNK_HOME/bin/splunk/restart Search Head Cluster: 
$SPLUNK_HOME/bin/splunk rolling-restart shcluster-members

4. Restart of Cluster manager and Indexing tier. 
 

$SPLUNK_HOME/bin/splunk/restart 

 

Related Links

https://docs.splunk.com/Documentation/Splunk/9.0.3/ReleaseNotes/KnownIssues#Search_issues

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release