Splunk Enterprise Security - Notable Events in the Incident Review are displayed twice/duplicated. Eventstats in Splunk Server creates an inconsistent result count.

Applicability

Splunk Enterprise Splunk Enterprise Security

Description

The issue has been independently reported on Enterprise Security (mostly) and distributed/Search Head Clustered Splunk Server environments (not Splunk Server standalone).

On Enterprise Security

Duplicates for the Incident Review Dashboard: Notable Events appeared in a situation when a greater number of new Enterprise Security incidents were created in a short time i.e. several thousand incidents in 24 hours.
The Bug also occurs for other Correlation Searches.

On Splunk Server (distributed or clustered)

Inconsistent result counts can occur on Splunk Server when using eventstats.
The behavior seems to be tied to a tiered/distributed/clustered Splunk Server environment (remote search) and is not reproducible with a different standalone instance.

 

Investigation

On Enterprise Security

Error messages similar to the following will appear on the Search Head in the file splunkd.log.
 

11-28-2018 14:45:15.958 +0100 WARN TcpOutputProc - Possible duplication of events with channel=source::/opt/splunk/var/log/splunk/splunkd_ui_access.log
|host::splsearchhead.company.corp|splunkd_ui_access|640056, streamId=0, offset=0 on host=10.0.66.200:999

On Splunk Server (distributed or clustered)

Example:

index=foo filter” returns 6 resultsindex=foo <filter> | eventstats count(host)” 2 results.

Reviewing the search.log for executions without and with | eventstats, the following message appears:
 

10-23-2018 10:42:52.343 ERROR Timeliner - Ignored 4 events because they were after the commit time (1540283798).

 

Resolution

The resolution for the two independent scenarios (Enterprise Security and Splunk Server) is the same.

It was found to be a known Bug:

SPL-160881  eventstats on an event search can create duplicate events in some scenarios

The workaround of setting:
 

phased_execution_mode=singlethreaded

in limits.conf will enable eventstats to report correctly. The workaround should be used until a patch is released for Splunk Server 7.1.x and 7.2.x.

A fix has been written and is currently undergoing testing that should be available in a future patch.

For versions of Splunk Server:  7.2.0, 7.2.1, 7.2.2, 7.2.3, you will encounter a known Bug with charting (e.g. Pie Charts).

SPL-165363  "phased_execution_mode = singlethreaded" causes issue with field ordering

This is fixed in Splunk Server 7.2.4 and above.  It does not affect Splunk Server 7.1.x.

 

Revision History

Revision	Publish Date	Comments
1.0	28-Sep-2026	Initial Release