Duplicate logging from Kubernetes via HEC ingestion is observed in Splunk

Applicability

Description

HEC data is not indexing as expected and it creates duplicates across multiple servers for a specific index which is having a specific source from Kubernetes

Investigation

 1.First check the HEC token status using the below command in Splunk SH GUI.

|rest /services/data/inputs/http/http
| table name, token , disabled

2.Duplicates may arise if the source is incorrectly tagging multiple sourcetypes. So check if the source or sourcetypes is misconfigured.

index=<index_name> | stats count by source ,sourcetype

 3.Confirm if duplicates events are coming more than once.

index=<index_name> | stats count by _time, source, host | where count > 1 

4. Inspect the HEC inputs settings in Splunk by navigating to Settings > Data Inputs > HTTP Event Collector in Splunk GUI.  Make sure no HEC input is defined twice for the same source or application.

5. Check the Splunk internal logs for any errors or warnings related to HEC ingestion that might indicate misconfigurations or issues.

index=_internal spurce*splunkd.log* component-HttpEventCollector*

Resolution

• For the Kubernetes source, the command "stats count by _raw ...field N where count > 1" method of detecting duplicates is unreliable because this is due to the events which does not have any unique fields like Event ID etc. so it seems like all the events are duplicates when checked in Splunk GUI. Unless a relatively unique field like kubernetes.docker_id or kubernetes.pod_id is included in the raw data and check the duplicates from the source end. 

• The duplicates took place as there were multiple load balancers present. Make sure that the load balancing is properly configured to direct the data to the correct indexers because incorrect settings may lead to the same data being set to multiple indexers and it will lead to duplicates.

Related Links

How do I find Duplicate Events