Splunk Connect for Kubernetes(EKS) is failing to send logs due to in-correct log format type.

Applicability

Splunk Apps & Add-ons Splunk Cloud Splunk Enterprise

Description

After upgrading the EKS Cluster from version 1.23 to 1.26, encountered an issue with Splunk Connect for Kubernetes. The application log ingestion completely stopped from the EKS cluster to Splunk.

Investigation

When looked into the Splunk Connect for Kubernetes Pods logs, it was showing the below warning log: 

2023-03-09 18:41:34 +0000 [warn]: #0 [containers.log] pattern not matched: "{\"log\":\"xx.xxx.xx.xxx - - [09/Mar/2023:18:41:33 +0000] \\\"GET /u/s/GB-S1C04-4E-9PB0/document_downloads/HJHZR-DS-48H8-DN.json HTTP/1.1\\\" 200 152 \\\"https://dme.parachutehealth.com/u/s/GB-S1C04-4E-9PB0/orders/IYSP-1TU-9HF-7BJ/review\\\" \\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/xx.xxx.xx.xxx Safari/537.36\\\" \\\"xx.xxx.xx.xxx\\\" \\\"dme.parachutehealth.com\\\" 0.023 0.024\\n\",\"stream\":\"stdout\",\"time\":\"2023-03-09T18:41:33.543117699Z\"}" 2023-03-09 18:41:34 +0000 [warn]: #0 [containers.log] pattern not matched: "{\"log\":\"xx.xxx.xx.xxx - - [09/Mar/2023:18:41:33 +0000] \\\"GET /u/f/MI4U-Q8-AZ26X-UQ/dashboard/followers.json?term= HTTP/1.1\\\" 200 516 \\\"https://dme.parachutehealth.com/u/f/MI4U-Q8-AZ26X-UQ\\\" \\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/5x37.36 Edg/xx.xxx.xx.xxx\\\" \\\"192.5.255.1\\\" \\\"dme.parachutehealth.com\\\" 0.048 0.048\\n\",\"stream\":\"stdout\",\"time\":\"2023-03-09T18:41:33.592547523Z\"}" 2023-03-09 18:41:34 +0000 [warn]: #0 [containers.log] pattern not matched: "{\"log\":\"xx.xxx.xx.xxx - - [09/Mar/2023:18:41:33 +0000] \\\"GET /active HTTP/1.1\\\" 200 14 \\\"https://dme.parachutehealth.com/u/s/N5-GC-F9EA-PDA5C/orders/SIV-KD-E18KS-C53/review\\\" \\\"Mozilla/5.0 (iPhone; CPU iPhone OS 16_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Mobile/15E148 Safari/604.1\\\" \\\"xx.xxx.xx.xxx\\\" \\\"dme.parachutehealth.com\\\" 0.020 0.019\\n\",\"stream\":\"stdout\",\"time\":\"2023-03-09T18:41:33.59474438Z\"}" 2023-03-09 18:41:34 +0000 [warn]: #0 [containers.log] pattern not matched: "{\"log\":\"xx.xxx.xx.xxx - - [09/Mar/2023:18:41:33 +0000] \\\"GET /u/s/JARX-3HLP9-RXBHM/orders/TDJ41-EJ2-VG-FSK/active_users.json?skip_extend_session=true HTTP/1.1\\\" 200 12 \\\"https://dme.parachutehealth.com/u/s/JARX-3HLP9-RXBHM/orders/TDJ41-EJ2-VG-FSK/review\\\" \\\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36\\\" \\\"97.99.100.0\\\" \\\"dme.parachutehealth.com\\\" 0.036 0.036\\n\",\"stream\":\"stdout\",\"time\":\"2023-03-09T18:41:33.604934874Z\"}"

 

Resolution

The issue arises from a configuration switch on a docker-based cluster. The conflicting element lies in the Helm chart configuration. Analysis of the logs suggests a potential error in applying the parsing logic to that specific cluster. It's noteworthy that the problematic cluster operates on Docker, not Containerd or Crio.

Update the following parameters in the values.yaml file as below: 

logFormatType: cri

and specify the log format for "cri" logFormatType

logFormat: "%Y-%m-%dT%H:%M:%S.%N%:z"

Related Links

GitHub documentation  for reference

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release