SC4S not sending logs to Splunk Cloud

Applicability

Description

Using SC4S to collect syslogs. However, SC4S stopped sending its logs to Splunk Cloud via Universal forwarder.

The below search in Cloud Search Heads yields no results:

index="main"

Investigation

Data for index=main data is coming from 3 different hosts: 2 Universal forwarders)and via HEC input (http-inputs-xxxxxxx.splunkcloud.com).

Steps to follow:
1. Confirm the HEC input is connected via the below method:

 Run the below command in SC4S syslog container:

/opt/sc4s$ docker logs SC4S

 If the instance is using Podman instead of Docker, run the below command:

[instance_name]# podman logs SC4S

2. Notice that the inputs are successful along with the below errors:

SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index="main" for sourcetype=xxxxxxx

SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=main for sourcetype=xxxxxxx

ERRORS:

WARNING: capacity-bytes() has changed since the last syslog-ng run. syslog-ng currently does not support changing the capacity-bytes() of existing disk-queues. Continuing with the old one; filename='/var/lib/syslog-ng/syslog-ng-00000.qf', active_old_capacity_bytes='53160876800', ignored_new_capacity_bytes='149797221529'

3. Observe the qfiles files getting filed up with its capacity on the SC4S.

On the SC4S perform the below steps to see Syslog-ng disk queue files: 
 

1. Identify the active queue files. 

   The syslog-ng PE application keeps track of active disk queue files, and the internal state of its source drivers in the syslog-ng.persist file. While running, syslog-ng PE uses the mmap() system call to map the file's contents into physical memory. This means that the actual contents of the file may not always contain the up-to-date internal state of syslog-ng PE. For this reason, while working with the syslog-ng.persist file, stop syslog-ng PE.

   The following command lists the destinations and the related queue files.

   # /opt/syslog-ng/bin/persist-tool dump /opt/syslog-ng/var/syslog-ng.persist | fgrep qfile

   The output if this command is similar to the following:

   afsocket_dd_qfile(stream,127.0.0.1:601) = { "queue_file": "\/\/syslog-ng-00001.rqf" }

2. Identify which queue files hold valid data. 

   To identify which queue files hold unsent data, use the following two commands for the disk queue files (the example shows a single file called syslog-ng-00000.rqf):

   # /opt/syslog-ng/bin/dqtool info syslog-ng-00000.rqf
   # /opt/syslog-ng/bin/dqtool cat syslog-ng-00000.rqf
   

Resolution

To make changes of qfiles on SC4S container, please refer the below documents:

1. Identify the active queue files

2. Identify which queue files still hold valid data

3. Configure a separate syslog instance to send queue files to the processing application
   
   Post running the above steps, it was determined that old capacity files are filling up. Finally, old capacity bytes were relocated to another mount file and fixed the problem by freeing up disk space.

Related Links

how-to-get-information-of-disk-buffer-files

sending-out-messages-stuck-in-syslog-ng-disk-queue-files