Splunk cannot forward syslog data - Disk Full - Logrotate is not configured.

Applicability

Splunk Forwarder rsyslog syslog-ng logrotate

Description

Splunk is running without error, but log data from file monitor inputs for syslog are not displaying new data in Splunk. _internal logs can be seen from the host collecting syslog and the files in the monitor path are readable by Splunk. 

Investigation

Reviewing the last logs in Splunk showed successful connections to the indexer. This means no data processing queues were blocked, and no problems with sending data can be identified. A successful connection to the indexer looks like this:

01-03-2024 12:02:24.398 -0500 INFO  AutoLoadBalancedConnectionStrategy [7794 TcpOutEloop] - Found currently active indexer. Connected to idx=X.X.X.X:9997:0, reuse=1.

Search for the following to look for successful connections. Be sure to specify a host. The data is logged in splunkd.log:

index=_internal sourcetype=splunkd host=<hostInQuestion> "Connected to"

Splunk is working fine, and in this scenario, there are no errors in Splunk preventing it from sending data. Permissions could prevent the files from being read, but trying to read them as the user Splunk is running as is successful. The Splunk user can read the files. The actual problem in this case is disk space. Running a command like "df -h" on the syslog receiver will show a full disk. Syslog cannot write to disk, so Splunk is not reading new data. This can be confirmed by searching the data in Splunk and then tailing the log path in-question on the forwarder to see if it is the same event. The source field in your search will have this information as well as the monitor input for the data in inputs.conf. As long as the event was not sent to the nullqueue per any custom configurations, the data should be consistent. 

If there are a large number of errors indicating that the cooked connection timed out or that it is putting indexer IP addresses in detention, this guide will not solve the problem. Data queues must be free and clear of congestion on the indexers and forwarders. If this search returns results, please fix forwarder congestion before using this guide:

index=_internal sourcetype=splunkd "cooked connection timed out"

The df command will display the disks available and their space metrics. The -h makes df results human-readable:

]# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             3.7G     0  3.7G   0% /dev
tmpfs                3.8G     0  3.8G   0% /dev/shm
tmpfs                3.8G   81M  3.7G   3% /run
tmpfs                3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/mapper/cs-root  191G   94G   98G  50% /
/dev/sda5           1014M  344M  671M  34% /boot
/dev/sda1            599M  7.3M  592M   2% /boot/efi
/dev/mapper/cs-home   24G  2.5G   21G  11% /home
/dev/mapper/splunk   150G  150G   10K 100% /opt/splunk/syslog
tmpfs                762M     0  762M   0% /run/user/1000

Resolution

DISCLAIMER: The following solution does not include Splunk components. All commands and changes should be verified to the version of the command on the operating system hosting Splunk. All logrotate examples are only examples and need to be configured specifically to the needs of the hardware and the organization. Linux installations have manpages installed that can be reviewed by pretending man to the command:

man <linuxCommand>man findman logrotate

Always be sure to use the corresponding documentation for the operating system installed.

Disk space must be recovered to restore log collection by syslog. If Logrotate is already configured, more disk space will be needed on the host. If Logrotate is not configured to remove Splunk files, this will need to be configured.

Logrotate is the most common way to manage this. An easy hack is to copy another logrotate file on your system and make it work for Splunk. Below is an example of a Logrotate configuration that will rotate and keep two files if they are bigger than 1 megabyte. It will copy the file and truncate the original, and it will kill syslog-ng after performing the rotation. It will do this every hour:

/opt/splunk/syslog/*/syslog.log {
hourly
missingok
rotate 2
copytruncate
minsize 1M
postrotate
/usr/bin/systemctl -s HUP kill syslog-ng.service >/dev/null 2>&1 || true
endscript
}

The postrotate command is for syslog-ng. If another Syslog collection is being used, adjust the command to cater to the Syslog collector installed. This is an operating system-specific area, and there are a lot of forums and other information online about different ways to approach logrotate. 

What Can Go Wrong?
Disk space can run out again if there is not enough available for log rotate and syslog to work. Just because data is flowing again does not mean there is sufficient disk space. Pay attention to how many rotated files are kept and how often logrotate runs to determine the needed space. More logs left on the syslog collector mean an easier/greater chance of recovering data that was not indexed. If Splunk goes down, syslog can continue to collect data. When Splunk recovers, it can send the data it didn't already send. Disaster recovery as well as how logrotate is configured will need to be considered to determine the required disk space. If logrotate was never set up, all of the syslog files are probably in single files that are very large. Confirm the first and last log in those files are in Splunk. If they are, remove the file to recover space.

Once the above is done to all data sources, get logrotate configured to run. Time is important here. If it takes a week to clear up the directories, the log files may get bigger than desirable for rotation again. Get this done fast and as soon as possible. A suggestion to speed things up is to use find to locate files to delete by date. 

find . -name 'syslog.log' -mtime +60 -exec rm -fv {} \;

This will remove files more than 60 days old in verbose mode. Test it by excluding " -exec rm -fv {} \;" from the command. Adjust mtime to fit the needs of the environment (60 days is a lot to hold onto in a lot of cases). mtime stands for last modified time.

Once disk space is sufficient for syslog to save data, data flow should resume. After logrotate runs at least as many times as needed to satisfy all of the criteria in the logrotate script the amount of times as the value in "rotate", the exact amount of disk space required will be known. It is fairly easy to estimate, and the rotated files can be packaged in gzip format by adding the word "compress" to the configuration file above on a new line between the curly brackets and before postrotate. Please keep in mind that spikes and increased trends in data may require more disk space in the future.

What Else Can Go Wrong?
Make sure that the rotated files are not collectible by Splunk, or there will be duplicate data and the potential for license violations. The example below will collect syslog.log from the directory, but it will not collect syslog.log.1, syslog.log.2, or syslog.log.gz. 

[monitor:///opt/splunk/syslog/*/syslog.log]
host =
host_segment = 4
sourcetype = ubnt
index = ubqt

Wildcarding the file extension or the whole last directory can lead to duplicate data and potential license violations. Also, please be certain to check with Linux system administrators and company policy before making any changes to logrotate and the retention of log data. 

Related Links

Redhat: setting-logrotate

logrotate manpage

Find Command

Revision History

Revision	Publish Date	Comments
1.0	29-Sep-2026	Initial Release