Introduction
This document describes the node-exporter disk full problem noticed in a user's network.
Background
When an audit of the Cluster Manager Common Execution Environment (CEE) is performed, the audit result indicates the node-exporter disk is full.
Problem
A critical severity alert condition exists because a disk full condition is projected to occur in the next 24 hours, this alert was noticed on CEE:
" Device /dev/sda3 of node-exporter cee03/node-exporter-4dd4a4dd4a is projected to be full within the next 24 hours"
Analysis
The alert reported is on the CEE that tracks hardware issues for the rack and projects the full disk condition to occur in the next 24 hours.
cisco@deployer-cm-primary:~$ kubectl get pods -A -o wide | grep node
cee03 node-exporter-4dd4a4dd4a 1/1 Running 1 111d 10.10.1.1 deployer-cm-primary <none> <none>
root@deployer-cm-primary:/# df -h
Filesystem Size Used Avail Use% Mounted on
overlay 568G 171G 368G 32% /
tmpfs 64M 0 64M 0% /dev
tmpfs 189G 0 189G 0% /sys/fs/cgroup
tmpfs 189G 0 189G 0% /host/sys/fs/cgroup
/dev/sda1 9.8G 3.5G 5.9G 37% /host/root
udev 189G 0 189G 0% /host/root/dev
tmpfs 189G 0 189G 0% /host/root/dev/shm
tmpfs 38G 15M 38G 1% /host/root/run
tmpfs 5.0M 0 5.0M 0% /host/root/run/lock
/dev/sda3 71G 67G 435M 100% /host/root/var/log
When an audit is performed, it appears to fill up the /dev/sda3 disc.
root@deployer-cm-primary:/host/root/var/log# du -h --max-depth=1
76M ./sysstat
16K ./lost+found
4.0K ./containers
4.0K ./landscape
9.3M ./calico
1.1G ./apiserver
808K ./pods
5.6G ./journal
60G ./audit
36K ./apt
67G .
A check of the audit shows it keeps the logs and as a result, the server condition of exporter-node disk full is likely to occur.
cisco@deployer-cm-primary:~$ sudo cat /etc/audit/auditd.conf
#
# This file controls the configuration of the audit daemon
#
local_events = yes
write_logs = yes
log_file = /var/log/audit/audit.log
log_group = adm
log_format = RAW
flush = INCREMENTAL_ASYNC
freq = 50
max_log_file = 8
num_logs = 5
priority_boost = 4
disp_qos = lossy
dispatcher = /sbin/audispd
name_format = NONE
##name = mydomain
max_log_file_action = keep_logs
space_left = 75
space_left_action = email
verify_email = yes
action_mail_acct = root
admin_space_left = 50
admin_space_left_action = halt
disk_full_action = SUSPEND
disk_error_action = SUSPEND
use_libwrap = yes
##tcp_listen_port = 60
tcp_listen_queue = 5
tcp_max_per_addr = 1
##tcp_client_ports = 1024-65535
tcp_client_max_idle = 0
enable_krb5 = no
krb5_principal = auditd
##krb5_key_file = /etc/audit/audit.key
distribute_network = no
cisco@deployer-cm-primary:~$
Solution
Preform the command code listed next, on both the deployer-cm-primary and the deployer-cm-secondary to remediate the potential node-exporter disk full condition.
sudo vim /etc/audit/auditd.conf
Then, use the code listed next to change the inside file from keep_logs to rotate.
max_log_file_action = rotate
After the code is changed, restart the service.
sudo systemctl restart auditd.service
Verify the critical alert is removed.
Feedback