Introduction
This document describes ACI fault codes F1527, F1528, F1529 and remediation steps.
Background Information
These three faults occur when the utilization of the storage capacity of a controller exceeds its threshold.
F1527 is a warning fault that occurs when the usage is greater than 75%.
F1528 is a major fault that occurs when the usage is between 85% and 90%.
F1529 is a critical fault that occurs when the usage is greater than 90%.
code : F1529
cause : equipment-full
descr : Storage unit /techsupport on node 1 with hostname rtp-aci08-apic1 mounted at /techsupport is 100% full
dn : topology/pod-1/node-1/sys/ch/p-[/techsupport]-f-[/dev/mapper/vg_ifc0-techsupport]/fault-F1529
rule : eqpt-storage-full-critical
severity : critical
Quick Start to Address Fault
- Identify the directory that is at capacity
- Verify that the capacity has been reached
- Clean up files in the directory
Detailed Steps to Address Fault
Identify the Directory
The directory that the fault is raised against will be identified in the fault description.
In the two examples below you can see that the F1527 fault is raised against the /firmware directory and the F1529 fault is tied to /techsupport.
We can also see in the description that the faults are raised on node 1.
code : F1527
descr : Storage unit /firmware on Node 1 mounted at /firmware is 76% full
dn : topology/pod-1/node-1/sys/ch/p-[/firmware]-f-[/dev/mapper/vg_ifc0-firmware]/fault-F1527
code : F1529
descr : Storage unit /techsupport on node 1 with hostname rtp-aci08-apic1 mounted at /techsupport is 100% full
dn : topology/pod-1/node-1/sys/ch/p-[/techsupport]-f-[/dev/mapper/vg_ifc0-techsupport]/fault-F1529
Verifying Capacity
Once you know which directory the fault is raised on you can use the CLI to verify we are using that much drive space.
Using the command df -h we can see the available disk space for each mount.
In the table below we can see the /firmware is using 76% of its available space and /data/techsupport is using 100%
rtp-aci08-apic1# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vg_ifc0/boot 40G 13G 25G 35% /bin
/dev/mapper/vg_ifc0_ssd-data 176G 4.2G 162G 3% /var/log/dme
devtmpfs 32G 0 32G 0% /dev
tmpfs 4.0G 182M 3.9G 5% /dev/shm
/dev/mapper/vg_ifc0-firmware 40G 28G 9.3G 76% /firmware
/dev/mapper/vg_ifc0-scratch 40G 49M 38G 1% /home
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/vg_ifc0-techsupport 40G 38G 0 100% /data/techsupport
tmpfs 16G 592K 16G 1% /tmp
/dev/sdc1 55M 1.2M 49M 3% /tmp/bootflash
tmpfs 2.0G 721M 1.3G 36% /var/log/dme/log
/dev/mapper/vg_ifc0-logs 40G 5.0G 33G 14% /var/log/dme/oldlog
/dev/mapper/vg_ifc0-data2 156G 11G 137G 8% /data2
/dev/mapper/vg_ifc0-dmecores 50G 53M 47G 1% /var/log/dme/core
tmpfs 32G 9.0G 23G 29% /var/run/utmp
Clean Up Files
After we have verified the fault condition is present we can then clean up files in the directory.
To do this you will navigate to that directory; then you can list the files by size (ls -lahS) and remove any large files (rm <fileName>) that are no longer needed.
You can then verify again with the df -h command that the space has been cleaned up.
rtp-aci08-apic1# cd /data/techsupport
rtp-aci08-apic1# ls -lahS
total 38G
-rw-r--r-- 1 admin admin 10G Aug 10 18:12 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-49UTC_logs_3of3.tgz
-rw-r--r-- 1 admin admin 9.4G Aug 10 18:13 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-79UTC_logs_3of3.tgz
-r--r----- 1 ifc admin 3.9G Jul 24 02:05 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T01-49UTC_logs_3of3.tgz
-r--r----- 1 ifc admin 3.7G Jul 24 01:55 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T01-49UTC_1of3.tgz
-r--r----- 1 ifc admin 2.5G May 15 19:33 dbgexp_tsod-upgrde427sto524d_rtp-aci08-apic1_sysid-1_2023-05-15T19-24UTC_logs_3of3.tgz
-r--r----- 1 ifc admin 2.1G May 4 19:17 dbgexp_tsod-failed_upgrade_repro_rtp-aci08-apic1_sysid-1_2023-05-04T19-08UTC_logs_3of3.tgz
-r--r----- 1 ifc admin 1.1G Aug 10 18:04 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-08-10T17-59UTC_logs_3of3.tgz
-rw-r--r-- 1 admin admin 1.1G Aug 10 18:11 1g.img
-r--r----- 1 ifc admin 952M May 4 19:17 dbgexp_tsod-failed_upgrade_repro_pod8-spine1_sysid-201_2023-05-04T19-08UTC_logs_3of3.tgz
-r--r----- 1 ifc admin 946M May 3 19:44 dbgexp_tsod-failed_upgrade_repro_pod8-spine1_sysid-201_2023-05-03T19-35UTC_logs_3of3.tgz
-r--r----- 1 ifc admin 894M May 15 19:27 dbgexp_tsod-upgrde427sto524d_rtp-aci08-apic1_sysid-1_2023-05-15T19-24UTC_1of3.tgz
-r--r----- 1 ifc admin 892M May 4 19:12 dbgexp_tsod-failed_upgrade_repro_rtp-aci08-apic1_sysid-1_2023-05-04T19-08UTC_1of3.tgz
-r--r----- 1 ifc admin 253M Mar 31 20:33 dbgexp_tsod-12345_12345_sysid-105_2023-03-31T20-25UTC_logs_3of3.tgz
-r--r----- 1 ifc admin 205M Jul 18 14:40 dbgexp_coreexp-default_pod8-spine3_sysid-203_2023-07-18T14-21UTC_1689690102_0x1b01_kernel_log.0.tar.gz
-r--r----- 1 ifc admin 141M Aug 10 18:02 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-08-10T17-59UTC_db_2of3.tgz
-r--r----- 1 ifc admin 134M Jul 24 02:00 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T01-49UTC_db_2of3.tgz
-r--r----- 1 ifc admin 130M May 15 19:29 dbgexp_tsod-upgrde427sto524d_rtp-aci08-apic1_sysid-1_2023-05-15T19-24UTC_db_2of3.tgz
<snip>
rtp-aci08-apic1# rm dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-49UTC_logs_3of3.tgz
rtp-aci08-apic1# rm dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-79UTC_logs_3of3.tgz
rtp-aci08-apic1# df -h | grep techsupport
/dev/mapper/vg_ifc0-techsupport 40G 18G 20G 49% /data/techsupport
Fault Raised on /
If the directory that is full is the / directory you may not be able to clean up the affected files without being root.
code : F1528
descr : Storage unit / on Node 1 with hostname rtp-aci08-apic1 mounted at / is 89% full
dn : topology/pod-1/node-1/sys/ch/p-[/]-f-[/dev/vg_ifc0/boot]/fault-F1528
When we use the df -h command here we do not see anything mounted on /.
We do use that /bin is 100% full. However, when looking at the files there we only see 606M being used not 40G.
rtp-aci08-apic1# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vg_ifc0/boot 40G 40G 0 100% /bin
/dev/mapper/vg_ifc0_ssd-data 176G 4.2G 162G 3% /var/log/dme
devtmpfs 32G 0 32G 0% /dev
tmpfs 4.0G 182M 3.9G 5% /dev/shm
/dev/mapper/vg_ifc0-firmware 40G 28G 9.3G 76% /firmware
/dev/mapper/vg_ifc0-scratch 40G 49M 38G 1% /home
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/vg_ifc0-techsupport 40G 18G 20G 49% /data/techsupport
tmpfs 16G 592K 16G 1% /tmp
/dev/sdc1 55M 1.2M 49M 3% /tmp/bootflash
tmpfs 2.0G 726M 1.3G 36% /var/log/dme/log
/dev/mapper/vg_ifc0-logs 40G 5.1G 33G 14% /var/log/dme/oldlog
/dev/mapper/vg_ifc0-data2 156G 11G 137G 8% /data2
/dev/mapper/vg_ifc0-dmecores 50G 53M 47G 1% /var/log/dme/core
tmpfs 32G 7.1G 25G 23% /var/run/utmp
rtp-aci08-apic1# cd /bin
rtp-aci08-apic1# ls -lahS | head
total 606M
-rwxr-xr-x 1 root root 103M Jul 26 20:44 nomad
-rwxr-xr-x 1 root root 60M Mar 1 2021 podman
-rwxr-xr-x 1 root root 51M Sep 9 2020 containerd
-rwxr-xr-x 1 root root 47M Aug 4 2021 consul
-rwxr-xr-x 1 root root 32M Apr 27 2021 atomix
-rwxr-xr-x 1 root root 30M Apr 27 2021 atomix-downgrade-grub
-rwxr-xr-x 1 root root 26M Sep 9 2020 ctr
-rwxr-xr-x 1 root root 25M Feb 13 2019 etcd
-rwxr-xr-x 1 root root 21M Feb 13 2019 etcdctl
In order to see the actual files taking up the space on / we would need to access the APIC CLI with the root login.
To do this you will need to contact Cisco TAC for assistance.
Feedback