Contents

Introduction

BackgroundInformation

Quick Start to Address Fault

Detailed Steps to Address Fault

Identify the Directory

Verifying Capacity

Clean Up Files

Fault Raised on /

Introduction

This document describes ACI fault codes F1527, F1528, F1529 and remediation steps.

Background Information

These three faults occur when the utilization of the storage capacity of a controller exceeds its threshold.

F1527 is a warning fault that occurs when the usage is greater than 75%.

F1528 is a major fault that occurs when the usage is between 85% and 90%.

F1529 is a critical fault that occurs when the usage is greater than 90%.

code : F1529

cause : equipment-full

descr : Storage unit /techsupport on node 1 with hostname rtp-aci08-apic1 mounted at /techsupport is 100% full

dn : topology/pod-1/node-1/sys/ch/p-[/techsupport]-f-[/dev/mapper/vg_ifc0-techsupport]/fault-F1529

rule : eqpt-storage-full-critical

severity : critical

Quick Start to Address Fault

1. Identify the directory that is at capacity

2. Verify that the capacity has been reached

3. Clean up files in the directory

Detailed Steps to Address Fault

Identify the Directory

The directory that the fault is raised against will be identified in the fault description.

In the two examples below you can see that the F1527 fault is raised against the /firmware directory and the F1529 fault is tied to /techsupport.

We can also see in the description that the faults are raised on node 1.

code : F1527

descr : Storage unit /firmware on Node 1 mounted at /firmware is 76% full

dn : topology/pod-1/node-1/sys/ch/p-[/firmware]-f-[/dev/mapper/vg_ifc0-firmware]/fault-F1527

code : F1529

descr : Storage unit /techsupport on node 1 with hostname rtp-aci08-apic1 mounted at /techsupport is 100% full

dn : topology/pod-1/node-1/sys/ch/p-[/techsupport]-f-[/dev/mapper/vg_ifc0-techsupport]/fault-F1529

Verifying Capacity

Once you know which directory the fault is raised on you can use the CLI to verify we are using that much drive space.

Using the command df -h we can see the available disk space for each mount.

In the table below we can see the /firmware is using 76% of its available space and /data/techsupport is using 100%

rtp-aci08-apic1# df -h

Filesystem                       Size  Used Avail Use% Mounted on

/dev/vg_ifc0/boot                 40G   13G   25G  35% /bin

/dev/mapper/vg_ifc0_ssd-data     176G  4.2G  162G   3% /var/log/dme

devtmpfs                          32G     0   32G   0% /dev

tmpfs                            4.0G  182M  3.9G   5% /dev/shm

/dev/mapper/vg_ifc0-firmware      40G   28G  9.3G  76% /firmware

/dev/mapper/vg_ifc0-scratch       40G   49M   38G   1% /home

tmpfs                             32G     0   32G   0% /sys/fs/cgroup

/dev/mapper/vg_ifc0-techsupport   40G   38G     0 100% /data/techsupport

tmpfs                             16G  592K   16G   1% /tmp

/dev/sdc1                         55M  1.2M   49M   3% /tmp/bootflash

tmpfs                            2.0G  721M  1.3G  36% /var/log/dme/log

/dev/mapper/vg_ifc0-logs          40G  5.0G   33G  14% /var/log/dme/oldlog

/dev/mapper/vg_ifc0-data2        156G   11G  137G   8% /data2

/dev/mapper/vg_ifc0-dmecores      50G   53M   47G   1% /var/log/dme/core

tmpfs                             32G  9.0G   23G  29% /var/run/utmp

Clean Up Files

After we have verified the fault condition is present we can then clean up files in the directory.

To do this you will navigate to that directory; then you can list the files by size (ls -lahS) and remove any large files (rm <fileName>) that are no longer needed.

You can then verify again with the df -h command that the space has been cleaned up.

rtp-aci08-apic1# cd /data/techsupport

rtp-aci08-apic1# ls -lahS

total 38G

-rw-r--r--  1 admin   admin    10G Aug 10 18:12 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-49UTC_logs_3of3.tgz

-rw-r--r--  1 admin   admin   9.4G Aug 10 18:13 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-79UTC_logs_3of3.tgz

-r--r-----  1 ifc     admin   3.9G Jul 24 02:05 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T01-49UTC_logs_3of3.tgz

-r--r-----  1 ifc     admin   3.7G Jul 24 01:55 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T01-49UTC_1of3.tgz

-r--r-----  1 ifc     admin   2.5G May 15 19:33 dbgexp_tsod-upgrde427sto524d_rtp-aci08-apic1_sysid-1_2023-05-15T19-24UTC_logs_3of3.tgz

-r--r-----  1 ifc     admin   2.1G May  4 19:17 dbgexp_tsod-failed_upgrade_repro_rtp-aci08-apic1_sysid-1_2023-05-04T19-08UTC_logs_3of3.tgz

-r--r-----  1 ifc     admin   1.1G Aug 10 18:04 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-08-10T17-59UTC_logs_3of3.tgz

-rw-r--r--  1 admin   admin   1.1G Aug 10 18:11 1g.img

-r--r-----  1 ifc     admin   952M May  4 19:17 dbgexp_tsod-failed_upgrade_repro_pod8-spine1_sysid-201_2023-05-04T19-08UTC_logs_3of3.tgz

-r--r-----  1 ifc     admin   946M May  3 19:44 dbgexp_tsod-failed_upgrade_repro_pod8-spine1_sysid-201_2023-05-03T19-35UTC_logs_3of3.tgz

-r--r-----  1 ifc     admin   894M May 15 19:27 dbgexp_tsod-upgrde427sto524d_rtp-aci08-apic1_sysid-1_2023-05-15T19-24UTC_1of3.tgz

-r--r-----  1 ifc     admin   892M May  4 19:12 dbgexp_tsod-failed_upgrade_repro_rtp-aci08-apic1_sysid-1_2023-05-04T19-08UTC_1of3.tgz

-r--r-----  1 ifc     admin   253M Mar 31 20:33 dbgexp_tsod-12345_12345_sysid-105_2023-03-31T20-25UTC_logs_3of3.tgz

-r--r-----  1 ifc     admin   205M Jul 18 14:40 dbgexp_coreexp-default_pod8-spine3_sysid-203_2023-07-18T14-21UTC_1689690102_0x1b01_kernel_log.0.tar.gz

-r--r-----  1 ifc     admin   141M Aug 10 18:02 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-08-10T17-59UTC_db_2of3.tgz

-r--r-----  1 ifc     admin   134M Jul 24 02:00 dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T01-49UTC_db_2of3.tgz

-r--r-----  1 ifc     admin   130M May 15 19:29 dbgexp_tsod-upgrde427sto524d_rtp-aci08-apic1_sysid-1_2023-05-15T19-24UTC_db_2of3.tgz

<snip>

rtp-aci08-apic1# rm dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-49UTC_logs_3of3.tgz

rtp-aci08-apic1# rm dbgexp_tsod-case-12345_rtp-aci08-apic1_sysid-1_2023-07-24T07-79UTC_logs_3of3.tgz

rtp-aci08-apic1# df -h | grep techsupport

/dev/mapper/vg_ifc0-techsupport   40G   18G   20G  49% /data/techsupport

Fault Raised on /

If the directory that is full is the / directory you may not be able to clean up the affected files without being root.

code : F1528

descr : Storage unit / on Node 1 with hostname rtp-aci08-apic1 mounted at / is 89% full

dn : topology/pod-1/node-1/sys/ch/p-[/]-f-[/dev/vg_ifc0/boot]/fault-F1528

When we use the df -h command here we do not see anything mounted on /.

We do use that /bin is 100% full. However, when looking at the files there we only see 606M being used not 40G.

rtp-aci08-apic1# df -h

Filesystem                       Size  Used Avail Use% Mounted on

/dev/vg_ifc0/boot                 40G   40G     0 100% /bin

/dev/mapper/vg_ifc0_ssd-data     176G  4.2G  162G   3% /var/log/dme

devtmpfs                          32G     0   32G   0% /dev

tmpfs                            4.0G  182M  3.9G   5% /dev/shm

/dev/mapper/vg_ifc0-firmware      40G   28G  9.3G  76% /firmware

/dev/mapper/vg_ifc0-scratch       40G   49M   38G   1% /home

tmpfs                             32G     0   32G   0% /sys/fs/cgroup

/dev/mapper/vg_ifc0-techsupport   40G   18G   20G  49% /data/techsupport

tmpfs                             16G  592K   16G   1% /tmp

/dev/sdc1                         55M  1.2M   49M   3% /tmp/bootflash

tmpfs                            2.0G  726M  1.3G  36% /var/log/dme/log

/dev/mapper/vg_ifc0-logs          40G  5.1G   33G  14% /var/log/dme/oldlog

/dev/mapper/vg_ifc0-data2        156G   11G  137G   8% /data2

/dev/mapper/vg_ifc0-dmecores      50G   53M   47G   1% /var/log/dme/core

tmpfs                             32G  7.1G   25G  23% /var/run/utmp

rtp-aci08-apic1# cd /bin

rtp-aci08-apic1# ls -lahS | head

total 606M

-rwxr-xr-x  1 root root  103M Jul 26 20:44 nomad

-rwxr-xr-x  1 root root   60M Mar  1  2021 podman

-rwxr-xr-x  1 root root   51M Sep  9  2020 containerd

-rwxr-xr-x  1 root root   47M Aug  4  2021 consul

-rwxr-xr-x  1 root root   32M Apr 27  2021 atomix

-rwxr-xr-x  1 root root   30M Apr 27  2021 atomix-downgrade-grub

-rwxr-xr-x  1 root root   26M Sep  9  2020 ctr

-rwxr-xr-x  1 root root   25M Feb 13  2019 etcd

-rwxr-xr-x  1 root root   21M Feb 13  2019 etcdctl

In order to see the actual files taking up the space on / we would need to access the APIC CLI with the root login.

To do this you will need to contact Cisco TAC for assistance.