Field Notice: FN74383 - Cisco Access Points Running Cisco IOS XE Releases 17.12.4, 17.12.5, 17.12.6 or 17.12.6a May Run Out of Flash Space, Preventing Upgrades - Software Upgrade Recommended

Certain Cisco Access Points (APs) may fail to download new software images or Access Point Service Packs (APSPs) if they are currently running, or have previously run, Cisco IOS XE Software releases 17.12.4, 17.12.5, 17.12.6, or 17.12.6a. For a list of affected APs, see the Products Affected section of this field notice.

A library update causes a specific file on the flash memory of an AP to grow by approximately 5MB per day. Over time, this file could consume a big portion of the available flash space. The longer an AP runs the affected software, the higher the probability that a software download will fail due to insufficient disk space.

This issue does not affect Cisco Catalyst IW9167 Heavy Duty Series products—including Cisco Catalyst IW9167E Heavy Duty Access Points for Hazardous Locations (HazLoc) and Stadium variants—that are running Cisco Unified Industrial Wireless (UIW) Software in either Ultra-Reliable Wireless Backhaul (URWB) or workgroup bridge (WGB) mode. However, these products can switch between operating modes, and the UIW and AP software releases are managed independently. If an IW9167 device has ever operated in AP mode using an affected software release, it may be affected.

A library update issue was introduced in Cisco IOS XE Release 17.12.4 and also affects Cisco IOS XE Releases 17.12.5, 17.12.6, and 17.12.6a. This issue causes the AP to generate a log file named cnssdaemon.log within the flash memory.

This flash memory is where partition 2 (part2) of the AP resides. This issue will affect an AP if the active boot partition is partition 1 (part1). The cnssdaemon.log file grows continuously and cannot be deleted through the AP CLI.

These symptoms can be identified both before and after an upgrade attempt. Cisco recommends performing upgrade prechecks to prevent this issue.

Upgrade Prechecks (Recommended)

Performing validation before an upgrade is the safest approach. This can be done manually or through automation.

Automated Procedure (Preferred)

Cisco offers the WLANPoller tool, which automates command execution for validation across multiple APs. WLANPoller collects and analyzes the outputs of commands to verify the status of APs. If any APs require recovery, WLANPoller provides the best recovery path.

Download WLANPoller tool here: https://developer.cisco.com/docs/wireless-troubleshooting-tools/wlan-poller-wlan-poller/

For instructions, see the Automated Precheck section of the Validate and Recover Catalyst APs on 17.12 Impacted by Upgrade Failure TechNote.

Manual Procedure

Use the show boot command to check the active boot partition.

• If part1 is active, the new code or APSP will attempt to install on part2, where the cnssdaemon.log file is stored. Use the show filesystems command to see the available space on part2. If part2 has less than 85M available, the upgrade will likely fail.
• If part2 is active, the new code or APSP will be installed on part1. In most situations, the installation will succeed.

AP commands for manual precheck:

• show boot: Identifies the active partition (part1 or part2)
• show filesystems: Confirms the available space on part2
• show flash | in cnss: Displays the cnssdaemon.log file and its current size
• show image integrity: Verifies the health of the images currently residing in both partitions

The following command outputs serve as an example for reference only:

AP# show boot
--- Boot Variable Table ---
BOOT path-list: part1
AP# show flash | in cnss
-rw-r--r-- 1 root root 151889248 Mar 9 03:18 cnssdaemon.log
AP# show filesystems
Filesystem                Size      Used Available Use% Mounted on
devtmpfs                876.4M         0    876.4M   0% /dev
/sysroot                879.3M    223.9M    655.4M  25% /
tmpfs                     1.0M     56.0K    968.0K   5% /dev/shm
tmpfs                   879.3M         0    879.3M   0% /run
tmpfs                   879.3M         0    879.3M   0% /sys/fs/cgroup
/dev/ubivol/part1       278.3M     95.8M    177.8M  35% /part1
/dev/ubivol/part2       387.8M    290.5M     92.6M  76% /part2

For more information, see the Manual Precheck section of the Validate and Recover Catalyst APs on 17.12 Impacted by Upgrade Failure TechNote.

Post-Upgrade Symptoms

If an upgrade is attempted without performing the upgrade precheck, the AP may fail due to insufficient space, potentially leading to a bootloop. Look for the following log errors from the APs.

Insufficient space errors:

capwapd: opening /storage/base_capwap_cfg_info.tmp_9410 failed!: No space left on device
upgrade.sh: /tmp space: ERROR! available 56608, required 122880
sync_log: no space to sync new syslogs: needed 1024, free 8
capwap-upgrade script returned failure when calling PREDOWNLOAD, return value 1

Bootloop/signature Errors:

verify signature failed for /bootpart/part2/ramfs_data_cisco.squashfs

For a successful upgrade, Cisco recommends the following:

• Run the precheck as close to the scheduled maintenance window as possible. Because the cnssdaemon.log file grows daily, Cisco recommends performing these checks no more than 48 hours before the maintenance window to ensure the most accurate results. Follow the instructions in the Upgrade Prechecks section of the Validate and Recover Catalyst APs on 17.12 Impacted by Upgrade Failure TechNote.
• If an AP requires recovery, see the Recovery section of the Validate and Recover Catalyst APs on 17.12 Impacted by Upgrade Failure TechNote. For recovery, Cisco offer automation tools such as WLANPoller and RADkit.

If issues occur during the prechecks, or if the recovery process indicates that an AP requires manual recovery, open a case with the Cisco Technical Assistance Center (TAC).