Validate and Recover Catalyst APs on 17.12 Impacted by Upgrade Failure

Introduction

This document describes the recovery procedure when when APs are affected by Cisco bug ID CSCwf25731  and CSCwf37271.

Context

Upgrades or APSPs applied to systems currently running 17.12.4/5/6/6a or previously ran these versions for considerable amount of time can cause affected AP models (please check Affected Access Points list below) to enter into boot loop under certain conditions, triggered by image install failure due to insufficient disk space on AP storage. While this doesn't affect daily operations or SMUs, it is a critical risk during ISSU, full controller code upgrades or APSP installs since these procedures involve Access Points image upgrades.

Additional process need to be followed which requires mandatory upgrade prechecks steps mentioned in this document, as no configuration workaround exists.

To resolve this, you must install the specific APSP version fix (shown in the Fixed Codes table below) in the WLC before the APs attempt to upgrade, or use the cleanup APSP (available for 17.15.4d and 17.18.2) if you have already moved to a later release but ran any of the affected versions. Even if you are unsure of your system's history, it is highly recommended to perform storage checks before any upgrade or APSP install if the affected versions were ever present in your environment.

Root Cause Details

APs running 17.12.4 through 17.12.6a are affected by a library bug that generates a persistent log file: /storage/cnssdaemon.log. This file grows by 5MB daily and is not cleared by reboots, eventually exhausting the device's storage space. Consequently, if the AP is running from boot partition 1 (Part1), and boot partition 2 (Part2) is full due to the log file, the upgrade process will fail because it cannot store the new image in boot partition 2, potentially leading to a boot loop. To prevent this, you must clear the storage or ensure the AP has booted from partition 2 before attempting any further upgrades by following the instructions in this document.

Affected Codes and Access Points

Affected Access Points

The following Access Point models are the only ones susceptible to this issue. If your network does not utilize any of these specific models, your environment is not affected and no further action is required.

• Catalyst 9124 (I/D/E)
• Catalyst 9130 (I/E)
• Catalyst 9136I
• Catalyst 9162I
• Catalyst 9163E
• Catalyst 9164I
• Catalyst 9166 (I/D1)
• Catalyst IW9167 (I/E)

Affected Codes

To verify this across your network, execute the following command from all the WLCs and verify if the code present in your WLCs is listed in the table below.

#show version | in Version

Alternatively, you can do the same from the APs. Check the output to see if either the Primary or Backup image is running an affected image from the ones listed in the table.

#show version | in Image

Note: Note: In general, if the network is not running and has not run 17.12.4, 17.12.5, 17.12.6/6a in the past, the issue is not applicable.

Fixed Codes

The following table lists the WLC software versions and their corresponding APSPs (Access Point Service Packs) that contain the fix for this bug. Please note that for the versions listed below, the fix is currently available only via APSP installation at the time of this writing.

Upgrade Path and Bug Applicability

Use the table below to determine if your current WLCs and APs software versions apply for this bug and what upgrade steps to take. If your current deployment matches any of these releases in the Bug applicable and upgrade path table, you must perform the storage prechecks explained in the Upgrade Prechecks section before attempting any further upgrades.

Bug not applicable and upgrade path

The following table shows if your WLCs and APs do not apply for this bug:

Bug applicable and upgrade path

The following table shows if your WLCs and APs do apply for this bug:

Upgrade Prechecks

If your environment is affected by this bug, follow these mandatory steps to ensure a safe recovery and upgrade:

1. Identify: Execute the manual or automated pre-checks to identify which specific Access Points are applicable for the bug. Automated precheck is highly recommended.
2. Recover: For any APs flagged, follow the recovery procedures mentioned in the Recovery section.
3. Verify: Re-run the prechecks to confirm all devices are healthy and the storage issue is resolved.
4. Upgrade: Proceed with the upgrade to the fixed APSPs listed in the Fixed Versions Table.

Manual Precheck

1. From the WLC you can check the Primary and Backup image columns to confirm if your Access Points are running any of the impacted releases (check the Affected Codes section above).

9800-l#show ap image
Total number of APs  : 4

Number of APs
        Initiated                  : 0
        Downloading                : 0
        Predownloading             : 0
        Completed downloading      : 0
        Completed predownloading   : 0
        Not Supported              : 0
        Failed to Predownload      : 0
        Predownload in progress    : No

AP Name             Primary Image          Backup Image            Predownload Status   Predownload Version  Next Retry Time   Retry Count   Method
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Ap117.12.5.41        17.12.4.201                None                  0.0.0.0             N/A                 0           N/A
Ap217.12.5.41        17.12.4.201                None                  0.0.0.0             N/A                 0           N/A
Ap317.12.5.41        17.12.4.201                None                  0.0.0.0             N/A                 0           N/A
Ap417.12.5.41        17.12.4.201                None                  0.0.0.0             N/A                 0           N/A

You can also perform a similar verification directly at the AP level by running the following command to check both image partitions:

AP# show version
AP Running Image     :17.12.5.41
Primary Boot Image   : 17.12.5.41
Backup Boot Image    : 17.12.5.209
Primary Boot Image Hash: 93ef1e703a5e7c5a4f97b8f59b220f52d94dd17c527868582c0048caad6397a9f3526c644f94a52bb70a104385690065ad0d652aa3fed607f24920d7e5ed5b5c
Backup Boot Image Hash: 4bbe4a0d9edc3cad938a7de399d3c2e08634643a2623bae65973ef00deb154b8eb7c7917eeecdd46e3e2ddc7be80139475e19fb3040b08aa715de196a733252b
1 Multigigabit Ethernet interfaces

Verify the active boot partition to ensure the backup image is on part2. Use the "show boot" command and confirm that the "BOOT path-list" points to part1; this indicates that the AP is currently running from the primary partition and attempting to upgrade into the secondary part which could trigger the problem.

AP# show boot
--- Boot Variable Table ---
BOOT path-list: part1
Console Baudrate: 9600 Enable Break:

2. Verify current filesystem usage by checking the /dev/ubivol/part2 partition. If the "Use%" is close to 100%, the partition is exhausted, which will cause the upgrade to fail and potentially trigger a boot loop.

AP# show filesystems
Filesystem                Size      Used Available Use% Mounted on
devtmpfs                880.9M         0    880.9M   0% /dev
/sysroot                883.8M    219.6M    664.1M  25% /
tmpfs                     1.0M     56.0K    968.0K   5% /dev/shm
tmpfs                   883.8M         0    883.8M   0% /run
tmpfs                   883.8M         0    883.8M   0% /sys/fs/cgroup
/dev/ubivol/part1       372.1M     79.7M    292.4M  21% /part1
/dev/ubivol/part2       520.1M    291.3M    228.9M  56% /part2

3. Verify the image integrity for both partitions to ensure they are not corrupted. All fields for both Primary and Backup images must display a status of "Good"; if any field shows otherwise, stop the process and open a TAC case immediately.

AP# show image integrity
/part1(Backup) 17.12.5.209
    part.bin : Good
    ramfs_data_cisco.squashfs : Good
    iox.tar.gz : Good
/part2(primary) 17.12.5.41
    part.bin : Good
    ramfs_data_cisco.squashfs : Good
    iox.tar.gz : Good

Automated Precheck

For the automated pre-check, utilize the WLAN Poller tool. This tool allows you to run the needed commands across all your Access Points simultaneously to identify impacted Access Points; you can download it directly from the following link: https://developer.cisco.com/docs/wireless-troubleshooting-tools/wlan-poller-wlan-poller/#wlan-poller

Steps:

1. Extraction – Extract the WLAN Poller files to your preferred directory.

2. Configuration – Update the "config.ini" file with the following parameters, ensuring you input your specific credentials and Controller IP Address.

wlc_type: 2
mode: ssh
ap_mode: ssh

; set global WLC credentials
wlc_user: <username>
wlc_pasw: <password>
wlc_enable: <enable_password>

; set global AP credentials
ap_user: <ap_username>
ap_pasw: <ap_password>
ap_enable: <ap_enable_password>

[WLC-1]
active: True
ipaddr: <Controller_IP>
mode: ssh

3. Command List Preparation – Comment out (add the hash symbol "#") the default contents in "cmdlist_cos" and "cmdlist_cos_qca", then add the following commands to both files.

# snippet to download the Debug image on COS APs
# show version | in Compiled
# archive download-sw /reload tftp://<tftp server ip-address>/<TAR filename>
# 
show clock
show version
show flash
show flash | i cnssdaemon.log
show boot
show filesystems
show image integrity

4. Execution – Run the tool using ".\wlanpoller.exe". The tool will SSH into all Access Points and collect the command outputs.

5. Data Retrieval – Once complete, navigate to the newly created /data folder. Follow the subdirectories to the final folder containing the individual AP output files.

6. Analysis – Download the "ap_detection_script.py" from the official Box link, place it in the "data" folder, and execute it.

7. Review Results – Open the generated "Status_check_results.log" to view the list of APs in a problematic state. These devices require recovery steps (explained in the Recovery section below) before you proceed with the upgrade, here an explanation of how to interpretate the results:

By design, APs can boot from either Partition 1 or Partition 2. When one partition is active, the other is used for image or APSP downloads. The logical storage partition is permanently mapped to Partition 2 and cannot be changed. This issue only impacts Access Points currently booting from Partition 1. You can verify this by checking the "current_boot_partition_check" column to verify what is the current partition used by the APs. Example:

From the example above we can conclude that from the 3 APs:

• AP name: Test AP1 (Action Required): If an AP shows part1 "True Susceptible" in the "current_boot_partition_check" column, you must then check the "part2_mem_utilization_check" column. If that column also shows "True Susceptible", the AP is impacted.

  • Example: Test AP1 is impacted (boot in Part1 and Part2 available space: 51.9 MB) and requires recovery.

• AP Name: Test AP2 (Partition 1): If an AP shows part1 "True Susceptible" but the "part2_mem_utilization_check" shows "False Not Susceptible", the AP is safe.

  • Example: Test AP2 is not impacted as it has enough space in partition 2 (part2), however, it is recommended to install the APSP to take care of future issues since the cnssdaemon.log file continues existing in the AP.

• AP Name: Test AP3 (Partition 2): If an AP shows part2 "False Not Susceptible" in the "current_boot_partition_check" column, it is not impacted. No further checks are necessary.

Note: Note: The numerical value appearing in the column "part2_mem_utilization_check" next to the "True/False Susceptible" status represents the amount of available space in Partition 2.

• Any Access Point: If any AP shows "Image Integrity Check: Failed" under the "image_integrity_check" column please proceed opening a TAC case.

Recovery

Based on each AP's specific state, the script will recommend the most efficient recovery method. Follow the detailed steps below for the identified affected APs:

AP Image partition swap

1. Isolate APs – make sure the APs have no connectivity with the WLC:
   1. Make sure SSH is enabled in the APs join profile and APs are SSH accessible (or use console)
   2. Ensure the APs do not have connectivity with the WLC, but, you still have SSH access to the APs. This can be accomplished by having an ACL in the gateway or moving the APs to an isolated Vlan. If the APs get access to the WLC the AP could revert back to boot Part1 and will be back to the affected state.

2. Configure Boot Path – On the impacted APs, set the boot path to partition 2:

AP# config boot path 2

3. Reboot – Restart the AP to load the image from partition 2:

AP# reload

4. Upgrade – Install in the WLC the needed APSP in your current code, or, if you were going to upgrade the code move to a new code and make sure APSP is installed as well.

5. Reconnect – Once the controller upgrade is complete, remove the communication stopper between the AP and the WLC. The AP will join the WLC and automatically download the new, fixed image in boot Part1.

6. Double Check – After upgrading to a fixed release, verify both partitions of the APs to ensure the backup slot does not still contain the buggy image.

7. Maintenance – To maintain long-term stability and prevent future boot loops, we recommend overwriting the backup partition with a known good image. For smaller groups, use archive "download-sw" directly on the AP; for larger deployments, perform a WLC AP pre-download to update the backup partition without triggering the AP image activation.

AP shell acces recovery

Technical Assistance Center (TAC) can perform manual recovery by clearing the cnssdaemon.log file directly from shell on affected Access Points. Depending on the scale of the impact, there are two methods mentioned below:

• Small Number of Affected APs: For a small amount of affected APs, TAC can proceed using one of two approaches:

  • Manual shell access: Access each Access Point individually via shell to manually clear the log file.

  • Automated (bulk) shell access: Use the RADKit tool to automate the cleanup process to all the affected APs.

• Large Number of Affected APs: TAC shall use Radkit tool, this allows for bulk shell access to all affected APs simultaneously to execute the cleanup process efficiently.

Note: We recommend using RADKit for the AP shell access recovery procedure to ensure efficiency and consistency.

Note: To download Radkit please use this link: Radkit Download and download the latest version. To Install Radkit please follow this video: Radkit Installation

When to open a TAC case

Open a TAC case immediately if any of the following conditions occur:

• Failed Recovery: The AP Image AP Image Partition Swap procedure fails or cannot be implemented in your environment.
• Integrity Issues: Manual or automated prechecks return an "Image Integrity Check: Failed" status for any AP.
• Storage Exhaustion: If after upgrade/APSP install the partition "/dev/ubivol/part2" shows still a critically high usage.

Cisco TAC can access the AP shell to manually clear the cnssdaemon.log file and perform advanced recovery actions to restore your devices.

FAQs

Q: Does this issue apply only to full code upgrades, or does it also affect APSP installations?

A: This issue affects both scenarios. If your environment meets the criteria for this bug, the issue can occur during a full code upgrade or APSP installation (including the APSP with the bug fix). Please complete the Upgrade prechecks section to determine if you need to follow the Recovery steps before applying any updates or APSPs.

Q: My WLC and APs are on 17.9.x (or earlier) and I need to upgrade to 17.12.x, What should I do?

A: You can perform a direct upgrade from 17.9.x to 17.12.x. However, if your AP models are susceptible to this bug, ensure you install the recommended APSP immediately following the upgrade.

Q: My WLC and APs are on 17.9.x (or earlier) and I need to upgrade to 17.15.x or higher.

A: There are two possible scenarios:

• Direct Upgrade: If your environment allows a direct upgrade (please verify via the Release Notes for your target code), proceed with the upgrade and then install the APSP for that target code.
• Intermediate Upgrade: If you must follow an upgrade path (e.g., 17.9.x → 17.12.x → 17.15.x), we recommend completing the entire sequence to 17.15.x within the same day. Because the cnssdaemon.log file grows by 5 MB daily, completing the upgrade quickly prevents the file from reaching a critical size. If a same-day upgrade is not possible, you must install the APSP at the 17.12.x stage before eventually proceeding to 17.15.x and installing its respective APSP.

Q: I am already on 17.15.x. Does that mean I am not affected by this bug?

A: Not necessarily. If your APs previously ran version 17.12.4, 17.12.5, or 17.12.6/6a (on 9800-L/40/80/CL), the problematic log file may have already been generated and remains in storage. We strongly recommend following the Upgrade Prechecks section to ensure any residual files are cleaned up.

Q: I use the 9800-M, 9800-H1, or 9800-H2 platforms, which were first supported in 17.15, Am I affected?

A: There are two possible scenarios:

• First ever joined WLC: If your APs joined a 9800-M/H1/H2 as their first ever controller, you are not impacted.
• Previous WLCs joined: If those APs were previously joined to a different controller running an affected version (17.12.4/5/6/6a) before moving to the 9800-M/H1/H2, they may still carry the problematic file. In this case, please follow the Upgrade prechecks section.

Q: We have separate WLCs for Lab testing and staggered upgrades, How should we handle them?

A: Ensure all WLCs in your environment are running the appropriate APSP. Because the cnssdaemon.log file grows by 5 MB daily, any AP that joins a WLC running affected code, even temporarily for testing, will potentially become susceptible to this bug.