Field Notice: FN - 70330 - Cisco IOS Access Point Stranded Due to Flash Corruption Issue - Workaround Provided

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	20-Dec-18	Initial Release
2.0	28-Jan-19	Updated the Workaround/Solution Section
3.0	24-Apr-19	Updated the Workaround/Solution Section

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	Wireless LAN Controller Software	8.0	8.0.100.0, 8.0.110.0, 8.0.115.0, 8.0.120.0, 8.0.121.0, 8.0.132.0, 8.0.133.0, 8.0.135.0, 8.0.140.0, 8.0.150.0, 8.0.152.0
NON-IOS	Wireless LAN Controller Software	8.1	8.1.101.0, 8.1.102.0, 8.1.111.0, 8.1.120.0, 8.1.121.0, 8.1.122.0, 8.1.131.0
NON-IOS	Wireless LAN Controller Software	8.2	8.2.100.0, 8.2.110.0, 8.2.111.0, 8.2.121.0, 8.2.130.0, 8.2.141.0, 8.2.151.0, 8.2.160.0, 8.2.161.0, 8.2.164.0, 8.2.166.0, 8.2.170.0
NON-IOS	Wireless LAN Controller Software	8.3	8.3.102.0, 8.3.111.0, 8.3.112.0, 8.3.121.0, 8.3.122.0, 8.3.130.0, 8.3.131.0, 8.3.132.0, 8.3.133.0, 8.3.140.0, 8.3.141.0, 8.3.143.0
NON-IOS	Wireless LAN Controller Software	8.4	8.4.100.0
NON-IOS	Wireless LAN Controller Software	8.5	8.5.103.0, 8.5.105.0, 8.5.110.0, 8.5.120.0, 8.5.131.0, 8.5.135.0
IOS	IOS	15.3	15.3(3)JA10, 15.3(3)JC1

Defect Information

Defect ID	Headline
CSCvk15043	Wave 1 APs - AP radio FW image install failure in the bootup loop
CSCvk15068	IOS APs, recovery logic for failure on primary Image
CSCvk26732	New Flash recovery logic
CSCvm33617	Configuration file should not be modified due to low flash memory
CSCvf16302	Flash on lightweight IOS APs gets corrupted
CSCvf28459	Write of the Private File nvram:/lwapp_ap.cfg Failed on compare RCA needed (try = 1)

Problem Description

Some Cisco IOS^® Access Points (APs) report no flash access or file system corruption after an AP reboot. This is most commonly seen when a Wireless LAN Controller (WLC) is upgraded.

The corruption can cause these conditions on the AP:

• The AP is unable to save the configuration.
• The AP is unable to perform an upgrade.
• The AP loses configuration.
• The AP is stuck in a booting loop.
• The AP is in bootloader status.

Background

Due to a number of software issues, the flash file system on some Cisco IOS APs might become corrupt over time during normal operation.

Problem Symptom

Problem #1: No Flash Access

These symptoms are indicative of problem #1, no flash access.

Command: show file system

The size and free space on flash is shown as "-" instead of the actual value.

Problem Output

==============================
       Size(b)       Free(b)      Type  Flags  Prefixes				
*            -             -     flash     rw   flash:	
==============================

Normal Output

================================
       Size(b)       Free(b)      Type    Flags  Prefixes				
*     40900608       25282560     flash     rw     flash:
===============================

 

Command: show flash

The AP cannot display its files.

Directory of flash:/
 %Error opening flash:/ (Invalid argument)

 

Command: show logging

The AP might display these messages.

*Mar  1 00:01:04.159: %LWAPP-3-CLIENTERRORLOG: Save LWAPP Config: error saving config file

.....

*Mar  1 00:01:04.159: %LWAPP-3-CLIENTERRORLOG: Load nvram:/lwapp_ap.cfg config
failed, trying backup...
*Mar  1 00:01:04.159: %LWAPP-3-CLIENTERRORLOG: Load nvram:/lwapp_ap.cfg.bak
config failed...

 

Command: fsck flash:

The AP cannot verify its file system.

AP# fsck flash:

Fsck operation may take a while. Continue? [confirm]

%Error fscking flash: (Device or resource busy)

 

Problem #2: Corrupted Files

This symptom is indicative of problem #2, corrupted files.

The AP is stuck in this loop. This output is seen on the AP console:

Loading "flash:/ap1g1-k9w8-mx.v153_80mr_esc.201603111020/ap1g1-k9w8-mx.v153_80mr_esc.201603111020"
...###############################################################################################
##################################################################################################
##################################################################################################
##################################################################################################
##################################################################################################
####################bad mzip file, unknown zip method?

Workaround/Solution

Solution

Caution: Before an upgrade, run the Wireless LAN (WLAN) Poller tool in order to identify affected Cisco IOS APs on the network and fix them.

Refer to the document Understanding Various AP-IOS Flash Corruption Issues in order to understand how to recover APs with flash issues and for details about how to install and run the WLAN Poller tool.

Upgrade the WLC to one of these AireOS releases:

• Release 8.3MR4Esc (Release 8.3.143.7 or later, available from the Technical Assistance Center (TAC))
• Release 8.3.150.0
• Release 8.5.140.0 or later
• Release 8.8.111.0 or later
• Release 8.9 or later

Notes:

• If you upgrade to a fixed version before you run the WLAN Poller tool, there is a risk that your Cisco IOS APs will be stranded during the upgrade.
• If the WLAN Poller tool needs to recover affected APs, it will cause those APs to reload once the image is downloaded and installed. Ensure that you run the WLAN Poller tool in a maintenance window.
• If it is not possible to upgrade the WLC to a fixed code version at this time, and you see that a significant number of APs enter into this state, you should run the WLAN Poller tool every 30 days or more frequently until you can upgrade.

How To Identify Affected Products

These are the only Cisco IOS AP models affected: 1600, 1700, 2600, 2700, 3600, and 3700 Series APs.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.