Cisco Aironet 1520 (1522, 1524), 1530, 1550 (1552), 1570, and Industrial Wireless 3700 Series outdoor and industrial wireless access points
Note: There is a much higher prevalence in Wave1 AP models like 1700/2700/3700 and 2600/3600 on this issue vs other AP types due to flash HW type.
As per FN70330 - IOS AP stranded due to flash corruption issue, due to a number of software bugs an AP in normal operation, the flash file system on some IOS APs may become corrupt over time. This is seen especially after an upgrade is performed to the WLC but not necessarily limited to this scenario.
AP may be working fine, servicing client, etc, while on this problem state which is not easily detectable.
Work before WLC Upgrade.
In order to identify affected APs on the network and fix them before an upgrade. You need to run the WLAN Poller
Caution: Before Upgrade, complete reading this document.
WLAN Poller Logic
Every time the script is run it verifies whether an AP’s flash is accessible or not.
If it is accessible, it runs the command fsck flash:
If all is OK, move on to the next AP
else repeat the command up to 4 times. If there is a failure, the script will report this on the final report and this AP is eligible to be recovered.
if it is inaccessible
the script flags AP on its final report and this AP is eligible to be recovered.
If it is accessible, AP will check MD5 values for critical files.
If all values good, move to the next AP
else, the script will report this on the final report and this AP is eligible to be recovered.
The script needs to be run three times.
The script builds the MD5 database looking at MD5 checksum value for every file on the AP. The final MD5 value for a specific file is the one that has the more hits across the same AP family on WLC.
The script compares MD5 checksum values vs its database. If value matches then the file is ok, if not then AP is flagged in order to recover on the third run
The script triggers command test capwap image capwap only for those APs that were flagged during the previous two steps.
Note: This recovery method causes the AP to reload once the image is downloaded and installed. Make sure you run it in a maintenance window.
Once the installation is completed and files are generated. We need to edit the file "config.ini"
Specify WLC/AP connection mode
; config global mode for WLC and AP connection: "ssh" or "telnet" mode: ssh ap_mode: ssh
Specify WLC/AP credentials
; set global WLC credentials wlc_user: <wlc_user> wlc_pasw: <wlc_pasw>
; set global AP credentials ap_user: <ap_user> ap_pasw: <ap_pasw> ap_enable: <ap_enable>
For the flash check/recover we have the following options
; ap file system checks (WARNING: recover will force IOS image download and AP reload) ap_fs_check: False ap_fs_recover: False
To indeitfy affected APs use:
; ap file system checks (WARNING: recover will force IOS image download and AP reload) ap_fs_check: True ap_fs_recover: False
To recover APs use:
; ap file system checks (WARNING: recover will force IOS image download and AP reload) ap_fs_check: True ap_fs_recover: True
Specify WLC information
In this example, WLC name is "2504-WLC". You can find this information on WLC Monitor Page.
; WLC sections must be named as [WLC-<wlcname>] [WLC-2504-Rafis] active: True ipaddr: <wlc-ip-addr>
You can add several WLCs, If so, copy/paste the above syntaxis using new WLC information.
Note: You do not need to specify any AP list. The script picks up AP from the WLC.
Run WLAN Poller
From the directory where config files were created (Previous section, step 3). Use the following command: wlanpoller --cli-logging
Once the script is done. It will provide a summary as follow:
============================================================ Summary ============================================================ Total APs : 1 Processed APs : 1 Failed APs : 0 ============================================================ Errors ============================================================ AP MD5 checksum mismatch : 2 AP FSCK recover : 1 ============================================================
Note: Remember, the script needs to be run 2 times to have accurate information on how many AP are impacted
WLAN Poller Output
On the path where the script was run. It creates the following files.
ap_md5_db.json: MD5 database
It stores all output display by the WLAN poller on the terminal
It breaks down reports into the following path: <year> / <month> / <day>
File: <timestamp>_ap_fs.csv - Summary of the checks executed on APs and their results
ap_name: Name of the AP.
ap_type: AP model.
ap_uptime: Uptime for the AP (days).
ap_ios_ver: IOS version.
fs_free_bytes: Number of free bytes in the flash file system.
flash_issue: True if any flash corruption has been observed.
fs_zero_size: True when flash hung has been detected file system showing "-" - (show file system - command)
fsck_fail: True if file system check has failed. - (fsck flash: - command)
fsck_busy: True device or resource busy when doing flash fsck.
fsck_recovered: True when an error occurred on fsck but it is fixed in next fsck.
fsck_attempts: Number of attempts of fsck to recover the AP (max 4)
md5_fail: True when md5 at least one file is different from the stored inthe database.
rcv_trigger: True when AP tried to download the image from WLC when the issue has been detected and recovery has been enabled.
File: <timestamp>_ap_md5.csv Details of the MD5 checksum values of all files (on all APs)
ap_name: Name of the AP.
ap_type: AP model.
ap_uptime: Uptime for the AP (days).
filename: IOS image file name.
md5_hash: md5 value for filename.
is_good: True md5 value matches with value stored in db. False md5 mismatch observed for this file.
is_zero_bytes: True when filename has 0 bytes based on md5checksum so file is incorrect.
md5_error: Error message retrieving md5 value if it was not possible to get md5 for the filename.
Note: There could be scenarios where the WLAN Poller recovery script is unable to recover certain AP's and those AP remains flagged as failed in the report. In those scenarios, manual AP recovery by telnet/SSH/console into AP CLI is recommended. Please open TAC SR if you needed assistance on this process. Attach all output generated from WLAN poller to the case.
If SSH/telnet connection
You can follow the next steps to try to recover AP:
You can compare CLI value vs value on the cisco web page.
Set boot variable to newly downloaded Recovery image
AP#show boot AP(config)#boot system flash:/RCV/RCV-image
If AP Rommon status
You can try the same as above, but from boot commands. Here the commands you can follow:
ap: tftp_init ap: ether_init ap: flash_init ap: format flash: ap: set IP_ADDR <IP Address> ap: set NETMASK <mask> ap: set DEFAULT_ROUTER < default router > ap: tar -xtract tftp://<IP address>/<file name> flash: ap: set BOOT flash:/<file name> ap: boot
You cannot SSH/telnet
Bounce switch port, few times, verify if that helps.
Step by step guide to install WLAN poller on a Windows 10 machine - 64 bits
Download and install Python 2.7.14 from the following link
Download and install the C++ Compiler for Python for Windows clients from the following link
Once it is installed go to the System Settings on your Control Panel and select Advanced System Settings(Make sure that all the windows terminals are closed):
In the window that pops-out select "Environment Variables…"
In there, select the “Path” variable from the "System variables" and click Edit…
On that window add the path to the base directory where you installed Python 126.96.36.199 and the C:\<Base directory>\Scripts so that the command line of the laptop recognizes python commands: Click on New and add the path manually.
Close all the settings windows and the terminals (command prompt) opened if any
Verify if pip is installed, open a new terminal and enter: pip --version
Another option is to check if there is a file called pip or pip2 or pip2.7 on the folder: C:\Python27\Scripts
If everything is OK go to the section upgrade pip. Step 8
If you get an error or you don’t find the folder/files keep reading.
Close the terminal and install pip from the below link
Download and save the file “get-pip.py”. On the website look for
Copy the “get-pip-py” file to the folder “C:\Python27”
Note: If you copy and paste the content from the website make sure that it does not have the py.txt extension, check this with a dir on the folder “C:\Python27”, if this happens, rename the file from the terminal.
Rename the file with the below command:
On the same folder “C:\Python27” execute: python get-pip.py
Upgrade PIP to the latest version with the below command: pip install --upgrade pip
Above steps will install all packets needed. Now open a command line for Windows and go to the directory where you stored the .tar.gz WLAN poller file (use: cd <Path to directory>)
Install the script by using the command “pip install wlanpoller-0.7.1.dev90_md5rcv.tar.gz”
Create a new directory which you want to store all WLAN poller information.
On the “Command Line” move to that directory and run the command “wlanpoller --generate-configs” to create the setup variables and configuration files needed for the script to run: