Repair C9120/C9115 Access Points from U-boot

Introduction

This document describes the recovery procedure of Catalyst 9115/91120 access points that boot in the U-boot prompt instead of normal operating system.

Background Information

Between Oct. 20 and Oct. 27, 2021, Cisco manufactured a number of C9115 and C9120 access points with a bad image.  These units log "uncorrectable ECC error" messages to the console during bootup, and are unable to boot successfully.  To recover them, you must console into each AP, and load in a special image via TFTP.

This issue is documented as Field Notice: FN - 72278 - Some C9120 and C9115 APs Might Get Stuck in Bootup with 'Uncorrectable ECC Error' Displayed - Workaround Provided and as the Cisco bug ID CSCwa12652.

Recovery Procedure - Overview

1. Download the special image bundle-axel-SS-8_10_130_0.img.
2. Put the image on a TFTP server.
3. Console into the affected AP.
4. Break into u-boot mode.
5. Use u-boot mode to copy the image into the AP via TFTP.

Detailed Steps

Download Special Image Bundle-Axel-SS-8_10_130.0.img

1. Browse to the hidden posting https://software.cisco.com/download/specialrelease/aa90a6d87be2275f1f0111c4973295c2
2. Download the Catalyst 9115AX/9120AX Access Point Manufacture Image bundle-axel-SS-8_10_130_0.img

Note: These images are only to be used to recover an unbootable AP.

Copy Special Image to TFTP Server

Set up a TFTP server in a network that is cabled to the AP's LAN port (labeled "2.5G").

See Appendix A for an example of how to set up the Tftpd64 server in Windows.

Get Into U-boot on the Console

1. Connect a serial cable to the AP's console port (this has a pale blue label that is not labeled "2.5G").  The  serial port can be configured for 9600 bps, 8 databits, 1 stopbit, no parity, no flow control.
   See Appendix B for an example that uses MobaXterm to connect to the serial port.
2. Power up the AP.

As soon as the AP starts powering up, enter the ESC key repetitively in the terminal window, until you see the u-boot prompt.

 
    Verify that the firmware has been loaded with good CRC: OK
    Firmware loading completed successfully
    bcm4908_eth-0
    MAC: 70:69:5a:76:40:3c

    Hit ESC key to stop autoboot: 
    u-boot>

Download Special Image into AP Flash

Configure u-boot to access the TFTP server via IP.  Text manually entered via the terminal is in bold.

1. Configure the AP with an unused IP address in the TFTP server's subnet.

   u-boot> setenv ipaddr 10.1.1.101

2. Configure the AP's netmask.

   u-boot> setenv netmask 255.255.255.0

3. Configure the IP address of the TFTP server.

   u-boot> setenv serverip 10.1.1.1

4. Configure the IP address of the default gateway. If the TFTP server is on the same subnet as the AP, you can enter the TFTP server's address here.

   u-boot> setenv gatewayip 10.1.1.1

5. Clear the target TFTP directory, and save the settings.
   
   

   Note: To verify the current settings, you can use the printenv command.

u-boot> setenv tftpdir
u-boot> saveenv

Saving Environment to SPI Flash...
Erasing at 0x0 -- 100% complete.
SF: Detected mx25l6405d with page size 256 Bytes, erase size 4 KiB, total 8 MiB
Erasing SPI flash...Writing to SPI flash...done

1. Ping the TFTP server to make sure it is reachable.

   u-boot> ping 10.1.1.1
   Use bcm4908_eth-0 device
   host 10.1.1.1 is alive

2. Download the image into AP flash and provide the answers to the prompts like in this example (the interactive prompts can apear after the image is loaded).
   
   

   Note: Do not power cycle the AP during this step.

   
   

   u-boot> boardinit bundle-axel-SS-8_10_130_0.img
   // Some output here was removed in order to emphasize the interactive prompts. Full outputs are shown after the final step. 
   Program PHY firmware? [y/N]: y
   Program UBIFS image? [y/N]: y
   Program bootloaders? [y/N]: n   //pay attention: the last option must be “n”
   

3. Reboot the AP and use the reset command. Once it is fully rebooted, it must respond to Enter with a "Username:" prompt.

   u-boot> reset

   
   Example output during flashing:
   

   u-boot> boardinit bundle-axel-SS-8_10_130_0.img
   Download image bundle-axel-SS-8_10_130_0.img...
   Use bcm4908_eth-0 device
   TFTP from server 10.1.1.1; our IP address is 10.1.1.104
   Filename 'bundle-axel-SS-8_10_130_0.img'.
   Load address: 0x501144b0
   Loading: #################################################################
   
   // Note: The # symbols can continue to print until the image is fully transfered, this can take a few minutes //
   
   done
   Bytes transferred = 147999664 (8d24bb0 hex)
   Program PHY firmware? [y/N]: y
   Writing PHY firmware to NOR flash...
   >>> Backup current PHY firmware image @300000:45000...[Success]
   >>> Erase flash blocks ..........[Success]
   >>> Write PHY firmware image ...[Success]
   [Success]
   Program UBIFS image? [y/N]: y
   Writing UBIFS to NAND...
   
   NAND erase.part: device 0 offset 0x6c0000, size 0x3f940000
   Skipping bad block at  0x00e00000 ……                                        
   Erasing at 0x3ffc0000 -- 100% complete.
   OK
   
   NAND write: device 0 offset 0x6c0000, size 0x8c80000
   Skip bad block 0x00e00000
    147324928 bytes written: OK
   [Success]
   Program bootloaders? [y/N]: n   //pay attention: the last option must be “n”
   u-boot>
   u-boot> reset

Appendix A - Set Up Tftpd64 in Windows

In this example, a Windows PC is manually configured with the address 10.1.1.1, and Ttftpd64 is set up to serve the special image.

Configure PC's Address as 10.1.1.1

Download Tftpd64

TFTPD64 Software

Copy Special Image to TFTP Base Folder

Copy the bundle-axel-SS-8_10_130_0.img file into the TFTP base directory (for example, C:\tftpboot).

Appendix B - Connect to AP Console via MobaXterm

Appendix C - Example Topologies

Option 1 - Multiple APs, PoE Switch and Terminal Server

Option 2 - Multiple APs, PoE Switch and USB Hub with Multiple USB-to-RS232 Adapters

Option 3 - Single AP, PoE Injector