Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information.

Configure

Network Diagram

Image Download via TFTP

TFTPD-64 (Windows)

TFTPD-HPA (Linux)

WLC Configuration

Image Download via SFTP

SFTP Server (Linux)

WLC Configuration

Verify

AP Image Download

Troubleshoot

Related Information

Introduction

This document describes in detail the Access Point Image download process for non-homogeneous EWC networks with TFTP and SFTP Servers.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Generals of the AP Join Process.

- Embedded Wireless LAN Controllers on Catalyst 9100 Series APs.

- TFTP File transfers.

- SFTP File transfers

- Linux Command Line Interface usage.

Components Used

The information in this document is based on these software and hardware versions:

- Embedded Catalyst 9800 WLC in a Catalyst 9120AXI AP, Cisco IOS® XE Cupertino 17.9.3.

- Catalyst 9105AXI AP.

- TFTPD-64 version 4.64.

- TFTPD-HPA Linux package.

- SSH Linux package

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information.

Access Pointss that act as EWC can only provide their own AP image type to other access points when they join the network. If your network consists of a non-homogeneous deployment (APs from a different image than the AP acting as EWC), you need to deploy a TFTP or SFTP Server and host the AP images there for the APs to download it from there.

Note: This applies only to AP image upgrade processes that download the image locally from within the network. The APs can also download image directly from the internet via CCO Upgrade.

Configure

Network Diagram

Network Diagram

Image Download via TFTP

TFTPD-64 (Windows)

TFTPD-64 is a well-known Free and Open Source (FOSS) utility that includes TFTP Capabilities. Refer to its website to download and install.

Make sure to unzip the AP Bundle Image in the adequate folder for the TFTP Server.

Unzipped Files in the TFTP Folder

Once the AP starts to download its image from the TFTP Server, a pop-up from TFTP shows up and details the image transfer progress.

TFTPD-64 File Transfer Progress

TFTPD-HPA (Linux)

TFTPD-HPA is a basic, well known package that can be get from the APT repositories. Refer to Ubuntu's TFTP documentation for further information.

Make sure that your TFTP Configurations are adequately pointed to your TFTP Folder and that the AP Bundle Image is unzipped.

TFTP Configurations and Unzipped Files in Ubuntu

You can track the image transfer process logged by default in /var/lib/syslog on Ubuntu.

TFTP File Transfer Logs on Ubuntu

WLC Configuration

In the GUI of the WLC, go to Administration > Software Management > Software Upgrade. Select TFTP in the drop-down list under Mode and provide the information of your TFTP Server.

Choose Save to save the image download profile and enable image download for new APs joining the EWC network or click on Save & Download to immediately trigger the download process on all APs, including the EWC's AP.

TFTP Configuration for Software Upgrade

CLI configuration:

9120-EWC(config)#wireless profile image-download default

9120-EWC(config-wireless-image-download-profile)#image-download-mode tftp

9120-EWC(config-wireless-image-download-profile)#tftp-image-server <TFTP-server>

9120-EWC(config-wireless-image-download-profile-tftp)#tftp-image-path <path>

Image Download via SFTP

SFTP Server (Linux)

Since SFTP works over SSH, you can use Linux's SSH Package to configure a simple SFTP Server in Linux.

Make sure to provide the adequate configurations for SFTP in the /etc/ssh/ssh_config file. Add permissions for the users (or groups) to the SFTP directories as needed and unzip the AP Bundle Image file in the desired path.

SFTP Configuration in Ubuntu

Similarly to the TFTP Server in Linux, you can track the SFTP activity as well. By default, logs are configured to be stored in /var/log/auth.log. Make sure to add the log level configurations as needed.

SFTP Log Activity and Configuration in Ubuntu.

Note: The device that connects to the SFTP Server is the EWC, not the AP that requests the image. This is because the credentials are provisioned in the EWC and not in the APs before they join the EWC. The image then gets forwarded to the actual AP that requests it.

WLC Configuration

In the GUI of the WLC, go to Administration > Software Management > Software Upgrade. Select SFTP in the drop-down list under Mode and provide the information and credentials of your STFTP Server.

Choose Save to save the image download profile and enable image download for new APs joining the EWC network or click on Save & Download to immediately trigger the download process on all APs, including the EWC's AP.

SFTP Configuration in the GUI

CLI Configuration:

9120-EWC(config)#wireless profile image-download default

9120-EWC(config-wireless-image-download-profile)#image-download-mode sftp

9120-EWC(config-wireless-image-download-profile-sftp)#sftp-image-server <SFTP-Server>

9120-EWC(config-wireless-image-download-profile-sftp)#sftp-image-path <path>

9120-EWC(config-wireless-image-download-profile-sftp)#sftp-username <user>

9120-EWC(config-wireless-image-download-profile-sftp)#sftp-password 0 <password>

Verify

The CAPWAP State Machine logs in the APs flow as you normally would expect for any other AP Image Download process.

[*01/30/2024 21:41:35.1120] CAPWAP State: Image Data

[*01/30/2024 21:41:35.1130] AP image version 17.3.3.26 backup 8.10.130.0, Controller 17.9.4.27

[*01/30/2024 21:41:35.1130] Version does not match.

[*01/30/2024 21:41:35.1130] Request to close the file..

[*01/30/2024 21:41:35.1130] wtpOpenImgFile: image file closed, dcb->fd set to -1.

[*01/30/2024 21:41:35.2040] status 'upgrade.sh: Script called with args:[PRECHECK]'

[*01/30/2024 21:41:35.3020] do PRECHECK, part2 is active part

[*01/30/2024 21:41:35.3350] status 'upgrade.sh: Cleanup tmp files ...'

[*01/30/2024 21:41:35.4620] status 'upgrade.sh: /tmp space: OK available 96064, required 50000 '

[*01/30/2024 21:41:35.4630] wtpOpenImgFile: request ap1g8, local /tmp/part.tar

[*01/30/2024 21:41:35.4630] wtpOpenImgFile: open (/tmp/part.tar) image file success

[*01/30/2024 21:41:35.4630] Using fd(37559296) for image writing to file(/tmp/part.tar)

[*01/30/2024 21:41:35.4650] Image Data Request sent to 172.16.4.26, fileName [ap1g8], replicaStatus 1

[*01/30/2024 21:41:35.4690] Image Data Response from 172.16.4.26

[*01/30/2024 21:41:35.4690] AC accepted previous sent request with result code: 0

[*01/30/2024 21:41:35.4760] <.......................................Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).

[*01/30/2024 21:41:50.6190] ...........

[*01/30/2024 21:41:54.7060] ..............................................Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).

[*01/30/2024 21:42:14.0820] ....

[*01/30/2024 21:42:15.5860] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).

[*01/30/2024 21:42:15.6430] .............................................

[*01/30/2024 21:42:34.2800] ...............................Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).

[*01/30/2024 21:42:46.0420] ...................

[*01/30/2024 21:42:53.0610] ..................................................

[*01/30/2024 21:43:11.6480] ......> 70512640 bytes, 51208 msgs, 601 last

[*01/30/2024 21:43:13.3940] Last block stored, IsPre 0, WriteTaskId 0

[*01/30/2024 21:43:13.3940] Request to close the file..

[*01/30/2024 21:43:13.3940] wtpOpenImgFile: image file closed, dcb->fd set to -1.

[*01/30/2024 21:43:13.3940] Image transfer completed from WLC, last 1

[*01/30/2024 21:43:13.3940] Request to close the file..

[*01/30/2024 21:43:13.3940] wtpOpenImgFile: image file closed, dcb->fd set to -1.

[*01/30/2024 21:43:13.3950] in (CAPWAP_MSGELE_IMAGE_DATA_msg_dec_cb) Enabling radCfg.is_oob_image_dnld_supported

[*01/30/2024 21:43:13.4190] wtp_delayed_event_handle_write_image_to_storage(10): fileName ap1g8, pre 0

[*01/30/2024 21:43:13.4190] wtp_delayed_event_handle_write_image_to_storage(10): fileName ap1g8, pre 0

[*01/30/2024 21:43:13.5110] status 'upgrade.sh: Script called with args:[PREDOWNLOAD]'

[*01/30/2024 21:43:13.6100] do PREDOWNLOAD, part2 is active part

[*01/30/2024 21:43:13.6420] status 'upgrade.sh: Creating before-upgrade.log'

[*01/30/2024 21:43:13.6990] status 'upgrade.sh: Start doing upgrade arg1=PREDOWNLOAD arg2= arg3= ...'

[*01/30/2024 21:43:13.8610] status 'upgrade.sh: Using image /tmp/part.tar on ax-bcm32 ...'

[*01/30/2024 21:43:20.9990] status 'Image signing verify success.'

In the WLC Syslog, the Image download is marked as Successful.

*Feb 1 17:05:37.108: %INSTALL-5-INSTALL_COMPLETED_INFO: Chassis 1 R0/0: install_engine: Completed install add sftp://******@172.16.5.62/Documents/sftp_files/EWC_17_9_4a/ap3g3

*Feb 1 17:07:00.720: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: AP-POD-2-2 Mac: 2c5a.0f40.6920 Session-IP: 172.16.4.33[5248] 172.16.4.26[5246] Disjoined Image Download Success

AP Image Download

Once you start an upgrade process, you can track the AP Image Predownload process with the "show ap image" command on the EWC. Once all APs finish to download the image, you are be able to see the target image in the APs' Backup Image.

9120-EWC#show ap image

Total number of APs  : 3

Number of APs

Initiated                  : 0

Downloading                : 0

Predownloading             : 0

Completed downloading      : 0

Completed predownloading   : 3

Not Supported              : 0

Failed to Predownload      : 0

Predownload in progress    : No

AP Name                           Primary Image          Backup Image            Predownload Status   Predownload Version  Next Retry Time   Retry Count   Method

------------------------------------------------------------------------------------------------------------------------------------------------------------------

AP-POD-2-2                        17.9.4.27              17.12.1.5                  Complete              17.12.1.5             0                 0                 CAPWAP

AP6C41.0E16.E79C                  17.9.4.27              17.12.1.5                  Complete              17.12.1.5             0                 0                 CAPWAP

9105-emorenoa                     17.9.4.27              17.12.1.5                  Complete              17.12.1.5             0                 0                 CAPWAP

Alternatively, in the GUI the progress bar reaches the Activate stage, at which point only the reload is needed to swap the EWC to the new code.

EWC Web UI Upgrade Progress Bar

Below, the EWC shows the Predownload status of the APs.

EWC Web UI APs Image Predownload Status

Troubleshoot

In the AP Image download process, you can see in the CAPWAP State Machine logs in the AP that the download is not able to start.

[*07/12/2023 07:41:00.7960] CAPWAP State: Image Data

[*07/12/2023 07:41:00.7970] AP image version 17.3.3.26 backup 8.10.130.0, Controller 17.9.4.27

[*07/12/2023 07:41:00.7970] Version does not match.

[*07/12/2023 07:41:00.8580] upgrade.sh: Script called with args:[PRECHECK]

[*07/12/2023 07:41:00.9540] do PRECHECK, part2 is active part

[*07/12/2023 07:41:01.0070] upgrade.sh: /tmp space: OK available 101272, required 40000

[*07/12/2023 07:41:01.0080] wtpImgFileReadRequest: request ap1g8, local /tmp/part.tar

[*07/12/2023 07:41:01.0100] Image Data Request sent to 172.16.4.26, fileName [ap1g8], slaveStatus 0

[*07/12/2023 07:41:01.0140] Image Data Response from 172.16.4.26

[*07/12/2023 07:41:01.0140] AC accepted join request with result code: 0

[*07/12/2023 07:41:09.5930] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).

[*07/12/2023 07:41:28.7700] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Image Data(10).

[*07/12/2023 07:41:29.7500]

[*07/12/2023 07:41:29.7500] Going to restart CAPWAP (reason : image download cannot start)...

[*07/12/2023 07:41:29.7500]

[*07/12/2023 07:41:29.7570] Restarting CAPWAP State Machine.

[*07/12/2023 07:41:29.7600] Image Data Request sent to 172.16.4.26, fileName [ap1g8], slaveStatus 1

[*07/12/2023 07:41:29.7970]

[*07/12/2023 07:41:29.7970] CAPWAP State: DTLS Teardown

[*07/12/2023 07:41:29.8330] Aborting image download(0x0): Dtls cleanup, ap1g8

[*07/12/2023 07:41:29.9560] upgrade.sh: Script called with args:[ABORT]

[*07/12/2023 07:41:30.0570] do ABORT, part2 is active part

[*07/12/2023 07:41:30.1050] upgrade.sh: Cleanup tmp files ...

[*07/12/2023 07:41:30.1590] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).

To understand why the AP is unable to download the image, you can check the Syslog in the EWC. It is common to see failed image downloads due to wrong specified paths to the TFTP and SFTP Servers, which is properly reflected in the logs:

For SFTP:

*Feb 1 20:29:14.108: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: AP-9117 Mac: 0cd0.f897.ade0 Session-IP: 172.16.4.34[5248] 172.16.4.26[5246] Disjoined Image Download Failed

*Feb 1 20:29:17.325: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R0/0: install_engine: Started install add sftp://******@172.16.5.62/Documents/Wrong-Path/ap1g6

*Feb 1 20:29:25.730: %INSTALL-3-OPERATION_ERROR_MESSAGE: Chassis 1 R0/0: install_engine: Failed to install_add package sftp://******@172.16.5.62/Documents/Wrong-Path/ap1g6, Error: Failed to download file sftp://******@172.16.5.62/Documents/Wrong-Path/ap1g6: No such file or directory

For TFTP:

*Feb 1 20:52:08.742: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: AP-9117 Mac: 0cd0.f897.ade0 Session-IP: 172.16.4.34[5248] 172.16.4.26[5246] Disjoined Image Download Failed

*Feb 1 20:52:11.894: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R0/0: install_engine: Started install add tftp://172.16.5.27/Wrong-Path/ap1g6

*Feb 1 20:52:13.977: %INSTALL-3-OPERATION_ERROR_MESSAGE: Chassis 1 R0/0: install_engine: Failed to install_add package tftp://172.16.5.27/Wrong-Path/ap1g6, Error: Failed to download file tftp://172.16.5.27/Wrong-Path/ap1g6: No such file or directory

Make sure that your TFTP or SFTP server is reachable by the APs and the EWC. Otherwise, a Timed Out log can be seen in the EWC Syslog.

*Feb 1 20:55:03.359: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: AP-9117 Mac: 0cd0.f897.ade0 Session-IP: 172.16.4.34[5248] 172.16.4.26[5246] Disjoined Image Download Failed

*Feb 1 20:55:06.512: %INSTALL-5-INSTALL_START_INFO: Chassis 1 R0/0: install_engine: Started install add tftp://172.16.5.199/EWC/17_9_4a/ap1g6

*Feb 1 20:55:46.579: %INSTALL-3-OPERATION_ERROR_MESSAGE: Chassis 1 R0/0: install_engine: Failed to install_add package tftp://172.16.5.199/EWC/17_9_4a/ap1g6, Error: Failed to download file tftp://172.16.5.199/EWC/17_9_4a/ap1g6: Timed out

Note: Ensure that UDP Port 69 for TFTP and TCP Port 22 for SFTP are not blocked between the APs and EWC and your TFTP or SFTP Server.

Related Information

- Cisco Embedded Wireless Controller on Catalyst Access Points (EWC) White Paper

- Cisco Embedded Wireless Controller on Catalyst Access Points Data Sheet

- Cisco Embedded Wireless Controller on Catalyst Access Points FAQ

- Understand the AP Join Process with the Catalyst 9800 WLC

- Release notes for Cisco Catalyst 9800 Series Wireless LAN Controller, Cisco IOS XE