Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

WorkFlow

Rolling AP Upgrade Algorithm

1.Candidate AP Set Selection

2.Client Steering

3. AP Re-load and Re-join

Restrictions

Topology

Configuration

From GUI

From CLI

Troubleshoot

Verify

On Source WLC

On Destination WLC

Introduction

This document describes how to perform an N+1 hitless software upgrade on Catalyst 9800 series Wireless LAN controllers.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Catalyst 9800 Wireless LAN Controllers and AP (Cisco IOS® and ClickOS) platforms

- Catalyst 9800 Wireless LAN Controllers software feature sets

Components Used

The information provided in this document is based on the these software and hardware components.

- Catalyst C9800-40 and C9800-L-F-K9 wireless LAN Controllers

- Click OS and Cisco IOS® APs

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The current CAPWAP implementation requires the WLC and the AP to be on the same software version. Therefore, a WLC upgrade is followed by APs upgrade which causes an inevitable network outage. With the current implementation, it is impossible to upgrade the WLC without scheduled downtime.

Hitless upgrade leverages the concept of N+1 high availability with a spare WLC (already upgraded to the target version) to upgrade the CAPWAP infrastructure. The APs are then upgraded in a staggered fashion, and they use the Rolling AP upgrade feature, which avoids network disruption and does not allow all the APs upgrade at once. This ensures that the clients are serviced by the neighbor APs while one of the APs undergoes the upgrade process.

WorkFlow

1. Upgrade the spare WLC to the target version.

2. Establish a mobility tunnel between the production WLC and the spare one.

3. Initiate the upgrade on the production controller with the install add file command.

4. Pre-download the APs.

5. Move the APs to the destination controller (spare WLC). APs are upgraded in a staggered fashion with the Rolling AP upgrade algorithm.

6. Once all the APs move to the spare WLC in multiple iterations activate the target image on production WLC.

7. The production WLC reloads for the new image to take effect.

8. Move all the APs back to the production controller.

Rolling AP Upgrade Algorithm

The algorithm works in three stages.

1. Candidate AP Set Selection

First, a set of candidates are selected based on nearby APs information. Rolling AP Upgrade algorithm selects the configured percentage of APs to be upgraded in each iteration while it maintains RF coverage

For wireless client service, coverage maintenance is important and hence, it takes precedence over selection of the required number of APs. Therefore,

For P = 25%, expected number of iterations for all APs to upgrade ~ 6

For P = 15%, expected number of iterations for all APs to upgrade ~ 12

For P = 5%, expected number of iterations for all APs to upgrade ~ 22

2. Client Steering

Clients on the candidate APs are steered to APs which are not in the candidate list before the candidate APs are rebooted. If the clients still persist on the candidate APs, they are sent a de-authentication frame and the AP reloads with the new image.

3. AP Re-load and Re-join

Past the client steering stage, the AP is reloaded with the new image.

At this point, a 3-minute timer is started for the APs to join back. When this timer expires, all candidate APs are checked and marked for the WLC they have connected to (self or the peer).

If at least 90% of the candidate APs have joined back, the iteration is concluded. If not, 3 minutes window is extended and the check is repeated for two more times until the count hits at least 90%.

At the end of the third try, the iteration is concluded anyway and the next iteration is initiated. Hence, each iteration lasts for, at most, 10 minutes.

Restrictions

- Non-client serving APs (like the ones that work in the monitor, sniffer mode) are upgraded in one go before the rest of the procedure starts.

- Mesh APs are not supported by rolling AP Upgrade. If the deployment has mesh APs, they are upgraded in one shot and at the end of all iterations.

- 16.10 only had a CLI option to configure.

- AP needs to be registered before GUI shows up the option to enable a hitless upgrade.

- The hitless upgrade is not supported by the controller running in BUNDLE mode.

Topology

Configuration

From GUI

Note: From 16.11 onwards, the GUI option for N+1 hitless upgrade is only available when an AP is registered to the controller.

1. Establish the mobility tunnel between the controllers.

2. Initiate the upgrade on the controller. Enable the hitless upgrade option as well. Optionally, enable Fallback after upgrade so that the APs move back to the parent controller (without a swap and reset) after activation of the new image and reloading of the parent controller.

Note: Before this step, ensure that the destination WLC is already upgraded to the target code.

3. Once all the stages are done, WLC prompts for a reload.

From CLI

1. Establish the mobility tunnel between the controllers.

9800-40(config)#wireless mobility group member mac-address d478.9b3c.4ecb ip 10.106.36.78 public-ip 10.106.36.78 group default

9800-L(config)#wireless mobility group member mac-address d4e8.80b2.dc8b ip 10.106.36.110 public-ip 10.106.36.110 group default

2. Initiate the upgrade on the controller.

9800-40#install add file flash:C9800-40-universalk9_wlc.17.01.01s.SPA.bin

Once the installation is successful, the new image is in an inactive state.

9800-40#show install summary

[ Chassis 1 ] Installed Package(s) Information:

State (St): I - Inactive, U - Activated & Uncommitted,

C - Activated & Committed, D - Deactivated & Uncommitted

--------------------------------------------------------------------------------

Type St Filename/Version

--------------------------------------------------------------------------------

IMG I 17.1.1s.0.351

IMG C 16.12.2s.0.47

--------------------------------------------------------------------------------

Auto abort timer: inactive

--------------------------------------------------------------------------------

3. Initiate the pre-download on APs to load the new image as the backup on the APs.

9800-40#ap image predownload

In order to check the status of pre-download, use this command.

9800-40#show ap image

Total number of APs: 5

Number of APs

Initiated                  : 0

Predownloading             : 1

Completed predownloading   : 3

Not Supported              : 0

Failed to Predownload      : 0

Predownload in progress    : Yes

AP Name                           Primary Image          Backup Image            Predownload Status   Predownload Version  Next Retry Time   Retry Count

-------------------------------------------------------------------------------------------------------------------------------------------------------

AP3800                            16.12.2.132            17.1.1.29                  Complete              17.1.1.29             0                 0

3800-2                            16.12.2.132            17.1.1.29                  Complete              17.1.1.29             0                 0

4800-1                            16.12.2.132            17.1.1.29                  Complete              17.1.1.29             0                 0

3702I-2                           16.12.2.132            0.0.0.0                    Predownloading        17.1.1.29             0                 0

4. Optionally, if you are required to configure the percentage of APs to be upgraded per iteration, this command can be used. The default value is 15.

9800-40(config)#ap upgrade staggered  ?

15        15 percent APs per iteration

25        25 percent APs per iteration

5         5 percent APs per iteration

one-shot  All APs in one shot, no staggering

5. Once the pre-download is complete on all the APs, move the APs to the spare controller that runs on the updated code.

9800-40#ap image upgrade destination 9800-L 10.106.36.78 fallback

This command moves the APs to the specified destination WLC with a swap and reset command. Swap command interchanges the AP image so that the target code is marked as the primary image for the APs, whereas, reset command reloads the AP. It is assumed that the destination WLC is on the same version as the APs backup image.

Optionally, you can use the fallback keyword to enable Fallback after Upgrade option so that the APs move back to the parent controller (without a swap and reset) after activation of the new image and reloading of the source controller.

6. Once all the APs have moved to the destination controller, activate the image on the source WLC.

On Destination WLC, verify if all the APs have moved successfully.

9800-L#show ap upgrade

AP upgrade is complete, fallback awaited

Fallback type: Fallback only

From version: 16.12.2.132

To version: 17.1.1.29

Started at: 04/13/2020 02:32:09 UTC

Configured percentage: N/A

Percentage complete: 100

End time: 04/13/2020 02:56:09 UTC

Progress Report

---------------

Iterations

----------

Iteration              Start time                          End time                    AP count

------------------------------------------------------------------------------------------------

0                      04/13/2020 02:32:09 UTC             04/13/2020 02:32:09 UTC     1

1                      04/13/2020 02:32:09 UTC             04/13/2020 02:38:09 UTC     1

2                      04/13/2020 02:38:09 UTC             04/13/2020 02:44:09 UTC     1

3                      04/13/2020 02:44:09 UTC             04/13/2020 02:47:09 UTC     1

4                      04/13/2020 02:47:09 UTC             04/13/2020 02:56:09 UTC     1

Upgraded

--------

Number of APs: 5

AP Name                          Radio MAC               Iteration       Status               Site

----------------------------------------------------------------------------------------------------

AP3800                           1880.9021.e0e0          0               Joined               default-site-tag

3800-2                           1880.9021.e280          1               Joined               default-site-tag

9130-1                           04eb.409f.9760          2               Joined               default-site-tag

4800-1                           dc8c.3746.b0e0          3               Joined               default-site-tag

3702I-2                          fc5b.39f1.c7e0          4               Joined               Unknown

In Progress

-----------

Number of APs: 0

AP Name                          Radio MAC

-------------------------------------------------

Remaining

---------

Number of APs: 0

AP Name                          Radio MAC

-------------------------------------------------

APs not handled by Rolling AP Upgrade

-------------------------------------

AP Name                          Radio MAC               Status          Reason for not handling by Rolling AP Upgrade

----------------------------------------------------------------------------------------------------------------------

On Source WLC, activate the image. Type yes to all the prompts. Once the install is complete, the controller proceeds to reload.

9800-40#install activate

install_add_activate_commit: Activating PACKAGE

These packages shall be activated:

/bootflash/C9800-L-rpboot.17.01.01s.SPA.pkg

/bootflash/C9800-L-mono-universalk9_wlc.17.01.01s.SPA.pkg

/bootflash/C9800-L-hw-programmables.17.01.01s.SPA.pkg

This operation requires a reload of the system. Do you want to proceed? [y/n]y

--- Starting Activate ---

Performing Activate on all members

[1] Activate package(s) on chassis 1

[1] Finished Activate on chassis 1

Checking status of Activate on [1]

Activate: Passed on [1]

Finished Activate

Post-reload, commit the image with this command.

9800-40#install commit

7. If you have not enabled Fallback after Upgrade option (as mentioned in Step 5) use this command on destination WLC to move back the APs to the source WLC, once the source WLC is upgraded to the latest code.

Note: Use this command on destination WLC, only if Fallback after upgrade (in step 5) option was not enabled while the hitless upgrade is initiated.

On source WLC:

9800-40#show version | i Version

Cisco IOS XE Software, Version 17.01.01s

Cisco IOS Software [Amsterdam], C9800 Software (C9800_IOSXE-K9), Version 17.1.1s, RELEASE SOFTWARE (fc4)

On Destination WLC:

9800-L#ap image move destination 9800-40 10.106.36.110

This command moves the APs back to the source WLC without a swap and reset command.

8. All the APs join back the source WLC and the latest image must be in a committed state.

9800-40#show install summary

[ Chassis 1/R0 ] Installed Package(s) Information:

State (St): I - Inactive, U - Activated & Uncommitted,

C - Activated & Committed, D - Deactivated & Uncommitted

--------------------------------------------------------------------------------

Type  St   Filename/Version

--------------------------------------------------------------------------------

IMG   C    17.1.1s.0.351

--------------------------------------------------------------------------------

Auto abort timer: inactive

--------------------------------------------------------------------------------

9800-40#show ap summary

Number of APs: 5

AP Name                            Slots    AP Model  Ethernet MAC    Radio MAC       Location                          Country     IP Address                                 State

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

9130-1                               2      9130AXI   04eb.409e.2620  04eb.409f.9760  default location                  IN          10.106.36.145                              Registered

AP3800                               2      3802I     a023.9fae.f48a  1880.9021.e0e0  default location                  IN          10.106.37.13                               Registered

3800-2                               2      3802I     a023.9fae.f4a4  1880.9021.e280  default location                  IN          10.106.36.187                              Registered

4800-1                               3      4800      dc8c.370e.b2da  dc8c.3746.b0e0  default location                  IN          10.106.36.130                              Registered

3702I-2                              2      3702I     fc5b.39d9.f4b4  fc5b.39f1.c7e0  default location                  IN          10.106.38.219                              Registered

Troubleshoot

It is possible to abort the AP predownload in case one or a few APs are stuck in the AP predownload section with the command:

WLC#ap image predownload abort

It is possible to abort the AP upgrade (that is the part of the workflow where APs have their image swapped and are rebooted progressively to join the N+1 WLC) with the command :

WLC#ap image upgrade abort

Verify

- Ensure that the WLC runs in INSTALL mode. The hitless upgrade is not supported in the BUNDLE mode.

9800-40#show version | i mode

Installation mode is INSTALL

- The mobility tunnel between the controllers must be UP.

9800-40#show wireless  mobility summary

Mobility Summary

Wireless Management VLAN: 36

Wireless Management IP Address: 10.106.36.110

Wireless Management IPv6 Address:

Mobility Control Message DSCP Value: 48

Mobility Keepalive Interval/Count: 10/3

Mobility Group Name: default

Mobility Multicast Ipv4 address: 0.0.0.0

Mobility Multicast Ipv6 address: ::

Mobility MAC Address: d4e8.80b2.dc8b

Mobility Domain Identifier: 0x34ac

Controllers configured in the Mobility Domain:

IP              Public Ip           MAC Address     Group Name    Multicast IPv4    Multicast IPv6     Status  PMTU

---------------------------------------------------------------------------------------------------------------------

10.106.36.110    N/A             d4e8.80b2.dc8b      default        0.0.0.0           ::                 N/A    N/A

10.106.36.78    10.106.36.78     d478.9b3c.4ecb      default        0.0.0.0           ::                 Up     1385

- In order to monitor the AP upgrade, use these commands.

On Source WLC

9800-40#show ap upgrade

AP upgrade is in progress

Fallback type: Fallback only

From version: 16.12.2.132

To version: 17.1.1.29

Started at: 04/12/2020 21:02:09 India

Configured percentage: 15

Percentage complete: 80

Expected time of completion: 04/12/2020 22:22:09 India

Progress Report

---------------

Iterations

----------

Iteration              Start time                          End time                    AP count

------------------------------------------------------------------------------------------------

0                      04/12/2020 21:02:09 India           04/12/2020 21:02:09 India   1

1                      04/12/2020 21:02:09 India           04/12/2020 21:08:09 India   1

2                      04/12/2020 21:08:09 India           04/12/2020 21:14:09 India   1

3                      04/12/2020 21:14:09 India           04/12/2020 21:17:09 India   1

4                      04/12/2020 21:17:09 India           ONGOING                     1

Upgraded

--------

Number of APs: 4

AP Name                          Radio MAC               Iteration       Status               Site

----------------------------------------------------------------------------------------------------

AP3800                           1880.9021.e0e0          0               Joined Member        default-site-tag

3800-2                           1880.9021.e280          1               Joined Member        default-site-tag

9130-1                           04eb.409f.9760          2               Joined Member        default-site-tag

4800-1                           dc8c.3746.b0e0          3               Joined Member        default-site-tag

In Progress

-----------

Number of APs: 1

AP Name                          Radio MAC

-------------------------------------------------

3702I-2                          fc5b.39f1.c7e0

Remaining

---------

Number of APs: 0

AP Name                          Radio MAC

-------------------------------------------------

APs not handled by Rolling AP Upgrade

-------------------------------------

AP Name                          Radio MAC               Status          Reason for not handling by Rolling AP Upgrade

----------------------------------------------------------------------------------------------------------------------

On Destination WLC

9800-L#show ap upgrade

AP upgrade is in progress

Fallback type: Fallback only

From version: 16.12.2.132

To version: 17.1.1.29

Started at: 04/13/2020 02:32:09 UTC

Configured percentage: N/A

Percentage complete: 80

Expected time of completion: 04/13/2020 03:52:09 UTC

Progress Report

---------------

Iterations

----------

Iteration              Start time                          End time                    AP count

------------------------------------------------------------------------------------------------

0                      04/13/2020 02:32:09 UTC             04/13/2020 02:32:09 UTC     1

1                      04/13/2020 02:32:09 UTC             04/13/2020 02:38:09 UTC     1

2                      04/13/2020 02:38:09 UTC             04/13/2020 02:44:09 UTC     1

3                      04/13/2020 02:44:09 UTC             04/13/2020 02:47:09 UTC     1

4                      04/13/2020 02:47:09 UTC             ONGOING                     0

Upgraded

--------

Number of APs: 4

AP Name                          Radio MAC               Iteration       Status               Site

----------------------------------------------------------------------------------------------------

AP3800                           1880.9021.e0e0          0               Joined               default-site-tag

3800-2                           1880.9021.e280          1               Joined               default-site-tag

9130-1                           04eb.409f.9760          2               Joined               default-site-tag

4800-1                           dc8c.3746.b0e0          3               Joined               default-site-tag

In Progress

-----------

Number of APs: 1

AP Name                          Radio MAC

-------------------------------------------------

3702I-2                          fc5b.39f1.c7e0

Remaining

---------

Number of APs: 0

AP Name                          Radio MAC

-------------------------------------------------

APs not handled by Rolling AP Upgrade

-------------------------------------

AP Name                          Radio MAC               Status          Reason for not handling by Rolling AP Upgrade

----------------------------------------------------------------------------------------------------------------------

9800-L#show ap upgrade summary

Report Name                                                      Start time

------------------------------------------------------------------------------------------

AP_upgrade_from_9800-40_13320202329                              04/13/2020 02:32:09 UTC

9800-L#show ap upgrade name AP_upgrade_from_9800-40_13320202329

AP upgrade is in progress

Fallback type: Fallback only

From version: 16.12.2.132

To version: 17.1.1.29

Started at: 04/13/2020 02:32:09 UTC

Configured percentage: N/A

Percentage complete: 60

Expected time of completion: 04/13/2020 03:52:09 UTC

Progress Report

---------------

Iterations

----------

Iteration              Start time                          End time                    AP count

------------------------------------------------------------------------------------------------

0                      04/13/2020 02:32:09 UTC             04/13/2020 02:32:09 UTC     1

1                      04/13/2020 02:32:09 UTC             04/13/2020 02:38:09 UTC     1

2                      04/13/2020 02:38:09 UTC             04/13/2020 02:44:09 UTC     1

3                      04/13/2020 02:44:09 UTC             ONGOING                     0

Upgraded

--------

Number of APs: 3

AP Name                          Radio MAC               Iteration       Status                    Site

---------------------------------------------------------------------------------------------------------

AP3800                           1880.9021.e0e0          0               Joined               default-site-tag

3800-2                           1880.9021.e280          1               Joined               default-site-tag

9130-1                           04eb.409f.9760          2               Joined               default-site-tag

In Progress

-----------

Number of APs: 1

AP Name                          Radio MAC

-------------------------------------------------

4800-1                           dc8c.3746.b0e0

Remaining

---------

Number of APs: 0

AP Name                          Radio MAC

-------------------------------------------------

APs not handled by Rolling AP Upgrade

-------------------------------------

AP Name                          Radio MAC               Status          Reason for not handling by Rolling AP Upgrade

----------------------------------------------------------------------------------------------------------------------