High Availability (SSO) Deployment Guide
High Availability in Release 7.3 and 7.4
HA Connectivity Using Redundant Port on the 5500/7500/8500 WLC
High Availability Connectivity Using Redundant VLAN on WiSM-2 WLC
Introduction of New Interfaces for HA Interaction
Redundancy Management Interface
Configure HA from the Configuration Wizard
Important Guidelines before Initiating a WLC Upgrade in HA Setup
Download/Upload Facts in HA Setup
Failover Process in the HA Setup
Steps to Simulate Box Failover
SSO Deployment with Legacy Primary/Secondary/Tertiary HA
SSO Deployment in Mobility Setup
One WLC has a valid AP Count license and the other WLC has a HA SKU UDI
Both the WLCs have a valid AP Count license
One WLC has an Evaluation license and the other WLC has a HA SKU UDI or Permanent license
High Availability in Release 7.5
Redundancy Port Connectivity in 7.5
Supported HA Topologies in Release 7.5
5500/7500/8500 Series Controllers
5508, 7500, or 8500 Connected to VSS Pair
Supported HA Topologies for WiSM2 Controllers
WiSM2 in Different Chassis: Redundancy VLAN over L2 Network
WiSM2 in Different Chassis: VSS Pair
Client SSO (Client Stateful Switchover)
Client SSO Behavior and Limitations
High Availability in Release 8.0
Debug/Show Command Enhancements
Configurable Keep-alive and Peer-Search Parameters
Peer RMI ICMP Ping Replaced with UDP Messages
Standby WLC On-the-Fly Maintenance Mode (MTC)
Default Gateway Reachability Check Enhancement
SSO Support for Internal DHCP Server
SSO Support for Sleeping Clients
High Availability in Release 8.1
HA Standby Monitoring Feature Introduction
Trap When WLC Turns Hot Standby
Trap when Standby WLC goes Down
Syslog notification when Admin login on Standby
Peer Process Statistics on CLI
Peer Process Statistics on GUI
This document provides information on the theory of operation and configuration for the Cisco Unified Wireless LAN Controller (WLC) as it pertains to supporting stateful switchover of access points and clients (AP and Client SSO).
The new High Availability (HA) feature (that is, AP SSO) set within the Cisco Unified Wireless Network software release version 7.3 and 7.4 allows the access point (AP) to establish a CAPWAP tunnel with the Active WLC and share a mirror copy of the AP database with the Standby WLC. The APs do not go into the Discovery state when the Active WLC fails and the Standby WLC takes over the network as the Active WLC.
There is only one CAPWAP tunnel maintained at a time between the APs and the WLC that is in an Active state. The overall goal for the addition of AP SSO support to the Cisco Unified Wireless LAN is to reduce major downtime in wireless networks due to failure conditions that may occur due to box failover or network failover.
To support High Availability without impacting service, there needs to be support for seamless transition of clients and APs from the active controller to the standby controller. Release 7.5 supports Client Stateful Switch Over (Client SSO) in Wireless LAN controllers. Client SSO will be supported for clients which have already completed the authentication and DHCP phase and have started passing traffic. With Client SSO, a client's information is synced to the Standby WLC when the client associates to the WLC or the client’s parameters change. Fully authenticated clients, i.e. the ones in Run state, are synced to the Standby and thus, client re-association is avoided on switchover making the failover seamless for the APs as well as for the clients, resulting in zero client service downtime and no SSID outage.
The information in this document is based on these software and hardware versions:
■WLCs 5500 Series, 7500/8500 Series, and WiSM-2
■APs 700, 1130, 1240, 1250, 1040, 1140, 1260, 1600, 2600, 3500, 3600 Series APs, and 1520 or 1550 Series Mesh APs (MAPs).
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
The new architecture for HA is for box-to-box redundancy. In other words, 1:1 where one WLC will be in an Active state and the second WLC will be in a Hot Standby state continuously monitoring the health of the Active WLC via a Redundant Port. Both the WLCs will share the same set of configurations including the IP address of the Management interface. The WLC in the Standby state does not need to be configured independently as the entire configuration (Bulk Configuration while boot up and Incremental Configuration in runtime) will be synced from the Active WLC to the Standby WLC via a Redundant Port. The AP's CAPWAP State (only APs which are in a run state) is also synced, and a mirror copy of the AP database is maintained on the Standby WLC. The APs do not go into the Discovery state when the Active WLC fails and the Standby WLC takes over the network's Active WLC.
There is no preempt functionality. When the previous Active WLC comes back, it will not take the role of the Active WLC, but will negotiate its state with the current Active WLC and transition to a Standby state. The Active and Standby decision is not an automated election process. The Active/Standby WLC is decided based on HA SKU (Manufacturing Ordered UDI) from release 7.3 onwards. A WLC with HA SKU UDI will always be the Standby WLC for the first time when it boots and pairs up with a WLC running a permanent count license. For existing WLCs having a permanent count license, the Active/Standby decision can be made based on manual configuration.
AP SSO is supported on 5500/7500/8500 and WiSM-2 WLCs. Release 7.3 only supports AP SSO that will ensure that the AP sessions are intact after switchover.
Client SSO is supported on 5500/7500/8500 and WiSM2 WLCs from release 7.5 onwards. For more information see High Availability in Release 7.5.
■5500/7500/8500 WLCs have a dedicated Redundancy Port which should be connected back to back in order to synchronize the configuration from the Active to the Standby WLC.
■Keep-alive packets are sent on the Redundancy Port from the Standby to the Active WLC every 100 msec (default timer) in order to check the health of the Active WLC.
■Both the WLCs in HA setup keep track of gateway reachability. The Active WLC sends an Internet Control Message Protocol (ICMP) ping to the gateway using the Management IP address as the source, and the Standby WLC sends an ICMP ping to the gateway using the Redundancy Management IP address. Both the WLCs send an ICMP ping to the gateway at a one-second interval.
■It is highly recommended to have back-to-back direct connectivity between Redundant Ports.
Here you can see the Redundant Port Connectivity between 5500 WLCs in an HA Setup:
Here you can see the Redundant Port Connectivity between Flex 7500 WLCs in an HA setup:
Note: A direct physical connection between Active and Standby Redundant Ports is highly recommended. The distance between the connections can go up to 100 meters at per Ethernet cable standards.
■WiSM-2 WLCs have a dedicated Redundancy VLAN which is used to synchronize the configuration from the Active WLC to the Standby WLC.
■A Redundancy VLAN should be a Layer 2 VLAN dedicated for the HA Pairing process. It should not be spanned across networks and should not have any Layer 3 SVI interface. No data VLAN should be used as a Redundancy VLAN.
■Keep-alive packets are sent on Redundancy VLAN from the Standby WLC to the Active WLC every 100 msec (default timer) in order to check the health of the Active WLC.
■Both the WiSMs in a HA setup keep track of gateway reachability. Active WLC sends an ICMP ping to the gateway using the Management IP address as the source, and the Standby WLC sends an ICMP ping to the gateway using the Redundancy Management IP address. Both the WLCs send an ICMP ping to the gateway at a one-second interval.
■To achieve HA between WiSM-2, it can be deployed in a single chassis or can also be deployed between multiple chassis using VSS as well as by extending Redundancy VLAN between two chassis.
This diagram shows HA Connectivity in a single chassis and extending Redundancy VLAN in a multiple chassis VSS setup:
Warning: The Redundancy VLAN should be a non routable VLAN. In other words, no layer 3 interface should be created for this VLAN and can be allowed on VSL Link to extend HA setup between multiple chassis in VSS setup. It is important to make sure this VLAN is dedicated for the HA process and is not part of any Data VLAN, or else it may result in unpredictable results.
Note: The Redundancy VLAN should be created like any normal Data VLAN on IOS® switches. Redundancy VLAN is configured for redundant port on WiSM-2 blades connected to a backplane. There is no need to configure an IP address for the Redundancy VLAN as it will receive an auto-generated IP which is discussed later in this document.
Note: On Cisco WiSM2 and Cisco Catalyst 6500 Series Supervisor Engine 2T, if HA is enabled, post switchover, the APs might disconnect and reassociate with the WiSM2 controller. To prevent this from occurring, before you configure HA, we recommend that you verify–in the port channel–the details of both the active and standby Cisco WiSM2 controllers, that the ports are balanced in the same order, and the port channel hash distribution is using fixed algorithm. If they are not in order, you must change the port channel distribution to be fixed and reset Cisco WiSM2 from the Cisco Catalyst 6500 Series Supervisor Engine 2T. You can use the command show etherchannel port-channel to verify the port channel member order and load value. You can use the config command port-channel hash-distribution fixed to make the distribution fixed.
Note: To support the active and standby WLCs in different data centers, in release 7.5, back-to-back redundancy port connectivity between peers is no longer mandatory and the redundancy ports can be connected via switches such that there is L2 adjacency between the two controllers. See Redundancy Port Connectivity in 7.5 for more information.
The IP address on this interface should be configured in the same subnet as the management interface. This interface will check the health of the Active WLC via network infrastructure once the Active WLC does not respond to Keepalive messages on the Redundant Port. This provides an additional health check of the network and Active WLC, and confirms if switchover should or should not be executed. Also, the Standby WLC uses this interface in order to source ICMP ping packets to check gateway reachability. This interface is also used in order to send notifications from the Active WLC to the Standby WLC in the event of Box failure or Manual Reset. The Standby WLC will use this interface in order to communicate to Syslog, the NTP server, and the TFTP server for any configuration upload.
This interface has a very important role in the new HA architecture. Bulk configuration during boot up and incremental configuration are synced from the Active WLC to the Standby WLC using the Redundant Port. WLCs in a HA setup will use this port to perform HA role negotiation. The Redundancy Port is also used in order to check peer reachability sending UDP keep-alive messages every 100 msec (default timer) from the Standby WLC to the Active WLC. Also, in the event of a box failure, the Active WLC will send notification to the Standby WLC via the Redundant Port. If the NTP server is not configured, a manual time sync is performed from the Active WLC to the Standby WLC on the Redundant Port. This port in case of standalone controller and redundancy VLAN in case of WISM-2 will be assigned an auto generated IP Address where last 2 octets are picked from the last 2 octets of Redundancy Management Interface (the first 2 octets are always 169.254).
1. Before you configure HA, it is mandatory to have both the controllers' management interface in the same subnet:
2. HA is disabled by default. Before you enable HA, it is mandatory to configure the Redundancy Management IP Address and Peer Redundancy Management IP Address. Both the interfaces should be in the same subnet as the Management Interface. In this example, 9.6.61.21 is the Redundancy Management IP Address for WLC 1, and 9.6.61.23 is the Redundancy Management IP Address for WLC 2. It also needs to be configured so that 9.6.61.23 is the Redundancy Management IP Address of WLC 2 and 9.6.61.21 is the Redundancy Management IP Address of WLC 1.
Use this CLI in order to configure the Redundancy and Peer Redundancy Management IP Address:
3. Configure one WLC as Primary (by default, the WLC HA Unit ID is Primary and should have a valid AP-BASE count license installed) and another WLC as Secondary (AP base count from the Primary WLC will be inherited by this unit) using the CLI in this step. In this example, WLC 1 is configured as Primary, and WLC 2 is configured as Secondary:
Note: You do not need to configure the unit as Secondary if it is a factory ordered HA SKU that can be ordered from release 7.3 onwards. A factory ordered HA SKU is a default Secondary unit, and will take the role of the Standby WLC the first time it is paired with an Active WLC that has a valid AP Count License.
If you want to convert any existing WLC as a Standby WLC, do so using the config redundancy unit secondary command in the CLI. This CLI command will only work if the WLC which is intended to work as Standby has some number of permanent license count. This condition is only valid for the 5508 WLC, where a minimum of 50 AP Permanent licenses are needed to be converted to Standby. There is no restriction for other WLCs such as the 5520, WiSM2, 7500, and 8500.
4. After the WLCs are configured with Redundancy Management and Peer Redundancy Management IP Addresses and Redundant Units are configured, it is time to enable SSO. It is important to make sure that physical connections are up between both the controllers (that is, both the WLCs are connected back to back via the Redundant Port using an Ethernet cable) and the uplink is also connected to the infrastructure switch and the gateway is reachable from both the WLCs before SSO is enabled.
Once SSO is enabled, it will reboot the WLCs. While it boots, the WLCs negotiate the HA role as per the configuration via Redundant Port. If the WLCs cannot reach each other via Redundant Port or via the Redundant Management Interface, the WLC configured as Secondary may go in to Maintenance Mode. Maintenance Mode is discussed later in this document.
5. Use the CLI in this step in order to enable AP SSO. Remember that enabling AP SSO will initiate a WLC reboot.
6. Enabling SSO will reboot the WLCs in order to negotiate the HA role as per the configuration performed. Once the role is determined, configuration is synced from the Active WLC to the Standby WLC via the Redundant Port. Initially, the WLC configured as Secondary will report XML mismatch and will download the configuration from Active and reboot again. During the next reboot after role determination, it will validate the configuration again, report no XML mismatch, and process further in order to establish itself as the Standby WLC.
These are the boot-up logs from both the WLCs:
WLC 2 on first reboot after enabling SSO:
Note: Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
WLC 2 on second reboot after downloading XML configuration from Active:
7. After SSO is enabled, WLC is rebooted, and the XML configuration is synced, WLC 1 will transition its state to Active and WLC 2 will transition its state to Standby HOT. From this point onwards, GUI/Telnet/SSH for WLC 2 on the management interface will not work, as all the configurations and management should be done from the Active WLC. If required, the Standby WLC (WLC 2, in this example) can only be managed via the Console or Service Port.
Also, once the Peer WLC transitions to the Standby Hot state, -Standby keyword is automatically appended to the Standby WLCs prompt name.
8. Complete these steps in order to check the redundancy status:
a. For WLC 1, go to Monitor > Redundancy > Summary:
b. For WLC 2, go to Console connection:
Note: Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
1. On primary controller, disable SSO using the command:
Config redundancy mode disable
The Active and Standby WLCs reboot once this command is executed.
The standby controller, when it comes back after the reboot, has the same IP address on interfaces as the primary controller and all the ports disabled.
2. On the standby controller, re-enter the correct IP addresses corresponding to the management and dynamic interfaces and execute the following command:
Config port adminmode all enable
3. Save the configuration on the controller.
4. To re-enable SSO, execute the command Config redundancy sso on the primary and secondary controllers.
Both controllers reboot and pair up in the SSO mode. The standby will sync its configuration from the primary and come back in Hot-standby mode.
1. Before you configure HA, it is mandatory to have both the controllers' management interface in the same subnet:
2. HA is disabled by default. Before you enable HA, it is mandatory to configure the Redundancy Management IP Address and the Peer Redundancy Management IP Address. Both interfaces should be in the same subnet as the Management Interface. In this example, 9.6.61.21 is the Redundancy Management IP Address for WLC 1, and 9.6.61.23 is the Redundancy Management IP Address for WLC 2. It needs to be configured on WLC 2 where 9.6.61.23 is the Redundancy Management IP Address of WLC 2 and 9.6.61.21 is the Redundancy Management IP Address of WLC 1.
Enter the IP Address for both interfaces, and click Apply.
3. Configure one WLC as Primary and the other WLC as Secondary from the Redundant Unit drop-down list. In this example, WLC 1 is configured as Primary and WLC 2 is configured as Secondary. Once configured, click Apply.
Note: You do not need to configure the unit as Secondary if it is a factory ordered HA SKU ordered from release 7.3 onwards. A factory ordered HA SKU is the default Secondary unit and will take the role of the Standby WLC the first time it is paired with an Active WLC with a valid AP Count License.
If you want to convert any existing WLC as a Standby WLC, do so by using the config redundancy unit secondary command in the CLI. This CLI only works if the WLC which is intended to work as standby has some number of permanent license count. This condition is only valid for the 5508 WLC, where a minimum of 50 AP Permanent licenses are needed to be converted to Standby. There is no restriction for other WLCs such as the 5520, WiSM2, 7500, and 8500.
4. After the WLCs are configured with Redundancy Management and Peer Redundancy Management IP Address and Redundant Units are configured, it is time to enable SSO. It is important to make sure that physical connections are up between both the controllers (that is, both the WLCs are connected back to back via Redundant Port using an Ethernet cable) and the uplink is also connected to the infrastructure switch and the gateway is reachable from both the WLCs before SSO is enabled.
Once SSO is enabled, it will reboot the WLCs. While it boots, the WLCs negotiate the HA role as per the configuration via Redundant Port. If the WLCs cannot reach each other via the Redundant Port or via the Redundant Management Interface, the WLC configured as Secondary may go in Maintenance Mode. Maintenance Mode is discussed later in this document.
5. In order to enable AP SSO, select Enabled from the drop-down list on both the WLCs, and click Apply. After you enable AP SSO, the WLCs reboot and the default information is populated in other fields like Peer Service Port IP, Peer Redundancy port IP, and so forth.
6. Enabling SSO will reboot the WLCs in order to negotiate the HA role as per the configuration performed. Once the role is determined, configuration is synced from the Active WLC to the Standby WLC via the Redundant Port. Initially WLC configured, as Secondary will report XML mismatch and will download the configuration from Active and reboot again. During the next reboot after role determination, it will validate the configuration again, report no XML mismatch, and will process further in order to establish itself as the Standby WLC.
These are the boot-up logs from both the WLCs:
WLC on first reboot after enabling SSO:
Note: Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
WLC 2 on second reboot after downloading XML configuration from Active:
7. After SSO is enabled, WLC is rebooted, and the XML configuration is synced, WLC 1 transitions its state as Active and WLC 2 transitions its state to STANDBY HOT. From this point onwards, GUI/Telnet/SSH for WLC 2 on the management interface will not work, as all the configurations and management should be done from the Active WLC. If required, the Standby WLC (WLC 2, in this case) can only be managed via the Console or Service Port.
Also, once Peer WLC transitions to the STANDBY HOT state, the -Standby keyword is automatically appended to Standby WLCs prompt name.
Note: Enable HA on the primary controller then 5 minutes later enable it on the secondary controller to force the primary to come up as Active..
8. Complete these steps in order to check the redundancy status:
a. For WLC 1, go to Monitor > Redundancy > Summary:
b. For WLC 2, go to Console connection:
Note: Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
1. HA between two WLCs can also be enabled from the configuration wizard. It is mandatory to configure the Management IP Address of both the WLCs in same subnet before you enable HA.
2. Once the Management IP is configured, the wizard will prompt you to enable HA. Enter yes in order to enable HA, which is followed by the configuration of the Primary/Secondary Unit and the Redundancy Management and Peer Management IP Address.
3. After you enable HA from the configuration wizard, continue to configure these legacy wizard parameters:
The WLCs will reboot after you save the configuration at the end.
4. While booting, the WLCs will negotiate the HA role as per the configuration done. Once the role is determined, the configuration is synced from the Active WLC to the Standby WLC via the Redundant Port. Initially WLC is configured, as Secondary will report XML mismatch and will download the configuration from Active and reboot again. During the next reboot after role determination, it will validate the configuration again, report no XML mismatch, and process further in order to establish itself as the Standby WLC.
These are the boot-up logs from both the WLCs:
WLC 2 on first reboot after enabling HA:
WLC 2 on second reboot after downloading XML configuration from Active:
Note: Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
5. After HA is enabled followed by WLC reboots and XML configuration is synced, WLC 1 will transition its state as Active and WLC 2 will transition its state as STANDBY HOT. From this point onwards GUI/Telnet/SSH for WLC 2 on management interface will not work, as all the configurations and management should be done from Active WLC. If required, the Standby WLC (WLC 2, in this case) can only be managed via the Console or Service Port.
Also, once the Peer WLC transitions to the STANDBY Hot state, the -Standby keyword is automatically appended to the Standby WLCs prompt name.
6. Complete these steps in order to check the redundancy status:
b. For WLC 2, go to Console connection:
Note: Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
1. Before you configure HA, it is mandatory to have both the controllers' management interface in the same subnet.
2. Add both the controllers in Cisco Prime using their individual Management IP Address. Once added, both the WLCs can be viewed under Operate > Device Work Center.
3. HA is disabled by default. Before you enable HA, it is mandatory to configure the Redundancy Management IP Address and the Peer Redundancy Management IP Address. Both the interfaces should be in the same subnet as the Management Interface. In this example, 9.6.61.21 is the Redundancy Management IP Address for WLC 1 and 9.6.61.23 is the Redundancy Management IP Address of WLC 2. It needs to be configured on WLC 2 where 9.6.61.23 is the Redundancy Management IP Address of WLC 2 and 9.6.61.21 is the Redundancy Management IP Address of WLC 1.
In order to configure from Cisco Prime, go to Operate > Device Work Center, and select the controller by clicking on the check box in front of the device on which HA should be configured. Once selected, click the Configuration tab, which provides all the options needed to configure the WLC 1, and repeat the steps for WLC 2.
In order to configure the HA parameters for WLC 1, go to Redundancy > Global Configuration, enter the Redundancy and Peer Redundancy-Management IP address, and click Save.
In order to configure the HA parameters for WLC 2, go to Redundancy > Global Configuration, enter the Redundancy and Peer Redundancy-Management IP address, and click Save.
4. Configure one WLC as Primary and the other WLC as Secondary from the Redundant Unit drop-down list. In this example, WLC 1 is configured as Primary and WLC 2 is configured as Secondary. Once configured, click Save.
5. After the WLCs are configured with Redundancy Management and Peer Redundancy Management IP Address, and the Redundant Units are configured, it is time to enable SSO. Once SSO is enabled, it will reboot the WLCs. While booting, the WLCs negotiate the HA role as per configuration via Redundant Port. If the WLCs cannot reach each other via the Redundant Port or via the Redundant Management Interface, the WLC configured as secondary may go in to Maintenance Mode. Maintenance Mode is discussed later in this document.
6. Check the Enabled check box, in order to enable redundancy mode, and click Save. The WLCs will reboot once redundancy mode is enabled.
7. Enabling SSO will reboot the WLCs in order to negotiate the HA role as per the configuration performed. Once the role is determined, the configuration is synced from the Active WLC to the Standby WLC via the Redundant Port. Initially WLC configured, as Secondary will report XML mismatch and will download the configuration from Active and reboot again. In the next reboot after role determination, it will validate the configuration again, report no XML mismatch, and process further in order to establish itself as the Standby WLC.
These are the boot-up logs from both the WLCs:
WLC 2 on first reboot after enabling SSO:
Note: Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.
WLC 2 on second reboot after downloading XML configuration from Active:
8. After SSO is enabled followed by the WLC reboot and XML configuration is synced, WLC 1 will transition its state as Active and WLC 2 will transition its state as STANDBY HOT. From this point onwards, the GUI/Telnet/SSH for WLC 2 on the management interface will not work, as all the configurations and management should be done from the Active WLC. If required, the Standby WLC (WLC 2, in this case) can only be managed via the Console or Service Port.
Also, once the Peer WLC transitions to the STANDBY Hot state, the -Standby keyword is automatically appended to the Standby WLCs prompt name.
9. Once the HA pairing is formed, Cisco Prime removes/deletes the WLC 2 entry from its database as both the WLCs have the same management IP address. For the network, it is the one box which is active in the network.
Note: From this image, it is clear that only WLC 1 (with an IP address of 9.6.61.2 and configured as Primary Unit) is active on Cisco Prime. WLC 2, which was initially added in Cisco Prime with an IP address 9.6.61.3, is deleted from Cisco Prime database after HA pairing is formed.
10. In order to check the redundancy state of the Active WLC from Cisco Prime, go to Device Details > Redundancy > Redundancy States.
The Standby WLC cannot be upgraded directly from the TFTP/FTP server. After executing all scripts, the Active WLC transfers the image to the Standby WLC. Once the Standby WLC receives the image from the Active WLC, it starts executing upgrade scripts. All the logs for image transfer and script execution on the Standby WLC can be seen on the Active WLC.
Note: The FUS image can be upgraded while the controllers have HA enabled. The secondary controller will get upgraded just like it does when upgrading the regular code. However, when you initiate the reboot on the primary controller both controllers will be unreachable until the FUS upgrade completes on both the active and the standby in the HA pair. This process will take around 30 to 40 minutes to complete just like in a non-HA FUS upgrade.
1. After the WLCs are configured in the HA setup, the Standby WLC cannot be upgraded directly from the TFTP/FTP server.
2. Initiate upgrade on the Active WLC in the HA setup via CLI/GUI, and wait for the upgrade to finish.
3. Once the Active WLC executes all the upgrade scripts, it will transfer the entire image to the Standby WLC via the Redundant Port.
4. When the Standby WLC receives the image from the Active WLC, it will start executing the upgrade scripts. The transfer of the image to standby and the execution of the upgrade scripts on the Standby WLC can be seen on the Active WLC Console/Telnet/SSH/Http connection.
5. After a successful message of Standby Upgrade is observed on the Active WLC, it is important to issue the show boot command on the Active WLC in order to make sure the new image is set as the primary image.
6. Once verified, initiate primary image pre-download on the Active WLC in order to transfer the new image to all the APs in the network.
7. After pre-image is completed on all the APs, issue the show AP image all command in order to verify that the primary image on the WLC is set as the backup image on APs.
8. Initiate swap option to interchange the backup image as primary on the APs. With this implementation, the WLC's and AP's primary image is set to the new image.
9. Issue the schedule-reset command as per planned outage with the no swap option in order to reset the APs and WLCs so that they can boot with the new image.
10. All the APs will reboot and join the new Active WLC.
11. Issue the show boot, show sysinfo, show ap image all, and show redundancy summary commands in order to verify that both the WLCs and APs have booted with the new image.
■Service Upgrade is not supported in this release, so network downtime should be planned before you upgrade the WLCs in the HA setup.
■The peer should be in the Hot Standby state before you start the upgrade in the HA setup.
■It is recommended to reboot both the WLCs almost together after upgrade so that there is no software version mismatch.
■Schedule Reset applies to both the WLCs in the HA setup.
■The Standby WLC can be rebooted from the Active WLC using the reset peer-system command if a scheduled reset is not planned.
■Debug transfer can be enabled on the Active WLC as well as the Standby WLC.
■If Active WLC unexpectedly reboot between software download and reboot both WLCs, you need to reboot both WLCs in order to complete software upgrade.
■No direct download and upload configuration is possible from the Standby WLC.
■All download file types like Image, Configuration, Web-Authentication bundle, and Signature Files will be downloaded on the Active WLC first and then pushed automatically to the Standby WLC.
■Once the configuration file is downloaded on the Active WLC, it is pushed to the Standby WLC. This results in the reset of the Standby WLC first, followed by the reset of the Active WLC.
■The Peer Service Port and Static route configuration is a part of a different XML file, and will not be applied if downloaded as part of the configuration file.
■The download of certificates should be done separately on each box and should be done before pairing.
■Uploading different file types like Configuration, Event Logs, Crash files, and so forth can be done separately from the Standby WLC. However, the CLI to configure different parameters for upload like Server IP, file type, path and name should be done on the Active WLC. Once the upload parameters are configured on the Active WLC, the transfer upload peer-start command should be issued on the Active WLC in order to initiate the upload from the Standby WLC.
■The service port state will be synced from the Active WLC to the Standby WLC. That is, if DHCP is enabled on the Active WLC service port, the Standby WLC will also use DHCP for getting the service port IP address. If the service port of the Active WLC is configured with a Static IP Address, the Standby WLC also needs to be configured with a different Static IP Address. The CLI to configure the IP Address for the Standby WLC service port is configure redundancy interface address peer-service-port <IP Address>. This command should be executed from the Active WLC. Also, in order to configure the route on the Standby WLC for out-of-band management on the service port, issue the configure redundancy peer-route add <Network IP Address > <IP Mask> <Gateway> command from the Active WLC.
In the HA setup, the AP's CAPWAP state is maintained on the Active WLC as well as the Standby WLC (only for APs which are in a Run state). That is, Up Time and Association Up Time is maintained on both the WLC, and when switchover is initiated, the Standby WLC takes over the network. In this example, WLC 1 is in an Active state and serving the network, and WLC 2 is in a Standby state monitoring the Active WLC. Although WLC 2 is in Standby state, it still maintains the CAPWAP state of the AP.
Failover for WLCs in HA setup can be categorized into two different sections:
In the case of Box Failover (that is, the Active WLC crashes / system hang / manual reset / force switchover), the direct command is sent from the Active WLC via the Redundant Port as well as from the Redundant Management Interface to the Standby WLC to take over the network. This may take 5-100 msec depending on the number of APs in the network. In the case of power failure on the Active WLC or some crash where the direct command for switchover cannot be sent, it may take 350-500 msec depending on the number of APs in network.
The time it takes for failover in case of power failure on an Active Box also depends on the Keepalive timer configured on the WLC (configured for 100 msec by default). The algorithm it takes to decide the failover is listed here:
■The Standby WLC sends Keepalive to the Active WLC and expects and acknowledgment within 100 msec as per the default timer. This can be configured in range from 100-400 msec.
■If there is no acknowledgment of Keepalive within 100 msec, the Standby WLC immediately sends an ICMP message to the Active WLC via the redundant management interface in order to check if it is a box failover or some issue with Redundant Port connection.
■If there is no response to the ICMP message, the Standby WLC gets aggressive and immediately sends another Keepalive message to the Standby WLC and expects an acknowledgment in 25% less time (that is, 75 msec or 25% less of 100 msec).
■If there is no acknowledgment of Keepalive within 75 msec, the Standby WLC immediately sends another ICMP message to the Active WLC via the redundant management interface.
■Again, if there is no response for the second ICMP message, the Standby WLC gets more aggressive and immediately sends another Keepalive message to the Standby WLC and expects an acknowledgment in time further 25% of actual timer less from last Keepalive timer (that is, 50 msec or last Keepalive timer of 75 msec - 25% less of 100 msec).
■If there is no acknowledgment of the third Keepalive packet within 50 msec, the Standby WLC immediately sends another ICMP message to the Active WLC via the redundant management interface.
■Finally, if there is no response from the third ICMP packet, the Standby WLC declares the Active WLC is dead and assumes the role of the Active WLC.
In the case of a Network Failover (that is, the Active WLC cannot reach its gateway for some reason), it may take 3-4 seconds for a complete switchover depending on the number of APs in the network.
1. Complete the steps as explained in the configuration section in order to configure HA between two WLCs, and make sure before force switchover is initiated that both the WLCs are paired up as the Active WLC and the Standby WLC.
For WLC 2, go to Console connection:
2. Associate an AP to the WLC and check the status of the AP on both the WLCs. In the HA setup, a mirror copy of the AP database is maintained on both the WLCs. That is, APs CAPWAP state in maintained on Active as well as Standby WLC (only for APs which are in Run state) and when switchover is initiated, the Standby WLC takes over the network. In this example, WLC 1 is an Active WLC, WLC 2 is in a Standby state, and the AP database is maintained on both the WLCs.
3. Create an open WLAN and associate a client to it. The client database is not synced on the Standby WLC, so the client entry will not be present on the Standby WLC. Once the WLAN is created on the Active WLC, it will also be synced to the Standby WLC via the Redundant Port.
4. Issue the redundancy force-switchover command on the Active WLC. This command will trigger a manual switchover where the Active WLC will reboot and the Standby WLC will take over the network. In this case, the client on the Active WLC will be de-authenticated and join back on the new Active WLC.
Note: Observe that the prompt in this example changed from 5508-Standby to 5508. This is because this WLC is now the Active WLC and the time taken for AP switchover is 1 msec.
Observe the AP CAPWAP State on WLC 2, which was the Standby WLC initially and is now the Active WLC after switchover. AP Up Time as well as Association Up Time is maintained, and the AP did not go in to the discovery state.
These matrices provide a clear picture of what condition the WLC Switchover will trigger:
■HA Pairing is possible only between the same type of hardware and software versions. Mismatch may result in Maintenance Mode. The Virtual IP Address should be the same on both the WLCs before configuring SSO.
■Direct connectivity is recommended between the Active and Standby Redundant Port for 5500/7500/8500 Series of WLCs.
■WiSM-2 WLCs should be in same 6500 chassis or can be installed in VSS setup for reliable performance.
■A physical connection between Redundant Port and Infrastructure Network should be done prior to HA configuration.
■The Primary units MAC should be used as Mobility MAC in the HA setup in order to form a mobility peer with another HA setup or independent controller. You also have the flexibility to configure a custom MAC address, which can be used as a Mobility MAC address using the configure redundancy mobilitymac <custom mac address> command. Once configured, you should use this MAC address to form a mobility peer instead of using the system MAC address. Once HA is configured, this MAC cannot be changed.
■It is recommended that you use DHCP address assignment for the service port in the HA setup. After HA is enabled, if the static IP is configured for service port, WLC loses the service port IP and it has to be configured again.
■When SSO is enabled, there is no SNMP/GUI access on the service port for both the WLCs in the HA setup.
■Configurations like changing virtual IP address, enabling secureweb mode, configuring web auth proxy, and so forth need a WLC reboot in order to get implemented. In this case, a reboot of the Active WLC will also trigger a simultaneous reboot of the Standby WLC.
■When SSO is disabled on the Active WLC, it will be pushed to the Standby WLC. After reboot, all the ports will come up on the Active WLC and will be disabled on the Standby WLC.
■Keepalive and Peer Discovery timers should be left with default timer values for better performance.
■Clear configuration on the Active WLC will also initiate clear configuration on the Standby WLC.
■Internal DHCP is not supported when SSO is enabled.
■With versions 7.5 and above, AP/Client SSO supports synchronization of L3 MGID between active and standby controllers.
■APs with LSC certificates are supported. The controller's LSC certificate and SCEP configuration must be implemented on the active and standby controllers before activating SSO.
Note: Upon a switchover, the behavior of the mobility peer depends on the version running on the anchor and foreign controllers. When both anchor and foreign controllers are running version 7.5 or higher, roamed clients are not impacted and the peer sends back the AP list, shun list, and Infrastructure MFP keys to the new active controller upon receiving a switchover message.
In a mobility group that has a mix of WLCs running versions lower than 7.5 which supports HA (7.3 and 7.4) and WLCs running versions 7.5 or higher, when a switchover occurs, the roamed clients will be cleaned up on both the anchor and foreign WLCs. Therefore, it is recommended to have a mobility group with WLCs running image versions 7.5 and higher, when an HA Pair is present in the mobility group. If the WLC mobility peer version is older than 7.3, which does not support HA, this problem does not exist.
There are few scenarios where the Standby WLC may go into Maintenance Mode and not be able to communicate with the network and peer:
■Non reachability to Gateway via Redundant Management Interface
■WLC with HA SKU which had never discovered peer
■Software version mismatch (WLC which boots up first goes into active mode and the other WLC in Maintenance Mode)
Note: The WLC should be rebooted in order to bring it out of Maintenance Mode. Only the Console and Service Port is active in Maintenance Mode.
HA (that is, AP SSO) can be deployed with Secondary and Tertiary Controllers just like today. Both Active and Standby WLCs combined in the HA setup should be configured as primary WLC. Only on failure of both Active and Standby WLCs in the HA setup will the APs fall back to Secondary and further to Tertiary WLCs.
Each WLC has its own unique MAC address, which is used in mobility configuration with an individual controller management IP address. In HA (that is, AP SSO) setup, both the WLCs (Primary and Standby) have their own unique MAC address. In the event of failure of the Primary box and Standby takes over the network if the MAC address of the Primary box is used on another controllers in mobility setup, control path and data path will be down and user has to manually change the MAC to standby MAC address on all the controllers in mobility setup. This is a really cumbersome process as a lot of manual intervention is required.
In order to keep the mobility network stable without any manual intervention and in the event of failure or switchover, the back-and-forth concept of Mobility MAC has been introduced. When the HA pair is set up, by default, the Primary WLC's MAC address is synced as the Mobility MAC address on the Standby WLC which can be seen via the show redundancy summary command on both the controllers.
In this output, captured from a Standby controller, the Mobility MAC address can be observed, which is different from the Standbys own MAC address seen as Unit ID. This MAC address is synced from the Active WLC and should be used in mobility configuration. With this implementation, if the Active WLC goes down or even if it is replaced, the Mobility MAC address is still available and active on the Standby WLC. In case the new controller is introduced in the network because of the replacement of the previous Active WLC, it will transition its state as Standby and the same Mobility MAC address is synced again to the new Standby WLC.
You have the flexibility to configure a custom MAC address as Mobility MAC instead of using the default behavior of using the Active WLC MAC address as Mobility MAC. This can be done using the configure redundancy mobilitymac <custom mac address> command on the Active WLC. Once configured, you should use this MAC address on other controllers in order to form a mobility peer instead of using the Active WLC MAC address. This MAC address should be configured before forming the HA pair. Once the HA pair is formed, the Mobility MAC cannot be changed or edited.
In this topology, the Primary and Standby have their own MAC address. With HA pairing, the Active WLC MAC address is synced as a Mobility MAC address, which is the default behavior if a custom MAC is not configured before HA pairing. Once the Active WLC MAC address is synced as the Mobility MAC address, the same MAC is used in mobility configuration on all the controllers in the mobility setup.
A HA Pair can be established between two WLCs running in these combinations:
■One WLC has a valid AP Count license and the other WLC has a HA SKU UDI
■Both the WLCs have a valid AP Count license
■One WLC has an Evaluation license and the other WLC has a HA SKU UDI or Permanent license
■HA SKU is a new SKU with a Zero AP Count License.
■The device with HA SKU becomes Standby the first time it pairs up.
■AP-count license info will be pushed from Active to Standby.
■On event of Active failure, HA SKU will let APs join with AP-count obtained and will start 90-day countdown. The granularity of this is in days.
■After 90-days, it starts nagging messages. It will not disconnect connected APs.
■With new WLC coming up, HA SKU at the time of paring will get the AP Count:
–If the new WLC has a higher AP count than the previous, the 90-day counter is reset.
–If the new WLC has a lower AP count than the previous, the 90-day counter is not reset.
–In order to lower AP count after switchover, the WLC offset timer will continue and nagging messages will be displayed after time expiry.
■The CLI should be used to configure one WLC as the Standby WLC (as mentioned in the configuration section) provided it satisfies the requirement of minimum permanent license count. This condition is only valid for the 5508 WLC, where a minimum of 50 AP Permanent licenses are needed to be converted to Standby. There is no restriction for other WLCs such as the 5520, WiSM2, 7500, and 8500.
■AP-count license information will be pushed from Active to Standby.
■In the event of a switchover, the new Active WLC will operate with the license count of the previous Active WLC and will start the 90-day countdown.
■The WLC configured as Secondary will not use its own installed license, and only the inherited license from the active will be utilized.
■After 90-days, it starts nagging messages. It will not disconnect connected APs.
■With the new WLC coming up, HA SKU at the time of paring will get the AP Count:
–If the new WLC has a higher AP count than the previous, the 90-day counter is reset.
–If the new WLC has a lower AP count than the previous, the 90-day counter is not reset.
–After switchover to a lower AP count, the WLC offset timer will continue and nagging messages will be displayed after time expiry.
■The device with HA SKU becomes the Standby WLC the first time it pairs up with an existing Active WLC running Evaluation License. Or, any WLC running a permanent license count can be configured as the Secondary unit using the CLI configuration provided if it satisfies the requirement of minimum permanent license count. This condition is only valid for the 5508 WLC, where a minimum of 50 AP Permanent licenses are needed to be converted to Standby. There is no restriction for other WLCs such as the 5520, WiSM2, 7500, and 8500.
■AP-count license information will be pushed from Active to Standby.
■In the event of a switchover, the new Active WLC will operate with the license count of the previous Active WLC and start the 90-day countdown.
■After 90-days, it starts nagging messages. It will not disconnect connected APs.
■With new the WLC coming up, HA SKU at the time of paring will get the AP Count:
–If the new WLC has a higher AP count than the previous, the 90-day counter is reset.
–If the new WLC has a lower AP count than the previous, the 90-day counter is not reset.
–After switchover to a lower AP count, the WLC offset timer will continue and nagging messages will be displayed after time expiry.
To support High Availability without impacting service, there needs to be support for seamless transition of clients and APs from the active controller to the standby controller. Release 7.5 supports Client Stateful Switch Over (Client SSO) in Wireless LAN controllers. Client SSO will be supported for clients which have already completed the authentication and DHCP phase and have started passing traffic. With Client SSO, a client's information is synced to the Standby WLC when the client associates to the WLC or the client’s parameters change. Fully authenticated clients, i.e. the ones in Run state, are synced to the Standby and thus, client re-association is avoided on switchover making the failover seamless for the APs as well as for the clients, resulting in zero client service downtime and no SSID outage.
■In controller release 7.3 and 7.4, back-to-back connectivity through redundancy port restrains the active and standby controllers to be in different locations. There are two mandatory interfaces for redundancy, redundancy port and redundancy management interface. Redundancy port uses dedicated physical port eth1 (similar to service port). It is used for all redundancy communication (AP, Client data, configuration sync, keep-alive messages and role negotiation messages). Redundancy management interface is used to check for the reachability of the peer and management gateway.
■To support the active and standby WLCs in different data centers, in release 7.5, back-to-back redundancy port connectivity between peers is no longer mandatory and the redundancy ports can be connected via switches such that there is L2 adjacency between the two controllers.
■Backward compatibility for release 7.3/7.4 will be supported, wherein back-to-back redundancy port connectivity is used for redundancy communication between the WLCs and the redundancy management interface is used to check the reachability to the peer and to management gateway.
■No additional configuration change is required for redundancy port and the configuration remains the same as in 7.3/7.4 release.
1. Back-to-back Redundancy Port (RP) connectivity between the two WLCs, Redundancy Management Interface (RMI) connectivity to check peer and management gateway reachability.
2. RP connectivity with L2 adjacency between the two WLCs, RMI connectivity to check peer and management gateway reachability. This can be within the same or different data centers.
3. Two 5508, 7500 or 8500 connected to a VSS pair. Primary WLC connected to one 6500 and the Stand-by WLC to the other 6500.
Figure 1 Back-to-back RP connectivity
■This is the same topology as was supported in controller release 7.3.
■Configuration Sync and Keepalive messages are sent via Redundancy Port.
■RMI interface is created as part of Management subnet and is used to check peer and management gateway reachability.
■RTT Latency is 80 milliseconds by default. The RTT should be 80% of the Keepalive timer which is configurable in the range 100-400 milliseconds.
■Failure detection time is 3*100 + 60 + jitter (12 msec) = ~400 msec
Note: In the above equation, 3 is the Keepalive retry count, 100 is the Keepalive timer, and 60 is
3*10 + 3*10 (3 RMI pings to peer + 3 pings to gateway).
configure interface address management 9.5.56.2 255.255.255.0 9.5.56.1
configure interface address redundancy-management 9.5.56.10 peer-redundancy-management 9.5.56.11
configure redundancy unit primary
Configuration on Hot Standby WLC:
configure interface address management 9.5.56.3 255.255.255.0 9.5.56.1
configure interface address redundancy-management 9.5.56.11 peer-redundancy-management 9.5.56.10
Figure 2 RP connectivity via switches
■Redundancy Port connectivity via switches across data centers is supported in this topology.
■Configuration sync and Keepalives via Redundancy Port.
■RMI interface is created as part of Management subnet and is used to check peer and management gateway reachability.
■RTT Latency is 80 milliseconds by default. The RTT should be 80% of the Keepalive timer which is configurable in the range 100-400 milliseconds.
■Failure detection time is 3*100 + 60 + jitter (12 msec) = ~400 msec
configure interface address management 9.5.56.2 255.255.255.0 9.5.56.1
configure interface address redundancy-management 9.5.56.10 peer-redundancy-management 9.5.56.11
configure redundancy unit primary
Configuration on Hot Standby WLC
configure interface address management 9.5.56.3 255.255.255.0 9.5.56.1
configure interface address redundancy-management 9.5.56.11 peer-redundancy-management 9.5.56.10
Figure 5 WiSM2 connectivity using Redundancy VLAN over L2 network
Configuration on Cat6k for WiSM2
wism service-vlan 192 ( service port VLAN)
wism redundancy-vlan 169 ( redundancy port VLAN)
Figure 6 WiSM2 connectivity using VSS Pair
Figure 7 Active and Standby VSS Pair connected via VSL Link
Figure 8 WiSM2 connectivity using VSS Pair
■Round trip latency on Redundancy Link should be less than or equal to 80 milliseconds.
■Preferred MTU on Redundancy Link is 1500 or above.
■Bandwidth on Redundancy Link should be 60 Mbps or more.
■If redundancy ports are connected via switches such that there is L2 adjacency between the two controllers, the RP VLAN should be excluded from the access VLAN configured on the switch for the management ports.
■For WiSM2 connectivity between two different chassis connected across the L2 network, the “redundancy-vlan” should be excluded from the access-VLAN configured on the switch for the management ports.
■It is highly recommended to use different sets of switches for the RP port connectivity and the management port traffic to avoid an Active-Active scenario.
■When deploying WiSM2 in VSS setup, it is recommended to set the peer search time to 180 seconds.
To support High Availability without impacting the service, there needs to be support for seamless transition of the clients and APs from the active controller to the standby controller. Release 7.5 supports Client Stateful Switch Over (Client SSO) in Wireless LAN controllers. Client SSO will be supported for clients, which have already completed the authentication and DHCP phase and have started passing traffic. With Client SSO, the client’s information is synced to the Standby WLC when client associates or the client parameters change. Fully authenticated clients, i.e. ones in Run state, are synced to the Standby and thus, client re-association is avoided on switchover making the failover seamless for the AP as well as for the client.
■Client SSO will work with Anchor-Foreign mobility setup as well as Guest Anchor scenarios.
■L3 MGIDs are synced to the Standby Controller.
■The failover time varies from ~2-996 milliseconds depending on the category of box failover.
■The management gateway failover time is in the order of ~15 seconds, which is the time taken for 12 pings to the management gateway.
■The default RTT latency between the two WLCs is 80 milliseconds. RTT latency should be less than or equal to 80% of the Keepalive timer. The Keepalive timer is configurable in the range 100-400 milliseconds
1. Before configuring HA it is mandatory to have both the controllers’ management interface in same subnet.
2. HA is disabled by default. Before enabling HA it is mandatory to configure Redundant Management IP address and Peer Redundant Management IP address. Both the interfaces should be in same subnet as Management Interface.
To configure Redundant Management and Peer Redundant Management IP address click Controller tab > Redundancy > Global Configuration and enter IP address in both the fields and then click Apply.
3. Now configure one controller as Primary and another controller as Secondary from Redundant Unit drop-down. In an example below WLC 1 is configured as Primary Unit and WLC 2 is configured as Secondary Unit (will work as HA SKU UDI). While pairing, the controller that is configured as Primary will push its AP Count License to Standby WLC. To configure one controller as Primary unit and second controller as Secondary unit, click Controller tab > Redundancy > Global Configuration and select Primary/Secondary from Redundant Unit drop-down list and then click Apply.
4. After controllers are configured with Redundant Management, Peer Redundant Management IP address and Redundant Units are configured, it is very important to make sure physical connection are up between both the controllers i.e. both the WLCs are connected via Redundant Port using Ethernet cable and uplink is also connected to infrastructure switch and gateway is reachable from both the WLCs. Initiate ping to management interface gateway IP Address from both the controllers and make sure reachability to management gateway is fine.
5. To enable SSO navigate to Controller >Redundancy > Global Configuration and select the Enable option from SSO drop-down list on both the WLCs and click Apply. This step will make controllers reboot.
6. Enabling SSO will reboot controllers to negotiate HA role as per configuration and once the role is determined, configuration is synced from Active to Standby WLC via the redundant port. Initially controller configured as Secondary will report XML mismatch after downloading the configuration from Active and reboot again. In next reboot after role determination it will validate the configuration again and will report no XML mismatch and will process further to establish itself as Standby WLC. Thus, controller configured as Primary will reboot once and controller configured as Secondary will reboot twice.
WLC 2 on first reboot after enabling SSO:
WLC 2 on second reboot after downloading XML configuration from Active:
While WLC2 is booting up, no configuration change is allowed on WLC1:
7. After SSO is enabled followed by controller reboots and XML configuration is synced, WLC 1 will transition its state as Active and WLC 2 will transition its state as Standby HOT. From this point onwards GUI/Telnet/SSH for WLC 2 on management interface will not work, as all the configurations should be done from the Active controller. Standby controller i.e. WLC 2 in this case if required can only be managed via Console or Service Port.
Also once Peer WLC transitions to Standby Hot state, -Standby keyword is automatically appended to Standby WLC’s prompt name.
8. To check the redundancy status
WLC 1 -> Click Monitor > Redundancy > Summary:
WLC 1 -> show redundancy summary:
WLC 2 -> show redundancy summary:
1. At this stage both the controllers are paired up in HA setup. Any configuration done on Active will be synced to Standby controller via redundant port. Check the WLAN summary and Interface summary on standby WLC from console connection.
2. In High Availability setup, APs’ CAPWAP state in maintained on Active as well as Standby controller (only for APs which are in Run state) i.e. UP time and Associated UP time is synced from the active to the standby controller. In an example below WLC 1 is an Active state and serving the network and WLC 2 is in Standby state monitoring active controller. Although WLC 2 is in standby state it still maintains CAPWAP state of AP.
Observe the AP UP Time and Association UP Time on Active WLC
Observe the AP Uptime and Association UP Time on Standby WLC will be in sync with active WLC.
3. In case of Box Failover i.e. Active controller crashes / system hang / manual reset / force switchover direct command is sent from Active controller via Redundant Port as well as from Redundant Management Interface to Standby controller to take over the network. Failover may take ~2-360 millisecond depending on number of APs/Clients on the active controller. In case of power failure on Active WLC or some crash where direct command for switchover cannot be sent to the standby controller, it may take ~360 – 990 msec depending upon number of APs/Clients on the active controller and the Keepalive timer configured. The default Keepalive timer is 100 milliseconds. Make sure that default RTT latency is less than or equal to 80 msec.
4. With release 7.5 as part of Client SSO, the client database is also synced to standby WLC so Run state client entries will be present on Standby WLC.
WLC 1-> Console/Telnet/SSH Connection:
Client entry is present on Active WLC.
Client entry is present on Standby WLC.
5. PMK cache is also synced between the two controllers
1. Issue a command redundancy force-switchover on Active controller. This command will trigger manual switchover where Active controller will reboot and Standby controller will take over the network. In this case Run state client on Active WLC will not be de-authenticated. The command save config is initiated before redundancy force-switchover command.
Observe the change in prompt in above screen capture.
Observe the AP CAPWAP State on WLC 2 which was standby initially and is Active now after switchover. AP uptime as well as Association UP Time is maintained and AP did not go in discovery state.
2. Also notice client connectivity when switchover is initiated. Client will be not be de-authenticated.
Ping from wireless client to its gateway IP Address and management IP Address during switchover shows minimal loss.
3. To check the redundancy status
WLC 1 -> Console connection issue a command show redundancy summary:
WLC 2 -> Console connection issue a command show redundancy summary:
WLC 2-> Click on Monitor > Redundancy > Summary:
4. Initiate a force switchover again on current active WLC.
WLC, which was configured as Primary Unit, should now be active and WLC, which was configured as Secondary Unit i.e., WLC 2 should be in Hot Standby State.
WLC 1: Make sure Local state should be Active and Unit should be Primary on WLC 1 after switchover:
Observe the switchover history. WLC maintains 10 switchover histories with switchover reason.
■The Bonjour dynamic database comprising of the services and service providers associated with a service and the domain name database is synced to standby.
■Only clients that are in Run state are synced between the Active and Standby WLC. Client SSO does not support seamless transitions for clients that are in the process of associating/joining the controller. The clients in the transition phase will be de-authenticated after switchover and will need to rejoin the controller.
■Posture and NAC OOB are not supported if the client is not in Run state.
■With release 8.2.111.0 WGB and clients associated to the WGB will be state fully switched over with client SSO.
■CCX based applications need to be re-started post Switchover.
■New mobility is not supported.
■Client statistics are not synced.
■PMIPv6, NBAR, SIP static CAC tree are not synced, need to be re-learned after SSO.
■OEAP (600) clients are not supported.
■Passive clients need to be re-associated after SSO.
■Device and root certificates are not automatically synced to the Standby controller.
■AP and Client Rogue information is not synced to the Standby controller and needs to be re-learnt when the hot standby becomes the active controller.
■Sleeping client information is not synced to the standby controller.
■NBAR statistics are not synced to the secondary controller.
■Native Profiling data is not synced to the secondary controller, therefore, clients will be re-profiled after switchover.
■The below table captures the behavior w.r.t SSO with MAPs and RAPs.
High Availability in release 8.0 introduces enhancements and improvements to the High Availability feature-set. The following enhancements are captured in this section:
■Enhanced debugs and serviceability for HA
■Configurable keep-alive timer/retries and peer-search timer value
■Peer RMI ICMP ping replaced with UDP messages
■Standby WLC on-the-fly maintenance mode
■Default gateway reachability check enhancement
High Availability in release 8.0 also introduces new features enabling SSO such as:
■Internal DHCP server support for SSO enabled controllers
■SSO support for sleeping client feature
Note: Release 8.0 onwards, it is mandatory to tag the RMI and management interfaces to avoid false switchovers.
Currently, the controller does not provide any indication for the completion of Bulk Sync configuration once it is initiated. The Bulk Sync can be verified only by user observation and by manually checking the number of clients synced to the standby WLC. As part of this feature, a mechanism is provided to convey the status of Bulk Sync (both AP and client sync) when standby WLC comes up.
A new field called BulkSync Status is added in the GUI under Controller > Redundancy > Summary. This field points to the status of the bulk sync to the standby WLC and the status can be Pending/In-progress/Complete.
The output of the CLI command show redundancy summary also displays the Bulk Sync status, which can be Pending/In-progress/Complete as shown below while pairing with the standby controller.
When the standby controller is booting up, the BulkSync status shows Pending.
Figure 10 BulkSync Status—Pending
Once the standby controller completes, the boot-up process and the bulk sync starts, the status changes to In-Progress.
Figure 11 BulkSync Status—In-Progress
When the bulk sync process is complete, the BulkSync status changes to Complete.
Figure 12 BulkSync Status—Complete
As HA plays a major role in avoiding network outage, it should also be pertinent to be able to debug the state changes on the boxes at the time of SSO or at a later point in time.
The following new categories of statistics are introduced under Monitor > Redundancy > Statistics:
Figure 13 Redundancy Statistics GUI
The Infra statistics contain RF Client details and Sanity Counters as shown in Figure 14.
Figure 14 Redundancy Statistics—Infra
Figure 15 Redundancy Statistics —Transport
The Heartbeat debugs include events of reception of heartbeats, loss of heartbeats, and subsequent actions related to them.
Figure 16 Redundancy Keep-alive Statistics
The HA system monitors management gateway reachability to reduce network outage.
On the Standby controller, serviceability debugs related to the gateway reachability of the active controller and standby controller, their health states, and actions taken based on this information is reported. While on the active controller, the reachability of active WLC to the gateway alone is reported.
Figure 17 Redundancy GW-Reachability Statistics
Figure 18 Redundancy Config-Sync Statistics
The following debug/show CLI commands are introduced for this feature:
1. debug redundancy infra detail/errors/event
2. debug redundancy transport detail/errors/events/packet
3. debug redundancy keepalive detail/errors/events
4. debug redundancy gw-reachability detail/errors/events
5. debug redundancy config-sync errors/events/detail
6. debug redundancy ap-sync errors/events/detail
7. debug redundancy client-sync errors/events/detail
8. debug redundancy mobility events/errors/detail
9. show redundancy infra statistics
10. show redundancy transport statistics
11. show redundancy keepalive statistics
12. show redundancy gw-reachability statistics
13. show redundancy config-sync statistics
To address the variable network latencies in different customer deployment scenarios, keep-alive and peer-search parameters are made configurable. As part of this enhancement, the maximum number of Keepalives between active and standby controllers to trigger a failover is now configurable. Also, peer-search timer and keep-alive timer are modified to support an extended range.
The following new CLI command is added to configure the number of redundancy keep-alive retries in the range of 3 to 10.
Figure 19 Redundancy retries CLI Command
The existing CLI command config redundancy timer keep-alive-timer of keep-alive timer is modified to support keep-alive timer from 100 to 1000 msecs.
The existing CLI command config redundancy timer peer-search-timer of peer-search timer is modified to support peer-search timer from 60 to 300 secs.
Figure 20 Redundancy timer CLI Command
The following CLI is introduced to view the redundancy keep-alive-retry value.
Figure 21 Show redundancy retries CLI Command
Use the show redundancy timers command to view the peer-search-timer and keep-alive-timer values.
Figure 22 Show redundancy timers CLI Command
Use the show redundancy detail command to display the keep-alive and peer-search timeout values.
Figure 23 Show redundancy detail CLI Command
The keep-alive timer, keep-alive retries, and peer-search timer can also be configured and viewed from the Controller > Redundancy > Global Configuration page in the GUI.
Figure 24 Redundancy Global Configuration GUI
Prior to release 8.0, ICMP ping is used to heart-beat with the peer WLC over the Redundancy Management Interface. As part of this feature for release 8.0, ICMP ping is replaced with a UDP message.
This will benefit due to the following factors:
■ICMP ping packets might get discarded under heavy loads.
■Any other device with the same IP might also reply to the ping.
It is recommended to tag the RMI and management interfaces to avoid false Switchovers. Tagging of the RMI and management interfaces is now mandatory in release 8.0 to pair WLCs in SSO mode.
Prior to release 8.0 when the standby controller looses reachability to the “Default Gateway” or “Peer RP”, the controller reboots and checks that condition while booting up and enters into the MTC mode. With this feature, the standby WLC will enter into the MTC mode “on-the-fly” without rebooting when such error scenarios occur. Once Peer-RP and the default gateway reachability is restored, the MTC mode auto-recovery mechanism introduced in release 7.6, will reboot the WLC and pair it with the active WLC. This mechanism is applicable only to the standby WLC. The active controller will still reboot before going to MTC mode.
As part of this enhancement, the gateway (GW) reachability check mechanism is modified to avoid false positives and it is also modified for the ideal time to start checking for gateway reachability once the controller boots up.
Prior to release 8.0, the “GW reachability check” is performed during Role negotiation. In release 8.0 and later, during Role negotiation, GW reachability check is not performed and is initiated only after the HA Pair-Up is complete.
Also, it is observed that certain Switch/Router configurations rate limits ICMP ping packets or drop them altogether. To avoid such conditions triggering false-positives, the new design ensures not to take switchover decisions purely based on ICMP ping losses. By the modified logic, upon 6 consecutive ping drops, an ARP request is sent to the GW IP address. A successful response to this request is considered as the GW being reachable.
Currently during the HA pairing process, once the active-standby role is determined, the configuration is synced from the active WLC to the standby WLC via the Redundancy Port. If the configuration is different, the secondary WLC reports XML mismatch and downloads the configuration from the active controller and reboots again. In the next reboot after role determination, it validates the configuration again, reports no XML mismatch, and process further in order to establish itself as the Standby WLC.
With this feature enhancement, the XMLs are sent from the to-be-Active to to-be-Standby controller at the time of initialization, just before the validation of the XMLs. This avoids the extra step of comparison and reboot since no other modules are initialized yet, resulting in faster pair up of Active and Standby WLCs.
As seen in the boot logs below, there are no comparison of XMLs and no reboot of standby WLC.
Figure 25 Standby WLC bootup log
Prior to release 8.0, configuration of “Internal DHCP Server” is not allowed on HA enabled controllers because the internal DHCP server data is not synced to the standby WLC. In release 8.0 and later, “Internal DHCP Server” is configured on HA enabled controllers and this data is synced to the standby WLC so that soon after a switchover, the “Internal DHCP Server” on the new active controller starts serving clients.
To configure the Internal DHCP server using the GUI, navigate to Controller > Internal DHCP Server
Figure 26 Internal DHCP Server GUI
The same is synced to the standby controller and is verified by executing the CLI command show dhcp summary
Figure 27 show dhcp summary on Active and Standby WLC
As part of this enhancement, Static CAC method bandwidth allocation parameters for Voice and Video and Call Statistics are synced to the Standby WLC, so that soon after a switchover, respective information is available on the new active controller that will be used for call admission control.
Release 7.5 did not provide SSO support for sleeping clients. The sleeping client database was not synced to the standby controller, which caused the sleeping clients to re-authenticate after a switchover occurred. With this release, sleeping client database is synced to the standby controller, allowing sleeping clients to avoid web re-authentication if they wake up within the sleeping client timeout interval.
The CLI command show custom-web sleep-client summary is used to verify the sleeping client database sync between the active and standby WLC.
Figure 28 Sleeping Client Database on Primary WLC
Figure 29 Sleeping Client Database on Standby WLC
Figure 30 Sleeping Client Details on Active and Standby WLC
Prior to release 8.0, when a switchover occurs on an HA pair, OEAP 600 APs restarts the CAPWAP tunnel and joins back the new active controller, and all the connected clients are de-authenticated. As a part of this feature, OEAP 600 APs ensure not to reset their CAPWAP tunnel. Also, clients continue their connection with the new active controller in a seamless manner.
As shown below, the output of show ap summary and show client summary command on the active and standby controllers displays the AP and client database sync.
Figure 31 OEAP 600 AP on Active WLC
Figure 32 OEAP 600 AP Sync to Standby WLC
Figure 33 Clients on Active WLC
Figure 34 Client Sync to Standby WLC
High Availability in release 8.1 introduces the HA Standby monitoring feature.
From the client’s perspective, although the Active and Hot Standby controllers constitute a single entity, from the administrator’s perspective, they are still considered, maintained, and monitored as two separate controllers. The administrator fetches the status and health information of Active and Standby WLCs separately to monitor and maintain the controllers on a continuous basis with the help of management infrastructure and various user interfaces.
This section outlines the interfaces to fetch the health state information and traps from the Standby controllers and also describes how to use these user interfaces through the CLI, GUI, and SNMP.
A trap is reported with time stamp when HA peer becomes Hot-Standby, and the following trap is reported:
A new trap type is added in CISCO-LWAPP-HA-MIB.my.
After the HA pairing is done and Bulk sync is complete, the following trap is reported:
A new trap type is added in CISCO-LWAPP-HA-MIB.my.
When the standby peer goes down due to any of the following events,
A new trap type is added in CISCO-RF-SUPPLEMENTAL-MIB.my.
On the CLI, the trap can be viewed by executing the command show traplog.
This generates an event in msglog / syslog and message snippet is as follows:
This message can be viewed on the Standby WLC by executing the CLI ‘ show msglog ’
This generates an event in msglog/syslog and message snippet is as follows:
This message can be viewed on the Standby WLC by executing the CLI ‘ show msglog ’
As part of this feature, CPU and Memory statistics of all the threads of the Standby WLC are synced to Active controller every 10 seconds. This information is displayed when the user queries for the Peer statistics on the active WLC.
New Commands on Active WLC to display peer process System, CPU and memory statistics are as follows:
■ show redundancy peer-system statistics
■ show redundancy peer-process cpu
■ show redundancy peer-process memory
MIB CISCO-LWAPP-HA-MIB.my is updated to capture these statistics.
Peer statistics on the GUI can be viewed under Monitor > Redundancy > Peer Statistics.
Figure 35 Peer Process System Statistics
MIB CISCO-LWAPP-HA-MIB.my is updated to capture these statistics.
Figure 36 Peer Process CPU Statistics
Figure 37 Peer Process Memory Statistics
■Cisco WLAN Controller Information: http://www.cisco.com/c/en/us/products/wireless/4400-series-wireless-lan-controllers/index.html http://www.cisco.com/c/en/us/products/wireless/2000-series-wireless-lan-controllers/index.html
■Cisco NCS Management Software Information: http://www.cisco.com/c/en/us/products/wireless/prime-network-control-system-series-appliances/
index.html
■Cisco MSE Information: http://www.cisco.com/c/en/us/products/wireless/mobility-services-engine/index.html
■Cisco LAP Documentation: http://www.cisco.com/c/en/us/products/wireless/aironet-3500-series/index.html
■Management—Management Interface
■WiSM-2—Wireless Service Module
■VACL—VLAN Access Control List
■DFC—Distributed Forwarding Card
■OIR—Online Insertion and Removal
■ISSU—In Service Software Upgrade
■MEC—Multichassis Ether Channel
■IDSM—Intrusion Detection Service Module