High Availability (SSO) Deployment Guide

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

• WLCs 5500 Series, 7500/8500 Series, and WiSM-2
• APs 700, 1130, 1240, 1250, 1040, 1140, 1260, 1600, 2600, 3500, 3600 Series APs, and 1520 or 1550 Series Mesh APs (MAPs).

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Topology

This document uses this network topology.

HA Connectivity Using Redundant Port on the 5500/7500/8500 WLC

• 5500/7500/8500 WLCs have a dedicated Redundancy Port which should be connected back to back in order to synchronize the configuration from the Active to the Standby WLC.
• Keep-alive packets are sent on the Redundancy Port from the Standby to the Active WLC every 100 msec (default timer) in order to check the health of the Active WLC.
• Both the WLCs in HA setup keep track of gateway reachability. The Active WLC sends an Internet Control Message Protocol (ICMP) ping to the gateway using the Management IP address as the source, and the Standby WLC sends an ICMP ping to the gateway using the Redundancy Management IP address. Both the WLCs send an ICMP ping to the gateway at a one-second interval.
• It is highly recommended to have back-to-back direct connectivity between Redundant Ports.

Here you can see the Redundant Port Connectivity between 5500 WLCs in an HA Setup:

Here you can see the Redundant Port Connectivity between Flex 7500 WLCs in an HA setup:

Note A direct physical connection between Active and Standby Redundant Ports is highly recommended. The distance between the connections can go up to 100 meters at per Ethernet cable standards.

High Availability Connectivity Using Redundant VLAN on WiSM-2 WLC

• WiSM-2 WLCs have a dedicated Redundancy VLAN which is used to synchronize the configuration from the Active WLC to the Standby WLC.
• A Redundancy VLAN should be a Layer 2 VLAN dedicated for the HA Pairing process. It should not be spanned across networks and should not have any Layer 3 SVI interface. No data VLAN should be used as a Redundancy VLAN.
• Keep-alive packets are sent on Redundancy VLAN from the Standby WLC to the Active WLC every 100 msec (default timer) in order to check the health of the Active WLC.
• Both the WiSMs in a HA setup keep track of gateway reachability. Active WLC sends an ICMP ping to the gateway using the Management IP address as the source, and the Standby WLC sends an ICMP ping to the gateway using the Redundancy Management IP address. Both the WLCs send an ICMP ping to the gateway at a one-second interval.
• To achieve HA between WiSM-2, it can be deployed in a single chassis or can also be deployed between multiple chassis using VSS as well as by extending Redundancy VLAN between two chassis.

This diagram shows HA Connectivity in a single chassis and extending Redundancy VLAN in a multiple chassis VSS setup:

Warning The Redundancy VLAN should be a non routable VLAN. In other words, no layer 3 interface should be created for this VLAN and can be allowed on VSL Link to extend HA setup between multiple chassis in VSS setup. It is important to make sure this VLAN is dedicated for the HA process and is not part of any Data VLAN, or else it may result in unpredictable results.

Note The Redundancy VLAN should be created like any normal Data VLAN on IOS® switches. Redundancy VLAN is configured for redundant port on WiSM-2 blades connected to a backplane. There is no need to configure an IP address for the Redundancy VLAN as it will receive an auto-generated IP which is discussed later in this document.

Note On Cisco WiSM2 and Cisco Catalyst 6500 Series Supervisor Engine 2T, if HA is enabled, post switchover, the APs might disconnect and reassociate with the WiSM2 controller. To prevent this from occurring, before you configure HA, we recommend that you verify–in the port channel–the details of both the active and standby Cisco WiSM2 controllers, that the ports are balanced in the same order, and the port channel hash distribution is using fixed algorithm. If they are not in order, you must change the port channel distribution to be fixed and reset Cisco WiSM2 from the Cisco Catalyst 6500 Series Supervisor Engine 2T. You can use the command show etherchannel port-channel to verify the port channel member order and load value. You can use the config command port-channel hash-distribution fixed to make the distribution fixed.

Note To support the active and standby WLCs in different data centers, in release 7.5, back-to-back redundancy port connectivity between peers is no longer mandatory and the redundancy ports can be connected via switches such that there is L2 adjacency between the two controllers. See Redundancy Port Connectivity in 7.5 for more information.

Introduction of New Interfaces for HA Interaction

Redundancy Management Interface

The IP address on this interface should be configured in the same subnet as the management interface. This interface will check the health of the Active WLC via network infrastructure once the Active WLC does not respond to Keepalive messages on the Redundant Port. This provides an additional health check of the network and Active WLC, and confirms if switchover should or should not be executed. Also, the Standby WLC uses this interface in order to source ICMP ping packets to check gateway reachability. This interface is also used in order to send notifications from the Active WLC to the Standby WLC in the event of Box failure or Manual Reset. The Standby WLC will use this interface in order to communicate to Syslog, the NTP server, and the TFTP server for any configuration upload.

Redundancy Port

This interface has a very important role in the new HA architecture. Bulk configuration during boot up and incremental configuration are synced from the Active WLC to the Standby WLC using the Redundant Port. WLCs in a HA setup will use this port to perform HA role negotiation. The Redundancy Port is also used in order to check peer reachability sending UDP keep-alive messages every 100 msec (default timer) from the Standby WLC to the Active WLC. Also, in the event of a box failure, the Active WLC will send notification to the Standby WLC via the Redundant Port. If the NTP server is not configured, a manual time sync is performed from the Active WLC to the Standby WLC on the Redundant Port. This port in case of standalone controller and redundancy VLAN in case of WISM-2 will be assigned an auto generated IP Address where last 2 octets are picked from the last 2 octets of Redundancy Management Interface (the first 2 octets are always 169.254).

Configure HA from the CLI

Complete these steps:

1. Before you configure HA, it is mandatory to have both the controllers' management interface in the same subnet:

WLC 1:

WLC 2:

2. HA is disabled by default. Before you enable HA, it is mandatory to configure the Redundancy Management IP Address and Peer Redundancy Management IP Address. Both the interfaces should be in the same subnet as the Management Interface. In this example, 9.6.61.21 is the Redundancy Management IP Address for WLC 1, and 9.6.61.23 is the Redundancy Management IP Address for WLC 2. It also needs to be configured so that 9.6.61.23 is the Redundancy Management IP Address of WLC 2 and 9.6.61.21 is the Redundancy Management IP Address of WLC 1.

Use this CLI in order to configure the Redundancy and Peer Redundancy Management IP Address:

WLC 1:

WLC 2:

3. Configure one WLC as Primary (by default, the WLC HA Unit ID is Primary and should have a valid AP-BASE count license installed) and another WLC as Secondary (AP base count from the Primary WLC will be inherited by this unit) using the CLI in this step. In this example, WLC 1 is configured as Primary, and WLC 2 is configured as Secondary:

WLC 1:

WLC 2:

Note You do not need to configure the unit as Secondary if it is a factory ordered HA SKU that can be ordered from release 7.3 onwards. A factory ordered HA SKU is a default Secondary unit, and will take the role of the Standby WLC the first time it is paired with an Active WLC that has a valid AP Count License.

If you want to convert any existing WLC as a Standby WLC, do so using the config redundancy unit secondary command in the CLI. This CLI command will only work if the WLC which is intended to work as Standby has some number of permanent license count. This condition is only valid for the 5500 WLC, where a minimum of 50 AP Permanent licenses are needed to be converted to Standby. There is no restriction for other WLCs such as the WiSM2, 7500, and 8500.

4. After the WLCs are configured with Redundancy Management and Peer Redundancy Management IP Addresses and Redundant Units are configured, it is time to enable SSO. It is important to make sure that physical connections are up between both the controllers (that is, both the WLCs are connected back to back via the Redundant Port using an Ethernet cable) and the uplink is also connected to the infrastructure switch and the gateway is reachable from both the WLCs before SSO is enabled.

Once SSO is enabled, it will reboot the WLCs. While it boots, the WLCs negotiate the HA role as per the configuration via Redundant Port. If the WLCs cannot reach each other via Redundant Port or via the Redundant Management Interface, the WLC configured as Secondary may go in to Maintenance Mode. Maintenance Mode is discussed later in this document.

5. Use the CLI in this step in order to enable AP SSO. Remember that enabling AP SSO will initiate a WLC reboot.

WLC 1:

WLC 2:

6. Enabling SSO will reboot the WLCs in order to negotiate the HA role as per the configuration performed. Once the role is determined, configuration is synced from the Active WLC to the Standby WLC via the Redundant Port. Initially, the WLC configured as Secondary will report XML mismatch and will download the configuration from Active and reboot again. During the next reboot after role determination, it will validate the configuration again, report no XML mismatch, and process further in order to establish itself as the Standby WLC.

These are the boot-up logs from both the WLCs:

WLC 1:

WLC 2 on first reboot after enabling SSO:

Note Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.

WLC 2 on second reboot after downloading XML configuration from Active:

7. After SSO is enabled, WLC is rebooted, and the XML configuration is synced, WLC 1 will transition its state to Active and WLC 2 will transition its state to Standby HOT. From this point onwards, GUI/Telnet/SSH for WLC 2 on the management interface will not work, as all the configurations and management should be done from the Active WLC. If required, the Standby WLC (WLC 2, in this example) can only be managed via the Console or Service Port.

Also, once the Peer WLC transitions to the Standby Hot state, -Standby keyword is automatically appended to the Standby WLCs prompt name.

8. Complete these steps in order to check the redundancy status:

a. For WLC 1, go to Monitor > Redundancy > Summary:

b. For WLC 2, go to Console connection:

Note Once SSO is enabled, the Standby WLC can be accessed via console connection or via SSH on the service port and on the redundant management interface.

Disabling SSO on HA Pair

1. On primary controller, disable SSO using the command:

Config redundancy mode disable

The Active and Standby WLCs reboot once this command is executed.

The standby controller, when it comes back after the reboot, has the same IP address on interfaces as the primary controller and all the ports disabled.

2. On the standby controller, re-enter the correct IP addresses corresponding to the management and dynamic interfaces and execute the following command:

Config port adminmode all enable

3. Save the configuration on the controller.

4. To re-enable SSO, execute the command Config redundancy sso on the primary and secondary controllers.

Both controllers reboot and pair up in the SSO mode. The standby will sync its configuration from the primary and come back in Hot-standby mode.

Upgrade Procedure in HA Setup

Complete these steps:

1. After the WLCs are configured in the HA setup, the Standby WLC cannot be upgraded directly from the TFTP/FTP server.

2. Initiate upgrade on the Active WLC in the HA setup via CLI/GUI, and wait for the upgrade to finish.

3. Once the Active WLC executes all the upgrade scripts, it will transfer the entire image to the Standby WLC via the Redundant Port.

4. When the Standby WLC receives the image from the Active WLC, it will start executing the upgrade scripts. The transfer of the image to standby and the execution of the upgrade scripts on the Standby WLC can be seen on the Active WLC Console/Telnet/SSH/Http connection.

5. After a successful message of Standby Upgrade is observed on the Active WLC, it is important to issue the show boot command on the Active WLC in order to make sure the new image is set as the primary image.

6. Once verified, initiate primary image pre-download on the Active WLC in order to transfer the new image to all the APs in the network.

7. After pre-image is completed on all the APs, issue the show AP image all command in order to verify that the primary image on the WLC is set as the backup image on APs.

8. Initiate swap option to interchange the backup image as primary on the APs. With this implementation, the WLC's and AP's primary image is set to the new image.

9. Issue the schedule-reset command as per planned outage with the no swap option in order to reset the APs and WLCs so that they can boot with the new image.

10. All the APs will reboot and join the new Active WLC.

11. Issue the show boot, show sysinfo, show ap image all, and show redundancy summary commands in order to verify that both the WLCs and APs have booted with the new image.

Important Guidelines before Initiating a WLC Upgrade in HA Setup

• Service Upgrade is not supported in this release, so network downtime should be planned before you upgrade the WLCs in the HA setup.
• The peer should be in the Hot Standby state before you start the upgrade in the HA setup.
• It is recommended to reboot both the WLCs almost together after upgrade so that there is no software version mismatch.
• Schedule Reset applies to both the WLCs in the HA setup.
• The Standby WLC can be rebooted from the Active WLC using the reset peer-system command if a scheduled reset is not planned.
• Debug transfer can be enabled on the Active WLC as well as the Standby WLC.
• If Active WLC unexpectedly reboot between software download and reboot both WLCs, you need to reboot both WLCs in order to complete software upgrade.

Download/Upload Facts in HA Setup

• No direct download and upload configuration is possible from the Standby WLC.
• All download file types like Image, Configuration, Web-Authentication bundle, and Signature Files will be downloaded on the Active WLC first and then pushed automatically to the Standby WLC.
• Once the configuration file is downloaded on the Active WLC, it is pushed to the Standby WLC. This results in the reset of the Standby WLC first, followed by the reset of the Active WLC.
• The Peer Service Port and Static route configuration is a part of a different XML file, and will not be applied if downloaded as part of the configuration file.
• The download of certificates should be done separately on each box and should be done before pairing.
• Uploading different file types like Configuration, Event Logs, Crash files, and so forth can be done separately from the Standby WLC. However, the CLI to configure different parameters for upload like Server IP, file type, path and name should be done on the Active WLC. Once the upload parameters are configured on the Active WLC, the transfer upload peer-start command should be issued on the Active WLC in order to initiate the upload from the Standby WLC.
• The service port state will be synced from the Active WLC to the Standby WLC. That is, if DHCP is enabled on the Active WLC service port, the Standby WLC will also use DHCP for getting the service port IP address. If the service port of the Active WLC is configured with a Static IP Address, the Standby WLC also needs to be configured with a different Static IP Address. The CLI to configure the IP Address for the Standby WLC service port is configure redundancy interface address peer-service-port <IP Address>. This command should be executed from the Active WLC. Also, in order to configure the route on the Standby WLC for out-of-band management on the service port, issue the configure redundancy peer-route add <Network IP Address > <IP Mask> <Gateway> command from the Active WLC.

Steps to Simulate Box Failover

Complete these steps:

1. Complete the steps as explained in the configuration section in order to configure HA between two WLCs, and make sure before force switchover is initiated that both the WLCs are paired up as the Active WLC and the Standby WLC.

For WLC 1:

For WLC 2, go to Console connection:

2. Associate an AP to the WLC and check the status of the AP on both the WLCs. In the HA setup, a mirror copy of the AP database is maintained on both the WLCs. That is, APs CAPWAP state in maintained on Active as well as Standby WLC (only for APs which are in Run state) and when switchover is initiated, the Standby WLC takes over the network. In this example, WLC 1 is an Active WLC, WLC 2 is in a Standby state, and the AP database is maintained on both the WLCs.

WLC 1:

WLC 2

3. Create an open WLAN and associate a client to it. The client database is not synced on the Standby WLC, so the client entry will not be present on the Standby WLC. Once the WLAN is created on the Active WLC, it will also be synced to the Standby WLC via the Redundant Port.

WLC 1:

WLC 2:

4. Issue the redundancy force-switchover command on the Active WLC. This command will trigger a manual switchover where the Active WLC will reboot and the Standby WLC will take over the network. In this case, the client on the Active WLC will be de-authenticated and join back on the new Active WLC.

WLC 1:

WLC 2:

Note Observe that the prompt in this example changed from 5508-Standby to 5508. This is because this WLC is now the Active WLC and the time taken for AP switchover is 1 msec.

WLC 2:

Observe the AP CAPWAP State on WLC 2, which was the Standby WLC initially and is now the Active WLC after switchover. AP Up Time as well as Association Up Time is maintained, and the AP did not go in to the discovery state.

These matrices provide a clear picture of what condition the WLC Switchover will trigger:

Maintenance Mode

There are few scenarios where the Standby WLC may go into Maintenance Mode and not be able to communicate with the network and peer:

• Non reachability to Gateway via Redundant Management Interface
• WLC with HA SKU which had never discovered peer
• Redundant Port is down
• Software version mismatch (WLC which boots up first goes into active mode and the other WLC in Maintenance Mode)

Note The WLC should be rebooted in order to bring it out of Maintenance Mode. Only the Console and Service Port is active in Maintenance Mode.

One WLC has a valid AP Count license and the other WLC has a HA SKU UDI

• HA SKU is a new SKU with a Zero AP Count License.
• The device with HA SKU becomes Standby the first time it pairs up.
• AP-count license info will be pushed from Active to Standby.
• On event of Active failure, HA SKU will let APs join with AP-count obtained and will start 90-day countdown. The granularity of this is in days.
• After 90-days, it starts nagging messages. It will not disconnect connected APs.
• With new WLC coming up, HA SKU at the time of paring will get the AP Count:

– If the new WLC has a higher AP count than the previous, the 90-day counter is reset.

– If the new WLC has a lower AP count than the previous, the 90-day counter is not reset.

– In order to lower AP count after switchover, the WLC offset timer will continue and nagging messages will be displayed after time expiry.

• Elapsed time and AP-count will be remembered on reboot.
• The factory default HA-SKU controller should not allow any APs to join.

Both the WLCs have a valid AP Count license

• The CLI should be used to configure one WLC as the Standby WLC (as mentioned in the configuration section) provided it satisfies the requirement of minimum permanent license count. This condition is only valid for the 5500 WLC, where a minimum of 50 AP Permanent licenses are needed to be converted to Standby. There is no restriction for other WLCs such as the WiSM2, 7500, and 8500.
• AP-count license information will be pushed from Active to Standby.
• In the event of a switchover, the new Active WLC will operate with the license count of the previous Active WLC and will start the 90-day countdown.
• The WLC configured as Secondary will not use its own installed license, and only the inherited license from the active will be utilized.
• After 90-days, it starts nagging messages. It will not disconnect connected APs.
• With the new WLC coming up, HA SKU at the time of paring will get the AP Count:

– If the new WLC has a higher AP count than the previous, the 90-day counter is reset.

– If the new WLC has a lower AP count than the previous, the 90-day counter is not reset.

– After switchover to a lower AP count, the WLC offset timer will continue and nagging messages will be displayed after time expiry.

One WLC has an Evaluation license and the other WLC has a HA SKU UDI or Permanent license

• The device with HA SKU becomes the Standby WLC the first time it pairs up with an existing Active WLC running Evaluation License. Or, any WLC running a permanent license count can be configured as the Secondary unit using the CLI configuration provided if it satisfies the requirement of minimum permanent license count. This condition is only valid for the 5500 WLC, where a minimum of 50 AP Permanent licenses are needed to be converted to Standby. There is no restriction for other WLCs such as the WiSM2, 7500, and 8500.
• AP-count license information will be pushed from Active to Standby.
• In the event of a switchover, the new Active WLC will operate with the license count of the previous Active WLC and start the 90-day countdown.
• After 90-days, it starts nagging messages. It will not disconnect connected APs.
• With new the WLC coming up, HA SKU at the time of paring will get the AP Count:

– If the new WLC has a higher AP count than the previous, the 90-day counter is reset.

– If the new WLC has a lower AP count than the previous, the 90-day counter is not reset.

– After switchover to a lower AP count, the WLC offset timer will continue and nagging messages will be displayed after time expiry.

Redundancy Port Connectivity in 7.5

• In controller release 7.3 and 7.4, back-to-back connectivity through redundancy port restrains the active and standby controllers to be in different locations. There are two mandatory interfaces for redundancy, redundancy port and redundancy management interface. Redundancy port uses dedicated physical port eth1 (similar to service port). It is used for all redundancy communication (AP, Client data, configuration sync, keep-alive messages and role negotiation messages). Redundancy management interface is used to check for the reachability of the peer and management gateway.
• To support the active and standby WLCs in different data centers, in release 7.5, back-to-back redundancy port connectivity between peers is no longer mandatory and the redundancy ports can be connected via switches such that there is L2 adjacency between the two controllers.
• Backward compatibility for release 7.3/7.4 will be supported, wherein back-to-back redundancy port connectivity is used for redundancy communication between the WLCs and the redundancy management interface is used to check the reachability to the peer and to management gateway.
• No additional configuration change is required for redundancy port and the configuration remains the same as in 7.3/7.4 release.

Supported HA Topologies in Release 7.5

5500/7500/8500 Series Controllers

1. Back-to-back Redundancy Port (RP) connectivity between the two WLCs, Redundancy Management Interface (RMI) connectivity to check peer and management gateway reachability.

2. RP connectivity with L2 adjacency between the two WLCs, RMI connectivity to check peer and management gateway reachability. This can be within the same or different data centers.

3. Two 5508, 7500 or 8500 connected to a VSS pair. Primary WLC connected to one 6500 and the Stand-by WLC to the other 6500.

Back-to-back RP Connectivity

Figure 1 Back-to-back RP connectivity

• This is the same topology as was supported in controller release 7.3.
• Configuration Sync and Keepalive messages are sent via Redundancy Port.
• RMI interface is created as part of Management subnet and is used to check peer and management gateway reachability.
• RTT Latency is 80 milliseconds by default. The RTT should be 80% of the Keepalive timer which is configurable in the range 100-400 milliseconds.
• Failure detection time is 3*100 + 60 + jitter (12 msec) = ~400 msec

Note In the above equation, 3 is the Keepalive retry count, 100 is the Keepalive timer, and 60 is
3*10 + 3*10 (3 RMI pings to peer + 3 pings to gateway).

• Bandwidth: 60 Mbps or more
• MTU: 1500

Configuration on Primary WLC:

configure interface address management 9.5.56.2 255.255.255.0 9.5.56.1

configure interface address redundancy-management 9.5.56.10 peer-redundancy-management 9.5.56.11

configure redundancy unit primary

configure redundancy mode sso

Configuration on Hot Standby WLC:

configure interface address management 9.5.56.3 255.255.255.0 9.5.56.1

configure interface address redundancy-management 9.5.56.11 peer-redundancy-management 9.5.56.10

configure redundancy unit secondary

configure redundancy mode sso

RP Connectivity via Switches

Figure 2 RP connectivity via switches

• Redundancy Port connectivity via switches across data centers is supported in this topology.
• Configuration sync and Keepalives via Redundancy Port.
• RMI interface is created as part of Management subnet and is used to check peer and management gateway reachability.
• RTT Latency is 80 milliseconds by default. The RTT should be 80% of the Keepalive timer which is configurable in the range 100-400 milliseconds.
• Failure detection time is 3*100 + 60 + jitter (12 msec) = ~400 msec
• Bandwidth: 60 Mbps or more
• MTU: 1500

Configuration on Primary WLC

configure interface address management 9.5.56.2 255.255.255.0 9.5.56.1

configure interface address redundancy-management 9.5.56.10 peer-redundancy-management 9.5.56.11

configure redundancy unit primary

configure redundancy mode sso

Configuration on Hot Standby WLC

configure interface address management 9.5.56.3 255.255.255.0 9.5.56.1

configure interface address redundancy-management 9.5.56.11 peer-redundancy-management 9.5.56.10

configure redundancy unit secondary

configure redundancy mode sso

5508, 7500 or 8500 Connected to VSS Pair

Figure 3 WLCs connected to VSS Pair

Supported HA Topologies for WiSM2 Controllers

WiSM2 in the Same Chassis

Figure 4 WiSM2 in Single Chassis

WiSM2 in Different Chassis: Redundancy VLAN over L2 Network

Figure 5 WiSM2 connectivity using Redundancy VLAN over L2 network

Configuration on Cat6k for WiSM2

wism service-vlan 192 ( service port VLAN)

wism redundancy-vlan 169 ( redundancy port VLAN)

wism module 6 controller 1 allowed-vlan 24-38 ( data VLAN)

WiSM2 HA configuration remains the same.

WiSM2 in Different Chassis: VSS Pair

Figure 6 WiSM2 connectivity using VSS Pair

Figure 7 Active and Standby VSS Pair connected via VSL Link

Figure 8 WiSM2 connectivity using VSS Pair

VSS Configuration

Recommendations

• Round trip latency on Redundancy Link should be less than or equal to 80 milliseconds.
• Preferred MTU on Redundancy Link is 1500 or above.
• Bandwidth on Redundancy Link should be 60 Mbps or more.
• If redundancy ports are connected via switches such that there is L2 adjacency between the two controllers, the RP VLAN should be excluded from the access VLAN configured on the switch for the management ports.
• For WiSM2 connectivity between two different chassis connected across the L2 network, the “redundancy-vlan” should be excluded from the access-VLAN configured on the switch for the management ports.
• It is highly recommended to use different sets of switches for the RP port connectivity and the management port traffic to avoid an Active-Active scenario.
• When deploying WiSM2 in VSS setup, it is recommended to set the peer search time to 180 seconds.

Client SSO (Client Stateful Switchover)

To support High Availability without impacting the service, there needs to be support for seamless transition of the clients and APs from the active controller to the standby controller. Release 7.5 supports Client Stateful Switch Over (Client SSO) in Wireless LAN controllers. Client SSO will be supported for clients, which have already completed the authentication and DHCP phase and have started passing traffic. With Client SSO, the client’s information is synced to the Standby WLC when client associates or the client parameters change. Fully authenticated clients, i.e. ones in Run state, are synced to the Standby and thus, client re-association is avoided on switchover making the failover seamless for the AP as well as for the client.

• Client SSO will work with Anchor-Foreign mobility setup as well as Guest Anchor scenarios.
• L3 MGIDs are synced to the Standby Controller.
• The failover time varies from ~2-996 milliseconds depending on the category of box failover.
• The management gateway failover time is in the order of ~15 seconds, which is the time taken for 12 pings to the management gateway.
• The default RTT latency between the two WLCs is 80 milliseconds. RTT latency should be less than or equal to 80% of the Keepalive timer. The Keepalive timer is configurable in the range 100-400 milliseconds

Configuration

1. Before configuring HA it is mandatory to have both the controllers’ management interface in same subnet.

WLC 1:

WLC 2:

2. HA is disabled by default. Before enabling HA it is mandatory to configure Redundant Management IP address and Peer Redundant Management IP address. Both the interfaces should be in same subnet as Management Interface.

To configure Redundant Management and Peer Redundant Management IP address click Controller tab > Redundancy > Global Configuration and enter IP address in both the fields and then click Apply.

WLC 1:

WLC 2:

3. Now configure one controller as Primary and another controller as Secondary from Redundant Unit drop-down. In an example below WLC 1 is configured as Primary Unit and WLC 2 is configured as Secondary Unit (will work as HA SKU UDI). While pairing, the controller that is configured as Primary will push its AP Count License to Standby WLC. To configure one controller as Primary unit and second controller as Secondary unit, click Controller tab > Redundancy > Global Configuration and select Primary/Secondary from Redundant Unit drop-down list and then click Apply.

WLC 1:

WLC 2:

4. After controllers are configured with Redundant Management, Peer Redundant Management IP address and Redundant Units are configured, it is very important to make sure physical connection are up between both the controllers i.e. both the WLCs are connected via Redundant Port using Ethernet cable and uplink is also connected to infrastructure switch and gateway is reachable from both the WLCs. Initiate ping to management interface gateway IP Address from both the controllers and make sure reachability to management gateway is fine.

5. To enable SSO navigate to Controller >Redundancy > Global Configuration and select the Enable option from SSO drop-down list on both the WLCs and click Apply. This step will make controllers reboot.

WLC 1:

WLC 2:

6. Enabling SSO will reboot controllers to negotiate HA role as per configuration and once the role is determined, configuration is synced from Active to Standby WLC via the redundant port. Initially controller configured as Secondary will report XML mismatch after downloading the configuration from Active and reboot again. In next reboot after role determination it will validate the configuration again and will report no XML mismatch and will process further to establish itself as Standby WLC. Thus, controller configured as Primary will reboot once and controller configured as Secondary will reboot twice.

WLC 1:

WLC 2 on first reboot after enabling SSO:

WLC 2 on second reboot after downloading XML configuration from Active:

While WLC2 is booting up, no configuration change is allowed on WLC1:

7. After SSO is enabled followed by controller reboots and XML configuration is synced, WLC 1 will transition its state as Active and WLC 2 will transition its state as Standby HOT. From this point onwards GUI/Telnet/SSH for WLC 2 on management interface will not work, as all the configurations should be done from the Active controller. Standby controller i.e. WLC 2 in this case if required can only be managed via Console or Service Port.

Also once Peer WLC transitions to Standby Hot state, -Standby keyword is automatically appended to Standby WLC’s prompt name.

8. To check the redundancy status

WLC 1 -> Click Monitor > Redundancy > Summary:

WLC 1 -> show redundancy summary:

WLC 2 -> show redundancy summary:

AP And Client State Sync

1. At this stage both the controllers are paired up in HA setup. Any configuration done on Active will be synced to Standby controller via redundant port. Check the WLAN summary and Interface summary on standby WLC from console connection.

2. In High Availability setup, APs’ CAPWAP state in maintained on Active as well as Standby controller (only for APs which are in Run state) i.e. UP time and Associated UP time is synced from the active to the standby controller. In an example below WLC 1 is an Active state and serving the network and WLC 2 is in Standby state monitoring active controller. Although WLC 2 is in standby state it still maintains CAPWAP state of AP.

WLC 1->Console Connection:

Observe the AP UP Time and Association UP Time on Active WLC

WLC 2->Console Connection:

Observe the AP Uptime and Association UP Time on Standby WLC will be in sync with active WLC.

3. In case of Box Failover i.e. Active controller crashes / system hang / manual reset / force switchover direct command is sent from Active controller via Redundant Port as well as from Redundant Management Interface to Standby controller to take over the network. Failover may take ~2-360 millisecond depending on number of APs/Clients on the active controller. In case of power failure on Active WLC or some crash where direct command for switchover cannot be sent to the standby controller, it may take ~360 – 990 msec depending upon number of APs/Clients on the active controller and the Keepalive timer configured. The default Keepalive timer is 100 milliseconds. Make sure that default RTT latency is less than or equal to 80 msec.

4. With release 7.5 as part of Client SSO, the client database is also synced to standby WLC so Run state client entries will be present on Standby WLC.

WLC 1-> Console/Telnet/SSH Connection:

Client entry is present on Active WLC.

WLC2-> Console Connection:

Client entry is present on Standby WLC.

5. PMK cache is also synced between the two controllers

WLC 1:

WLC 2:

Failover Process

1. Issue a command redundancy force-switchover on Active controller. This command will trigger manual switchover where Active controller will reboot and Standby controller will take over the network. In this case Run state client on Active WLC will not be de-authenticated. The command save config is initiated before redundancy force-switchover command.

WLC 1-> Console Connection:

WLC 2-> Console Connection:

Observe the change in prompt in above screen capture.

WLC 2->Console Connection:

Observe the AP CAPWAP State on WLC 2 which was standby initially and is Active now after switchover. AP uptime as well as Association UP Time is maintained and AP did not go in discovery state.

2. Also notice client connectivity when switchover is initiated. Client will be not be de-authenticated.

Ping from wireless client to its gateway IP Address and management IP Address during switchover shows minimal loss.

3. To check the redundancy status

WLC 1 -> Console connection issue a command show redundancy summary:

WLC 2 -> Console connection issue a command show redundancy summary:

WLC 2-> Click on Monitor > Redundancy > Summary:

4. Initiate a force switchover again on current active WLC.

WLC, which was configured as Primary Unit, should now be active and WLC, which was configured as Secondary Unit i.e., WLC 2 should be in Hot Standby State.

WLC 2:

WLC 1: Make sure Local state should be Active and Unit should be Primary on WLC 1 after switchover:

Observe the switchover history. WLC maintains 10 switchover histories with switchover reason.

Client SSO Behavior and Limitations

• The Bonjour dynamic database comprising of the services and service providers associated with a service and the domain name database is synced to standby.
• Only clients that are in Run state are synced between the Active and Standby WLC. Client SSO does not support seamless transitions for clients that are in the process of associating/joining the controller. The clients in the transition phase will be de-authenticated after switchover and will need to rejoin the controller.
• Posture and NAC OOB are not supported if the client is not in Run state.
• WGB and the clients associated to the WGB need to be re-associated post switchover.
• CCX based applications need to be re-started post Switchover.
• New mobility is not supported.

• Client statistics are not synced.
• PMIPv6, NBAR, SIP static CAC tree are not synced, need to be re-learned after SSO.
• OEAP (600) clients are not supported.
• Passive clients need to be re-associated after SSO.
• Device and root certificates are not automatically synced to the Standby controller.
• AP and Client Rogue information is not synced to the Standby controller and needs to be re-learnt when the hot standby becomes the active controller.
• Sleeping client information is not synced to the standby controller.
• NBAR statistics are not synced to the secondary controller.
• Native Profiling data is not synced to the secondary controller, therefore, clients will be re-profiled after switchover.
• The below table captures the behavior w.r.t SSO with MAPs and RAPs.

Enhancements and Improvements

Bulk Sync Status

Currently, the controller does not provide any indication for the completion of Bulk Sync configuration once it is initiated. The Bulk Sync can be verified only by user observation and by manually checking the number of clients synced to the standby WLC. As part of this feature, a mechanism is provided to convey the status of Bulk Sync (both AP and client sync) when standby WLC comes up.

A new field called BulkSync Status is added in the GUI under Controller > Redundancy > Summary. This field points to the status of the bulk sync to the standby WLC and the status can be Pending/In-progress/Complete.

Figure 9 BulkSync Status GUI

The output of the CLI command show redundancy summary also displays the Bulk Sync status, which can be Pending/In-progress/Complete as shown below while pairing with the standby controller.

When the standby controller is booting up, the BulkSync status shows Pending.

Figure 10 BulkSync Status—Pending

Once the standby controller completes, the boot-up process and the bulk sync starts, the status changes to In-Progress.

Figure 11 BulkSync Status—In-Progress

When the bulk sync process is complete, the BulkSync status changes to Complete.

Figure 12 BulkSync Status—Complete

Debug/Show Command Enhancements

As HA plays a major role in avoiding network outage, it should also be pertinent to be able to debug the state changes on the boxes at the time of SSO or at a later point in time.

The following new categories of statistics are introduced under Monitor > Redundancy > Statistics:

a. All

b. Infra

c. Transport

d. Keep-Alive

e. GW-Reachability

f. Config-Sync

Figure 13 Redundancy Statistics GUI

The Infra statistics contain RF Client details and Sanity Counters as shown in Figure 14.

Figure 14 Redundancy Statistics—Infra

Figure 15 Redundancy Statistics —Transport

The Heartbeat debugs include events of reception of heartbeats, loss of heartbeats, and subsequent actions related to them.

Figure 16 Redundancy Keep-alive Statistics

The HA system monitors management gateway reachability to reduce network outage.

On the Standby controller, serviceability debugs related to the gateway reachability of the active controller and standby controller, their health states, and actions taken based on this information is reported. While on the active controller, the reachability of active WLC to the gateway alone is reported.

Figure 17 Redundancy GW-Reachability Statistics

Figure 18 Redundancy Config-Sync Statistics

The following debug/show CLI commands are introduced for this feature:

1. debug redundancy infra detail/errors/event

2. debug redundancy transport detail/errors/events/packet

3. debug redundancy keepalive detail/errors/events

4. debug redundancy gw-reachability detail/errors/events

5. debug redundancy config-sync errors/events/detail

6. debug redundancy ap-sync errors/events/detail

7. debug redundancy client-sync errors/events/detail

8. debug redundancy mobility events/errors/detail

9. show redundancy infra statistics

10. show redundancy transport statistics

11. show redundancy keepalive statistics

12. show redundancy gw-reachability statistics

13. show redundancy config-sync statistics

14. show redundancy ap-sync statistics

15. show redundancy client-sync statistics

Configurable Keep-alive and Peer-Search Parameters

To address the variable network latencies in different customer deployment scenarios, keep-alive and peer-search parameters are made configurable. As part of this enhancement, the maximum number of Keepalives between active and standby controllers to trigger a failover is now configurable. Also, peer-search timer and keep-alive timer are modified to support an extended range.

The following new CLI command is added to configure the number of redundancy keep-alive retries in the range of 3 to 10.

Figure 19 Redundancy retries CLI Command

The existing CLI command config redundancy timer keep-alive-timer of keep-alive timer is modified to support keep-alive timer from 100 to 1000 msecs.

The existing CLI command config redundancy timer peer-search-timer of peer-search timer is modified to support peer-search timer from 60 to 300 secs.

Figure 20 Redundancy timer CLI Command

The following CLI is introduced to view the redundancy keep-alive-retry value.

Figure 21 Show redundancy retries CLI Command

Use the show redundancy timers command to view the peer-search-timer and keep-alive-timer values.

Figure 22 Show redundancy timers CLI Command

Use the show redundancy detail command to display the keep-alive and peer-search timeout values.

Figure 23 Show redundancy detail CLI Command

The keep-alive timer, keep-alive retries, and peer-search timer can also be configured and viewed from the Controller > Redundancy > Global Configuration page in the GUI.

Figure 24 Redundancy Global Configuration GUI

Peer RMI ICMP Ping Replaced with UDP Messages

Prior to release 8.0, ICMP ping is used to heart-beat with the peer WLC over the Redundancy Management Interface. As part of this feature for release 8.0, ICMP ping is replaced with a UDP message.

This will benefit due to the following factors:

• ICMP ping packets might get discarded under heavy loads.
• Any other device with the same IP might also reply to the ping.

It is recommended to tag the RMI and management interfaces to avoid false Switchovers. Tagging of the RMI and management interfaces is now mandatory in release 8.0 to pair WLCs in SSO mode.

Standby WLC On-the-Fly Maintenance Mode (MTC)

Prior to release 8.0 when the standby controller looses reachability to the “Default Gateway” or “Peer RP”, the controller reboots and checks that condition while booting up and enters into the MTC mode. With this feature, the standby WLC will enter into the MTC mode “on-the-fly” without rebooting when such error scenarios occur. Once Peer-RP and the default gateway reachability is restored, the MTC mode auto-recovery mechanism introduced in release 7.6, will reboot the WLC and pair it with the active WLC. This mechanism is applicable only to the standby WLC. The active controller will still reboot before going to MTC mode.

Default Gateway Reachability Check Enhancement

As part of this enhancement, the gateway (GW) reachability check mechanism is modified to avoid false positives and it is also modified for the ideal time to start checking for gateway reachability once the controller boots up.

Prior to release 8.0, the “GW reachability check” is performed during Role negotiation. In release 8.0 and later, during Role negotiation, GW reachability check is not performed and is initiated only after the HA Pair-Up is complete.

Also, it is observed that certain Switch/Router configurations rate limits ICMP ping packets or drop them altogether. To avoid such conditions triggering false-positives, the new design ensures not to take switchover decisions purely based on ICMP ping losses. By the modified logic, upon 6 consecutive ping drops, an ARP request is sent to the GW IP address. A successful response to this request is considered as the GW being reachable.

Faster HA Pair Up

Currently during the HA pairing process, once the active-standby role is determined, the configuration is synced from the active WLC to the standby WLC via the Redundancy Port. If the configuration is different, the secondary WLC reports XML mismatch and downloads the configuration from the active controller and reboots again. In the next reboot after role determination, it validates the configuration again, reports no XML mismatch, and process further in order to establish itself as the Standby WLC.

With this feature enhancement, the XMLs are sent from the to-be-Active to to-be-Standby controller at the time of initialization, just before the validation of the XMLs. This avoids the extra step of comparison and reboot since no other modules are initialized yet, resulting in faster pair up of Active and Standby WLCs.

As seen in the boot logs below, there are no comparison of XMLs and no reboot of standby WLC.

Figure 25 Standby WLC bootup log

New Features Support in SSO

SSO Support for Internal DHCP Server

Prior to release 8.0, configuration of “Internal DHCP Server” is not allowed on HA enabled controllers because the internal DHCP server data is not synced to the standby WLC. In release 8.0 and later, “Internal DHCP Server” is configured on HA enabled controllers and this data is synced to the standby WLC so that soon after a switchover, the “Internal DHCP Server” on the new active controller starts serving clients.

To configure the Internal DHCP server using the GUI, navigate to Controller > Internal DHCP Server

Figure 26 Internal DHCP Server GUI

The same is synced to the standby controller and is verified by executing the CLI command show dhcp summary

Figure 27 show dhcp summary on Active and Standby WLC

AP Radio CAC Statistics Sync

As part of this enhancement, Static CAC method bandwidth allocation parameters for Voice and Video and Call Statistics are synced to the Standby WLC, so that soon after a switchover, respective information is available on the new active controller that will be used for call admission control.

SSO Support for Sleeping Clients

Release 7.5 did not provide SSO support for sleeping clients. The sleeping client database was not synced to the standby controller, which caused the sleeping clients to re-authenticate after a switchover occurred. With this release, sleeping client database is synced to the standby controller, allowing sleeping clients to avoid web re-authentication if they wake up within the sleeping client timeout interval.

The CLI command show custom-web sleep-client summary is used to verify the sleeping client database sync between the active and standby WLC.

Figure 28 Sleeping Client Database on Primary WLC

Figure 29 Sleeping Client Database on Standby WLC

Figure 30 Sleeping Client Details on Active and Standby WLC

SSO Support for OEAP 600 APs

Prior to release 8.0, when a switchover occurs on an HA pair, OEAP 600 APs restarts the CAPWAP tunnel and joins back the new active controller, and all the connected clients are de-authenticated. As a part of this feature, OEAP 600 APs ensure not to reset their CAPWAP tunnel. Also, clients continue their connection with the new active controller in a seamless manner.

As shown below, the output of show ap summary and show client summary command on the active and standby controllers displays the AP and client database sync.

Figure 31 OEAP 600 AP on Active WLC

Figure 32 OEAP 600 AP Sync to Standby WLC

Figure 33 Clients on Active WLC

Figure 34 Client Sync to Standby WLC