Field Notice: FN74222 - Full or Partial Cisco 9800 Series Wireless Controller Configuration Loss after High-Availability Stateful Switchover Failover - Software Upgrade Recommended

After a High-Availability Stateful Switchover (HA SSO) failover or reboot, a partial or complete configuration loss may be seen on Cisco C9800 Series Wireless Controllers (WLCs).

Observable symptoms include access points (APs) not being able to join the WLC and the WLC being unreachable or unable to be configured.

This is a problem specific to HA scenarios. Standalone controllers are not impacted.

This does not apply to standalone controllers in N+1 or other mobility scenarios.

Due to this software defect, if the Replication Manager (repm) process increases CPU utilization and the WLC reloads or undergoes a failover for any reason, the configuration may be lost.

Observable symptoms vary and may appear as follows:

• The WLC becomes unreachable.
• The WLC is not able to apply configuration changes. Doing so may result in the following error:

  Country <XX> is unsupported. Please configure supported country code.

• APs can no longer join the WLC.
• APs stop broadcasting some or all Service Set Identifiers (SSIDs).
• Total or partial configuration is missing.
• There is no telemetry.
• The WLC reverts to the Day 0 configuration in the CLI and the Day 0 wizard in the GUI.
• The repm process indicates high CPU utilization over the time (e.g., greater than 60% and increasing) before the failover or reboot. Use the following command to monitor the repm process:

  Device# show processes cpu platform | include repm
  
  Pid PPid 5Sec 1Min 5Min Status Size Name 
  -------------------------------------------------------------------------------- 
  17795 17785 83% 89% 89% S 910372 repm

• The running configurations for Standby and Active WLCs show differences. View and compare the two running wireless configurations using the following instructions:
  • Active: Use the show running-config command.
  • Standby: Use SSH to access the Redundancy Management Interface (RMI) and use the show running-config command.

You can proactively monitor for one or more of the symptoms below:

• Monitor the repm for high CPU usage: 

  show processes cpu platform | include repm
  Pid    PPid    5Sec    1Min    5Min  Status        Size  Name
  --------------------------------------------------------------------------------
   17795   17785     83%     89%     89%  S           910372  repm

• Monitor the flap count field of the show platform software replication chassis st R0 peer details command
• Monitor the Standby and Active chassis' running-configs for parity.
  • For Active, use show running-config.
  • For Standby, SSH to the RMI ip address of the standby controller and use show running-config.

If you have one or more of the preceding symptoms, you can do the following to recover:

1. Delete the persistent binary config file from both Active and Standby. All commands are issued in the Active WLC:

   delete /force /recursive bootflash:.dbpersist/persistent-config.tar.gz
   delete /force /recursive bootflash:.dbpersist/persistent-config.meta
   delete /force /recursive stby-bootflash:.dbpersist/persistent-config.tar.gz
   delete /force /recursive stby-bootflash:.dbpersist/persistent-config.meta

2. Reload the whole stack (Active and Standby together). This step will cause downtime for a few minutes until the Active controller reloads and comes back.
   Execute command on the active HA member:

   WLCx#reload
   Reload command is being issued on Active unit, this will reload the whole stack
   Proceed with reload? [confirm]

If you have high repm CPU usage, do the following to recover:

1. Delete the persistent binary config file from both Active and Standby. All commands are issued in the Active WLC:

   delete /force /recursive bootflash:.dbpersist/persistent-config.tar.gz
   delete /force /recursive bootflash:.dbpersist/persistent-config.meta
   delete /force /recursive stby-bootflash:.dbpersist/persistent-config.tar.gz
   delete /force /recursive stby-bootflash:.dbpersist/persistent-config.meta

2. Reload the whole stack (Active and Standby together). This step will cause downtime for a few minutes until the Active controller reloads and comes back.
3. Install the SMU for the hot fix. This SMU is available for Cisco IOS XE releases 17.9.5 and 17.12.3. For other software releases, contact the Cisco Technical Assistance Center (TAC).

To determine if a specific WLC is affected by this issue, confirm the software version the WLC is running and whether HA SSO is enabled.

To determine which software version a WLC is running, use the show version | include 9800|IOSXE  command.

To verify that an HA SSO pair is running on the device, use the show redundancy command.

If the output of the show redundancy command returns Hardware Mode = Duplex and the WLC is running an affected software version, the WLC is susceptible to this issue.