Troubleshoot High CPU on Switches with dot1x/Mab Due to EAP Framework and AAA Manager

Introduction

This document describes how to troubleshoot High CPU/memory due to  Extensible Authentication Protocol (EAP) framework and Authentication, Authorization, and Accounting (AAA) manager. This is seen on switches that use dot1x/mab authentication.

Background Information

The Cisco IOS Auth Manager handles network authentication requests and enforces authorization policies regardless of the authentication method. The Auth Manager maintains operational data for all port-based network connection attempts, authentications, authorizations, and disconnections and, serves as a session manager.

The switch acts as an intermediary (proxy) between the client and the authentication server, it requests identity information from the client, verifies that information with the authentication server, and relays a response to the client. The switch includes the RADIUS client, which encapsulates and decapsulates the EAP frames and interacts with the authentication server.

Configuration

This section shows a Cisco switch that does MAB/DOT1X (MAC AuthenticationBypass) authentication.

You should understand the concepts of port-based network access control and have an understanding of how to configure port-based network access control on your Cisco platform. This image illustrates workstations that have dot1x/MAB authentication.

This is of a sample configuration:

interface FastEthernet0/8
 switchport access vlan 23
 switchport mode access
 switchport voice vlan 42
 authentication host-mode multi-domain
 authentication order mab dot1x
 authentication priority mab dot1x---> Priority order 
 authentication port-control auto
 authentication periodic
 authentication timer reauthenticate <value in sec>---->(Time after which the client auth would be re-negotiated)
 authentication violation protect
 mab
 mls qos trust dscp
 dot1x pae authenticator
 dot1x timeout tx-period 3
 storm-control broadcast level 2.00
 no cdp enable
 spanning-tree portfast
 spanning-tree bpduguard enable
 service-policy input Marking
end

Troubleshoot

Switches that use dot1x/MAB authentication sometimes have high CPU/memory spikes due to the EAP Framework and AAA manager. This can impact the production since authentication requests are dropped.

In order to resolve this, these steps are recommended:

Step 1. Enter the show proc cpu sort command in order to check the high CPU usage on the switch and make sure that the EAP Framework and Auth manager processes have the highest usage as shown in this example:

PU utilization for five seconds: 

97%

/2%; one minute: 90%; five minutes: 89%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
149   178566915 140683416       1269 

64.04% 47.11% 45.63%   0 EAP Framework

141   130564594  55418491       2355 

21.61% 29.05% 29.59%   0 Auth Manager

121   305295906 487695245        519  1.74%  1.84%  1.78%   0 Hulc LED Process
144    12070918  31365536        384  0.63%  0.43%  0.49%   0 MAB Framework
258   117344878 885817567        132  0.47%  0.79%  0.86%   0 RADIUS

Step 2. Check the memory usage on the switch for processes like Auth Manager and RADIUS with the show process cpu memory command as shown in this example.

Processor Pool Total:   22559064 Used:   16485936 Free:    6073128
      I/O Pool Total:    4194304 Used:    2439944 Free:    1754360
Driver te Pool Total:    1048576 Used:         40 Free:    1048536

PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0   29936164   13273256   13856236          0          0 *Init*
   0   0   34797632   32603736    1091560    2481468     263240 *Dead*
  59   0     366860       6760     317940          0          0 Stack Mgr Notifi
141   0 

 569580564 3357129696

174176    2986956

          0 

Auth Manager

258   0 

1212276148 2456764884     140684   21066696

          0 

RADIUS

131   0  552345134 541235441      90736      20304          0 HRPC qos reque

Step 3. If you face high resource usage on the switch, you might see the following logs for the authentication failures as shown:

Enter the show logging command.

%DOT1X-5-FAIL: Authentication failed for client (7446.a04b.1495) on Interface Fa0/17 AuditSessionID 0A73340200000224870C28AA
%AUTHMGR-7-RESULT: 

Authentication result 'no-response'

 from 'dot1x' for client (7446.a04b.1495) on Interface Fa0/17 AuditSessionID 0A73340200000224870C28AA
%AUTHMGR-7-FAILOVER: Failing over from 'dot1x' for client (7446.a04b.1495) on Interface Fa0/17 AuditSessionID 0A73340200000224870C28AA

Step 4. Set the re-authenticate timer to a higher value (for example, 3600 seconds) in order to ensure that you do not authenticate frequently for the clients, which thereby increases the load on the switch.

In order to validate the configuration enter show run interface <interface-name> command:

interface FastEthernet0/8
switchport access vlan 23
switchport mode access
switchport voice vlan 42
authentication host-mode multi-domain
authentication order mab dot1x
authentication priority mab dot1x
authentication port-control auto
authentication periodic

authentication timer reauthenticate 60---------->Make sure we do not have any

 aggressive timers set
authentication violation protect

Step 5. Determine how many sessions are seen for MAB/dot1x processes, because sometimes a high number of authenticated sessions can also lead to high CPU. In order to check the number of active sessions, enter these commands:

SW#

show authentication registrations

Auth Methods registered with the Auth Manager:
  Handle  Priority  Name
    100       0      dot1x
    3        1      mab
    1        2      webauth

SW#Show authentication method dot1x

SW#Show authentication method mab

SW#Show authentication sessions

Step 6. In order to check the version and potential bugs, enter the show version command.

If the bug is not listed in the "Bugs" section, open a case with the Technical Assistance Center (TAC) and attach all of the logs from steps 1 to 5.

Bugs

CSCus46997 Memory Leak and High CPU in IP Host Track and Auth Manager

CSCtz06177 A catalyst 2960 may run low on memory.

CSCty49762 EAP Framework and AAA AttrL Sub Uses All Process Memory 

Tip: For further details, refer to Cisco bug IDs CSCus46997, CSCtz06177 and CSCty49762.