Understand AnyConnect Network Access Manager Logging

Introduction

This document describes how to enable AnyConnect Network Access Manager (NAM) logging as well as to collect and interpret the logs. The examples included in the document describe different authentication scenarios and the logs that reflect the steps taken by Network Access Manager to authenticate the client.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Enable NAM Logging

If an issue is identified that may be related to NAM module, the first step is to enable Extended Logging feature. This must be done on the client endpoint while NAM module is running.

Step 1. Open AnyConnect window and make sure it's in focus.

Step 2. Press this key combination, Left Shift + Left Alt + L. There is no response.

Step 3. Right click on AnyConnect icon in Windows System Tray. A menu pops up.

Step 4. Select Extended Logging so it has a check mark displayed. NAM now logs detailed debug messages.

Configure NAM Packet Capture

When Extended Logging is enabled, NAM also keeps a packet capture buffer going. The buffer is by default limited to about 1MB. If packet capture is needed, it may be beneficial to increase buffer size so it captures more activities. To extend the buffer, a XML setting file must be manually modified.

Step 1. On the Windows PC, browse to:
C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Network Access Manager\system\

Step 2. Open file internalConfiguration.xml.

Step 3. Locate XML tag <packetCaptureFileSize>1</packetCaptureFileSize> and adjust the value to 10 for a 10MB buffer size, and so on.

Step 4. Reboot the client PC for the change to take effect.

Log Collection

NAM log collection is done via Diagnostic And Reporting Tool (DART), which is a module of AnyConnect suite. In the installer, select a module and use AnnyConnect full installation ISO to install. The Cisco Media Services Interface (MSI) installer can also be found inside the ISO.

After you enable Extended Logging and perform a test, simply run DART and go through the dialogue, the log bundle is located by default on the Windows Desktop.

In addition to DART bundle, the NAM message log is also helpful to locate the relevant data in the NAM log. In order to find the NAM message log, navigate to AnyConnect settings window > Network Access Manager > Message History. The message log contains timestamp of each network connection event, which can be used to find the logs relevant to the event.

Reading NAM Logs

NAM logs, especially after you enable Extended Logging, contains a large amount of data, most of which are irrelevant and can be ignored. This section lists out the debug lines to demonstrate each step NAM takes to establish a network connection. When you work through a log, these key phrases may be helpful to locate part of the log relevant to the issue. 

Log Summary of a Network Connection without 802.1x Enabled Authentication

2016 17:20:37.974 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Network test123: AccessStateMachine current state = ACCESS_STOPPED, received userEvent = START

Explanation: This indicates that the user has selected a network from NAM module, and NAM has received a userEvent of START.

538: TESTPC: May 16 2016 17:20:37.974 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Network test123: AccessStateMachine new state = ACCESS_STARTED
539: TESTPC: May 16 2016 17:20:37.974 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Network test123: NetworkStateMachine current state USER_T_DISCONNECTED, received access event ACCESS_STARTED

Explanation: Both Access State Machine and Network State Machine have been started.

545: TESTPC: May 16 2016 17:20:37.974 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Ipv4 {EFDAF0F0-CF25-4D88-B125-E748CD539DFF}: received Cancel event [state: COMPLETE]

Explanation: The IPv4 instance got cancelled in order to reset the states.

547: TESTPC: May 16 2016 17:20:37.974 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: starting makeMatches...
549: TESTPC: May 16 2016 17:20:37.989 +0600: %NAM-6-INFO_MSG: %[tid=1412]: matching adapter {484E4FEF-392C-436F-97F0-CD7206CD7D48} and network test123 ...

Explanation: The adapter with ID 484E4FEF-392C-436F-97F0-CD7206CD7D48 was selected to connect to network test123, which is the name of the network connection configured in NAM.

551: TESTPC: May 16 2016 17:20:37.989 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Network test123: AccessStateMachine new state = ACCESS_ATTACHED
557: TESTPC: May 16 2016 17:20:37.989 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Network test123: AccessStateMachine current state = ACCESS_ATTACHED, received userEvent = CONNECT

Explanation: NAM has successfully engaged the adapter for this network. Now NAM tries to associate (connect) to this network (which happens to be wireless):

561: TESTPC: May 16 2016 17:20:37.989 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: ACE: adapter SM current: state(STATE_DISCONNECTED_LINK_DOWN), event(EVENT_CONNECT)
562: TESTPC: May 16 2016 17:20:37.989 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: ACE: adapter SM state change: STATE_DISCONNECTED_LINK_DOWN -> STATE_ASSOCIATING
567: TESTPC: May 16 2016 17:20:37.989 +0600: %NAM-6-INFO_MSG: %[tid=1412]: Starting wifi connection, trying ssid test123 ...
568: TESTPC: May 16 2016 17:20:37.989 +0600: %NAM-6-INFO_MSG: %[tid=1412]: Connection Association Started(openNoEncryption)

Explanation: openNoEncryption indicates that the network is configured as open. On the Wireless Lan Controller it uses MAC Authentication Bypass (MAB) to authenticate.

234: TESTPC: May 16 2016 17:20:38.020 +0600: %NAMSSO-7-DEBUG_MSG: %[tid=1912]: waiting for cs...

Explanation: cs can be seen a lot in NAM logs. These are irrelevant logs and should be ignored.

575: TESTPC: May 16 2016 17:20:38.020 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Network test123: NetworkStateMachine new state USER_T_DISCONNECTED 236: TESTPC: May 16 2016 17:20:38.020 +0600: %NAMSSO-7-DEBUG_MSG: %[tid=1912]: Tx CP Msg: <?xml version="1.0" encoding="UTF-8"?><SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ssc="http://www.cisco.com/ssc" encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> <SOAP-ENV:Body> <networkStateEvent> <sequenceNumber>16</sequenceNumber> <groupName>Local networks</groupName> <networkName>test123</networkName> <networkState>Associating</networkState> <adapterName>Intel(R) Centrino(R) Ultimate-N 6300 AGN</adapterName> <serverVerifiedName></serverVerifiedName> </networkStateEvent> </SOAP-ENV:Body></SOAP-ENV:Envelope>
 

Explanation: These are Simple Object Access Protocol (SOAP) messages used to tell AnyConnect GUI to display the connection status message such as Associating in this case. Any error messages displayed on NAM window can be found in one of the SOAP messages in the log which can be used to locate the issue easily.

582: TESTPC: May 16 2016 17:20:38.020 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: {484E4FEF-392C-436F-97F0-CD7206CD7D48} - Received STATE_AUTHENTICATED
583: TESTPC: May 16 2016 17:20:38.020 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: ACE: adapter SM current: state(STATE_ASSOCIATING), event(EVENT_AUTH_SUCCESS)

Explanation: NAM receives an AUTH_SUCCESS event, which misleads because there is no authentication which currently happened. You are get this event simply because you connect to an open network, so by default authentication is successful.

595: TESTPC: May 16 2016 17:20:38.738 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Network test123: AccessStateMachine current state = ACCESS_ASSOCIATING, received adapterState = associated

Explanation: Association to Service Set Identifier (SSID) is successful, time to handle authentication.

603: TESTPC: May 16 2016 17:20:38.754 +0600: %NAM-6-INFO_MSG: %[tid=1412][mac=1,6,3c:a9:f4:33:ab:50]: Authentication not required.
604: TESTPC: May 16 2016 17:20:38.754 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: ACE: adapter SM current: state(STATE_ASSOCIATED), event(EVENT_AUTH_SUCCESS)
605: TESTPC: May 16 2016 17:20:38.754 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: ACE: adapter SM state change: STATE_ASSOCIATED -> STATE_AUTHENTICATED

Explanation: Since this is an open network, it is by default authenticated. At this point, NAM is connected to the network and now starts DHCP process:

610: TESTPC: May 16 2016 17:20:38.754 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: {484E4FEF-392C-436F-97F0-CD7206CD7D48} creating a new DHCP work
612: TESTPC: May 16 2016 17:20:38.754 +0600: %NAM-6-INFO_MSG: %[tid=1412][mac=1,6,3c:a9:f4:33:ab:50]: {484E4FEF-392C-436F-97F0-CD7206CD7D48}: DHCP: Sending DHCP request
613: TESTPC: May 16 2016 17:20:38.754 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: queueing DHCP work
642: TESTPC: May 16 2016 17:20:40.830 +0600: %NAM-7-DEBUG_MSG: %[tid=1448]: Ipv4 {484E4FEF-392C-436F-97F0-CD7206CD7D48}: connectivity test[03]: IP:10.201.230.196(255.255.255.224) GW:10.201.230.193 [Success]
643: TESTPC: May 16 2016 17:20:40.830 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Ipv4 {484E4FEF-392C-436F-97F0-CD7206CD7D48}: received Success event [state: WAIT_FOR_CONNECTIVITY]
645: TESTPC: May 16 2016 17:20:40.845 +0600: %NAM-6-INFO_MSG: %[tid=1412][mac=1,6,3c:a9:f4:33:ab:50]: {484E4FEF-392C-436F-97F0-CD7206CD7D48}: IP Address Received: 10.201.230.196
646: TESTPC: May 16 2016 17:20:40.845 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Ipv4 Connectivity Result: SUCCESS

Explanation: NAM successfully acquires an IP address.

648: TESTPC: May 16 2016 17:20:40.845 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: ACE: adapter SM current: state(STATE_AUTHENTICATED), event(EVENT_IP_CONNECTIVITY)
649: TESTPC: May 16 2016 17:20:40.845 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: ACE: adapter SM state change: STATE_AUTHENTICATED -> STATE_CONNECTED

Explanation: Once an IP address is received NAM will send ARP (Address Resolution Protocol) request to the gateway (Get-Connectivity). Once the ARP response is received the client is connected.

Log Summary of a Network Connection using 802.1x and PEAP over Wired Network

1286: TESTPC: May 16 2016 17:55:17.138 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Network WiredPEAP: AccessStateMachine new state = ACCESS_STARTED
 

Explanation: NAM started to connect to network WiredPEAP.

1300: TESTPC: May 16 2016 17:55:17.138 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Binding adapter Intel(R) 82579LM Gigabit Network Connection and user auth for network WiredPEAP
1303: TESTPC: May 16 2016 17:55:17.138 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Network WiredPEAP: AccessStateMachine new state = ACCESS_ATTACHED
 

Explanation: NAM matched an adapter to this network.

1309: TESTPC: May 16 2016 17:55:17.138 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Network WiredPEAP: AccessStateMachine current state = ACCESS_ATTACHED, received userEvent = CONNECT
1342: TESTPC: May 16 2016 17:55:17.154 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: STATE (4) S_enterStateAux called with state = CONNECTING (dot1x_sm.c 142)
 

Explanation: NAM started connecting to this wired network.

1351: TESTPC: May 16 2016 17:55:17.154 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: 8021X (4) Sent start frame (dot1x_sm.c 117)
 

Explanation: Client sends EAPOL_START.

1388: TESTPC: May 16 2016 17:55:17.154 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: PORT (3) net: RECV (status: UP, AUTO) (portMsg.c 658)
1389: TESTPC: May 16 2016 17:55:17.154 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: 8021X (4) recvd EAP IDENTITY frame (dot1x_util.c 264)
1397: TESTPC: May 16 2016 17:55:17.154 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: EAP (0) EAP State: EAP_STATE_IDENTITY (eap_auth_client.c 940)
 

Explanation: Client receives Identity Request from the switch, it now looks for a credential to send back.

1406: TESTPC: May 16 2016 17:55:17.154 +0600: %NAM-7-DEBUG_MSG: %[tid=1464]: EAP-CB: credential requested: sync=8, session-id=1, handle=00AE1FFC, type=AC_CRED_SESSION_START
1426: TESTPC: May 16 2016 17:55:17.169 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: EAP: processing credential request: sync=8, session-id=1, eap-handle=00AE1FFC, eap-level=0, auth-level=0, protected=0, type=CRED_REQ_SESSION_START
1458: TESTPC: May 16 2016 17:55:17.169 +0600: %NAM-6-INFO_MSG: %[tid=1412]: Trying fast reauthentication for unprotected identity anonymous
1464: TESTPC: May 16 2016 17:55:17.169 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: EAP: credential request completed, response sent: sync=9
 

Explanation: By default, Anyconnect sends anonymous as unprotected identity (outter identity), so here it tries anonymous and see if the server is OK with it. The fact that the identity is anonymous as opposed to host/anonymous indicates that it's a user authentication, rather than machine authentication.

1492: TESTPC: May 16 2016 17:55:17.185 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: 8021X (4) recvd EAP TLS frame (dot1x_util.c 293)
 

Explanation: RADIUS server sends an Extensible Authentication Protocol-Transport Layer Security (EAP-TLS) frame without any content. Its purpose is to negotiate EAP-TLS protocol with the client.

1516: TESTPC: May 16 2016 17:55:17.185 +0600: %NAM-6-INFO_MSG: %[tid=1412]: EAP: EAP suggested by server: eapTls
1517: TESTPC: May 16 2016 17:55:17.185 +0600: %NAM-6-INFO_MSG: %[tid=1412]: EAP: EAP requested by client: eapPeap
1518: TESTPC: May 16 2016 17:55:17.185 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: EAP: EAP methods sent: sync=10
1519: TESTPC: May 16 2016 17:55:17.185 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: EAP: credential request 10: state transition: PENDING -> RESPONDED
 

Explanation: NAM recognizes server's request to use EAP-TLS but the client is configured to use Protected Extensible Authentication Protocol (PEAP). This is the reason that NAM sends back a counter-offer for PEAP.

1520: TESTPC: May 16 2016 17:55:17.185 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Auth[WiredPEAP:user-auth]: Authentication state transition: AUTH_STATE_UNPROTECTED_IDENTITY_SENT_FOR_FAST_REAUTHENTICATION -> AUTH_STATE_UNPROTECTED_IDENTITY_ACCEPTED
 

Explanation: RADIUS server accepts the outter/unprotected identity.

1551: TESTPC: May 16 2016 17:55:17.200 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: 8021X (4) recvd EAP PEAP frame (dot1x_util.c 305)
1563: TESTPC: May 16 2016 17:55:17.200 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: EAP (0) EAP-PEAP: SSL handshake start (eap_auth_tls_p.c 409)
 

Explanation: The Protected portion of PEAP (to establish a secure tunnel to exchange inner credentials) starts, after client receives a confirmation from RADIUS server to continue the use of PEAP.

1565: TESTPC: May 16 2016 17:55:17.200 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: EAP (0) SSL STATE: SSLv3 write client hello A (eap_auth_tls_p.c 394)
1566: TESTPC: May 16 2016 17:55:17.200 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: EAP (0) SSL STATE: SSLv3 read server hello A (eap_auth_tls_p.c 394)
 

Explanation: NAM sends a client hello encapsulated in EAP message and waits for server hello to come. The server's hello contains ISE certificate, so it takes some time to finish transferring.

1622: TESTPC: May 16 2016 17:55:17.216 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: 8021X (4) recvd EAP PEAP frame (dot1x_util.c 305)
1632: TESTPC: May 16 2016 17:55:17.216 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: EAP (0) SSL STATE: SSLv3 read server hello A (eap_auth_tls_p.c 394)
1633: TESTPC: May 16 2016 17:55:17.216 +0600: %NAM-6-INFO_MSG: %[tid=1468][comp=SAE]: CERT (0) looking up: "/CN=ISE20-1.kurmai.com" (lookup.c 100)
1634: TESTPC: May 16 2016 17:55:17.232 +0600: %NAM-6-INFO_MSG: %[tid=1468][comp=SAE]: CERT (0) Certificate not found: "/CN=ISE20-1.kurmai.com" (lookup.c 133)
1646: TESTPC: May 16 2016 17:55:17.232 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: EAP (0) SSL_ERROR_WANT_X509_LOOKUP (eap_auth_tls_p.c 193)
 

Explanation: NAM extracted the subject name of the ISE server from server certificate. Since it doesn't have server certificate installed in the trust store, you do not find it there.

1649: TESTPC: May 16 2016 17:55:17.232 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: EAP (5) EAP_EVENT_CRED_REQUEST queued (eapCredProcess.c 496)
1650: TESTPC: May 16 2016 17:55:17.232 +0600: %NAM-7-DEBUG_MSG: %[tid=1464][comp=SAE]: EAP (5) EAP: CRED_REQUEST (eapMessage.c 355)
1662: TESTPC: May 16 2016 17:55:17.232 +0600: %NAM-6-INFO_MSG: %[tid=1412]: Getting credentials from logon.
1685: TESTPC: May 16 2016 17:55:17.232 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Auth[WiredPEAP:user-auth]: ...resumed
 

Explanation: NAM looks for the inner/protected identity to be sent to RADIUS server after tunnel is established. In this case, "Automatically use my Windows logon name and password" option has been enabled on the wired adapter, so NAM uses windows logon credentials instead of asking the user for it.

1700: TESTPC: May 16 2016 17:55:17.247 +0600: %NAM-7-DEBUG_MSG: %[tid=1464][comp=SAE]: EAP (0) SSL STATE: SSLv3 write client key exchange A (eap_auth_tls_p.c 394)
1701: TESTPC: May 16 2016 17:55:17.247 +0600: %NAM-7-DEBUG_MSG: %[tid=1464][comp=SAE]: EAP (0) SSL STATE: SSLv3 write change cipher spec A (eap_auth_tls_p.c 394)
1750: TESTPC: May 16 2016 17:55:17.278 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: EAP (0) SSL STATE: SSL negotiation finished successfully (eap_auth_tls_p.c 394)
1751: TESTPC: May 16 2016 17:55:17.278 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: EAP (0) EAP-PEAP: SSL handshake done (eap_auth_tls_p.c 425)
1752: TESTPC: May 16 2016 17:55:17.278 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: EAP (0) EAP-PEAP: New session. (eap_auth_tls_p.c 433)
1753: TESTPC: May 16 2016 17:55:17.278 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: EAP (0) EAP-PEAP: session cipher AES256-SHA. (eap_auth_tls_p.c 441)
 

Explanation: NAM sent client key and cipher spec to server and received confirmation. SSL negotation is successful and a tunnel is established.

1810: TESTPC: May 16 2016 17:55:17.294 +0600: %NAM-6-INFO_MSG: %[tid=1412]: Protected identity/(Username) sent.
1814: TESTPC: May 16 2016 17:55:17.294 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Auth[WiredPEAP:user-auth]: Authentication state transition: AUTH_STATE_UNPROTECTED_IDENTITY_ACCEPTED -> AUTH_STATE_PROTECTED_IDENTITY_SENT
1883: TESTPC: May 16 2016 17:55:17.310 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Auth[WiredPEAP:user-auth]: Authentication state transition: AUTH_STATE_PROTECTED_IDENTITY_SENT -> AUTH_STATE_PROTECTED_IDENTITY_ACCEPTED
 

Explanation: Protected identity is sent to the server, who accepts the identity. Now server requests password.

1902: TESTPC: May 16 2016 17:55:17.310 +0600: %NAM-7-DEBUG_MSG: %[tid=1464][comp=SAE]: EAP (5) deferred password request (eapRequest.c 147)
1918: TESTPC: May 16 2016 17:55:17.310 +0600: %NAM-6-INFO_MSG: %[tid=1412]: Protected password sent.
1921: TESTPC: May 16 2016 17:55:17.325 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Auth[WiredPEAP:user-auth]: Authentication state transition: AUTH_STATE_PROTECTED_IDENTITY_ACCEPTED -> AUTH_STATE_CREDENTIAL_SENT
 

Explanation: NAM receives password request and sends password to server.

2076: TESTPC: May 16 2016 17:55:17.856 +0600: %NAM-7-DEBUG_MSG: %[tid=1412]: Auth[WiredPEAP:user-auth]: Authentication state transition: AUTH_STATE_CREDENTIAL_SENT -> AUTH_STATE_SUCCESS
2077: TESTPC: May 16 2016 17:55:17.856 +0600: %NAM-7-DEBUG_MSG: %[tid=1468][comp=SAE]: STATE (4) S_enterStateAux called with state = AUTHENTICATED (dot1x_sm.c 142)
 

Explanation: Server receives the password, verifies it and sends EAP-Success. Authentication is successful at this point, and client proceeds as it gets the IP address from DHCP.