Introduction

This document describes a systematic approach and list of commands to collect to troubleshoot 9800 client connectivity issues.

Prerequisites

Cisco recommends that you have knowledge of these topics:

Basic knowledge of Cisco WLC 9800
Basic knowledge of Cisco Wave2 and/or 11AX APs

Topology

Generic outputs to collect from WLC

0. Enable term exec prompt timestamps to have time reference for all the commands.

1. We can start checking number of clients, client states distribution and excluded clients.
show wireless summary !!Total number of Aps and clients
show wireless exclusionlist !!In case any client is seen as excluded
show wireless exclusionlist client mac-address MAC@ !!To get more details about concrete client excluded.

2. Next we can identify clients that are not in run state. Collect these outputs several times and identify clients not changing of state, stuck in authenticating, iplearn.
We can also check stats and deletions of a concrete SSID.
show wireless cli summary | ex _Run_ !!Identify client mac address not changing state
show wireless stats client detail !!Check counters for global errors and failures, retransmissions and info about webauth clients. Here we can also check client distribution (show wireless stats client detail | i Authenticating|Mobility__|IP Learn__|Webauth Pending|Run__|Delete-in-Progress)
show wlan name WLAN-NAME client stats !!Check client stats counters per SSID to identify if issue is only seen in one concrete SSID.

3. Check ip address assigment for clients, look for incorrect addresses or unexpected static address learning, vlans marked as dirty due to no response from DHCP server, or packets drops in SISF that is handling DHCP/ARP.
show wireless device-tracking database ip !! Check by ip and see how address learning occurred:
show wireless device-tracking database mac !! Check by mac and see vlan to which client is assigned
show wireless vlan details !!Check that vlan is not marked as dirty due to DHCP failures
show wireless device-tracking feature drop !!Drops in SISF

4. Check client RF health, uptime tx/rx and retries to identify clients with bad RSSI and lots of retries.
show wireless stats client summary

5. Device classification. Identify type of clients connected and if there is any classification issue with some of the clients.
show wireless client device summary

6. Check for predefined errors and unexpected client delete reasons. Collect those outputs several times and identify unexpected failures/deletes.
show wireless stats trace-on-failure | ex :_0
show wireless stats client delete reasons | ex _:_0

7. Once identified unexpected delete reasons, lets try to identify client mac-addressed facing that concrete delete reason. Proactive troubleshooting.
show logging profile wireless | i DELETE !!Find client mac address matching unexpected delete reason.

Specific outputs from WLC for concrete client MAC@

8. Get details about one concrete client:
show wireless client mac MAC@ detail
show wireless client mac MAC@ mobility history !!Check roaming events
show wireless client mac MAC@ stats mobility !!Check roam counts for client intra/inter wncd

9. Check always-on-tracing for this client. Identify steps that lead to the delete reason.
show logging profile wireless start last 1 hours filter mac MAC@ to-file bootflash:Always_on_tracing_MAC@.log !! start last time can be changed to match when client faced issued.

10. Collect all details for client using show tech wireless commands for client and for client dataplane:
show tech wireless client mac-address MAC@
show tech wireless datapath client mac-address MAC@

Advanced logs from WLC and AP for concrete client MAC@

11. If with previous outputs root cause could not be identified then we need to collect ra_trace and packet captures from client to identify root cause on next event.

Logs from WLC:

!! Enable ra-trace for AP using default monitor-time is 1800s increse it to max in case you do not know when AP disconnection occurs.
debug wireless mac <Client_MAC> internal monitor-time 2085978494 !!Using AP radio mac to capture traces with verbose level from WLC. Setting time allows us to enable traces for up to 24 days
!!Reproduce
no debug wireless mac <Client_MAC> internal monitor-time 2085978494
!!WLC generates an ra_trace file with Client_info, command to check for ra_trace file generated.
dir bootflash: | i ra_trace

!!Embedded Captures filtered by client mac address in both directions and have a circular buffer to ensure that we get latest captures in case buffer exceeds 100M. Client inner mac filter available after 17.1
monitor capture MYCAP clear
monitor capture MYCAP interface Po1 both
monitor capture MYCAP buffer size 100
monitor capture MYCAP match any
monitor capture MYCAP inner mac CLIENT_MAC@
monitor capture MYCAP start
!!Reproduce
monitor capture MYCAP stop
monitor capture MYCAP export flash:|tftp:|http:.../filename.pcap

Logs from AP

show tech !! Collect show tech to have all config details and client stats for the AP.
term mon
!!Basic
debug client MAC@

!!Option is to use Efficient client troubleshooting: https://developer.cisco.com/docs/wireless-troubleshooting-tools/#9800-guestshelleem-scripts-9800-guestshelleem-scripts

List of all commands

List of all commands from WLC

term exec prompt timestamps
show wireless summary
show wireless exclusionlist
show wireless exclusionlist client mac-address MAC@
show wi cli summary | ex _Run_
show wireless stats client detail | i Authenticating|Mobility__|IP Learn__|Webauth Pending|Run__|Delete-in-Progress
show wireless stats client detail
show wireless device-tracking database ip
show wireless device-tracking database mac
show wireless vlan details
show wireless device-tracking feature drop
show wireless stats client summary
show wireless client device summary
show wireless stats trace-on-failure | ex :_0
show wireless stats client delete reasons | ex _:_0
show logging profile wireless | i DELETE
show wireless client mac MAC@ detail
show wireless client mac MAC@ mobility history
show wireless client mac MAC@ stats mobility
show logging profile wireless start last 1 hours filter mac MAC@ to-file bootflash:Always_on_tracing_MAC@.log
show tech wireless client mac-address MAC@
show tech wireless datapath client mac-address MAC@
debug wireless mac <Client_MAC> internal monitor-time 2085978494

List of all commands from AP

show tech
term mon
debug client MAC@