Monitor Catalyst 9800 KPIs (Key Performance Indicators)

Introduction

This document describes a list of commands that can be used to check and monitor status of a 9800 and proactively identify any possible issues.

Monitor KPIs

There are various categories of perofrmance indicators and this document covers the following categories :

• General WLC (Wireless LAN Controllers)  KPIs : check WLC status, crashes, CPU and memory utilization
• AP KPIs : AP disconnections, crashes, misconfig tags
• Wireless Client related KPIs: Client state distribution, client delete reasons, radius failures
• RF stats: AP channel/tx power distribution, load, channel changes, DFS events
• Packet drops and punted to CPU: Check for drops, packets punted, buffer failures, datapath utilization
• Connection with other WLCs, DNAC or DNAS: Check mobility and connectivity with DNAC or DNAS
• Automate datacollection for KPIs: how to automate KPIs datacollection and how to do periodical collection.

General WLC KPIS

• Check WLC uptime and version and confirm that the WLC is in install mode

show version | i uptime|Installation mode|Cisco IOS Software

• Check redundancy and identify if there are any switchovers

show redundancy | i ptime|Location|Current Software state|Switchovers

• Crashes/cores for WLC: Identify if there are any WLC coredumps or system-reports
  Physical WLCs 9800-40 and 9800-80:
  

  !!Active:
  dir harddisk:/core/ | i core|system-report
  !!Standby:
  dir stby-harddisk:/core/| i core|system-report

    Virtual WLC and 9800-L:

!!Active
dir bootflash:/core/ | i core|system-report
!!Standby
dir stby-bootflash:/core/ | i core|system-report

• Check CPU utilization: Identify if there is any process using lot of CPU.

sh processes cpu platform sorted | ex 0% 0% 0%

     Note : for 9800-CL and 9800-L we can see high cpu in ucode_pkt_PPE0 as expected behavior.

• Check CPU utilization per wncd process

show processes cpu platform sorted | i wncd

• Check Memory utilization: Identify if device is running out of memory and if there is some process holding lot of memory.

show platform resources (basic)
show process memory platform sorted
show processes memory platform accounting (advanced)

• Hardware: Confirm Fans, power, SFPs, temperature,

show platform
show inventory
show environment (Only for physical 9800-40, 9800-80 and 9800-L)

• Check licensing: Confirm that WLC is licensed

show license summary | i Status:

Monitor Access Point KPIs

• Check total number of APs:

show ap summary | i Number of APs

• Check for AP reloads or capwap restarts. Identify APs with lower "AP Up Time" -->reload or lower "Association Up Time"--> capwap restart

show ap uptime | ex ____([0-9])+ day

     Note : focus on APs reloaded or restarted within 1 day

• Check for AP crashes

show ap crash

•  Check for highest number of events or unexpected events of AP session termination

show wireless stats ap session termination

• Check for disconnect reasons and time of disconnections

show wireless stats ap history | i Disjoined

• Check for APs with misconfigured tags

show ap tag summary | i Yes

• Check for APs with no clients connected

show ap sum sort descending client-count | i __0_

Wireless client related KPIs

•  Total number of connected clients and states for those clients

show wireless summary
show wireless stats client detail | i Authenticating :|Mobility :|IP Learn :|Webauth Pending :|Run :|Delete-in-Progress :

• Check client delete reasons and look for highest values and counters increasing

show wireless stats client delete reasons | e : 0
show wireless stats trace-on-failure

• Check for radius failures, access-rejects, unresponsed packets, long queues

show radius statistics

• Check for radius servers been marked as DEAD

show aaa servers | i Platform Dead: total|RADIUS: id

RF stats

• Check channels, txpowers, radio downs:

show ap dot11 5ghz summary
show ap dot11 24ghz summary
show ap dot11 6ghz summary

• Check channel changes and DFS events.

show ap auto-rf dot11 5ghz | i Channel changes due to radar|AP Name|Channel Change Count
show ap auto-rf dot11 24ghz | i Channel Change Count|AP Name

• Check channel utilization and number of clients. Identify APs with few clients and high channel utilization.

show ap dot11 5ghz load-info
show ap dot11 24ghz load-info
show ap dot11 6ghz load-info

Packet drops and punted to CPU

• Check volume of traffic received and tx by WLC:

show int po1 | i line protocol|put rate|drops|broadcast

• Check for packet drops:

show platform hardware chassis active qfp statistics drop

• Check for packets punted to CPU:

show platform hardware chassis active qfp feature wireless punt statistics

• Check for buffer failures:

show buffers | i buffers|failures

• Check dataplane utilization:

show platform hardware chassis active qfp datapath utilization | i Load

Monitor connections with other WLCs, DNAC or DNAS

• Check mobility, confirm that mobility tunnels are up and with right PMTU

show wireless mobility summary

• Check connection with DNAC

show telemetry internal connection  or  show telemetry connection all after 17.7.1

• Check connection with DNASpaces/CMX:

show nmsp status

Automate data collection of KPIs

There is a script to collect and store outputs using guestshell: https://developer.cisco.com/docs/wireless-troubleshooting-tools/#9800-guestshelleem-scripts-9800-guestshelleem-scripts

You can also use EEM to run guestshell script every certain time:

Ex Collecting KPIs from Monday to Friday at 23:59:

event manager applet KPIs
event timer cron cron-entry "59 23 * * 1-5" maxrun 600
action 1.0 cli command "enable"
action 2.0 cli command "guestshell run python3 /bootflash//guest-share/KPI.py"
action 3.0 cli command " end"