Introduction

This document describes how to configure and troubleshoot Smart Licensing Using Policy (SLUP) on Catalyst 9800 Wireless LAN Controller (WLC) .

Background information

1. Direct connect to Cisco Smart Software Manager Cloud (CSSM Cloud)
2. Connected to CSSM via CSLU (Cisco Smart License Utility Manager)
3. Connected to CSSM via On-Prem Smart Software Manager (On-Prem SSM)

This article does not try to cover all the Smart Licensing scenarios on Catalyst 9800 that are best explained in the Smart Licensing Using Policy configuration guide. However, it gives a series of useful commands to troubleshoot direct connect, CSLU and On-Prem SSM Smart Licensing Using Policy issues on the Catalyst 9800.

Option 1. Direct connect to Cisco Smart Licensing Cloud Servers (CSSM)

Option 2. Connection via CSLU

Option 3. Connection via On-Prem Smart Software Manager (On-Prem SSM)

Note: All commands mentioned in this article are applicable only to WLCs that run version 17.3.2 or later.

Traditional Licensing vs SLUP

Smart Licensing Using Policy has been introduced to the Catalyst 9800 with the code version 17.3.2. The initial 17.3.2 release misses SLUP configuration menu in the WLC webUI, which was introduced with the 17.3.3 release. The SLUP is different from traditional smart licensing in couple of ways:

• WLC now communicates with CSSM through smartreceiver.cisco.com domain, instead of the old tools.cisco.com
• Instead of "Registering", the WLC now "Establishes Trust" with the CSSM or On-Prem SSM
• CLI commands have been slightly altered
• There is no Smart Licensing Reservation (SLR) anymore. Instead you can periodically report your usage manually
• There is no evaluation mode anymore. The WLC continues to function at full capacity even without license. The system is honor based and you are supposed to report your license usage periodically (automatically or manually in case of airgapped networks)

Note: Means reader take note. Notes contain helpful suggestions or references to material not covered in the document.

Warning: If you are using a Cisco Catalyst 9800-CL Wireless Controller, ensure that you are familiar with the mandatory ACK requirement that starts with Cisco IOS XE® Cupertino 17.7.1. See RUM Reporting and Acknowledgment Requirement for Cisco Catalyst 9800-CL Wireless Controller.

Configuration

Direct Connect CSSM

Once the token has been created on the CSSM, in order to establish trust, these commands need to be executed:

Note: Token Max. Number of Uses count must be at least 2 in a case of WLC in HA SSO.

configure terminal
ip http client source-interface <interface>
ip http client secure-trustpoint <TP>
license smart transport smart
license smart url default
exit
write memory
terminal monitor
license smart trust idtoken <token> all force

• "ip http client source-interface" command specifies the L3 interface that licensing related packets are going to be sourced from
• "ip http client secure-trustpoint" command specifies which trustpoint/certificate is used for CSSM communication. The trustpoint name can be found using "show crypto pki trustpoints" command. It is recommended to use a self signed cert TP-self-signed-xxxxxxxxxx certificate or Manufacturer Installed Certificate (also known as MIC, available only on 9800-40, 9800-80 and 9800-L), usually called CISCO_IDEVID_SUDI.
• "Terminal monitor" command makes the WLC print the logs to the console and help confirm that trust has been successfully established. It can be disabled using "terminal no monitor".
• Keyword "all" in the last command tells tells all of the WLCs in the HA SSO cluster to establish the trust with the CSSM.
• Keyword "force" tells the WLC to override any of the previously established trusts and attempt a new one.

Note: If the trust is not being established, the 9800 tries again 1 minute later after the command is executed and then does not try again for some time. Enter the token command again to force  a new trust establishment.

Connected to CSLU

Cisco Smart License Utility Manager (CSLU) is a Windows-based application (also available on Linux) that enables customers to administer licenses and their associated Product Instances from their premises instead of having to directly connect their Smart Licensed enabled Product Instances to Cisco Smart Software Manager (CSSM).

This section only covers the 9800 Wirless configuration. There are others steps to perform to configure licensing with CSLU (such as install CSLU, configure the CSLU software and so on), which is covered in the configuration guides.Whether you want to implement a product instance-initiated or CSLU-initiated method of communication, complete the corresponding sequence of tasks.

Product instance-initiated 

1. Ensure network reachability from the controller to CSLU 
2. Ensure that transport type is set to cslu : 

   (config)#license smart transport cslu
   (config)#exit
   #copy running-config startup-config

3. If you want CSLU to be discovered by the controller, you need to perform the action. If you want CSLU to be discovered using DNS, no action is required. If you want to discover it using an URL, please enter these commands : 
   

   (config)#license smart url cslu http://<cslu_ip>:8182/cslu/v1/pi
   (config)#exit
   #copy running-config startup-config

CSLU-initiated 

When you configure CSLU-initiated communication, there is nothing else to do than check and ensure network reachability to CSLU from the controller. 

Connected to SSM On-Prem

Configuration with On-Prem SSM is fairly similar to direct connection. On-Prem needs to run version  8-202102 or newer. For SLUP releases (17.3.2 and later), it is advised to use the CSLU URL and transport type. The URL can be obtained from the On-prem Web UI interface under Smart Licensing -> Inventory -> <Virtual Account> -> General section

configure terminal
 ip http client source-interface <interface>
 ip http client secure-trustpoint <TP>
 license smart transport cslu
 license smart url https://<on-prem-ssm-domain>/SmartTransport
 crypto pki trustpoint SLA-TrustPoint
  revocation-check none 
 exit
write memory
terminal monitor

On-Prem SSM does not require the use of a trust token.

Note : if you are getting the message " %PKI-3-CRL_FETCH_FAIL: CRL fetch for trustpoint SLA-TrustPoint failed", it is because you have not configured revocation-check none under the SLA-TrustPoint. This is the trustpoint used for Smart Licensing. In case of on-prem, the certificate on the licensing server is most often a self-signed certificate for which CRL verificaiton is not possible, hence the requirement to configure no revocation checks.

Configuring Smart Transport Through an HTTPs Proxy

Note: Means reader take note. Notes contain helpful suggestions or references to material not covered in the document.

Note: Authenticated proxies are not yet supported as of publication of this article (code release 17.9.2). If you are using authenticated proxies in your infrastructure, consider using the Cisco Smart License Utility Manager (CSLU), itself supporting this type of servers.

To use a proxy server to communicate with CSSM when using the Smart transport mode, complete these steps:

configure terminal
  ip http client source-interface <interface>
  ip http client secure-trustpoint <TP>
  license smart transport smart
  license smart url default
  license smart proxy address <proxy ip/fqdn>
  license smart proxy port <proxy port>
exit
write memory
terminal monitor
license smart trust idtoken <token> all force

Communication Frequency

The reporting interval you can configure in CLI or GUI has no effect.

The 9800 WLC communicates with CSSM or On-Prem Smart Software Manager every 8 hours, no matter what reporting interval is configured via web interface or CLI. This means that newly joined access points can appear on the CSSM up to 8 hours after they initially joined.

You can figure out the next time licenses are calculated and reported with "show license air entities summary". This command is not part of the typical "show tech" or "show license all" output:

WLC#show license air entities summary 
Last license report time........................: 07:38:15.237 UTC Fri Aug 27 2021
Upcoming license report time....................: 15:38:15.972 UTC Fri Aug 27 2021
No. of APs active at last report................: 3
No. of APs newly added with last report.........: 0
No. of APs deleted with last report..............: 0

License Factory Reset

Catalyst 9800 WLC can have all of its licensing configuration and trust factory reset and still keep all the other configuration. This requires a WLC reload:

WLC-1#license smart factory reset
%Warning: reload required after "license smart factory reset" command

In Case of RMA or Hardware Replacement

If the 9800 WLC needs to be replaced, the new device has to register with CSSM/On-Prem Smart Software Manager and it is perceived as a new device. Releasing the license count of the previous device requires manual deletion under "Product Instances":

Upgrading From Specific License Registration (SLR)

Older WLC releases <17.3.2 used a special offline licensing method called Specific License Registration (SLR). This licensing method has been deprecated in the releases using SLUP (17.3.2 and later).

If you upgrade a 9800 controller that was using SLR to a release post 17.3.2 or 17.4.1, it is recommended to move to offline SLUP reporting rather than relying on the SLR commands.Save the license usage RUM file and register that with the Smart Licensing Portal. Since SLR does not exist anymore in newer releases, this reports the correct license count and release any unused license. Licenses are not "blocked" anymore but the exact usage count is reported.

Troubleshooting

Internet Access, Port Checks & Pings

Instead of the tools.cisco.com that traditional smart licensing used, the new SLUP uses smartreceiver.cisco.com domain to establish trust. At the time of writing of this article, this domain currently resolves to the 3 different IP addresses. There is a discrepancy between the ability to ping these 3 IPs. Only one of them allows it. Pings must not be used as an internet reachability test from WLC.

Not being able to ping these servers does not mean that they are not working properly. Smart licensing team currently works to fix this and make all 3 servers pingable:

72.163.15.144 – responding to PING
173.36.127.16 – not responding to PING
192.133.220.90 – not responding to PING

Instead of pings, telnet over port 443 must be used as a reachability test. Telnet can be checked either against smartreceiver.cisco.com domain or directly against the server IP addresses. If traffic is not being blocked, port must show up as open in the output:

WLC-1#telnet smartreceiver.cisco.com 443
Trying smartreceiver.cisco.com (192.133.220.90, 443)... Open <-------
[Connection to 192.133.220.90 closed by foreign host]
WLC-1#telnet 72.163.15.144 443
Trying 72.163.15.144, 443 ... Open <-------
[Connection to 72.163.15.144 closed by foreign host]
WLC-1#telnet 192.133.220.90 443
Trying 192.133.220.90, 443 ... Open <-------
[Connection to 192.133.220.90 closed by foreign host]
WLC-1#telnet 173.36.127.16 443
Trying 173.36.127.16, 443 ... Open <-------
[Connection to 173.36.127.16 closed by foreign host]

Syslog

If "terminal monitor" command is enabled while the token is being configured, the WLC prints out the relevant logs in the CLI. These messages can also be obtained if you run "show logging". Logs of a successfully established trust looks like this:

WLC-1#license smart trust idtoken <token> all force
Aug 22 12:13:08.425: %CRYPTO_ENGINE-5-KEY_DELETED: A key named SLA-KeyPair has been removed from key storage
Aug 22 12:13:08.952: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair has been generated or imported by crypto-engine
Aug 22 12:13:08.975: %PKI-6-CONFIGAUTOSAVE: Running configuration saved to NVRAM
Aug 22 12:13:11.879: %SMART_LIC-6-TRUST_INSTALL_SUCCESS: A new licensing trust code was successfully installed on P:C9800-CL-K9,S:9PJK8D9OCNB.

Logs of a WLC without a defined DNS server or with a non-functioning DNS server:

Aug 23 09:19:43.486: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart Software Manager (CSSM) : Unable to resolve server hostname/domain name 

Logs of a WLC with a functioning DNS server, but without internet access:

Aug 23 09:23:30.701: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart Software Manager (CSSM) : No detailed information given

Packet captures

Even though communication between WLC and CSSM/On-Prem SSM is encrypted and going over HTTPS, performing packet captures can give some insight into what can cause the trust not to be established. Easiest way to collect packet captures is through the WLC web interface.

Navigate to Troubleshooting > Packet Capture. Create a new capture point:

Make sure that "Monitor Control Plane" checkbox is enabled. Increase the buffer size to the maximum 100MB. Add the interface which must be captured. Smart licensing traffic is sourced out of the wireless management interface by default or from the interface defeined via "ip http client source-interface " command:

Start the captures and run "license smart trust idtoken <token> all force" command:

The packet captures of a trust establishment must contain these steps:

1. TCP session establishment using SYN, SYN-ACK & ACK sequence
2. TLS session establishment with both server and client certificate exchange. Establishment ends with "New Session Ticket" packet
3. Encrypted packet exchange ("Application data" frames) where WLC reports license usage
4. TCP session termination via FIN-PSH-ACK, FIN-ACK & ACK sequence

Note: The packet captures contains a lot more frames, including multiples of TCP window update and "Application Data" frames

Since CSSM Cloud uses 3 different public IP addresses, in order to filter out all of the packet captures between WLC and CSSM, use these wireshark filter:

ip.addr==173.36.127.16 or ip.addr==192.133.220.90 or ip.addr==72.163.15.144 

If using an On-Prem SSM, filter for the SSM IP address:

ip.addr==<on-prem-ssm-ip>

Example packet captures of a successful trust establishment with directly connected CSSM with all of the significant packet captures filtered:

Show Commands

These show commands contain useful information about trust establishment:

show license status
show license summary
show tech-support license
show license tech-support
show license air entities summary

show license history message (useful to see the history and content of messages sent to SL)

show tech wireless (actually gets show log and show run on top of the rest which can be useful)

Command "show license history message" is one of the more useful ones since it can display the actual messages sent from WLC and received back from CSSM.

A successful trust establishment has both "REQUEST: Aug 23 10:18:08 2021 Central" and "RESPONSE: Aug 23 10:18:10 2021 Central" messages printed. If there is nothing after the "RESPONSE" line, that means that the WLC did not receive a response from the CSSM.

Example of a "show license history message" output for a successful trust establishment:

REQUEST: Aug 23 10:18:08 2021 Central
{"request":"{\"header\":{\"request_type\":\"POLL_REQ\",\"sudi\":{\"udi_pid\":\"C9800-CL-K9\",\"udi_serial_number\":\"9PJK8D7OC
NB\"},\"version\":\"1.3\",\"locale\":\"en_US.UTF-8\",\"signing_cert_serial_number\":\"3\",\"id_cert_serial_number\":\"59152896
\",\"product_instance_identifier\":\"\",\"connect_info\":{\"name\":\"C_agent\",\"version\":\"5.0.9_rel/68\",\"production\":tru
e,\"additional_info\":\"\",\"capabilities\":[\"UTILITY\",\"DLC\",\"AppHA\",\"MULTITIER\",\"EXPORT_2\",\"OK_TRY_AGAIN\",\"POLIC
Y_USAGE\"]}},\"request_data\":\"{\\\"sudi\\\":{\\\"udi_pid\\\":\\\"C9800-CL-K9\\\",\\\"udi_serial_number\\\":\\\"9PJK8D7OCNB\\
\"},\\\"timestamp\\\":1629713888600,\\\"nonce\\\":\\\"11702702165338740293\\\",\\\"product_instance_identifier\\\":\\\"\\\",\\
\"original_request_type\\\":\\\"LICENSE_USAGE\\\",\\\"original_piid\\\":\\\"2e84a42f-c903-44c5-83b2-e62e258c780f\\\",\\\"id\\\
":7898262236}\"}","signature":{"type":"SHA256","key":"59152896","value":"eiJ7IuQaTCFxgUkwls76WZxa5DRI5AvRl2Fi1trn6H1x4HrKS/0fc
OgMqQd5POU6VNsH2j9dHco4T1NJ/aCMbR1MRmkfxyVSWsx4lmjJL1lmpOSi3ZS4FBMvlF/EBOUfowREe2oz2lrQp1cAFpPn5SlaFezW80tMdJm08nv29pO8O7Bffyy
/Nu6SQZfIW+IdF+2qnJeNFAIZbNpg0B5d5HIJvDmDImvDu3bMRHhQAWr2KKzGFr6jPz0hs7bGY/+FlfTLQk5LFEUaKTNH/tuxJPFHlF0BtjIRQtAqy5qDpXdjVJokD
h9//uhsd+NaQyfdRFludkbfUBTFkvPxHW9/5w=="}}

  RESPONSE: Aug 23 10:18:10 2021 Central
{"signature":{"type":"SHA256","value":"TXZE034fqAul2jy9V4+HoB2hDShl9au/5sgodiCVatmu671/6MyN7kZfEzREufY8\nOOsh4l+BZ1ZAXyQ/hVf+Q
SLrjTfO4grGeQTcH7yEj0D+gztWXCOu8RBT7/Bo9aBs\n4x1i0E6flPB3BP6yu7KIEUQZ8yHzlwDT+mVtJGi6TRrtYnV3KQMpCUmF5F5Jby78\nVNHOakQHVE0Ozrg
wOksf3SfXreNZJuzWXzjHvtmlusCQXw7ZTBzffYsNKO0lkJlr\nvgB2PkV7JUlsA481kpIvlPul6IiJXqk+2PC2IzCrCLG57lVN3XgX6lcU581P7HK7\nrSt3mfdyK
1pEl2SHyQ/DAw==","piid":null,"cert_sn":null},"response":"{\"header\":{\"version\":\"1.3\",\"locale\":\"en_US.UTF-8\",\"timesta
mp\":1629713890172,\"nonce\":null,\"request_type\":\"POLL_REQ\",\"sudi\":{\"udi_pid\":\"C9800-CL-K9\",\"udi_serial_number\":\"
9PJK8D7OCNB\"},\"agent_actions\":null,\"connect_info\":{\"name\":\"SSM\",\"version\":\"1.3\",\"production\":true,\"capabilitie
s\":[\"DLC\",\"AppHA\",\"EXPORT_2\",\"POLICY_USAGE\",\"UTILITY\"],\"additional_info\":\"\"},\"signing_cert_serial_number\":\"3
\",\"id_cert_serial_number\":\"59152896\",\"product_instance_identifier\":\"\"},\"status_code\":\"FAILED\",\"status_message\":
\"Invalid ProductInstanceIdentifier: 2e84a42f-c903-44c5-83b2-e62e258c780f provided in the polling request for polling id: 7898
262236\",\"retry_time_seconds\":0,\"response_data\":\"\"}","sch_response":null}

Debugs/btrace

Run this command a few minutes after a trust establishment has been attempted using a "license smart trust idtoken all force" command. IOSRP logs are extremely verbose. Append " | include smart-agent" to the command to get only smart licensing logs.

show logging process iosrp start last 5 minutes
show logging process iosrp start last 5 minutes | include smart-agent

You can also run these debugs and then reconfigure the licensing commands to force a new connection :

debug license events
debug license errors
debug license agent all

Common issues

WLC Has No Internet Access or Firewall blocks/alters Traffic

Embedded packet captures on the WLC are an easy way to see if WLC receives anything back from the CSSM or On-Prem SSM. If there was no response, chances are that the firewall blocks something.

The "show license history message" command prinst an empty response 1 second after the request is sent out if no response was received from the CSSM Cloud or On-Prem SSM.

For example, this can lead an engineer to believe that an empty response was received, but in reality there was no response at all:

REQUEST: Jun 29 11:12:39 2021 CET
{"request":"{\"header\":{\"request_type\":\"ID_TOKEN_TRUST\",\"sudi\":{\"udi_pid\":\"C9800-CL-K9\",\"udi_serial_number\":\"9V4ZPZPN8DW\"},\"version\":\"1.3\",\"locale\":\"en_US.UTF-8\",\"product_instance_identifier\":\"\",\"connect_info\":{\"name\":\"C_agent\",\"version\":\"5.0.9_rel/68\",\"production\":true,\"additional_info\":\"\",\"capabilities\":[\"UTILITY\",\"DLC\",\"AppHA\",\"MULTITIER\",\"EXPORT_2\",\"OK_TRY_AGAIN\",\"POLICY_USAGE\"]}},\"request_data\":\"{\\\"sudi\\\":{\\\"udi_pid\\\":\\\"C9800-CL-K9\\\",\\\"udi_serial_number\\\":\\\"9V4ZPZPN8DW\\\"},\\\"timestamp\\\":1624957959810,\\\"nonce\\\":\\\"12527456165463158693\\\",\\\"product_instance_identifier\\\":\\\"\\\",\\\"hostname\\\":\\\"myc9800-CL\\\",\\\"token\\\":\\\"ZmI3YmNmYzYtNTdhZC00N2QwLTkyMjUtOTVmMjM5YmYzNzNlLTE2Mjc1NDkx%0AODEyMjN8Tkw4YU9zaTJDa045K2U3aG5xdlp2SE9VOGJxMkJmc0dNMWpKT0FJ%0AeUZqUT0%3D%0A\\\",\\\"mode\\\":\\\"PERMANENT\\\",\\\"force\\\":false,\\\"software_tag_identifier\\\":\\\"regid.2018-05.com.cisco.WLC_9500C,1.0_85665885-b865-4e32-8184-5510412fcb54\\\",\\\"device_list\\\":[{\\\"sudi\\\":{\\\"udi_pid\\\":\\\"C9800-CL-K9\\\",\\\"udi_serial_number\\\":\\\"9V4ZPZPN8DW\\\"},\\\"software_tag_identifier\\\":\\\"regid.2018-05.com.cisco.WLC_9500C,1.0_85665885-b865-4e32-8184-5510412fcb54\\\",\\\"product_instance_identifier\\\":\\\"\\\",\\\"csr\\\":\\\"MIICnjCCAYYCAQAwODE2MDQGA1UEAxMtMjlCQUM2OEQ3MDQ0NzQ1NzE2QjVDOEYxODdGM0Q0Q0QxNTIwOEVGRjo6MSwyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmGCVOO1Q/LiZs3je+Zvo+OsJilTb1vpL7sA80TjprHyLkqQ9mMCub1GunwtF18jzIRa1COUPk9vDsY7bD9btvIG80x2MQFAPPmiNFR4ccvuLYzGl+XjjTjgjbsZpov6S6n3rRGJVyd5aS0S8+JCCljzwtv2rHV628RlWFwlEyzcp4Kt5dm8QBnc+dcYWWIn/X9aExPbf9P96HVL6T6+SomKT/h5hQP8wyz8ycol3XjcXN6gD/XZFyOydZ3oayi/RWjRlA0NThHPQy03bdtFiEakA8ZEWmduNRvTnmSsqKQBVVa43gjgoBzANTAOyXv6e33TRN3bQdV6UuSLcqrw1PwIDAQABoCEwHwYJKoZIhvcNAQkOMRIwEDAOBgNVHQ8BAf8EBAMCBaAwDQYJKoZIhvcNAQEFBQADggEBAHwhn+YcbEuJyYrkcFlaEfzRbJcz9KXXAZASykW7ql3fSlv+PaHePdrh9vsxmkXiVo+UnBbk4Kr5a/mdXEG/4NOeToGUwEqf38ktZMJyL81Whl/FEj83Lhmy7Z5paJrSCWwPc14nHFEFJ4BuelbfmPwiK+Y6X0RHEL92ySH6UNNlVgsWsQbHnHLbacDBsjOcyf9N3xAZc/IDxakSUSYMgFGnBXwkP4aN6XZLfKKGvXtYuIs7D+omRpMijYxQfHzvVnauGi7jiDXEAJjUueNHyztgpPi191j84oEVFHvAEhhMRNakrEEp4jyD2bYOr+CVbbxbYiBd2GKDs1/I1N1XSgM=\\\",\\\"id_cert_sn\\\":\\\"\\\"}]}\"}"}
RESPONSE: Jun 29 11:12:40 2021 CET

Note: There is currently an enhancement request "CSCvy84684 - show license history message prints empty response when there are no response" which aims to enhance the output of the "show license history message" command

Unknown CA Alert in Packet Captures

Communication with CSSM or On-Prem SSM requires a "decent" certificate on the 9800 side. It can be self-signed, but it cannot be invalid or expired. In such a case, a packet capture shows a TLS alert for "unknown CA" sent by CSSM when the 9800 http client certificate has expired.

Smart licensing uses "ip http client" configuration, which is different from the "ip http server" that WLC web interface uses. This means that these commands needs to be configured properly:

ip http client source-interface <interface>
ip http client secure-trustpoint <TP>

The trustpoint name can be found with "show crypto pki trustpoints" command. It is recommended to use a self signed cert TP-self-signed-xxxxxxxxxx certificate or Manufacturer Installed Certificate (MIC) which is usually called CISCO_IDEVID_SUDI and is available only on 9800-80, 9800-40 and 9800-L.

It is important to note that devices that do TLS interception, such as a firewall with the SSL decrypt feature, can prevent the C9800 to establish a successful handshake with the Cisco Licensing server as the HTTPS certificate presented is the firewall certificate instead of the Cisco Licensing server certificate.

Note: Make sure to configure both source-interface and secure-trustpoint commands. Source-interface command is needed even if WLC has only one L3 interface.