Field Notice: FN72578 - Cisco IOS XE - Smart Licensing Using Policy Might Cause High CPU/Memory Usage - Software Upgrade Recommended

For affected versions of the Cisco IOS^® XE software, devices might report high CPU or memory usage. In some scenarios, devices might report both high CPU and memory usage. 

This issue is seen only when the 'Smart Licensing Using Policy' feature is implemented on the device.

This issue is seen when the "Smart Licensing Using Policy" feature is implemented, and Resource Utilization Measurement (RUM) reports are accumulated in large quantities on the device.

The underlying reason for the accumulation could be a communication issue between the device and Cisco Smart Software Manager (CSSM). When RUM reports do not get the required acknowledgement, they accumulate on the device and can cause high CPU/memory usage.

The underlying reason for the RUM report accumulation could be a communication issue between the device and Cisco Smart Software Manager (CSSM).

This error log might be observed in the affected device:

%SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart Software Manager (CSSM) : Communications failure

This issue can occur when there is an increasing trend of accumulated RUM reports in the device. The increasing trend of accumulated RUM reports can be viewed with multiple iterations of these commands:

Sample output is shown here:

system#license smart save usage all file flash:report.txt

system#more flash:report.txt | count RUMReport

Number of lines which match regexp = 214 <<<< This counter may increase over multiple iterations

Cisco has observed that at 1000 RUM reports, high CPU/memory usage might be seen. This value, however, is not fixed and will vary depending on network conditions and topology. It is highly recommended that the mitigation steps (see the Workaround/Solution section) are taken as soon as RUM reports start to accumulate.

For high CPU, the processes “SAGetRUMIds” and “SAUtilRepSave” will be seen.

Sample output is shown here:

system#show processes cpu sorted

CPU utilization for five seconds: 99%/99%; one minute: 99%; five minutes: 99% 

 PID  Runtime(ms)    Invoked      uSecs   5Sec   1Min   5Min TTY Process

 725  4042749037   313719798      12886 75.29% 77.09% 76.24%   0 SAGetRUMIds 

 154   164791260   261212986        630 21.62% 21.81% 22.97%   0 SAUtilRepSave

For high memory usage, there will be an increasing trend in memory held by the “MallocLite” process.

Sample output is shown here:

system#show processes memory sorted

Processor Pool Total: 1348707052 Used:  317607088 Free: 1031099964

reserve P Pool Total:     102404 Used:         88 Free:     102316

 lsmpi_io Pool Total:    6295128 Used:    6294296 Free:        832

 PID TTY  Allocated      Freed      Holding    Getbufs    Retbufs Process

  0   0          0          0    1205727320          0          0 *MallocLite*

  0   0  335560736   76485816     238693656          0          0 *Init*

Workaround  

The underlying reason for the RUM report accumulation could be a communication issue between the device and CSSM. Ensure that the underlying connectivity operates as expected for your network as per the smart licensing implementation.

Either of these workarounds can be implemented in order to temporarily resolve the issue:

• Clean Up RUM Report Accumulation

  Enter the license smart factory reset command followed by a device reload with the reload command.

  system#license smart factory reset
  
  %Warning: reload required after "license smart factory reset" command
  
  system#reload

• Manual Sync of RUM Reports

  The RUM reports can be manually synced with CSSM. For instructions on how to manually sync the RUM Reports, see Uploading Data or Requests to CSSM and Downloading a File.

Solution

In order to resolve this issue, upgrade to one of these fixed Cisco IOS XE software releases:

• Cisco IOS XE Release 17.3.7 or later
• Cisco IOS XE Release 17.6.5 or later
• Cisco IOS XE Release 17.9.1 or later

The software can be downloaded from the Cisco Software Download website.

Cisco strongly recommends that customers review the software download page for the current recommended starred releases and upgrade to those releases.

These products are affected:

• Cisco 1000 Series Integrated Services Routers
• Cisco 1100 Integrated Services Router
• Cisco 4000 Series Integrated Services Routers
• Cisco 4221 Integrated Services Router
• Cisco 4321 Integrated Services Router
• Cisco 4331 Integrated Services Router
• Cisco 4351 Integrated Services Router
• Cisco 4431 Integrated Services Router
• Cisco 4451-X Integrated Services Router
• Cisco 4461 Integrated Services Router
• Cisco ASR 1000 Series Aggregation Services Routers
• Cisco ASR 1000 Series IOS XE SD-WAN
• Cisco ASR 1001-HX Router
• Cisco ASR 1001-X Router
• Cisco ASR 1002-HX Router
• Cisco ASR 1002-X Router
• Cisco ASR 1004 Router
• Cisco ASR 1006 Router
• Cisco ASR 1006-X Router
• Cisco ASR 1009-X Router
• Cisco ASR 1013 Router
• Cisco Catalyst 8000V Edge Software
• Cisco Catalyst 8200 Series Edge Platforms
• Cisco Catalyst 8300 Series Edge Platforms
• Cisco Catalyst 8500 Series Edge Platforms
• Cisco Catalyst 8500L Series Edge Platforms
• Cisco Catalyst 9200 Series Switches
• Cisco Catalyst 9200L Switch Stack
• Cisco Catalyst 9300 Series Switches
• Cisco Catalyst 9300L Series Switches
• Cisco Catalyst 9400 Series Switches
• Cisco Catalyst 9500 Series Switches
• Cisco Catalyst 9600 Series Switches
• Cisco Catalyst 9800-40 Wireless Controller
• Cisco Catalyst 9800-80 Wireless Controller
• Cisco Catalyst 9800-CL Wireless Controller for Cloud
• Cisco Catalyst 9800-L-C Wireless Controller
• Cisco Catalyst 9800-L-F Wireless Controller
• Cisco Cloud Services Router 1000V Series
• Cisco CSR 1000V Series IOS XE SD-WAN
• Cisco Embedded Wireless Controller on Catalyst 9115AX Access Points
• Cisco Embedded Wireless Controller on Catalyst 9117AX Access Points
• Cisco Embedded Wireless Controller on Catalyst 9120AX Access Points
• Cisco Embedded Wireless Controller on Catalyst 9130AX Access Points
• Cisco Embedded Wireless Controller on Catalyst Access Points
• Cisco Integrated Services Virtual Router
• Cisco ISR 1000 Series IOS XE SD-WAN
• Cisco ISR 4000 Series IOS XE SD-WAN
• Cisco XE SD-WAN Routers