Monitor CPU Usage on ISR4300 Series

Introduction

This document describes how to read the Central Process Unit (CPU) usage on Integrated Service Routers (ISR) from the 4300 series family.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco IOS® XE
• ISR43XX

Components Used

The information in this document is based on the hardware and software version:

• ISR4321/K9
• ISR4331/K9
• ISR4351/K9
• 03.16.01a.S // 15.5(3)S1a
• 03.16.04b.S // 15.5(3)S4b
• 16.9.7
• 16.12.4

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Architecture

Cisco ISR 4000 Series platforms run Cisco IOS XE that has a distributed software architecture that runs a Linux kernel where Cisco IOS® runs as one of many Linux processes. Cisco IOS runs as a daemon, which is referred as Cisco IOS-Daemon (IOSd).

CPU Usage on Cisco IOSd

In order to monitor the CPU usage on IOSd run the show process cpu command:

#show process cpu
CPU utilization for five seconds: 1%/0%; one minute: 1%; five minutes: 0%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
   1           2           8        250  0.00%  0.00%  0.00%   0 Chunk Manager
   2           5          18        277  0.07%  0.00%  0.00%   0 Load Meter
   3           0           2          0  0.00%  0.00%  0.00%   0 DiagCard4/-1
   4           0           1          0  0.00%  0.00%  0.00%   0 Retransmission o
   5           0           1          0  0.00%  0.00%  0.00%   0 IPC ISSU Dispatc

The output displays two values for the CPU usage, the first value is the total amount of CPU utilization and the second value is the amount of CPU by interrupts sent to IOSd:

The difference between the total amount of CPU and the amount of CPU by interrupts are the values of CPU consumed by processes; in order to corroborate add all the processes usage for the last five seconds:

• CPU consumption of processes = 1% - 0% = 1% = All the processes CPU consumption listed on the command

In order to display the processes that consume the most amount of CPU on the top, run the show process cpu sorted command:

#show process cpu sorted
CPU utilization for five seconds: 1%/0%; one minute: 0%; five minutes: 0%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
  64         103          10      10300  0.33%  0.02%  0.00%   0 Licensing Auto U
  83          26         231        112  0.27%  0.00%  0.00%   0 PuntInject Keepa
 235         555       48176         11  0.11%  0.09%  0.07%   0 Inline Power
   1           2           8        250  0.00%  0.00%  0.00%   0 Chunk Manager

Note: The addition of all the processes can result into floating point values, IOSd rounds the result to the next integer.

CPU Usage by Traffic

The ISR4300 family design, in order to forward traffic, is through an element referred as QuantumFlow Processor (QFP).

Caution: QFP is found on ASR1K as one or several physical chips, on the ISR4400 the same functionality is done with Cavium Octeon co-processors, on the ISR4300 that functionality is done on certain cores of the main Intel CPU. You can think of the QFP on the ISR4300 family as a piece of software which forwards packets.

In order to determine the amount of CPU consumed by traffic you can run the show platform hardware qfp active datapath utilization command:

#show platform hardware qfp active datapath utilization
  CPP 0: Subdev 0            5 secs        1 min        5 min       60 min
Input:  Priority (pps)            0            0            0            0
                 (bps)            0            0            0            0
    Non-Priority (pps)            3            2            2            1
                 (bps)         1448          992          992          568
           Total (pps)            3            2            2            1
                 (bps)         1448          992          992          568
Output: Priority (pps)            0            0            0            0
                 (bps)            0            0            0            0
    Non-Priority (pps)            3            2            2            1
                 (bps)        12216         8024         8024         4576
           Total (pps)            3            2            2            1
                 (bps)        12216         8024         8024         4576
Processing: Load (pct)            0            0            0            1

The command lists the input and output CPU usage for priority and non-priority packets, the information is displayed with packets per second (PPS) and bits per second (BPS), the last line displays the total amount of CPU load due to packet forward in percentage (PCT) values.

CPU Cores Installed

The ISR4300 family have a different amount of CPU cores installed that depends on the model, to identify the number of cores installed on your device run the show processes cpu platform command:

#show processes cpu platform
CPU utilization for five seconds: 30%, one minute: 29%, five minutes: 29%
Core 0: CPU utilization for five seconds: 13%, one minute: 13%, five minutes: 13%
Core 1: CPU utilization for five seconds:  2%, one minute:  3%, five minutes:  3%
Core 2: CPU utilization for five seconds:  0%, one minute:  0%, five minutes:  0%
Core 3: CPU utilization for five seconds: 99%, one minute: 99%, five minutes: 99%
   Pid    PPid    5Sec    1Min    5Min  Status        Size  Name
--------------------------------------------------------------------------------
     1       0      0%      0%      0%  S          1863680  init
     2       0      0%      0%      0%  S                0  kthreadd

Alternatively, run the show platform software status control-processor command:

#show platform software status control-processor
<output omitted>
Per-core Statistics
CPU0: CPU Utilization (percentage of time spent)
  User:  4.80, System: 10.30, Nice:  0.00, Idle: 84.50
  IRQ:  0.40, SIRQ:  0.00, IOwait:  0.00
CPU1: CPU Utilization (percentage of time spent)
  User:  2.00, System:  3.40, Nice:  0.00, Idle: 94.59
  IRQ:  0.00, SIRQ:  0.00, IOwait:  0.00
CPU2: CPU Utilization (percentage of time spent)
  User:  0.50, System:  0.00, Nice:  0.00, Idle: 99.49
  IRQ:  0.00, SIRQ:  0.00, IOwait:  0.00
CPU3: CPU Utilization (percentage of time spent)
  User: 24.72, System: 75.27, Nice:  0.00, Idle:  0.00
  IRQ:  0.00, SIRQ:  0.00, IOwait:  0.00

 On the other hand, run the show platform software status control-processor brief command, and any of these commands displays the amount of cores installed:

#show platform software status control-processor brief
<output omitted>
CPU Utilization
 Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
  RP0    0   4.30   9.80   0.00  85.90   0.00   0.00   0.00
         1   0.79   0.99   0.00  98.20   0.00   0.00   0.00
         2   0.50   0.00   0.00  99.50   0.00   0.00   0.00
         3  24.60  75.40   0.00   0.00   0.00   0.00   0.00

CPU Cores Distribution

The design of the ISR4300 family results in specific cores used for packet process. Cores four to seven are reserved for packet process on ISR4331 and 4351, while cores two and three are used for ISR4321.

Until and including Cisco IOS XE versions 16.5.x due to performance reasons, the Hierarchical Queue Framework (HQF) features thread always hot-spins and runs at high CPU utilization no matter what configuration is on the box or what amount of traffic goes through the system. On the ISR4300 platforms, this can appear as high CPU utilization on one or more of the cores, because the QFP software runs on the main CPU.

But, after and including Cisco IOS XE versions 16.6.x, a change was implemented so that these platforms would not make the threads hot-spin. In this case, CPU utilization is more distributed through the cores.

To display the hot-spin usage run the show processes cpu platform sorted command, before Cisco IOS XE 16.6.x:

#show processes cpu platform sorted
CPU utilization for five seconds: 28%, one minute: 29%, five minutes: 29%
Core 0: CPU utilization for five seconds: 12%, one minute: 13%, five minutes: 14%
Core 1: CPU utilization for five seconds:  2%, one minute:  3%, five minutes:  3%
Core 2: CPU utilization for five seconds:  0%, one minute:  0%, five minutes:  0%
Core 3: CPU utilization for five seconds: 99%, one minute: 99%, five minutes: 99% <<< hot-spin
   Pid    PPid    5Sec    1Min    5Min  Status        Size  Name
--------------------------------------------------------------------------------
  2541    1955     99%     99%     99%  S       1073807360  qfp-ucode-utah <<< high CPU process
  1551     929      7%      7%      7%  S       2038525952  fman_fp_image

On an eight-core architecture you can see the same result, with a different core on hot-spin, before Cisco IOS XE 16.6.x:

#show processes cpu platform sorted
CPU utilization for five seconds: 15%, one minute: 14%, five minutes: 15%
Core 0: CPU utilization for five seconds:  6%, one minute:  4%, five minutes:  8%
Core 1: CPU utilization for five seconds:  1%, one minute:  0%, five minutes:  2%
Core 2: CPU utilization for five seconds:  9%, one minute: 10%, five minutes:  7%
Core 3: CPU utilization for five seconds:  1%, one minute:  2%, five minutes:  1%
Core 4: CPU utilization for five seconds:  1%, one minute:  1%, five minutes:  1%
Core 5: CPU utilization for five seconds:  0%, one minute:  0%, five minutes:  0%
Core 6: CPU utilization for five seconds: 99%, one minute: 99%, five minutes: 99% <<< hot-spin
Core 7: CPU utilization for five seconds:  0%, one minute:  0%, five minutes:  0%
   Pid    PPid    5Sec    1Min    5Min  Status        Size  Name
--------------------------------------------------------------------------------
  3432    2779     99%     99%     99%  S       1086341120  qfp-ucode-utah <<< high CPU process
  2612    1893      7%      7%      7%  S       2038697984  fman_fp_image
 26114   25132      4%      5%      5%  R         42803200  hman

After and including Cisco IOS XE 16.6.x, however you can see that there is a distribution of load between Core 2 and Core 3:

------------------ show process cpu platform sorted ------------------

CPU utilization for five seconds: 31%, one minute: 32%, five minutes: 29%
Core 0: CPU utilization for five seconds:  3%, one minute:  3%, five minutes:  3%
Core 1: CPU utilization for five seconds:  3%, one minute:  2%, five minutes:  2%
Core 2: CPU utilization for five seconds: 39%, one minute: 41%, five minutes: 34% <<< load distributed
Core 3: CPU utilization for five seconds: 84%, one minute: 83%, five minutes: 79% <<< load distributed
   Pid    PPid    5Sec    1Min    5Min  Status        Size  Name                  
--------------------------------------------------------------------------------
 26939   26344    127%    126%    116%  S       1195311104  qfp-ucode-utah <<< high CPU process

After and including Cisco IOS XE 16.6.x, the same as the previous output applies, but for Cores 4 to 7:

------------------ show process cpu platform sorted ------------------

CPU utilization for five seconds: 30%, one minute: 24%, five minutes: 27%
Core 0: CPU utilization for five seconds: 41%, one minute: 13%, five minutes: 13%
Core 1: CPU utilization for five seconds: 23%, one minute: 11%, five minutes: 13%
Core 2: CPU utilization for five seconds: 19%, one minute: 10%, five minutes: 12%
Core 3: CPU utilization for five seconds: 38%, one minute: 12%, five minutes: 12%
Core 4: CPU utilization for five seconds: 28%, one minute: 26%, five minutes: 28% <<< load distributed
Core 5: CPU utilization for five seconds: 53%, one minute: 40%, five minutes: 37% <<< load distributed
Core 6: CPU utilization for five seconds: 18%, one minute: 16%, five minutes: 17% <<< load distributed
Core 7: CPU utilization for five seconds: 93%, one minute: 81%, five minutes: 81% <<< load distributed
   Pid    PPid    5Sec    1Min    5Min  Status        Size  Name  
--------------------------------------------------------------------------------
26049    25462    164%    165%    170%  S           394128  qfp-ucode-utah <<< high CPU process

Caution: If you suspect a problem with the core CPU usage, open a Technical Assistance Center (TAC) case in order to get assistance and confirm the device stability.

Best Practices to Monitor CPU

Is best to use the specific commands for datapath utilization or IOSd usage, the result of the core display commands can lead to false positive alerts.

The command to monitor the datapath utilization is:

• show platform hardware qfp active datapath utilization

The command to monitor IOSd usage is:

• show process cpu sorted

Use any of these Object Identifiers (OID) to monitor the IOSd CPU usage with Simple Network Management Protocol (SNMP):

• busyPer = IOSd CPU busy percentage in the last 5 second period
• avgBusy1 = IOSd one minute exponentially-decayed moving average of the CPU busy percentage
• avgBusy5 = IOSd five minutes exponentially-decayed moving average of the CPU busy percentage