Understand High CPU Usage Reported for Dataplane on Catalyst 9800

Introduction

This document describes the reasons why a high CPU usage is expected for the Catalyst 9800 if it concerns the dataplane.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

• 9800-CL or 9800-L

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The 9800-CL and 9800-L do not have a hardware dataplane forwarding chip, unlike the 9800-40 and 9800-80. Therefore, the dataplane is emulated by a software process (that runs on the CPU).

A particularity of this software emulated dataplane is that the dataplane process is nearly always using a CPU core at 100%, because its only job is to constantly poll the network interface to see if a packet has arrived, even when there is little to no network traffic.

Symptoms

The Catalyst 9800 can sometimes display a high CPU utilization on given CPU cores. It is very important to identify if it is really a high CPU software problem or a dataplane symptom you can ignore.

The 9800 web UI homepage has a CPU trend widget which shows you the utilization of each CPU core. That widget also shows you what the CPU core is dedicated to.

In this example, you can see a 9800-CL medium OVA (not high throughput) deployed which has 4 control plane core and 2 data planes cores.

In this case, if core 4 and 5 show high utilization, you can ignore it.

CPU Utilization Widget on 9800 Web UI

In the CLI, the command show platform software cpu alloc shows you the role assigned to each CPU core (as shown in the previous web UI widget). The command show process cpu platform sorted shows you the CPU utilization of each core and the top processes.

The process ucode_pkt_PPE0 is the software process which emulates the dataplane and therefore that process can be ignored from the list of processes that contribute to CPU utilization.

The 9800-L assigns statically the last 5 CPU cores to the dataplane. It is typically only core 7 which shows a high utilization (and it is expected) due to the ucode_pkt_PPE0 process that runs on it.

Here is an example of a small 9800-CL OVA where core 2 and 3 are dedicated to the dataplane. The process ucode_pkt_PPE0 shows very high utilization but this is normal and expected even when little network traffic is taking place.

myc9800-CL#show processes cpu platform sorted
CPU utilization for five seconds:  4%, one minute:  5%, five minutes:  6%
Core 0: CPU utilization for five seconds:  2%, one minute:  2%, five minutes:  1%
Core 1: CPU utilization for five seconds:  2%, one minute:  2%, five minutes:  2%
Core 2: CPU utilization for five seconds:  2%, one minute:  1%, five minutes:  2%
Core 3: CPU utilization for five seconds: 99%, one minute: 99%, five minutes: 99%
   Pid    PPid    5Sec    1Min    5Min  Status        Size  Name
--------------------------------------------------------------------------------
 20268   19784     99%     99%     99%  S           271872  ucode_pkt_PPE0
 28292    8568      1%      1%      1%  S          1087088  linux_iosd-imag
 19801   19314      1%      1%      1%  S           250064  fman_fp_image
 32539   32538      0%      0%      0%  S             1692  inet_gethost
 32538   31946      0%      0%      0%  S              364  inet_gethost
 32507   29493      0%      0%      0%  S             5592  confd-startup.s

How to Verify the Dataplane Utilization

You possibly wonder how to verify if the Catalyst 9800 is overwhelmed with network traffic if the dataplane CPU utilization is always high. There are specific widgets and CLI commands for this.

In the Web UI, navigate to Monitor > System > CPU utilization. On the right, you have Datapath utilization.

Caution: The 9800-L web UI does not display the datapath utilization at this time. Enhancement Cisco bug ID CSCwo11247 was submitted for it.

The most important metric is Packet Processed (PP) in the form of a percentage. In this example, the datapath is 99.56% idle and is therefore logically 0.44% busy with the process of network packets, which is very low. RX depicts time spent receiving traffic and TM time spent transmitting traffic. Idle is basically the opposite of PP (100% - PP), that is the time the dataplane spends waiting for something to do.

Datapath Utilization

The equivalent CLI command of this dashboard is :

WLC#show platform hardware chassis active qfp datapath utilization sum
  CPP 0:                     5 secs        1 min        5 min       60 min
Input:     Total (pps)           10           10           10            2
                 (bps)         5760         9792        12760         2672
Output:    Total (pps)            6            4            8            5
                 (bps)        19816        11808        47184        42536
Processing: Load (pct)            0            0            0            0

WLC#