Quantum Virtualized Packet Core Distributed Instance (QvPC-DI) : iftask and Network Processor Unit (NPU) performance monitoring

Available Languages

Updated:June 9, 2018

Document ID:213397

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

Components Used

Iftask Architecture

Monitoring iftask performance

Bulkstats

Introduction

This document describes how to monitor performance of iftask / NPU on QvPC-DI.

Components Used

The information in this document is based on QvPC-DI.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Iftask Architecture

iftask is a process in QvPC-DI. It enables Data Plane Development Kit (DPDK) functionality on the Service Function Virtual Card (SF) and Control Function Virtual Card (CF) for the DI network ports and the service ports. DPDK is a more efficient way to handle Input/output in virtualized environments.

The device drivers of high performance Network interface controllers (NIC) are now moved to userspace, that avoids expensive context switches (userspace/kernelspace).

The drivers run in un-interruptable mode in userspace, and threads have direct access to the HW queues/ring buffers in these NIC drivers.

Documentation about architecture is available at :

Ultra Services Platform (USP) Introduction from Ultra Gateway Platform System Administration Guide.

Availability for different versions.

In depth iftask architecture (for SF) is seen in this diagram:

During deployment, certain amount of virtual central processing units (vCPU's) are statically allocated to the iftask process. This reduces the amount of cores for userspace applications (sessmgr etc), but it greatly enhance the performance of I/O.

This allocation is done via below parameter in that param.cfg template that is associated with each SF/CF during deployment:

IFTASK_CORES (% of available cores to be assigned with iftask)
(IFTASK_CRYPTO_CORES - not discussed here)
(IFTASK_MCDMA_CORES - not discussed here)

On an SF, The iftask process internally will distribute its assigned cores into:
- Poll Mode Drivers (PMD) vCPU’s (doing tx/rx/vnpu activity
- MCDMA vCPU’s, doing transfer of packets from iftask to sessmgr and back
On a CF, no MCDMA vCPU's are required, since SF's are not hosting sessmgr processes.

The command 'show cloud hardware iftask' gives more details about this on your QVPC-DI deployment:

[local]UGP# show cloud hardware iftask
Card 1:
  Total number of cores on VM:      8
  Number of cores for PMD only:     0
  Number of cores for VNPU only:    0
  Number of cores for PMD and VNPU: 2     <--  CF: 2 out of 8 cores are assigned to iftask PMD/VNPU
  Number of cores for MCDMA:        0     <--  CF: no cores allocated to MCDMA as there is no sessmgr process on CF
  Number of cores for Crypto:        0
  Hugepage size:           2048 kB
  Total hugepages:      3670016 kB
  NPUSHM hugepages:           0 kB
  CPU flags: avx sse sse2 ssse3 sse4_1 sse4_2
  Poll CPU's: 1 2
  KNI reschedule interval: 5 us
 ...
Card 3:
  Total number of cores on VM:      8
  Number of cores for PMD only:     0
  Number of cores for VNPU only:    0
  Number of cores for PMD and VNPU: 2     <-- SF: 2 out of 8 core are assigned to iftask PMD/VNPU
  Number of cores for MCDMA: 1            <-- SF: 1 out of 8 cores is assigned to iftak MCDMA
  Number of cores for Crypto: 0 
  Hugepage size: 2048 kB 
  Total hugepages: 4718592 kB 
  NPUSHM hugepages: 0 kB 
  CPU flags: avx sse sse2 ssse3 sse4_1 sse4_2 
  Poll CPU's: 1 2 3 
  KNI reschedule interval: 5 us

Command 'show cloud configuration' will give more details on the parameters used:

[local]UGP# show cloud configuration
Card 1:
  Config Disk Params:
-------------------------
CARDSLOT=1
CPUID=0
CARDTYPE=0x40010100
DI_INTERFACE=BOND:TYPE:ixgbevf-1,TYPE:ixgbevf-2
DI_INTERFACE_VLANID=2111
VNFM_INTERFACE=MAC:fa:16:3e:23:aa:e9
VNFM_PROXY_ADDRS=172.16.180.3,172.16.180.5,172.16.180.6
MGMT_INTERFACE=MAC:fa:16:3e:87:23:9b
VNFM_IPV4_ENABLE=true
VNFM_IPV4_DHCP_ENABLE=true

  Local Params:
-------------------------
CARDSLOT=1
CARDTYPE=0x40010100
CPUID=0

...

Card 3:
  Config Disk Params:
-------------------------
CARDSLOT=3
CPUID=0
CARDTYPE=0x42030100
DI_INTERFACE=BOND:TYPE:ixgbevf-1,TYPE:ixgbevf-2
SERVICE1_INTERFACE=BOND:TYPE:ixgbevf-3,TYPE:ixgbevf-4
SERVICE2_INTERFACE=BOND:TYPE:ixgbevf-5,TYPE:ixgbevf-6
DI_INTERFACE_VLANID=2111
VNFM_INTERFACE=MAC:fa:16:3e:29:c6:b7
IFTASK_CORES=30
VNFM_IPV4_ENABLE=true
VNFM_IPV4_DHCP_ENABLE=true

  Local Params:
-------------------------
CARDSLOT=3
CARDTYPE=0x42010100
CPUID=0

Monitoring iftask performance

The iftask process can be monitored in several ways.

Consolidate list of show commands:

show subscribers data-rate 
show npumgr dinet utilization pps 
show npumgr dinet utilization pps
show cloud monitor di-network summary
show cloud hardware iftask
show cloud configuration
show iftask stats summary
show port utilization table
show npu utilization table
show npumgr utilization information
show processes cpu

Command #show cpu info verbose will not give information about the iftask cores. They will always be listed at 100% utilization.

In the example below, core 1,2,3 are associated with iftask, and are listed at 100% utilization, this is expected.

Card 3, CPU 0:
  Status                : Standby, Kernel Running, Tasks Running
  Load Average          : 3.12, 3.12, 3.13 (3.95 max)
  Total Memory          : 16384M
  Kernel Uptime         : 4D 21H 56M
  Last Reading:
    CPU Usage All       : 1.9% user, 0.3% sys, 0.0% io, 0.0% irq, 97.8% idle
        Core 0          : 5.8% user, 0.2% sys, 0.0% io, 0.0% irq, 94.0% idle
        Core 1          : Not Averaged (Poll CPU)
        Core 2          : Not Averaged (Poll CPU)
        Core 3          : Not Averaged (Poll CPU)
        Core 4          : 2.2% user, 0.2% sys, 0.0% io, 0.0% irq, 97.6% idle
        Core 5          : 0.8% user, 0.5% sys, 0.0% io, 0.0% irq, 98.7% idle
        Core 6          : 0.4% user, 0.5% sys, 0.0% io, 0.0% irq, 99.1% idle
        Core 7          : 0.1% user, 0.3% sys, 0.0% io, 0.0% irq, 99.6% idle
    Poll CPUs           : 3 (1, 2, 3)
        Core 1          : 100.0% user, 0.0% sys, 0.0% io, 0.0% irq, 0.0% idle
        Core 2          : 100.0% user, 0.0% sys, 0.0% io, 0.0% irq, 0.0% idle
        Core 3          : 100.0% user, 0.0% sys, 0.0% io, 0.0% irq, 0.0% idle
    Processes / Tasks   : 143 processes / 16 tasks
    Network mcdmaN      : 0.002 kpps rx, 0.001 mbps rx, 0.002 kpps tx, 0.001 mbps tx
    File Usage          : 1504 open files, 1627405 available
    Memory Usage        : 7687M 46.9% used
    Memory Details:
      Static            : 330M kernel, 144M image
      System            : 10M tmp, 0M buffers, 54M kcache, 79M cache
      Process/Task      : 6963M (120M small, 684M huge, 6158M other)
      Other             : 104M shared data
      Free              : 8696M free
      Usable            : 5810M usable (8696M free, 0M reclaimable, 2885M reserved by tasks)

Command #show npu utilization table will give a good summary on the utilization of each core associated with iftask process (on each card).

Note: Important here is to identify if some cores are consistently higher in utilization than other cores.

[local]UGP# show npu utilization table
          -------iftask-------
   lcore     now   5min  15min
--------  ------ ------ ------
  01/0/1      0%     0%     0%
  01/0/2      0%     0%     0%
  02/0/1      0%     0%     0%
  02/0/2      2%     1%     0%
  03/0/1      0%     0%     0%
  03/0/2      0%     0%     0%
  03/0/3      0%     0%     0%
  04/0/1      0%     0%     0%
  04/0/2      0%     0%     0%
  04/0/3      0%     0%     0%
  05/0/1      0%     0%     0%
  05/0/2      0%     0%     0%
  05/0/3      0%     0%     0%

Command #show npumgr utilization information (hidden command)

This command gives more information about each iftask core, and what is consuming CPU on these cores.

Note: PMD cores are having their CPU consumed on PortRX, PortTX, KNI, Cipher.

Note: MCDMA cores are having their CPU consumed by MCDMA.

Both PMD and MCDMA cores have even load.

If this is not the case, some tuning might be required (allocating more/less MDMA cores for example).

******** show npumgr utilization information 3/0/0 *******
    5-Sec Avg: lcore01| lcore02| lcore03| lcore04| lcore05| lcore06| lcore07| lcore08| lcore09| lcore10| lcore11| lcore12|
         Idle:     41%|     47%|     73%|     62%|     48%|     49%|     69%|     71%|     49%|     64%|     48%|     69%|
       PortRX:     32%|     36%|      0%|      0%|      0%|      0%|     31%|     29%|      0%|      0%|      0%|     31%|
       PortTX:     16%|     18%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|
        KniRX:      3%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|
      McdmaRX:      0%|      0%|     11%|     15%|     22%|     21%|      0%|      0%|     20%|     14%|     20%|      0%|
        Mcdma:      0%|      0%|      1%|      2%|      3%|      2%|      0%|      0%|      3%|      3%|      3%|      0%|
   McdmaFlush:      0%|      0%|     15%|     21%|     27%|     27%|      0%|      0%|     27%|     18%|     28%|      0%|
       Cipher:      7%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|
 rx kbits/sec:  173914|  257249|  248543|  224081|  319973|  299437|  450992|  380112|  391400|  325099|  355809|  399515|
rx frames/sec:   30557|   36549|   37465|   32560|   46914|   43938|   70711|   54818|   58482|   47462|   53477|   58793|
 tx kbits/sec:  169641|  251958|  253930|  228760|  326714|  305750|  440773|  372187|  399806|  331914|  363491|  391002|
tx frames/sec:   30551|   36524|   37465|   32560|   46914|   43938|   70681|   54785|   58483|   47462|   53477|   58749|

    5-Min Avg: lcore01| lcore02| lcore03| lcore04| lcore05| lcore06| lcore07| lcore08| lcore09| lcore10| lcore11| lcore12|
         Idle:     18%|     36%|     60%|     62%|     46%|     45%|     65%|     62%|     44%|     53%|     39%|     65%|
       PortRX:     29%|     45%|      0%|      0%|      0%|      0%|     35%|     38%|      0%|      0%|      0%|     35%|
       PortTX:     17%|     20%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|
        KniRX:      3%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|
          Kni:      1%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|
      McdmaRX:      0%|      0%|     17%|     17%|     23%|     24%|      0%|      0%|     24%|     19%|     26%|      0%|
        Mcdma:      0%|      0%|      2%|      2%|      3%|      3%|      0%|      0%|      4%|      3%|      4%|      0%|
   McdmaFlush:      0%|      0%|     21%|     20%|     28%|     28%|      0%|      0%|     28%|     24%|     30%|      0%|
       Cipher:     32%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|
 rx kbits/sec:  217296|  297699|  261605|  268546|  389380|  397101|  479195|  528945|  438931|  360583|  454262|  468575|
rx frames/sec:   39194|   42022|   40122|   40086|   58217|   59507|   80931|   76094|   67224|   54592|   68565|   67013|
 tx kbits/sec:  211773|  291616|  267373|  274308|  397747|  405655|  467493|  517944|  448590|  368412|  464116|  458868|
tx frames/sec:   39182|   41998|   40122|   40086|   58217|   59507|   80895|   76058|   67224|   54592|   68565|   66973|

   15-Min Avg: lcore01| lcore02| lcore03| lcore04| lcore05| lcore06| lcore07| lcore08| lcore09| lcore10| lcore11| lcore12|
         Idle:     22%|     37%|     60%|     61%|     45%|     46%|     64%|     64%|     44%|     49%|     41%|     66%|
       PortRX:     33%|     43%|      0%|      0%|      0%|      0%|     36%|     36%|      0%|      0%|      0%|     34%|
       PortTX:     18%|     20%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|
        KniRX:      4%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|
      McdmaRX:      0%|      0%|     16%|     16%|     24%|     24%|      0%|      0%|     24%|     22%|     25%|      0%|
        Mcdma:      0%|      0%|      2%|      2%|      3%|      3%|      0%|      0%|      4%|      4%|      4%|      0%|
   McdmaFlush:      0%|      0%|     21%|     20%|     28%|     28%|      0%|      0%|     28%|     26%|     30%|      0%|
       Cipher:     23%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|      0%|
 rx kbits/sec:  225682|  297062|  267631|  272770|  419821|  405286|  489633|  511100|  476834|  401810|  467134|  466549|
rx frames/sec:   39670|   42772|   40892|   40834|   62740|   61170|   83540|   76519|   72158|   60242|   70707|   67961|
 tx kbits/sec:  220089|  290875|  273510|  278639|  428840|  414080|  477557|  500037|  487201|  410421|  477298|  456711|
tx frames/sec:   39657|   42748|   40892|   40834|   62740|   61170|   83504|   76484|   72158|   60242|   70707|   67925|

@ tick 896633 (+ve-skew-cnt=123633, -ve-skew-cnt=0), failed samples 0

Commands #show npumgr dinet utilization pps, #show npumgr dinet utilization bbps and #show port utilization table

They provide information about the load on the DI ports, and the services ports.

The actual performance depends on NIC's/CPU's and CPU allocation to iftask.

[local]UGP# show npumgr dinet utilization pps
                           ------ Average DINet Port Utilization (in kpps) ------
Port   Type                       Current           5min             15min
                                 Rx      Tx      Rx       Tx       Rx     Tx
----- ------------------------  ------- -------  ------- -------  ------- -------
 1/0  Virtual Ethernet          0       0        0       0        0       0
 2/0  Virtual Ethernet          0       0        0       0        0       0
 3/0  Virtual Ethernet          0       0        0       0        0       0
 4/0  Virtual Ethernet          0       0        0       0        0       0
 5/0  Virtual Ethernet          0       0        0       0        0       0

[local]UGP# show npumgr dinet utilization bps
                           ------ Average DINet Port Utilization (in mbps) ------
Port   Type                       Current           5min             15min
                                 Rx      Tx      Rx       Tx       Rx     Tx
----- ------------------------  ------- -------  ------- -------  ------- -------
 1/0  Virtual Ethernet          1       1        1       1        1       1
 2/0  Virtual Ethernet          1       0        1       0        1       0
 3/0  Virtual Ethernet          0       0        0       0        0       0
 4/0  Virtual Ethernet          0       0        0       0        0       0
 5/0  Virtual Ethernet          0       0        0       0        0       0

[local]UGP# show port utilization table
                                ------ Average Port Utilization (in mbps) ------
Port   Type                       Current           5min             15min
                                 Rx      Tx      Rx       Tx       Rx     Tx
----- ------------------------  ------- -------  ------- -------  ------- -------
 1/1  Virtual Ethernet          0       0        0       0        0       0
 2/1  Virtual Ethernet          0       0        0       0        0       0
 3/10 Virtual Ethernet          0       0        0       0        0       0
 3/11 Virtual Ethernet          0       0        0       0        0       0
 4/10 Virtual Ethernet          0       0        0       0        0       0
 4/11 Virtual Ethernet          0       0        0       0        0       0
 5/10 Virtual Ethernet          0       0        0       0        0       0
 5/11 Virtual Ethernet          0       0        0       0        0       0

Command #show cloud monitor di-network summary

This command monitors the health of the DI network. Cards are sending heartbeats to each other, and the Loss is monitored. In a healthy system, no loss is reported.

[local]UGP# show cloud monitor di-network summary
Card 3 Heartbeat Results:
ToCard   Health   5MinLoss   60MinLoss
   1      Good       0.00%      0.00%
   2      Good       0.00%      0.00%
   4      Good       0.00%      0.00%
   5      Good       0.00%      0.00%
Card 4 Heartbeat Results:
ToCard   Health   5MinLoss   60MinLoss
   1      Good       0.00%      0.00%
   2      Good       0.00%      0.00%
   3      Good       0.00%      0.00%
   5      Good       0.00%      0.00%
Card 5 Heartbeat Results:
ToCard   Health   5MinLoss   60MinLoss
   1      Good       0.00%      0.00%
   2      Good       0.00%      0.00%
   3      Good       0.00%      0.00%
   4      Good       0.00%      0.00%

Command #show iftask stats summary

With higher NPU loads, it might be possible that traffic is getting dropped.

To evaluate this, command #show iftask stats summary output can be taken.

Note: DISCARDS can be non-zero.

Note: all other counters should not increment.

[local]VPC# show iftask stats summary^M
Thursday January 18 16:01:29 IST 2018
 
-----------------------------------------------------------------------------------------------
Counter                               SF3         SF4         SF5         SF6         SF7         SF8         SF9         SF10         SF11         SF12         ___TOTAL___  
------------------------------------------------------------------------------------------------
svc_rx                       32491861127   16545600654   37041906441   37466889835   32762859630   34931554543   38861410897   16025531220   33566817747   32823851780  312518283874
svc_tx                       46024774071   14811663244   40316226774   39926898585   40803541378   48718868048   35252698559    1738016438    4249156512   40356388348  312198231957
di_rx                        42307187425   14637310721   40072487209   39584697117   41150445596   44534022642   31867253533    1731310419    4401095653   40711142205  300996952520
di_tx                        28420090751   16267050562   36423298668   36758561246   32731606974   30366650898   35201117980   16009902791   33536789041   32815316570  298530385481
__ALL_DROPS__                    1932492           252         17742        790473         11228        627018        844812         60402             0        460830       4745249
svc_tx_drops                           0             0             0             0             0             0             0             0             0             0             0
di_rx_drops                            0             1             0             0            49           113           579         30200             0          4888         35830
di_tx_drops                            0             0             0             0             0             0             0             0             0             0             0
sw_rss_enq_drops                       0             0             0             0             0             0             0             0             0             0             0
kni_thread_drops                       0             0             0             0             0             0             0             0             0             0             0
kni_drops                              0             1             0             0             0             0           124         30200             0             0         30325
mcdma_drops                            0             0             0           168            80        194535        758500             0             0         11628        964911
mux_deliver_hop_drops                  0             0             0             0             0             0             0             0             0          1019          1019
mux_deliver_drops                      0             0             0             0             0             0             0             0             0             0             0
mux_xmit_failure_drops                 0             3             0             0             0             0             7             2             0             0            12
mc_dma_thread_enq_drops                0             0             0             0            49           113           580             0             0          3457          4199
sw_tx_egress_enq_drops           1904329             0             0        787971          9004        429214         85022             0             0        429810       3645350
cpeth0_drops                           0             0             0             0             0             0             0             0             0             0             0
mcdma_summary_drops                28163           247         17742          2334          2046          3043             0             0             0         10028         63603
fragmentation_err                      0             0             0             0             0             0             0             0             0             0             0
reassembly_err                         0             0             0             0             0             0             0             0             0             0             0
reassembly_ring_enq_err                0             0             0             0             0             0             0             0             0             0             0
__DISCARDS__                    20331090       9051092      23736055      23882896      23807520      24231716      24116576       8944291      22309474      20135799      20135799

Bulkstats

Bulkstat scheme is developed for QPVC-DI performance related to iftask/dinet. This is useful for monitoring the dinet, service ports, and npu utilization from a performance/load perspective:

card schema iftask-dinet format EMS,IFTASKDINET,%date%,%time%,%dinet-rxpkts-curr%,%dinet-txpkts-curr%,%dinet-rxpkts-5minave%,%dinet-txpkts-5minave%,%dinet-rxpkts-15minave%,%dinet-txpkts-15minave%,%dinet-txdrops-curr%,%dinet-txdrops-5minave%,%dinet-txdrops-15minave%,%npuutil-now%
   file 2
port schema iftask-port format EMS,IFTASKPORT,%date%,%time%,%util-rxpkts-curr%,%util-txpkts-curr%,%util-rxpkts-5min%,%util-txpkts-5min%,%util-rxpkts-15min%,%util-txpkts-15min%,%util-txdrops-curr%,%util-txdrops-5min%,%util-txdrops-15min%
  file 3
card schema npu-util format EMS,NPUUTIL,%date%,%time%,%npuutil-now%,%npuutil-5minave%,%npuutil-15minave%,%npuutil-rxbytes-5secave%,%npuutil-txbytes-5secave%,%npuutil-rxbytes-5minave%,%npuutil-txbytes-5minave%,%npuutil-rxbytes-15minave%,%npuutil-txbytes-15minave%,%npuutil-rxpkts-5secave%,%npuutil-txpkts-5secave%,%npuutil-rxpkts-5minave%,%npuutil-txpkts-5minave%,%npuutil-rxpkts-15minave%,%npuutil-txpkts-15minave%

Contributed by Cisco Engineers

Steven Loos

Was this Document Helpful?

Feedback

Contact Cisco

Open a Support Case
(Requires a Cisco Service Contract)

Quantum Virtualized Packet Core Distributed Instance (QvPC-DI) : iftask and Network Processor Unit (NPU) performance monitoring

Available Languages

Bias-Free Language

Contents

Introduction

Components Used

Iftask Architecture

Monitoring iftask performance

Bulkstats

Contributed by Cisco Engineers

Was this Document Helpful?

Contact Cisco

This Document Applies to These Products