Increase Data Plane QFP exmem Memory on CSR1000V

Introduction

This document describes how to increase Data Plane (DP) Quantum Flow Processor (QFP) External Memory Manger (exmem) memory on Cisco Cloud Services Router 1000V (CSR1000V).

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• CSR1000V routers.
• Cisco IOS®-XE.
• ESXi vCenter.

Components Used

The information in this document is based on these software and hardware versions:

• CSR1000V router deployed in VMware ESXi, 6.7.0
• Cisco IOS®-XE 16.12.6

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background

It is important to understand how Cisco IOS®-XE is built. With Cisco IOS®-XE, Cisco has moved to a Linux kernel and all the subsystems have been broken down into processes. All the subsystems that were inside Cisco IOS® before, now run as software processes within the Linux Operating System (OS). Cisco IOS® itself runs as a daemon (Cisco IOSd) within the Linux OS. Cisco IOS®-XE retains not only the same look and feel of the classic Cisco IOS®, but also its operation, support, and management.

Memory assigned to Virtual Machine (VM) is allocated to Cisco IOS® daemon (Cisco IOSd) and to Data Plane memory.

By default, memory allocation on Cisco IOSd is 2.5 GB and for Data Plane is 1.5 GB, the amount of memory available to Cisco IOSd is controlled by memory add-on licenses and certain feature licenses (broadband feature license) and it does not change when more memory is allocated to VM.

In Data Plane memory, exmem memory size on CSR1000V by default is 256MB (268435456 Bytes), which can very small amount for certain specific features configurations, like QoS Polices, NAT sessions, IPSec Security Associations, and so on. and when exmem high utilization is observed -

CSR_ESXi#show platform hardware qfp active infrastructure exmem statistics
QFP exmem statistics

Type: Name: DRAM, QFP: 0
  Total: 268435456
  InUse: 263066746
  Free: 5368710
  Lowest free water mark: 206885888
Type: Name: IRAM, QFP: 0
  Total: 2097152
  InUse: 218112
  Free: 1879040
  Lowest free water mark: 1879040
Type: Name: SRAM, QFP: 0
  Total: 0
  InUse: 0
  Free: 0
  Lowest free water mark: 0

Jul 5 08:016:32.486: %QFPOOR-2-LOWRSRC_PERCENT_CRIT: R0/0: cpp_ha: QFP 0 DRAM(EXMEM) at 99 percent, exceeds critical level 95

Exmem is the dynamic memory that allows the actual packet handling, used for features to scale and manages static and dynamic memory used by Cisco Packet Processor (CPP).

Note: Refer to Cisco CSR 1000v Memory Allocation for detailed Cisco IOSd memory add-on information.

Configure CSR1000V exmem allocation

This section describes how to configure Data Plane templates to increase QFP exmem size.

Template options available are -

control-plane-extra-heavy 
control-plane-heavy 
data-plane-heavy
data-plane-normal
service-plane-heavy
service-plane-medium 

Note: This document does not cover control plane templates and service plane templates, for more details about these templates please refer to - Cisco CSR 1000v and Cisco ISRv Software Configuration Guide

Template: Data Plane Heavy

The next table shows the QFP exmem memory allocation for the Data Plane Heavy template.

Router#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Router(config)#platform resource data-plane-heavy
Please reboot to activate this template

Router(config)#end
Router#write memory 
Building configuration...
[OK]
CSR_ESXi#reload

Summary Steps

1. enable
2. configure terminal
3. platform resource data-plane-heavy
4. end
5. write memory
6. reload

Verify

1. 8GB assigned to VM .

cisco CSR1000V (VXE) processor (revision VXE) with 2295972K/3075K bytes of memory.
Processor board ID 949XJJNMSHM
3 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
8105924K bytes of physical memory.
7774207K bytes of virtual hard disk at bootflash:.
0K bytes of WebUI ODM Files at webui:.

2. No changes observed in terms of memory allocated to Cisco IOSd and Data Plane after template change.

Router#show platform software vmemory info

Memory Upgrade Limits:
        Total System Memory:            7915 MB
        Memory From Upgrade Licenses:   N/A(Smart License Enabled)
        Memory From Feature Licenses:   N/A(Smart License Enabled)
Memory Available For Upgrade:
        Available System Memory:        3819 MB
        Available Upgrade Licensed Memory:      N/A(Smart License Enabled)
        Available Feature Licensed Memory:      N/A(Smart License Enabled)
Current Memory Allocation:
        Cisco IOSD:   2560 MB (default) + 0 MB upgrade
        Data Plane:     1536 MB (default) + 0 MB upgrade

3. QFP exmem total available is now 512MB.

Router#show platform hardware qfp active infrastructure exmem statistics
QFP exmem statistics

Type: Name: DRAM, QFP: 0
  Total: 536870912
  InUse: 61156352
  Free: 475714560
  Lowest free water mark: 475321344
Type: Name: IRAM, QFP: 0
  Total: 2097152
  InUse: 218112
  Free: 1879040
  Lowest free water mark: 1879040
Type: Name: SRAM, QFP: 0
  Total: 0
  InUse: 0
  Free: 0
  Lowest free water mark: 0

Router#show platform software cpu alloc
CPU alloc information:

  Control plane cpu alloc: 0

  Data plane cpu alloc: 1-7

  Service plane cpu alloc: 0

  Template used: CLI-data_plane_heavy

Template: Data Plane Normal

Router#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Router(config)#platform resource data-plane-normal
Please reboot to activate this template

Router(config)#end
Router#write memory 
Building configuration...
[OK]
CSR_ESXi#reload

Summary Steps

1. enable
2. configure terminal
3. platform resource data-plane-normal
4. end
5. write memory
6. reload

Data Plane Normal, uses 256MB in QFP exmem regardless of amount of VM Memory assigned.

cisco CSR1000V (VXE) processor (revision VXE) with 2295972K/3075K bytes of memory.
Processor board ID 949XJJNMSHM
3 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
16363460K bytes of physical memory.
7774207K bytes of virtual hard disk at bootflash:.
0K bytes of WebUI ODM Files at webui:.

Router#show platform software vmemory info

Memory Upgrade Limits:
        Total System Memory:            15979 MB
        Memory From Upgrade Licenses:   N/A(Smart License Enabled)
        Memory From Feature Licenses:   N/A(Smart License Enabled)
Memory Available For Upgrade:
        Available System Memory:        11883 MB
        Available Upgrade Licensed Memory:      N/A(Smart License Enabled)
        Available Feature Licensed Memory:      N/A(Smart License Enabled)
Current Memory Allocation:
        Cisco IOSD:   2560 MB (default) + 0 MB upgrade
        Data Plane:     1536 MB (default) + 0 MB upgrade

Router#show platform hardware qfp active infrastructure exmem statistics
QFP exmem statistics

Type: Name: DRAM, QFP: 0
  Total: 268435456
  InUse: 61156352
  Free: 207279104
  Lowest free water mark: 206885888
Type: Name: IRAM, QFP: 0
  Total: 2097152
  InUse: 218112
  Free: 1879040
  Lowest free water mark: 1879040
Type: Name: SRAM, QFP: 0
  Total: 0
  InUse: 0
  Free: 0
  Lowest free water mark: 0

Note: QFP exmem memory allocation depends solely on the amount of memory assigned to VM and the data plane template applied. It does not depend on memory add-on licenses installed.

Troubleshoot

There is currently no specific information available on how to troubleshoot this configuration.