Introduction
On Aggregation Services Router 9000 (ASR9K) running Enhanced XR (eXR), also known as 64-bit XR, release 6.2.1 and newer the command “show memory summary” doesn’t show all internal memory documented on the corresponding Data Sheet, which might confuse some customers.
Problem Details
Below we will use an ASR9901, running 6.4.2, as an example.
According to the datasheet the integrated Route Processor (RP) has 32 GigaBytes (GB) of Random Access Memory (RAM).
RP/0/RSP0/CPU0:R1#show platform
Node Type State Config state
--------------------------------------------------------------------------------
0/RSP0/CPU0 ASR9901-RP(Active) IOS XR RUN NSHUT
0/FT0 ASR-9901-FAN OPERATIONAL NSHUT
0/FT1 ASR-9901-FAN OPERATIONAL NSHUT
0/FT2 ASR-9901-FAN OPERATIONAL NSHUT
0/0/CPU0 ASR9901-LC IOS XR RUN NSHUT
"Show memory summary" only shows around 27GB, despite the system recognizing the total memory as 32GB under “admin show system resources”.
RP/0/RSP0/CPU0:R1#show memory summary
node: node0_RSP0_CPU0
------------------------------------------------------------------
Physical Memory: 27089M total (22185M available)
Application Memory : 27089M (22013M available)
Image: 4M (bootram: 0M)
Reserved: 0M, IOMem: 0M, flashfsys: 0M
Total shared window: 133M
RP/0/RSP0/CPU0:R1#admin show system resources
Node Physical Application Boot Partition CPUs Shmwin
Total Available Cached Total Available Cached Total Available
0/RSP0-Host 32415M^ 512M^ 135M 31655M^ 500M^ 132 923M 542M 4 N/A
0/RSP0-Admin 1940M 1072M 144M 1894M 1047M 141 2308M 1252M 1 N/A
0/RSP0-XR 27739M 22548M 999M 27089M 22020M 975 N/A N/A 2 6655M
<snip>
The same behavior can be observed on other ASR9K devices, for example: ASR9010 with eXR 6.2.3
Looking at the datasheet A9K-RSP880-TR should have 16GB of RAM.
RP/0/RSP0/CPU0:R2#show platform
Node Type State Config state
-------------------------------------------------------------------------------
0/RSP0/CPU0 A9K-RSP880-TR(Active) IOS XR RUN NSHUT
0/FT0 ASR-9010-FAN-V2 OPERATIONAL NSHUT
0/FT1 ASR-9010-FAN-V2 OPERATIONAL NSHUT
0/1/CPU0 A9K-MOD400-TR IOS XR RUN NSHUT
0/1/0 A9K-MPA-20X10GE OK
0/1/1 A9K-MPA-1X100GE OK
0/PT0 A9K-AC-PEM-V3 OPERATIONAL NSHUT
0/PT1 A9K-AC-PEM-V3 OPERATIONAL NSHUT
"Show memory summary" shows 12.5GB and "admin show system resource shows 16GB
RP/0/RSP0/CPU0:R2#show memory summary
node: node0_RSP0_CPU0
------------------------------------------------------------------
Physical Memory: 12496M total (8465M available)
Application Memory : 12496M (8287M available)
Image: 4M (bootram: 0M)
Reserved: 0M, IOMem: 0M, flashfsys: 0M
Total shared window: 128M
RP/0/RSP0/CPU0:R2#admin show system resou
Node Physical Application Boot Partition CPUs Shmwin
Total Available Cached Total Available Cached Total Available
0/RSP0-Host 16217M^ 276M^ 29M 15837M^ 271M^ 28 923M 592M 8 N/A
0/RSP0-Admin 1940M 1158M 78M 1894M 1131M 76 2308M 1451M 1 N/A
0/RSP0-XR 12796M 8488M 888M 12496M 8288M 867 N/A N/A 6 3071M
0/1-Host 24491M 11510M 127M 23917M 11241M 124 923M 592M 6 N/A
0/1-Admin 1008M 471M 90M 984M 460M 88 2308M 1528M 1 N/A
0/1-XR 10948M 5902M 2343M 10691M 5764M 2288 N/A N/A 5 3071M
However, if we check 32-bit XR, it shows the total physical memory as documentated in the datasheet:
RP/0/RSP0/CPU0:R3#show memory summary
Physical Memory: 16384M total (12600M available)
Application Memory : 16043M (12600M available)
Image: 100M (bootram: 100M)
Reserved: 224M, IOMem: 0, flashfsys: 0
Total shared window: 51M
RP/0/RSP0/CPU0:R3#show platform
Node Type State Config State
-----------------------------------------------------------------------------
0/RSP0/CPU0 A99-RSP-TR(Active) IOS XR RUN PWR,NSHUT,MON
0/RSP1/CPU0 A99-RSP-TR(Standby) IOS XR RUN PWR,NSHUT,MON
0/1/CPU0 A9K-MOD400-SE IOS XR RUN PWR,NSHUT,MON
0/1/0 A9K-MPA-20X10GE OK PWR,NSHUT,MON
0/1/1 A9K-MPA-2X100GE OK PWR,NSHUT,MON
0/2/CPU0 A99-8X100GE-SE IOS XR RUN PWR,NSHUT,MON
0/4/CPU0 A9K-MOD400-TR IOS XR RUN PWR,NSHUT,MON
0/4/0 A9K-MPA-8X10GE OK PWR,NSHUT,MON
0/4/1 A9K-MPA-2X40GE OK PWR,NSHUT,MON
Analysis
From release 6.2.1 onwards, eXR on ASR9K will use the Virtual Machine (VM) virtualisation model, switching from using Linux Container (LXC) model which was used in releases 6.1.2 and 6.1.3.
Containers can pack a lot more applications into a single physical server than a virtual machine (VM) can.
VM's can take up a lot of system resources. Each VM runs not just a full copy of an operating system, but a virtual copy of all the hardware that the operating system (OS) needs to run. This may add upto a lot of memory and CPU cycles. In contrast, all that a container requires is enough of an operating system, supporting programs and libraries, and system resources to run a specific program. However, VM's give much more flexibilty to accomodate functions like In-Service Software Upgrade (ISSU).
Since Linux Containers shares the physical resources and kernel of the host, while each VM requires its own OS and virtualized hardware, this would explain why each VM is allocated a specific amount of resources from the overall host which results in the show memory summary not displaying the overall total physical memory on eXR releases 6.2.1 and later. Linux Conainers can also have limit for allocated memory as it appears on NCS5000 and NCS5500 platforms and the same behavios for the 'show memory summary" CLI will be observed on these platforms.
Feedback