Configure ltrace Scale Factors on ASR9K Route Processors and Line Cards for Efficient Memory Management

Introduction

This document describes ltrace reduction with its benefits and debugability tradeoff for Aggregation Services Router (ASR) 9K on Cisco IOS® XR.

What are ltraces?

Ltraces are used extensively in Cisco IOS XR for debugging purposes. Every process in Cisco IOS XR uses ltraces which are stored in memory and thus consume memory as they grow during the time of operation of the router.

Configurations to Change ltrace Use

Cisco IOS XR release 5.3.2 onwards brought changes to provide options to change ltrace usage via two ddtses:

1. Cisco bug ID CSCus39188 Provide option to limit ltrace shmem usage with scale option
2. Cisco bug ID CSCus39159 Provide more detail on shmem usage and details in show memory summary

A Reload is required after the configuration in order to bring the changes into effect.

Note: Due to DDTS Cisco bug ID CSCuz22580 ltrace dynamic scale factor cannot be used on release 5.3.x as it results in fabric cards continually crash. This command can be used on routers on Cisco IOS XR Release 6.0.x and later.

Default Behaviour

Please note the default behaviour from ltrace point of view on the ASR9K router on Cisco IOS XR 5.3.2 and onwards.

RP/0/RSP1/CPU0:#show shmem summary location 0/1/cPU0 
Total Shared memory: 1608M
 ShmWin: 389M
 Image: 882M
 LTrace: 148M
 AIPC: 27M
 SLD: 3M
 SubDB: 1016K
 CERRNO: 144K
 GSP-CBP: 62M
 EEM: 0
 XOS: 4M
 CHKPT: 2M
 CDM: 3M
 XIPC: 1M
 DLL: 64K
 SysLog: 0
 Miscellaneous: 81M

LTrace usage details:
 Used: 148M, Max: 1884M
 Current: default(dynamic)
 Configured: dynamic with scale-factor: 16 (changes take effect after reload)

Configuration Changes

After configuration, here are the changes that occur in order to reduce the ltrace by the scale factor of 16 on the Slot 1 Line Card.

RP/0/RSP1/CPU0:#config t
RP/0/RSP1/CPU0:(config)#ltrace mode dynamic scale-factor 16 location 0/1/cpu0
RP/0/RSP1/CPU0:(config)#commit
RP/0/RSP1/CPU0:(config)#

After Reload

Ltrace use after LC reload:

RP/0/RSP1/CPU0:#show run | i ltrace
ltrace mode dynamic scale-factor 16 location 0/1/CPU0

RP/0/RSP1/CPU0:#show shmem summary location 0/1/CPU0 
Total Shared memory: 1484M
 ShmWin: 383M
 Image: 882M
 LTrace: 31M
 AIPC: 27M
 SLD: 3M
 SubDB: 1016K
 CERRNO: 144K
 GSP-CBP: 62M
 EEM: 0
 XOS: 4M
 CHKPT: 2M
 CDM: 3M
 XIPC: 1M
 DLL: 64K
 SysLog: 0
 Miscellaneous: 80M

LTrace usage details:
 Used: 31M, Max: 141M
 Current: dynamic with scale-factor: 16
 Configured: dynamic with scale-factor: 16 
RP/0/RSP1/CPU0:#

Note: This configuration change reduced maximum ltrace from a previous of 1884 M to 141 M and previous use of 148 M to 31 M. The memory gained from ltrace reduction can be available for Cisco IOS XR Router operations and processes.

Comparison of Different Scale Factors

Default ltrace

LTrace usage details:

Used: 563M, Max: 3612M

Current: default(dynamic)

Configured: default(dynamic)

On the Scale Factor of 8

LTrace usage details:

Used: 76M, Max: 436M

Current: dynamic with scale-factor: 8

Configured: dynamic with scale-factor: 8

On the Scale Factor of 16

Used: 31M, Max: 141M

Current: dynamic with scale-factor: 16

Configured: dynamic with scale-factor: 16

CLI Differences in 32 and 64-bit on ASR9K

32-bit Command

RP/0/RSP1/CPU0:#show shmem summary location 0/2/CPU0

64-bit Command

RP/0/RSP1/CPU0:# show shared-memory location 0/RSP0/CPU0>>>>>> This cli is not supported on releases prior to 712/721

Shell Command

# show_memory -m

How to Avoid Chassis Reload if ltrace Reduction is Needed on RPs Only

• Commit config changes for both route processors (RPs) only.
• Standby Route Switch Processor (RSP) reload.
• Please check show redundancy to make sure RSPs are both standby-ready and NSR-ready.
• RP FO, which results in previous active RSP reload as part of RP FO.

Questions and Answers

1. What are ltraces and their purpose in Router operation?

Ltraces are used for debugging. Almost all XR processes use them to log all kinds of events during their operation. These ltraces help in debugging unexpected operations of the router.

2. How is ltrace reduction calculated?

If the value with the default scale factor was 1825 MB. With a scale factor of 8, it becomes 1825/8 = 228.125 ~= 248MB.

3. How is the maximum ltrace value calculated?

Maximum value indicates the maximum memory ltraces can consume. It is based on a number of processes and their cumulative ltrace memory requested from the process responsible for ltrace management.

By default, ltrace does not provide clients with the memory requested in one chunk. It is done dynamically based on their usage (that’s where static vs dynamic configuration comes into the picture.)

‘Used’ on the other hand, provides the current shared memory that is currently used by all clients of ltrace.

RP/0/RSP0/CPU0:#show shmem summary location 0/0/CPU0

Total Shared memory: 1395M

ShmWin: 630M

Image: 407M

LTrace: 173M

AIPC: 28M

SLD: 3M

SubDB: 1016K

CERRNO: 144K

GSP-CBP: 58M

EEM: 0

XOS: 4M

CHKPT: 2M

CDM: 3M

XIPC: 1M

DLL: 64K

SysLog: 0

Miscellaneous: 80M




LTrace usage details:

Used: 173M, Max: 1825M

Current: default(dynamic)

Configured: default(dynamic)

4. What is the trade-off of ltraces reduction by configuration change?

By configuration commit change with a scale factor different from the default, more shared memory is available for the Router with a trade-off on debuggability as the buffer of ltrace storage is reduced.

5. What are the recommended values?  

This really depends on the need for ltrace reduction. However, it is recommended to reduce ltrace usage on RSPs with less physical memory installed. For example, on A9K-RSP440-TR with 6 GBs of physical memory, reduction of ltraces with a scale factor of 8 or 16 results in significant memory available for system use and thus help in outage mitigation. 

6. How to find ltrace consumer components?  

Attach to RP or Line Card LC via run and execute shell CLI of du -sk /dev/shmem/ltrace/* | sort -n.