Modular QoS Configuration Guide for Cisco 8000 Series Routers, Cisco IOS XR Release

PDF

Modular QoS Configuration Guide for Cisco 8000 Series Routers, Cisco IOS XR Release

High Bandwidth Memory congestion detection

Updated: June 9, 2026

Want to summarize with AI?

Log in

Get detailed insights into congestion on the High Bandwidth Memory (HBM), such as the devices on which congestion has occurred, the timestamps, and when the device returned to its normal state.

High Bandwidth Memory (HBM) congestion detection is a Priority Flow Control (PFC) congestion monitoring functionality that

  • helps monitor buffer occupancy across the Shared Memory System (SMS) and High-Bandwidth Memory (HBM)

  • detects when Virtual Output Queue (VOQ) spillover into HBM indicates congestion, and

  • records time-stamped congestion events and history for post-event analysis and troubleshooting.

Table 1. Feature History Table

Feature Name

Release Information

Feature Description

Detect High Bandwidth Memory Congestion

Release 25.4.1

Introduced in this release on: Fixed Systems (8010 [ASIC: A100])(select variants only*)

*This feature is supported on:

  • 8011-32Y8L2H2FH

  • 8011-12G12X4Y-A/D

Detect High Bandwidth Memory Congestion

Release 25.1.1

Introduced in this release on: Fixed Systems (8700 [ASIC: K100], 8010 [ASIC: A100])(select variants only*)

This feature is supported on:

  • 8712-MOD-M

  • 8011-4G24Y4H-I

Detect High Bandwidth Memory Congestion

Release 24.4.1

Introduced in this release on: Fixed Systems (8200 [ASIC: P100], 8700 [ASIC: P100])(select variants only*); Modular Systems (8800 [LC ASIC: P100])(select variants only*)

*This feature is supported on:

  • 8212-48FH-M

  • 8711-32FH-M

  • 88-LC1-36EH

  • 88-LC1-12TH24FH-E

  • 88-LC1-52Y8H-EM

Detect High Bandwidth Memory Congestion

Release 7.5.3

We provide detailed insights into congestion on the High Bandwidth Memory (HBM), such as the devices on which congestion has occurred, the time stamps, and when the device returned to its normal state. With such details, you can investigate the cause of congestion and identify the source ports causing congestion for future preventive actions.

You must configure PFC in the buffer-extended mode for this option.

The feature introduces the following to enable the option to detect HBM congestion:

It also introduces the following to view the congestion and memory usage details:

How HBM congestion detection works

This process explains how queue congestion moves from the Shared Memory System (SMS) to the High-Bandwidth Memory (HBM) on your router. It also describes how the HBM congestion detection feature captures buffer utilization details for post-congestion analysis.

Summary

The key components involved in High Bandwidth Memory congestion detection are:

  • Shared Memory System (SMS): the on-chip buffer where queues are normally stored and monitored for occupancy thresholds.

  • High Bandwidth Memory (HBM): the external, off-chip buffer that temporarily stores excess packets evicted from SMS during congestion.

  • Virtual Output Queues (VOQs): logical queues that carry per-class traffic from SMS to HBM.

  • Detection logic within the NPU: tracks queue depth, HBM use, and packet age to identify congestion.

Workflow

These stages describe how HBM congestion detection works.

  1. Normal queue operation: Under normal conditions, packets egressing from an interface are enqueued into the Shared Memory System (SMS), the router's on-chip buffer memory.
  2. Onset of congestion: When congestion begins, the SMS continues to buffer packets until queue occupancy exceeds per queue usage criteria. These criteria are based on two parameters.
    • Buffer space threshold per VOQ, and
    • Packet age, measured in milliseconds.
  3. Eviction to HBM: Once the usage threshold is crossed, the router evicts packets from the SMS to the HBM.
  4. Accumulation of HBM load: A sustained HBM load marks the onset of HBM congestion conditions.
  5. HBM congestion metrics: By configuring the command hw-module profile npu memory buffer-extended bandwidth-congestion-detection enable, you can view information such as devices on which congestion has occurred and the timestamps, and the current as well as highest buffer memory usage watermark since the last reading.

Result

Use this information for post-event analysis and reporting.

Best practices for HBM congestion detection

Use these practices to ensure accurate monitoring when enabling High Bandwidth Memory (HBM) congestion detection on your router.

  • Priority Flow Control mode: You must configure Priority Flow Control (PFC) in buffer-extended mode to enable this functionality.

  • Line card reload: No line card reload is required after running the hw-module profile npu memory buffer-extended bandwidth-congestion-detection enable command.

  • Supported hardware: This functionality is supported on:

    • Cisco Silicon One Q200-based routers and line cards

    • 8201-32FH routers

    • 88-LC0-48TH-MO, 88-LC0-36FH, and 88-LC0-36FH-M line cards

Configure High Bandwidth Memory congestion detection

Capture congestion events and memory usage data for High Bandwidth Memory (HBM) and Shared Memory System (SMS).

Before you begin

Enable Priority Flow Control (PFC) buffer-extended mode.

Follow these steps to enable HBM congestion detection and view the details.

Procedure

1.

Enable congestion detection.

Example:

Router#configure
Router(config)#hw-module profile npu buffer-extended location 0/6/CPU0
bandwidth-congestion-detection enable
Router(config)#commit
Router(config)#exit

Line card reload is not required.
2.

View the buffer usage and HBM congestion details.

If ...

Then...

You want to view buffer details

Run

  • show controllers npu packet-memory usage instance all location all

  • show controllers npu packet-memory usage verbose instance all location all

Note

  • These commands provide data for the current use and the highest watermark reached since the last reading for both SMS and HBM.

  • The refresh interval for the information is 30 seconds.

You want to view HBM bandwidth congestion

Run

  • show controllers npu packet-memory congestion instance all location all

  • show controllers npu packet-memory congestion detail instance all location all

  • show controllers npu packet-memory congestion verbose instance all location all

Note

  • These commands provide data for when the HBM congestion occurred or is about to happen.

  • The output maintains a history of the last 120 events regardless of the elapsed time.

  • The refresh interval for new events to be added is 30 seconds.
  • View the buffer details.
    Router#show controllers npu packet-memory usage instance all location all
HW memory Information For Location: 0/6/CPU0
 
-------------------------------------------------------------------------------------
Timestamp(msec)                  | Device | Buff-int | Buff-int | Buff-ext | Buff-ext
                                            Usage      Max WM   | Usage    | Max WM
-------------------------------------------------------------------------------------
Wed 2022-09-21 01:54:11.154 UTC       0          7          8          0          0
Wed 2022-09-21 01:54:12.154 UTC       0          7          8          0          0
Wed 2022-09-21 01:54:24.023 UTC       1         22         22          0          0
Wed 2022-09-21 01:54:34.088 UTC       2         11         12          0          0
Wed 2022-09-21 01:54:35.088 UTC       2         11         12          0          0
Wed 2022-09-21 01:54:36.088 UTC       2         11         12          0          0
Wed 2022-09-21 01:54:37.088 UTC       2         11         12          0          0
Wed 2022-09-21 01:54:38.089 UTC       2         11         12          0          0
Wed 2022-09-21 01:54:39.089 UTC       2         11         12          0          0
Wed 2022-09-21 01:54:40.089 UTC       2         11         12          0          0

    This example displays:

    • The timestamp at which data is sampled.

    • The network processor name (Device ).

    • The packet memory usage for that timestamp for SMS (Buff-int Usage in units of buffers) and HBM (Buff-ext Usage in units of 8 KB blocks).

    • The highest maximum watermark reached for SMS (Buff-int Max WM ) and HBM (Buff-ext Max WM ) since the last reading.

  • View the timestamp in milliseconds and the buffer details.
    Router#show controllers npu packet-memory usage verbose instance all location all 

HW memory Information For Location: 0/RP0/CPU0

* Option 'verbose' formatted data is for internal consumption.
--------------------------------------------------------------------
Timestamp(msec) | Device | Buff-int | Buff-int | Buff-ext | Buff-ext
                           Usage      Max WM   | Usage    | Max WM
--------------------------------------------------------------------
   1663958881006       0       2455       2676        637        640
   1663958882007       0       2461       2703        635        640
   1663958883007       0       2364       2690        635        640
   1663958884007       0      71603      75325       3183      18336
   1663958885008       0       2458       2852       1275       1279
   1663958886008       0       2484       2827       1275       1279
  • Check if HBM congestion has occurred and view the timestamp of the congestion state.
    Router#show controllers npu packet-memory congestion instance all location all 
HW memory Information For Location: 0/6/CPU0
 
----------------------------------------------------------------
Timestamp(msec)                   |      Buff-ext       | Device 
                                  |      Event Type     |        
----------------------------------------------------------------
Wed 2022-09-21 02:14:41.709 UTC            Congest         1
Wed 2022-09-21 02:14:41.959 UTC            Congest         1
Wed 2022-09-21 02:14:42.960 UTC            Congest         1
Wed 2022-09-21 02:14:43.960 UTC            Congest         1
Wed 2022-09-21 02:14:45.210 UTC            Congest         1
Wed 2022-09-21 02:14:45.710 UTC            Congest         1
Wed 2022-09-21 02:14:47.711 UTC            Normal          1

    The system displays the last 120 events and adds new events every 30 seconds. To view updated data, run the command again.

    After 120 events, the system replaces the oldest event with the newest. You cannot remove events from the list.

  • View additional details about HBM congestion.
    Router#show controllers npu packet-memory congestion detail instance all location all
Fri Sep 23 18:49:50.640 UTC
HW memory Information For Location: 0/RP0/CPU0

* Option 'detail' formatted data is for internal consumption.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
Timestamp(msec)                  |      Buff-ext         | Device | Slice |    VOQ |VOQ-buff| Evicted-buff | Buff-int | Buff-int | Buff-int | Buff-ext | Buff-ext
                                 |      Event Type       |                         |int-WM  | int-WM       | UC-WM    | Usage    | Max WM   | Usage    | Max WM
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
Fri 2022-09-23 18:42:30.349 UTC                Congest        0        5      534    16011        63969        65451      70410      70410       34405    34405
Fri 2022-09-23 18:42:31.101 UTC                 Normal        0        5      534        0            0          900       2440       2440           0        0
Fri 2022-09-23 18:42:37.354 UTC                Congest        0        5      534    16011        63984        65493      70573      70573       34408    34408
Fri 2022-09-23 18:42:38.354 UTC                 Normal        0        5      534        0            0          915       2455       2455           0        0
Fri 2022-09-23 18:42:44.606 UTC                Congest        0        5      534    16011        64002        65520      70081      70081       34532    34532
…	…	…

    This example displays:

    • The network processor name (Device ) and the slice number (Slice ) for that device. Every network processor has a fixed number of slices, and each slice, in turn, has a set number of ports.

    • Single VOQ buffer and aggregated SMS VOQ buffers.

    • The packet memory usage for that timestamp for SMS (Buff-int Usage in units of buffers) and HBM (Buff-ext Usage in units of 8 KB blocks).

    • The highest maximum watermark reached for SMS (Buff-int Max WM ) and HBM (Buff-ext Max WM ) since the last reading.

  • View additional HBM congestion details.
    Router#show controllers npu packet-memory congestion verbose instance all location all
HW memory Information For Location: 0/RP0/CPU0
* Option 'verbose' formatted data is for internal consumption.
-------------------------------------------------------------------------------------------------------------------------------------
Timestamp(msec) |      Event | Device | Slice |    VOQ |VOQ-buff| Evicted-buff | Buff-int | Buff-int | Buff-int | Buff-ext | Buff-ext
                |      Type  |                         |int-WM  | int-WM       | UC-WM    | Usage    | Max WM   | Usage    | Max WM
-------------------------------------------------------------------------------------------------------------------------------------
1663958550349          0         0        5      534     16011        63969        65451      70410      70410      34405     34405
1663958551101          1         0        5      534         0            0          900       2440       2440          0         0
1663958557354          0         0        5      534     16011        63984        65493      70573      70573      34408     34408
1663958558354          1         0        5      534         0            0          915       2455       2455          0         0
1663958564606          0         0        5      534     16011        64002        65520      70081      70081      34532     34532
1663958565356          1         0        5      534         0            0          915       2417       2417          0         0
    This example also displays:

    • Event type, where 0 is single VOQ-based congestion and 1 is single VOQ-based congestion backoff (VOQ-buff int-WM ), 2 is congestion in aggregated SMS buffers for VOQ and 3 is congestion backoff in aggregated SMS buffers for VOQ (Evicted-buff int-WM ).

    • The buffer internal for unicast (Buff-int UC-WM ), which is for information only.

Check available Shared Memory System and High Bandwidth Memory buffers

By enabling instantaneous display of available or free buffers for Shared Memory System (SMS) and High Bandwidth Memory (HBM), you can analyze the congestion affecting buffer occupancy accurately, especially during rapid traffic fluctuations.

Before you begin

To detect HBM and SMS congestion, use the hw-module profile npu memory buffer-extended bandwidth-congestion-detection enable command.

Procedure

1.

Run the command to view buffer usage statistics.

Example:

Router#show controller npu packet-memory usage instance all location all 
HW memory Information For Location: 0/6/CPU0
 
---------------------------------------------------------------------------------------------------------------------
Timestamp(msec)                  | Device | Buff-int | Buff-int | Buff-ext | Buff-ext | Buff-int-free | Buff-ext-free
                                            Usage      Max WM   | Usage    | Max WM   | Min WM        | Min WM
---------------------------------------------------------------------------------------------------------------------
Wed 2023-08-30 23:47:40.918 UTC       0       1518       6668      17154      17656     293394          982846
Wed 2023-08-30 23:47:41.918 UTC       0       1227       5631      16010      16427     293685          983990
Wed 2023-08-30 23:47:42.919 UTC       0       1398       8295      15041      15734     293514          984959
Wed 2023-08-30 23:47:43.919 UTC       0       1765       8892      14744      15678     293147          985256
Wed 2023-08-30 23:47:41.011 UTC       1      10380      12419      37532      38165     284532          962468
Wed 2023-08-30 23:47:42.011 UTC       1      10463      11977      37315      38326     284449          962685
Wed 2023-08-30 23:47:43.013 UTC       1       9145      12604      37714      38242     285767          962286
Wed 2023-08-30 23:47:44.013 UTC       1      10996      13272      37429      38051     283916          962571
2.

Review the output. Focus on:

  • Buff-int-free Min WM (available SMS buffer)

  • Buff-ext-free Min WM (available HBM buffer)
The current availability of SMS and HBM buffers is displayed, allowing you to analyze congestion and buffer occupancy in real-time.

Available buffer in Shared Memory System and High Bandwidth Memory

This functionality provides instantaneous visibility of available, or free, buffer space in both Shared Memory System (SMS) and High Bandwidth Memory (HBM), complementing existing watermark data.

These counters for available buffers are accounted for in this way:

  • Available SMS buffer (Buff-int-free Min WM ) at a given instant= Maximum SMS buffer - highest maximum watermark reached for SMS (Buff-int Max WM )

  • Available HBM buffer (Buff-ext-free Min WM ) at a given instant = Maximum HBM buffer - highest maximum watermark reached for HBM (Buff-ext Max WM )

Table 2. Feature History Table

Feature Name

Release Information

Feature Description

Available Shared Memory System and High Bandwidth Memory Buffers

Release 25.4.1

Introduced in this release on: Fixed Systems (8010 [ASIC: A100])(select variants only*)

*This feature is supported on:

  • 8011-32Y8L2H2FH

  • 8011-12G12X4Y-A/D

Available Shared Memory System and High Bandwidth Memory Buffers

Release 25.1.1

Introduced in this release on: Fixed Systems (8700 [ASIC: K100], 8010 [ASIC: A100])(select variants only*)

This feature is supported on:

  • 8712-MOD-M

  • 8011-4G24Y4H-I

Available Shared Memory System and High Bandwidth Memory Buffers

Release 24.4.1

Introduced in this release on: Fixed Systems (8200 [ASIC: P100], 8700 [ASIC: P100])(select variants only*); Modular Systems (8800 [LC ASIC: P100])(select variants only*)

*This feature is supported on:

  • 8212-48FH-M

  • 8711-32FH-M

  • 88-LC1-36EH

  • 88-LC1-12TH24FH-E

  • 88-LC1-52Y8H-EM

Available Shared Memory System and High Bandwidth Memory Buffers

Release 24.1.1

You can now view buffer availability for Shared Memory System (SMS) and High Bandwidth Memory (HBM) with higher accuracy without any lag between the minimum and maximum watermark readings, especially when the packet buffers are used and released rapidly. This is possible because we’ve enabled the instantaneous display of available or free SMS and HBM.

Previously, you could view details only for the highest watermark readings for SMS and HBM.

You must configure PFC in the buffer-extended mode for this option, and this functionality is available only for Cisco Silicon One Q200-based routers and line cards.

This functionality modifies the following: