Guest

Cisco Interface Processors

Understanding Buffer Misses and Failures

Cisco - Understanding Buffer Misses and Failures

Document ID: 14620

Updated: Sep 09, 2005

   Print

Introduction

This document discusses buffer misses and failures on the Routing Processor (RP).

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Buffer Misses and Failures

The RP divides its processor memory into pools. Each pool contains a number of memory blocks of equal size. These memory blocks are called buffers.

Buffer Pools

There are six buffer pools:

  • Small—104 bytes buffers

  • Middle—600 byte buffers

  • Big—1524 byte buffers

  • VeryBig—4520 byte buffers

  • Large—5024 byte buffers

  • Huge—18024 byte buffers

For example, if an interface processor needs to pass a 20 byte packet to the RP, it “asks” for a Small buffer. If an interface processor needs to pass a 500 byte packet to the RP, it asks for a Middle buffer, and so forth.

Note: The interface processor must ask for a buffer of a certain size.

When the interface processor asks for a buffer, this occurs:

  • If a free buffer exists within the requested pool, the buffer is granted. Otherwise, the request generates a “miss” and the buffer algorithm tries to “create” more buffers for that pool .

  • When IOS fails to get a Small buffer, it does not drop the packet. It increments the failed counter and falls through to the next level buffer, which is the Middle buffer and requests a buffer there. If it fails to get a Middle buffer, it requests the next level buffer, which is a Big buffer. This process continues until it hits the Huge buffer pool. If it fails to get a Huge buffer, then it drops the packet.

  • When you use the IBM feature set, a miss almost always generates a failure.

  • Although the IBM features may be process-switched, the code to get a buffer to pass a packet from an interface to the RP executes at interrupt level.

  • Buffers can not be created at interrupt level; consequently, a miss queues its request for more buffers to the RP.

  • Because an additional buffer can not be created on the spot, the buffer request fails, and the packet is dropped.

Buffer failures are one of the most common reasons for packet drops. When packet drops occur because of buffer failure, this occurs:

  • After a buffer failure, the RP has an outstanding request to create more buffers of the appropriate size for the particular pool.

  • While the RP is servicing the create buffers request, there may be additional failures in the pool.

  • The RP may even fail to create more buffers, because of memory constraints in the system when the extra buffers are required.

  • Essentially, the create buffers operation could take several microseconds, in which packets are continually dropped because of the buffer shortage.

  • In addition, if buffers are used as quickly as they are created, the RP could be forced to spend more time on buffer creation than on packet processing.

  • This may cause the RP to begin to drop packets so quickly that performance degrades and sessions are lost.

Fortunately, as this document discusses, buffer failure problems are not difficult to identify and resolve. This show buffers command output shows the current state of the router’s buffer pools:

dspu-7k#show buffers

Buffer elements:
     500 in free list (500 max allowed)
     2370 hits, 0 misses, 0 created

Public buffer pools:
Small buffers, 104 bytes (total 16, permanent 10):
     11 in free list (0 min, 10 max allowed)
     1770 hits, 33 misses, 22 trims, 28 created
     9 failures (0 no memory)
Middle buffers, 600 bytes (total 90, permanent 90):
     89 in free list (10 min, 200 max allowed)
     590 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
Big buffers, 1524 bytes (total 90, permanent 90):
     90 in free list (5 min, 300 max allowed)
     126 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
     10 in free list (0 min, 300 max allowed)
     50 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
Large buffers, 5024 bytes (total 10, permanent 10):
     10 in free list (0 min, 30 max allowed)
     0 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
Huge buffers, 18024 bytes (total 2, permanent 0):
     0 in free list (0 min, 13 max allowed)
     2 hits, 2 misses, 0 trims, 2 created
     0 failures (0 no memory)

In the show buffers output:

  • Total identifies the total number of buffers in the pool, which include used and unused buffers.

  • Permanent identifies the permanent number of allocated buffers in the pool. These buffers are always in the pool and can not be trimmed away.

  • In free list identifies the number of buffers currently in the pool that are available for use.

  • Min identifies the minimum number of buffers that the RP should attempt to keep in the free list:

    • The min parameter is used to anticipate demand for buffers from the pool at any given time.

    • If the number of buffers in free list falls below the min value, the RP attempts to create more buffers for that pool.

  • Max-allowed identifies the maximum number of buffers that are allowed in the free list:

    • The max-allowed parameter prevents a pool from monopolizing buffers that it does not need anymore. It also frees this memory back to the system for further use.

    • If the number of buffers in the free list is greater than the max-allowed value, the RP should attempt to trim buffers from the pool.

  • Hits identifies the number of buffers that have been requested from the pool. The hits counter provides a mechanism to determine which pool must meet the highest demand for buffers.

  • Misses identifies the number of times that a buffer has been requested and the RP detected in which pool additional buffers were required. In other words, the number of buffers in the free list has dropped below min level. The misses counter represents the number of times the RP has been forced to create additional buffers.

  • Trims identifies the number of buffers that the RP has trimmed from the pool, when the number of buffers in the free list exceeded the number of max-allowed buffers.

  • Created identifies the number of buffers that have been created in the pool. The RP creates buffers in these situations:

    • When demand for buffers has increased until the number of buffers in the free list is less than the min buffers.

    • A miss occurs because there are no buffers in the free list.

    • Both of the previous situations.

  • Failures identifies when IOS fails to get a Small buffer, it does not drop the packet. It increments the failed counter and falls through to the next level buffer, which is the Middle buffer and requests a buffer there. If it fails to get a middle buffer, it requests the next level buffer, which is a Big buffer. This process continues until it hits the Huge buffer pool. If it fails to get a Huge buffer, then it drops the packet.

  • No memory identifies the number of failures caused by insufficient memory to create additional buffers.

You can can examine the characteristics of each pool, to determine which pools (if any) are encountering problems. The parameters for a pool can be tuned to allow the router to be better prepared to handle the load, if the pool seems to exhibit these characteristics:

  • The number of misses and creates increment at a high rate (as a percentage of hits).

  • There is a consistently low number of buffers in the free list.

  • The number of failures or no memory increment.

buffers Configuration Command

With the buffers configuration command, you can tune these parameters for each buffer pool:

  • initial—Temporary buffers that are allocated at system reload.

  • max-free—Maximum number of free buffers.

  • min-free—Minimum number of free buffers.

  • permanent—Number of permanent buffers.

Initial Buffers

Tune initial buffers to accommodate the burst of session-establishment traffic after router reload.

buffers small initial 250

These buffers are eventually “trimmed” and returned to the system.

The initial buffers are designed to handle session establishment, which is always process-switched.

During session establishment, the fastswitching cache (used by other route protocols) is populated; process-switched buffers are no longer required and may be returned to the system.

To tune initial buffers may not be the correct solution for the IBM feature set, because almost all packets (after session establishment) are process-switched and require the additional buffering anyway.

Note: For the IBM process-switched features, you should tune permanent buffers rather than tune the temporary initial buffers.

Max-Free Buffers

Tune max-free buffers so that the value is equal to or greater than the permanent buffers. If all permanent buffers are in the free list, then the RP should not try to trim permanent buffers. Max-free can be used to ensure that unused buffers that are created during irregular bursts are returned to the system memory.

buffers small max-free 175

buffers small permanent 125

Min-Free Buffers

Tune min-free buffers so that the value represents the estimated minimum number of buffers required at any time. Min-free can be used to anticipate buffer shortage conditions and to ensure that a minimum number of buffers are always available.

buffers small min-free 50

Permanent Buffers

Tune permanent buffers so that the value represents the estimated number of buffers required for normal processing.

buffers small permanent 125

Permanent buffers are used to accommodate the normal buffer requirements (including frequent bursts) of the router. Determination of the normal buffer requirements is an interactive process, where the show buffer output should show the total buffers used in a pool at a given time. Permanent buffers should be tuned with regard to the consistent "total" buffers required. When you tune permanent buffers, you should focus on the reduction of creates and the elimination of misses and failures.

Additional show Commands

There are two other show commands that you can use to identify problems with buffer allocation:

  • show interfaces interface-identifier

  • show source-bridge

This show interfaces interface-identifier sample command output includes a counter for no buffer:

dspu-7k#show interfaces channel 4/2

Channel4/2 is up, line protocol is up
  Hardware is cxBus IBM Channel
  MTU 4472 bytes, BW 98304 Kbit, DLY 100 usec, rely 255/255, load 1/255
  Encapsulation CHANNEL, loopback not set, keepalive not set
  Virtual interface
  Last input 0:00:04, output 0:00:04, output hang never
  Last clearing of "show interface" counters never
  Output queue 0/40, 0 drops; input queue 0/75, 8 drops
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     646 packets input, 27760 bytes, 8 no buffer
     Received 0 broadcasts, 0 runts, 0 giants
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     328 packets output, 16959 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets, 0 restarts
     0 output buffer failures, 0 output buffers swapped out

In the show interfaces interface-identifier command output:

  • The no buffer counter increments when the interface fails to obtain a buffer for an inbound packet.

  • Both the no buffer and drops (input queue) counters increment when the interface fails to obtain a buffer for an inbound packet.

  • A no buffer counter that increments in the show interfaces output correlates to the misses counter that increments in the show buffers output. The appropriate buffer pool may be tuned.

This show source-bridge sample command output includes an interface counter for throttles, when source-route bridging (SRB) is configured for the interface:

dspu-7k#show source-bridge

Local Interfaces:                        receive            transmit
       srn bn  trn r p s n  max hops     cnt:bytes          cnt:bytes      drops
Ch4/2  666  1   99 *   f    7  7  7      652:26020            6:266           0

Global RSRB Parameters:
 TCP Queue Length maximum: 100

Ring Group 99:
  This TCP peer: 150.10.20.2
   Maximum output TCP queue length, per peer: 100
  Peers:                 state     bg lv  pkts_rx  pkts_tx  expl_gn   drops TCP
   TCP 150.10.20.1       open         *3      261      266        0       0   0
   TCP 150.10.20.2       -            *3        0        0        0       0   0
  Rings:
   bn: 1  rn: 888  locvrt ma: 4000.7000.fff1 Buff Ring888          fwd: 0
   bn: 1  RN: 666  local  ma: 4000.0c48.2e80 Channel4/2            fwd: 261
   bn: 1  RN: 88   remote ma: 4000.4000.fff1 TCP 150.10.20.1       fwd: 322
   bn: 1  RN: 250  remote ma: 4000.300f.7c09 TCP 150.10.20.1       fwd: 0

Explorers: ------- input -------             ------- output -------
         spanning  all-rings     total      spanning  all-rings     total
Ch4/2           0          0         0             0          1         1

  Local: fastswitched 0         flushed 0         max Bps 256000

         rings      inputs         bursts         throttles output drops
         Ch4/2           0              0                 8            0

In the show source-bridge command output:

  • The throttles counter increments when the interface fails to obtain a buffer for an inbound packet.

  • The throttles counter that increments in the show interfaces command output correlates to a misses counter that increments in the show buffers command output. The appropriate buffer pool may be tuned.

Related Information

Updated: Sep 09, 2005
Document ID: 14620