Guest

Cisco MDS 9500 Series Multilayer Directors

Generation 4 (gen4) Linecard Slow Drain Counters and Commands Troubleshooting (MDS 9500 Series Multilayer Directors)

Document ID: 116098

Updated: Jul 17, 2013

Contributed by Ed Mazurek, Cisco TAC Engineer.

   Print

Contents

Introduction

This document describes the commands and counters that increment on Cisco MDS 9500 Series Multilayer Directors generation 4 (gen4) linecards with the Thunderbird ASIC, models DS-X9232-256K9 and DS-X9248-256K9, when a device is behaving as a slow drain device. If you understand which counters increment and where they increment, you can properly identify and resolve these types of problems.

All data traffic between end devices in a storage area network (SAN) fabric is carried by Fiber Channel Class 3 and, in some cases, Class 2 services that use link-level, per-hop-based, and buffer-to-buffer flow control. These classes of service do not support end-to-end flow control. When there are slow devices attached to the fabric, the end devices do not accept the frames at the configured or negotiated rate. The slow devices, referred to as slow drain devices, lead to Inter-Switch Link Protocol ISL) credit shortage in the traffic destined for these devices and congest the links. The credit shortage affects the unrelated flows in the fabric that use the same ISL link even though destination devices do not experience slow drain.

Three tests were run:

  1. Slow Port Emulation - R_RDY Delay 10000us (10 ms)
  2. Slow Port Emulation - R_RDY Delay 1500000us (1.5 sec)
  3. Port-Monitor - Slow Port Emulation - R_RDY Delay 1500000us (1.5 sec)

Topology

All ports are 4 Gbps.

                       172.18.121.64     fc13/1         fc4/9 172.18.121.16
Agilent 103/3--fc13/13 rtp-san-23-02-9513 =======PC 237======= rtp-san-23-01-9513 fc4/13--Agilent 103/2
                       NX-OS 5.2(2d)                           NX-OS 5.2(8) rtp-san-23-02-9513 9513 sup-2a/fab3 NX-OS 5.2(2d)
rtp-san-23-02-9513# show mod
Mod  Ports  Module-Type                         Model              Status
---  -----  ----------------------------------- ------------------ ----------
1    8      IP Storage Services Module          DS-X9308-SMIP      powered-dn
2    8      10 Gbps FCoE Module                 DS-X9708-K9        powered-dn
3    8      10 Gbps FCoE Module                 DS-X9708-K9        powered-dn
4    22     4x1GE IPS, 18x1/2/4Gbps FC Module   DS-X9304-18K9      ok
5    24     1/2/4/8 Gbps FC Module              DS-X9224-96K9      ok
6    8      10 Gbps FCoE Module                 DS-X9708-K9        powered-dn
7    0      Supervisor/Fabric-2a                DS-X9530-SF2AK9    active *
8    0      Supervisor/Fabric-2a                DS-X9530-SF2AK9    ha-standby
9    8      IP Storage Services Module          DS-X9308-SMIP      powered-dn
10   22     4x1GE IPS, 18x1/2/4Gbps FC Module   DS-X9304-18K9      ok
11   22     4x1GE IPS, 18x1/2/4Gbps FC Module   DS-X9304-18K9      ok
12   24     1/2/4/8 Gbps FC Module              DS-X9224-96K9      ok
13   48     1/2/4/8/10 Gbps Advanced FC Module  DS-X9248-256K9     ok

rtp-san-23-01-9513 9513 sup-2a/fab3 NX-OS 5.2(8)
rtp-san-23-01-9513# show mod
Mod  Ports  Module-Type                         Model              Status
---  -----  ----------------------------------- ------------------ ----------
1    4      10 Gbps FC Module                   DS-X9704           ok
2    48     1/2/4/8/10 Gbps Advanced FC Module  DS-X9248-256K9     ok
3    16     16x1GE, Storage Services Node       DS-X9316-SSNK9     ok
4    32     1/2/4/8/10 Gbps Advanced FC Module  DS-X9232-256K9     ok
7    0      Supervisor/Fabric-2a                DS-X9530-SF2AK9    ha-standby
8    0      Supervisor/Fabric-2a                DS-X9530-SF2AK9    active *

Background Information

This section describes the restrictions on commands in the Cisco NX-OS network operating system as they apply to gen4 linecards.

Note: Use the Command Lookup Tool (registered customers only) in order to obtain more information on the commands used in this section.

NX-OS Release 5.2(1)

The show interface counters and show interface aggregate-counters commands return 'receive B2B credit remaining' and 'transmit B2B credit remaining' with the TBBZ and RBBZ values swapped.

The fcmac counters are correct.

See Cisco bug ID CSCts28865, "B2B credit 0 transitions incorrect for generation 4 linecards."

NX-OS Release 5.2(2d)

These commands do not work:

  • show logging onboard flow-control timeout-drops module 13 - Syntax error.
  • show logging onboard flow-control request-timeout module 13 - Syntax error.
  • show hardware internal packet-flow dropped - Points to the show hardware internal statistics module 13 pktflow dropped command.
  • show hardware internal packet-dropped-reason module 2 - Not supported.
  • show hardware internal up-xbar 0 queued-packet-info - Invalid command.

NX-OS Release 5.2(6a)

These commands do not work:

  • show hardware internal packet-dropped-reason module 2 - Not supported.
  • show hardware internal up-xbar 0 queued-packet-info - Invalid command.

NX-OS Release 5.2(8)

The show hardware internal packet-dropped-reason module 2 command is not supported.

A new linecard command, show hardware internal que inst 0-3 memory iqm-statusmem0 & 1, is available. This command replaces the show hardware internal up-xbar 0 queued-packet-info command in generation 1-3 linecards.

Discards do not increment on the interface. See Cisco bug ID CSCud77292, "Gen 4 linecards do not increment output discards on interface statistics."

fcIfOutDiscards do not appear to be sent out.

All counters have different names than generation1-3 counters.

The fcmac counters do not contain these five software counters; they appear only in the on-board failure logging (OBFL) error-stats:

  • FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO
  • FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO
  • FCP_SW_CNTR_FORCE_TIMEOUT_ON
  • FCP_SW_CNTR_FORCE_TIMEOUT_OFF
  • FCP_SW_CNTR_CREDIT_LOSS

See Cisco bug ID CSCud93629, "Gen 4 fcmac counters don't contain several software counters."

Test 1: Slow Port Emulation - R_RDY Delay 10000us (10 ms)

Overview of Steps 1-9

This is an overview of the test procedure for slow port emulation with a R-RDY delay of 1000us (10 ms).

fc4/13 is the port connected to the slow device, and fc13/13 is the port connected to sender.

  1. Start Agilent traffic 103/3 > 103/2.
  2. Let it run for 30 seconds or so.
  3. Issue set of commands on rtp-san-23-02-9513.
  4. Wait 30 seconds.
  5. Issue set of commands on rtp-san-23-02-9513.
  6. Issue set of commands on rtp-san-23-01-9513.
  7. Wait 30 seconds.
  8. Issue set of commands on rtp-san-23-01-9513.
  9. Stop test.

MDS Values and Counters after Step 3: rtp-san-23-02-9513 fc13/13 - Port Connected to Sender

Interface Counters - fc13/13

These are the changes, if any, on the MDS counters after Step 3:

input discards - No change from previous value.
input OLS - No change from previous value.
input LRR - No change from previous value.
input NOS - No change from previous value.

output discards - No change from previous value.
output OLS - No change from previous value.
output LRR - No change from previous value.
output NOS - No change from previous value.

receive B2B credit transitions from zero - Not captured.
transmit B2B credit transitions from zero - Not captured.
receive B2B credit remaining 65530 - 65530 - No change from previous value.
transmit B2B credit remaining 128 - 128 - No change from previous value.

Supervisor Level Hardware-Detected Errors

This command was issued:

show hardware internal errors module 13

No counters applicable to port fc13/13 changed.

Supervisor Packet-Flow Dropped

A new command-line interface (CLI) command, show hardware internal statistics module 13 pktflow dropped, is available.

No counters applicable to port fc13/13 changed.

Linecard OBFL Error-Stats

This command was issued:

show logging onboard module 13 error-stats

No counters applicable to port fc13/13 changed.

Linecard OBFL Flow-Control Timeout-Drops

This command was issued:

show logging onboard flow-control timeout-drops module 13 

There is a syntax error when show logging onboard timeout-drops is parsed.

Linecard OBFL Flow-Control Request-Timeout

This command was issued:

show logging onboard flow-control request-timeout module 13 

There is a syntax error when show logging onboard request-timeout is parsed.

Creditmon

This command was issued:

show process creditmon credit-loss-events module 13

No counters applicable to port fc13/13 changed.

Linecard Statistics

This command gives this example output:

slot 13 show hardware internal statistics device fcmac all port 13

|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod:13    |
| Last cleared @ Wed Nov 28 20:36:11 2012
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
60   THB_RCM_RCP3_RBBZ_CH0                    0000000001209670   13 -   0000000001243266  +33596
1044 IP_FCMAC_CNT_STATS_DATA_RX_CLASS3_FRAMES 0000000072446822   13 -   0000000072485256  +38434

Linecard Credit-Info

This command was issued:

slot 13 show hardware internal credit-info port 13

There is no change from previous value. This is example output:

    ======== Device Credit Information - RX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|  13  |  0/0 |     Tbird Mac Driver |       0x20 |    0x26 | Shared |
+------+------+----------------------+------------+---------+--------+
    ======== Device Credit Information - TX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|  13  |  0/0 |     Tbird Mac Driver |       0x80 |     0x0 | Shared |
+------+------+----------------------+------------+---------+--------+

Linecard Queued Packets

These commands were issued:

slot 13 show hardware internal que inst 1 memory iqm-statusmem0
slot 13 show hardware internal que inst 1 memory iqm-statusmem1

The output is not usable.

MDS Values and Counters after Step 5: rtp-san-23-02-9513 port-channel 237 - Sender Side - fc13/1

Interface Counters

These are the changes, if any, on the MDS counters after Step 5:

input discards - No change from previous value.
input OLS - No change from previous value.
input LRR - No change from previous value.
input NOS - No change from previous value.

output discards - No change from previous value.
output OLS - No change from previous value.
output LRR - No change from previous value.
output NOS - No change from previous value.

receive B2B credit transitions from zero - Not captured.
transmit B2B credit transitions from zero - Not captured.
receive B2B credit remaining - 250 but no change from previous value.
transmit B2B credit remaining - 0 but no change from previous value.

Supervisor Level Hardware-Detected Errors

This command gives this example output:

show hardware internal errors module 13

|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod:13    |
| Last cleared @ Wed Nov 28 20:36:11 2012
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:0
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
119  THB_TMM_TIMEOUT_STATS_DROP               0000000000011975   1-6 -  0000000000015552   +3577
1023 THB_TMM_PORT_FRM_DROP_CNT                0000000000011991   1 -    0000000000015568   +3577
1122 THB_TMM_TOLB_TIMEOUT_DROP_CNT            0000000000011991   1 -    0000000000015568   +3577

|------------------------------------------------------------------------|
| Device:Tbird Fwd Driver         Role:L2                      Mod:13    |
| Last cleared @ Wed Nov 28 20:36:11 2012
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:0
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
493  THB_EPR3_CNT_GOOD_SF_DROP                0000000000000103   1-12 - 0000000000000127     +24
497  THB_EPR3_CNT_GOOD_PKT_DROP               0000000000000103   1-12 - 0000000000000127     +24
509  THB_EPR3_CNT_OTHER_DROP                  0000000000000103   1-12 - 0000000000000127     +24

Supervisor Packet-Flow Dropped

A new CLI command, show hardware internal statistics module 13 pktflow dropped, is available.

|------------------------------------------------------------------------|
| Device:Tbird Fwd Driver         Role:L2                      Mod:13    |
| Last cleared @ Wed Nov 28 20:36:11 2012
|------------------------------------------------------------------------|
Instance:0
ID   Name                                   Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
493  THB_EPR3_CNT_GOOD_SF_DROP                0000000000000107   1-12 - 0000000000000128     +21
497  THB_EPR3_CNT_GOOD_PKT_DROP               0000000000000107   1-12 - 0000000000000128     +21
509  THB_EPR3_CNT_OTHER_DROP                  0000000000000107   1-12 - 0000000000000128     +21
|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod:13    |
| Last cleared @ Wed Nov 28 20:36:11 2012
|------------------------------------------------------------------------|
Instance:0
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
119  THB_TMM_TIMEOUT_STATS_DROP               0000000000012560   1-6 -  0000000000016021   +3461
1023 THB_TMM_PORT_FRM_DROP_CNT                0000000000012576   1 -    0000000000016037   +3461
1122 THB_TMM_TOLB_TIMEOUT_DROP_CNT            0000000000012576   1 -    0000000000016037   +3461

Linecard OBFL Error-Stats

This command gives this example output:

show logging onboard module 13 error-stats

--------------------------------------------------------------------------------
 ERROR STATISTICS INFORMATION FOR DEVICE DEVICE: FCMAC
--------------------------------------------------------------------------------
    Interface      |                                |         |    Time Stamp  
     Range        |    Error Stat Counter Name     |  Count  |MM/DD/YY HH:MM:SS
                   |                                |         |                
--------------------------------------------------------------------------------
fc13/1             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |14924    |12/13/12 16:23:03
fc13/1             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |12576    |12/13/12 16:22:43
fc13/1             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |10182    |12/13/12 16:22:23

Linecard OBFL Flow-Control Timeout-Drops

This command was issued:

show logging onboard flow-control timeout-drops module 13

There is a syntax error when show logging onboard timeout-drops is parsed.

Linecard OBFL Flow-Control Request-Timeout

This command was issued:

show logging onboard flow-control request-timeout module 13

There is a syntax error when show logging onboard request-timeout is parsed.

Creditmon

This command was issued:

show process creditmon credit-loss-events module 13

No counters applicable to port fc13/1 changed.

Linecard Statistics

This command gives this example output:

slot 13 show hardware internal statistics device fcmac all port 13

|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod:13    |
| Last cleared @ Wed Nov 28 20:36:11 2012
|------------------------------------------------------------------------|
Instance:0
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
119  THB_TMM_TIMEOUT_STATS_DROP               0000000000012678   1-6 -  0000000000016138   +3460v
1000 THB_RPB_IN_PKT_CNT                       0000000000505935   1 -    0000000000505967     +32
1001 THB_RPB_OUT_PKT_CNT                      0000000000505935   1 -    0000000000505967     +32
1002 THB_RPB_NC_FRM_CNT                       0000000000253790   1 -    0000000000253796      +6
1003 THB_RPB_LP_FRM_CNT                       0000000000253790   1 -    0000000000253796      +6
1016 THB_TMM_PORT_TBBZ_CH0                    0000000000021060   1 -    0000000000024433   +3373
1021 THB_TMM_PORT_NP_CNT                      0000000000482819   1 -    0000000000517817  +34998
1022 THB_TMM_PORT_TOLB_FRM_SENT_CNT           0000000000253790   1 -    0000000000253796      +6
1048 IP_FCMAC_CNT_STATS_DATA_TX_CLASS3_FRAMES 0000000000230751   1 -    0000000000265723  +34972
1111 THB_TMM_TO_CNT_CLASS_3                   0000000000012694   1 -    0000000000016154   +3460
1122 THB_TMM_TOLB_TIMEOUT_DROP_CNT            0000000000012694   1 -    0000000000016154   +3460

Linecard Credit-Info

This command was issued:

slot 13 show hardware internal credit-info port 13

There is no change from previous value. This is example output:

   ======== Device Credit Information - RX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|   1  |  0/0 |     Tbird Mac Driver |       0xfa |     0x0 |   Full |
+------+------+----------------------+------------+---------+--------+
    ======== Device Credit Information - TX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|   1  |  0/0 |     Tbird Mac Driver |      0x1f4 |   0x1f4 |   Full |
+------+------+----------------------+------------+---------+--------+

Linecard Queued Packets

These commands were issued:

slot 13 show hardware internal que inst 1 memory iqm-statusmem0
slot 13 show hardware internal que inst 1 memory iqm-statusmem1

The output is not usable.

MDS Values and Counters after Step 6: rtp-san-23-01-9513 port-channel 237 - Slow Drain Side - fc4/9

Interface Counters - fc4/9

These are the changes, if any, on the MDS counters after Step 6:

input discards - No change from previous value.
input OLS - No change from previous value.
input LRR - No change from previous value.
input NOS - No change from previous value.

output discards - No change from previous value.
output OLS - No change from previous value.
output LRR - No change from previous value.
output NOS - No change from previous value.

receive B2B credit transitions from zero - 31413 - at end (no delta). If you go by THB_RCM_RCP5_RBBZ_CH0, it is 29541-26055=3486.
transmit B2B credit transitions from zero - 5 - at end (no delta). If you go by THB_TMM_PORT_TBBZ_CH0, it is 5-5=0.
receive B2B credit remaining - 0 - No change from previous value.
transmit B2B credit remaining - 250 - No change from previous value.

Supervisor Level Hardware-Detected Errors

This command was issued:

show hardware internal errors module 4

No counters applicable to port fc4/9 changed.

Supervisor Packet-Flow Dropped

This command was issued:

show hardware internal packet-flow dropped module 4

No counters applicable to port fc4/9 changed.

Linecard OBFL Error-Stats

This command was issued:

show logging onboard module 4 error-stats

No counters applicable to port fc4/9 changed.

Linecard OBFL Flow-Control Timeout-Drops

This command was issued:

show logging onboard flow-control timeout-drops module 4

No counters applicable to port fc4/9 changed.

Linecard OBFL Flow-Control Request-Timeout

This command was issued:

show logging onboard flow-control request-timeout module 4

No counters applicable to port fc4/9 changed.

Creditmon

This command was issued:

show process creditmon credit-loss-events module 4

Module: 04      Credit Loss Events: NO

Linecard Statistics

This command gives this example output:

slot 4 show hardware internal statistics device fcmac all port 9

|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod: 4    |
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
68   THB_RCM_RCP5_RBBZ_CH0                    0000000000026055   9 -    0000000000029541   +3486
1002 THB_RPB_NC_FRM_CNT                       0000000000499012   9 -    0000000000499018      +6
1003 THB_RPB_LP_FRM_CNT                       0000000000499012   9 -    0000000000499018      +6
1022 THB_TMM_PORT_TOLB_FRM_SENT_CNT           0000000000499012   9 -    0000000000499018      +6
1044 IP_FCMAC_CNT_STATS_DATA_RX_CLASS3_FRAMES 0000000000282573   9 -    0000000000318142  +35569
1097 THB_RXF_TO_LB_CNT                        0000000000499012   9 -    0000000000499018      +6

Linecard Credit-Info

This command was issued:

slot 4 show hard internal credit-info port 9

There is no change from previous value. This is example output:

    ======== Device Credit Information - RX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|   9  |  0/0 |     Tbird Mac Driver |      0x1f4 |   0x1fa |   Full |
+------+------+----------------------+------------+---------+--------+
    ======== Device Credit Information - TX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|   9  |  0/0 |     Tbird Mac Driver |       0xfa |     0x0 |   Full |
+------+------+----------------------+------------+---------+--------+

Linecard Queued Packets

These commands give this example output:

slot 4 show hardware internal que inst 1 memory iqm-statusmem0
slot 4 show hardware internal que inst 1 memory iqm-statusmem1

rtp-san-23-01-9513#slot 4 show hardware internal que inst 1 memory iqm-statusmem0
+-------------------------------------------------------------------------------
| IQM: PG0 Status Memory for Tbird Que Driver
| Inst 1; port(s) 9-16
|

Only non-zero entries are displayed. Each non-zero bit indicates a pending frame in the virtual output queuing (VOQ) for that InfiniBand (IB).

+----------+--------+--------+--------+--------+
| GI (Hex) | Prio 0 | Prio 1 | Prio 2 | Prio 3 |
+----------+--------+--------+--------+--------+
|        c | 000000 | 000000 | 000000 | 000001 |
+----------+--------+--------+--------+--------+
         ^                                   ^
         |                                   |
egress port (slow)                       ingress port

This indicates that one or more packets are queued to destination index xC, which, in this test, is port fc4/13:

rtp-san-23-01-9513# show port internal info interface fc4/13
fc4/13 - if_index: 0x0118C000, phy_port_index: 0xc
     local_index: 0xc

000001 indicates the first port in Thunderbird ASIC instance 1. For model DS-X9232-256K9, instance 1 is for ports 9-16, so that indicates fc4/9 is where the packets are ingressing going to fc4/13.

For model DS-X9248-256K9, each Thunderbird instance handles 12 ports, so instance 1 would start with port 13. Only non-zero entries are displayed.

rtp-san-23-01-9513# slot 4 show hardware internal que inst 1 memory iqm-statusmem1
+-------------------------------------------------------------------------------
| IQM: PG1 Status Memory for Tbird Que Driver
| Inst 1; port(s) 9-16
|

MDS Values and Counters after Step 8: rtp-san-23-01-9513 fc4/13 - Port Connected to the Slow Device

Interface Counters - fc4/13

These are the changes, if any, on the MDS counters after Step 8:

input discards - No change from previous value.
input OLS - No change from previous value.
input LRR - No change from previous value.
input NOS - No change from previous value.

output discards - No change from previous value.
output OLS - No change from previous value.
output LRR - No change from previous value.
output NOS - No change from previous value.

receive B2B credit transitions from zero - Not captured. If you go by THB_RCM_RCP5_RBBZ_CH0, it is 11-11=0.
transmit B2B credit transitions from zero - Not captured. If you go by THB_TMM_PORT_TBBZ_CH0, it is 43034-39908=3126.
receive B2B credit remaining 32 - No change from previous value.
transmit B2B credit remaining 0 - No change from previous value.

Supervisor Level Hardware-Detected Errors

This command gives this example output:

show hardware internal errors module 4

-------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod: 4    |
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:0
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
272  THB_TMM_TIMEOUT_STATS_DROP               0000000000407690   13-16 -0000000000440678  +32988
1823 THB_TMM_PORT_FRM_DROP_CNT                0000000000407706   13 -   0000000000440694  +32988
1922 THB_TMM_TOLB_TIMEOUT_DROP_CNT            0000000000407706   13 -   0000000000440694  +32988

Supervisor Packet-Flow Dropped

This command gives this example output:

show hardware internal packet-flow dropped module 4

|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod: 4    |
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
272  THB_TMM_TIMEOUT_STATS_DROP               0000000000411953   13-16 -0000000000444935  +32982
1823 THB_TMM_PORT_FRM_DROP_CNT                0000000000411969   13 -   0000000000444951  +32982
1922 THB_TMM_TOLB_TIMEOUT_DROP_CNT            0000000000411969   13 -   0000000000444951  +32982

show hardware internal statistics module 4 pktflow dropped

This is the new CLI command mentioned in MDS Values and Counters after Step 5, Supervisor Packet-Flow Dropped.

Linecard OBFL Error-Stats

This command gives this example output:

show logging onboard module 4 error-stats

--------------------------------------------------------------------------------
 ERROR STATISTICS INFORMATION FOR DEVICE DEVICE: FCMAC
--------------------------------------------------------------------------------
    Interface      |                                |         |    Time Stamp  
      Range        |    Error Stat Counter Name     |  Count  |MM/DD/YY HH:MM:SS
                   |                                |         |                
--------------------------------------------------------------------------------
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |445487   |12/13/12 22:23:25
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |424732   |12/13/12 22:23:05
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |403982   |12/13/12 22:22:45

Linecard OBFL Flow-Control Timeout-Drops

This command gives this example output:

show logging onboard flow-control timeout-drops module 4

--------------------------------------------------------------------------------
ERROR STATISTICS INFORMATION FOR DEVICE DEVICE: FCMAC
--------------------------------------------------------------------------------
    Interface      |                                |         |    Time Stamp  
      Range        |    Error Stat Counter Name     |  Count  |MM/DD/YY HH:MM:SS
                   |                                |         |                
--------------------------------------------------------------------------------
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |445487   |12/13/12 22:23:25
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |424732   |12/13/12 22:23:05
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |403982   |12/13/12 22:22:45

Linecard OBFL Flow-Control Request-Timeout

This command was issued:

show logging onboard flow-control request-timeout module 4

There is no output.

Creditmon

This command was issued:

show process creditmon credit-loss-events module 4

Module: 04      Credit Loss Events: NO

Linecard Statistics

This command gives this example output:

slot 4 show hardware internal statistics device fcmac all port 13

|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod: 4    |
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
272  THB_TMM_TIMEOUT_STATS_DROP               0000000000415139   13-16 -0000000000447595  +32456
1816 THB_TMM_PORT_TBBZ_CH0                    0000000000039908   13 -   0000000000043034  +3126
1828 THB_TMM_PORT_TWAIT_CNT                   0000042231489340   13 -   0000045541420902 +3309931562
1918 THB_TMM_TOLB_TO_CNT                      0000000000415155   13 -   0000000000447611  +32456
1922 THB_TMM_TOLB_TIMEOUT_DROP_CNT            0000000000415155   13 -   0000000000447611  +32456

Linecard Credit-Info

This command was issued:

slot 4 show hard internal credit-info port 13

All Tx credits are used. The device is not returning R_RDYs on a timely basis. This is example output:

   ======== Device Credit Information - RX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|  13  | 16/0 |     Tbird Mac Driver |       0x20 |     0x0 |   Full |
+------+------+----------------------+------------+---------+--------+
    ======== Device Credit Information - TX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|  13  | 16/0 |     Tbird Mac Driver |       0x80 |    0x80 |   Full |
+------+------+----------------------+------------+---------+--------+

Linecard Queued Packets

These commands were issued:

slot 4 show hardware internal que inst 1 memory iqm-statusmem0
slot 4 show hardware internal que inst 1 memory iqm-statusmem1

The output is not usable.

Refer to MDS Values and Counters after Step 6, Linecard Queued Packets.

Test 2: Slow Port Emulation - R_RDY Delay 1500000us (1.5 sec)

Overview of Steps 1-10

This is an overview of the test procedure for slow port emulation with a R-RDY delay of 1500000us (1.5 seconds).

fc4/13 is the port connected to the slow device, and fc13/13 is the port connected to sender.

  1. Start Agilent traffic 103/3 -> 103/2.
  2. Let it run for 30 seconds or so.
  3. Issue set of commands on rtp-san-23-02-9513.
  4. Wait 30 seconds.
  5. Issue set of commands on rtp-san-23-02-9513.
  6. Issue set of commands on rtp-san-23-01-9513.
  7. Wait 30 seconds.
  8. Issue set of commands on rtp-san-23-01-9513.
  9. Stop test.

MDS Values and Counters after Step 4: rtp-san-23-02-9513 fc13/13 - Port Connected to Sender

Interface Counters - fc13/13

This command was issued:

show interface fc13/13 counters

These are the changes, if any, on the MDS counters after Step 4:

input discards - No change from previous value.
input OLS - No change from previous value.
input LRR - No change from previous value.
input NOS - No change from previous value.

output discards - No change from previous value.
output OLS - No change from previous value.
output LRR - No change from previous value.
output NOS - No change from previous value.

transmit B2B credit transitions from zero - 0 - No change from previous value.
receive B2B credit transitions from zero - +24483.
receive B2B credit remaining - 65530 - No change from previous value.
transmit B2B credit remaining- 128 - No change from previous value.

Supervisor Level Hardware-Detected Errors

This command was issued:

show hardware internal errors module 13

No counters applicable to port fc13/13 changed.

Supervisor Packet-Flow Dropped

This command was issued:

show hardware internal packet-flow dropped module 13

No counters applicable to port fc13/13 changed.

Linecard OBFL Error-Stats

This command gives this example output:

show logging onboard module 13 error-stats

rtp-san-23-02-9513# show logging onboard module 13 error-stats

----------------------------
    Module: 13
----------------------------

--------------------------------------------------------------------------------
 ERROR STATISTICS INFORMATION FOR DEVICE DEVICE: FCMAC
--------------------------------------------------------------------------------
    Interface      |                                |         |    Time Stamp  
     Range        |    Error Stat Counter Name     |  Count  |MM/DD/YY HH:MM:SS
                   |                                |         |                
--------------------------------------------------------------------------------
fc13/13            |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |776      |12/17/12 08:28:38
fc13/1             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |777      |12/17/12 08:28:38
fc13/13            |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |695      |12/17/12 08:28:18
fc13/1             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |697      |12/17/12 08:28:18
fc13/13            |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |628      |12/17/12 08:27:58
fc13/1             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |27139    |12/17/12 08:27:58
fc13/1             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |630      |12/17/12 08:27:58
fc13/13            |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |541      |12/17/12 08:27:38
fc13/1             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |543      |12/17/12 08:27:38

Linecard OBFL Flow-Control Timeout-Drops

This command was issued:

show logging onboard flow-control timeout-drops module 13

There is a syntax error when show logging onboard timeout-drops is parsed.

Linecard OBFL Flow-Control Request-Timeout

This command was issued:

show logging onboard flow-control request-timeout module 13

There is a syntax error when show logging onboard request-timeout is parsed.

Creditmon

This command was issued:

show process creditmon credit-loss-events module 13

No counters applicable to module 13 changed.

Linecard Statistics

This command gives this example output:

slot 13 show hardware internal statistics device fcmac all port 13

|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod:13    |
| Last cleared @ Wed Nov 28 15:36:11 2012
------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
60   THB_RCM_RCP3_RBBZ_CH0                    0000000001448221   13 -   0000000001471293  +23072
1002 THB_RPB_NC_FRM_CNT                       0000000000320356   13 -   0000000000320362      +6
1022 THB_TMM_PORT_TOLB_FRM_SENT_CNT           0000000000320356   13 -   0000000000320362      +6
1090 THB_RHP_FRM_CNT                          0000000072752061   13 -   0000000072785537      +6
1096 THB_RXF_NP_CNT                           0000000072752061   13 -   0000000072785537      +6

Linecard Credit-Info

This command was issued:

slot 13 show hardware internal credit-info port 13

There is no change from previous value. This is example output:

rtp-san-23-02-9513# slot 13 show hardware internal credit-info port 13

    ======== Device Credit Information - RX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|  13  |  0/0 |     Tbird Mac Driver |       0x20 |    0x26 | Shared |
+------+------+----------------------+------------+---------+--------+
    ======== Device Credit Information - TX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|  13  |  0/0 |     Tbird Mac Driver |       0x80 |     0x0 | Shared |
+------+------+----------------------+------------+---------+--------+

Linecard Queued Packets

These commands were issued:

slot 13 show hardware internal que inst 1 memory iqm-statusmem0
slot 13 show hardware internal que inst 1 memory iqm-statusmem1

There is no usable output.

Port Link-Events

This command was issued:

slot 13 show port-config internal link-events

There is no change from previous value.

MDS Values and Counters after Step 6: rtp-san-23-02-9513 port-channel 237 - Sender Side - fc13/1

Interface Counters

This command was issued:

show interface fc13/1 counters

These are the changes, if any, on the MDS counters after Step 6:

input discards - No change from previous value.
input OLS - No change from previous value.
input LRR - No change from previous value.
input NOS- No change from previous value.

output discards - No change from previous value.
output OLS - No change from previous value.
output LRR- No change from previous value.
output NOS- No change from previous value.

transmit B2B credit transitions from zero - +3531.
receive B2B credit transitions from zero - No change from previous value.
receive B2B credit remaining - 250 - No change from previous value.
transmit B2B credit remaining - 0 - No change from previous value.

Supervisor Level Hardware-Detected Errors

This command gives this example output:

show hardware internal errors module 13

|------------------------------------------------------------------------|
| Device:Tbird Fwd Driver         Role:L2                      Mod:13    |
| Last cleared @ Wed Nov 28 15:36:11 2012| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:0
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
493  THB_EPR3_CNT_GOOD_SF_DROP                0000000000000537   1-12 - 0000000000000591     +54
497  THB_EPR3_CNT_GOOD_PKT_DROP               0000000000000537   1-12 - 0000000000000591     +54
509  THB_EPR3_CNT_OTHER_DROP                  0000000000000537   1-12 - 0000000000000591     +54

Supervisor Packet-Flow Dropped

This command gives this example output:

show hardware internal packet-flow dropped module 13

|------------------------------------------------------------------------|
| Device:Tbird Fwd Driver         Role:L2                      Mod:13    |
| Last cleared @ Wed Nov 28 15:36:11 2012
|------------------------------------------------------------------------|
Instance:0
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
493  THB_EPR3_CNT_GOOD_SF_DROP                0000000000000537   1-12 - 0000000000000591     +54
497  THB_EPR3_CNT_GOOD_PKT_DROP               0000000000000537   1-12 - 0000000000000591     +54
509  THB_EPR3_CNT_OTHER_DROP                  0000000000000537   1-12 - 0000000000000591     +54

Linecard OBFL Error-Stats

This command gives this example output:

show logging onboard module 13 error-stats

rtp-san-23-02-9513# show logging onboard module 13 error-stats

----------------------------
    Module: 13
----------------------------
--------------------------------------------------------------------------------
 ERROR STATISTICS INFORMATION FOR DEVICE DEVICE: FCMAC
--------------------------------------------------------------------------------
    Interface      |                                |         |    Time Stamp  
     Range        |    Error Stat Counter Name     |  Count  |MM/DD/YY HH:MM:SS
                   |                                |         |                
--------------------------------------------------------------------------------
fc13/13            |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |776      |12/17/12 08:28:38
fc13/1             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |777      |12/17/12 08:28:38
fc13/13            |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |695      |12/17/12 08:28:18
fc13/1             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |697      |12/17/12 08:28:18
fc13/13            |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |628      |12/17/12 08:27:58
fc13/1             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |27139    |12/17/12 08:27:58
fc13/1             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |630      |12/17/12 08:27:58
fc13/13            |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |541      |12/17/12 08:27:38
fc13/1             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |543      |12/17/12 08:27:38

Linecard OBFL Flow-Control Timeout-Drops

This command was issued:

show logging onboard flow-control timeout-drops module 13

There is a syntax error when show logging onboard timeout-drops is parsed.

Linecard OBFL Flow-Control Request-Timeout

This command was issued:

show logging onboard flow-control request-timeout module 13

There is a syntax error when show logging onboard request-timeout is parsed.

Creditmon

This command was issued:

show process creditmon credit-loss-events module 13

There is no change from previous value.

Linecard Statistics

This command gives this example output:

slot 13 show hardware internal statistics device fcmac all port 13
rtp-san-23-02-9513# slot 13 show hardware internal statistics device fcmac all port 1

|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod:13    |
| Last cleared @ Wed Nov 28 15:36:11 2012
|------------------------------------------------------------------------|
Instance:0
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
119  THB_TMM_TIMEOUT_STATS_DROP               0000000000027128   1-6 -  0000000000027230    +102
1001 THB_RPB_OUT_PKT_CNT                      0000000000897031   1 -    0000000000897076     +45
1002 THB_RPB_NC_FRM_CNT                       0000000000320352   1 -    0000000000320358     +6
1003 THB_RPB_LP_FRM_CNT                       0000000000320352   1 -    0000000000320358     +6
1016 THB_TMM_PORT_TBBZ_CH0                    0000000000050587   1 -    0000000000053961  +3374
1028 THB_TMM_PORT_TWAIT_CNT                   0000032055488369   1 -    0000034152485776  +2096997407
1033 THB_TMM_PORT_TX_FRAME_CNT                0000000001098361   1 -    0000000001131514  +33153
1048 IP_FCMAC_CNT_STATS_DATA_TX_CLASS3_FRAMES 0000000000521794   1 -    0000000000554912  +33118
1097 THB_RXF_TO_LB_CNT                        0000000000320357   1 -    0000000000320363     +6
1117 THB_TMM_TOLB_LB_CNT                      0000000000320352   1 -    0000000000320358     +6
1118 THB_TMM_TOLB_TO_CNT                      0000000000027144   1 -    0000000000027246   +102
1122 THB_TMM_TOLB_TIMEOUT_DROP_CNT            0000000000027139   1 -    0000000000027241   +102

Linecard Credit-Info

This command gives this example output:

slot 13 show hard internal credit-info port 1

rtp-san-23-02-9513# slot 13 show hard internal credit-info port 1

  ======== Device Credit Information - RX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|   1  |  0/0 |     Tbird Mac Driver |       0xfa |     0x0 |   Full |
+------+------+----------------------+------------+---------+--------+
    ======== Device Credit Information - TX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|   1  |  0/0 |     Tbird Mac Driver |      0x1f4 |   0x1f4 |   Full |
+------+------+----------------------+------------+---------+--------+

Linecard Queued Packets

These commands were issued:

slot 13 show hardware internal que inst 1 memory iqm-statusmem0
slot 13 show hardware internal que inst 1 memory iqm-statusmem1

There is no usable output.

Port Link-Events

This command was issued:

slot 13 show port-config internal link-events

There is no change from previous value.

MDS Values and Counters after Step 7: rtp-san-23-01-9513 port-channel 237 - Slow Drain Side - fc4/9

Interface Counters - fc4/9

This command was issued:

show interface fc4/9 counters

These are the changes, if any, on the MDS counters after Step 7:

input discards - No change from previous value.
input OLS - No change from previous value.
input LRR - No change from previous value.
input NOS - No change from previous value.

output discards - No change from previous value.
output OLS - No change from previous value.
output LRR - No change from previous value.
output NOS - No change from previous value.

transmit B2B credit transitions from zero - 0 - No change from previous value.
receive B2B credit transitions from zero - +57467.
receive B2B credit remaining - 0 - No change from previous value.
transmit B2B credit remaining - 250 - No change from previous value.

Supervisor Level Hardware-Detected Errors

This command gives this example output:

show hardware internal errors module 4

|------------------------------------------------------------------------|
| Device:Tbird Fwd Driver         Role:L2                      Mod: 4    |
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
423  THB_EPR1_CNT_GOOD_SF_DROP                0000000000000615   9-16 - 0000000000000708     +93
427  THB_EPR1_CNT_GOOD_PKT_DROP               0000000000000615   9-16 - 0000000000000708     +93
439  THB_EPR1_CNT_OTHER_DROP                  0000000000000615   9-16 - 0000000000000708     +93

|------------------------------------------------------------------------|
| Device:Tbird Xbar Driver        Role:FABRIC                  Mod: 4    |
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:1
ID   Name                                          Value              Ports  Next value(30sec) Delta
--   ----                                          -----              -----  ----------------- ------
161  EFI-1: Dropped packets                        0000000000000615   9-12 - 0000000000000708     +93
2142 EFI1_INT: Super-frame EOF before EOP errors   0000000000000170   9-12 - 0000000000000197     +27
2143 EFI1_INT: Vegas-2 super-frame too short error 0000000000000170   9-12 - 0000000000000197     +27

Supervisor Packet-Flow Dropped

This command gives this example output:

show hardware internal packet-flow dropped module 4

rtp-san-23-01-9513# show hardware internal packet-flow dropped module 4

|------------------------------------------------------------------------|
| Device:Tbird Xbar Driver        Role:FABRIC                  Mod: 4    |
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
161  EFI-1: Dropped packets                   0000000000000626   9-12 - 0000000000000719     +93

|------------------------------------------------------------------------|
| Device:Tbird Que Driver         Role:QUE                     Mod: 4    |
|------------------------------------------------------------------------|
|------------------------------------------------------------------------|
| Device:Tbird Fwd Driver         Role:L2                      Mod: 4    |
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
423  THB_EPR1_CNT_GOOD_SF_DROP                0000000000000626   9-16 - 0000000000000719     +93
427  THB_EPR1_CNT_GOOD_PKT_DROP               0000000000000626   9-16 - 0000000000000719     +93
439  THB_EPR1_CNT_OTHER_DROP                  0000000000000626   9-16 - 0000000000000719     +93

Linecard OBFL Error-Stats

This command gives this example output:

show logging onboard module 4 error-stats 

rtp-san-23-01-9513# show logging onboard module 4 error-stats

----------------------------
    Module:  4
----------------------------
--------------------------------------------------------------------------------
 ERROR STATISTICS INFORMATION FOR DEVICE DEVICE: FCMAC
--------------------------------------------------------------------------------
    Interface      |                                |         |    Time Stamp  
     Range        |    Error Stat Counter Name     |  Count  |MM/DD/YY HH:MM:SS
                   |                                |         |                
--------------------------------------------------------------------------------
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |688408   |12/17/12 09:28:40
fc4/13             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |1885     |12/17/12 09:28:40
fc4/13             |FCP_SW_CNTR_CREDIT_LOSS         |188      |12/17/12 09:28:40
fc4/9              |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |886      |12/17/12 09:28:40
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |667625   |12/17/12 09:28:20
fc4/13             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |1711     |12/17/12 09:28:20
fc4/13             |FCP_SW_CNTR_CREDIT_LOSS         |171      |12/17/12 09:28:20
fc4/9              |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |791      |12/17/12 09:28:20
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |647760   |12/17/12 09:28:00
fc4/13             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |1538     |12/17/12 09:28:00
fc4/13             |FCP_SW_CNTR_CREDIT_LOSS         |153      |12/17/12 09:28:00
fc4/9              |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |716      |12/17/12 09:28:00
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |627337   |12/17/12 09:27:40
fc4/13             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |1365     |12/17/12 09:27:40
fc4/13             |FCP_SW_CNTR_CREDIT_LOSS         |136      |12/17/12 09:27:40
fc4/9              |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |636      |12/17/12 09:27:40
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |606474   |12/17/12 09:27:20
fc4/13             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |1192     |12/17/12 09:27:20
fc4/13             |FCP_SW_CNTR_CREDIT_LOSS         |119      |12/17/12 09:27:20
fc4/9              |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |574      |12/17/12 09:27:20

Linecard OBFL Flow-Control Timeout-Drops

This command gives this example output:

show logging onboard flow-control timeout-drops module 4 

rtp-san-23-01-9513# show logging onboard flow-control timeout-drops module 4

----------------------------
    Module:  4
----------------------------
--------------------------------------------------------------------------------
 ERROR STATISTICS INFORMATION FOR DEVICE DEVICE: FCMAC
--------------------------------------------------------------------------------
    Interface      |                                |         |    Time Stamp  
     Range        |    Error Stat Counter Name     |  Count  |MM/DD/YY HH:MM:SS
                   |                                |         |                
--------------------------------------------------------------------------------
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |688408   |12/17/12 09:28:40
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |667625   |12/17/12 09:28:20
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |647760   |12/17/12 09:28:00
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |627337   |12/17/12 09:27:40
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |606474   |12/17/12 09:27:20

Linecard OBFL Flow-Control Request-Timeout

This command was issued:

show logging onboard flow-control request-timeout module 4

There is no change from previous value.

Creditmon

This command was issued:

show process creditmon credit-loss-events module 4

There are no events for fc4/9.

Linecard Statistics

This command gives this example output:

slot 4 show hardware internal statistics device fcmac all port 9 

rtp-san-23-01-9513# slot 4 show hardware internal statistics device fcmac all port 9

|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod: 4    |
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
68   THB_RCM_RCP5_RBBZ_CH0                    0000000000054989   9 -    0000000000058489   +3500
1002 THB_RPB_NC_FRM_CNT                       0000000000565760   9 -    0000000000565766      +6
1044 IP_FCMAC_CNT_STATS_DATA_RX_CLASS3_FRAMES 0000000000571568   9 -    0000000000607292  +35724
1045 IP_FCMAC_CNT_STATS_DATA_RX_CLASSF_FRAMES 0000000000329947   9 -    0000000000329966     +19
1048 IP_FCMAC_CNT_STATS_DATA_TX_CLASS3_FRAMES 0000000000000032   9 -    0000000000000032      +0
1049 IP_FCMAC_CNT_STATS_DATA_TX_CLASSF_FRAMES 0000000000329973   9 -    0000000000329998     +25
1097 THB_RXF_TO_LB_CNT                        0000000000565760   9 -    0000000000565766      +6
1117 THB_TMM_TOLB_LB_CNT                      0000000000565760   9 -    0000000000565766      +6

Linecard Credit-Info

This command was issued:

slot 4 show hard internal credit-info port 9

There is no change from previous value, but all RX credits are used, and six additional credits are used. This is example output:

rtp-san-23-01-9513# slot 4 show hard internal credit-info port 9
    ======== Device Credit Information - RX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|   9  |  0/0 |     Tbird Mac Driver |      0x1f4 |   0x1fa |   Full |
+------+------+----------------------+------------+---------+--------+
    ======== Device Credit Information - TX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|   9  |  0/0 |     Tbird Mac Driver |       0xfa |     0x0 |   Full |
+------+------+----------------------+------------+---------+--------+

Linecard Queued Packets

These commands were issued:

slot 4 show hardware internal que inst 1 memory iqm-statusmem0
slot 4 show hardware internal que inst 1 memory iqm-statusmem1

This indicates that one or more packets are queued to destination index xC, which, in this test, is port fc4/13:

rtp-san-23-01-9513# show port internal info interface fc4/13
fc4/13 - if_index: 0x0118C000, phy_port_index: 0xc
     local_index: 0xc

000001 indicates the first port in Thunderbird ASIC instance 1. For model DS-X9232-256K9, instance 1 is for ports 9-16, so that indicates fc4/9 is where the packets are ingressing going to fc4/13.

For model DS-X9248-256K9, each Thunderbird instance handles 12 ports, so instance 1 would start with port 13.

rtp-san-23-01-9513# slot 4 show hardware internal que inst 1 memory iqm-statusmem0 
+-------------------------------------------------------------------------------
| IQM: PG0 Status Memory for Tbird Que Driver
| Inst 1; port(s) 9-16
|

Only non-zero entries are displayed. Each non-zero bit indicates pending frame in VOQ for that IB.

+----------+--------+--------+--------+--------+
| GI (Hex) | Prio 0 | Prio 1 | Prio 2 | Prio 3 |
+----------+--------+--------+--------+--------+
|        c | 000000 | 000000 | 000000 | 000001 |
+----------+--------+--------+--------+--------+
         ^                                   ^
         |                                   |
egress port (slow)                       ingress port

rtp-san-23-01-9513# slot 4 show hardware internal que inst 1 memory iqm-statusmem1
+-------------------------------------------------------------------------------
| IQM: PG1 Status Memory for Tbird Que Driver
| Inst 1; port(s) 9-16
|

Only non-zero entries are displayed.

Port Link-Events

This command was issued:

slot 4 show port-config internal link-events

There is no change from previous value.

MDS Values and Counters after Step 9: rtp-san-23-01-9513 fc4/13 - Port Connected to the Slow Device

Interface Counters - fc4/13

This command was issued:

show interface fc4/13 counters

These are the changes, if any, on the MDS counters after Step 9:

input discards - No change from previous value.
input OLS - No change from previous value.
input LRR - +18.
input NOS - No change from previous value.

output discards - No change from previous value.
output OLS - No change from previous value.
output LRR - No change from previous value.
output NOS - No change from previous value.

transmit B2B credit transitions from zero +27.
receive B2B credit transitions from zero +27
.
receive B2B credit remaining - 32 - No change from previous value.
transmit B2B credit remaining - 0 - No change from previous value.

Supervisor Level Hardware-Detected Errors

This command gives this example output:

show hardware internal errors module 4

rtp-san-23-01-9513# show hardware internal errors module 4

----------------------------------------
Hardware errors as reported in module 4
----------------------------------------
|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod: 4    |
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
272  THB_TMM_TIMEOUT_STATS_DROP               0000000000668774   13-16 - 0000000000701235 +32461
1823 THB_TMM_PORT_FRM_DROP_CNT                0000000000668790   13 -    0000000000701251 +32461
1922 THB_TMM_TOLB_TIMEOUT_DROP_CNT            0000000000668790   13 -    0000000000701251 +32461

|------------------------------------------------------------------------|
| Device:Tbird Fwd Driver         Role:L2                      Mod: 4    |
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
423  THB_EPR1_CNT_GOOD_SF_DROP                0000000000000615   9-16 - 0000000000000708     +97
427  THB_EPR1_CNT_GOOD_PKT_DROP               0000000000000615   9-16 - 0000000000000708     +97
439  THB_EPR1_CNT_OTHER_DROP                  0000000000000615   9-16 - 0000000000000708     +97

|------------------------------------------------------------------------|
| Device:Tbird Xbar Driver        Role:FABRIC                  Mod: 4    |
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:1
ID   Name                                          Value              Ports  Next value(30sec) Delta
--   ----                                          -----              -----  ----------------- ------
161  EFI-1: Dropped packets                        0000000000000615   9-12 - 0000000000000708     +97
2142 EFI1_INT: Super-frame EOF before EOP errors   0000000000000170   9-12 - 0000000000000197     +27
2143 EFI1_INT: Vegas-2 super-frame too short error 0000000000000170   9-12 - 0000000000000197     +27

Supervisor Packet-Flow Dropped

This command gives this example output:

show hardware internal packet-flow dropped module 4 

rtp-san-23-01-9513# show hardware internal packet-flow dropped module 4

|------------------------------------------------------------------------|
| Device:Tbird Xbar Driver        Role:FABRIC                  Mod: 4    |
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
161  EFI-1: Dropped packets                   0000000000000626   9-12 - 0000000000000719     +93

|------------------------------------------------------------------------|
| Device:Tbird Fwd Driver         Role:L2                      Mod: 4    |
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
423  THB_EPR1_CNT_GOOD_SF_DROP                0000000000000626   9-16 - 0000000000000719     +93
427  THB_EPR1_CNT_GOOD_PKT_DROP               0000000000000626   9-16 - 0000000000000719     +93
439  THB_EPR1_CNT_OTHER_DROP                  0000000000000626   9-16 - 0000000000000719     +93
|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod: 4    |
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
272  THB_TMM_TIMEOUT_STATS_DROP               0000000000673431   13-16 -0000000000705719  +32288
1823 THB_TMM_PORT_FRM_DROP_CNT                0000000000673447   13 -   0000000000705735  +32288
1922 THB_TMM_TOLB_TIMEOUT_DROP_CNT            0000000000673447   13 -   0000000000705735  +32288

Linecard OBFL Error-Stats

This command gives this example output:

show logging onboard module 4 error-stats 

rtp-san-23-01-9513# show logging onboard module 4 error-stats

----------------------------
    Module:  4
----------------------------
--------------------------------------------------------------------------------
 ERROR STATISTICS INFORMATION FOR DEVICE DEVICE: FCMAC
--------------------------------------------------------------------------------
    Interface      |                                |         |    Time Stamp  
      Range        |    Error Stat Counter Name     |  Count  |MM/DD/YY HH:MM:SS
                   |                                |         |                
--------------------------------------------------------------------------------
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |688408   |12/17/12 09:28:40
fc4/13             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |1885     |12/17/12 09:28:40
fc4/13             |FCP_SW_CNTR_CREDIT_LOSS         |188      |12/17/12 09:28:40
fc4/9              |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |886      |12/17/12 09:28:40
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |667625   |12/17/12 09:28:20
fc4/13             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |1711     |12/17/12 09:28:20
fc4/13             |FCP_SW_CNTR_CREDIT_LOSS         |171      |12/17/12 09:28:20
fc4/9              |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |791      |12/17/12 09:28:20
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |647760   |12/17/12 09:28:00
fc4/13             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |1538     |12/17/12 09:28:00
fc4/13             |FCP_SW_CNTR_CREDIT_LOSS         |153      |12/17/12 09:28:00
fc4/9              |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |716      |12/17/12 09:28:00
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |627337   |12/17/12 09:27:40
fc4/13             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |1365     |12/17/12 09:27:40
fc4/13             |FCP_SW_CNTR_CREDIT_LOSS         |136      |12/17/12 09:27:40
fc4/9              |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |636      |12/17/12 09:27:40
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |606474   |12/17/12 09:27:20
fc4/13             |FCP_SW_CNTR_TX_WT_AVG_B2B_ZERO  |1192     |12/17/12 09:27:20
fc4/13             |FCP_SW_CNTR_CREDIT_LOSS         |119      |12/17/12 09:27:20
fc4/9              |FCP_SW_CNTR_RX_WT_AVG_B2B_ZERO  |574      |12/17/12 09:27:20

Linecard OBFL Flow-Control Timeout-Drops

This command gives this example output:

show logging onboard flow-control timeout-drops module 4 

rtp-san-23-01-9513# show logging onboard flow-control timeout-drops module 4

----------------------------
    Module:  4
----------------------------
--------------------------------------------------------------------------------
 ERROR STATISTICS INFORMATION FOR DEVICE DEVICE: FCMAC
--------------------------------------------------------------------------------
    Interface      |                                |         |    Time Stamp  
     Range        |    Error Stat Counter Name     |  Count  |MM/DD/YY HH:MM:SS
                   |                                |         |                
--------------------------------------------------------------------------------
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |688408   |12/17/12 09:28:40
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |667625   |12/17/12 09:28:20
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |647760   |12/17/12 09:28:00
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |627337   |12/17/12 09:27:40
fc4/13             |THB_TMM_TOLB_TIMEOUT_DROP_CNT   |606474   |12/17/12 09:27:20

Linecard OBFL Flow-Control Request-Timeout

This command was issued:

show logging onboard flow-control request-timeout module 4

There is no change from previous value.

Creditmon

This command gives this example output:

show process creditmon credit-loss-events module 4

rtp-san-23-01-9513# show process creditmon credit-loss-events module 4
       Module: 04      Credit Loss Events: YES
----------------------------------------------------
| Interface |  Total |          Timestamp          |
|           | Events |                             |
----------------------------------------------------
| fc4/13    |    205 | 1. Mon Dec 17 09:28:59 2012 |
|           |        | 2. Mon Dec 17 09:28:58 2012 |
|           |        | 3. Mon Dec 17 09:28:57 2012 |
|           |        | 4. Mon Dec 17 09:28:56 2012 |
|           |        | 5. Mon Dec 17 09:28:55 2012 |
|           |        | 6. Mon Dec 17 09:28:54 2012 |
|           |        | 7. Mon Dec 17 09:28:52 2012 |
|           |        | 8. Mon Dec 17 09:28:51 2012 |
|           |        | 9. Mon Dec 17 09:28:50 2012 |
|           |        |10. Mon Dec 17 09:28:49 2012 |
----------------------------------------------------

Linecard Statistics

This command gives this example output:

slot 4 show hardware internal statistics device fcmac all port 13

rtp-san-23-01-9513# slot 4 show hardware internal statistics device fcmac all port 13
|------------------------------------------------------------------------|
| Device:Tbird Mac Driver         Role:FCMAC                   Mod: 4    |
|------------------------------------------------------------------------|
Instance:1
ID   Name                                     Value              Ports  Next value(30sec) Delta
--   ----                                     -----              -----  ----------------- ------
201  THB_RCM_RCP0_RBBZ_CH0                    0000000000000189   13 -   0000000000000216     +27
272  THB_TMM_TIMEOUT_STATS_DROP               0000000000675772   13-16 -0000000000708489  +32717
1816 THB_TMM_PORT_TBBZ_CH0                    0000000000045018   13 -   0000000000045045     +27
1821 THB_TMM_PORT_NP_CNT                      0000000000068432   13 -   0000000000071888   +3456
1822 THB_TMM_PORT_TOLB_FRM_SENT_CNT           0000000000565760   13 -   0000000000565766      +6
1823 THB_TMM_PORT_FRM_DROP_CNT                0000000000675788   13 -   0000000000708505  +32717
1828 THB_TMM_PORT_TWAIT_CNT                   0000069165935092   13 -   0000072511847155 +3345912063
1833 THB_TMM_PORT_TX_FRAME_CNT                0000000000068432   13 -   0000000000071888   +3456
1844 IP_FCMAC_CNT_STATS_DATA_RX_CLASS3_FRAMES 0000000000000237   13 -   0000000000000264     +27
1848 IP_FCMAC_CNT_STATS_DATA_TX_CLASS3_FRAMES 0000000000068427   13 -   0000000000071883   +3456
1896 THB_RXF_NP_CNT                           0000000000000261   13 -   0000000000000288     +27
1897 THB_RXF_TO_LB_CNT                        0000000000565760   13 -   0000000000565766      +6
1900 THB_RXF_SPARE_CNT                        0000000000566021   13 -   0000000000566054     +33
1901 THB_TMM_CH_CNT_CH0                       0000000000068432   13 -   0000000000071888   +3456
1911 THB_TMM_TO_CNT_CLASS_3                   0000000000675788   13 -   0000000000708505  +32717
1917 THB_TMM_TOLB_LB_CNT                      0000000000565760   13 -   0000000000565766      +6
1918 THB_TMM_TOLB_TO_CNT                      0000000000675788   13 -   0000000000708505  +32717
1922 THB_TMM_TOLB_TIMEOUT_DROP_CNT            0000000000675788   13 -   0000000000708505  +32717

Linecard Credit-Info

This command was issued:

slot 4 show hard internal credit-info port 13

There is no change from previous value, but all TX credits are used. This is example output:

rtp-san-23-01-9513

    ======== Device Credit Information - RX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|  13  | 16/0 |     Tbird Mac Driver |       0x20 |     0x0 |   Full |
+------+------+----------------------+------------+---------+--------+
    ======== Device Credit Information - TX ========
+------+------+----------------------+------------+---------+--------+
| PORT | SI/  |     DEVICE NAME      |  CREDITS   | CREDITS |   BW   |
|  NO  | PRIO |                      | CONFIGURED |  USED   |  MODE  |
+------+------+----------------------+------------+---------+--------+
|  13  | 16/0 |     Tbird Mac Driver |       0x80 |    0x80 |   Full |
+------+------+----------------------+------------+---------+--------+

Linecard Queued Packets

These commands were issued:

slot 4 show hardware internal que inst 1 memory iqm-statusmem0
slot 4 show hardware internal que inst 1 memory iqm-statusmem1

The output is not usable.

Refer to MDS Values and Counters after Step 7, Linecard Queued Packets.

Port Link-Events

This command was issued:

slot 4 show port-config internal link-events

There is no change from previous value.

Test 3: Port-Monitor - Slow Port Emulation - R_RDY Delay 1500000us (1.5 sec)

This port-monitor, slow port emulation test with an R-RDY delay of 1500000us (1.5 seconds) is based upon the Cisco Data Center Network Manager (DCNM), Release 6.1(1a).

Create Port-Monitor Policy

These commands give this example output:

rtp-san-23-01-9513(config)# port-monitor name edmgen4slowdrain
rtp-san-23-01-9513(config-port-monitor)#
rtp-san-23-01-9513(config)# port-monitor activate edmgen4slowdrain
rtp-san-23-01-9513(config)# show port-monitor active

Policy Name  : edmgen4slowdrain
Admin status : Active
Oper status  : Active
Port type    : All Ports
---------------------------------------------------------------------------------------------------------
Counter                  Threshold  Interval Rising Threshold event Falling Threshold  event PMON Portguard
-------                  ---------  -------- ---------------- ----- ------------------ ----- --------------
Link Loss                Delta      60       5                4     1                  4     Not enabled
Sync Loss                Delta      60       5                4     1                  4     Not enabled
Signal Loss              Delta      60       5                4     1                  4     Not enabled
Invalid Words            Delta      60       1                4     0                  4     Not enabled
Invalid CRC's            Delta      60       5                4     1                  4     Not enabled
TX Discards              Delta      60       200              4     10                 4     Not enabled
LR RX                    Delta      60       5                4     1                  4     Not enabled
LR TX                    Delta      60       5                4     1                  4     Not enabled
Timeout Discards         Delta      60       200              4     10                 4     Not enabled
Credit Loss Reco         Delta      1        1                4     0                  4     Not enabled
TX Credit Not Available  Delta      1        10               4     0                  4     Not enabled
RX Datarate              Delta      60       80%              4     20%                4     Not enabled
TX Datarate              Delta      60       80%              4     20%                4     Not enabled
----------------------------------------------------------------------------------------------------------
rtp-san-23-01-9513(config)#

Rerun Test

Rerun Test 2 with R_RDY Delay 1500000us (1.5 sec) for approximately 60 seconds.

View Threshold Manager Log

Navigate to Device Manager > Logs > Switch Resident > Threshold Manager in order to see the Threshold Manager Log.

116098-trouble-gen4-01.jpg

Note: This image was taken at a different time and does not reflect this example test.

In text format, the Threshold Manager Log for Test 3 is:

EventId, Id    Time     Description     
4, 1    2012/12/10-14:51:14    fcIfTxWtAvgBBCreditTransitionToZero.18399232=20 >= 10:65500, 4 WARNING(4)Rising
4, 2    2012/12/10-14:51:15    fcIfCreditLoss.18399232=1 >= 1:65500, 4 WARNING(4)Rising
4, 3    2012/12/10-14:51:19    fcIfCreditLoss.18399232=0 <= 0:65500, 4 WARNING(4)Falling
4, 4    2012/12/10-14:51:20    fcIfCreditLoss.18399232=1 >= 1:65500, 4 WARNING(4)Rising
4, 5    2012/12/10-14:51:29    fcIfCreditLoss.18399232=0 <= 0:65500, 4 WARNING(4)Falling
4, 6    2012/12/10-14:51:30    fcIfCreditLoss.18399232=1 >= 1:65500, 4 WARNING(4)Rising
4, 7    2012/12/10-14:51:40    fcIfCreditLoss.18399232=0 <= 0:65500, 4 WARNING(4)Falling
4, 8    2012/12/10-14:51:41    fcIfCreditLoss.18399232=1 >= 1:65500, 4 WARNING(4)Rising
4, 9    2012/12/10-14:51:43    fcIfLinkResetOuts.18399232=25 >= 5:65500, 4 WARNING(4)Rising
4, 10    2012/12/10-14:51:43    fcIfTimeOutDiscards.18399232=30387 >= 200:65500, 4 WARNING(4)Rising
4, 11    2012/12/10-14:51:51    fcIfCreditLoss.18399232=0 <= 0:65500, 4 WARNING(4)Falling
4, 12    2012/12/10-14:51:52    fcIfCreditLoss.18399232=1 >= 1:65500, 4 WARNING(4)Rising
4, 13    2012/12/10-14:51:53    fcIfCreditLoss.18399232=0 <= 0:65500, 4 WARNING(4)Falling
4, 14    2012/12/10-14:51:53    fcIfTxWtAvgBBCreditTransitionToZero.18399232=0 <= 0:65500, 4 WARNING(4)Falling

Note: 18399232 is ifindex, which is 0x0118C000 and corresponds to fc4/13.

rtp-san-23-01-9513# show port internal info interface-id 0x0118C000 
fc4/13 - if_index: 0x0118C000, phy_port_index: 0xc
     local_index: 0xc

Object Identifiers (OIDs)

fcIfTxWtAvgBBCreditTransitionToZero 1.3.6.1.4.1.9.9.289.1.2.1.1.38

Note: There does not appear to be an OID in the Rx direction.

fcIfCreditLoss 1.3.6.1.4.1.9.9.289.1.2.1.1.37
fcIfLinkResetOuts - 1.3.6.1.4.1.9.9.289.1.2.1.1.10
fcIfTimeOutDiscards - 1.3.6.1.4.1.9.9.289.1.2.1.1.35
fcIfOutDiscards - 1.3.6.1.4.1.9.9.289.1.2.1.1.36 - not seen above - from notes

Appendix

Counter Definitions

These are the counters and their definitions:

CounterDefinition

201 THB_RCM_RCP0_RBBZ_CH0

Total count of transitions to zero for Rx B2B credits on ch0; this typically indicates the switch is applying back pressure to the attached device due to perceived congestion. This perceived congestion can be due to a lack of Tx B2B credits being returned on an interface over which this device is communicating.
272 THB_TMM_TIMEOUT_STATS_DROPTimeout stats dropped because stats FIFO full.
423 THB_EPR1_CNT_GOOD_SF_DROPNumber of superframe access control list (ACL) dropped.
427 THB_EPR1_CNT_GOOD_PKT_DROPNumber of packet ACL dropped.
439 THB_EPR1_CNT_OTHER_DROPNumber of multicast packets (MCST) dropped.

1122 THB_TMM_TOLB_TIMEOUT_DROP_CNT

Timeout drops at egress.

1816 THB_TMM_PORT_TBBZ_CH0Total count of transitions to zero for Tx B2B credits on ch0; this is typically due to the attached device withholding R_RDYs from the switch due to congestion in that device.
1823 THB_TMM_PORT_FRM_DROP_CNTNumber of frames dropped in tolb_path or np path.
1821 THB_TMM_PORT_NP_CNTNumber of frames in normal path, including error frames.
1822 THB_TMM_PORT_TOLB_FRM_SENT_CNTNumber of frames sent in tolb_path.
1823 THB_TMM_PORT_FRM_DROP_CNTNumber of frames dropped in tolb_path or np path; represents all types of packet drops like timeout, offline, and abort drops at egress.
1828 THB_TMM_PORT_TWAIT_CNTPacket is available to send but no credit is available; increments every cycle.
1833 THB_TMM_PORT_TX_FRAME_CNTNo description listed.
1844 IP_FCMAC_CNT_STATS_DATA_RX_CLASS3_FRAMESNumber of class 3 frames received.
1848 IP_FCMAC_CNT_STATS_DATA_TX_CLASS3_FRAMESNumber of class 3 frames transmitted.
1896 THB_RXF_NP_CNTCount of normal frames serviced.
1897 THB_RXF_TO_LB_CNTLB frames serviced; count of timeout/LB frames serviced.
1900 THB_RXF_SPARE_CNTSpare cnt.
1901 THB_TMM_CH_CNT_CH0Not found.
1911 THB_TMM_TO_CNT_CLASS_3 
1917 THB_TMM_TOLB_LB_CNTNot found.
1918 THB_TMM_TOLB_TO_CNTNot found.
1922 THB_TMM_TOLB_TIMEOUT_DROP_CNTNot listed.
FCP_SW_CNTR_RX_WT_AVG_B2B_ZEROCount of the number of times an interface was at 0 Rx B2B credits for 100ms; this typically indicates the switch is withholding R_RDYs to the device attached on that interface due to congestion in the path to devices it is communicating with.
FCP_SW_CNTR_TX_WT_AVG_B2B_ZEROCount of the number of times an interface was at 0 Tx B2B credits for 100ms; this typically indicates congestion at the device attached on that interface.
FCP_SW_CNTR_FORCE_TIMEOUT_ONCount of times the 'system timeout no-credit-drop' threshold has been hit by this port; when a port is at 0 Tx B2B credits for the time specified, the port starts to drop packets at line rate.
FCP_SW_CNTR_FORCE_TIMEOUT_OFFCount of times the port has recovered from the 'system timeout no-credit-drop' condition; this typically means R_RDYs have been returned or possibly that an LR/LRR has occurred.
FCP_SW_CNTR_CREDIT_LOSSCount of times the port has gone through credit loss recovery; this occurs when an F port is at 0 Tx B2B credits for one full second or when an E port is at 0 Tx B2B credits for 1.5 seconds; the switch initiates a Link Reset (LR) in order to recover lost credits.

Commands Issued to Each Switch

These are the commands issued to switch rtp-san-23-02-9513:

  • show clock
  • show interface fc13/1
  • show interface fc13/13
  • show interface fc13/1 counters
  • show interface fc13/13 counters
  • show hardware internal errors module 13
  • show hardware internal packet-flow dropped module 13
  • show hardware internal statistics module 13 pktflow dropped
  • show logging onboard module 13 error-stats
  • show logging onboard flow-control timeout-drops module 13
  • show logging onboard flow-control request-timeout module 13
  • show process creditmon credit-loss-events module 13
  • slot 13 show hardware internal statistics device fcmac all port 1
  • slot 13 show hardware internal statistics device fcmac all port 13
  • slot 13 show hard internal credit-info port 1
  • slot 13 show hardware internal credit-info port 13
  • slot 13 show hardware internal que inst 0 memory iqm-statusmem0
  • slot 13 show hardware internal que inst 0 memory iqm-statusmem1
  • slot 13 show hardware internal que inst 1 memory iqm-statusmem0
  • slot 13 show hardware internal que inst 1 memory iqm-statusmem1
  • slot 13 show port-config internal link-events
  • **end

These are the commands issued to switch rtp-san-23-01-9513:

  • show clock
  • show interface fc4/9
  • show interface fc4/13
  • show interface fc4/9 counters
  • show interface fc4/13 counters
  • show hardware internal errors module 4
  • show hardware internal packet-flow dropped module 4
  • show hardware internal statistics module 4 pktflow dropped
  • show logging onboard module 4 error-stats
  • show logging onboard flow-control timeout-drops module 4
  • show logging onboard flow-control request-timeout module 4
  • show process creditmon credit-loss-events module 4
  • slot 4 show hardware internal statistics device fcmac all port 9
  • slot 4 show hardware internal statistics device fcmac all port 13
  • slot 4 show hard internal credit-info port 9
  • slot 4 show hard internal credit-info port 13
  • slot 4 show hardware internal que inst 1 memory iqm-statusmem0
  • slot 4 show hardware internal que inst 1 memory iqm-statusmem1
  • slot 4 show port-config internal link-events
  • **end
Updated: Jul 17, 2013
Document ID: 116098