Introduction
This document describes the receive and transmit buffer-to-buffer (B2B) credits in different interfaces, which are useful when you troubleshoot performance issues in a Fibre Channel fabric.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- Cisco Multilayer Data Switch (MDS)
- Fibre Channel protocol
Components Used
The information in this document is based on Cisco MDS 9000 Series Multilayer Switches.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Understand Priority Transmit B2B Credits in MDS Switch
When you troubleshoot the slow drain issue for Fibre Channel fabric network, an easy way to confirm the available credits for interface is the show interface command in the MDS switches.
The following explains the transmit B2B credits in an MDS interface.
For an Inter-Switch Link Protocol (ISL) Trunk Interface
Low priority credits are used for data traffic while the high priority credits are for inter-switch-link control traffic. This is just a logical division on sender side to limit itself so that it does not consume all the available Receive (Rx) credits on the receiver.
For example, switch MDS Tempest (receiver) tells switch MDS Maestro (sender) that it has 64 Rx credits.
Switch MDS Maestro marks 56 credits as low priority Transmit (Tx) credits and 8 as high priority Tx credits.
In case of congestion, MDS Maestro stops sending data traffic when none of the 56 low priority credits is available.
However, it can still send control traffic to MDS Tempest using the available 8 high priority credits.
There is no need for MDS Tempest to make a differentiation between low or high priority credits.
Note: The number of high priority TX transmit credit depends on different hardware platforms.
This is the general concept: one out of eight TX transmit credit will be a high priority credit and the maximum is 15 - sufficient to handle the ELP or other ISL control messages.
This high/low priority differentiation is only for ISL links between E ports or TE ports and is not used in F ports.
MDS 9148S Platform Example:
For an E port interface:
Tempest# show int fc1/26
fc1/26 is trunking
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:1a:00:2a:6a:b9:d2:30
Peer port WWN is 20:1a:00:2a:6a:56:f0:00
Admin port mode is auto, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 2
Admin Speed is auto
Operating Speed is 4 Gbps
Rate mode is dedicated
Port flow-control is R_RDY
Transmit B2B Credit is 32
Receive B2B Credit is 64
B2B State Change Number is 14
Receive data field Size is 2112
Beacon is turned off
Logical type is core
Trunk vsans (admin allowed and active) (2)
Trunk vsans (up) (2)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
5 minutes input rate 160 bits/sec,20 bytes/sec, 0 frames/sec
5 minutes output rate 96 bits/sec,12 bytes/sec, 0 frames/sec
22464 frames input,1801536 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
22977 frames output,1318564 bytes
0 discards,0 errors
0 input OLS,0 LRR,0 NOS,2 loop inits
1 output OLS,2 LRR, 0 NOS, 1 loop inits
64 receive B2B credit remaining
32 transmit B2B credit remaining
28 low priority transmit B2B credit remaining
Interface last changed at Thu Jul 20 15:40:32 2017
Last clearing of "show interface" counters : never
Tempest#
The orther end of the link shows total transmit credit is 64 and low priority is 56:
Maestro# show int fc1/26
fc1/26 is trunking
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:1a:00:2a:6a:56:f0:00
Peer port WWN is 20:1a:00:2a:6a:b9:d2:30
Admin port mode is auto, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 2
Speed is 4 Gbps
Rate mode is dedicated
Transmit B2B Credit is 64
Receive B2B Credit is 32
B2B State Change Number is 14
Receive data field Size is 2112
Beacon is turned off
admin fec state is down
oper fec state is down
Trunk vsans (admin allowed and active) (2)
Trunk vsans (up) (2)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
5 minutes input rate 160 bits/sec,20 bytes/sec, 0 frames/sec
5 minutes output rate 216 bits/sec,27 bytes/sec, 0 frames/sec
23189 frames input,1330368 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
22677 frames output,1818076 bytes
0 discards,0 errors
1 input OLS,2 LRR,0 NOS,3 loop inits
1 output OLS,0 LRR, 0 NOS, 1 loop inits
32 receive B2B credit remaining
64 transmit B2B credit remaining
56 low priority transmit B2B credit remaining
Interface last changed at Wed Jul 19 16:39:58 2017
Last clearing of "show interface" counters 1w 5d
Maestro#
For an F Port Interface:
In the case of an F port, this calculation does not apply since there is no control traffic between the MDS switch and the initiator/target device.
However, you can see all the transmit credits in an F port which are marked as low priority for data traffic.
For example, total transmit credit is 3 and low priority is 3 in the following output:
switch# show int fc1/47
fc1/47 is up
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:2f:00:2a:6a:56:f0:00
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is F, FCID is 0x570f00
Port vsan is 2
Speed is 4 Gbps
Rate mode is dedicated
Transmit B2B Credit is 3
Receive B2B Credit is 32
Receive data field Size is 2112
Beacon is turned off
admin fec state is down
oper fec state is down
5 minutes input rate 0 bits/sec,0 bytes/sec, 0 frames/sec
5 minutes output rate 16 bits/sec,2 bytes/sec, 0 frames/sec
1220 frames input,74256 bytes
0 discards,0 errors
0 invalid CRC/FCS,0 unknown class
0 too long,0 too short
2231 frames output,220056 bytes
0 discards,0 errors
1 input OLS,1 LRR,0 NOS,0 loop inits
1 output OLS,0 LRR, 1 NOS, 0 loop inits
32 receive B2B credit remaining
3 transmit B2B credit remaining
3 low priority transmit B2B credit remaining
Interface last changed at Wed Jul 19 15:56:15 2017
Last clearing of "show interface" counters 1w 5d
switch#
MDS 9710 switch example:
There are no high priority Tx transmit credits in MDS97xx switches. However, this can be changed by the command system default tx-credit double-queue.
The maximum Tx transmit credits can also be seen from the interface of the connected switch.
Topology: MDS 9710 Leonard fc3/14 is connected to MDS 9148S Tempest fc1/13 back to back via an E port.
Default setting:
By default, MDS 9710 DS-X9448-768K9 line cards do not have high priority credits, all 64 transmit credits are deemed low priority.
Leonard(config-if)# show int fc3/14 counter
fc3/14
5 minutes input rate 320 bits/sec, 40 bytes/sec, 3 frames/sec
5 minutes output rate 2688 bits/sec, 336 bytes/sec, 3 frames/sec
1416819 frames input, 77233324 bytes
0 discards, 0 errors, 0 CRC/FCS
0 unknown class, 0 too long, 0 too short
1416813 frames output, 138598368 bytes
0 discards, 0 errors
0 timeout discards, 0 credit loss
53 input OLS, 85 LRR, 119 NOS, 0 loop inits
77 output OLS, 7 LRR, 93 NOS, 0 loop inits
14 link failures, 37 sync losses, 2 signal losses
140 Transmit B2B credit transitions to zero
92 Receive B2B credit transitions to zero
0 2.5us TxWait due to lack of transmit credits
Percentage Tx credits not available for last 1s/1m/1h/72h: 0%/0%/0%/0%
500 receive B2B credit remaining
64 transmit B2B credit remaining
64 low priority transmit B2B credit remaining
Last clearing of "show interface" counters : never
Leonard(config-if)#
Using tx-credit double-queue:
After issuing the command on switch Leonard system default tx-credit double-queue
(Need to reset the interface by shut/no shut)
Leonard(config)# show system default switchport
System default port state is down
System default trunk mode is on
System default link failure syslog logging level is 5
System default tx credit queue type is 0
Leonard(config)# system default tx-credit double-queue
Leonard(config)# show system default switchport
System default port state is down
System default trunk mode is on
System default link failure syslog logging level is 5
System default tx credit queue type is 1
Leonard(config)# int fc3/14
Leonard(config-if)# shut
Leonard(config-if)# no shut
it can be seen that there are 8 high priority Tx transmit B2B credits and 56 low priority credits:
Leonard(config-if)# show int fc3/14 counter
fc3/14
5 minutes input rate 320 bits/sec, 40 bytes/sec, 2 frames/sec
5 minutes output rate 256 bits/sec, 32 bytes/sec, 2 frames/sec
1420613 frames input, 77459300 bytes
0 discards, 0 errors, 0 CRC/FCS
0 unknown class, 0 too long, 0 too short
1420607 frames output, 138968464 bytes
0 discards, 0 errors
0 timeout discards, 0 credit loss
54 input OLS, 87 LRR, 122 NOS, 0 loop inits
79 output OLS, 7 LRR, 95 NOS, 0 loop inits
14 link failures, 38 sync losses, 2 signal losses
143 Transmit B2B credit transitions to zero
94 Receive B2B credit transitions to zero
0 2.5us TxWait due to lack of transmit credits
Percentage Tx credits not available for last 1s/1m/1h/72h: 0%/0%/0%/0%
500 receive B2B credit remaining
64 transmit B2B credit remaining
56 low priority transmit B2B credit remaining
Last clearing of "show interface" counters : never
Leonard(config-if)#
Maximum RX transmit credit is 15 from switch MDS 9148S Tempest:
From the connected MDS 9148S,Tempest, it can be seen the maximum Tx transmit B2B credits is 15.
MDS9710 switch Leonard has 500 receive credits by default hence MDS9148S switch Tempest will also have 500 Tx transmit B2B credits remaining but 15 credits (the maximum) is allocated as high priority (500 - 485 = 15).
Tempest# show int fc1/13 counter
fc1/13
5 minutes input rate 2624 bits/sec, 328 bytes/sec, 3 frames/sec
5 minutes output rate 1504 bits/sec, 188 bytes/sec, 3 frames/sec
1343331 frames input, 131489464 bytes
0 discards, 0 errors, 0 CRC/FCS
0 unknown class, 0 too long, 0 too short
1343332 frames output, 72878124 bytes
0 discards, 0 errors
0 timeout discards, 0 credit loss
9 input OLS, 2 LRR, 0 NOS, 0 loop inits
12 output OLS, 20 LRR, 9 NOS, 0 loop inits
9 link failures, 0 sync losses, 0 signal losses
66 Transmit B2B credit transitions to zero
24 Receive B2B credit transitions to zero
0 2.5us TxWait due to lack of transmit credits
Percentage Tx credits not available for last 1s/1m/1h/72h: 0%/0%/0%/0%
64 receive B2B credit remaining
500 transmit B2B credit remaining
485 low priority transmit B2B credit remaining
Last clearing of "show interface" counters :never
Tempest#
When the receive credits from MDS 9710 Leonard is changed to lower than 120, then one out of eight Tx credits will be marked as high priority:
Leonard(config)# int fc3/14
Leonard(config-if)# switchport fcrxbbcredit 112
Tempest# show int fc1/13 count
fc1/13
5 minutes input rate 1024 bits/sec, 128 bytes/sec, 3 frames/sec
5 minutes output rate 1536 bits/sec, 192 bytes/sec, 3 frames/sec
1360026 frames input, 133105756 bytes
0 discards, 0 errors, 0 CRC/FCS
0 unknown class, 0 too long, 0 too short
1360027 frames output, 73885304 bytes
0 discards, 0 errors
0 timeout discards, 0 credit loss
14 input OLS, 2 LRR, 0 NOS, 0 loop inits
17 output OLS, 30 LRR, 14 NOS, 0 loop inits
14 link failures, 0 sync losses, 0 signal losses
91 Transmit B2B credit transitions to zero
34 Receive B2B credit transitions to zero
0 2.5us TxWait due to lack of transmit credits
Percentage Tx credits not available for last 1s/1m/1h/72h: 0%/0%/0%/0%
64 receive B2B credit remaining
112 transmit B2B credit remaining
98 low priority transmit B2B credit remaining
Last clearing of "show interface" counters :never
Tempest#
Feedback