Reaping the Benefits of Cisco MDS 9000 Buffer-to-Buffer Credit Recovery Solution Overview
Available Languages
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
At times, there are technical features that contribute significantly to smooth operations despite receiving little attention. In the case of storage area networks built on the Fibre Channel (FC) protocol, Buffer-to-Buffer credit recovery (BB_credit recovery) is one such feature. The Cisco MDS 9000 Series Multilayer Switches are purpose built to enable applications access their data on shared storage devices and provide a standards compliant implementation of the BB_credit recovery feature. The MDS 9000 series has now enabled BB_credit recovery on all port types by default. However, for the feature to become fully functional, the HBAs (initiators) and storage arrays (targets) need to also enable it on their side. Enabling this on Fibre Channel links improves resiliency, enhances application performance, and reduces troubleshooting efforts for both traditional and AI workloads.
● Improved network resiliency: automatically restores lost credits caused by physical-layer errors, ensuring lossless frame delivery (enabled by default on all Cisco MDS 9000 ports)
● Enhanced application performance: maintains smooth data flow for both traditional and AI workloads by addressing corrupted or lost R_RDYs (leverages the INCITS T11 Fibre Channel standard for BB_credit recovery)
● Simplified configuration: reduces the configuration burden because the feature is enabled by default on all port types from Cisco NX-OS 8.4(1) release onward (default enablement on all port types)
● Reduced troubleshooting efforts: automatically recovers from credit loss, minimizing manual intervention and troubleshooting time (enabled across all Fibre Channel links)
Fibre Channel flow control
Fibre Channel uses a buffer-to-buffer credit flow control mechanism to ensure lossless frame delivery. The flow-control mechanism is very efficient, foundational to how Fibre Channel functions and an important part of its success. Frame transmitters only send frames when the receiver has available buffer space to accept them. Communication of initial buffer capacity occurs during link initialization. Maintaining the current buffer counts and availability in each direction occurs through R_RDY primitive signals (credits).
When R_RDYs are corrupted or lost due to physical layer issues, the transmitter loses credits. Similarly, corrupted frames that are unrecognized by the receiver prevent R_RDY signals from being returned, which leads again to loss of credits. If either of the above happens repeatedly, all credits could be lost, halting frame transmission in one direction for a minimum of 1 second. During this 1-second interval, there could be thousands of frames dropped on the interface as well as significant congestion reflected back to the source of the frames, affecting many devices in the fabric. This can all be prevented by ensuring that BB_credit recovery is enabled and operational on all Fibre Channel interfaces.
Buffer-to-Buffer credit recovery
The INCITS T11 Fibre Channel standard (FC-FS-5) addresses these issues with BB_ credit recovery, a mechanism that restores lost credits. The local port and peer port agree to send checkpoint primitives to each other for both frames and R_RDYs, starting from the time the link comes up:
● During link initialization, both sides exchange BB_credit recovery support through FLOGI (F ports) or ELP (E ports), agreeing on a BB_ SC_N (buffer-to-buffer state change number) as the highest number offered by both ends. This is a small number used as an exponent of base 2. For example, if the number is 14 then it is 214 = 16,384. We will call this the BB_SC_N (value).
● After every BB_SC_N (value) frames, a frame transmitter sends a BB_SCs primitive to the receiver. If fewer frames were received than expected, the receiver sends the owed R_ RDYs back. This recovers the lost credits.
● Similarly, after every BB_SC_N (value) R_RDYs, a frame receiver sends a BB_SCr primitive to the transmitter. If fewer R_RDYs were received, the transmitter updates its count. This recovers the lost credits.
These primitives ensure that lost credits are recovered, allowing the link to maintain optimal performance.
Cisco MDS 9000 Series Multilayer Switches implementation
Cisco MDS 9000 Series Multilayer Switches have now enabled this important feature on all interface types. BB_credit recovery has long been supported on Fibre Channel ISLs (E ports). With Cisco NX-OS 8.2(1) release, Cisco added support for F ports, and with Cisco NX-OS 8.4(1) release for NP ports (uplink ports while operating in NPV mode). Moreover, since NX-OS 8.4(1) the BB_credit recovery feature is enabled by default on all MDS 9000 port types, alleviating configuration burden and favoring adoption.
Both E ports and NP ports enable recovery automatically, but F ports may require configuration for HBAs (initiators) and storage arrays (targets). Experience shows that old versions of HBAs (up to 16G FC) are not capable of BB_credit recovery. All current generation HBAs (initiators) and storage arrays (targets) are capable, but the feature might be disabled. Considering the importance of BB_ credit recovery to achieve long-term smooth operations, Cisco is constantly monitoring field installations when customers contact Cisco® Technical Assistance Center (Cisco TAC) for support. A quick check of one customer’s Cisco MDS 9710 Multilayer Director showed 114 F ports with BB_credit recovery not functional. This is approximately 30 percent of the F ports on that switch. Cisco recommends adjusting the HBA configuration accordingly or migrating to a newer HBA version. Consult the individual HBA configuration guide for information on support and enablement of BB_credit recovery.
Verify buffer-to-buffer credit recovery with Marvell QLogic Fibre Channel HBAs
As an example, to verify if BB_credit recovery is enabled on Marvell Qlogic Fibre Channel HBAs or to change the settings, administrators can connect to the QConvergeConsole CLI.
● On the Main Menu: select Option #2 -- Adapter Configuration
◦ Select the HBA port to configure
● On the Sub Menu: select Option # 13 -- Buffer – Buffer Credits
◦ Option # 1 = Display Settings
◦ Option # 2 = Enable
◦ Option # 3 = Disable
● Make the change to other ports if desired.
● Exit and reboot
Verify buffer-to-buffer credit recovery with Cisco MDS 9000 Multilayer Series Switches
As already said, BB_credit recovery is enabled by default on all Cisco MDS 9000 switch ports since Cisco NX-OS 8.4(1) release. To verify:
● Show interface: Look for the BB State Change line:
MDS9710# show interface fc1/5
fc1/5 is up
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
…
Transmit B2B Credit is 16
Receive B2B Credit is 32
B2B State Change: Admin (on), Oper (up), Negotiated Value (14)
◦ Admin (on): BB_credit recovery is enabled in the MDS 9000 switch.
◦ Oper (up): Adjacent device also supports BB_credit recovery, and BB_credit recovery is operational.
◦ Oper (down): Adjacent device does not support BB_credit recovery, and BB_credit recovery is not operational
◦ Negotiated Value (for example, 14): reflects the BB_SC_N value (e.g., 2¹⁴ = 16,384).
● To check recovery activity, use the show interface counters detailed command. Example output:
MDS9710# show interface fc16/1 counters detailed
fc16/1
…
Link Stats:
BB_SCs credit resend actions: 8
BB_SCr Tx credit increment actions: 8
For detailed timestamps of when each counter incremented, use the show logging onboard interrupt-stats command. Example:
Output of the show logging onboard interrupt-stats command
Notes:
IP_FCMAC_INTR_ERR_BB_SCS_RESEND indicates that a BB_SCs primitive was received and that the count of frames received did not match the BB_SC_N (value). The owed R_RDYs are sent back to recover the lost credits. The count does not indicate the actual number of frames lost. It simply indicates that frame loss did occur and that credits were recovered.
IP_FCMAC_INTR_ERR_BB_SCR_INCREMENT (not shown in the example above) indicates that a BB_SCr primitive was received and that the count of R_RDYs(credits) received did not match the BB_SC_N (value). The count of Tx credits is incremented to recover the lost credits. The count does not indicate the actual number of R_RDYs that were lost; it simply indicates that R_RDY loss did occur and that credits were recovered.
Conclusion
BB_credit recovery is automatically enabled on all Cisco MDS 9000 switch ports, thereby ensuring resiliency against physical-layer errors. For F ports showing Oper (down), adjust the HBA configuration. With the BB_credit recovery feature enabled, FC links become more robust, maintaining full credit recovery and optimal performance.
“BB_credit recovery is automatically enabled on all Cisco MDS 9000 switch ports, ensuring resiliency against physical-layer errors.”
“Enabling BB_credit recovery on Fibre Channel links improves resiliency, enhances application performance, and reduces troubleshooting efforts for both traditional and AI workloads.”
BB_credit recovery is automatically enabled on all Cisco MDS 9000 switch ports, ensuring resiliency against physical-layer errors. For F ports showing Oper (down), adjust the HBA configuration. With the BB_credit recovery feature enabled, FC links become more robust, maintaining full credit recovery and optimal performance.
Financing to help you achieve your objectives
Cisco Capital® can help you acquire the technology you need to achieve your objectives and stay competitive. We can help you reduce CapEx. Accelerate your growth. Optimize your investment dollars and ROI. Cisco Capital financing gives you flexibility in acquiring hardware, software, services, and complementary third-party equipment. And there’s just one predictable payment. Cisco Capital is available in more than 100 countries. Learn more.
Learn more about Cisco MDS 9000 Series Multilayer Switches at: https://www.cisco.com/site/us/en/products/networking/cloud-networking-switches/storage-area-networking/mds-9000-series-multilayer-switches/index.html.