Cisco ONS 15500 Series

Fibre Channel Extension Over Metropolitan DWDM

  • Viewing Options

  • PDF (442.1 KB)
  • Feedback


Explosive growth in mission-critical business applications has resulted in an exponential growth in data-storage requirements. The need to protect and secure critical data and the growing size of that data are creating real challenges for enterprises as they seek affordable solutions to interconnect their storage area network (SAN) infrastructures and implement geographically dispersed data-recovery solutions. SAN-extension technologies such as dense wavelength division multiplexing (DWDM) help customers interconnect data centers to reduce the effects of a catastrophic failure at a single site and reduce the total cost of ownership. Inherently, the Fibre Channel protocol limits the throughput of the data stream for long-distance transmission, but buffer-to-buffer credit spoofing technology can effectively remove that limitation and allow data traffic to flow at maximum throughput even at great distances.

Buffer-to-buffer credit spoofing technology can effectively enhance data throughput when data is transmitted over long distances.


The tremendous growth of business applications is quickly heightening data-storage requirements and network expectations. Employees and customers demand uninterrupted access to corporate systems and data. Enterprises must respond to this surge in demand by providing robust, highly secure, interconnected SANs and geographically dispersed data-recovery solutions.
Remote data-replication solutions commonly include disk array-based technologies that replicate and synchronize an exact copy of an application's data to a remote disk array. Although data replication can take the form of synchronous or asynchronous replication, for a guaranteed exact copy of the data, synchronous replication is the predominantly used method. Synchronous replication ensures an exact copy of a data set is present at the remote site. Therefore network latency is a critical consideration when building a synchronous-replication solution and DWDM is the ideal technology for low latency, scalability, and resiliency.


Data distribution, data protection, and business-continuance strategies are critical components of today's information-centric businesses. Increasing the resiliency of important enterprise applications often involves extending the recovery capability of applications to remote data centers. SAN extension technologies help customers interconnect their data centers to reduce costs and mitigate the impact of a failure at a single site.


Cisco Systems ® offers several solutions for SAN extension, each with its own unique benefits and effectiveness. The application criticality and data-resiliency requirements would dictate which of the following solutions is the most suitable:

• Fibre Channel over DWDM-DWDM offers a cost-effective mechanism to increase network bandwidth, while minimizing the number of fibers required between locations. DWDM provides high-density aggregation over a high-speed, low-latency network that can support important protocols such as Fibre Channel, IBM Fiber Connection (FICON), Enterprise Systems Connection (ESCON), and the IBM Sysplex and Coupling Facility protocols.

• Fibre Channel over IP (FCIP)-The primary advantages of FCIP for remote connectivity is its ability to extend distances using TCP/IP and the wide availability of IP. With advanced quality of service (QoS) traffic engineering, FCIP can be an effective solution for SAN extension. However, one must also take into consideration the impact on performance due to the unreliable and unpredictable nature of TCP/IP in large networks.

Fibre Channel over SONET/SDH-SONET/SDH networks are readily accessible and are the primary optical backbone network for most service providers. Fibre Channel coexists with other time-division multiplexing (TDM) and Ethernet traffic over a SONET/SDH network. Bandwidth can be assigned per application requirements. A SONET/SDH network can deliver the low latency, high performance, and QoS required to extend Fibre Channel reliably over a large geographical area.


Figure 1 illustrates an overview of Cisco ® SAN extension solutions.

Figure 1

Cisco Storage Extension Solutions



This application note will focus on the limitations and requirements of SAN extension as well as how the Cisco ONS 15530 DWDM Multiservice Aggregation Platform effectively solve these challenges.


Each layer of the Fibre Channel architecture imposes specific physical, protocol, and application limitations that inhibit the normal operation of Fibre Channel beyond certain distances.

Physical Limitations

The maximum reach of a Fibre Channel link depends on the physics of light signals traveling in the fiber. The physical parameters that determine the effective reach include transmitter power of the laser, receiver sensitivity to decode the laser signal, dispersion shift, and the optical signal-to-noise ratio. Some of these physical limitations can be overcome by using optical amplifiers (erbium-doped fiber amplifiers) to boost the signal strength and dispersion compensation units (DCUs) to correct the dispersion shift.

Protocol Limitations

The problem with transparently extending a Fibre Channel SAN over a long distance arises from the nature of the flow-control mechanism employed by Fibre Channel and its potential effect on an application's I/O performance. The Fibre Channel protocol uses a credit-based algorithm to allow or throttle frame flow based on the capacity of a receiving device's input buffer. To ensure input buffers are not overrun and frames are not dropped, each ingress Fibre Channel port gives out credits called buffer-to-buffer credits to permit a transmitting device to send frames. During initial login process, the number of credits that a device interface can support is exchanged with the neighbor port. This number determines the number of consecutive frames that may be transmitted by a neighboring device or switch port before it must receive acknowledgment that the earlier frames have been delivered successfully. In the example shown in Figure 2, the negotiated number of buffer credits is four. Additional credits can only be given when input buffers have been emptied on either end of the link. If the distance is large enough between the two communicating ports, the inherent transmission latency can effectively limit the maximum amount of sustainable bandwidth over the link due to credit starvation. Figure 2 shows how as distance increases, a set number of buffer credits cannot fill up the pipe.

Figure 2

Fibre Channel Flow Control



Application Limitations

Certain applications such as synchronous mirroring are sensitive to long round-trip delay between the local and remote node. In synchronous mode, data between the primary and secondary sites are copied, validated, and committed at the same time. Therefore synchronous applications require an acknowledgment for every sequence of "write" operation. This level of high reliability comes at the cost of the time required to complete the handshake, greatly reducing the throughput of the link.
The write operation can take up to several microseconds, depending on the distance and remote-disk processing speed. Hence, application performance and throughput degrade as distance increases because round-trip latency also increases.
Because of all the physical, protocol, and application limitations, interface throughput and latency must be addressed for a SAN extension solution. Advanced buffer-credit spoofing mechanisms can dramatically maximize the throughput of a Fibre Channel link over a DWDM network.


Benefits of DWDM

DWDM not only offers high-density aggregation over a high-speed, low-latency network of storage protocols including Fibre Channel, FICON, ESCON, and IBM Sysplex and Coupling Facilities, the technology can also be used to support network consolidation of multiple other protocols including data, voice, and video.

Figure 3

SAN Extension over DWDM



WDM technology offers significant return on investment by combining multiple services into a single pair of fiber. The scalability of WDM technology can be achieved in two ways. First, with each end-user service mapped to a specific wavelength, WDM combines all the different wavelengths onto the fiber pair. Second, instead of mapping each end-user service into a wavelength, one can first multiplex multiple client interfaces into the wavelength, achieving a multiplying effect in the amount of data a single pair of fiber can carry. While CWDM technology is limited in terms of the wavelengths per fiber and supported distances, DWDM offers significantly greater scalability and flexibility. DWDM technology typically provides the capability to multiplex 32 or more services onto a fiber. Additionally, with the use of optical amplifiers and dispersion compensation units (DCU), the reach of a DWDM network can be up to several hundred kilometers.
DWDM is a transparent, low-latency technology. Because the raw data stream is passed through the network transparently, the latency of the signal is limited to the Optical-Electrical-Optical conversion and the propagation delay of the light signal as it travels through the fiber. This extremely low and very consistent latency is an important requirement in any synchronous data-replication scheme. Furthermore, because of the transparency of DWDM, it is less complicated to provision and manage as compared to traditional storage extension methods over ATM or SONET/SDH.
Additionally, a complete DWDM network offers the manageability of Layer 1 transport. This manageability includes robust protection schemes, internode communication, and an overall network management and provisioning system. All these capabilities work in synchronization to help ensure a robust and resilient Layer 1 transport network.
Finally, customers can use the DWDM infrastructure for more than SAN extension; the transparency of DWDM allows any other protocols to be multiplexed into the same fiber pair. This added flexibility gives the customers the ability to consolidate and scale their data, voice, video, storage, and SONET/SDH infrastructures.


Service aggregation can significantly improve the wavelength efficiency in Fibre Channel extension. Because the availability of fiber is limited, it becomes very cost effective to maximize the use of each wavelength in the fiber. Service aggregation refers to the technology that combines several 1-G Fibre Channels or FICON links into higher-bandwidth data streams through the intelligence in the electronics. Commonly, the optics support up to 2.5-Gbps and 10-Gbps rates today, and most service-aggregation technology will maximize the service efficiency into these aggregated rates.
The Cisco ONS 15530 8-Port Fibre Channel/Gigabit Ethernet Aggregation Card supports service aggregation of up to two 1-G Fibre Channels into a 2.5-Gbps data stream, or up to eight 1-G Fibre Channels into a 10-Gbps data stream. The implementation maximizes the efficiency of the fiber, providing up to 256 Fibre Channel services in a single pair of fiber. Additionally, the ability to aggregate two 1-G Fibre Channels into a 2.5-Gbps data stream creates unparallel flexibility in network design. Furthermore, Cisco ONS 15530 platform has the capability to simultaneously combine Fibre Channel, FICON, Gigabit Ethernet, and ESCON traffic into a single pair of fiber.


Buffer Credit Spoofing-Based Flow Control

As noted above, native Fibre Channel protocol presents some challenges for operating over extended distances, including the negative impact on throughput due to strict buffer-to-buffer credit flow-control mechanism to guarantee data delivery. One solution to this problem is buffer credit spoofing technology. Spoofing is a technique used to transport Fibre Channel over extended distances beyond the distance for buffer credits negotiated by the end nodes. The R_RDY signal, normally used by the Fibre Channel switch nodes to communicate a frame has been received, is intercepted and simulated by the DWDM node for every frame the Fibre Channel switch transmits. This technology is becoming more widely adopted as Fibre Channel Extension becomes an important requirement, whether the underlying transport technology is WDM, SONET/SDH, or FCIP.
The Cisco ONS 15530 8-port card provides the capacity to extend Fibre Channel over great distances with the capability to spoof the R_RDY signals from the end Fibre Channel switches. Figures 4-7 explain how the card carries out Fibre Channel spoofing to maximize bandwidth utilization and protect the data stream integrity. In Figure 4, the blue squares represent the Fibre Channel data frames, and the red squares are corresponding R_RDY acknowledgment signals. The implementation is bidirectional for any Fibre Channel link. For simplicity, Figure 4 only illustrates unidirectional operation.

Figure 4

Buffer Credit Spoofing Setup and Transmit



Step 1: Through the login process, the Cisco MDS 9000 Series multilayer switches #1 and #2 negotiate the number of credits. In this example, four credits are used. This translates to four credits between Cisco MDS 9000 Series #1 and Cisco ONS 15530 platform #1 as well as between Cisco MDS 9000 Series #2 and Cisco ONS 15530 #2. Cisco MDS 9000 Series #1 can send up to four data frames to Cisco ONS 15530 #1 before it must wait for R_RDY acknowledgment to continue.
Step 2: Cisco ONS 15530 #1 sends back four R_RDY messages to Cisco MDS 9000 Series #1 after it has received the first four data frames. Cisco MDS 9000 Series #1 acknowledges the incoming R_RDY from Cisco ONS 15530 #1 and continues to send additional data frames to the Cisco ONS 15530 #1.
Step 3: The 8-port Fibre Channel/Gigabit Ethernet aggregation module on Cisco ONS 15530 #2 contains the necessary memory storage that is used to store and forward the incoming frames from Cisco ONS 15530 #1. Additionally, there is continuous communication between Cisco ONS 15530 #1 and Cisco ONS 15530 #2 on the memory utilization status.

Figure 5

R_RDY Acknowledgment and Continuous Flow



Step 4: Cisco ONS 15530 #2 receives and forwards the data frames to Cisco MDS 9000 Series #2. After receiving the data frames, Cisco MDS 9000 Series #2 sends R_RDY acknowledgment back to Cisco ONS 15530 #2. Because Cisco ONS 15530 #2 has received R_RDY acknowledgment from Cisco MDS 9000 Series #2, it continues to forward additional data frames to Cisco MDS 9000 Series #2. IF R_RDY is not received from Cisco MDS 9000 Series #2, Cisco ONS 15530 #2 will immediately stop forwarding any additional data frames and store the data.

Buffer-Credit Spoofing-Data Integrity

Because of the continuous communication, Cisco ONS 15530 #1 is always aware of the memory utilization on Cisco ONS 15530 #2 and the number of transmitted frames already in the fiber path from Cisco ONS 15530 #1 to Cisco ONS 15530 #2. Based on this information, Cisco ONS 15530 #1 will control when and how many R_RDY signals it sends back to Cisco MDS 9000 Series #1 to maximize utilization of the link and protect the integrity of the data stream.
Figures 6 and 7 illustrate how flow control helps ensure data-flow sequence and prevents dropped frames when back pressure is applied from the receiving Fibre Channel switch.

Figure 6

No R_RDY Acknowledgment from Fibre Channel Switch


The first four data frames from Cisco MDS 9000 Series #1 are sent and received by Cisco MDS 9000 Series #2 through the DWDM infrastructure. Cisco MDS 9000 Series #2 is not ready to receive any additional data frames. It does not send back R_RDY acknowledgment to Cisco ONS 15530 #2. Because Cisco ONS 15530 #2 does not receive any R_RDY acknowledgment from Cisco MDS 9000 Series #2, the aggregation card stores all incoming data frames. Cisco ONS 15530 #2 communicates the memory utilization status and the lack of R_RDY back to Cisco ONS 15530 #1.

Figure 7

Stored Fibre Channel Frames



Knowing the memory utilization at Cisco ONS 15530 #2 is approaching full, Cisco ONS 15530 #1 stops sending R_RDY back to Cisco MDS 9000 Series #1. This effectively stops any additional transmission from Cisco MDS 9000 Series #1. Cisco ONS 15530 #2 continues to store all incoming frames that were already in transit from Cisco ONS 15530 #1 to Cisco ONS 15530 #2. When Cisco MDS 9000 Series #2 is ready to receive additional frames, it sends R_RDY acknowledgment back to Cisco ONS 15530 #2. Upon receiving R_RDY acknowledgment, Cisco ONS 15530 #2 sends the stored frames to Cisco MDS 9000 Series #2 in the original sequence. The memory utilization at Cisco ONS 15530 #2 begins to decrease and this information is communicated to Cisco ONS 15530 #1. Based on that, Cisco ONS 15530 #1 sends R_RDY to Cisco MDS 9000 Series #1 to reinitiate the Fibre Channel data traffic.
The R_RDY spoofing algorithm is implemented on Cisco ONS 15530 8-port module. This technology has been tested for interoperability with major storage vendors including Cisco MDS 9000 Series, Brocade Silkworm, McData 6000, InRange Fibre Channel, and Qlogic SanBox. In the lab, the algorithm has been shown to support up to 320 kilometers while maintaining full throughput of Fibre Channel at 1-G line rate.
Because end Fibre Channel switches interoperate transparently with the DWDM nodes, Cisco ONS 15530 nodes do not participate in the Fibre Channel login process and are not involved in the termination of Fibre Channel links. Therefore, Cisco ONS 15530 8-port module can support any point-to-point links. Whether it is E_port to E_port, F_port to N_port, or N_port to N_port, Cisco ONS 15530 will automatically detect the type of port attached and start the flow-control mechanism automatically.


There are many different technologies to implement Fibre Channel extension today. The best choice depends on various requirements including the criticality of user applications, the SAN implementation, and the availability of resources. DWDM offers the unique benefits of low and consistent latency, tremendous bandwidth scalability, and the ability to build a flexible infrastructure.
SAN extension, specifically Fibre Channel extension, has several technical challenges that must be overcome to ensure efficiency and scalability. Cisco ONS 15530 is an industry-leading metro DWDM platform that overcomes these challenges. The service aggregation capability and the R_RDY spoofing algorithm of the Cisco ONS 15530 allows the enterprises to build a scalable, efficient, and resilient business continuance strategy over a metro DWDM network today.