Solution Overview
Summary
Enterprises appreciate the cost and administrative advantages of consolidating storage platforms and deploying storage area networks (SANs). Evolving business requirements, however, underscore the need for high-density, high-speed networks that enable data center scalability and improve manageability while controlling IT costs. This white paper describes how Cisco Systems® can help customers protect their investment in mission-critical application services by meeting their storage networking and transport requirements in a way that redefines the data center and supports disaster recovery and business continuity.
Introduction
Early SANs were generally configured for the storage needs of a particular application or department. But as the number of applications and departments grew, so did the requirements for storage. Customers understand that such unplanned growth in SANs is no longer viable and appreciate the benefits that come from data center consolidation. Now the enterprise faces still more challenges:
• Rapidly and proactively respond to changes in business conditions
• Reduce risk by implementing a disaster recovery and business continuance plan
• Assure that application services meet availability and performance expectations
• Provision, utilize, and administer data center resources more efficiently to control costs
• Improve information management by classifying, tiering, archiving, and discarding data according to policies
Cisco® can help the enterprise meet these goals with the Cisco MDS 9000 Family of storage products for next-generation, scalable SANs and the Cisco ONS 15000 Series optical platforms for transporting high-density, high-bandwidth, and low-latency SAN protocols. These products comprise the Cisco Storage and Optical solution, which facilitates data center interconnection and SAN extension. The solution offers a choice of transport technologies for greater geographic flexibility and network interoperability between data centers. With this solution, customers have a wide range of options for disaster recovery, business continuance, and compliance with government regulations.
Cisco offers significant advantages in terms of security, scalability, and manageability of the SAN and transport network. The virtual SAN (VSAN) feature of the Cisco MDS 9000 storage products allows consolidating services onto fewer wavelengths. The high per-card density offered by the Cisco MDS 9000 Family and the buffer-credit management capabilities implemented in both the Cisco MDS 9000 and Cisco ONS product portfolios lowers the per-bit cost of transporting storage protocols and other services over varying distances. Cisco integrated management systems provide a consolidated view of storage and optical networks, thus simplifying operational and management functions. With the Cisco Storage and Optical solution, Cisco offers a fully integrated and feature-rich multiprotocol solution to customers today.
SAN extension is only one of several applications supported by the Cisco Storage Optical solution. Geo-clustering, for example, is used in the mainframe environment for critical business applications because it helps assure instantaneous failover to a backup data center with no loss of data. IBM's Geographically Dispersed Parallel Sysplex (GDPS) solution synchronizes mainframe computing resources and, in combination with Cisco Multinode Load Balancing capability, load balances across computing resources.
Choices in SAN Technologies
Customers have a choice of SAN technologies for the data center, the most popular SAN protocols being Fibre Channel and IBM Enterprise Systems Connection (ESCON)/Fiber Connectivity (FICON). ESCON operates at 200 Mbps over a limited distance (a few kilometers). FICON operates at 2 Gbps, typically over a distance of about 10 kilometers (km) or, with repeaters, as much as 100 km. It can, however, go much longer distances depending upon what transport technology is used.
Fibre Channel, a nonproprietary protocol, offers the widest choice of bandwidth. It supports speeds of 1, 2, 4, and 10 Gbps for distances up to 10 km but can be extended far beyond that. The three topologies supported by Fibre Channel are:
• Point-to-point is the most common.
• Arbitrated Loop requires the transmitting node to gain control of the loop to establish a virtual point-to-point connection with the node with which it wants to communicate. The communicating nodes have full use of the bandwidth until the transfer is complete.
• Fabric switches and directors perform much like traditional network switches, providing increased bandwidth, scalability, performance, security, and redundancy.
Cisco Storage and Optical: an End-to-End Solution
The Cisco MDS 9000 Family of storage products provides industry-leading port densities to allow for data center growth. The Cisco MDS 9000 products deliver robust and flexible hardware architecture, with multiple layers of network and storage intelligence for building highly available and scalable SANs with comprehensive security and management. As a result, customers can deploy large networks in the data center with confidence and, at the same time, lower their total cost of ownership.
The Cisco MDS 9513 Multilayer Director offers from 12 to 528 ports to meet the most demanding connectivity needs of large application server farms. In addition to providing 1- and 2-Gbps Fibre Channel interfaces, the Cisco MDS 9513 also supports Fibre Channel at 4 Gbps and 10 Gbps, rates that also will be supported on other Cisco MDS 9500 Series products for backwards compatibility.
In addition to these capabilities, two Cisco MDS 9000 technologies - Virtual SAN (VSAN) and Inter-VSAN Routing (IVR) - facilitate building very large, yet stable, SAN fabrics. VSAN maps previously isolated SANs onto a common physical infrastructure, in effect collapsing the switch fabric while maintaining per-VSAN management and control of each virtual fabric. IVR takes SAN scalability and manageability to the next level by creating a path for a device in one VSAN to communicate with devices on other VSANs without sacrificing fabric isolation. The addition of 10-Gbps Fibre Channel support means that multiple VSANs operating across 1-, 2-, and 4-Gbps Fibre Channel networks can be aggregated onto a 10-Gbps Fibre Channel connection for cost-effective transport over varying distances using Cisco optical solutions (see Figure 1).
Figure 1
DWDM Transport for Aggregated 10-Gbps Fibre Channel
The Cisco ONS 15454 Multiservice Transport Platform (MSTP) provides capital and operational efficiency by addressing the increasing demand for multiple services, greater transport capacity, networking flexibility, multiple distance options, and management simplicity in a single platform. In a single network element, this next-generation multiservice platform integrates transparent wavelength and SONET/SDH transport as well as switching capabilities. The Cisco ONS 15454 MSTP supports up to 64 2.5- or 10-Gbps protected wavelengths, giving the service provider or the enterprise a high-bandwidth, high-density platform for aggregating and transporting time-division multiplexing (TDM)-based services (T1/T3, OC-3/STM-1, OC-48/STM-16, etc.) as well as data services (Ethernet/IP, FICON/ESCON, Fibre Channel). Data cards for the Cisco ONS 15454 MSTP allow multiple SAN or Gigabit Ethernet signals to be carried over a single wavelength.
The Cisco ONS 15454 MSTP's dynamic wavelength services are enabled by reconfigurable optical add/drop multiplexer (ROADM) technology. ROADMs make it much easier and quicker to deploy wavelengths where needed because optical capacity can be added or dropped remotely at any node without impacting existing traffic. Network upgrades are done without service disruption and the cost of onsite maintenance is reduced. Cisco ROADM technology helps to meet quickly changing traffic demands in the SAN environment while supporting a choice of network topologies (such as point-to-point, ring, and mesh).
The Cisco Storage and Optical solution provides a way to easily upgrade SANs to 4- and 10-Gbps Fibre Channel, delivering scalable, dynamic optical transport for SAN extension and a host of other applications. The end-to-end Cisco solution consists of the Cisco MDS 9513 and Cisco ONS 15454 MSTP. The Cisco MDS 9513 aggregates 1-, 2-, and 4-Gbps Fibre Channel connections into a 10-Gbps Fibre Channel signal. The 10-Gbps Fibre Channel can then be transparently transported via the Cisco 10-Gbps Multirate Transponder Card over an ITU-compliant wavelength provided by the Cisco ONS 15454 MSTP (see Figure 2).
Figure 2
Easy SAN Scalability and Transport
Protecting and Accessing Mission-Critical Data
Customers want cost-effective storage solutions for growing server farms and they also want to protect and assure 24-hour access to a variety of mission-critical applications, such as enterprise resource planning, supply chain management, customer relationship management, etc. Given the increased reliance on an ever-increasing number of application services in today's competitive market, customers must have an effective strategy for disaster recovery and business continuity. Mission-critical information must be protected against natural disasters, human error, and malicious acts.
Although some businesses may choose to avoid or postpone their protection plans, governments worldwide have mandated disaster recovery and business continuity regulations for important sectors of the economy. In the United States, for example, the government requires the financial sector to resume operations the same business day; and the Health Insurance Portability and Accountability Act of 1996 (HIPAA) mandates backup of patient information. In some cases, regulations can be very stringent: financial sector businesses are required to locate the backup data center at least 200 miles from the primary data center and have the facilities on separate power grids. The Securities and Exchange Commission requires archiving of e-mail at an offsite facility. In Europe, the Basel II Accord encompasses disaster recovery and business continuance rules for international banking.
A backup data center, which is central to any disaster recovery and business continuity plan, can operate in a hot or cold standby mode for nondisruptive or delayed access to application services. Either model, however, requires the secondary site to have a copy of the data from the primary data center. To accomplish this, the SAN at the primary facility must extend to the backup site.
Transport Options for Fibre Channel Extension
Customers have several choices for metro, cross-country, or transcontinental Fibre Channel-based SAN extension.
Fibre Channel Over Dark Fiber
Fibre Channel interfaces can be directly connected with dark fiber, but the distance the Fibre Channel network can operate will be limited by physical parameters such as the power of the transmitter, optical signal-to-noise ratio (OSNR), sensitivity of the receiver, and bit error rate (BER) of the system. With a standard Fibre Channel gigabit interface converter (GBIC), the maximum reach of Fibre Channel is 10 kilometers (km) or, if extended GBIC is used, 80 km.
Fibre Channel Over IP/Ethernet
Fibre Channel over IP (FCIP) enables the Fibre Channel signal to be extended across great distances and allows the enterprise to take advantage of high-bandwidth Metro Ethernet services for point-to-point or multipoint-to-point connectivity between geographically dispersed locations. Many enterprises will favor Metro Ethernet because of their long years of comfort with this technology in corporate LANs. In addition to offering economical bandwidth scalable to 10 Gbps, Metro Ethernet services enable data center interconnection even when these facilities are well outside metro or regional boundaries, because a Fibre Channel signal can be transported over an IP (FCIP) network. Service providers can use Metro Ethernet to offer a low-cost interface for SAN transport and also to provide other revenue-generating services such as virtual LAN (VLAN), Layer 2 and Layer 3 VPNs, and high-speed Internet access.
Fibre Channel Over SONET/SDH
SONET/SDH services are widely available and allow Fibre Channel traffic, TDM, and Ethernet traffic to coexist over a common transport infrastructure. SONET/SDH technology's low latency, scalable bandwidth, and high reliability can be used to meet the needs of any application. The Cisco ONS 15454 Multiservice Provisioning Platform (MSPP) provides a single element for delivering private line (such as T1/T3) and high-speed optical (OC-n/STM-n) services, as well as data services such as Ethernet, Fibre Channel, and FICON, all of which can take advantage of a common SONET/SDH infrastructure.
The Cisco ONS 15454 MSPP's capability to economically transport data services has been enhanced by the insertion of technologies such as virtual concatenation (VCAT), which dynamically right-sizes an optical channel for data traffic, and link capacity adjustment scheme (LCAS), which adjusts channel capacity to meet changing service demands.
Fibre Channel Over DWDM
Although more costly than the other transport options, the high bandwidth and low latency of DWDM networks make wavelength services an ideal choice for Fibre Channel transport within the metro area. The primary benefits of DWDM systems are that they transparently transport any type of traffic (SAN, Ethernet, SONET/SDH, etc.) and greatly increase the capacity of the fiber by transmitting multiple channels or wavelengths on a single fiber strand.
In addition to the DWDM-based Cisco ONS 15454 MSTP, the Cisco optical product portfolio includes several other metro DWDM platforms, such as the Cisco ONS 15216, ONS 15530, and ONS 15540. These systems offer increased capacity (more services, such as Gigabit Ethernet and Fibre Channel/FICON on a fiber pair), high-density service aggregation (many services on a single wavelength), scalability from 2.5- to 10-Gbps wavelengths, service mixing (different services on the same wavelength), and support of different network topologies.
Distance Constraints on Fibre Channel Transmission
Fibre Channel's capacity and deterministic behavior make it an ideal protocol for mission-critical applications. To avoid loss of frames during transmission, the protocol uses a flow control mechanism by which the local Fibre Channel port informs the remote Fibre Channel port that it cannot receive any more frames. After the flow control request is received, the transmitting node throttles back to a rate acceptable to the receiver.
Fibre Channel uses buffer-to-buffer flow control between link partners connected directly to each other. A buffer-to-buffer credit system is implemented by the local port knowing exactly how many buffers the far-end port has before the local port communicates with the remote port. This is done during the login process by the remote node informing the local node logging in of the number of receive buffers it has for all the frames sent by the local node. For each frame the remote port receives, it returns an R_RDY acknowledgement frame to the local port to indicate that one of the receive buffers is now free. The local port then increments its available credits by one for each R_RDY frame it receives. At no time can the number of outstanding frames exceed the buffer capacity of the receiving node.
But as distances increase, so does the time it takes the transmitting node to receive an R_RDY frame from the far node because of signal propagation delays. For this reason, full utilization of the available bandwidth is not possible beyond the distance supported by the buffer credit limitation. The standard practice for a 1-Gbps Fibre Channel link is to allow 1 buffer credit for each 2 km of distance. Figure 3 shows how the buffer credit mechanism works over distance.
Figure 3
Buffer-to-Buffer Flow Control and Impact Due to Distance
In addition to Fibre Channel's physical constraints (the previously mentioned OSNR, BER, etc.) and protocol constraints, how data is replicated between primary and secondary data centers also impacts the allowable maximum distance between facilities. In synchronous mirroring, data is copied, validated, and committed at the same time at both the primary and secondary data centers. This means each write (input) sequence receives an acknowledgment from the remote site for the operation to be complete. A write operation can take up to several microseconds depending upon distance and remote disk processing speed. So while synchronous mirroring assures the maximum degree of data protection, it is sensitive to large roundtrip delays. Asynchronous mirroring, however, is generally not sensitive to roundtrip delays. This is because inputs at the primary and secondary sites occur independently of each other, making distance much less of a factor in asynchronous replication. In short, synchronous replication is real-time data mirroring and asynchronous replication is near-real-time data mirroring.
Storage array vendors have specific requirements regarding distances that can be supported in their environment and it is imperative to understand the delay sensitivity of an application before designing a Fibre Channel transport network.
Solutions for Overcoming Fibre Channel Protocol Constraints
There are two techniques for overcoming Fibre Channel protocol limitations to extend the SAN over longer distances.
• Large buffer-to-buffer credits - As previously discussed, Fibre Channel's flow control mechanism is sensitive to roundtrip delays that occur when an acknowledgment from the remote node is slow in being returned to the transmitting port because of the distance. This impacts the source port's ability to transmit frames, thus reducing effective throughput, a phenomenon called drooping. A Fibre Channel system designed to support a large number of buffer-to-buffer credits can overcome Fibre Channel's flow control constraints because such a system would continue to send frames before waiting to receive acknowledgments. The Cisco MDS 9000 Family of products supports up to 6000 buffer-to-buffer credits per interface card or 4095 buffer-to-buffer credits per SAN extension port, allowing for a maximum distance of 8000 km for 1-Gbps Fibre Channel.
• Spoofing of acknowledgments - Using this technique, the optical gear itself creates R-RDY acknowledgments instead of waiting for the SAN gear at the remote end to generate them. This is done by the buffering capability of the optical transport node to store Fibre Channel frames that are "in flight" in the event the remote SAN gear cannot accept them because of congestion. Cisco DWDM and SONET/SDH solutions include spoofing and so can spoof the buffer-to-buffer client credit by using a proprietary exchange of memory information between Cisco data muxponder cards. This allows client equipment to run at full rate across hundreds of kilometers, without data loss, while interoperating transparently with the spoofed solution.
Neither solution can totally eliminate the impact of propagation delay because higher-level applications, such as EMC's SRDF software suite of remote storage replication solutions, still need to respond both before and after Fibre Channel data transfer.
Next-Generation SAN Extension with Cisco Optical Solutions
The Cisco ONS 15454 can function as an MSTP or an MSPP, which makes it a highly flexible platform for transporting SAN as well as other data and TDM traffic.
High-Availability Design Using PortChanneling
The Cisco ONS 15454 MSTP uses ROADM technology to deliver up to 64 2.5-Gbps or 10-Gbps wavelengths. The Cisco 2.5-Gbps data muxponder card, which is a multiservice aggregation card, has eight client interfaces for transporting Fibre Channel, FICON, ESCON, and Gigabit Ethernet services over a single 2.5-Gbps wavelength. Another higher-speed card, the 10-Gbps muxponder card, aggregates Fibre Channel, FICON, and Gigabit Ethernet services over a single 10-Gbps wavelength. These cards support spoofing and buffer-to-buffer credit: Fibre Channel traffic can be transported with buffer-to-buffer credit, up to 1600 or 800 km at 1- and 2-Gbps Fibre Channel, respectively.
The Cisco MDS 9000 capability to aggregate Fibre Channel signals allows for full use of wavelength capacity. The Cisco MDS 9000 can aggregate 1-, 2-, and 4-Gbps Fibre Channel services into a 10-Gbps Fibre Channel signal. This aggregation, combined with the Cisco 10-Gbps multirate transponder card, allows 10-Gbps Fibre Channel as well as other 10-Gbps services (such as Ethernet), to be transported over 10-Gbps wavelengths. Figure 4 shows the Cisco MDS 9000 PortChannel feature being used to bundle redundant Extended Interswitch Links (EISLs) for transport across a metro DWDM network. Multiple VSANs can use the same EISL and, with PortChanneling, up to 16 links between Cisco MDS 9000 switches can be bundled. PortChannel member ports are selected from different Cisco MDS 9000 line cards for increased availability.
Figure 4
PortChanneling for High SAN Availability
SAN Transport Over SONET/SDH
As an MSPP, the Cisco ONS 15454 is a next-generation SONET/SDH platform that supports SAN as well as other data and traditional TDM services over the SONET/SDH infrastructure. The Cisco SL-Series card gives the Cisco MDS 9000 Family of products a direct Fibre Channel connection to the Cisco ONS 15454 MSPP. The card's spoofing feature allows 1- and 2-Gbps Fibre Channel signals to be transported up to 2800 and 1400 km, respectively, independent of the buffer-to-buffer credit capabilities of the Cisco MDS 9000. Figures 5 and 6 illustrate SAN extension over SONET/SDH and DWDM networks.
Figure 5
SAN Extension with the Cisco ONS 15454 MSPP
Figure 6
SAN Extension with the Cisco ONS 15454 MSTP
Going Global With FCIP
FCIP offers the most reach for SAN extension, requiring only the availability of an IP network with the required bandwidth for transport in campus, metro, regional, or global application environments. The Cisco MDS 9000 14/2-Port Multiprotocol Services Module and the Cisco MDS 9000 4-Port and 8-Port IP Storage Services Modules support FCIP across a distance of 22,000 km for 1-Gbps Fibre Channel.
Figure 7
SAN Extension with FCIP
Integrated Management with Cisco Optical and Storage Solution
As the storage network grows, the need for comprehensive management services becomes more apparent. Cisco offers integrated management of the Cisco MDS 9000 Family and Cisco ONS 15000 Series to lower both the capital and operational expenditures (CapEx and OpEx) of storage networks and optical transport.
Cisco Transport Manager, an integrated element management system for the Cisco ONS 15000 Series optical transport systems, simplifies management and accelerates deployment of the Cisco ONS 15454. Cisco Transport Manager has advanced capabilities for configuration, fault, performance, and security management of Cisco optical network elements, networks, and subnetworks. With inherent support for SONET/SDH, DWDM, Fibre Channel, FICON, ESCON, and Ethernet, open interfaces to operations support systems (OSSs), and a proven record of reliability and scalability, Cisco Transport Manager delivers the full power of the wide range of advanced Cisco optical systems.
The Cisco Fabric Manager is an element and fabric manager for the Cisco MDS 9000 Family. Element management allows the network administrator to configure the switch and track statistics and events. The Cisco Fabric Manager provides the capability to manage the fabric as a collection or network of devices. The fabric manager application is built upon a topology representation of the fabric. Once Cisco Fabric Manager is invoked, a topology discovery process begins. Using information polled from a seed Cisco MDS 9000 Family switch, including NameServer registrations, and from a Fibre Channel fabric configuration server, Cisco Fabric Manager can re-create a fabric topology and represent it for the user in a customizable map.
Both Cisco Transport Manager and Fabric Manager can cross-launch each other. Cisco Transport Manager can launch Cisco Fabric Manager and display Cisco MDS 9000 nodes, and Cisco Fabric Manager can cross-launch Cisco Transport Manager. (Future releases will support link- and domain-level inventory.) Figure 8 shows a single view of Cisco ONS 15000 and Cisco MDS 9000 nodes, using Cisco Transport Manager.
Figure 8
Integrated Storage and Transport Management - Consolidated View of Storage and Optical Network
Cisco Storage and Optical Solution Differentiators
Table 1. Cisco Storage and Optical Differentiation
Summary
Consolidating data centers and supporting an increasing number of application services has become essential for businesses to remain competitive. The Cisco MDS 9000 Family of storage switches help consolidation of SANs by offering unprecedented port density, support of multiple SAN protocols including 4- and 10-Gbps Fibre Channel, and the ability to create VSANs across a common physical infrastructure. Cisco ONS 15000 Series optical platforms provide several high-speed, all-distance transport solutions for Fibre Channel traffic, from IP, SONET/SDH, to DWDM. With Fibre Channel aggregation, customers can make optimal use of 10-Gbps wavelength services. Together, the Cisco MDS 9000 Family and Cisco ONS 15000 Series offer effective, easy-to-use storage and transport solutions for disaster recovery and business continuity, as well as other applications. Comprehensive and integrated management of the Cisco Storage and Optical solution, using Cisco Transport Manager and Fabric Manager, further reduces total cost of ownership for data center scalability.