The Cisco MDS 9000 platform is the first to offer mainframe IBM Fiber Connection (FICON) intermix with complete traffic and management isolation and advanced distance extension, quality of service, and scaling.
Over the past few years, Fibre Channel has become the dominant protocol for connecting servers to storage. In open-systems environments, Fibre Channel Protocol (FCP) is the upper-layer protocol for transporting Small Computer System Interface 3 (SCSI-3) over Fibre Channel transports. In 1998, IBM introduced IBM Fiber Connection (FICON) to replace the aging Enterprise Systems Connection (ESCON) architecture. Like FCP, FICON is an upper-layer protocol that uses the lower layers of Fibre Channel (FC-0 to FC-3). This common Fibre Channel transport enables mainframes and open systems to share a common network and I/O infrastructure commonly known as "intermix."
Intermix has several advantages: a reduced fiber plant and a reduced and common set of switches and directors.
Although this solution appears technically sound, it does present a few challenges. Two stand out:
• Management-Open-systems storage area networks (SANs) and mainframes are usually managed by separate teams with different skills and responsibilities. Traditional intermix solutions combine the management domains of both environments, allowing open-systems administrators access to mainframe channels. Mainframe operators will also have management access to open-systems SANs.
• Device and control unit exposure-In an intermix environment, open-systems nodes are exposed to mainframe channels and control units. Although zoning limits some of this exposure, it cannot keep errant devices from affecting mainframe channels and control units.
THE CISCO MDS 9000 INTERMIX SOLUTION
The Cisco® MDS 9000 product line delivers the benefits of FICON intermix without the compromises outlined above. A Cisco MDS 9000 intermix solution provides the following:
• Traffic separation-Virtual SANs (VSANs) allow customers to isolate mainframe channel traffic and events from open-systems traffic and events over the same physical infrastructure.
• Management-The Cisco MDS 9000 offers role-based access control (RBAC), limiting operators to custom sets of commands within defined VSANs. Customers can separate the mainframe operators from the open-systems administrators.
• Scaling-Customers can further scale their mainframe environment with as many as eight FICON VSANs from a total of 256 VSANs per Cisco MDS 9000. Cascaded switches can use up to 16 Fibre Channel-over-IP (FCIP) links in a logical PortChannel for interconnection.
• Advanced traffic management-Customers can reduce congestion and apply different service policies to FICON and open-systems traffic according to origin and destination per VSAN.
• Consolidated distance extension capabilities-Customers can transport open-systems and mainframe VSANs over the same Fibre Channel or FCIP distance extension infrastructure while optionally applying QoS policies to the different traffic types.
Traffic and Event Isolation Using VSANs
Traditional intermix solutions use zoning to isolate open-systems traffic from mainframe traffic. This solution is only partially successful. Mainframe channels and control units are still exposed to open-systems events within the fabric. Errant host bus adapters (HBAs) and targets, zone set changes, and fabric reconfigurations (intended or not), can all affect the mainframe channel environment.
Using Cisco MDS 9000 VSANs, all traffic and events are contained within the boundaries of the defined VSAN. Events occurring on an open-systems VSAN will not affect a mainframe VSAN, and vice versa. Errant devices cannot affect devices outside their own VSAN. Figure 1 illustrates two sample topologies: One shows a single director environment with a mainframe FICON VSAN and open-systems Fibre Channel VSAN. The other topology shows a cascaded FICON topology coexisting with two open-systems VSANs.
Figure 1. VSANs Isolate Open-Systems Traffic from Mainframe Traffic
Customers can assign any Cisco MDS 9000 interface to any VSAN, whether FICON or open system. Customers can assign new open-systems or mainframe equipment to "test" VSANs and confidently perform testing and system assurance without affecting production channels, machines, and applications. When the time comes to accept and put the new equipment into production, an authorized operator can simply assign that Cisco MDS 9000 interface to the appropriate production VSAN without a physical cable change.
Traditional FICON intermix solutions require customers to compromise their operational management policies, allowing open-systems administrators access to mainframe channels and mainframe operators access to interfaces and devices on open-systems SANs.
The Cisco MDS 9000 offers IBM Control Unit Port (CUP) protocol for in-band mainframe-based management and operation of the FICON VSAN. For out-of band management and operation, the Cisco MDS 9000 offers role-based management that limits operators to commands and VSANs that are within their span of control. For example, customers could define a "systems programmer" role, a "shift manager" role, and a "mainframe operator" for their mainframe environment and a "SAN-admin" and "SAN-operator" role for their open systems. They could define further roles to restrict operators to particular mainframes or open-systems applications based upon VSANs.
Customers need only define the roles once. Cisco Fabric Services will distribute the roles, rule sets, and policies to all switches and directors within the fabric. All users are authenticated against a central authentication, authorization, and accounting (AAA) server (RADIUS or TACACS+) or alternatively against an internal Cisco MDS 9000 database.
Scaling the Infrastructure
Each Cisco MDS 9000 can accommodate as many as eight FICON VSANs and 256 VSANs, allowing customers to create new FICON or open-systems VSANs on demand to accommodate growth.
FICON topologies with traffic traversing two switches or directors are called "cascaded" topologies. As many as 16 Fibre Channel or FCIP links can interconnect cascaded switches in a logical PortChannel. Open-systems VSANs can share the PortChannel or an individual link using a method called VSAN trunking. In this way, a VSAN trunk of just one Fibre Channel optical link or FCIP link can carry any number of VSANs of any type: mainframe FICON or open systems. In other words, customers can independently scale the interconnecting network from the number of implemented VSANs.
FICON addressing limits each director or switch to 256 addressable ports per domain. VSANs allow customers to implement a FICON/FICON intermix and logically divide high-density directors into smaller logical directors. This provides unlimited addressability to all director ports, regardless of switch density.
Advanced Traffic Management
The Cisco MDS 9000 offers several unique traffic management features for Fibre Channel and FCIP. Using these features, customers can minimize network congestion within their intermix environment and prioritize traffic.
The Fibre Channel features are as follows:
• Fibre Channel Congestion Control (FCC)-This Cisco proprietary flow control mechanism alleviates congestion in Fibre Channel (including FICON) networks. It uses an edge quench feedback mechanism to control the rate at which frames are admitted to the network when congestion occurs.
• Quality of service (QoS)-The Cisco MDS 9000 offers four classes of service for Fibre Channel traffic using a Differentiated Services (DiffServ) model. Control traffic uses the highest-priority queue. Users can customize the weighting of the remaining three queues, which are serviced according to a Deficit Weighted Round Robin (DWRR) mechanism. Users can select a service class based on the source and destination and apply policies for each VSAN.
Consolidated Distance Extension Capabilities
The Cisco MDS 9000 is the only single-chassis solution capable of both long-distance Fibre Channel and FCIP for cascading. Having sufficient buffer credits is critical to avoid "data droop" when cascading FICON or Fibre Channel over optical transports (coarse wavelength-division multiplexing [CWDM], dense wavelength-division multiplexing [DWDM], SONET, or SDH). Each port on the 16-port line card has 255 buffer credits, enabling 250 kilometers (km) at 2 Gbps or 500 km at 1 Gbps. The Cisco MDS 9000 14/2-Port Multiprotocol Services Module and Cisco MDS 9216i Multilayer Fabric Switch Fibre Channel ports boast 3500 buffer credits, enabling Fibre Channel over optical stretches of 3000 km at 2 Gbps without data droop.
FCIP capability is available on any of the Gigabit Ethernet interfaces in the Cisco MDS 9000 product family (Cisco MDS 9216i, the Cisco MDS 9000 4-Port IP Storage Services Module, the Cisco MDS 9000 8-Port IP Storage Services Module, and the Cisco MDS 9000 14/2-Port Multiprotocol Services Module). FCIP uses TCP for flow control, and so instead of buffer credits, a large maximum TCP window is required to avoid data droop. Each FCIP link on the Cisco MDS 9000 can allocate a window up to 32 MB, allowing distances between almost any two points on the globe without data droop at 1 Gbps.
For situations in which bandwidth is at a premium, three compression modes are available on all Gigabit Ethernet interfaces for FCIP. The Cisco MDS 9216i and Cisco MDS 9000 14/2-Port Multiprotocol Services Module support compression rates up to 1500 Mbps over each Gigabit Ethernet interface. For data confidentiality, integrity and authentication, the same interfaces can optionally employ IP Security (IPSsec) at wire rate. Each interface can use 128- or 256-bit Advanced Encryption Standard (AES), Data Encryption Standard (DES), or Tripe Data Encryption Standard (3DES) encryption.
QoS capabilities are critical in low-bandwidth cascaded situations to help ensure that high-priority applications get appropriate and adequate service. The Cisco MDS 9000 offers the following features and mechanisms for FCIP:
• QoS tagging-The Cisco MDS 9000 can tag control and data traffic for every FCIP link with a selectable differentiated services code point (DSCP) value between 0 and 63. QoS-aware WANs can then recognize and treat the tagged traffic according to enterprise QoS policies. For example, a customer might create two FICON VSANs-one for production and one for testing-with each mapped to a different FCIP link. The production FCIP traffic could be tagged at a higher DiffServ priority and treated more favorably if congestion occurs than the FCIP link carrying the test VSAN traffic.
• Advanced TCP capabilities-The Cisco MDS 9000 FCIP TCP stack is optimized for transporting storage traffic. It employs packet shaping to minimize the possibility of dropped packets and uses Selective Acknowledgement (SACK) and Fast Retransmit to recover quickly from congestion conditions. The Cisco MDS 9000 FCIP implementation also allows the customer to select a minimum and maximum bandwidth to avoid low throughput from TCP slow-start and congestion-avoidance mechanisms, as well as placing a ceiling on the maximum path bandwidth a given FCIP link will consume. In this way, customers can aggressively use dedicated bandwidth for FCIP or route them over shared IP WANs with a cap on bandwidth that each link can consume.