Storage Area Network Solutions

Fibre Channel over Ethernet Storage Networking Evolution

  • Viewing Options

  • PDF (447.8 KB)
  • Feedback


Evolving business requirements underscore the need for high-density, high-speed, and low-latency consolidated networks that enable data center scalability and improve manageability while controlling IT costs.

Data centers typically run multiple separate networks, including an Ethernet network for client-to-server and server-to-server communications and a Fibre Channel storage area network (SAN). To support various types of networks, data centers use separate redundant interface modules for each network: Ethernet network interface cards (NICs) and Fibre Channel interfaces in their servers, and redundant pairs of switches at each layer in the network architecture. Use of parallel infrastructures increases capital costs, makes data center management more difficult, and diminishes business flexibility.

Consolidation of I/O in the data center, allowing Fibre Channel and Ethernet networks to share a single, integrated infrastructure, helps enterprises address these challenges efficiently. An important pillar of this consolidated approach is Fibre Channel over Ethernet (FCoE).This document discusses the need for unified I/O and describes the phases of unified I/O along with FCoE deployment and the benefits of each phase.

Need for Unified I/O

As shown in Figure 1, a server deployed in a data center today has many discrete I/O interfaces to support specific application requirements:

Ethernet NICs (LAN): Used for client-to-server connectivity, server-to-server connectivity, and out-of-band management; typically 10/100/1000-Mbps Ethernet NICs

Fibre Channel host bus adapters (HBAs) (SAN): Used for server-to-storage connectivity and unique in their ability to combine lossless operation with the highest levels of network resiliency and availability in the data center; typically 1/2/4-Gbps Fibre Channel HBAs

Figure 1. Typical Server Connectivity in data center environments

Typical servers in data centers have five to seven I/O interfaces. A unified I/O adaptor that adequately supports the unique and varied traffic requirements of data center applications can reduce the number of network devices, server-network interfaces, and cables used to interconnect them. Unified I/O can also lead to a major reduction in data center power requirements; power is the most limited resource available to data center managers today.

Fibre Channel over Ethernet for Unified I/O

Fibre Channel over Ethernet (FCoE) is a protocol specification to encapsulate Fibre Channel frames in Ethernet packets to enable unified I/O connectivity on a server (Figure 2).

Figure 2. Encapsulation of Fibre Channel Frame in an Ethernet Packet

With FCoE technology, servers, instead of having multiple discrete I/O adaptors for LAN and SAN traffic, will have a smaller number of converged network adapters (CNAs) that support both LAN and Fibre Channel SAN traffic (Figure 3).

Figure 3. Server with Converged Network Adaptor supporting both SAN and LAN

The Ethernet network that supports FCoE is required to be a lossless Ethernet network, with switching devices that have internal architectures designed to offer a no-drop packet capability and network flow control mechanisms to enable lossless transmission of packets across the Ethernet infrastructure.

FCoE will provide several advantages over existing approaches to I/O consolidation:

Forward compatibility with existing Fibre Channel SANs by preserving well-known Fibre Channel concepts such as virtual SANs (VSANs), World Wide Names (WWNs), Fibre Channel IDs (FCIDs), and zoning to servers and storage arrays

A high level of performance, comparable to the performance of current Ethernet and Fibre Channel networks, achieved by using a hardware-based Ethernet network infrastructure that is not limited by the overhead of higher-layer TCP/IP protocols

The exceptional scalability of Ethernet at the highest available speeds (1, 10, and 40 Gigabit Ethernet and eventually 100 Gigabit Ethernet)

Gateway-less technology, avoiding the overhead of higher-layer TCP/IP protocols and simplifying operations and management (no change to the management infrastructure currently deployed in SANs)

FCoE Storage Networking Evolution

As with other networking technologies, FCoE-based storage networking will be evolutionary, with implementation in phases. The speed of migration will be determined by several factors, including progress on the standards and the availability of interoperable hardware and software drivers. As the standards are ratified and products become available, FCoE will likely be adopted in three phases:

1. FCoE server enablement

2. FCoE server proliferation

3. FCoE arrays and SANs

Phase 1: FCoE Server Enablement

The first phase of implementation of FCoE technology will enable FCoE at the network edge on standalone servers in the data center. Server vendors as well as HBA and NIC vendors will offer FCoE support in their data center products as an added value to Fibre Channel and Ethernet connectivity, respectively.

FCoE devices, including FCoE-capable servers, will be interconnected with top-of-rack (ToR) network switches capable of supporting unified I/O on servers. These ToR switches will provide access to existing Ethernet LANs and Fibre Channel SANs (Figure 4).

Figure 4. FCoE server enablement—Phase 1

The main benefits of phase 1 include the following:

Consolidation of server I/O and cables on a single interface whereas the traditional configuration included multiple Ethernet and Fibre Channel interfaces and cables

Significantly improved energy efficiency due to fewer interfaces and adapters, resulting in less power required per server

More affordable model for more servers to gain access to the SAN for the first time

Fibre Channel SAN investment protection: no changes to the SAN operating and management model; FCoE-enabled servers continue to access SAN-attached Fibre Channel storage

Phase 1 Implementation Using Cisco Nexus 5000 Series Switches and the Cisco MDS 9000 Family

Cisco® Nexus 5000 Series Switches, part of the Cisco Nexus family of data center–class switches, delivers unified fabric at the network access layer or edge, where servers connect to the LAN, SAN, and server clusters. The server connection is the best initial location for deploying a unified fabric in the data center because the rate of technology change is greater there than at the core, and the access layer or edge typically has many physical network interconnects to consolidate. To integrate with existing infrastructure, the Cisco Nexus 5000 Series provides native Fibre Channel uplinks to facilitate connection with installed SANs and available SAN switches. Other Cisco products, including the Cisco MDS 9000 family, will support FCoE in the future to extend consolidation beyond the access layer (Figure 5).

Figure 5. I/O consolidation using Cisco Nexus + MDS 9000 Family

Phase 2: FCoE Server Proliferation

In the second phase, FCoE technology will proliferate, moving to blade servers and to standalone servers attached to unified I/O–capable embedded blade switches and end-of-row (EoR) networking devices. Also, the existing Fibre Channel switches will evolve to support FCoE traffic from blade switches and EoR networking devices (Figure 6).

Figure 6. FCoE Server Proliferation—Phase 2

In addition to the benefits of phase 1, phase 2 will offer the benefits of agility and flexibility in enabling new applications and services in the data center, because every server in the data center will now be cost-effectively attached to the SAN and will have the flexibility to mount any Fibre Channel storage array in the SAN.

Phase 3: FCoE Arrays and SANs

In the third phase, storage arrays and tape libraries will support native FCoE interfaces (Figure 7).

Figure 7. FCoE Arrays and SANs—Phase 3

FCoE-enabled storage arrays and tape libraries will be attached to FCoE switches, creating an FCoE SAN that is natively accessible to FCoE-attached servers. As in phase 1, all native Fibre Channel equipment and new FCoE equipment will continue to be able to access existing Fibre Channel SANs because forward compatibility with native Fibre Channel is required of FCoE implementations.

With support for ubiquitous Ethernet extensions that provide a lossless Ethernet fabric, FCoE traffic will be carried with the same levels of reliability and guarantee of delivery as on Fibre Channel SANs today but over lossless Ethernet. This setup will enable the LAN and FCoE SAN traffic to converge onto a single, unified network fabric in the future. (Figure 8).

Nevertheless, most early implementations of FCoE SANs will likely be deployed in parallel with existing LANs to provide a highly resilient fabric to help ensure the performance required for storage traffic. As Ethernet evolves and becomes a lossless fabric, these two networks can be combined to form a single consolidated network for data and storage traffic. Forward compatibility with existing Fibre Channel SANs will be possible with FCoE implementations, preserving and protecting SAN investments in the future.

Figure 8. Unified I/O and FCoE Evolution Roadmap


FCoE is a compelling technology for enabling unified I/O in the data center. Cisco is committed to developing, supporting, and promoting the FCoE technologies required to facilitate this transition in a smooth, evolutionary way. As is the case with any new networking technology, adoption of FCoE-based storage networking will be evolutionary and will happen in phases: FCoE server enablement, FCoE server proliferation, and FCoE arrays and SANs. The arrival of the Cisco Nexus 5000 Series, in conjunction with the availability of cost-effective 10-Gbps LAN technology for the server, enables unified I/O in the data center. Data center administrators can now begin consolidating infrastructure to achieve economic benefits. IT departments can take advantage of what Cisco FCoE storage networking can offer today and be prepared to gain additional benefits from operational efficiencies and performance enhancements as the unified fabric becomes reality.