Figure 1. The Current Trends That Are Driving the Demand for Higher-Speed Ethernet
The business case for 40 Gigabit Ethernet is becoming inescapably compelling. While 10 Gigabit Ethernet is still making its way into the data centers, CIOs and IT managers must now consider how they are going to handle what's coming next: high-bandwidth applications such as server virtualization and cloud computing, fabric consolidation within the data center, and a greater demand for high-performance computing among end users (see Figure 1). The need for faster data transfer rates is relentless and carries significant implications with regard to network productivity as well as OpEx costs.
So when the IEEE officially adopted IEEE Std. 802.3ba in June, paving the way for both 40 Gigabit Ethernet and 100 Gigabit Ethernet, it came not a moment too soon. The increased speed will allow networks to move newfound 10 Gigabit Ethernet resources to the access layer, allowing the more powerful 40 Gigabit Ethernet-enabled equipment to handle traffic at the aggregation and core layers. Based on analysts' forecast and the robust development efforts by OEMs, it is no longer a question as to if 40 Gigabit Ethernet will become an accepted part of the IT landscape, but when and how.
This white paper addresses the impending move to 40 Gigabit Ethernet (Figure 2), how it may change the network architecture, and what IT managers can do now to prepare to migrate to the new standard.
Introduction: The Business Case for 40 Gigabit Ethernet
Since February 1980, when the first IEEE 802 standards committee convened, speeds in Ethernet delivery to all layers have made increasingly greater leaps over increasingly shorter intervals. Now, eight years to the month after the adoption of 10 Gigabit Ethernet, the IEEE has adopted 802.3ba, paving the way for 40 Gigabit Ethernet and 100 Gigabit Ethernet.
As illustrated by Figure 2, I/O data transfer rates within the access layer are doubling every 24 months, while transfer rates at the core layer double approximately once every 18 months. A primary driver behind the push to 40 Gigabit Ethernet is a new generation of high-speed, high-demand, computing applications and technologies. These include the spreading deployment of virtual servers and cloud computing.
By the end of 2009, nearly one in five new servers were virtualized.1 At the same time, the financial pressures of a challenging economy are forcing networks to look for ways to consolidate their resources in an effort to reduce OpEx and total cost of ownership.
With four times the capacity and the ability to cost-effectively migrate to 100 Gigabit Ethernet (see Figure 2), 40 Gigabit Ethernet on multimode fiber is the next logical step in the evolution of the data network. The increase in urgency and the drop in prices have led analysts at the market research firm The Dell'Oro Group to predict that shipments in the 40-Gbps optical market will reach $14.5 billion by 2013.2
Figure 2. Leading Edge Deployment Trends; General Market Deployment Is Typically Delayed by Several Years
There has been some debate as to whether IT managers should hold off on deploying the 40 Gigabit Ethernet technology and bide their time waiting for 100 Gigabit Ethernet to become commercially available. But that question is fast becoming moot because 40 Gigabit Ethernet provides design flexibility and cost advantage over 100 Gigabit Ethernet. 40 Gigabit Ethernet can be effectively deployed today in aggregation links in data center networks. By 2016, 40 Gigabit Ethernet will also be commonly applied to access links to connect servers, as Figure 3 indicates. Complementing 40 Gigabit Ethernet, 100 Gigabit Ethernet is a perfect choice for carrier service providers and core links in data centers.
Figure 3. Gigabit Ethernet for Multimode and Single-Mode
For IT Managers and CIOs who are intent on remaining competitive, 40 Gigabit Ethernet represents the best option for adding the required bandwidth. But without the proper planning, network operators may very well be caught unprepared when the time comes to make the move.
Let's examine the process to thoroughly evaluate, plan, and specify a network upgrade to 40 Gigabit Ethernet. As Figure 4 illustrates, 40 Gigabit Ethernet is still a couple of years from wide-scale adoption, which gives IT managers and CIOs time to start their migration planning.
40 Gigabit Ethernet and 100 Gigabit Ethernet are Ethernet standards developed by the IEEE 802.3ba Task Force to support sending Ethernet frames at 40 and 100 gigabits per second. They also address physical layer specifications for communication across backplanes, copper cabling, multimode-fiber, and single-mode fiber. Official development of the 40 Gigabit Ethernet and 100 Gigabit Ethernet standards began in January 2008, and the standards were officially ratified in June 2010.
At the heart of the 40 Gigabit Ethernet network layer is a pair of transceivers connected by a cable - for example, OM4 or OM3 fiber cable. The transceivers, in turn, are plugged into either network servers or a variety of components including interface cards and switches.
Figure 5. Transceiver Form Factors Planned for 1st Generation Implementation
40 Gigabit Ethernet transceivers (see Figure 6) are being developed along several standard form factors. The C Form-Factor Pluggable (CFP) transceiver features 12 transmit and 12 receive 10-Gbps lanes to support one 100 Gigabit Ethernet port, or up to three 40 Gigabit Ethernet ports. Its larger size is suitable for the needs of single-mode optics and can easily serve multimode optics or copper as well. The CXP transceiver form factor also provides 12 lanes in each direction, but is much smaller than the CFP and serves the needs of multimode optics and copper. The Quad Small-Form-Factor Pluggable (QSFP) is similar in size to the CXP and provides four transmit and four receive lanes to support 40 Gigabit Ethernet applications for multimode fiber and copper today and may serve single-mode in the future. Another future role for the QSFP may be to serve 100 Gigabit Ethernet when lane rates increase to 25 Gbps.
Figure 6. Transceiver Form Factors Planned for First-Generation Implementation
Cables and Connectors
Cabling for 40 Gigabit Ethernet, summarized in Figure 5, can be optical fiber or copper. The supportable channel length depends on the cable and the transceiver type. With regard to connectors, the only significant change outlined in the 802.3ba standard is the use of MPO Multi-Fiber Push On (MPO)-type connectors at the multimode transceivers to support the multifiber parallel optics channels.
For data center environments operating at 40 Gbps or 100 Gbps, OM3 and OM4 multimode cabling is generally recommended because its reach supports a wider range of deployment configurations compared to copper solutions, and the cost is lower compared to single-mode solutions.
Use of Parallel Optics
Traditionally, the Ethernet standard has relied upon duplex fiber cabling with each channel using one fiber to transmit and the other to receive. However, the 802.3ab standard requires multiple lanes of traffic per channel. To get multiple lanes, the 40 and 100 Gigabit Ethernet standard uses parallel optics, as indicated in Figure 8.
Figure 8. Layout Showing Ethernet Channel Distribution for 10/40/100 Gigabit Ethernet Using Multimode Fiber
The 40 Gigabit Ethernet specification calls for a 12-fiber cabling solution, with each channel featuring four dedicated transmit fibers and four dedicated receive fibers. The middle four fibers remain unused, or dark. The 100 Gigabit Ethernet solution specifies 24 fibers divided into two 12-fiber arrays, with one array dedicated to transmit and the other dedicated to receive. In each array, the middle 10 fibers are dedicated to traffic while the two fibers on either end remain dark.
For 100 Gigabit Ethernet, several interface variants have been described with the preferred option being a single 24-fiber MPO connector. Alternatively, two 12-fiber connectors can be positioned either vertically or side by side to make up the channel.
Deploying 40 Gigabit Ethernet in the Network
The new 40 Gigabit Ethernet technology will, most likely, first be deployed in the data center as shown in Figure 7. This will help alleviate bottlenecks in the layer that connects access switches to distribution switches.
"With enterprises beginning to deploy 10 Gigabit Ethernet uplinks on client side switches in the wiring closet, as well as 10 Gigabit Ethernet direct server connections, the aggregation of these links is becoming a bottleneck in the network until higher speeds are widely available," said Alan Weckel, director, Ethernet Switch & Enterprise Telephony Market Research at Dell`Oro Group.3
"For example, at present enterprises must deploy complicated fat tree or spanning tree architectures to aggregate 10 Gigabit Ethernet using the same speed as both the aggregator and the aggregated. We believe that 40 Gigabit Ethernet and 100 Gigabit Ethernet will be critical to meeting the increasing demands for bandwidth in the data center," Weckel explained.2
Figure 9. 40/100 Gigabit Ethernet Deployment Throughout the Data Center Network
One of the most attractive characteristics of 40 Gigabit Ethernet is broad applications and design flexibility. Considering the productivity gains and decrease in OpEx, migrating to 40 Gigabit Ethernet will prove very cost-effective for those who do it right.
In migrating to 40 Gigabit Ethernet, some networks will be able to use their current 10 Gigabit Ethernet switch chassis and just upgrade their line cards and transceivers. Deploying a CFP transceiver will provide the flexibility to migrate from 40 Gigabit Ethernet to 100 Gigabit Ethernet.
With regard to cabling, OM3 or OM4 is optimal for the 40 Gigabit Ethernet or 100 Gigabit Ethernet data center environment. The major difference is in the maximum span distances. In a 10 Gigabit Ethernet network, OM3 fiber can span up to 300m while OM4 supports even longer channels. In a 40 Gigabit Ethernet or 100 Gigabit Ethernet environment, OM3 can be used up to 100m and OM4 up to 150m, according to the IEEE802.3ba standard. For applications approaching 150m, the cable should be terminated with low-loss connectors.
There are a number of alternative upgrade paths leading from 10 Gigabit Ethernet to 40 Gigabit Ethernet. To make an optimal decision for your network, here are some key issues to consider:
• What factors determine when it more effective to deploy a 40 Gigabit Ethernet physical layer compared to aggregations of 10 Gigabit Ethernet channels?
• What installed hardware or cabling, if any, will need to be replaced or reconfigured?
• What are the capabilities of various 40 Gigabit Ethernet transmission alternatives?
Introduction of the Cisco Catalyst 6500 4 Port 40 Gigabit Ethernet and 16 port 10 Gigabit Ethernet Line Card
The new Cisco® Catalyst® 6900 Series 4-Port 40 Gigabit Ethernet Fiber Module (WS-X6904-40G) for the Cisco Catalyst 6500 Series Switches was developed for deployment in the enterprise campus distribution layer and core layer, in traditional data centers with 10 Gigabit aggregation, and in metro-Ethernet-based data center interconnections, as well as in multipurpose service provider networks, which require high-performance data throughput coupled with security, manageability, virtualization, and quality of service.
The WS-X6904-40G module supports either (and both) 10 Gigabit Ethernet or 40 Gigabit Ethernet interfaces. This new module is the only solution that supports both 10 Gigabit and 40 Gigabit Ethernet connections in a single format, for flexible migration. There are also many other new and exciting capabilities unique to the WS-X6904-40G module, such as CTS (Cisco Trust Sec) for end to end role based security deployment using SGT (Security Group Tagging) and SGACL (Security Group Access Control List) and link by link encryption based on IEEE 802.1AE MACsec (MAC based Security) standard., VNTag (Virtual Network Tag), extensive quality of services features.
Figure 10 shows the port assignment for the 40 Gigabit Ethernet line card. Figure 11 shows the port assignment for the 10 Gigabit Ethernet line card.
Figure 10. 4 Port 40 Gigabit Line Card Port Assignment
Figure 11. 16 Port 10 Gigabit Ethernet - Line Card Port Assignment
Figure 12 shows the 40 Gigabit Ethernet CFP Transceiver and FourX 10 Gigabit Ethernet SFP+ Converter
As links to access switches move to 10 Gigabit Ethernet, customers need interfaces greater than 10 Gigabit to link the aggregation and core layers together (Figure 13).
Enterprise and Service Delivery Data Center: Aggregation and Core
For switch uplinks supporting higher bandwidths as end hosts move towards 10 Gigabit Ethernet. This is the principal market for 40 Gigabit Ethernet (Figure 14).
Figure 13. Campus Deployment of Cisco Catalyst 6500 Series 4 Port 40 Gigabit Ethernet Line Card
Figure 14. Data Center Deployment of Cisco Catalyst 6500 Series 4 Port 40 Gigabit Ethernet Line Card
Don't Wait to Start Future-Proofing Your Network.
The risk in delaying migration planning is that you may underestimate what is involved in evaluating and selecting the best migration path. While it is true that upgrading from 10 Gigabit Ethernet to 40 Gigabit Ethernet should be a relatively smooth process, today's network manager must be thinking a step beyond. That means considering not only how best to transform today's legacy systems to a 40 Gigabit Ethernet environment, but what implications that transformation will have on the eventual migration to 100 Gigabit Ethernet and beyond. By thoroughly considering these issues now and developing an implementation plan, IT can help ensure a future-proof evolution of their networks.
It helps to remember that the preplanning process need not involve any immediate purchases. With the 802.3ba specifications now in place, networks can safely plan without fearing that the landscape will change dramatically.
In fact Cisco Catalyst 6500 is the perfect for this with it's newest 40 Gigabit Ethernet line card, you can deploy it today in 10 Gigabit Ethernet mode using the FourX adapter to convert each of the 40 Gigabit Ethernet ports into a standard 10 Gigabit Ethernet port without any upgrade and therefore disruption to your current network and slowly but seamlessly migrate your network to 40 Gigabit Ethernet as when you are ready to meet the ever increasing demand for bandwidth. The Catalyst 6500 continues to evolve with the release of the Supervisor Engine 2T and now with it 6900 Series 40 Gigabit Ethernet line card.
1Worldwide Quarterly Server Virtualization Tracker, International Data Corporation, April 2010.
2InformationWeek, January 29th, 2009, reported by W. David Gardner.
3Ethernet Alliance® Congratulates IEEE on the Ratification of 40 and 100 Gigabit Ethernet Standard; Announces Demonstration and Interoperability Plans, Press Release, June 21, 2010.