Today's data centers expand, contract, and make infrastructure changes with increasing frequency. To meet the challenges of this dynamic environment, Cisco delivers on its vision for a consolidated virtualized, unified, and radically simplified access layer. Cisco Nexus® 5000 Series Switches combined with Cisco Nexus 2000 Series Fabric Extenders support a highly scalable, low-latency access layer that supports the rapid deployment of new data center technologies in heterogeneous data centers along with incremental upgrades from 1 to 10 Gigabit Ethernet and Fibre Channel over Ethernet (FCoE). The solution's predictable low latency makes it a more scalable, low-cost option for high-speed interconnects in high-performance computing (HPC) and high-frequency financial environments, The result is an access layer based on a unified fabric that dramatically reduces cabling and infrastructure costs-savings that are proven by Cisco customers using these products.
Challenge: Data Centers in Transition
Today's data centers are dynamic entities, expanding and contracting with business cycles that include growth, consolidation, mergers, and acquisitions. Constant deployment and redeployment is the norm, and movement of assets within data centers is commonplace. Connectivity demands are increasing as the transition to 10 Gigabit Ethernet and 8-Gbps Fibre Channel is becoming a reality across servers, storage systems, and their respective networks.
The rapid changes of equipment combined with the migration to 10 Gigabit Ethernet and 8-Gbps Fibre Channel makes a number of problems even more acute:
• Cabling cost and complexity limit growth and agility in the data center. Older servers are configured with multiple Gigabit Ethernet and dual Fibre Channel connections, often to end-of-row (EoR) switches. The massive amount of cabling in cable trays makes it difficult to change a rack's cabling after installation, limiting mobility. Trays are reaching their weight limits, and having a large number of cables within racks impedes airflow and increases cooling costs.
• Operational complexity is becoming a more acute problem because of the number of switching devices that must be managed to handle a wide range of connection speeds and to make deployment architectures as resilient as possible. Data center growth adds a further complication because access-layer equipment was not designed to grow and scale with increasing bandwidth requirements.
• The proliferation of cables and network modalities requires more resources that consume power, including network interface cards (NICs), host bus adapters (HBAs), transceivers, and upstream Ethernet and Fibre Channel switch ports. While top-of-rack (ToR) switching makes growing and reconfiguring the data center easier, if not done correctly it can create hot spots that further challenge a data center's limited power and cooling resources.
Unified Data Center with Cisco Nexus 2000 Series Fabric Extenders
The combination of Cisco Nexus 5000 Series Switches and Cisco Nexus 2000 Series Fabric Extenders delivers a unified, consolidated, virtualized, and simplified access layer that addresses the challenges facing data center operators today. The product family is based on the concept of a remote I/O module in which fabric extenders are physically separate devices that are logically part of the parent switches to which they are connected. All management, configuration, troubleshooting, and firmware upgrade is handled by the parent switch. In combination, Cisco Nexus 5000 Series Switches and Cisco Nexus 2000 Series Fabric Extenders represent a virtual modular system that delivers the benefits of both fixed and modular systems (Figure 1)
Figure 1. The Combination of Cisco Nexus 5000 Series Switches and Cisco Nexus 2000 Series Fabric Extenders Delivers the Benefits of Both Fixed and Modular Systems.
The Cisco Nexus 2000 Series product line transforms data center designs, giving network architects greater design flexibility while simplifying cabling infrastructure and management and reducing power requirements. The Cisco Nexus 2000 Series provides a virtualization-aware fabric that scales across a wide range of 1 and 10 Gigabit Ethernet, unified fabric, rack, and blade server environments.
The expanded product line (Figure 2) delivers a flexible and consistent access-layer architecture with a physical topology that easily adapts to any combination of transport options, server form factors, and network topologies. It incorporates the best features of both ToR and EoR architectures with products that can be deployed in ToR, EoR, and middle-of-row (MoR) designs. The product line helps reduce total cost of ownership (TCO) through simplified management, reduced cabling, and a "wire-once" unified fabric. Most important, the combination of products helps data centers make a smooth transition to next-generation data centers with a single, unified access architecture that connects every server port, from 100-Mbps management network connections to 10-Gbps unified fabric with FCoE. The product line supports a "rack-and-roll" model in which preconfigured racks are deployed or moved within data centers and connected easily to upstream Cisco Nexus 5000 Series Switches.
Figure 2. Cisco Nexus 2000 Series Fabric Extenders
Scalability Without Increased Complexity
The Cisco Nexus 2000 Series expands the management domain and logical port count of Cisco Nexus 5000 Series Switches to up to 576 Gigabit Ethernet ports or 384 10 Gigabit Ethernet ports without increasing access-layer complexity. For applications requiring the low-latency, line-rate, lossless characteristics of the Cisco Nexus 5000 Series, direct connections to the switch deliver the high performance. For the vast majority of applications, however, connections to the access layer through fabric extenders provide an even more scalable and cost-effective approach. Cisco Nexus 2000 Series Fabric Extenders forward all traffic from their server ports to a parent Cisco Nexus 5000 Series Switch through one or more 10 Gigabit Ethernet uplinks. Passing all traffic to the parent switch allows all traffic to be shaped according to policies established on the parent Cisco Nexus 5000 Series Switch.
Logically, the access layer is a single entity because a Cisco Nexus 2000 Series Fabric Extender exists as a logical extension of the parent Cisco Nexus 5000 Series Switch to which it is connected (Figure 3). The parent switch handles firmware and device configuration as if the fabric extender were a line card installed directly on the switch. Physically, the fabric extenders can be dispersed throughout the data center, most commonly in ToR configurations that make each rack an easy-to-deploy unit.
Low Latency for Demanding High-Performance Computing and High-Frequency Trading
The low-latency characteristics of Cisco Nexus 5000 Series Switches and Cisco Nexus 2000 Series Fabric Extenders make a compelling case for high-performance computing and high-frequency trading applications. The Cisco Nexus 5000 Series Switches provide a strong foundation with their line-rate performance and cut-through, low-latency design. Using these switches for direct server connections provides line-rate bandwidth, while virtual PortChannel (vPC) technology allows customers to combine two active switches for redundancy while scaling out the Layer 2 network without having to resort to the use of Spanning Tree Protocol.
When Cisco Nexus 5000 Series Switches are combined with Cisco Nexus 2000 Series Fabric Extenders, organizations with low-latency requirements have a new, more cost-effective HPC interconnect that enables compute clusters to scale with predictable low latency and jitter. Even during traffic bursts, the infrastructure maintains predictable latency without packet loss. A single pod can link up to 384 servers into a fine-tuned, high-performance pod with latency of less than 5 microseconds-without the need for multiple tiers of interconnect switches, which introduce variable latencies. For organizations not yet standardized on 10 Gigabit Ethernet, the Cisco Nexus 2000 Series provides a smooth migration path that protects investments and interoperates between multiple generations of server platforms, enabling compute clusters to scale across both 1 and 10 Gigabit Ethernet-equipped servers.
Figure 3. The Unified Access Layer Is a Single Logical Entity That Is Physically Distributed Across the Data Center.
Rapid Deployment with a Plug-and-Play Model
With an access layer based on Cisco Nexus 5000 Series Switches, server racks preconfigured with fabric extenders can be rolled into position and their uplinks connected to integrate them into the data center access layer. Because all fabric extender firmware and configuration is managed by the parent switch, fabric extenders can be integrated into the access layer simply by connecting their uplinks. The simple way in which the access layer can be expanded makes rapid deployment a reality: servers can be installed into racks along with fabric extenders, rolled into position, and integrated into the access layer just by connecting the uplinks. Both Ethernet and Fibre Channel connectivity can be handled with one set of uplinks on 10-Gbps fabric extenders with unified fabric support.
Support for Multiple Connectivity Requirements with Investment Protection
Within each rack, 100-Mbps Ethernet connections can be made to management consoles, Gigabit Ethernet connections can be made to older rack-mount and blade systems, and 10 Gigabit Ethernet and FCoE connections can be made to state-of-the-art systems including Cisco® UCS C-Series Rack-Mount Servers.
A main feature of the solution is that a single switch-the Cisco Nexus 5000 Series-provides the access layer's switching capacity while a set of satellite fabric extenders satisfies the range of connectivity requirements found in any data center (Figure 4). Racks of older servers with 100-Mbps and Gigabit Ethernet connectivity requirements can be rolled out and replaced with newer racks of servers with Gigabit Ethernet and 10 Gigabit Ethernet connections. In time, those servers can be replaced with racks of servers and blade systems connected to a 10 Gigabit unified fabric that carries both IP and Fibre Channel traffic over a simple rack-level wiring plan. As the data center evolves and changes and racks come and go, the access layer based on Cisco Nexus 5000 Series Switches remains constant, and customer investments are protected.
Figure 4. Cisco Nexus 2000 Series Fabric Extenders Support a Range of Connectivity Requirements
Cost-Effective Transition Path
The access layer augmented with fabric extender technology provides a granular, cost-effective transition path for data centers while supporting server connectivity requirements now and in the future. The technology also helps reduce data center power and cooling costs. The low power requirements of the Cisco Nexus 2000 Series place only an incremental demand on rack-level power while helping evenly spread the heat generated by the access layer across all fabric extender-equipped racks.
Operational Simplicity with Unified Fabric
The Cisco Nexus 5000 Series Switches and all 10 Gigabit fabric extenders support a unified fabric that brings together Ethernet and FCoE with standards-based management extensions to make the fabric even more reliable and manageable. Advanced, standards-based extensions support the lossless requirements of Fibre Channel traffic on the same physical link as standard IP traffic. When the unified fabric is accessed by servers equipped with converged network adapters (CNAs), data centers can deploy a unified fabric in a way completely transparent to the operating system and its drivers, allowing IT departments to continue using their traditional server, storage, and networking administration roles without disruption.
Increased Agility with a Wire-Once Cabling Model
The unified fabric radically simplifies rack-level cabling with a wire-once model that allows racks to be cabled once, with advanced networking features (such as FCoE) configured in software. Instead of each server's being equipped with multiple Gigabit Ethernet NICs and dual Fibre Channel HBAs for both redundancy and bandwidth, each server can connect through a single pair of low-cost CX-1 copper connectors to a fabric extender at the top of each rack. A single pair of CNAs replaces multiple NICs and HBAs, and the high cost of fiber cabling, transceivers, and upstream switch ports is eliminated. The unified fabric brings such simplicity and cost effectiveness to the data center that nearly every server can be supported with Fibre Channel storage without the corresponding costs.
Cisco VN-Link technology allows network administrators to handle virtual machine connectivity similar to the way they handle physical machine connectivity. The concept is both simple and powerful. Physical servers are connected through physical links, and network profiles are applied to the links within the switch. With Cisco VN-Link technology, virtual links connect from virtual NICs (vNICs) to virtual interfaces within the switch. Virtual links give network administrators fine-grained control over each virtual link's network profiles, including the capability to simply turn off connectivity to an individual rogue virtual machine. Virtual links can move freely between physical ports, so when a virtual machine moves between servers, every aspect of its network connectivity moves along with it with complete consistency. This feature dramatically simplifies the management of complex virtualized environments, and it allows administrators to support mobility with more control over quality of service (QoS) and security than ever before.
Cisco VN-Link technology is supported by the Cisco Nexus 5000 Series and the Cisco Nexus 2000 Series through a software switch that replaces the default switch in VMware vSphere environments. The Cisco Nexus 1000V Switch offers tight integration between server and network environments to help ensure consistent, policy-based network capabilities for all virtual machines. The Cisco Nexus 1000V brings standard Cisco switch capabilities to each virtual machine, giving network administrators the same control over virtualized environments as they have over physical ones.
Multiple Options for High Availability
Fabric extenders can be deployed to meet a range of availability requirements using two different models, both of which avoid the complexity of introducing Spanning Tree Protocol into the access layer:
• Virtual PortChannels: Support for VPC technology allows fabric extenders to connect to two different Cisco Nexus 5000 Series Switches in an efficient active-active configuration. This approach provides continuous availability through the loss of an uplink port, cable, switch port, or switch. The active-active nature of the vPC allows efficient use of bandwidth during normal operation.
• Dual homing: The risk that a single fabric extender will fail can also be mitigated by dual homing each server to two Cisco Nexus 5000 Series Switches through two fabric extenders using vPC technology. In addition, this configuration mitigates NIC failures on servers.
Data Center Mobility with Simplified Cabling
The Cisco unified data center access layer supports a simplified, modular cabling strategy that increases mobility and enables data centers to easily accommodate expansion, consolidation, and movement of equipment within the data center.
Cabling Within Racks
With the ToR deployment model supported by the Cisco Nexus 2000 Series Fabric Extenders, all cabling within racks can use inexpensive copper cabling. For 100-Mbps and Gigabit Ethernet, standard patch panels with RJ-45 connectors can be installed. For 10 Gigabit Ethernet, CX-1 cabling with built-in transceivers is an extremely cost-effective and low-latency alternative to fiber within racks.
Cabling Between Racks
For cabling between racks, specifically between fabric extenders and Cisco Nexus 5000 Series Switches, fiber is one of the best long-term investments. It is the most flexible transmission medium, and advancements in Ethernet technology tend to be available first on fiber. Today, fiber can be used to support 1 and 10 Gigabit Ethernet and 2-,
4-, and 8-Gbps Fibre Channel. In the future, 40- and 100-Gbps Ethernet standards are likely to be supported first on fiber. Fiber in overhead cable trays running to each rack position supports a "rack- and- roll" model where in which any rack with top-of-rack switching can be rolled into any position.
For pod-based deployments, using fabric extenders in every rack and switches at the middle of the row, CX-1 cabling offers a lower-cost alternative to fiber.
The Cisco Nexus 2000 Series brings together the flexibility and agility of the ToR model with the management simplicity and efficiency of the EoR model. Cisco Nexus 2000 Series Fabric Extenders require only one rack unit (1RU) of space, and they use only as much power as a few light bulbs. With the expanded product line, Cisco now offers a fabric extender to support every bandwidth requirement.
Products for Implementing a Unified Access Layer
Cisco has a complete vision for the data center access layer, and Cisco innovations make implementation straightforward, with a comprehensive set of products and updates. The basic parameters of today's Cisco Nexus 2000 Series are outlined in Table 1, with new and updated features as follows:
• The Cisco Nexus 2232PP 10GE Fabric Extender brings the 10 Gigabit unified fabric with FCoE and Cisco VN-Link technology to the ToR model. Supporting 10 Gigabit Ethernet using Small Form-Factor Pluggable Plus (SFP+) transceivers, the Cisco Nexus 2232PP 10GE provides an ideal connectivity solution for mixed racks. This model helps data centers establish a migration path to a 10-Gbps unified fabric while protecting investments through the transition. The fabric extender's 32 ports can support the densest server racks.
• The Cisco Nexus 2248TP GE Fabric Extender is a 48-port device supporting 100/1000-Mbps Ethernet. The product helps reduce costs by providing standard RJ-45 interfaces for use with standard patch cables. It introduces the capability to consolidate server console management ports without the need for long cable runs to an EoR switch. Because it can provide Gigabit Ethernet connectivity as well as Fast Ethernet connectivity, the fabric extender can be used to consolidate connectivity for both older server ports and management networks. Establishing consistency across the product line, this fabric extender also supports Cisco VN-Link technology.
Table 1. Basic Parameters of the Cisco Nexus 2000 Series Fabric Extenders
Fabric Extender Model
Cisco Nexus 2232PP 10GE
Cisco Nexus 2248TP GE
Server port speed
32 SFP+ (10 Gbps)
10-Gbps uplink ports
Advanced networking features
Unified management and configuration through parent Cisco Nexus 5000 Series Switch
Unified management and configuration through parent Cisco Nexus 5000 Series Switch
Cisco Nexus 5000 Series Switches complete the unified access layer with their low-latency, lossless, cut-through switching architecture. In unified access layers that implement FCoE, the switches pass Fibre Channel traffic onto native Fibre Channel connections using expansion modules that can be installed on the switch. Customers can now choose a six-port, 8-Gbps Fibre Channel expansion module for compatibility with the most recent Fibre Channel speed upgrades. This new expansion module complements the existing choices: a six-port, 10 Gigabit Ethernet module; a four-port 10 Gigabit Ethernet and four-port 4-Gbps Fibre Channel module; and an eight-port 4-Gbps Fibre Channel module.
Unified Access Layer in Heterogeneous Data Centers
Heterogeneous data centers are the norm, with most having some mix of equipment: blade and rack-mount servers, systems from different vendors, older systems, and state-of-the-art servers. Even data centers fully committed to a unified fabric must support existing systems as they upgrade as part of their normal IT refresh cycle.
To see how a unified access layer can address all the connectivity needs of a heterogeneous environment, consider the example in Figure 5. The unified access layer is implemented with a set of Cisco Nexus 5000 Series Switches and Cisco Nexus 2000 Series Fabric Extenders. Each example illustrates a fully redundant configuration with no single point of failure. From left to right, the server racks connected with ToR fabric extenders are as follows:
• Dense Server Racks: A dense rack consisting of Cisco UCS C200 M1 High-Density Rack-Mount Servers is connected to the switches through a redundant pair of 32-port Cisco Nexus 2232PP 10GE Fabric Extenders. These servers use a 10-Gbps unified fabric and require only two data connections per server. (Figure 5 has been simplified by not showing a Cisco Nexus 2248TP GE Fabric Extender in each rack that provides ToR connectivity for server and blade management consoles.)
• Mixed Environments: A mixed server environment consisting of Cisco UCS C210 M1 General-Purpose Rack-Mount Servers and C250 M1 Extended-Memory Rack-Mount Servers plus third-party servers connects to a pair of Cisco Nexus 2232PP 10GE Fabric Extenders using 10 Gigabit Ethernet,
• 10 Gigabit Ethernet to Blade Servers with Pass-Through Modules: A rack of third-party blade servers with 16 blades uses pass-through modules to bring a total of 32 network connections per server to the chassis. Pass-through modules offer transparent, 1:1, 10 Gigabit Ethernet connectivity to each blade server, eliminating switching elements within the chassis and increasing network and management transparency. The reduction in managed switching elements reduces capital expenditures and operating expenses for each blade chassis and leads to a more stable server access-layer network. Transparency enables delivery of the Cisco Nexus solution to traditional blade servers and helps ensure support for unified fabric and server virtualization without the need for costly blade switch upgrades. Adopting a unified fabric further reduces cost and complexity within the blade server access layer through the removal of Fibre Channel blade switches and HBAs. In this example, pass-through modules are supported by four Cisco Nexus 2232PP 10GE Fabric Extenders. Blade servers from Dell, HP, and IBM can use pass-through modules.
• Blade Servers with Internal Switches: The next two racks illustrate the use of blade systems with internal switches. Because oversubscription is a standard practice when blade switches are employed, the systems are connected directly to low-latency, lossless Cisco Nexus 5000 Series Switches. One of the two racks uses a Cisco Nexus 4000 Series Blade Switch within the blade chassis, an option that is available for blade servers from IBM. The second rack uses blade systems equipped with 10 Gigabit Ethernet blade switches, illustrating how these devices can uplink to Cisco Nexus 5000 Series Switches.
• Cisco Unified Computing System™: The final two racks illustrate a Cisco Unified Computing System consisting of a total of 12 chassis and a redundant pair of Cisco UCS 6100 Series Fabric Interconnects. The interconnects uplink via 10 Gigabit Ethernet to Cisco Nexus 5000 Series Switches, and native Fibre Channel traffic is separated from the unified fabric by the fabric interconnects and uplinked directly to the Cisco MDS 9000 Series Multilayer Switches. This design provides a uniform architecture for aggregating multiple Cisco Unified Computing System deployments into a common IP network layer based on Cisco Nexus 5000 Series Switches.
Figure 5. A Unified Access Layer Based on Cisco Nexus Technologies Supports a Wide Range of Rack-Mount and Blade Server Connectivity Requirements
Any of the systems illustrated in Figure 5 could use a Cisco Nexus 1000V switch along with VMware vCenter software if virtualization is being used.
Proven TCO Reduction by Cisco Customers
While the example of the previous section illustrates how a unified access layer could be deployed to efficiently and effectively meet the needs of a mixed data center environment, the real proof of the value of the unified access layer with Cisco Nexus technologies is in the experiences of Cisco customers:
• St. Joseph Health System deployed a unified access layer using Cisco Nexus technologies and funded new data center switches from the savings on cabling alone: rather than spending an estimated US$1.3 million on cabling, with the unified access layer St. Joseph spent only US$190,000, an 85 percent savings. St. Joseph estimated its space savings at 80 percent, and its power savings at 25 percent. For the full story, see http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/Healthcare_Provider_case_study.pdf.
• NetApp reduced its network edge costs by 40 percent by deploying a cost-effective access layer based on the Cisco Nexus 5000 Series and Cisco Nexus 2000 Series. The cloud computing environment that NetApp deployed using this unified access layer required rapid scalability, and the Cisco Nexus technology-based environment provided exactly that. The solution's energy efficiency, including front-to-back airflow, helped NetApp achieve a power use effectiveness (PUE) rating of 1.2, far exceeding the rating of 2.0 or greater of most data centers. For the full story, see http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/case_study_c36-554435.html.
• Lawrence Livermore National Laboratories (LLNL) used Cisco Nexus technologies as a foundation to support future growth with increased availability. The LLNL is now prepared to migrate to 10 Gigabit Ethernet and virtualization on the lab's own schedule, all with a simplified management structure. One attribute of the unified access layer that particularly interested LLNL was the fabric extender's pay-as-you grow model that incorporates scalability with lower costs. For the full story, see http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/case_study_c36-562603_ps10110_Products_Case_Study.html.
Cisco has a complete vision for a consolidated, virtualized, unified, and radically simplified access layer. Cisco implements that vision with innovative products that are available today, and with products that enhance and protect the investment of products already in customer data centers. The comprehensive line of Cisco Nexus 2000 Series Fabric Extenders offers a unified access layer that is scalable without being complex. It provides connectivity for the wide range of server technologies that are at work in data centers today, and it consolidates the access layer with a unified fabric that dramatically reduces cabling and infrastructure costs.