Cisco Nexus 5000 Series Switches: Decrease Data Center Costs with Consolidated I/O
PDF(241.9 KB) View with Adobe Reader on a variety of devices
Updated:Apr 22, 2008
Data centers are growing at an unprecedented rate, creating challenges for enterprises. Enterprise-level applications require increased computing power; Web servers have become central to enterprises' customer-facing strategies; and the amount of data, including e-mail, documents, and multimedia, central to enterprises' operations continues to expand. In addition, compliance regulations require data to be kept for longer periods of time. All these demands make running a data center increasingly complex and expensive.
In addition, data centers typically run multiple separate networks including: an Ethernet network for client-to-server and server-to-server communications, and a Fibre Channel storage area network (SAN). To support various types of networks, data centers use separate redundant Interface Module for each network, i.e. Ethernet network interface cards (NICs) and Fibre Channel interfaces in their servers, and redundant pairs of switches at each layer in the network architecture. Use of parallel infrastructures increases capital costs, makes data center management more difficult, and diminishes business flexibility.
Unified fabric can meet these challenges, consolidating I/O in the data center and allowing Fibre Channel and Ethernet networks to share a single, integrated infrastructure. With the Cisco
® Nexus 5000 Series Switches, part of the Cisco Nexus family of data center-class switches, Cisco now offers a product that delivers unified fabric at the network access layer or edge, where servers connect to the LAN ,SAN and Server Clusters. The server connection is the best initial location for deploying a unified fabric in the data center because the rate of technology change is greater there than at the core, and because the access layer or edge typically has many physical network interconnects to consolidate. To integrate with existing infrastructure, the Cisco Nexus 5000 Series provides native Fibre Channel uplinks to facilitate connection with installed SANs and available SAN switches. Other Cisco products supporting Fibre Channel over Ethernet (FCoE) will be available in the future to extend consolidation beyond the access layer.
This document describes the business benefits offered by Cisco Nexus 5000 Series Switches, using real-world scenarios encountered in data centers today. The Analysis reveals that the cost of building a 10-Gbps consolidated infrastructure using the Cisco Nexus 5000 Series can be competitive with the cost for similar, unconsolidated configurations at 1 Gbps. Thus, at little or no additional cost, and often at less cost, enterprises can get all the benefits of a 10-Gbps infrastructure. The analyses also show that as the number of connections increase, a consolidated 10-Gbps infrastructure with the Cisco Nexus 5000 Series becomes even more financially attractive.
Cisco Nexus 5000 Series Benefits
The innovative architecture of the Cisco Nexus 5000 Series simplifies data center transformation by enabling a high-performance, standards-based, Ethernet unified fabric. The platform consolidates separate LAN, SAN, and server cluster network environments into a single unified fabric. Backed by a broad system of industry-leading technology partners, the Cisco Nexus 5000 Series is designed to meet the challenges of the next-generation data centers, which include dense multisocket, multicore, virtual machine-optimized services, where infrastructure sprawl and increasingly demanding workloads are commonplace.
The Cisco Nexus 5000 Series extends Fibre Channel traffic over 10 Gigabit Ethernet networks, consolidating I/O onto one set of cables, eliminating redundant adapters, cables, and ports. A single card and set of cables connects servers to the Ethernet and Fibre Channel networks and also allows the use of a single cabling infrastructure within server racks (Figure 1).
Figure 1. Existing Infrastructure with Dual LAN and SAN Networks (Left) and Consolidated I/O Using the Cisco Nexus 5000 Series (Right)
The benefits of I/O consolidation enabled by the Cisco Nexus 5000 Series include the following:
• Lower total cost of ownership (TCO): Infrastructure reduction decreases capital expenditures (CapEx), and infrastructure simplification decreases operating expenses (OpEx).
• Simplified data center infrastructure. IT staff manage a simplified, homgeneous infrastructure, with fewer adapter cards, cables, and ports, freeing staff for more productive tasks. Simplification of server I/O requirements can also reduce the interface cost per server and allow the provisioning of smaller, less-expensive server form factors, because fewer slots are required.
• Data center consolidation with investment protection for existing server, network storage, and facilities assets: FCoE deployed between servers and Cisco Nexus 5000 Series Switches simplifies infrastructure and reduces costs without disrupting the existing storage architecture, network architecture, or management and operating processes.
• Increased business agility through virtual machine-aware network services: The Cisco Nexus 5000 Series delivers virtual machine-aware network services, enabling applications to move between physical servers while maintaining consistent levels of security, quality of service (QoS), and performance. It standardizes cabling requirements and enables full connectivity between servers and storage devices.
Methodology and Scope of the Business Analysis
The methodology, scenarios, and models presented here are based on Cisco's considerable experience with enterprise-level data centers and have been validated by interviews in the financial, services, and manufacturing sectors. In the scenarios, all prices for switches and optics are list prices. Prices for all other components and equipment are actual marketplace prices at the time of writing.
Cisco recognizes that the scenarios presented here are selected examples and do not represent the full range and complexity of data center deployments. Nevertheless, the scenarios are useful starting points for understanding TCO, return on investment (ROI), and other financial metrics as they apply to the Cisco Nexus 5000 Series.
The business analysis in the scenarios considers TCO related to the access layer or edge. Components analyzed include adapters, switches, and connectivity, to quantify capital expenses and maintenance and cable lifecycle costs. In general, savings in capital expenses dominate Cisco Nexus 5000 Series unified fabric TCO quantify-able benefits. Interviewees contributing to this paper could not yet provide analytic data for operating expense items in the areas of benefits of standardization of server I/O and infrastructure management. A numeric contribution to the TCO will require future real-world user experience.
Cable costs can be significant. Cable installation within a rack can run up to US$200 per cable, and running cabling to patch panels can cost more than US$600 per cable, depending on labor rates and the state of cable infrastructure. In addition, though not quantified in this analysis, cable bulk can impede airflow, consuming up to 25 percent more power in increased airflow. Also, abandoned cable can be a hidden cost; decommissioned cable is often left in place and over time can cause airflow and service problems. Installation of fewer cables controls these problems. Cabling Cost are a significant component of I/O consolidation cost savings.
One long-term benefit of unified fabric is power reduction. Cisco cites a goal of 8 percent power reduction in the data center through network consolidation. In this first generation of products deployed at the access layer, power savings will be more modest because consolidation is not yet fully achieved and because often technology is being upgraded by moving from a 1-Gbps LAN and 4-Gbps SAN to a 10-Gbps network. Depending on the configurations, gains exceeding 30 percent may be achieved, but quantifying those gains at US$0.10/kwh does not contribute meaningfully to this TCO analysis.
Adoption of 10-Gbps LAN technology is accelerating in the marketplace, promoted by lower costs, viable copper connections, and the imminent arrival of 10 Gigabit Ethernet embedded on the server motherboard. 10 Gigabit Ethernet will require smaller switches per rack or per rack multiple to take advantage of inexpensive but limited-length copper connections and reduce the number of expensive optical links. If 10 Gigabit Ethernet is embedded in the server or otherwise required by an application, such as server virtualization, justification of SAN consolidation is relatively easy. By eliminating SAN adapters and reducing the number of SAN cables, optics, and SAN switches, consolidation can offer per-rack savings of US$10,000 to US$20,000 or more. The Cisco Nexus 5000 Series can be deployed as a standard rack-oriented LAN switch for exclusively 10 Gigabit Ethernet environments and, for the servers connected to the SAN, as the vehicle for consolidation Figure 2). Example 1 below illustrates the incremental benefit I/O consolidation for a 10GbE LAN plus 4GFC SAN implementation.
Figure 2. Cisco Nexus 5000 Series Saves Costs by Reducing the Number of Adapters, Switches, and Connections
The benefits of consolidation are not restricted to 10 Gigabit Ethernet adopters; 10-Gbps unified fabrics can be competitive for Gigabit Ethernet LANs and 4-Gbps SANs as well. The examples that follow show the possible benefits of a unified fabric compared to the Gigabit Ethernet and 4-Gbps Fibre Channel networks commonly in place today. However, in this first generation, unified fabric will not provide a capital equipment expense saving in all cases. In general, the more network connections at the server and the higher the percentage of servers connected to SAN, the greater the savings.
Unified fabric is most efficiently implemented in conjunction with new server installations or upgrades to support, for example, virtual machine technology.
The Cisco Nexus 5000 Series offers both CapEx and OpEx savings along with the benefits of 10 Gigabit Ethernet access. As the scenarios in the two examples described here show, data centers can often move to 10-Gbps Ethernet at little or no additional cost compared to unconsolidated, 1-Gbps LAN plus 4-Gbps SAN deployments. The scenarios also show that the Cisco Nexus 5000 Series can significantly reduce cable costs and both consolidate and eliminate SAN and LAN adapters and switches. Substantial inter-rack and intra-rack cable reductions were achieved in all scenarios, showing that, depending on the configuration, inter-rack cables can be reduced by up to 90 percent, and intra-rack cables can be reduced by up to 75 percent.
Example 1: 10 Gigabit Ethernet plus 4-Gbps Fibre Channel Unconsolidated Network Compared to Consolidated LAN plus SAN
Example 1 (Figure 3) compares the TCO and ROI for a Cisco Nexus 5000 Series 10-Gbps top-of-rack installation to that for a top-of-rack installation with 10-Gbps LAN and 4-Gbps SAN I/O. Both racks have 16 servers. For unconsolidated I/O, two 10-Gbps LAN adapters and two 4-Gbps SAN adapters per server are used. For consolidated I/O, two consolidated network adapters (CNAs) per server are used.
Figure 3. Example 1 Scenarios
Table 1 compares the number of switches and cables for the two installations.
Table 1. Comparison of Switches and Cables for Unconsolidated and Consolidated I/O
Number of Switches per Rack
Number of Intra-Rack Cables
Number of Inter-Rack Cables
Capital costs are 30 percent lower for consolidated switching in this case because of the elimination of two SAN switches and associated hardware (Figure 4). This example scales with the deployment of additional racks. Note that, at this writing, CNA prices have not been completely defined. The high end of CNA pricing may reduce the CapEx benefit to as low as 15 percent.
Figure 4. Comparison of CapEx for Unconsolidated and Consolidated I/O: Example 1
In Figure 4, the "Other" categories include adapters, cables, and optical transceivers.
The quantifiable elements of the TCO analysis include hardware maintenance service and cable installation costs. Service is applied to active devices such as switches and adapters. At 8 percent per year over 4 years with a 1-year warranty, 19.5 percent is added in expense for service costs, Combining capital and service costs yield a 31% advantage of consolidated I/O vs unconsolidated I/O.
Cable installation expenses can be a factor in new data centers and also in situations where rewiring is required. Using metrics of US$50 per cable for intra-rack cabling and US$200 per cable for intra-rack cabling, cable costs do not materially affect this analysis. Other scenarios will yield more significant results.
Table 2. Summary of Example 1 TCO Benefits
Capital Equipment Savings Only
Capital Equipment plus Maintenance and Cable Installation
In this example, capital costs dominate the recognizable TCO. However, every data center is different, and some have additional quantifiable aspects, so this example is meant to be suggestive of the benefits of consolidation rather than definitive. [ Note: While OpEx costs are recognizable with respect to personnel, interviewing the Data Center operations could not provide a mechanism for verifying the OpEx].
Example 2: Gigabit Ethernet plus 4-Gbps Fibre Channel Unconsolidated Network Compared to Consolidated LAN plus SAN
Consolidated I/O does not require use of 10 Gbps at the server to compete economically. In general, the greater the number of connections per server, the more competitive consolidated I/O becomes. End-of-row (EoR) server connectivity is the connectivity most widely deployed today. With EoR, servers connect inter-rack to a modular switch positioned nearby as an access layer or edge switch. Top-of-rack (ToR) switches can be deployed as an alternative.
In this example, 100 servers are deployed, with 16 servers per rack. The unconsolidated switches are EoR modular switches. The consolidated I/O switch connects 32 servers over two racks. Two, four, and six 1-Gbps LAN connections per server plus two 4-Gbps SAN connections per servers are compared to consolidated I/O with two 10-Gbps connections per server. The size of the LAN modular switch varies to accommodate the different connectivity needs across the analysis without exacting a switch-cost penalty. Two 1-Gbps LAN connections are always embedded on the motherboard, and all switches are redundantly deployed. Table 3 summarizes the four scenarios.
Table 3. Comparison of Unconsolidated and Consolidated I/O Scenarios
Unconsolidated I/O: 2 LAN plus 2 SAN Connections per Server
Unconsolidated I/O: 4 LAN plus 2 SAN Connections per Server
Unconsolidated I/O: 6 LAN plus 2 SAN Connections per Server
Consolidated I/O: 2 Connections per Server
Number of Switches per 100 Servers
Number of Intra-Rack Cables
Number of Inter-Rack Cables
48, 56, or 64*
* Inter-rack cables for consolidated I/O are for uplinks only. Although a rack module consisting of two racks is used, the short-run cabling between each rack module is considered intra-rack. Three inter-rack cable numbers are provided as the uplink count varies in comparison to each of the three unconsolidated scenarios.
The data reveals a considerable advantage of consolidation in long-distance cable reduction, with ratios around 10:1. Switch device count increases because of the top-of-rack strategy.
Consolidation reduces CapEx by 8, 22, and 32 percent compared to the unconsolidated scenarios, illustrating the increasing benefit of consolidation with increasing number of connections (Figure 5). (Note that analysis on the basis of list price switches does not adequately reflect market conditions and can vary; this example is meant only to be suggestive of benefits available.)
Figure 5. Comparison of CapEx for Unconsolidated and Consolidated I/O: Example 2
As in Example 1, TCO improves with consideration of maintenance and cable installation (for example, a 22 percent reduction improves to 28 percent), but the analysis continues to be dominated by capital costs.
Considerations for Blade Servers
The comparisons in the two examples used rack-mount servers. Consolidated I/O will have even greater benefits for blade servers. Blade servers are constrained by form factor. Consolidated I/O will allow greater I/O loads in smaller blade server form factors and increasing I/O capability in high-end form factors. Additionally, expensive embedded blade server switches for SANs can be eliminated, and the number of LAN switches can be reduced, as can the number of pass-through blades. The products to support consolidated I/O in blade servers is not yet available but is expected soon.
A ToR switch is also useful for consolidating the multiple 10-Gbps I/O ports emanating from pass-through blades. These ports would be connected with inexpensive copper connects and aggregated and uplinked optically in the same manner as in the previous examples.
A ToR deployment is useful for embedded consolidated I/O blade servers as well. Although the port density emanating from the blade server will be less than with pass-through blades, ToR switches connecting multiple racks of blade servers can provide benefits with inexpensive copper connectivity, aggregation, and long-link transfer.
The arrival of the Cisco Nexus 5000 Series, in conjunction with the availability of cost-effective 10-Gbps LAN technology for the server, enables unified I/O in the data center. Data center administrators can now begin consolidation of infrastructure to achieve economic benefits.
As the analyses presented here show, the cost of building a 10-Gbps consolidated infrastructure using the Cisco Nexus 5000 Series can be competitive with the cost for similar, unconsolidated configurations at 1 Gbps. Thus, at little or no additional cost, and often at less cost, enterprises can get all the benefits of a 10-Gbps infrastructure. The analyses also show that as the number of connections increase, a consolidated 10-Gbps infrastructure with the Cisco Nexus 5000 Series becomes even more financially attractive.
In addition, a unified 10-Gbps fabric provides numerous benefits beyond the measurable financial ones. With the Cisco Nexus 5000 Series, data centers gain a unified network, with better interconnectivity access and faster-running applications. With a unified fabric, enterprises can have a standard, single I/O pipe allowing servers to be configured in a uniform manner, and any server can have access to storage without a need to modify its hardware configuration.
The Cisco Nexus 5000 Series can be used for SAN and LAN consolidation today or in the future: for example, it can be deployed for a LAN and latter include the SAN as virtualization needs increase.
1The term "top-of-rack" is used in a broad sense. Switches do not need to literally be placed at the top of the rack but can appear anywhere in the field of servers. Also, ToR switches do not necessarily need to connect only to servers in the rack in which they reside. For example a 40-port switch capable of consolidated I/O can easily address 32 four-rack-unit (4RU) servers distributed over four adjacent racks. The limit of connectivity is dictated by the number of ports per switch and the 10m maximum length of Twinax cable. Connection over multiple racks in this fashion is sometimes referred to as multi-rack, and multiple racks are referred to as a rack module.