Cisco UCS, Nimble Unified Flash Fabric, and VMware vSphere 6.5 Design Guide

NOTE: Works with document’s Advanced Properties “First Published” property. Click File | Properties | Advanced Properties | Custom.

Last Updated: May 30, 2017

 

 

NOTE: Works with document’s Advanced Properties “Last Updated” property. Click File | Properties | Advanced Properties | Custom.

                                                                                                                                                            

About Cisco Validated Design (CVD)

The CVD program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information visit:

http://www.cisco.com/go/designzone.

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS.  CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.  IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE.  USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS.  THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS.  USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS.  RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2017 Cisco Systems, Inc. All rights reserved.

 

Table of Contents                                                             

Executive Summary. 5

Solution Overview.. 6

Introduction. 6

Audience. 6

SmartStack Program Benefits. 6

SmartStack System Overview.. 7

New Features/Additions in this SmartStack CVD.. 8

Technical Overview.. 10

Cisco Unified Compute System... 10

Cisco UCS Differentiators. 11

Cisco UCS 5108 Blade Server Chassis. 12

Cisco UCS 6200 Series Fabric Interconnects. 13

Cisco UCS 6300 Series Fabric Interconnects. 14

Cisco UCS Fabric Extenders. 15

Cisco UCS Manager 16

Cisco UCS B-Series M4 Servers. 17

Cisco UCS C-Series M4 Servers. 18

Cisco UCS Performance Manager 18

Cisco Nexus 9000 Series Platform Switches. 19

Cisco MDS 9100 Series Fabric Switches. 20

Nimble Storage Unified Flash Fabric. 22

Solution Architecture. 28

Solution Design. 34

Compute. 34

LAN Network. 35

Storage and SAN Network. 36

Design Considerations. 38

Management Connectivity. 38

QoS and Jumbo Frames. 38

Cisco UCS C-Series Server Connectivity Options. 38

Cisco UCS Server – High Availability and Virtual Networking Design. 39

High Availability. 43

Scalability. 45

Validation. 47

Validated Hardware and Software. 47

Bill of Materials (BOM) 50

Summary. 53

About Authors. 54

Acknowledgements. 54

Cisco Validated Designs (CVDs) are systems and solutions that are pre-designed, tested and documented to facilitate and accelerate customer deployments. CVDs deliver solutions that address customer requirements and incorporate a wide range of technologies, products and best practices into the design. CVDs combine best-in-class hardware and software to optimize the value and performance a customer receives and reduce both integration and performance risks to ensure a high-performance solution. CVD documents provide a detailed design with deployment steps and validation details that customers can use as a reference for their rollouts. From design to configuration, instructions to bills of materials (BOMs), CVDs provide everything Enterprises need to deploy the solutions in a most efficient manner. Everything is clearly and precisely laid out.  An additional benefit is that all CVDs are backed by solution-level support, providing seamless support throughout the solution life cycle.

SmartStack™ solutions are a portfolio of data center infrastructure and application solutions from Cisco and Nimble Storage.

·         Cisco UCS is a preferred choice among customers, with 87 percent of Fortune 500 customers invested in Cisco UCS

·         Nimble Storage is identified as Visionary on Gartner's Magic Quadrant for general-purpose disk arrays

SmartStack combines Cisco UCS Integrated Infrastructure, Cisco Nexus family of switches with Nimble Storage Predictive Flash technology to deliver a series of pre-designed, validated and integrated solutions that address a wide range of use cases and workloads. SmartStack delivers greater performance and adaptability with a small footprint.

This document focusses on a SmartStack infrastructure solution for enterprise and cloud data centers based on Nimble Unified Flash Fabric (UFF) – a single unified storage architecture for both general purpose workloads and for high performance, latency sensitive applications. The design uses Cisco UCS B-series blade and C-series rack servers running VMware vSphere 6.5 hypervisor for compute, with unified management provided by Cisco UCS Manager on Cisco UCS 6200 Fabric Interconnects. The LAN and SAN fabric in the design comprises of Cisco Nexus 9000 Series switches and Cisco MDS Fibre Channel switches respectively. Storage is provided by a unified cluster of Nimble Storage AF7000 all flash array and Nimble CS5000 adaptive flash array.

Introduction

Organizations continue to be challenged by achieving IT agility and efficiency necessary to meet business demands. IT staff is faced with an ever-increasing burden of rolling out new applications and generating new revenue streams while maintaining existing applications and infrastructure. A critical step in addressing these challenges and modernizing the data center is a converged infrastructure platform with the following characteristics.

·         Availability: Resilient design to ensure applications and services are accessible at all times with non-disruptive operations and no single points of failure.

·         Flexibility: Architectural flexibility to support different deployment models and workloads

·         Efficiency: Facilitate infrastructure management efficiency through re-usable policies, API management and non-disruptive operations

·         Manageability: A management strategy that minimizes operational complexity

·         Scalability: Ability to expand and grow non-disruptively with investment protection where possible

·         Compatibility: Ensure compatibility of integrated components and design to minimize risk

·         Simplicity: Ability to deploy and manage with minimal operational overhead

SmartStack uses the above characteristics as design goals to deliver a production ready, converged infrastructure platform for Enterprise and cloud data centers. SmartStack integrates best-of-breed components, technologies and best practices for compute, networking, and storage to maximize performance. All SmartStack solutions are designed for high availability with no single points of failure and can accommodate disparate workloads, including different hypervisors and bare metal workloads. SmartStack designs are also flexible and can be sized to support different use cases and requirements. IT staff can use the SmartStack CVDs for design and deployment guidance, which are backed by validation and solution-level support.

This document describes the SmartStack® integrated infrastructure solution for Enterprise and cloud deployments built using Cisco UCS, VMware vSphere 6.5, Cisco Nexus and Cisco MDS switches, and both a Fibre Channel-attached Nimble Storage AF7000 all flash array as well as a Nimble CS5000 adaptive flash array.

Audience

The intended audience of this document includes, but is not limited to, sales engineers, field consultants, professional services, IT managers, partner engineering, and customers who want to take advantage of an infrastructure built to deliver IT efficiency and enable IT innovation.

SmartStack Program Benefits

SmartStack program is the result of a joint partnership between Cisco and Nimble Storage to deliver a series of infrastructure and application solutions optimized and validated on Cisco UCS, Nimble Storage arrays and Cisco Nexus switches. To be a valid SmartStack solution, customers must use Cisco UCS, Nimble Storage, and an approved application stack. Customers also require a valid support contract with Cisco and Nimble Storage.

Each SmartStack CVD will include (though not limited to) the following line of documentation:

·         CVD Design Guide with the architecture, design and best practices validated

·         CVD Deployment guide with the validated hardware and software details and step-by-step implementation details to deploy the solution

Cisco and Nimble Storage have a solid, joint support program focused on SmartStack solutions, from customer account and technical support engineers. The support alliance provided by Cisco and Nimble Storage provides customers and channel partners with direct access to technical experts who can collaborate with cross vendors and have access to the shared lab resources to resolve the potential issues. Customers receive single point-of-contact support via Cisco, with back-end engineering communication between Cisco and Nimble.

Cisco and Nimble have developed several SmartStack reference architecture-based solutions and Cisco Validated Designs targeting a range of use cases. Each design has been extensively tested, validated, and documented by Cisco and Nimble architects and engineering experts. This proven integrated infrastructure portfolio, consisting of the Cisco Unified Computing System integrated infrastructure and the Nimble Adaptive Flash platform, has been deployed by over 750 customers across a wide range of industries, including financial services, healthcare, education and service provider.

SmartStack System Overview      

SmartStack is a data center architecture for Enterprise or Cloud deployments and uses the following infrastructure components for compute, network and storage:

·         Cisco Unified Computing System (Cisco UCS)

·         Cisco Nexus and Cisco MDS Switches

·         Nimble Storage Unified Flash Fabric

The validated SmartStack design covered in this document uses the following models of the aforementioned infrastructure components:

·         Cisco UCS 5108 Blade Server chassis with 2200 Series Fabric Extenders (FEX)

·         Cisco UCS B-series Blade servers

·         Cisco UCS C-series Rack mount servers

·         Cisco UCS 6200 Series Fabric Interconnects (FI)

·         Cisco Nexus 9300 Series Platform switches

·         Cisco MDS 9100 Series Fabric switches

·         Nimble Storage AF7000 All Flash Array

·         Nimble Storage CS5000 Adaptive Flash Array

The optional and alternate components in the SmartStack solution are:

·         Cisco UCS 6300 Series Fabric Interconnects is the 3^rd generation Cisco® Unified Fabric for the Cisco UCS system, delivering both Ethernet and Fibre Channel connectivity at higher speeds than previous generation models. Cisco UCS 6300 FIs provide line-rate connectivity with low-latency, lossless, 40Gbps Ethernet and 16G Fibre Channel speeds for high-capacity data centers. Fabric Interconnects are the backbone of the Cisco Unified Computing System, delivering a unified fabric with centralized management.

·         Cisco UCS C-Series Rack Mount Server  delivers unified computing innovations and benefits to rack server to reduce total cost of ownership and increase agility with performance and density to support a wide range of workloads. 

·         Cisco UCS Central provides a scalable management platform for managing multiple, globally distributed Cisco UCS domains with consistency. Cisco UCS Central integrates with Cisco UCS Manager in each domain to provide global and uniform configuration and maintenance of service profiles and other configuration.

·         Cisco UCS Performance Manager is purpose-built data center management solution that provides a single pane-of-glass visibility of a converged heterogeneous infrastructure data center for performance monitoring and capacity planning.

·         Cisco Cloud Center, formerly Cliqr, can securely extend SmartStack resources between private and public cloud.  Application blueprints rapidly scale the user’s needs, while implementing them in a manner that is secure, and provides the controls for proper governance.

·         Cisco UCS Director replaces time-consuming, manual provisioning and de-provisioning of data center resources with automated workflows.  It supports multiple hardware vendors and multiple hypervisors to deliver heterogeneous infrastructure management.  A custom adapter is available for Nimble Storage array for integrating and managing SmartStack resources.

·         Other models of Nimble Storage Arrays, not explicitly listed in this document, are still supported in SmartStack even though they were not the models used in validating this design. SmartStack solutions support the use of any Nimble Storage array (all flash or adaptive flash) running the appropriate NimbleOS revision (as validated by this design). A key benefit of the Nimble Storage Unified Flash Fabric is that all Nimble Storage arrays run the same operating system.

The above components are integrated using component and design best practices to deliver a foundational data center architecture for enterprise and cloud data centers. SmartStack solution is designed for high availability, scale and performance necessary to support high-performance applications. SmartStack can scale-up by adding or upgrading compute, network and storage components individually or scale-out by adding multiple SmartStack units as a whole while maintaining design consistency and best practices.

New Features/Additions in this SmartStack CVD

Features and capabilities added in this SmartStack solution are:   

Cisco UCS Release 3.1.2(e): vNIC/vHBA Template Redundancy Pair, Consistent Device Naming, UCS Backup enhancements, and HTML5 User Interface Improvements.

Nimble Storage: Unified Flash Fabric with Nimble All Flash array and Predictive Flash Arrays

VMware vSphere 6.5: VMware vCenter Server Appliance 6.5 with high availability, integrated VUM, built-in backup/restore, ESXi 6.5, Network aware DRS, Host Profiles, vSphere Distributed Switch (vDS)

This section provides a technical overview of the compute, network, storage and management components of the SmartStack solution. 

Cisco Unified Compute System

The Cisco Unified Computing System™ (Cisco UCS) is a next-generation data center platform that integrates computing, networking, storage access, and virtualization resources into a cohesive system designed to reduce total cost of ownership (TCO) and increase business agility. The system integrates a low-latency; lossless 10 Gigabit Ethernet unified network fabric with enterprise-class, x86-architecture servers. The system is an integrated, scalable, multi-chassis platform where all resources are managed through a unified management domain.

The main components of the Cisco UCS are:

Compute - The system is based on an entirely new class of computing system that incorporates rack mount servers and blade servers based on Intel processors. 


Network - The system is integrated onto a low-latency, lossless, 10-Gbps unified network fabric. This network foundation consolidates LANs, SANs, and high-performance computing networks which are separate networks today. The unified fabric lowers costs by reducing the number of network adapters, switches, and cables, and by decreasing the power and cooling requirements. 


Virtualization - The system unleashes the full potential of virtualization by enhancing the scalability, performance, and operational control of virtual environments. Cisco security, policy enforcement, and diagnostic features are now extended into virtualized environments to better support changing business and IT requirements. 


Storage access – Cisco UCS system provides consolidated access to both SAN storage and Network Attached Storage over the unified fabric. This provides customers with storage choices and investment protection. Also, the server administrators can pre-assign storage-access policies to storage resources, for simplified storage connectivity and management leading to increased productivity.

Management: The system uniquely integrates all the system components, enabling the entire solution to be managed as a single entity through Cisco UCS Manager software. Cisco UCS Manager provides an intuitive graphical user interface (GUI), a command-line interface (CLI), and a robust application-programming interface (API) to manage all system configuration and operations. Cisco UCS Manager helps in increasing the IT staff productivity, enabling storage, network, and server administrators to collaborate on defining service profiles for applications. Service profiles are logical representations of desired physical configurations and infrastructure policies. They help automate provisioning and increase business agility, allowing data center managers to provision resources in minutes instead of days.

Key components of the Cisco Unified Computing System in this SmartStack architecture are:

·         Cisco UCS Manager provides unified management of all software and hardware components in the Cisco Unified Computing System and manages servers, networking, and storage from a single interface.

·         Cisco UCS 6200 Series Fabric Interconnects is a family of line-rate, low-latency, lossless, 10-Gbps Ethernet and Fibre Channel over Ethernet interconnect switches providing the management and communication backbone for the Cisco Unified Computing System.

·         Cisco UCS 5108 Blade Server Chassis supports up to eight blade servers and up to two fabric extenders in a six-rack unit (RU) enclosure.

·         Cisco UCS B-Series Blade Servers increase performance, efficiency, versatility and productivity with these Intel based blade servers.

·         Cisco UCS Adapters wire-once architecture offers a range of options to converge the fabric, optimize virtualization and simplify management.

Cisco’s Unified Compute System has revolutionizing the way servers are managed in data-center.  The next section takes a detailed look at the unique differentiators of Cisco UCS and Cisco UCS Manager.

Cisco UCS Differentiators

·         Embedded Management — Servers in the system are managed by embedded software in the Fabric Interconnects, eliminating need for any external physical or virtual devices to manage the servers.

·         Unified Fabric — There is a single Ethernet cable to the FI from the server chassis (blade or rack) for LAN, SAN and management traffic. This converged I/O results in reduced cables, SFPs and adapters – reducing capital and operational expenses of overall solution.

·         Auto Discovery — By simply inserting a blade server in the chassis or connecting a rack server to the FI, discovery and inventory of compute resource occurs automatically without any intervention. Auto-discovery combined with unified fabric enables the wire-once architecture of Cisco UCS, where compute capability of Cisco UCS can be extended easily without additional connections to the external LAN, SAN and management networks.

·         Policy Based Resource Classification — Once a compute resource is discovered by Cisco UCS Manager, it can be automatically classified to a resource pool based on policies defined. This capability is useful in multi-tenant cloud computing. This CVD showcases the policy based resource classification of Cisco UCS Manager.

·         Combined Rack and Blade Server Management — Cisco UCS Manager can manage B-series blade servers and C-series rack servers under the same Cisco UCS domain. Along with stateless computing, this feature makes compute resources truly agnostic to the hardware form factor.

·         Model based Management Architecture — Cisco UCS Manager architecture and management database is model based and data driven. An open XML API is provided to operate on the management model. This enables easy and scalable integration of Cisco UCS Manager with other management systems.

·         Policies, Pools, and Templates — The management approach in Cisco UCS Manager is based on defining policies, pools and templates, instead of cluttered configuration, which enables a simple, loosely coupled, data driven approach in managing compute, network and storage resources.

·         Loose Referential Integrity — In Cisco UCS Manager, a service profile, port profile or policies can refer to other policies or logical resources with loose referential integrity. A referred policy cannot exist at the time of authoring the referring policy or a referred policy can be deleted even though other policies are referring to it. This provides different subject matter experts from different domains, such as network, storage, security, server and virtualization the flexibility to work independently to accomplish a complex task.

·         Policy Resolution — In Cisco UCS Manager, a tree structure of organizational unit hierarchy can be created that mimics the real life tenants and/or organization relationships. Various policies, pools and templates can be defined at different levels of organization hierarchy. A policy referring to another policy by name is resolved in the organization hierarchy with closest policy match. If no policy with specific name is found in the hierarchy of the root organization, then special policy named “default” is searched. This policy resolution logic enables automation friendly management APIs and provides great flexibility to owners of different organizations.

·         Service Profiles and Stateless Computing — A service profile is a logical representation of a server, carrying its various identities and policies. This logical server can be assigned to any physical compute resource as far as it meets the resource requirements. Stateless computing enables procurement of a server within minutes, which used to take days in legacy server management systems.

·         Built-in Multi-Tenancy Support — The combination of policies, pools and templates, loose referential integrity, policy resolution in organization hierarchy and a service profiles based approach to compute resources makes Cisco UCS Manager inherently friendly to multi-tenant environment typically observed in private and public clouds.

·         Extended Memory — The enterprise-class Cisco UCS B200 M4 blade server extends the capabilities of Cisco’s Unified Computing System portfolio in a half-width blade form factor. The Cisco UCS B200 M4 harnesses the power of the latest Intel® Xeon® E5-2600 v3 and v4 Series processor family CPUs with up to 1536 GB of RAM (using 64 GB DIMMs), two solid-state drives (SSDs) or hard disk drives (HDDs), and up to 80 Gbps throughput connectivity – allowing huge VM to physical server ratio required in many deployments, or allowing large memory operations required by certain architectures like big data.

·         Virtualization Aware Network — Cisco VM-FEX technology makes the access network layer aware about host virtualization. This prevents domain pollution of compute and network domains with virtualization when virtual network is managed by port-profiles defined by the network administrators’ team. VM-FEX also off-loads hypervisor CPU by performing switching in the hardware, thus allowing hypervisor CPU to do more virtualization related tasks. VM-FEX technology is well integrated with VMware vCenter, Linux KVM and Hyper-V SR-IOV to simplify cloud management.

·         Simplified QoS — Even though Fibre Channel and Ethernet are converged in Cisco UCS fabric, built-in support for QoS and lossless Ethernet makes it seamless. Network Quality of Service (QoS) is simplified in Cisco UCS Manager by representing all system classes in one GUI panel.

Cisco UCS 5108 Blade Server Chassis

The Cisco UCS 5108 Blade Server Chassis is a fundamental building block of the Cisco Unified Computing System, delivering a scalable and flexible blade server architecture. A Cisco UCS 5108 Blade Server chassis is six rack units (6RU) high and can house up to eight half-width or four full-width Cisco UCS B-series blade servers.

For a complete list of blade servers supported, see: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-b-series-blade-servers/index.html

There are four hot-swappable power supplies that are accessible from the front of the chassis. These power supplies are 94 percent efficient and can be configured to support non-redundant, N+1, and grid-redundant configurations. The rear of the chassis contains eight hot-swappable fans, four power connectors (one per power supply), and two I/O bays that can support Cisco UCS 2000 Series Fabric Extenders. Two fabric extenders can be deployed to provide higher uplink bandwidth with redundancy. A passive mid-plane provides up to 80 Gbps of I/O bandwidth per server slot and up to 160 Gbps of I/O bandwidth for two slots. The chassis is capable of supporting future 40 Gigabit Ethernet standards.

Cisco UCS 5108 blade server chassis uses a unified fabric and fabric-extender technology to simplify and reduce cabling by eliminating the need for dedicated chassis management and blade switches. The unified fabric also reduces TCO by reducing the number of network interface cards (NICs), host bus adapters (HBAs), switches, and cables that need to be managed, cooled, and powered. This architecture enables a single Cisco UCS domain to scale up to 20 chassis with minimal complexity.

For more information, see: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-5100-series-blade-server-chassis/index.html

Figure 1       Cisco UCS 5108 Blade Server Chassis

Cisco UCS 6200 Series Fabric Interconnects

Cisco UCS Fabric Interconnects are a family of line-rate, low-latency, lossless 1/10 Gigabit Ethernet and Fibre Channel over Ethernet (FCoE), and 4/2/1 and 8/4/2 native Fibre Channel switches. Cisco UCS Fabric Interconnects are the management and connectivity backbone of the Cisco Unified Computing System. Each chassis or rack server connects to the FI using a single Ethernet cable for carrying all network, storage and management traffic. Cisco UCS Fabric Interconnects provides uniform network and storage access to servers and is typically deployed in redundant pairs.

Cisco UCS^® Manager provides unified, embedded management of all software and hardware components of the Cisco Unified Computing System^™ (Cisco UCS) across multiple chassis with blade servers, rack servers and thousands of virtual machines. The Cisco UCS Management software (Cisco UCS Manager) runs as an embedded device manager in a clustered pair fabric interconnects and manages the resources connected to it. An instance of Cisco UCS Manager with all Cisco UCS components managed by it forms a Cisco UCS domain, which can include up to 160 servers. 

The Cisco UCS Fabric Interconnect family is currently comprised of the Cisco 6100 Series, Cisco 6200 Series and Cisco 6300 Series of Fabric Interconnects.

For more information, see:  http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-6200-series-fabric-interconnects/index.html

Figure 2       Cisco UCS 6248UP Fabric Interconnect

Cisco UCS 6300 Series Fabric Interconnects

The Cisco UCS Fabric interconnects provide a single point for connectivity and management for the entire system. Typically deployed as an active-active pair, the system’s fabric interconnects integrate all components into a single, highly-available management domain controlled by Cisco UCS Manager. The fabric interconnects manage all I/O efficiently and securely at a single point, resulting in deterministic I/O latency regardless of a server or virtual machine’s topological location in the system.

Fabric Interconnect provides both network connectivity and management capabilities for the Cisco UCS system. Cisco UCS Fabric Extenders (IOM) in the blade chassis support power supply, along with fan and blade management. They also support port channeling and, thus, better use of bandwidth. The IOMs support virtualization-aware networking in conjunction with the Fabric Interconnects and Cisco Virtual Interface Cards (VIC).

Cisco UCS 6300 Series Fabric Interconnects and Cisco UCS 2304 Fabric Extenders provide a few key advantages over the existing products. FI 6300 Series and IOM 2304 support 40GbE / FCoE port connectivity that enables an end-to-end 40GbE / FCoE solution. Unified ports support 4/8/16Gbps FC ports for higher density connectivity to SAN ports.

Table 1     Cisco UCS 6200 and 6300 series Fabric Interconnects

   * Using 40G to 4x10G breakout cables          ** Requires QSA module

Fabric Interconnect 6300 Series are new 40Gbps Fabric Interconnect products that double the switching capacity of 6200 Series data center fabric to improve workload density. Two 6300 Fabric Interconnect models have been introduced supporting Ethernet, FCoE, and FC ports.

Cisco UCS 6332 Fabric Interconnects is a one-rack-unit (1RU) 40 Gigabit Ethernet, and FCoE switch offering up to 2.56 Tbps throughput and up to 32 ports. The switch has 32 40 Gbps fixed Ethernet, and FCoE ports. This Fabric Interconnect is targeted for IP storage deployments requiring high-performance 40 Gbps FCoE connectivity to Cisco MDS switches.

Figure 3       Cisco UCS 6332 Fabric Interconnect – Front and Rear

Cisco UCS Fabric Interconnects 6332-16UP is a one-rack-unit (1RU) 40 Gigabit Ethernet/FCoE switch and 1/10 Gigabit Ethernet, FCoE and Fibre Channel switch offering up to 2.24 Tbps throughput and up to 40 ports. The switch has 24 40Gbps fixed Ethernet/FCoE ports and 16 1/10Gbps Ethernet/FCoE or 4/8/16Gbps Fibre Channel ports. This Fabric Interconnect is targeted for FC storage deployments requiring high performance 16Gbps FC connectivity to Cisco MDS switches.

Figure 4       Cisco 6332-16UP Fabric Interconnect – Front and Rear

Cisco UCS Fabric Extenders

The Cisco UCS Fabric extenders multiplexes and forwards all traffic from servers in a chassis to a parent Cisco UCS Fabric Interconnect over a 10-Gbps or 40Gbps unified fabric links. All traffic, including traffic between servers on the same chassis, or between virtual machines on the same server, is forwarded to the parent fabric interconnect, where network profiles and polices are maintained and managed by the Cisco UCS Manager. The Fabric extender technology was developed by Cisco. Up to two fabric extenders can be deployed in a Cisco UCS chassis.

The Cisco UCS Fabric Extender family comprises of the Cisco UCS 2200 and 2300 series fabric extenders for the blade server chassis and the Cisco Nexus 2000 Series for the rack mount servers. Three models of Cisco UCS Fabric Extenders are available for the blade server chassis as outlined below.

·         The Cisco UCS 2304XP Fabric Extender has four 40 Gigabit Ethernet, FCoE-capable, Quad Small Form-Factor Pluggable (QSFP+) ports that connect the blade chassis to the fabric interconnect. Each Cisco UCS 2304XP has eight 40 Gigabit Ethernet ports connected through the midplane to each half-width slot in the chassis. Typically configured in pairs for redundancy, two fabric extenders provide up to 320 Gbps of I/O to the chassis.

Figure 5       Cisco UCS 2304 Fabric Extender

·         The Cisco UCS 2204XP Fabric Extender has four 10 Gigabit Ethernet, FCoE-capable, SFP+ ports that connect the blade chassis to the fabric interconnect. Each Cisco UCS 2204XP has sixteen 10 Gigabit Ethernet ports connected through the mid-plane to each half-width slot in the chassis. Typically configured in pairs for redundancy, two fabric extenders provide up to 80 Gbps of I/O to the chassis.

Figure 6       Cisco UCS 2204XP Fabric Extender

·         The Cisco UCS 2208XP Fabric Extender has eight 10 Gigabit Ethernet, FCoE-capable, SFP+ ports that connect the blade chassis to the fabric interconnect. Each Cisco UCS 2208XP has thirty-two 10 Gigabit Ethernet ports connected through the mid-plane to each half-width slot in the chassis. Typically configured in pairs for redundancy, two fabric extenders provide up to 160 Gbps of I/O to the chassis.

Figure 7       Cisco UCS 2208XP Fabric Extender

    

For more information, see: http://www.cisco.com/c/en/us/products/switches/nexus-2000-series-fabric-extenders/index.html

Cisco UCS Manager

Cisco Unified Computing System (UCS) Manager provides unified, embedded management for all software and hardware components in the Cisco UCS. Using Cisco Single Connect technology, it manages, controls, and administers multiple chassis for thousands of virtual machines. Administrators use the software to manage the entire Cisco Unified Computing System as a single logical entity through an intuitive GUI, a command-line interface (CLI), or an XML API. The Cisco UCS Manager resides on a pair of Cisco UCS 6200 or 6300 Series Fabric Interconnects using a clustered, active-standby configuration for high availability.

Cisco UCS Manager offers unified embedded management interface that integrates server, network, and storage. Cisco UCS Manger performs auto-discovery to detect inventory, manage, and provision system components that are added or changed. It offers comprehensive set of XML API for third part integration, exposes 9000 points of integration and facilitates custom development for automation, orchestration, and to achieve new levels of system visibility and control.

Service profiles benefit both virtualized and non-virtualized environments and increase the mobility of non-virtualized servers, such as when moving workloads from server to server or taking a server offline for service or upgrade. Profiles can also be used in conjunction with virtualization clusters to bring new resources online easily, complementing existing virtual machine mobility.

For more information on Cisco UCS Manager, see: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-manager/index.html

Cisco UCS B-Series M4 Servers

The enterprise-class Cisco UCS B200 M4 Blade Server extends the capabilities of Cisco’s Unified Computing System portfolio in a half-width blade form factor. The Cisco UCS B200 M4 uses the power of the latest Intel® Xeon® E5-2600 v3 and v4 Series processor family CPUs with up to 1536 GB of RAM (using 64 GB DIMMs), two solid-state drives (SSDs) or hard disk drives (HDDs), and up to 80 Gbps throughput connectivity. The Cisco UCS B200 M4 Blade Server mounts in a Cisco UCS 5100 Series blade server chassis or Cisco UCS Mini blade server chassis. It has 24 total slots for registered ECC DIMMs (RDIMMs) or load-reduced DIMMs (LR DIMMs) for up to 1536 GB total memory capacity. It supports one connector for the Cisco VIC 1340 or 1240 adapters, which provides Ethernet and FCoE.

For more information, see: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-b200-m4-blade-server/index.html

Figure 8       Cisco UCS B200 M4 Blade Server

 

Cisco VIC 1340

The Cisco UCS Virtual Interface Card (VIC) 1340 is a 40-Gbps Ethernet or 4 x 10-Gbps Ethernet, FCoE-capable modular LAN on motherboard (mLOM) designed exclusively for the B200 M3 and M4 generation of Cisco UCS B-Series Blade Servers. When used in combination with an optional port expander, the Cisco UCS VIC 1340 capabilities provides an additional 40Gbps of uplink bandwidth. The Cisco UCS VIC 1340 enables a policy-based, stateless, agile server infrastructure that can present over 256 PCIe standards-compliant interfaces to the host that can be dynamically configured as either network interface cards (NICs) or host bus adapters (HBAs). In addition, the Cisco UCS VIC 1340 supports Cisco® Virtual Machine Fabric Extender (VM-FEX) technology, which extends the Cisco UCS Fabric interconnect ports to virtual machines, simplifying server virtualization deployment and management.

For more information, see: http://www.cisco.com/c/en/us/products/interfaces-modules/ucs-virtual-interface-card-1340/index.html

Figure 9       Cisco VIC 1340

      

Cisco UCS C-Series M4 Servers

The enterprise-class Cisco UCS C220 M4 server extends the capabilities of the Cisco Unified Computing System (UCS) portfolio in a one rack-unit (1RU) form-factor. The Cisco UCS C220 M4 uses the power of the latest Intel® Xeon® E5-2600 v3 and v4 Series processor family CPUs with up to 1536 GB of RAM (using 64 GB DIMMs), 8 Small Form-Factor (SFF) drives or 4 Large Form-Factor (LFF) drives, and up to 80 Gbps throughput connectivity. The Cisco UCS C220 M4 Rack Server can be used standalone, or as integrated part of the Unified Computing System. It has 24 DIMM for up to 1536 GB total memory capacity. It supports one connector for the Cisco VIC 1225, 1227 or 1380 adapters, which provide Ethernet and FCoE.

For more information, see: http://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c220-m4-rack-server/datasheet-c78-732386.html.

Figure 10     Cisco UCS C220 M4 Rack Server

Cisco UCS Performance Manager

Cisco UCS Performance Manager can help unify the monitoring of critical infrastructure components, network connections, applications, and business services across dynamic heterogeneous physical and virtual data centers powered by Cisco UCS. Cisco UCS Performance Manager delivers detailed monitoring from a single customizable console. The software uses APIs from Cisco UCS Manager and other Cisco® and third-party components to collect data and display comprehensive, relevant information about your Cisco UCS integrated infrastructure.

Cisco UCS Performance Manager does the following:

·         Unifies performance monitoring and management of Cisco UCS integrated infrastructure 

·         Delivers real-time views of fabric and data center switch bandwidth use and capacity thresholds

·         Discovers and creates relationship models of each system, giving your staff a single, accurate view of all components

·         Provides coverage for Cisco UCS servers, Cisco networking, storage, hypervisors, and operating systems

·         Allows you to easily navigate to individual components for rapid problem resolution

An Express version of Cisco UCS Performance Manager is also available for Cisco UCS-based compute platform coverage only (physical and virtual). The Express version covers Cisco UCS servers, hypervisors, and operating systems.

Cisco UCS Performance Manager provided a scalable and highly available architecture. It can be deployed in a distributed environment across geographical locations as needed. The monitored Cisco devices in the SmartStack architecture include the following:

·         Cisco UCS domain components, including Cisco UCS Mini and Cisco UCS Central software

·         Cisco Nexus® 9000 Series Switches

·         Cisco MDS 9000 Series Multilayer Switches

·         Cisco Nexus 2000 Series Fabric Extenders

Cisco Nexus 9000 Series Platform Switches

The Cisco Nexus 9000 family of switches offers both modular (9500 switches) and fixed (9300 switches) 1/10/40/100 Gigabit Ethernet switch configurations designed to operate in one of two modes:

·         Application Centric Infrastructure (ACI) mode that uses an application centric policy model with simplified automation and centralized management

·         Cisco NX-OS mode for traditional architectures – the SmartStack design in this document uses this mode

Architectural Flexibility

·         Delivers high performance and density, and energy-efficient traditional 3-tier or leaf-spine architectures

·         Provides a foundation for Cisco ACI, automating application deployment and delivering simplicity, agility, and flexibility

Scalability

·         Up to 60-Tbps of non-blocking performance with less than 5-microsecond latency

·         Up to 2304 10-Gbps or 576 40-Gbps non-blocking layer 2 and layer 3 Ethernet ports

·         Wire-speed virtual extensible LAN (VXLAN) gateway, bridging, and routing

High Availability

·         Full Cisco In-Service Software Upgrade (ISSU) and patching without any interruption in operation

·         Fully redundant and hot-swappable components

·         A mix of third-party and Cisco ASICs provide for improved reliability and performance

Energy Efficiency

·         The chassis is designed without a mid-plane to optimize airflow and reduce energy requirements

·         The optimized design runs with fewer ASICs, resulting in lower energy use

·         Efficient power supplies included in the switches are rated at 80 Plus Platinum

Investment Protection

·         Cisco 40-Gb bidirectional transceiver allows for reuse of an existing 10 Gigabit Ethernet cabling plant for 40 Gigabit Ethernet

·         Designed to support future ASIC generations

·         Easy migration from NX-OS mode to ACI mode

For more information, refer to: http://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/index.html

Cisco MDS 9100 Series Fabric Switches

The Cisco® MDS 9148S 16G Multilayer Fabric Switch is the next generation of the highly reliable, flexible, and low-cost Cisco MDS 9100 Series switches. It combines high performance with exceptional flexibility and cost effectiveness. This powerful, compact one rack-unit (1RU) switch scales from 12 to 48 line-rate 16 Gbps Fibre Channel ports. Cisco MDS 9148S is powered by Cisco NX-OS and delivers advanced storage networking features and functions with ease of management and compatibility with the entire Cisco MDS 9000 Family portfolio for reliable end-to-end connectivity. Cisco MDS 9148S is well suited as a:

·         Top-of-rack switch in medium-sized deployments

·         Edge switch in a two-tiered (core-edge) data center topology

·         Standalone SAN in smaller departmental deployments

The main features and benefits of Cisco MDS 9148S are summarized in the table below.

Table 2     Cisco MDS 9148S Features and Benefits

Nimble Storage Unified Flash Fabric

The Nimble Storage Unified Flash Fabric enables enterprise IT organizations to implement a single architectural approach to dynamically cater to the needs of varying workloads, driven by the power of predictive analytics. Predictive Flash arrays are the only storage platform that optimizes across performance, capacity, data protection, and reliability within a dramatically smaller footprint.

Predictive Flash is built upon NimbleOS and InfoSight™, the company’s cloud-connected predictive analytics and management system. NimbleOS scales performance and capacity seamlessly and independently. InfoSight leverages the power of deep data analytics to provide customers with precise guidance on the optimal approach to scaling flash, CPU, and capacity around changing application needs, while ensuring peak storage health.

The Nimble Storage All Flash Arrays

The AF series array family starts at the entry level with the AF1000, then expands up to the AF3000, AF5000, AF7000, and then finally, at the high end, to the AF9000. All flash arrays can be non-disruptively upgraded from the entry level, all the way to the high-end array model.

Figure 11     The Nimble All Flash Array Family

This CVD specifically uses the Nimble Storage AF7000 all flash array. The AF7000 is designed to deliver up to 230,000 IOPS at sub-millisecond response times.  The AF7000 delivers the best price for performance value within the Nimble Storage all flash array family.

Each 4U chassis supports 48 SSD drives with additional flash capacity available via expansion shelves.  The storage subsystem uses SAS 3.0 compliant 12Gbps SAS connectivity, with an aggregated bandwidth of 48Gbps across 4 channels per port.  The AF7000 all flash array can scale flash capacity up to 323TB raw, extensible to 1.2PB effectively with a 5:1 data reduction through data deduplication and compression.

For additional information, refer to: https://www.nimblestorage.com/technology-products/all-flash-arrays.

Figure 12     Nimble All Flash Array
 

 

Nimble Storage Adaptive Flash Arrays

The CS series array family starts at the entry level with the CS1000/CS1000H, then expands up to the CS3000, CS5000, and then finally, at the high end, to the CS7000. Adaptive flash arrays can be nondisruptively upgraded from the entry level, all the way to the high-end array model.

Figure 13     The Nimble Adaptive Flash Array Family

     

 

This CVD specifically uses the Nimble Storage CS5000 adaptive flash array. The CS5000 is designed to support up to 120,000 IOPS with 21TB to 882TB of raw capacity.  Effective capacity scales to 1.4PB (assumes a conservative 2:1 data reduction multiplier).

Each 4U chassis supports 21 HDDs with 6 SSDs (ratio of HDD to SSD is configurable), with additional HDD and flash capacity available via expansion shelves. The array has 24 drive bays. A single HDD occupies a single drive bay while the flash drives use the dual drive carrier.  The dual drive carrier allows up to two flash drives to be installed in a single bay (each flash drive can be individually removed). The storage subsystem uses SAS 3.0 compliant 12Gbps SAS connectivity, with an aggregated bandwidth of 48Gbps across 4 channels per port.

For additional information, refer to: https://www.nimblestorage.com/technology-products/adaptive-flash-arrays.

 

Figure 14     Nimble Adaptive Flash Array

 

Nimble Storage InfoSight

Using systems modeling, predictive algorithms, and statistical analysis, InfoSight™ solves storage administrators’ most difficult problems. InfoSight also ensures storage resources are dynamically and intelligently deployed to satisfy the changing needs of business-critical applications, a key facet of Nimble Storage’s Predictive Flash platform.  At the heart of InfoSight is a powerful engine comprised of deep data analytics applied to telemetry data gathered from Nimble arrays deployed across the globe. More than 30 million sensor values are collected per day per Nimble Storage array. InfoSight automatically collects and analyzes storage performance every four hours using predictive analytics to correlate vast amounts of data to known issues in order to identify and fix problems before they become outages.

The InfoSight Engine transforms the millions of gathered data points into actionable information that allows customers to realize significant operational efficiency through:

·         Maintaining optimal storage performance

·         Projecting storage capacity needs

·         Proactively monitoring storage health and getting granular alerts

·         Proactively diagnoses and automatically resolves complex issues, freeing up IT resources for value-creating projects

·         Ensures a reliable data protection strategy with detailed alerts and monitoring

·         Expertly guides storage resource planning, determining the optimal approach to scaling cache, IOPS to meet changing SLAs

·         Identifies latency and performance bottlenecks through the entire virtualization stack

For more information, refer to: http://www.nimblestorage.com/technology-products/infosight

NimbleOS Architecture

The Nimble Storage operating system, NimbleOS is based on its patented CASL™ architecture. CASL is a flash friendly architecture that is pointer-based and natively handles variable block sizes, which minimizes write amplification and write wear on SSDs.  NimbleOS and InfoSight™ form the foundation of the Predictive Flash platform, which allows for the dynamic and intelligent deployment of storage resources to meet the growing demands of business-critical applications.

 

In-Line Compression

NimbleOS uses fast, in-line compression for variable application block sizes to decrease the footprint of inbound write data by as much as 75 percent. Once there are enough variable-sized blocks to form a full write stripe, NimbleOS writes the data to SSD (note that, the write is acknowledged once placed in NVDIMM). All reads will come from SSD and as a result be very fast. Written data is protected with triple-parity RAID.

In-Line Deduplication

Nimble Storage all flash arrays include in-line data deduplication in addition to in-line compression. The combination of in-line deduplication and in-line compression delivers the most comprehensive data reduction capability that allows NimbleOS to minimize the data footprint on SSD, maximize usable space, and greatly minimize write amplification.

Thin-Provisioning and Efficient Capacity Utilization

Capacity is consumed as data gets written. NimbleOS efficiently reclaims free space on an ongoing basis, preserving write performance with higher levels of capacity utilization. This avoids fragmentation issues that hamper other architectures.

Acknowledging Writes at Memory Speed

Once writes are placed in NVDIMM (made persistent and mirrored to the passive partner controller) they are acknowledged back to the host and sent to SSD at a later time (generally when there is a full stripe to be written).  As a result, writes to a Nimble Storage array are acknowledged at memory speeds.

Maximizing Flash Write Cycles

By sequencing random write data, NimbleOS sends full stripes of data to SSD.  By compressing and deduplicating the data in-line (versus post-process), the data footprint on SSD as well as the block changes are kept to an absolute minimum.  Additionally, the data being sent to disk is of variable block size, which means NimbleOS doesn’t have to break up data into smaller, fixed-sized chunks, to be placed on SSD.  As a result, data is efficiently sent to SSD. This allows Nimble Storage arrays to maximize the deployable life of a flash drive by minimizing write wear on the flash cells.  This is demonstrated by the use of the TLC SSDs in Nimble Storage arrays, which are covered by a 7-year warranty.

Read Performance

NimbleOS and all flash arrays deliver sub-millisecond read latency and high throughput across a wide variety of demanding enterprise applications.  This is because all reads come from SSD.

Efficient, Fully Integrated Data Protection

All-inclusive snapshot-based data protection is built into the Adaptive Flash platform. Snapshots and production data reside on the same array, eliminating the inefficiencies inherent to running primary and backup storage silos. And, InfoSight ensures that customers’ data protection strategies work as expected through intuitive dashboards and proactive notifications in case of potential issues.

SmartSnap: Thin, Redirect-on Write Snapshots

Nimble snapshots are point-in-time copies capturing just changed data, allowing three months of frequent snapshots to be easily stored on a single array. Data can be instantly restored, as snapshots reside on the same array as primary data.

SmartReplicate: Efficient Replication

Only compressed, changed data blocks are sent over the network for simple and WAN-efficient disaster recovery. 

Zero-Copy Clones

Nimble snapshots allow fully functioning copies, or clones of volumes, to be quickly created. Instant clones deliver the same performance and functionality as the source volume, an advantage for virtualization, VDI, and test/development workloads.

Application-Consistent Snapshots

Nimble enables instant application/VM-consistent backups using VSS framework and VMware integration, using application templates with pre-tuned storage parameters.

SmartSecure: Flexible Data Encryption

NimbleOS enables individual volume level encryption with little to no performance impact. Encrypted volumes can be replicated to another Nimble target, and data can be securely shredded at the volume level of granularity.

REST API

NimbleOS has a RESTful API that allows for powerful, secure and scriptable management of storage objects.  Using this API, an administrator can use the scripting or orchestration to interact with infrastructure components in a well-defined, repeatable manner.

The SmartStack solution delivers an integrated infrastructure platform that incorporates compute, network and storage best practices from Cisco and Nimble to deliver a resilient, scalable and flexible data center architecture for enterprise and cloud deployments.

The key elements of the SmartStack architecture are:

·         Cisco Unified Computing System (UCS) – provides the compute resources

·         Cisco UCS 6200 Series Fabric Interconnects (FI) – manages the compute domain, provides access layer switching and unified SAN and LAN access

·         Cisco Nexus 9300 Switches – forms the datacenter LAN and provides connectivity between Cisco UCS domains, end users and other LAN networks

·         Cisco MDS Fabric Switches – forms the SAN fabric and provides Fibre Channel (FC) connectivity to storage

·         Nimble Storage AF7000 All Flash array and Nimble Storage CS5000 Adaptive Flash array – addresses all storage needs, including SAN boot of hosts

·         VMware vSphere - Hypervisor

The compute resources supported in this architecture are Cisco UCS B-series blade servers and C-series rack mount servers. Cisco UCS B-series servers are housed in a Cisco UCS 5108 blade server chassis that can support up to eight half-width blades or four full-width blades. Cisco UCS servers support different models of converged network adapters (CNAs) and Virtual Interface Cards (VICs) that converge LAN and SAN traffic onto a single interface. These cards eliminate the need for multiple physical network interface cards (NICs) and host bus adapters (HBAs) on a server for LAN and SAN connectivity. Cisco VICs can be virtualized to create up to 256 virtual interfaces (vNIC or vHBA) and provide up to 80Gbps of uplink bandwidth per server. The virtualized interfaces are presented as standards-compliant PCIe endpoints to the host. Cisco VICs are available for both B-Series and C-Series form factors. VIC is available as a Modular LAN on Motherboard (mLOM) or Mezzanine Slot card for the blade servers and as a half-height PCI Express (PCIe) card or MLOM card for the rack-mount servers. For more information on the different models of adapters available on Cisco UCS servers, see:  http://www.cisco.com/c/en/us/products/interfaces-modules/unified-computing-system-adapters/index.html

Cisco UCS servers are typically deployed with a single VIC card for unified network and storage access. The single VIC connects into a redundant unified fabric provided by a pair of Cisco UCS Fabric Interconnects. Fabric Interconnects provide a lossless and deterministic Fibre Channel over Ethernet (FCoE) fabric for the UCS servers connected to it. Fabric Interconnects are an integral part of the Cisco Unified Computing System, providing unified management and connectivity to all attached blades, chassis and rack servers. Cisco UCS 6200 series is the second generation fabric interconnect providing a unified 10GbE FCoE fabric with 10GbE uplinks for LAN/IP based storage and 2/4/8G FC uplinks for SAN access. Cisco UCS 6300 series Fabric Interconnects is the newest model of fabric interconnect, providing a 40GbE unified FCoE fabric with 40GbE uplinks for LAN/IP based storage and4/8/16G FC uplinks for SAN access.

On the blade server chassis, the servers connect into the fabric interconnect through Fabric Extenders (FEX) or Input Output Modules (IOM). Fabric Extenders extend the unified fabric from the FI to the chassis and serves as a consolidation point for all blade server I/O traffic. FEX is managed as an extension of the fabric interconnects, simplifying diagnostics, cabling and operations with a single point of management and policy enforcement. This approach reduces the overall infrastructure complexity and enables the Cisco UCS to scale with multiple blade and rack servers managed as a single, highly available management domain. For additional benefits of FEX, see: http://www.cisco.com/c/en/us/solutions/data-center-virtualization/fabric-extender-technology-fex-technology/index.html.

A Cisco UCS 5100 Series blade server chassis can support up to two fabric extenders that fit into the back of the chassis. Fabric Extenders operate in an active-active mode for data forwarding and active-passive mode for management functions. For the UCS blade server chassis, three FEX models are currently available. For each FEX model, the number of external 10GbE/40GbE links to fabric interconnects and internal links to blade servers within the chassis are shown in the table below. The maximum uplink I/O bandwidth for each FEX is also provided.

For additional info on the blade server FEX models, see: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-6300-series-fabric-interconnects/datasheet-listing.html

Table 3     Blade Server Fabric Extender Models

 

On the rack mount servers, the Cisco VIC can connect either directly or through a FEX into the fabric interconnects. Using fabric extenders will consolidate all I/O traffic coming from the rack mount servers connected to it. Unlike blade servers where the FEX modules fit into the back of the chassis, rack mount servers require a standalone platform FEX. Two currently available options are the Cisco Nexus 2200 and 2300 Series platforms. The discovery, configuration, and management are same as that of a blade server FEX once the rack mount servers are physically cabled to the FEX. For more info on the Cisco Nexus 2000 Series FEX platform, see: http://www.cisco.com/c/en/us/products/switches/nexus-2000-series-fabric-extenders/index.html

The fabric extenders can connect to the unified fabric using multiple Ethernet (10GbE/40GbE) links – the number of links determines the uplink I/O bandwidth available through each FEX. Each FEX connect into a fabric interconnect using either 1, 2, 4 or 8 Ethernet links depending on the model of FEX used. The links can be deployed as independent links (discrete Mode) or grouped together using link aggregation (port channel mode). In discrete mode, each server is pinned to a FEX link going to a port on the fabric interconnect and if the link goes down, the server’s connection also goes down through the FEX link. In port channel mode, the flows from the server will be redistributed across the remaining port channel members. This is less disruptive overall and therefore port channel mode is preferable.

Figure 15     FEX Connection in Discrete Mode and PortChannel Mode

    

Cisco UCS offers multiple options for the server (blade, rackmount, models), network adapters, FEX and Fabric Interconnects that make up a Cisco UCS system, giving customers the flexibility to select the components that meet the needs of the customer.

In this SmartStack architecture, a pair of Cisco UCS 6200 series fabric interconnects form the unified fabric. Cisco UCS B-series servers connect into the fabric through Cisco UCS 2200 series FEX while the UCS C-series servers attach directly without a FEX as shown below.

Figure 16     SmartStack Architecture - High Level

The compute resources in this architecture are grouped into an infrastructure layer and application data layer. Servers in the infrastructure layer are dedicated for hosting virtualized infrastructure services that are necessary for deploying and managing the entire data center. Servers in the application data layer are for hosting business applications and supporting services that Enterprise users will access to fulfill their business function.

The architecture can support any hypervisor but VMware vSphere was used to validate this architecture. High availability features available in VMware vSphere was leveraged to provide virtualization layer resiliency during validation. 

A SmartStack architecture can support either iSCSI or Fibre Channel (FC) based access to a Nimble Storage array. An iSCSI based access can be used if an end-to-end unified fabric architecture from compute to storage is preferred, but the focus of this CVD will be on a fibre channel based access to Nimble Storage, specifically the AF7000 All Flash array and the CS5000 Adaptive Flash array. In this architecture, the traffic will traverse a unified fabric between Cisco UCS and Fabric Interconnects but then diverge onto separate LAN and SAN networks. For other SmartStack design options, including iSCSI and FC connectivity spanning both adaptive flash and all flash arrays, see the following CVDs that are located in the Cisco Design Zone:

·         CS300 iSCSI Design Guide: http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/smartstack_cs300_mini.html

·         CS300 iSCSI Deployment Guide: http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/smartstack_cs300_mini_deploy.html

·         CS700 FC Design Guide: http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/smartstack_cs700_design.html

·         CS700 FC Deployment Guide: http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/smartstack_cs700_deploy.html

·         AF7000 FC Design Guide: http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/smartstack_afa_design.html

·         AF7000 FC Deployment Guide: http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/smartstack_afa.html

Nimble Storage’s all flash and adaptive flash arrays are built upon Nimble’s Predictive Flash CPU-driven architecture that provides enterprises with the ability to achieve peak flash-based performance and capacity.  The Predictive Flash platform is based on NimbleOS and InfoSight™, a cloud-based management and support system.  NimbleOS maximizes the capabilities of SSD to deliver the best flash performance while achieving flash-friendly results with intelligent, space efficient, variable block, data layout on SSD.  NimbleOS and InfoSight™ form the foundation of the Predictive Flash platform, which allows for the dynamic and intelligent deployment of storage resources to meet the growing demands of business-critical applications.

Nimble Storage AF7000 used in this architecture is designed to deliver the best flash price for performance capability.  The base array supports up to 24 solid-state drives (SSD). The maximum raw flash capacity of the base array is 323TB, with an effective capacity of around 1.2PB (applying a 5:1 data reduction). The AF7000 supported up to 2 all flash expansion shelves which can be utilized to meet the performance and capacity needs of enterprise data centers. For more information on the Nimble Storage AF7000 all flash array, see https://www.nimblestorage.com/technology-products/all-flash-array-specifications.

The Nimble Storage CS5000 used in this architecture is designed to deliver large capacity with highly scalable flash caching capacity.  The base array supports up to 21 HDDs with 6 flash drives (cache sizing is configurable).  The 21TB to 882TB of raw capacity can scale to an effective capacity of 1.4PB (applying a 2:1 data reduction). Up to 6 HDD expansion shelves and 1 all flash expansion shelf, or 2 all flash expansion shelves, can be added to scale capacity and flash cache size. For additional information on the Nimble Storage CS5000 adaptive flash array, see https://www.nimblestorage.com/technology-products/adaptive-flash-arrays.

This SmartStack architecture utilizes a dedicated SAN network for block storage traffic.  The figure below shows a unified fabric (blue lines) between Cisco UCS servers and Fabric Interconnects which then splits into separate SAN (green lines) and LAN (orange lines) networks. As stated earlier, if a unified fabric is required or preferable end-to-end from compute to storage, iSCSI block storage access can be used.  FCoE is another option but would require Cisco Nexus 5000 Series switches to be used instead of Cisco Nexus 9000 Series switches to support FCoE and to provide FCoE to FC connectivity to the Nimble arrays. However, this SmartStack solution provides a highly scalable FC based storage access with a dedicated SAN network to address the needs of Enterprises and Cloud deployments.

The SmartStack architecture is highly resilient with redundant Unified, LAN and SAN fabrics that include both component and link level redundancy when accessing these fabrics. The unified fabric and LAN connections are 10Gb Ethernet links. The SAN connections are 8Gb FC links from Cisco UCS FI to the SAN Fabric and 16Gb FC links from the SAN Fabric to Nimble Storage AF7000 array. Multiple 10GbE links and FC links are deployed in the SmartStack architecture with Link Aggregation Control Protocol (LACP) based aggregation to provide higher aggregate bandwidth and resiliency across the different fabrics. Use of LACP is strongly recommended when available to improve failover convergence time and protection from misconfigurations. Static or manual bundling is an alternative but is less preferable and therefore not used in this architecture.

The data center LAN network in this SmartStack architecture uses a pair of Cisco Nexus 9300 switches that serve as the access/aggregation layer of the data center network. In this design, the Cisco Nexus 9300s provide reachability to users and other parts of the network. In larger deployments, an additional layer of hierarchy can be added using Cisco Nexus 9500 series switches as an aggregation/core layer in a classic data center tiered design or as a spine in a spine/leaf design. Cisco Nexus 9000 family of switches can operate in one of two modes: Application Centric Infrastructure (ACI) for newer cloud architectures or in Cisco NX-OS standalone mode for the more traditional data center architectures. In the SmartStack architecture, the Cisco Nexus 9300s are deployed in NX-OS standalone mode and provide investment protection by enabling a pathway to Cisco’s Application Centric Infrastructure (ACI). Cisco ACI was designed to help Enterprises transition their data centers to a cloud architecture that is dynamic, where applications can be quickly deployed and scaled, and adapt with the needs of the business. To enable this, Cisco ACI provides a flexible, scalable, application centric, policy-based framework based on open APIs that accelerate and simplify data center deployments while providing centralized orchestration, visibility and management.

Virtual Port Channel (vPC) or link-aggregation capabilities of the Cisco Nexus 9000 family of switches are used on the network links to Cisco UCS Fabric Interconnects. A vPC allows links that are physically connected to two different Cisco Nexus 9000 Series devices to appear as a single PortChannel. vPC provides Layer 2 multipathing with load balancing by allowing multiple parallel paths between nodes that result in increased bandwidth and redundancy. A vPC-based architecture is therefore highly resilient and robust and scales the available Layer 2 bandwidth by using all available links. Other benefits of vPCs include:

·         Provides a loop-free topology

·         Eliminates Spanning Tree Protocol blocked ports

·         Uses all available uplink bandwidth

·         Provides fast convergence for both link and device failures

·         Provides higher aggregate bandwidth by adding links – same as Port Channels

·         Helps ensure high availability

The SAN network in this SmartStack architecture is both 8Gb FC (FI to Cisco MDS) and 16Gb FC (Cisco MDS to Nimble Storage arrays) FC and uses Cisco MDS 9100 switches to form the SAN Fabric.

The aggregate throughput of the two fabrics depends on the number of 10GbE and 8Gb/16Gb FC links deployed to meet the throughput needs of a given customer deployment. The figure above shows a 4 x 10GbE access to the unified fabric from Cisco UCS blade server chassis and 10GbE from the rack mount server. This can be scaled up to max of 8x10 GbE on the Cisco UCS blade server chassis when using the 8-port Fabric Extender. The LAN and SAN fabric provides a 2x10 GbE and 4x8Gb FC access respectively from each FI. This can also be scaled higher by adding additional 10GbE and 8Gb FC links. 16Gb FC links are used between the Nimble Storage array and Cisco MDS 9148 FC switches. For higher access bandwidth from a given Cisco UCS domain, Cisco UCS 6300 Series FIs can be deployed to enable 40Gb Ethernet and Fibre Channel over Ethernet (FCoE) connectivity within the Unified Fabric with 40GbE uplinks to Cisco Nexus 9000 switches and 16G FC connectivity to Cisco MDS 9000 SAN switches.

Link aggregation or port channeling is also used on the SAN network between Cisco UCS Fabric Interconnects and Cisco MDS 9100 Series switches. A SAN port channel provides a single point-to-point connection with higher aggregate bandwidth by distributing traffic among all individual links in the channel. Load balancing maintains optimum bandwidth utilization. Two primary load balancing methods exist on FC links – exchange or flow based. FC header contains an Originator Exchange ID (OX_ID) and a Responder Exchange ID (RX_ID). These fields are unique per exchange. Frames with different OX_ID or RX_ID belong to different exchanges. By default, Cisco MDS switches and Fabric interconnects use exchange based load balancing based on SRC-DST-OX_ID for a given virtual SAN. Using this method, frames belonging to the same exchange will always take the same physical link in the port-channel to keep the frames in sequence but the next exchange between the same endpoints can use one of the other links in the channel. Also, if additional capacity is needed, the number of links between these platforms can be increased to a max. of 16 per port-channel.

The SmartStack platform will be managed locally using Cisco UCS Manager, VMware vCenter and Nimble Management software. The storage array will also be remotely monitored from the cloud using Nimble InfoSight™ to provide insights into I/O and capacity usage, trend analysis for capacity planning and expansion, and for proactive ticketing and notification when issues occur.

The end-to-end SmartStack design is shown in Figure 17.

Figure 17     SmartStack Design

Compute

SmartStack design supports both Cisco UCS B-series and C-series deployments connecting to either 6200 or 6300 series fabric interconnects. For validation, 4x Cisco B200M4 half-width blades with Cisco 1340 VIC adapter and a Cisco C220 M4 rack mount server with Cisco 1227 VIC adapter connecting to a pair of 6248 fabric interconnects were used. The hypervisor used in the design is VMware ESXi 6.5. Features available in Cisco UCS and VMware (for example, VMware clustering, high-availability) are leveraged where possible to provide a robust and highly manageable design. The blade server chassis is connected via FEX modules on the back of the chassis to a pair of Cisco UCS 6248 FIs. A pair of Cisco 2204 XP fabric extenders is used in this design. The FEX to FI connectivity uses 8x10GbE links, 4 from FEX-A to FI-A and 4 from FEX-B to FI-B to provide an aggregate access bandwidth of 80Gbps to the unified fabric. The FIs are connected to LAN and SAN network using 10GbE and 8Gb FC links. The FI provides 40Gbps of aggregate bandwidth to the LAN network and 64Gbps to the SAN network. Link aggregation using port channels are used on the unified fabric FEX-FI connections and virtual port channels on the Cisco Nexus-FI LAN connections.  The hosts were allocated to a VMware DRS and HA enabled clusters supporting either infrastructure services (for example, vSphere Virtual Center, Microsoft Active Directory) that manage the virtualized environment or application clusters that host the VMs users access to perform business functions.  

LAN Network

The LAN network design is shown in Figure 18.

Figure 18     SmartStack LAN Fabric Design

The LAN network provides network reachability to the applications hosted on Cisco UCS servers in the data center. The infrastructure consists of a pair of Cisco Nexus 9372 PX switches deployed in NX-OS standalone mode. Redundant 10Gbps links from each Cisco Nexus switch are connected to ports on each FI and provide 20Gbps of bandwidth through each Cisco Nexus. Virtual PortChannels (vPCs) are used on the Cisco Nexus links going to each FI. VLAN trunking is enabled on these links as multiple application data VLANs, management and vMotion traffic needs to traverse these links. Jumbo Frames are also enabled in the LAN network to support vMotion between multiple Cisco UCS domains. See Design Practices section for other Cisco Nexus 9000 best practices in the design.

Storage and SAN Network

The Nimble storage design is shown in Figure 19.

Figure 19     SmartStack Storage Design

   

SmartStack design uses a Nimble Storage AF7000 all flash array and CS5000 adaptive flash array to provide block storage. The AF7000 used in this validated design contains a base configuration with 11.25TB of raw flash capacity. The CS5000 used in this validated design contains a base configuration with 16.30TB of raw HDD with 1.44TB of raw flash capacity.

Any mix of adaptive flash and all flash arrays can be deployed in an up to 4 array scale-out cluster to increase performance, capacity, and to provide tiering of data between all flash and hybrid flash storage.

Each Nimble Storage controller supports up to 3 FC interface cards, each with dual 16G FC ports. This SmartStack design uses 16Gb fabric connectivity between the storage array and Cisco MDS switches, with two FC interface cards providing 64G of FC bandwidth per controller. For additional FC bandwidth, a third FC card can be deployed on each controller but this interface is typically used for 10GbE connections to other arrays in a scale-out cluster for data replication traffic. The links between a pair of Cisco MDS and Fabric Interconnect switches are aggregated using 4x8G FC links to deliver 32G of bandwidth across the SAN fabric to each controller.

This SmartStack design uses FC SAN boot to boot the servers. The Service Profile used to configure and deploy Cisco UCS servers is configured to include a boot policy that points to the Nimble Storage array.  The boot policy specifies a primary and secondary SAN path to the two controllers on the array where the boot volumes reside. A second boot policy is also configured but with the primary and secondary paths reversed from that of the first boot profile. The second boot policy is used to load balance SAN boot across different paths when multiple servers are booting. This is an optional aspect of the design that can be helpful in larger deployments for distributing load when multiple servers have to be simultaneously booted. Each server has a dedicated boot volume (40GB) on the Nimble storage array. Nimble Storage arrays provide an ACL at the initiator level to only allow connections from the appropriate Cisco UCS blade. During the initial SAN boot, the server attaches to all primary and secondary connections to both active and standby controller. This provides for normal boot operations even when a controller or primary path is offline. The hosts are configured with the Nimble Connection Manager and Path Selection Policy which optimize MPIO settings. This will allow for proper FC path management and failover connectivity.  

The SAN network provides fibre channel connectivity to the Nimble storage array and consists of a pair of Cisco MDS switches. The Cisco MDS switches form completely separate fabrics (SAN fabric A, SAN fabric B) and use a dual vSAN (vSAN-A, vSAN-B) design to provide two redundant and completely diverse paths to the Nimble arrays.

Link aggregation using port channels are used to aggregate 4 x 8G FC links to provide 32G of FC bandwidth on each SAN Fabric between Cisco FI and Cisco MDS switches. Link aggregation is not used on the links to Nimble array but 2 links from each SAN fabric connects to both controllers to provide 32G of FC bandwidth to the active controller. Four links from the SAN fabric, 2 from SAN Fabric A and 2 from SAN Fabric B, also connect to the backup controller so that both controllers have 32B FC access to the SAN fabric.

Cisco MDS switches are deployed with N-Port ID Virtualization (NPIV) enabled to support the virtualized environment running on Cisco UCS blade and rack servers. NPIV is necessary to provide isolation in virtualized environments where multiple virtual machines are running on a single server but a LUN needs to be presented to only one VM and not all VMs running on the server. Without NPIV, LUNs would be presented to the host and as a result, all VMs running on that host. To support NPIV on the Cisco UCS servers, the Cisco UCS Fabric Interconnects that connect the servers to the SAN fabric, are enabled for N-Port Virtualization (NPV) mode by configuring to operate in end-host mode (as opposed to FC switching mode). NPV enables Cisco FIs to proxy fabric login, registration and other messages from the servers to the SAN Fabric without being a part of the SAN fabric. This is important for keeping the limited number of Domain IDs that Fabric switches require to a minimum.

The design also uses the following best practices:

·         Jumbo frames on unified fabric links between Cisco UCS and fabric interconnects

·         QoS policy for traffic prioritization on the unified fabric

·         Port-channels with multiple links are used in both the unified fabric and SAN network for higher aggregate bandwidth and redundancy

·         Zoning is single initiator (vHBA) to multiple targets

The following sections provide more details on the connectivity and high availability aspects of this design.

Design Considerations

Management Connectivity

This SmartStack design uses a separate out-of-band management network to configure and manage compute, storage and network components in the solution. Management ports on each physical device (Cisco UCS FI, Nimble CS700 Array Controllers, Cisco Nexus and Cisco MDS switches) in the solution are connected to a separate, dedicated management switch.

Management access is also needed to ESXi hosts, vCenter VM and other management VMs but this is done in-band in this design. However, if out-of-band management to these components are required, the disjoint layer 2 feature available in Cisco UCS running end-host mode can be used. This would require additional uplink port(s) on the Cisco UCS FI to connect to the management switches. Additional out-of-band management VLANs and vNICs will also be needed and associated with the uplink ports. The additional vNICs are necessary since a server vNIC can only be associated with a single uplink.

QoS and Jumbo Frames

Cisco UCS, Cisco Nexus, and Cisco MDS switches in the SmartStack solution provide QoS policies and features for handling congestion and traffic spikes that can occur in a SmartStack environment. SmartStack support different types of traffic (For example, vMotion, FCOE) and the QoS capabilities in these components can alleviate and provide the priority that certain types of traffic require.

SmartStack design also uses jumbo frames with an MTU of 9000 Bytes across the LAN and Unified Fabric links. Jumbo frames increase the throughput between devices by enabling larger sized frames to be sent and received on the wire while reducing the CPU resources necessary to process them. Jumbo frames were enabled during validation on the LAN network links in the Cisco Nexus switching layer and on the Unified Fabric links.

Cisco UCS C-Series Server Connectivity Options

As of Cisco UCS Manager Release 2.2, the Cisco UCS C-Series servers can be connected to the Cisco UCS Fabric Interconnects using one of the two design options and still can leverage the benefits of unified management that the Cisco UCS Manager provides.  This CVD uses Cisco UCS Manager Release 3.1.

Direct Attached Design

Customers can directly connect the Cisco UCS C-series rack servers to Cisco UCS Fabric Interconnects as of Cisco UCS Manager Release 2.2 and above releases.  This CVD uses Cisco UCS Manager Release 3.1.

This design is well suited for smaller deployments with limited scale.  For more details on this connectivity option, see: http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c-series_integration/ucsm2-2/b_C-Series -Integration_UCSM2-2/b_C-Series-Integration_UCSM2-2_chapter_0110.html#reference_EF9772524 CF3442EBA65813C2140EBE6

This SmartStack solution was validated with Cisco UCS C-series using the Direct Attached design.

Fabric Extender Attached Design

For larger scale deployments, Cisco UCS C-series can be connected using standalone Fabric Extenders, namely the 1RU FEX 2232PP. Functionally, the standalone FEX is identical to the Cisco UCS 2204 and 2208 IOM modules that are deployed on the Cisco UCS 5108 blade server chassis for connecting to Cisco UCS Fabric Extenders. Similarly, the Cisco VIC on each Cisco UCS C-series server connect to both Fabric Interconnects using two Cisco FEX 2232PPs. The FEX and Fabric Interconnects form port channels automatically based on the chassis discovery policy providing a link resiliency to the Cisco UCS C-series server. This is identical to the behavior of the IOM to Fabric Interconnect connectivity. Logically, the virtual circuits formed within the Cisco UCS domain are consistent between B and C series deployment models and the virtual constructs formed at the vSphere are unaware of the platform in use.

Cisco UCS Server – High Availability and Virtual Networking Design

The SmartStack design Cisco UCS servers used in the SmartStack design are assigned to a VMware High Availability (HA) cluster in either management & services infrastructure or in the application infrastructure to mitigate against host failures. VMware HA enables automatic failover of active VMs to other hosts in the cluster in the event of a host failure. Admission control is used to ensure that application VMs has the resources it needs before a failover is allowed.

The Cisco UCS server models used in this design are Cisco UCS B200M4 blade servers with Cisco 1340 VIC and Cisco UCS C220M4 rack servers with Cisco 1227 VIC running VMware vSphere 6.5. Cisco VICs provide 256 virtual adapters that can be presented as storage (vHBAs) or networking (vNICs) adapters to operating system. The virtual adapters are presented as vPCIe devices to ESXi and are referred to as vmnics within VMware vSphere.

In this SmartStack design, the following virtual adapters are used with –A connected to unified fabric A and –B to unified fabric B resulting in each ESXi node being dual homed to the external network. The virtual adapters are assigned to different virtual switches depending on the type of traffic.

·         Two vNICs (vMGMT-A, vMGMT-B) for in-band management and vCenter HA traffic

·         Two vNICs (vMotion-A, vMotion-B) for vMotion traffic

·         Two vNICs (vNIC-A, vNIC-B) for application VM traffic

VMware vSwitch is used for management, vCenter HA and vMotion traffic. VMware vDS is used for application traffic with different application port groups to isolate the applications. The connectivity within each ESXi server is as shown below.

Figure 20     Cisco UCS Server Virtual Networking

 

The design also used two VMkernel NICs (vmk), each with its own port group for:

·         vmk0 - ESXi management

·         vmk1 - vMotion interface

The ESXi management interface is for host to vCenter connectivity, direct access to ESXi shell and VMware cluster communication. The vMotion interfaces are private subnets supporting data access and VM migration across the SmartStack infrastructure

Cisco Nexus 9000 Series – vPC Best Practices

SmartStack incorporates the following Cisco Nexus 9000 design best practices and recommendations.

vPC Peer Keepalive Link Considerations

It is recommended to have a dedicated 1Gbps layer 3 link for vPC peer keepalive, followed by out-of-band management interface (mgmt0) and lastly, routing the peer keepalive link over an existing Layer3 infrastructure between the existing vPC peers. vPC peer keepalive link should not be routed over a vPC peer-link. The out-of-band management network is used as the vPC peer keepalive link in this SmartStack design.

vPC Peer Link Considerations

·         Only vPC VLANs are allowed on the vPC peer-links. For deployments that require non-vPC VLAN traffic to be exchanged between vPC peer switches, deploy a separate Layer 2 link for this traffic.

·         Only required VLANs are allowed on the vPC peer links and member ports – prune all others to minimize internal resource consumption.

·         Ports from different line cards should be used to provide redundancy for vPC peer links. It was not possible to do this on the fixed module Cisco Nexus 9372PX switches used in this SmartStack design.

vPC General Considerations:

·         vPC peer switches deployed using same bridge-id and spanning tree VLAN priority by configuring the peer-switch command on both vPC peer switches. This feature improves convergence and allows peer switches to appear as a single spanning-tree root in the Layer 2 topology.

·         vPC role priority specified on both Cisco Nexus peer switches. vPC role priority determines which switch will be primary and which one will be secondary. The device with the lower value will become the primary. By default, this value is 32677. Cisco recommends that the default be changed on both switches. Primary vPC devices are responsible for BPDU and ARP processing. Secondary vPC devices are responsible for shutting down member ports and VLAN interfaces when peer-links fail.

·         vPC convergence time of 30s (default) was used to give routing protocol enough time to converge post-reboot. The default value can be changed using delay-restore <1-3600> and delay-restore interface-VLAN <1-3600> commands. If used, this value should be changed globally on both peer switches to meet the needs of your deployment.

·         vPC peer switches enabled as peer-gateways using peer-gateway command on both devices. Peer-gateway allows a vPC switch to act as the active gateway for packets that are addressed to the router MAC address of the vPC peer allowing vPC peers to forward traffic.

·         vPC auto-recovery enabled to provide a backup mechanism in the event of a vPC peer-link failure due to vPC primary peer device failure or if both switches reload but only one comes back up. This feature allows the one peer to assume the other is not functional and restore the vPC after a default delay of 240s. This needs to be enabled on both switches. The time to wait before a peer restores the vPC can be changed using the command: auto-recovery reload-delay <240-3600>.

·         Cisco NX-OS can synchronize ARP tables between vPC peers using the vPC peer links. This is done using a reliable transport mechanism that the Cisco Fabric Services over Ethernet (CFSoE) protocol provides. For faster convergence of address tables between vPC peers, IP ARP synchronize command was enabled on both peer devices in this SmartStack design.

vPC Member Link Considerations

·         LACP used for port channels in the vPC. LACP should be used when possible for graceful failover and protection from misconfigurations

·         LACP mode active-active used on both sides of the port channels in the vPC. LACP active-active is recommended, followed by active-passive mode and manual or static bundling if the access device does not support LACP.  Port-channel in mode active-active is preferred as it initiates more quickly than port-channel in mode active-passive.

·         LACP graceful-convergence disabled on port-channels going to Cisco UCS FI. LACP graceful-convergence is ON by default and should be enabled when the downstream access switch is a Cisco Nexus device and disabled if it is not.

·         Only required VLANs are allowed on the vPC peer links and member ports – prune all others to minimize internal resource consumption.

·         Source-destination IP, L4 port and VLAN are used load-balancing hashing algorithm for port-channels. This improves fair usage of all member ports forming the port-channel. The default hashing algorithm is source-destination IP and L4 port.

vPC Spanning Tree Considerations:

·         Bridge Assurance enabled on vPC peer links by specifying spanning-tree port type network. Bridge Assurance should be disabled on vPC member ports.

·         Spanning port type specified as edge or edge trunk on host facing interfaces connecting to Cisco UCS FI.

·         BPDU Guard feature enabled on host-facing interfaces connecting to Cisco UCS FI. This was done by globally enabling it on all edge ports using the spanning-tree port type edge bpduguard default command.

·         BPDU Filtering feature enabled on host-facing interfaces connecting to Cisco UCS FI. This was done by globally enabling it on all edge ports using the spanning-tree port type edge bpdufilter default command.

·         Loop Guard was disabled (default setting) in this design. If necessary, they can be enabled globally using spanning-tree loopguard default or at the interface level using spanning-tree guard loop.

·         Root Guard enabled on vPC member ports connected to access devices to ensure that vPC peer switches remain the spanning tree root – using interface level command spanning-tree guard root

Other Considerations

Unidirectional Link Detection (UDLD) was enabled globally using feature UDLD and on vPC peer links and member ports to Cisco UCS FI.

·         HSRP specific

—     Interface VLANs should be defined as passive interfaces to avoid routing peer information

—     Disable IP redirection on HSRP interface VLANs

—     Use default timer for HSRP/VRRP

If the SmartStack design outlined in this CVD is connected to additional aggregation/core layer Cisco Nexus switches in a two-tiered design for scalability or other expansion purposes, the following guidelines should be followed.

·         In a two-tiered data center design using Cisco Nexus switches, vPCs can also be used between the Cisco Nexus switches in each tier using a double-sided vPC topology. In such a design, the vPC domain identifiers must be different as this information is exchanged through LACP protocol and using the same vPC domain identifiers will generate continuous flaps on vPC between the different Cisco Nexus network layers.

·         If modular Cisco Nexus switches are used, redundancy should be provided by using ports from different line cards.

·         Deploy dedicated Layer 3 link(s) for exchanging routing information between peer switches in a two-tiered design or other topologies where Layer 3 is used between the tiers. The vPC peer-link should not be used.

Last but not least, review the criteria for vPC Type-1 and Type-2 consistency checks in the link provided below to avoid issues in your deployment.

·         vPC Design Guide: http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf

·         Cisco Nexus 9000 NX-OS Release 6.x Configuration Guide: http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6x/interfaces/configuration/guide/b_Cisco_Nexus_9000_Series_NXOS_Interfaces_Configuration_Guide.html

High Availability

SmartStack platform was designed for maximum availability of the complete infrastructure (compute, network, storage, and virtualization) with no single points of failure. 

Compute and Virtualization

·         Cisco UCS system provides redundancy at the component and link level and end-to-end path redundancy to storage array and LAN network.

·         Cisco UCS 5108 blade server platform is highly redundant with redundant power supplies, fans and fabric extender modules. C-series rack mount servers also have redundant power supplies and fans. The rack mount servers are directly connected to the upstream fabric interconnects in this SmartStack design.

·         Each fabric extender on the Cisco UCS 5108 blade server is deployed with 4x10GbE links to the unified fabric. The links are part of a port channel to ensure rerouting of flows to other links in the event of a link failure.

·         Each server is deployed using vNICs and vHBAs that provide redundant connectivity to the unified fabric. NIC failover is enabled between Cisco UCS Fabric Interconnects using Cisco UCS manager. This is done for all management and virtual machine vNICs.

·         VMware vCenter is used to deploy VMware HA clusters to allow VMs to failover in the event of a server failure. VMware vMotion and VMware HA are enabled to auto restart VMs after a failure. Host Monitoring is enabled to monitor heartbeats of all ESXi hosts in the cluster for faster detection. Admission Control is also enabled on the blade servers to ensure the cluster has enough resources to accommodate a single host failure.

·         VMware vCenter in the Management Cluster is deployed with the native high availability that is available in vCenter Server Appliance (VCSA) as of 6.5. vCenter HA requires the deployment of 3 nodes on separate hosts for maximum availability and requires a separate HA network for synchronization and replication traffic. vCenter HA fails over to a backup node either when a host fails or when certain key services fail.

·         VMware vSphere hosts use SAN multipathing to access LUNs on the Nimble Storage array. If any component (NIC, HBA, FEX, FI, Cisco MDS, Nimble controller, cables) along a path fails, all storage traffic will reroute to an alternate path. When both paths are active, traffic is load balanced.

Network

·         Link aggregation using port channels and virtual port channels are used throughout the SmartStack design for higher bandwidth and availability.

·         Port channels are used on unified fabric links between fabric extender and fabric interconnects. Virtual port channels are used between FIs and Cisco Nexus switches. vPCs provider higher availability than port channels as it can continue to forward traffic even if one of the Cisco Nexus switches fail because vPCs distribute member links of port-channel across different Cisco Nexus switches.

·         Port channels are also used on the FC links between FIs and Cisco MDS switches. Exchange based load balancing method is used.  4 x 8G FC links are used on the port channel between each FI and MDS pair.

·         Cisco Nexus 9000 series switches are used in the data center LAN fabric to provide redundancy in the event of a switch failure. Cisco MDS switches are used in the SAN fabric to provide redundancy in the event of a switch failure.

·         Cisco MDS and Cisco Nexus switches are highly redundant with redundant power supplies, fans and have out-of-band management access.

·         The two Cisco MDS switches form two separate fabrics and provide two distinct physical paths to storage for redundancy. FI-A to Cisco MDS-A to Nimble Storage array is SAN Fabric A and FI-B to Cisco MDS-B to Nimble Storage array is SAN Fabric B. Dual VSANs are used across these fabrics with vSAN-A on Fabric A and vSAN-B on Fabric B. The dual vSANs represent two redundant paths to the storage array with traffic load balanced across both vSANs when there is no failure. 

Storage                                    

·         Both the Nimble Storage AF7000 all flash array and CS5000 adaptive flash array have redundant storage controllers in an active-standby configuration.  Using a full standby controller means that performance will not be degraded in the event of a controller failover.

·         The AF7000 all flash array and CS5000 adaptive flash array have redundant power supplies with diverse cabling and multiple data paths to each controller.

·         Each Nimble storage controller is redundantly connected to the SAN fabric. Each controller is connected using 4x16Gb FC links to upstream Cisco MDS switches with 2x16Gb links going to Cisco MDS-A switch and 2x16Gb links going to Cisco MDS-B switch in the SAN fabric. This will allow 64GB network bandwidth for each controller. 

·         FC target connections are configured in a Dual fabric / dual vSAN switch fabric.  This configuration is used across the SAN fabric and unified fabric for redundant connectivity to storage.

·         Each Service Profile has a boot profile with redundant paths to primary and secondary FC targets on the Nimble Storage array. 

·         All VMware datastore volumes utilize Nimble PSP_Directed for proper path failover and load distribution.

Scalability

For higher performance and capacity, SmartStack solutions can scale up by adding compute, network, storage or management subsystems individually or scale out with consistent and predictable performance by deploying additional units of the complete SmartStack solution.

Management

·         Cisco UCS Manager running on a clustered pair of Cisco UCS Fabric Interconnects forms a single Cisco UCS domain and can manage up to 160 servers (8 servers per chassis x 20 chassis) in that one UCS domain.

·         Cisco UCS Central can be deployed to provide a single pane for managing multiple Cisco UCS domains – for up to 10,000 servers. Cisco UCS Central complements Cisco UCS Manager to provide centralized inventory, faults, and logs, global policies and pools, firmware management, global visibility and flexible administrative controls. Cisco UCS Central is a manager of managers that interfaces with Cisco UCS Manager in each domain to manage multiple, globally distributed Cisco UCS domains.

Compute

SmartStack can support multiple Cisco UCS domains with each domain supporting up to 20 blade server chassis (up to 8 blade server per chassis) per domain. Each Cisco UCS domain consists of a pair of Cisco 6200 or 6300 Series Fabric Interconnects that can provide a unified fabric and management with up to 2.56 Tbps of bandwidth. High-performance ports capable of line-rate, low-latency, lossless 1-, 10-, and 40-Gigabit Ethernet (6300 Series only) and Fibre Channel over Ethernet (FCoE), and 16-, 8-, and 4-Gbps (6300 Series) and 8-, 4-, and 2-Gbps (6200 Series) Fibre Channel.

Each Cisco UCS domain can support multiple compute form factors (Blade Server, Rack mount, Cisco UCS Mini) to meet the needs of the deployment and the applications deployed. 

Storage

Scale-to-Fit

The Nimble Storage Predictive Flash platform provides for application growth by scaling performance, capacity, or both – efficiently and non-disruptively.  With Nimble Storage scale-to-fit, organizations can:

·         Scale capacity by adding expansion shelves. The AF7000 array can support an effective capacity of up to 1.2PB (assumes 5:1 data reduction) of flash capacity through the addition of expansion shelves while the CS5000 array can support an effective capacity of up to 1.4PB (assumes 2:1 data reduction)

·         Scale up performance by upgrading compute for greater throughput and IOPS

·         Scale capacity and performance together by clustering any combination of Nimble Storage all flash or adaptive flash arrays – see next section for the Nimble Storage scale-out capabilities

Scale-Out

Nimble Storage arrays feature a scale-out architecture that makes it easy to scale capacity and performance beyond the physical limitations of a single array.  With Nimble Storage scale-out, organizations can:

·         Group up to 4 Nimble arrays (any model or series) for higher scalability and performance

·         One management console to administer all storage hardware resources in the group as a single entity

·         Dynamic load balancing across arrays in a group to eliminate performance hot spots

·         Multi-array data striping, enabling any application to fully leverage the collective hardware resources of the scale-out group

·         Flexible configuration of any combination of Nimble Storage arrays in the group, maximizing storage ROI

·         Seamless reconfiguration and hardware refreshes, without downtime

Storage Management Nimble vCenter Plugin

Nimble Storage provides a plugin that works specifically with vCenter to manage datastores residing on Nimble arrays. You can use either the desktop vCenter plugin, or the web-based vCenter plugin. Both provide the same functionality, with slight variances in the user views, and minor enhancements.

The Nimble vCenter plugin allows the following capability directly from vCenter:

·         Create, clone, grow, and edit datastores

·         Take, clone, and delete snapshots

·         Add Nimble-specific capabilities to vCenter server which can be used to create vCenter roles

·         Edit protection schedules

A high level summary of the validation done for the SmartStack design is provided in this section. The solution was validated for basic data forwarding by deploying virtual machine running IO Meter tool. The system was validated for resiliency by failing various aspects of the system under load. Examples of the types of tests executed are as follows:

·         Failure and recovery of ESXi hosts in a cluster including rebooting hosts, powering down hosts – all hosts will boot from SAN

·         Service Profile migration to a new server

·         Failure and recovery of redundant links between Cisco UCS FI and Cisco MDS 

·         Failure and recovery of redundant Cisco MDS switches

·         Failure and recovery of redundant links between Cisco MDS and Nimble controllers

·         Failure and recovery of SSDs on the Nimble Storage arrays

·         Managed failover and recovery of Nimble controllers (active & backup) from Nimble management interface

·         Failure and recovery of Nimble controllers (active & backup) – simulating a hard failure

·         Upgrading Nimble OS while system is actively processing I/O

·         Failure and recovery of Cisco Nexus switches

·         Failure and recovery of redundant links between Cisco UCS FI and Cisco Nexus switches

·         Failure and recovery of array nodes in the cluster.

·         Ensuring access to volumes with an array down in a 2-node cluster.

Load was generated using IOMeter tool and different IO profiles were used to reflect the different profiles that are seen in customer networks. See table below for the profiles used. 

Table 4     Traffic Profiles

Validated Hardware and Software

Table 5  lists all the components and software versions used in validating the SmartStack design.

Table 5     Validated Infrastructure Components and Software Revisions

 

Supported SmartStack Software Versions

SmartStack was validated using the software versions outlined in the table above but other software versions are also supported provided the components and releases are listed in the interoperability matrices below that Cisco and Nimble Storage provides. 

·         Cisco UCS Hardware and Software Interoperability Tool: http://www.cisco.com/web/techdoc/ucs/interoperability/matrix/matrix.html

·         Interoperability Matrix for Cisco Nexus and Cisco MDS 9000 Products: http://www.cisco.com/c/en/us/support/storage-networking/mds-9000-nx-os-san-os-software/products-device-support-tables-list.html

·         Nimble Support Matrix: https://infosight.nimblestorage.com/InfoSight/media/cms/active/pubs_support_matrix_3_rev_b.pdf  (requires login)

·         VMware Compatibility Guide: http://www.vmware.com/resources/compatibility/search.php

A detailed view of the Nimble Storage array configurations are shown in the figures below.

Figure 21     Nimble Storage AF7000 Array Configuration

        

Figure 22     Nimble Storage CS5000 Array Configuration

       

For ease of ordering SmartStack solutions, a Solution ID based process is available to configure and order a given solution as outlined in the following steps:

1.     Go to the Main CCW page : https://apps.cisco.com/Commerce/home

2.     Click “Create Estimate” on the Estimates tab and then click “Search For Solutions” below the Search field.

3.     Click “Find Products and Solutions."

4.     Click the “Solution” Radio button and then “View All Solutions” to the right of the Search field. Under the Main Category, select “Data Center Solutions.”

5.     From the “Sub Category” list for Data Center Solutions, select SmartStack. Select the specific SmartStack Solution ID and click “View components”.

6.     Click “Add Solution” to add all components from the selected SmartStack Solution ID.

7.     Click “Add” at the bottom of the Selected Items tab.

8.     Click “Edit Options” on the estimates page and select required items to complete configuration of each component

9.     When all components show “Valid”, the Estimate ID is now ready for import into CCW Quote/Order.

For this SmartStack solution, the BOM below lists the major components validated but it is not intended to be a comprehensive list.

Table 6     SmartStack Bill of Materials

SmartStack delivers an infrastructure platform for Enterprise and cloud data centers using Cisco UCS and Fibre Channel-attached Nimble Storage AF7000 all flash and Nimble CS5000 adaptive flash arrays. SmartStack is designed and validated using compute, network and storage best practices and high availability to reduce deployment time, project risk and IT costs while maintaining scalability and flexibility for addressing a multitude of IT initiatives.

Archana Sharma, Technical Leader, Cisco UCS Solutions Engineering, Cisco Systems Inc.

Archana Sharma has 20 years of experience at Cisco focused on Data Center, Desktop Virtualization, Collaboration and related technologies. Archana has been working on Enterprise and Service Provider systems and solutions and delivering Cisco Validated designs for over 10 years. Archana holds a CCIE (#3080) in Routing and Switching and a Bachelor’s degree in Electrical Engineering from North Carolina State University.

Jay White, Principal Technical Marketing Engineer, Nimble Storage Inc.

Jay has over 20 years of experience in both the network and storage industries, spanning roles in engineering, technical sales, data center consulting, and technical marketing. Jay leads the converged infrastructure (SmartStack) solutions technical activity at Nimble Storage and covers cloud solutions. In the past, he has provided subject matter expertise on nearly all aspects of network storage systems, including performance, file systems, protocols, storage efficiency, disaster recovery, and more.

Acknowledgements

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, the authors would like to thank:

·         Chris O' Brien, Director, Cisco UCS Solutions Technical Marketing Team, Cisco Systems Inc.

·         Nivas Iyer, Product Manager, Cisco UCS Product Management and Solutions, Cisco Systems Inc.

·         Ashish Prakash, VP Solutions Engineering, Nimble Storage Inc.

·         Arun Garg, Director, Solutions Product Management, Nimble Storage Inc.

·         Matt Miller, Senior Product Marketing Manager, Nimble Storage Inc.

NOTE: Available paragraph styles are listed in the Quick Styles Gallery in the Styles group on the Home tab. Alternatively, they can be accessed via the Styles window (press Alt + Ctrl + Shift + S).