Cisco Virtualization Solution for EMC VSPEX with VMware vSphere 5.0 for 250 Virtual Machines
Available Languages
Table Of Contents
About Cisco Validated Design (CVD) Program
Cisco Solution for EMC VSPEX VMware Architectures
Cisco Unified Computing System
EMC Storage Technologies and Benefits
Memory Configuration Guidelines
ESX/ESXi Memory Management Concepts
Virtual Machine Memory Concepts
Allocating Memory to Virtual Machines
VSPEX VMware Memory Virtualization
Virtual Networking Best Practices
Non-Uniform Memory Access (NUMA)
VSPEX VMware Storage Virtualization
Architecture for 250 VMware Virtual Machines
Defining the Reference Workload
Applying the Reference Workload
VSPEX Configuration Guidelines
Preparing Cisco UCS Components
Preparing Cisco Nexus Switches
Topology Diagram for 250 Virtual Machines
Configuring Cisco Nexus Switches
Create Service Profile Template
Create service profile instances from the template
Preparing and Configuring Storage Array
Configure SAN Boot Infrastructure
Installing ESXi Servers and vCenter Infrastructure
Installing and Configuring Microsoft SQL Server Database
VMware vCenter Server Deployment
Configuring Cluster, HA and DRS on the vCenter
Jumbo MTU Validation And Diagnostics
Template-Based Deployments for Rapid Provisioning
Validating Cisco Solution for EMC VSPEX VMware Architectures
Verify the Redundancy Of The Solution Components
Customer Configuration Data Sheet
Cisco Solution for EMC VSPEX VMware ArchitecturesDesign for 250 Virtual MachinesLast Updated: October 24, 2013
Building Architectures to Solve Business Problems
About the Authors
Mehul Bhatt, Virtualization Architect, Server Access Virtualization Business Unit, Cisco SystemsMehul Bhatt has over 12 years of Experience in virtually all layers of computer networking. His focus area includes Unified Compute Systems, network and server virtualization design. Prior to joining Cisco Technical Marketing team, Mehul was Technical Lead at Cisco, Nuova systems and Bluecoat systems. Mehul holds a Masters degree in computer systems engineering and holds various Cisco career certifications.
Vijay Kumar D, Technical Marketing Engineer, Server Access Virtualization Business Unit, Cisco Systems
Vijay Kumar has over 10 years of experience in UCS, network, storage and server virtualization design. Vijay has worked on performance and benchmarking on Cisco UCS and has delivered benchmark results on SPEC CPU2006 and SPECj ENT 2010. Vijay holds certification in VMware Certified Professional and Cisco Unified Computing systems Design specialist.
Vadiraja Bhatt, Performance Architect, Server Access Virtualization Business Unit, Cisco SystemsVadiraja Bhatt is a Performance Architect at Cisco, managing the solutions and benchmarking effort on Cisco Unified Computing System Platform. Vadi has over 17 years of experience in performance and benchmarking the large enterprise systems deploying mission critical applications. Vadi specializes in optimizing and fine tuning complex hardware and software systems and has delivered many benchmark results on TPC and other industry standard benchmarks. Vadi has 6 patents to his credits in the Database (OLTP and DSS) optimization area.
Acknowledgements
For their support and contribution to the design, validation, and creation of the Cisco Validated Design, we would like to thank:
•
Bathu Krishnan-Cisco
•
•
•
•
About Cisco Validated Design (CVD) Program
The CVD program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information visit www.cisco.com/go/designzone.
ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.
CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.
All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)
© 2012 Cisco Systems, Inc. All rights reserved
Cisco Solution for EMC VSPEX VMware Architectures
Executive Summary
Cisco solution for the EMC VSPEX is a pre-validated and modular architecture built with proven best-of-breed technologies to create and complete an end-to-end virtualization solution. The end-to-end solutions enable you to make an informed decision while choosing the hypervisor, compute, storage and networking layers. VSPEX eliminates the server virtualization planning and configuration burdens. The VSPEX infrastructures accelerate your IT Transformation by enabling faster deployments, greater flexibility of choice, efficiency, and lower risk. This Cisco Validated Design document focuses on the VMware architecture for 250 virtual machines with Cisco solution for the EMC VSPEX.
Introduction
Virtualization is a key and critical strategic deployment model for reducing the Total Cost of Ownership (TCO) and achieving better utilization of the platform components like hardware, software, network and storage. However, choosing an appropriate platform for virtualization can be challenging. Virtualization platforms should be flexible, reliable, and cost effective to facilitate the deployment of various enterprise applications. In a virtualization platform to utilize compute, network, and storage resources effectively, the ability to slice and dice the underlying platform is essential to size to the application requirements. The Cisco solution for the EMC VSPEX provides a very simplistic yet fully integrated and validated infrastructure to deploy VMs in various sizes to suit various application needs.
Target Audience
The reader of this document is expected to have the necessary training and background to install and configure VMware vSphere 5.0, EMC VNX5500, Cisco Nexus 5548UP switch, and Cisco Unified Computing (UCS) B200 M3 Blade Servers. External references are provided wherever applicable and it is recommended that the reader be familiar with these documents.
Readers are also expected to be familiar with the infrastructure and database security policies of the customer installation.
Purpose of this Guide
This document describes the steps required to deploy and configure the Cisco solution for the EMC VSPEX for VMware architecture. The document provides the end-to-end solution for the VMware vSphere 5.0 for 250 Virtual Machine Architecture.
The readers of this document are expected to have sufficient knowledge to install and configure the products used, configuration details that are important to the deployment models mentioned above.
Business Needs
The VSPEX solutions are built with proven best-of-breed technologies to create complete virtualization solutions that enable you to make an informed decision in the hypervisor, server, and networking layers. The VSPEX infrastructures accelerate your IT transformation by enabling faster deployments, greater flexibility of choice, efficiency, and lower risk.
Business applications are moving into the consolidated compute, network, and storage environment. The Cisco solution for the EMC VSPEX using VMware reduces the complexity of configuring every component of a traditional deployment model. The complexity of integration management is reduced while maintaining the application design and implementation options. Administration is unified, while process separation can be adequately controlled and monitored. The following are the business needs for the Cisco solution for EMC VSPEX VMware architectures:
•
•
•
•
Solution Overview
The Cisco solution for EMC VSPEX using VMware vSphere 5.0 provides an end-to-end architecture with Cisco, EMC, VMware, and Microsoft technologies that demonstrate support for up to 250 generic virtual machines and provide high availability and server redundancy.
The following are the components used for the design and deployment:
•
•
•
•
•
•
•
•
•
•
•
•
The solution is designed to host scalable, and mixed application workloads. The scope of this CVD is limited to the Cisco solution for EMC VSPEX VMware solutions for 250 virtual machines only.
Technology Overview
Cisco Unified Computing System
The Cisco Unified Computing System is a next-generation data center platform that unites compute, network, and storage access. The platform, optimized for virtual environments, is designed using open industry-standard technologies and aims to reduce total cost of ownership (TCO) and increase business agility. The system integrates a low-latency; lossless 10 Gigabit Ethernet unified network fabric with enterprise-class, x86-architecture servers. It is an integrated, scalable, multi chassis platform in which all resources participate in a unified management domain.
The main components of Cisco Unified Computing System are:
•
•
•
•
•
The Cisco Unified Computing System is designed to deliver:
•
•
•
•
•
Cisco UCS Fabric Interconnect
The Cisco® UCS 6200 Series Fabric Interconnect is a core part of the Cisco Unified Computing System, providing both network connectivity and management capabilities for the system. The Cisco UCS 6200 Series offers line-rate, low-latency, lossless 10 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE) and Fibre Channel functions.
The Cisco UCS 6200 Series provides the management and communication backbone for the Cisco UCS B-Series Blade Servers and Cisco UCS 5100 Series Blade Server Chassis. All chassis, and therefore all blades, attached to the Cisco UCS 6200 Series Fabric Interconnects become part of a single, highly available management domain. In addition, by supporting unified fabric, the Cisco UCS 6200 Series provides both the LAN and SAN connectivity for all blades within its domain.
From a networking perspective, the Cisco UCS 6200 Series uses a cut-through architecture, supporting deterministic, low-latency, line-rate 10 Gigabit Ethernet on all ports, 1Tb switching capacity, 160 Gbps bandwidth per chassis, independent of packet size and enabled services. The product family supports Cisco low-latency, lossless 10 Gigabit Ethernet unified network fabric capabilities, which increase the reliability, efficiency, and scalability of Ethernet networks. The Fabric Interconnect supports multiple traffic classes over a lossless Ethernet fabric from a blade server through an interconnect. Significant TCO savings come from an FCoE-optimized server design in which network interface cards (NICs), host bus adapters (HBAs), cables, and switches can be consolidated.
Cisco UCS 6248UP Fabric Interconnect
The Cisco UCS 6248UP 48-Port Fabric Interconnect is a one-rack-unit (1RU) 10 Gigabit Ethernet, FCoE and Fiber Channel switch offering up to 960-Gbps throughput and up to 48 ports. The switch has 32 1/10-Gbps fixed Ethernet, FCoE and FC ports and one expansion slot.
Figure 1 Cisco UCS 6248UP Fabric Interconnect
Cisco UCS Fabric Extenders
The Cisco UCS 2100 Series Fabric Extenders multiplex and forward all traffic from blade servers in a chassis to a parent Cisco UCS fabric interconnect over from 10-Gbps unified fabric links. All traffic, even traffic between blades on the same chassis or virtual machines on the same blade, is forwarded to the parent interconnect, where network profiles are managed efficiently and effectively by the fabric interconnect. At the core of the Cisco UCS fabric extender are application-specific integrated circuit (ASIC) processors developed by Cisco that multiplex all traffic.
Cisco UCS 2104XP Fabric Extender
The Cisco UCS 2104XP Fabric Extender has eight 10GBASE-KR connections to the blade chassis midplane, with one connection per fabric extender for each of the chassis' eight half slots. This configuration gives each half-slot blade server access to each of two 10-Gbps unified fabric-based networks through SFP+ sockets for both throughput and redundancy. It has four ports connecting the fabric interconnect.
Figure 2 Cisco UCS 2104XP Fabric Extender
Cisco UCS Blade Chassis
The Cisco UCS 5100 Series Blade Server Chassis is a crucial building block of the Cisco Unified Computing System, delivering a scalable and flexible blade server chassis.
The Cisco UCS 5108 Blade Server Chassis, is six rack units (6RU) high and can mount in an industry-standard 19-inch rack. A single chassis can house up to eight half-width Cisco UCS B-Series Blade Servers and can accommodate both half-width and full-width blade form factors.
Four single-phase, hot-swappable power supplies are accessible from the front of the chassis. These power supplies are 92 percent efficient and can be configured to support non-redundant, N+ 1 redundant and grid-redundant configurations. The rear of the chassis contains eight hot-swappable fans, four power connectors (one per power supply), and two I/O bays for Cisco UCS 2104XP Fabric Extenders.
A passive mid-plane provides up to 20 Gbps of I/O bandwidth per server slot and up to 40 Gbps of I/O bandwidth for two slots. The chassis is capable of supporting future 40 Gigabit Ethernet standards. The Cisco UCS Blade Server Chassis is shown in Figure 3.
Figure 3 Cisco Blade Server Chassis (front and back view)
Cisco UCS Blade Servers
Delivering performance, versatility and density without compromise, the Cisco UCS B200 M3 Blade Server addresses the broadest set of workloads, from IT and Web Infrastructure through distributed database.
Building on the success of the Cisco UCS B200 M2 blade servers, the enterprise-class Cisco UCS B200 M3 server, further extends the capabilities of Cisco's Unified Computing System portfolio in a half blade form factor. The Cisco UCS B200 M3 server harnesses the power and efficiency of the Intel Xeon E5-2600 processor product family, up to 768 GB of RAM, 2 drives or SSDs and up to 2 x 20 GbE to deliver exceptional levels of performance, memory expandability and I/O throughput for nearly all applications. In addition, the Cisco UCS B200 M3 blade server offers a modern design that removes the need for redundant switching components in every chassis in favor of a simplified top of rack design, allowing more space for server resources, providing a density, power and performance advantage over previous generation servers. The Cisco UCS B200M3 Server is shown in Figure 4.
Figure 4 Cisco UCS B200 M3 Blade Server
Cisco UCS Manager
Cisco UCS Manager provides unified, embedded management of all software and hardware components of the Cisco Unified Computing System through an intuitive GUI, a command line interface (CLI), or an XML API. The Cisco UCS Manager provides unified management domain with centralized management capabilities and controls multiple chassis and thousands of virtual machines.
Cisco UCS Service Profiles
Programmatically Deploying Server Resources
Cisco UCS Manager provides centralized management capabilities, creates a unified management domain, and serves as the central nervous system of the Cisco UCS. Cisco UCS Manager is embedded device management software that manages the system from end-to-end as a single logical entity through an intuitive GUI, CLI, or XML API. Cisco UCS Manager implements role- and policy-based management using service profiles and templates. This construct improves IT productivity and business agility. Now infrastructure can be provisioned in minutes instead of days, shifting IT's focus from maintenance to strategic initiatives.
Dynamic Provisioning with Service Profiles
Cisco UCS resources are abstract in the sense that their identity, I/O configuration, MAC addresses and WWNs, firmware versions, BIOS boot order, and network attributes (including QoS settings, pin groups, and threshold policies) all are programmable using a just-in-time deployment model. The manager stores this identity, connectivity, and configuration information in service profiles that reside on the Cisco UCS 6100 Series Fabric Interconnect. A service profile can be applied to any blade server to provision it with the characteristics required to support a specific software stack. A service profile allows server and network definitions to move within the management domain, enabling flexibility in the use of system resources. Service profile templates allow different classes of resources to be defined and applied to a number of resources, each with its own unique identities assigned from predetermined pools.
Service Profiles and Templates
A service profile contains configuration information about the server hardware, interfaces, fabric connectivity, and server and network identity. The Cisco UCS Manager provisions servers utilizing service profiles. The Cisco UCS Manager implements a role-based and policy-based management focused on service profiles and templates. A service profile can be applied to any blade server to provision it with the characteristics required to support a specific software stack. A service profile allows server and network definitions to move within the management domain, enabling flexibility in the use of system resources.
Service profile templates are stored in the Cisco UCS 6100 Series Fabric Interconnects for reuse by server, network, and storage administrators. Service profile templates consist of server requirements and the associated LAN and SAN connectivity. Service profile templates allow different classes of resources to be defined and applied to a number of resources, each with its own unique identities assigned from predetermined pools.
The Cisco UCS Manager can deploy the service profile on any physical server at any time. When a service profile is deployed to a server, the Cisco UCS Manager automatically configures the server, adapters, Fabric Extenders, and Fabric Interconnects to match the configuration specified in the service profile. A service profile template parameterizes the UIDs that differentiate between server instances.
This automation of device configuration reduces the number of manual steps required to configure servers, Network Interface Cards (NICs), Host Bus Adapters (HBAs), and LAN and SAN switches. Figure 5 shows the Service profile which contains abstracted server state information, creating an environment to store unique information about a server.
Figure 5 Service Profile
Cisco Nexus 5548UP Switch
The Cisco Nexus 5548UP is a 1RU 1 Gigabit and 10 Gigabit Ethernet switch offering up to 960 gigabits per second throughput and scaling up to 48 ports. It offers 32 1/10 Gigabit Ethernet fixed enhanced Small Form-Factor Pluggable (SFP+) Ethernet/FCoE or 1/2/4/8-Gbps native FC unified ports and three expansion slots. These slots have a combination of Ethernet/FCoE and native FC ports. The Cisco Nexus 5548UP switch is shown in Figure 6.
Figure 6 Cisco Nexus 5548UP switch
Cisco I/O Adapters
Cisco UCS Blade Servers support various Converged Network Adapter (CNA) options. Cisco UCS Virtual Interface Card (VIC) 1240 is used in this EMC VSPEX solution.
The Cisco UCS Virtual Interface Card 1240 is a 4-port 10 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE)-capable modular LAN on motherboard (mLOM) designed exclusively for the M3 generation of Cisco UCS B-Series Blade Servers. When used in combination with an optional Port Expander, the Cisco UCS VIC 1240 capabilities can be expanded to eight ports of 10 Gigabit Ethernet.
The Cisco UCS VIC 1240 enables a policy-based, stateless, agile server infrastructure that can present up to 256 PCIe standards-compliant interfaces to the host that can be dynamically configured as either network interface cards (NICs) or host bus adapters (HBAs). In addition, the Cisco UCS VIC 1240 supports Cisco Data Center Virtual Machine Fabric Extender (VM-FEX) technology, which extends the Cisco UCS fabric interconnect ports to virtual machines, simplifying server virtualization deployment.
Figure 7 Cisco UCS VIC 1240
Cisco VM-FEX Technology
The Virtual Interface Card provides hardware based implementation of the Cisco VM-FEX technology. The Cisco VM-FEX technology eliminates the standard virtual switch within the hypervisor by providing individual virtual machine virtual ports on the physical network switch. Virtual machine I/O is sent directly to the upstream physical network switch, in this case, the Cisco UCS 6200 Series Fabric Interconnect, which takes full responsibility for virtual machine switching and policy enforcement.
In a VMware environment, the VIC presents itself as three distinct device types to the hypervisor OS as:
•
•
•
The Fibre Channel and Ethernet interfaces are consumed by the standard VMware vmkernel components and provide standard capabilities. The dynamic Ethernet interfaces are not visible to the vmkernel layers and are preserved as raw PCIe devices.
Using the Cisco vDS VMware plug-in and Cisco VM-FEX technology, the VIC provides a solution that is capable of discovering the dynamic Ethernet interfaces and registering all of them as uplink interfaces for internal consumption of the vDS. The vDS component on each host discovers the number of uplink interfaces that it has and presents a switch to the virtual machines running on a host as shown in the Figure 8. All traffic from an interface on a virtual machine is sent to the corresponding port of the vDS switch. The traffic is mapped immediately to a unique dynamic Ethernet interface presented by the VIC. This vDS implementation guarantees the 1:1 relationship with a virtual machine interface and an uplink port. The dynamic Ethernet interface selected, is a precise proxy for the virtual machine's interface.
The dynamic Ethernet interface presented by the VIC has a corresponding virtual port on the upstream fabric interconnect.
Figure 8 VM Interfaces Showing their Virtual Ports on the Physical Switch
Cisco UCS Manager running on the Cisco UCS Fabric Interconnect works in conjunction with the VMware vCenter software to coordinate the creation and movement of virtual machines. Port profiles are used to describe the virtual machine interface attributes such as VLAN, port security, rate limiting, and QoS marking. Port profiles are managed and configured by network administrators using Cisco UCS Manager. To facilitate integration with the VMware vCenter, Cisco UCS Manager pushes the catalog of port profiles into VMware vCenter, where they are represented as distinct port groups. This integration allows the virtual machine administrators to simply select from a menu of port profiles as they create virtual machines. When a virtual machine is created or moved to a different host, it communicates its port group to the Virtual Interface Card. The VIC gets the port profile corresponding to the requested profile from the Cisco UCS Manager and the virtual port on the fabric interconnect switch is configured according to the attributes defined in the port profile.
The Cisco VM-FEX technology addresses the common concerns of server virtualization and virtual networking by providing the following benefits:
•
•
•
Modes of Operations for VM-FEX technology
Cisco VM-FEX technology supports virtual machine interfaces that run in the following modes:
•
The hypervisor emulates a NIC (also referred to as a back-end emulated device) to replicate the hardware it virtualizes for the guest virtual machine. The emulated device presents descriptors, for read and write, and interrupts to the guest virtual machine just as a real hardware NIC device would. One such NIC device that VMware ESXi emulates is the vmxnet3 device. The guest OS in turn instantiates a device driver for the emulated NIC. All the resources of the emulated devices' host interface are mapped to the address space of the guest OS.
•
Virtual Interface Card uses PCIe standards-compliant IOMMU technology from Intel and VMware's VMDirectPath technology to implement PCIe Pass-Through across the hypervisor layer and eliminate the associated I/O overhead. The Pass-Through mode can be requested in the port profile associated with the interface using the "high-performance" attribute.
VMware vSphere 5.0
VMware vSphere 5.0 is a next-generation virtualization solution from VMware which builds upon ESXi 4 and provides greater levels of scalability, security, and availability to virtualized environments. vSphere 5.0 offers improvements in performance and utilization of CPU, memory, and I/O. It also offers users the option to assign up to thirty two virtual CPU to a virtual machine—giving system administrators more flexibility in their virtual server farms as processor-intensive workloads continue to increase.
The vSphere 5.0 provides the VMware vCenter Server that allows system administrators to manage their ESXi hosts and virtual machines on a centralized management platform. With the Cisco Fabric Interconnects Switch integrated into the vCenter Server, deploying and administering virtual machines is similar to deploying and administering physical servers. Network administrators can continue to own the responsibility for configuring and monitoring network resources for virtualized servers as they did with physical servers. System administrators can continue to "plug-in" their virtual machines into the network ports that have Layer 2 configurations, port access and security policies, monitoring features, and so on, that have been pre-defined by the network administrators; in the same way they need to plug in their physical servers to a previously-configured access switch. In this virtualized environment, the network port configuration/policies move with the virtual machines when the virtual machines are migrated to different server hardware.
EMC Storage Technologies and Benefits
The EMC VNX™ family is optimized for virtual applications delivering industry-leading innovation and enterprise capabilities for file, block, and object storage in a scalable, easy-to-use solution. This next-generation storage platform combines powerful and flexible hardware with advanced efficiency, management, and protection software to meet the demanding needs of today's enterprises.
VNX series is designed to meet the high-performance, high-scalability requirements of midsize and large enterprises. The EMC VNX storage arrays are multi-protocol platform that can support the iSCSI, NFS, Fibre Channel, and CIFS protocols depending on the customer's specific needs. This solution was validated using NFS for data storage of Virtual Machines and Fibre Channel for hypervisor SAN boot.
VNX series storage arrays have the following customer benefits:
•
•
•
•
•
Software Suites
The following are the available EMC software suites:
•
•
•
Software Packs
Total Value Pack—Includes all protection software suites, and the Security and Compliance Suite.
This is the available EMC protection software pack.
EMC Avamar
EMC's Avamar® data deduplication technology seamlessly integrates into virtual environments, providing rapid backup and restoration capabilities. Avamar's deduplication results in vastly less data traversing the network, and greatly reduces the amount of data being backed up and stored; resulting in storage, bandwidth and operational savings.
The following are the two most common recovery requests used in backup and recovery:
•
•
The Avamar System State protection functionality adds backup and recovery capabilities in both of these scenarios.
Architectural Overview
This Cisco Validated Design discusses the deployment model of the VMware solution for 250 virtual machines.
Table 1 lists the hardware components required in the solution:
Table 2 lists the various firmware and software components which occupies different tiers of the Cisco solution for EMC VSPEX VMware architectures under test.
Table 3 outlines the B200 M3 server configuration details (per server basis) across all the VMware architectures.
Table 3 Server configuration details
Memory (RAM)
64 GB (8X8 MB DIMM)
Processor
2 x Intel® Xenon ® E5-2650 CPUs, 2 GHz, 8 cores, 16 threads
This architecture assumes that there is an existing infrastructure/ management network available in which a virtual machine hosting vCenter server and Windows Active Directory/ DNS server are present. Figure 9 shows a high level Cisco solution for EMC VSPEX VMware architecture for 250 virtual machines.
Figure 9 Reference Architecture for 250 Virtual Machines
The following are the high level design points of the architecture:
•
•
•
This design does not recommend or require any specific layout of infrastructure network. The vCenter server, SQL server, and AD/ DNS virtual machines are hosted on the infrastructure network. However, design does require accessibility of certain VLANs from the infrastructure network to reach the servers.
ESXi 5.0 is used as hypervisor operating system on each server and is installed on fibre channel SAN. The defined load is 25 virtual machines per server.
Memory Configuration Guidelines
This section provides guidelines for allocating memory to the virtual machines. The guidelines outlined here take into account vSphere memory overhead and the virtual machine memory settings.
ESX/ESXi Memory Management Concepts
vSphere virtualizes guest physical memory by adding an extra level of address translation. Shadow page tables make it possible to provide this additional translation with little or no overhead. Managing memory in the hypervisor enables the following:
•
•
•
For more information about vSphere memory management concepts, see the VMware vSphere Resource Management Guide at: http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf
Virtual Machine Memory Concepts
The Figure 10 shows the use of memory settings parameters in the virtual machine.
Figure 10 Virtual Machine Memory Settings
The vSphere memory settings for a virtual machine include the following parameters:
•
•
•
Allocating Memory to Virtual Machines
Memory sizing for a virtual machine in VSPEX architectures is based on many factors. With the number of application services and use cases available determining a suitable configuration for an environment requires creating a baseline configuration, testing, and making adjustments, as discussed later in this paper. Table 4 outlines the resources used by a single virtual machine:
Following are the recommended best practices:
•
•
•
–
–
–
As with over committing CPU resources, proactive monitoring is a requirement. Table 5 lists counters that can be monitored to avoid performance issues resulting from overcommitted memory.
Storage Guidelines
VSPEX architecture for VMware 250 VMs scale, uses NFS to access storage arrays. This simplifies the design and implementation for the small to medium level businesses. vSphere provides many features that take advantage of EMC storage technologies such as VNX VAAI plug-in for NFS storage and storage replication. Features such as VMware vMotion, VMware HA, and VMware Distributed Resource Scheduler (DRS) use these storage technologies to provide high availability, resource balancing, and uninterrupted workload migration.
Virtual Server Configuration
Figure 11 shows that the VMware storage virtualization can be categorized into three layers of storage technology:
•
•
•
Figure 11 VMware Storage Virtualization Stack
Storage Protocol Capabilities
VMware vSphere provides vSphere and storage administrators with the flexibility to use the storage protocol that meets the requirements of the business. This can be a single protocol datacenter wide, such as iSCSI, or multiple protocols for tiered scenarios such as using Fibre Channel for high-throughput storage pools and NFS for high-capacity storage pools.
For VSPEX solution on vSphere NFS is a recommended option because of its simplicity in deployment.
For more information, see the VMware white paper Comparison of Storage Protocol Performance in VMware vSphere 5: http://www.vmware.com/files/pdf/perf_vsphere_storage_protocols.pdf
Storage Best Practices
Following are the vSphere storage best practices:
•
•
•
•
•
VSPEX VMware Memory Virtualization
VMware vSphere 5.0 has a number of advanced features that help to maximize performance and overall resources utilization. This section describes the performance benefits of some of these features for the VSPEX deployment.
Memory Compression
Memory over-commitment occurs when more memory is allocated to virtual machines than is physically present in a VMware ESXi host. Using sophisticated techniques, such as ballooning and transparent page sharing, ESXi is able to handle memory over-commitment without any performance degradation. However, if more memory than that is present on the server is being actively used, ESXi might resort to swapping out portions of a VM's memory.
For more details about Vsphere memory management concepts, see the VMware Vsphere Resource Management Guide at: http://www.VMware.com/files/pdf/mem_mgmt_perf_Vsphere5.pdf
Virtual Networking
The Cisco VMFEX collapses virtual and physical networking into a single infrastructure. The Cisco VM-FEX technology allows data center administrators to provision, configure, manage, monitor, and diagnose virtual machine network traffic and bare metal network traffic within a unified UCS infrastructure.
The VM-FEX technology extends Cisco data-center networking technology to the virtual machine with the following capabilities:
•
•
•
•
Benefits
•
•
•
•
Virtual Networking Best Practices
Following are the vSphere networking best practices:
•
•
•
•
This solution suggests 32 dynamic VNICs per ESXi host based on following assumptions and calculation:
•
•
•
–
–
–
•
–
–
Three dynamic vNICs are required per hypervisor, but we have provisioned one extra for head room.
vSphere VMware Performance
With every release of vSphere the overhead of running an application on the vSphere virtualized platform is reduced by the new performance improving features. Typical virtualization overhead for applications is less than 10%. Many of these features not only improve performance of the virtualized application itself, but also allow for higher consolidation ratios. Understanding these features and taking advantage of them in your environment helps guarantee the highest level of success in your virtualized deployment. Table 6 provides details on vSphere VMware performance.
Table 6 vSPhere VMware performance
NUMA Support
ESX/ESXi uses a NUMA load-balancer to assign a home node to a virtual machine. Because memory for the virtual machine is allocated from the home node, memory access is local and provides the best performance possible. Even applications that do not directly support NUMA benefit from this feature.
See The CPU Scheduler in VMware ESXi 5: http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf
Transparent page sharing
Virtual machines running similar operating systems and applications typically have identical sets of memory content. Page sharing allows the hypervisor to reclaim the redundant copies and keep only one copy, which frees up the total host memory consumption. If most of your application virtual machines run the same operating system and application binaries then total memory usage can be reduced to increase consolidation ratios.
See Understanding Memory Resource Management in VMware ESXi 5.0: http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf
Memory ballooning
By using a balloon driver loaded in the guest operating system, the hypervisor can reclaim host physical memory if memory resources are under contention. This is done with little to no impact to the performance of the application.
See Understanding Memory Resource Management in VMware ESXi 5.0: http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf
Memory compression
Before a virtual machine resorts to host swapping, due to memory over commitment the pages elected to be swapped attempt to be compressed. If the pages can be compressed and stored in a compression cache, located in main memory, the next access to the page causes a page decompression as opposed to a disk swap out operation, which can be an order of magnitude faster.
See Understanding Memory Resource Management in VMware ESXi 5.0: http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf
Large memory page support
An application that can benefit from large pages on native systems, such as MS SQL, can potentially achieve a similar performance improvement on a virtual machine backed with large memory pages. Enabling large pages increases the memory page size from 4KB to 2MB.
See Performance Best Practices for VMware vSphere 5.0: http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf
and see Performance and Scalability of Microsoft SQL Server on VMware vSphere 4:
http://www.vmware.com/files/pdf/perf_vsphere_sql_scalability.pdf
Physical and Virtual CPUs
VMware uses the terms virtual CPU (vCPU) and physical CPU to distinguish between the processors within the virtual machine and the underlying physical x86/x64-based processor cores. Virtual machines with more than one virtual CPU are also called SMP (symmetric multiprocessing) virtual machines. The virtual machine monitor (VMM), or hypervisor, is responsible for CPU virtualization. When a virtual machine starts running, control transfers to the VMM, which virtualizes the guest OS instructions.
Virtual SMP
VMware Virtual Symmetric Multiprocessing (Virtual SMP) enhances virtual machine performance by enabling a single virtual machine to use multiple physical processor cores simultaneously. vSphere supports the use of up to thirty two virtual CPUs per virtual machine. The biggest advantage of an SMP system is the ability to use multiple processors to execute multiple tasks concurrently, thereby increasing throughput (for example, the number of transactions per second). Only workloads that support parallelization (including multiple processes or multiple threads that can run in parallel) can really benefit from SMP.
The virtual processors from SMP-enabled virtual machines are co-scheduled. That is, if physical processor cores are available, the virtual processors are mapped one-to-one onto physical processors and are then run simultaneously. In other words, if one vCPU in the virtual machine is running, a second vCPU is co-scheduled so that they execute nearly synchronously. Consider the following points when using multiple vCPUs:
•
•
In VMware ESXi 5 and ESXi, the CPU scheduler underwent several improvements to provide better performance and scalability; for more information, see the CPU Scheduler in VMware ESXi 5:
http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf. For example, in VMware ESXi 5, the relaxed co-scheduling algorithm was refined so that scheduling constraints due to co-scheduling requirements are further reduced. These improvements resulted in better linear scalability and performance of the SMP virtual machines.
Overcommitment
VMware conducted tests on virtual CPU overcommitment with SAP and SQL, showing that the performance degradation inside the virtual machines is linearly reciprocal to the overcommitment. Because the performance degradation is "graceful," any virtual CPU overcommitment can be effectively managed by using VMware DRS and VMware vSphere® vMotion® to move virtual machines to other ESX/ESXi hosts to obtain more processing power. By intelligently implementing CPU overcommitment, consolidation ratios of applications Web front-end and application servers can be driven higher while maintaining acceptable performance. If it is chosen that a virtual machine not participate in overcommitment, setting a CPU reservation provides a guaranteed CPU allocation for the virtual machine. This practice is generally not recommended because the reserved resources are not available to other virtual machines and flexibility is often required to manage changing workloads. However, SLAs and multi-tenancy may require a guaranteed amount of compute resources to be available. In these cases, reservations make sure that these requirements are met.
When choosing to overcommit CPU resources, monitor vSphere and applications to be sure responsiveness is maintained at an acceptable level. Table 7 lists counters that can be monitored to help achieve higher drive consolidation while maintaining the system performance.
Hyper-threading
Hyper-threading technology (recent versions of which are called symmetric multithreading, or SMT) enables a single physical processor core to behave like two logical processors, essentially allowing two independent threads to run simultaneously. Unlike having twice as many processor cores which can roughly double performance, hyper-threading can provide anywhere from a slight to a significant increase in system performance by keeping the processor pipeline busier.
Non-Uniform Memory Access (NUMA)
Non-Uniform Memory Access (NUMA) compatible systems contain multiple nodes that consist of a set of processors and memory. The access to memory in the same node is local, while access to the other node is remote. Remote access can take longer because it involves a multihop operation. In NUMA-aware applications, there is an attempt to keep threads local to improve performance.
The VMware ESX/ESXi provides load-balancing on NUMA systems. To achieve the best performance, it is recommended that the NUMA be enabled on compatible systems. On a NUMA-enabled ESX/ESXi host, virtual machines are assigned a home node from which the virtual machine's memory is allocated. Because it is rare for a virtual machine to migrate away from the home node, memory access is mostly kept local.
In applications that scale out well it is beneficial to size the virtual machines with the NUMA node size in mind. For example, in a system with two hexa-core processors and 64GB of memory, sizing the virtual machine to six virtual CPUs and 32GB or less, means that the virtual machine does not have to span multiple nodes.
VSPEX VMware Storage Virtualization
Disk provisioning on the EMC VNX series requires administrators to choose disks for each of the storage pools.
Storage Layout
The architecture diagram in this section shows the physical disk layout. Disk provisioning on the VNX5500 storage array is simplified through the use of wizards, so that administrators do not choose which disks belong to a given storage pool. The wizard may choose any available disk of the proper type, regardless of where the disk physically resides in the array.
The reference architecture uses the following configuration:
•
Note
•
•
•
•
•
The VNX family storage array is designed for five 9s availability by using redundant components throughout the array. All of the array components are capable of continued operation in case of hardware failure. The RAID disk configuration on the array provides protection against data loss due to individual disk failures, and the available hot spare drives can be dynamically allocated to replace a failing disk.
Storage Virtualization
NFS is a cluster file system that provides UDP based stateless storage protocol to access storage across multiple hosts over the network. Each virtual machine is encapsulated in a small set of files and NFS datastore mount points are used for the operating system partitioning and data partitioning.
It is preferable to deploy virtual machine files on shared storage to take advantage of VMware VMotion, VMware High Availability™ (HA), and VMware Distributed Resource Scheduler™ (DRS). This is considered a best practice for mission-critical deployments, which are often installed on third-party, shared storage management solutions.
Architecture for 250 VMware Virtual Machines
Figure 12 shows the logical layout of 50 VMware virtual machines. The following are the key aspects of this solution:
•
•
•
•
•
•
Figure 12 Cisco Solution VMware Architecture for 250 VMs
Stateless Computing
UCS Manager (UCSM) provides the concept of Service Profile for server running on a physical hardware. Service profile is a logical entity, which can be associated to the physical server. Among other things, service profile includes various identities of the server or server components, such as:
•
•
•
•
•
•
All these identities can be assigned to any physical server managed by the UCSM. All other configuration of the service profile is based on templates, pools and policies, providing immense flexibility to the administrator. This includes firmware and BIOS versions required by the server. These concepts enable UCSM to provide stateless computing across entire UCSM managed compute hardware. If remote storage is used to boot operating system of the server (such as SAN boot, PXE boot, iSCSI boot etc), then a given service profile can be associated to any physical server hardware and downtime for migrating such server can be reduced to few minutes. Solution presented in this CVD makes use of identity pools and SAN storage to simplify the server procurement and provide stateless computing capability.
Sizing Guidelines
In any discussion about virtual infrastructures, it is important to first define a reference workload. Not all servers perform the same tasks, and it is impractical to build a reference that takes into account every possible combination of workload characteristics.
Defining the Reference Workload
To simplify the discussion, we have defined a representative customer reference workload. By comparing your actual customer usage to this reference workload, you can extrapolate which reference architecture to choose.
For the VSPEX solutions, the reference workload was defined as a single virtual machine. This virtual machine has the following characteristics:
This specification for a virtual machine is not intended to represent any specific application. Rather, it represents a single common point of reference to measure other virtual machines.
Applying the Reference Workload
When considering an existing server which will move into a virtual infrastructure, you have the opportunity to gain efficiency by right-sizing the virtual hardware resources assigned to that system.
The reference architectures create a pool of resources sufficient to host a target number of reference virtual machines as described above. It is entirely possible that customer virtual machines may not exactly match the specifications above. In that case, you can say that a single specific customer virtual machine is the equivalent of some number of reference virtual machines, and assume that number of virtual machines have been used in the pool. You can continue to provision virtual machines from the pool of resources until it is exhausted. Consider these examples:
Example 1 Custom Built Application
A small custom-built application server needs to move into this virtual infrastructure. The physical hardware supporting the application is not being fully utilized at present. A careful analysis of the existing application reveals that the application can use one processor, and needs 3 GB of memory to run normally. The IO workload ranges between 4 IOPS at idle time to 15 IOPS when busy. The entire application is only using about 30 GB on local hard drive storage.Based on these numbers, following resources are needed from the resource pool:- CPU resources for 1 VM- Memory resources for 2 VMs- Storage capacity for 1 VM- IOPS for 1 VMIn this example, a single virtual machine uses the resources of two of the reference VMs. Once this VM is deployed, the solution's new capability would be 248 VMs.Example 2 Point of Sale System
The database server for a customer's point-of-sale system needs to move into this virtual infrastructure. It is currently running on a physical system with four CPUs and 16 GB of memory. It uses 200 GB storage and generates 200 IOPS during an average busy cycle.The following resources that are needed from the resource pool to virtualize this application:- CPUs of 4 reference VMs- Memory of 8 reference VMs- Storage of 2 reference VMs- IOPS of 8 reference VMsIn this case the one virtual machine uses the resources of eight reference virtual machines. Once this VM is deployed, the solution's new capability would be 242 VMs.Example 3 Web Server
The customer's web server needs to move into this virtual infrastructure. It is currently running on a physical system with two CPUs and 8GB of memory. It uses 25 GB of storage and generates 50 IOPS during an average busy cycle.The following resources that are needed from the resource pool to virtualize this application:- CPUs of 2 reference VMs- Memory of 4 reference VMs- Storage of 1 reference VMs- IOPS of 2 reference VMsIn this case the virtual machine would use the resources of four reference virtual machines. Once this VM is deployed, the solution's new capability would be 246 VMs.Example 4 Decision Support Database
The database server for a customer's decision support system needs to move into this virtual infrastructure. It is currently running on a physical system with 10 CPUs and 48 GB of memory. It uses 5 TB of storage and generates 700 IOPS during an average busy cycle.The following resources that are needed from the resource pool to virtualize this application:- CPUs of ten reference VMs- Memory of 24 reference VMs- Storage of 52 reference VMs- IOPS of 28 reference VMsIn this case the one virtual machine uses the resources of fifty-two reference virtual machines. Once this VM is deployed, the solution's new capability would be 198 VMs.Summary of Example
The four examples show the flexibility of the resource pool model. In all the four cases the workloads simply reduce the number of available resources in the pool. If all four examples were implemented on the same virtual infrastructure, with an initial capacity of 250 virtual machines they can all be implemented, leaving the capacity of one hundred eighty six reference virtual machines in the resource pool.
In more advanced cases, there may be tradeoffs between memory and I/O or other relationships where increasing the amount of one resource, decreases the need for another. In these cases, the interactions between resource allocations become highly complex, and are out of the scope of this document. However, once a change in the resource balance is observed, and the new level of requirements is known; these virtual machines can be added to the infrastructure using the method described in the above examples.
VSPEX Configuration Guidelines
This sections provides the procedure to deploy the Cisco solution for EMC VSPEX VMware architecture.
Follow these steps to configure the Cisco solution for EMC VSPEX VMware architectures:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
These steps are described in detail in the following sections.
Pre-deployment Tasks
Pre-deployment tasks include procedures that do not directly relate to environment installation and configuration, but whose results will be needed at the time of installation. Examples of pre-deployment tasks are collection of hostnames, IP addresses, VLAN IDs, license keys, installation media, and so on. These tasks should be performed before the customer visit to decrease the time required onsite.
•
•
•
Customer Configuration Data
To reduce the onsite time, information such as IP addresses and hostnames should be assembled as part of the planning process.
The section Customer Configuration Data Sheet provides tabulated record of relevant information (to be filled at the customer's end). This form can be expanded or contracted as required, and information may be added, modified, and recorded as the deployment progresses.
Additionally, complete the VNX Series Configuration Worksheet, available on the EMC online support website, to provide the most comprehensive array-specific information.
Physical setup
Physical setup includes the following tasks:
1.
2.
3.
Preparing Cisco UCS Components
For information on mounting the hardware, see the Cisco UCS B-Series Hardware Installation Guide. Care must be taken about efficient cooling and proper airflow while mounting any equipment in the data center. Similarly, you need to pay attention to power requirements of chassis, servers and fabric interconnects.
Cisco UCS 5108 chassis, including its embedded blade servers and fabric extenders do not require management connectivity as they are managed by the fabric interconnects. Fabric interconnects are deployed in pair for high availability. Both the fabric interconnects require 100 Mbps peer connectivity for synchronizing the management plane between them. In addition, both the FIs require 1Gbps out-of-band management connectivity.
Cisco UCS Manager software runs on the Cisco UCS Fabric Interconnects. The Cisco UCS 6000 Series Fabric Interconnects expand the UCS networking portfolio and offer higher capacity, higher port density, and lower power consumption. These interconnects provide the management and communication backbone for the Cisco UCS B-Series Blades and Cisco UCS Blade Server Chassis. All chassis and the blade servers attached to the interconnects are part of a single, highly available management domain. By supporting unified fabric, the Cisco UCS 6000 Series provides the flexibility to support LAN and SAN connectivity for all blade servers within its domain right at the configuration time. Typically deployed in redundant pairs, the Cisco UCS Fabric Interconnect provides uniform access to both network and storage, facilitating a fully virtualized environment.
Initial setup steps of Cisco UCS 6248UP Fabric Interconnects and the Cisco UCS Manager are similar to those of the Nexus 5548UP switches:
1.
2.
3.
4.
5.
Cisco UCS 5108 Chassis, Cisco UCS 2104XP Fabric Extenders and UCS B200 M3 blade servers would be part of the UCS Manager (UCSM) management domain, so no special configuration is required for them.
Preparing Cisco Nexus Switches
Cisco Nexus 5548UP switches are 1RU top of the rack 10Gbps Ethernet and Fibre Channel switches. For information on how to deploy these switches, see Nexus 5548UP Product Documentation.
For initial configuration of these switches, follow these steps:
1.
2.
3.
4.
Preparing EMC VNX5500
For information on mounting the storage array EMC VNX5500 and initial configuration, see the EMC product documentation. Proper connectivity of storage controllers and DAEs are crucial for high availability of the storage.
Topology Diagram for 250 Virtual Machines
Following diagrams show connectivity details cable connectivity of solution covered in this document. At high level, cable connectivity can be divided in two parts:
•
•
As it is apparent from the following figure, there are five major cabling sections for the Ethernet connectivity:
•
•
•
•
•
Figure 13 Topology Diagram for 250 VMs
Table 10, Table 11 and Figure 14 provides the detailed cable connectivity for the EMC VSPEX 250 virtual machines configuration. Table 10 lists all the device port links from the Cisco Nexus 5548UP Switch perspective. Table 11 lists all the device port links from the Cisco UCS 6248UP Fabric Interconnects.
Figure 14 Port Connectivity from the Cisco UCS Fabric Interconnect Perspective
After connecting all the cables as per Table 10 and Table 11, you can configure the switch and the fabric interconnect.
Following are the important points to note:
•
•
•
•
Fibre Channel connectivity
This solution uses Fibre Channel over Ethernet (FCoE) protocol from UCS B200 M3 servers to UCS fabric interconnects. This reduces number of cables required between fabric interconnect and UCS blade server chassis by half. Native fibre channel cables are required from FIs to Nexus 5548UP switches and from there to storage devices. Use following guideline to connect the fibre channel links:
•
•
Connect all the cables as shown in Figure 14 you will be ready to configure UCSM and switches.
Configuring Cisco Nexus Switches
This section explains switch configuration needed for the Cisco solution for EMC VSPEX VMware architectures. For information on configuring password, and management connectivity, see the Cisco Nexus 5000 Series Configuration Guide.
Configure Global VLANs and VSANs
Figure 15 shows how to configure VLAN on a switch.
Figure 15 Creating VLAN
Following VLANs in Table 12 need to be configured on both switches A and B in addition to your application specific VLANs:
For actual VLAN IDs of your deployment, see Customer Configuration Data Sheet.
We have used one VSAN in this solution. Table 13 gives the VSAN name and the description.
Table 13 Configured Vsan To Access Storage Array
Storage
VSAN to access storage array from the servers over fibre channel
For actual VSAN ID of your deployment, see Customer Configuration Data Sheet.
Figure 16 and Figure 17 show the creation of VSAN and assigning VSAN to the fibre channel interface.
Figure 16 Creating VSAN
After creating the VSAN. VSAN membership is assigned, and the peer interfaces on the links need to be configured properly, a healthy fibre channel port is shown in Figure 17.
Figure 17 Assigned VSAN Membership
It is also crucial to enable NPIV feature on the Cisco Nexus 5548UP switches. Figure 18 show how to enable NPIV feature on Nexus 5548UP switches.
Figure 18 Enabling Npiv Feature On Cisco Nexus Switches
Configuring Virtual Port Channel (VPC)
Virtual port-channel effectively enables two physical switches to behave like a single virtual switch, and port-channel can be formed across the two physical switches. Following are the steps to enable vPC:
1.
2.
3.
4.
Figure 19 Configuring Peer-Gateway
5.
Figure 20 Configured VPC Peer-link on Port-Channel
l
6.
7.
Figure 21 Window Showing Successful vPC Configuration
Configuring Port-Channels Connected to Cisco UCS Fabric Interconnects
Interfaces connected to the fabric interconnects need to be in the trunk mode. Storage, vMotion, infra, and application VLANs are allowed on this port. From the switch side, interfaces connected to Cisco UCS FI-A and Cisco UCS FI-B are in a vPC, and from the FI side the links connected to Cisco Nexus 5548UP A and B switches are in LACP port-channels. Ensure that you give a right description for each port and port-channel on the switch for better diagnosis in case of any problem. Figure 22 shows the configuration commands.
Figure 22 Port-channel Configuration
Configuring Storage Connectivity
From each switch one link connects to each storage processor on the VNX5500 storage array. A virtual port-channel is created for the two links connected to a single storage processor, but connected to two different switches. In this example configuration, links connected to the storage processor A (SP-A) of VNX5500 storage array are connected to Ethernet port 1/26 on both the switches and links connected to the storage processor B (SP-B) are connected to Ethernet port 1/25 on both the switches. A virtual port-channel (id 26) is created for the Ethernet port 1/26 on both the switches and another virtual port-channel (id 25) is created for the Ethernet port 1/25 on both the switches.
Note
Figure 23 shows the configuration on the port-channels and interfaces.
Figure 23 Configuration of Port-channel and Interfaces
Configuring Ports Connected To Infrastructure Network
Port connected to infrastructure network need to be in trunk mode, and they require at least infrastructure VLAN, N1k control and packet VLANs at the minimum. You may require enabling more VLANs as required by your application domain. For example, Windows virtual machines may need to access to active directory / DNS servers deployed in the infrastructure network. You may also want to enable port-channels and virtual port-channels for high availability of infrastructure network.
Verify VLAN and Port-channel Configuration
At this point of time, all ports and port-channels are configured with necessary VLANs, switchport mode and vPC configuration. Validate this configuration using the "show vlan", "show port-channel summary" and "show vpc" commands as shown in Figure 24.
Note
Figure 24 Validating Created Port-Channels with VLANs
"show vlan" command can be restricted to a given VLAN or set of VLANs as shown in Figure 24. Ensure that on both switches, all required VLANs are in "active" status and right set of ports and port-channels are part of the necessary VLANs.
Port-channel configuration can be verified using "show port-channel summary" command. Figure 25 shows the expected output of this command.
Figure 25 Verifying Port-Channel Configuration
In this example, port-channel 1 is the vPC peer-link port-channel, port-channels 25 and 26 are connected to the storage arrays and port-channels 18 and 19 are connected to the Cisco UCS FI A and B. Make sure that the state of the member ports of each port-channel is "P" (Up in port-channel).
Note
Common reasons for port-channel port being down are:
•
•
vPC status can be verified using "show vpc" command. Example output is shown in Figure 26.
Figure 26 Verifying VPC Status
Ensure that the vPC peer status is "peer adjacency formed ok" and all the port-channels, including the peer-link port-channel status are "up".
Configuring QoS
The Cisco solution for the EMC VSPEX VMware architectures require MTU to be set at 9216 (jumbo frames) for efficient storage and vMotion traffic. MTU configuration on Cisco Nexus 5000 fall under global QoS configuration. You may need to configure additional QoS parameters as needed by the applications. For more information on the QoS configuration, see Cisco Nexus 5000 Series Configuration Guide.
To configure jumbo MTU on the Cisco Nexus 5000 series switches, follow these steps on both switch A and B:
1.
2.
3.
Figure 27 shows the exact Cisco Nexus CLI for the steps mentioned above.
Figure 27 Configuring MTU on Cisco Nexus Switches
Note
Configuring Cisco Unified Computing System Using Cisco UCS Manager
We would use web interface of Cisco UCS Manager (UCSM) to configure Cisco Unified Computing System. Cisco Unified Computing System configuration is broadly divided in two parts:
•
•
To launch UCSM, access https://<UCSM-Virtual-IP>/. By default, UCSM uses self-signed certificate, and so, browser would give untrusted SSL certificate warning. Ignore the warning and allow the initial Web UI page to load. Click Launch UCSM button. A Java applet gets automatically downloaded and the Cisco UCS Manager login page appears. Enter the administrator's username/ password. Provide the right credential and let the Java based UCSM client application run.
Configuring VLANs
To create and configure VLANs, follow these steps:
1.
Figure 28 Creating VLANs
2.
Figure 29 VLAN Details
3.
4.
Figure 30 Window Showing all the Created VLANs
Configuring VSANs
1.
Figure 31 Creating VSANs
2.
Figure 32 VSAN Details
3.
Configure Fibre Channel Uplink Ports
Fibre Channel ports on the Cisco UCS FIs are classified as uplink ports by default, but they are under VSAN by default. Click the SAN tab in the UCSM window and select the uplink FC interface connected to the Cisco Nexus 5548UP switches. From the drop-down text box for VSAN, select Boot Vsan created in step 2 of Configuring VSANs as shown in Figure 33. Click Save Changes button. Repeat this for all the uplink FC interfaces on both the Cisco UCS FIs.
Figure 33 Mapping FC Uplink Ports to Created VSAN
Configuring Ethernet Uplink Port-Channels
Virtual port-channels (vPC) on the Nexus 5548UP switches terminate on the UCS FIs as regular LACP port-channels. Follow these steps to configure uplink port-channels. Note that Ethernet ports on the UCS FIs are classified as "Unconfigured" by default, and need to be classified as uplink or server ports.
1.
Figure 34 Creating Port-Channel
2.
Figure 35 Port-Channel Details
3.
Figure 36 Adding Ports to the Port-Channel
4.
5.
Figure 37 Enabling Port-Channels in Cisco UCS Manager
6.
7.
Figure 38 Overall Status of all the Port-Channels
8.
Chassis and Server Discovery
After configuring uplink connectivity and global VLANs and VSANs, we need to configure server side connectivity for chassis and server discovery steps.
When the initial configuration in UCSM is completed through the serial console, the cluster state of the Cisco UCS Manager remains as "HA Not Ready" as shown in Figure 39. This is because there is no shared storage between two fabric interconnects due to lack of blade server chassis configuration on the UCS domain. Upon configuring two chassis in this solution, the HA state of the Cisco UCS Manager would transition to "HA Ready".
Figure 39 Cluster State of Cisco UCS Manager
As this solution requires 25 VMs per server, we need 32 dynamic VNICs, two static VNICs and two more VHBAs per server, we need four links between Cisco UCS Fabric Interconnects and Cisco UCS Fabric Extenders. The default chassis discovery policy supports one link between chassis and FI, so, we need to change the chassis discovery policy to "4 Link".
To change chassis discovery policy, click the Equipment tab in the UCSM window, expand the Equipment tree root, and select the Policies tab on the right pane of the UCSM window as shown in Figure 40. Select the option "4 Link" from the "Action" drop down list in the "Chassis Discovery Policy" and click Save Changes.
Figure 40 Changing Chassis Discovery Policy Settings
Marking Server Ports
After changing chassis discovery policy, next step is to classify interfaces connected to fabric extender as server ports on the fabric interconnect. Follow these steps for the same:
1.
Figure 41 Selecting Port to Configure as Server Port
2.
3.
Figure 42 Overall Status of the Configured Port
4.
Once all the server ports are configured, Cisco UCS Manager will auto-discover the chassis connected to the Cisco UCS Fabric Interconnects. Chassis objects will show up under the Equipment tab in Cisco UCS Manager, and upon successful deep discovery of chassis, the "Overall Status" of the chassis will change to "Operable" as shown in Figure 43. Also ensure that the two IOMs (Fabric Extenders) are listed under each chassis by expanding the individual chassis.
Figure 43 Change in the Overall Status of the Chassis
Once the chassis is discovered, deep discovery of the embedded Cisco UCS B200 M3 Blade Servers would also get started. You can see the progress of the Blade Server discovery FSM. To see the discovery status, expand Chassis, expand Servers, select the required server and then click the FSM tab on right pane of the UCSM window as shown in Figure 44.
Figure 44 Discovery Status
When deep discovery of the server is complete, the "Status of Last Operation" will change to "Discover Success" and "Progress Status" will reach 100%. The success notification of the server discovery will also show up and "Overall Status" becomes "Unassociated" in the General tab of UCSM.
Figure 45 Status Details of the Server
QoS Configuration
We need to configure system QoS classes and vNIC in QoS policies, which plays part in the end-to-end jumbo MTU configuration of the solution. Follow these steps to configure QoS on the UCSM:
1.
Figure 46 Enabling Classes in UCSM
2.
Figure 47 Creating QoS Policies
3.
Figure 48 Details to Create QoS Policies
4.
Figure 49 Details for Creating Another QoS Policy
These QoS policies would be used in the port-profiles created at a later point during the solution deployment.
Configuring Identifier Pools
As described in the "Stateless Computing" section of the "Architectural Overview", Service Profile - a logical representation of the server - includes various identities. Best way to manage various identities is to configure identifier pools. We will begin with defining external management IP address pool for the servers. Most common use of external management IP address of the server is launch of KVM of the server. KVM also includes virtual CD-ROM media launch, which we would use at later point in deployment of this solution.
Configuring external management IP address pool
An IP address pool named "ext-mgmt" is predefined in UCSM. Follow these steps to populate the pool with IP addresses for the out-of-band management of the Cisco UCS B200 M3 blade servers.
1.
Figure 50 Creating Block of IP Addresses for the Pool
2.
Figure 51 Entering Parameters to Create Block of IP Addresses
3.
4.
Figure 52 List of Assigned IP Addresses
Configure UUID Pool
Follow these steps to create B200 M3 servers' BIOS UUID pool:
1.
Figure 53 Creating UUID Suffix Pool
2.
Figure 54 Entering Details for Creating UUID Suffix Pool
3.
Figure 55 Adding UUID Blocks
4.
Figure 56 Entering Parameters to Create a Block of UUID Suffixes
5.
Configure MAC Address Pools
For each ESXi host in the solution, we would need two vNIC cards, so we need at least 20 unique MAC addresses defined in a MAC address pool. Follow these steps to create MAC address pool.
1.
Figure 57 Creating MAC Address Pool
2.
Figure 58 Entering Details for Creating MAC Address Pool
3.
Figure 59 Adding MAC Addresses
4.
Figure 60 Entering Parameters to Create a Block of MAC Addresses
5.
Configuring WWNN Pool
Each ESXi host requires a unique Fibre Channel World Wide Node Name (WWNN), so we need at least 10 unique WWNN addresses defined in a WWN address pool. Follow these steps to create WWNN address pool.
1.
Figure 61 Creating WWNN Pool
2.
Figure 62 Entering Details for Creating WWNN Address Pool
3.
Figure 63 Adding WWNN Blocks
4.
Figure 64 Entering Parameters to Create WWNN Block
5.
Configuring WWPN Pool
Each ESXi host in this solution has two vHBAs. Each vHBA requires a unique Fibre Channel World Wide Port Name (WWPN), so we need at least 20 unique WWPN addresses defined in a WWN address pool. Follow these steps to create WWPN address pool.
1.
Figure 65 Creating WWPN Pool
2.
Figure 66 Entering Details for Creating WWPN Address Pool
3.
Figure 67 Adding WWNN Blocks
4.
Figure 68 Entering Parameters to Create WWPN Block
5.
Create Service Profile Template
After configuring all necessary pools, next step is to define Service Profile Template. Given that all ten B200 M3 Blade Servers have identical ESXi 5 hypervisor configuration, Service Profile Template is the most convenient approach. Follow these steps to configure service profile template.
1.
Figure 69 Creating Service Profile Template
2.
Figure 70 Identify Service Profile Template Window
3.
Figure 71 Storage Window of the Create Service Profile Template Wizard
4.
Figure 72 Creating vHBA on Fabric A
5.
Figure 73 Creating vHBA on Fabric B
6.
Figure 74 Networking Window of the Create Service Profile Template Wizard
7.
Figure 75 Adding vNIC on Fabric A
8.
Figure 76 Adding vNIC on Fabric B
9.
10.
11.
Figure 77 Create Boot Policy Window
12.
Figure 78 Adding SAN Boot
13.
Figure 79 Adding SAN Boot Target
14.
Figure 80 Entering Details in Add SAN Boot Target Window
15.
Figure 81 Successfully Created Boot Policy Window
16.
17.
18.
19.
20.
Figure 82 Success Notification of Created Service Profile Template
Create service profile instances from the template
In this section we will create ten service profile instances from the template created in the previous section. Follow these steps to create service profile instances:
1.
Figure 83 Creating Service Profiles from Template
2.
Figure 84 Entering Details for Creating Service Profile instance
3.
4.
Figure 85 Created Service Profile Instances
Associate Service Profiles
As mentioned before, Service Profile is a logical representation of a server. When a Service Profile is associated with available physical server, the server assumes the role described by the Service Profile and corresponding server is booted. we need to associate Service Profile instances created in previous section to the Cisco UCS B200 M3 Blade Servers available. Follow these steps to associate Service Profiles:
1.
Figure 86 Changing Service Profile Association
2.
Figure 87 Associating Service Profile
3.
Figure 88 Service Profile Association Process in Progress
4.
Figure 89 Overall Status of the Server
This completes the UCS server configuration. We need to configure the UCSM/ vCenter integration in UCSM and Cisco VMFEX architecture after the vSphere infrastructure is setup.
Preparing and Configuring Storage Array
To configure the EMC VNX5500 storage array follow these steps:
1.
2.
3.
4.
Configure SAN Boot Infrastructure
This section explains how to configure end-to-end SAN Boot Infrastructure for the Cisco UCS B200 M3 Blade Servers. Most of the configuration is on the EMC VNX5500, but part of it is on Cisco Nexus 5548UP switches and UCSM. we have the following tasks completed already:
1.
2.
Follow these steps to configure SAN Boot Infrastructure:
1.
Figure 90 Assigned WWPN Values
2.
Figure 91 Creating Zones for Each of the ESX Hosts
3.
Figure 92 Validating the Activation of Zoneset on Fabric A
4.
Figure 93 Validating the Activation of Zoneset on Fabric B
5.
Figure 94 Validating the Created Zoneset Across SAN Fabric
6.
–
–
–
Similarly, verify the FLogI entries on SAN fabric B.
Figure 95 Total Number of flogi Sessions
7.
Figure 96 Creating Storage Pools in EMC VNX5500 Unisphere
8.
Figure 97 Entering Details for Creating Storage Pool
9.
Figure 98 Selecting Disks for ESXi 5 Hypervisor
10.
Figure 99 Confirmation Window to Initiate RAID Group Operation
11.
Figure 100 Success Notification of RAID Group Creation
12.
Figure 101 Creating LUN in EMC Unisphere
13.
Figure 102 Entering Details to Create LUNs
14.
Figure 103 Window Showing LUN Creation in Progress
15.
Figure 104 Adding Hosts to the Host List in the EMC Unisphere Window
16.
Figure 105 Host Initiator Connectivity Status
17.
Figure 106 Entering Details for Editing Initiators
18.
Figure 107 Confirmation to Register Initiator with Existing Host Information
19.
Figure 108 Editing the Host Initiators
20.
Figure 109 Browsing for the Existing Host
21.
Figure 110 Selecting the Existing Host Initiators
22.
Figure 111 Entering Initiator Information
23.
Figure 112 Window Showing WWPNs of ESXi Host
24.
Figure 113 Connectivity Status of All the TEN Servers
25.
Figure 114 Managing Storage Groups in the EMC Unisphere Window
26.
Figure 115 Creating Storage Groups
27.
Figure 116 Creating Host Groups
28.
Figure 117 Confirmation to Add LUNs
29.
Figure 118 Adding LUNs
30.
Figure 119 Selecting the Hosts to be Connected
31.
Figure 120 Confirmation to Connect the Hosts to the Storage Group
32.
Figure 121 Storage Group After Adding All the Hosts
33.
Figure 122 Launch the KVM Console
34.
Figure 123 Reboot the Server
35.
Figure 124 Details of Cisco UCS B 200 M3 Blade Server
As there is no bootable image yet which is installed on the LUN, the server will not actually boot; however this is a validation of end-to-end SAN boot infrastructure from Cisco UCS B200 M3 Blade Servers to the VNX5500 LUN.
Configure NFS Storage
This section covers the configuration of NFS storage on VNX5500 storage array.
1.
Figure 125 Selecting Storage Pools in the EMC Unisphere
2.
Figure 126 Creating Storage Pools
3.
Note
Figure 127 Entering Details for Creating Storage Pools
4.
Figure 128 Confirmation on Creating Storage Pool
5.
Figure 129 Window Showing Successful Creation of Storage Pool
6.
Refresh button, until initialization state shows "Ready". To add 75 more disks to the pool, select "PerformancePool" and click Expand.
Figure 130 Adding More Disks to the Pool
7.
Figure 131 Expanding Storage Pool
8.
Figure 132 Completion of Storage Pool Expansion
9.
Figure 133 Window Showing Storage Pools After the Expansion
10.
Figure 134 Selecting Hot Spares in EMC Unisphere
11.
Figure 135 Creating Hot Spares
12.
Figure 136 Entering Storage Pool Parameters
13.
Figure 137 Window Showing Successful Creation of RAID Group 1
14.
Figure 138 Window Showing Successful Creation of RAID Group 7
15.
Figure 139 Window Showing Hot Spare Status
16.
Figure 140 Creating LUN
17.
Figure 141 Entering Details to Create LUNs
18.
Figure 142 Confirmation to Create LUNs
19.
Figure 143 Window Showing LUN Creation Progress Indicator
20.
Figure 144 Window Showing Successful LUN Creation
21.
Figure 145 Adding the Created LUNs to Storage Group
22.
Figure 146 Adding Storage Groups
23.
Figure 147 Ensuring the Storage Group is Added
24.
Figure 148 Confirmation Window to Add LUNs
25.
Figure 149 Host Information for All the Added LUNs
26.
Figure 150 Window Showing System Volumes Created by Default
27.
Figure 151 CLI Showing List of NAS Disks
28.
Figure 152 Command to Show All the LUNs as Disk Volumes
29.
Figure 153 CLI Showing Discovery Process
30.
Figure 154 Command Showing All the LUNs Discovered as Disk Volumes
31.
Figure 155 Verify the Storage Capacity for All the Disk Volumes
32.
Figure 156 Creating LACP Interface
33.
Figure 157 Creating Network Device
34.
Figure 158 Entering Details to Create Network Device
35.
Figure 159 New Network Device is Created
36.
Figure 160 Creating Interfaces
37.
Figure 161 Entering Details to Create Network Interface
38.
Figure 162 New Network Interface is Created
39.
Figure 163 Entering Details to Create File System
40.
Figure 164 Window Showing NFS-OS File System Creation in Progress
41.
Figure 165 Window Showing NFS-OS Created with 2 TB Capacity
42.
Figure 166 Creating File System with 2.5 TB Storage Capacity
43.
Figure 167 Creating NFS Data File Systems at 2.5 TB Storage Capacity
44.
Figure 168 Window Showing All the NFS Data File Systems Created
45.
Figure 169 Mounts Tab of File System Window
46.
Figure 170 Window Showing the Path for NFS File Systems
47.
Figure 171 Enabling Read/Write for NFS-OS
48.
Figure 172 Enabling Parameters for NFS-OS
49.
50.
Figure 173 Creating NFS Exports
51.
Figure 174 Entering Details for Creating NFS Exports
52.
Figure 175 Window Showing Created NFS Export for NFS-OS
53.
Installing ESXi Servers and vCenter Infrastructure
1.
Figure 176 Launch KVM Console
2.
Figure 177 Adding Image in Virtual Media
3.
Figure 178 Reset the Server After Adding the ISO Image
4.
Figure 179 Selecting Power Cycle Option to Restart the Server
5.
–
–
–
See, Customer Configuration Data Sheet for appropriate values.
6.
Installing and Configuring Microsoft SQL Server Database
SQL server is used as database for the VMware vCenter server. Follow these steps to configure Microsoft SQL server.
1.
Note
2.
3.
4.
Note
5.
6.
VMware vCenter Server Deployment
This section describes the installation of VMware vCenter for VMware environment and to complete the following configuration:
•
•
•
For detailed information on Installing a vCenter Server, see the link:
http://pubs.vmware.com/vsphere-50/index.jsp?topic=/com.vmware.vsphere.install.doc_50/GUID-A71D7F56-6F47-43AB-9C4E-BAA89310F295.html.
For detailed information on vSphere Virtual Machine Administration, see the link:
For detailed information on creating a Virtual Machine in the vSphere 5 client, see the link:
Following steps provides high level configuration to configure vCenter server:
1.
2.
3.
For instructions on how to create the necessary ODBC connections see, vSphere Installation and Setup and Installing and Administering VMware vSphere Update Manager.
4.
5.
Configuring Cluster, HA and DRS on the vCenter
To add all the VMware on virtual machine vCenter, follow these steps:
1.
2.
3.
a.
b.
4.
Figure 180 Window Showing vCenter Cluster in VMware vSphere Client
Configure Cisco VMFEX
Technology Overview section detailed about benefits of Cisco VMFEX technology. This section explains step by step configuration guide for UCSM/ vCenter management plane integration and Cisco VMFEX technology implementation. Follow these steps to configure Cisco VMFEX architecture.
1.
Figure 181 Modifying Extension Key in the UCSM Window
2.
Figure 182 Modifying Extension Key
3.
Figure 183 Exporting vCenter Extension
4.
Figure 184 Specifying Path for vCenter Extension File
5.
Figure 185 Managing Plug-ins in VMware vSphere Client
6.
Figure 186 Creating New Plug-in—Plug-in Manager Window
7.
Figure 187 Registering the Plug-in
8.
Figure 188 Window Showing the Plug-in Registered in vCenter
9.
Figure 189 UCSM Plug-in Listed in the Available Plug-ins
10.
Figure 190 Configuring vCenter in UCSM
11.
Figure 191 Entering Details in the Configure vCenter Wizard
12.
Figure 192 Folders Window of the Configure vCenter Wizard
13.
Figure 193 Datacenters Window of the Configure vCenter Wizard
14.
Figure 194 Identifying the Datacenter
15.
Figure 195 Adding the Datacenter
16.
Figure 196 Creating Folder
17.
Figure 197 Adding a Distributed Virtual Switch
18.
Figure 198 Creating a Distributed Virtual Switch
19.
Figure 199 Wizard Showing Created UCS-DVS
20.
Figure 200 Wizard Showing Created Folder UCSM
21.
Figure 201 Wizard Showing Created Datacenter V250
22.
Figure 202 Windows Showing vCenter VC Created Successfully
23.
Figure 203 vCenter Window Showing the Folder and DVS Created
24.
Figure 204 Creating Port profile in UCSM
25.
Figure 205 Entering Details to Create Port Profile
26.
Figure 206 Creating Profile Client in UCSM
27.
Figure 207 Entering Details for Creating Profile Client
28.
Figure 208 Creating Port Profile
29.
Figure 209 Creating Port Profile for Storage Traffic
30.
Figure 210 Window Showing Sample Port Profile Created
31.
Figure 211 UCS-DVS Showing All the Created Port Profiles
32.
Figure 212 Adding Host in UCS-DVS
33.
Figure 213 Selecting Hosts and Physical Adapters Window in Adding Hosts Wizard
34.
Figure 214 Network Connectivity Window in Adding Hosts Wizard
35.
Figure 215 Virtual Machine Networking Window in Adding Hosts Wizard
36.
Figure 216 Ready to Complete Window in Adding Hosts Wizard
37.
Figure 217 UCS-DVS Showing Hosts Details in vCenter
38.
Figure 218 Window Showing Host Server Status in UCSM
We need to create two more kernel interfaces per host, one for vMotion and one for NFS storage access by the kernel. Choose the appropriate port profiles for the same. For both the vMotion and the NFS kernel interfaces, choose the MTU to be jumbo 9000 MTS for bulk transfer efficiency. Follow these steps:
1.
Figure 219 Managing Virtual Adapters in vCenter
2.
Figure 220 Adding Virtual Kernel Interface
3.
Figure 221 Creation Type Window in Adding Virtual Adapter Wizard
4.
Figure 222 Connection Settings Window in Adding Virtual Adapter Wizard
5.
Figure 223 Entering IP Address Details
6.
Figure 224 Ready to Complete Window of Adding Virtual Adapters Wizard
7.
8.
Figure 225 Editing the VMkernel Interface
9.
Figure 226 Changing the MTU Size for the VMkernel Interface
10.
11.
Figure 227 VIrtual Machine Properties in UCSM
Jumbo MTU Validation And Diagnostics
To validate the jumbo MTU from end-to-end, SSH to the ESXi host. By default, SSH access is disabled to ESXi hosts. Enable SSH to ESXi host by editing hosts' security profile under "Configuration" tab.
Once connected to the ESXi host through SSH, initiate ping to the NFS storage server with large MTU size and set the "Do Not Fragment" bit of IP packet to 1. Use the vmkping command as shown in the example:
Example 5
~ # vmkping -d -s 8972 10.10.40.6411PING 10.10.40.64 (10.10.40.64): 8972 data bytes8980 bytes from 10.10.40.64: icmp_seq=0 ttl=64 time=0.417 ms8980 bytes from 10.10.40.64: icmp_seq=1 ttl=64 time=0.518 ms8980 bytes from 10.10.40.64: icmp_seq=2 ttl=64 time=0.392 ms--- 10.10.40.64 ping statistics ---3 packets transmitted, 3 packets received, 0% packet lossround-trip min/avg/max = 0.392/0.442/0.518 ms~ #Ensure that the packet size is 8972 due to various L2/L3 overhead. Also ping all other hosts' vMotion and NFS vmkernel interfaces. Ping must be successful. If ping is not successful verify that 9000 MTU configured. Follow these steps to verify:
1.
2.
3.
4.
5.
Template-Based Deployments for Rapid Provisioning
Figure 228 Rapid Provisioning
In an environment with established procedures, deploying new application servers can be streamlined, but this can still take many hours or days to complete. Not only should you complete an OS installation, but downloading and installing service packs and security updates can add a significant amount of time. Many applications require features that are not installed with Microsoft Windows by default and must be installed before installing the applications. Inevitably, those features require more security updates and patches. By the time all the deployment aspects are considered, more time is spent waiting for downloads and installs than that spent on configuring the application.
Virtual machine templates can help speed up this process by eliminating most of these monotonous tasks. By completing the core installation requirements, typically to the point where the application is ready to be installed, you can create a golden image which can be sealed and used as a template for all the virtual machines. Depending on how granular you want to make a specific template, the time to deploy can be as less as the time it takes to install, configure, and validate the application. You can use PowerShell tools and VMware vSphere Power CLI to bring down the time and manual effort dramatically, especially when you have a large number of VMs to deploy.
Validating Cisco Solution for EMC VSPEX VMware Architectures
This section provides a list of items that should be reviewed once the solution has been configured. The goal of this section is to verify the configuration and functionality of specific aspects of the solution, and ensure that the configuration supports core availability requirements.
Post Install Checklist
The following configuration items are critical to functionality of the solution, and should be verified before deploying for production.
•
•
•
•
Verify the Redundancy Of The Solution Components
Following redundancy checks were performed at the Cisco lab to verify solution robustness:
1.
2.
3.
4.
5.
6.
7.
Cisco Validation Test Profile
"vdbench" testing tool was used with Windows 2008 R2 SP1 server to test scaling of the solution in Cisco labs. Table 14 details on the test profile used.
Bill of Material
Table 15 gives the list of the components used in the CVD for 250 virtual machines configuration
For more information on details of the hardware components, see: http://www.cisco.com/en/US/prod/collateral/ps10265/ps10280/B200M3_SpecSheet.pdf
Customer Configuration Data Sheet
Before you start the configuration, gather the customer-specific network and host configuration information. Table 16, Table 17, Table 18, Table 19, Table 20, Table 21, Table 22 provide information on assembling the required network, host address, numbering, and naming information. This worksheet can also be used as a "leave behind" document for future reference.
Table 16 Common Server Information
Domain Controller
DNS Primary
DNS Secondary
DHCP
NTP
SMTP
SNMP
vCenter Console
SQL Server
Table 17 ESXi Server Information
Host 1
ESXi
....
....
Host 10
ESXi
Table 18 Array Information
Admin account
Management IP
Storage pool name
Datastore name
NFS Server IP
Table 19 Network Infrastructure Information
Cisco Nexus 5548UP A
Cisco Nexus 5548UP B
Cisco UCSM Virtual IP
Cisco UCS FI-A
Cisco UCS FI-B
Table 21 VSAN Information
vsan-storage
FC connectivity from server to storage
References
Cisco UCS:
http://www.cisco.com/en/US/solutions/ns340/ns517/ns224/ns944/unified_computing.html
VMware vSphere:
http://www.vmware.com/products/vsphere/overview.html
Cisco Nexus 5000 Series NX-OS Software Configuration Guide:
EMC VNX 5xxx series resources:
http://www.emc.com/storage/vnx/vnx-series.htm#!resources
Microsoft SQL Server installation guide:
http://msdn.microsoft.com/en-us/library/ms143219.aspx
Feedback