Cisco UCS Solution for EMC VSPEX Microsoft Private Cloud Fast Track 4.0

 

                                                                                                                                                                       

June 6, 2015

                                     

The CVD program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information visit

http://www.cisco.com/go/designzone.

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS.  CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.  IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE.  USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS.  THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS.  USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS.  RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2015 Cisco Systems, Inc. All rights reserved.

 

 

 

 

About Cisco Validated Design (CVD) Program... 2

Table of Contents.. 3

Introduction.. 16

Private Cloud Fast Track Program Description.. 16

Business Value.. 17

Technical Benefits.. 18

Program Requirements and Validation.. 18

Design Patterns Overview... 19

Core Fast Track Infrastructure.. 20

Architecture.. 22

Prerequisite Infrastructure.. 24

Active Directory Domain Services (AD DS) 24

Domain Name System (DNS) 24

IP Address Assignment and Management. 25

Software Revisions.. 25

PowerShell Deployment Toolkit (PDT) 33

Configuration Guidelines.. 33

Configuration Workstation.. 35

Windows 8.1 Workstation.. 35

Windows Server 2012 R2 Workstation.. 36

Both Workstations.. 36

Cisco PowerShell Modules.. 37

Cisco Nexus 1000V PowerShell Module.. 41

EMC ESI PowerShell 41

Deployment. 45

Cabling Information.. 46

Cisco Nexus 9396 Deployment. 50

Set up Initial Cisco Nexus 9396 Switch.. 50

Cisco Nexus 9396 A.. 50

Cisco Nexus 9396 B.. 52

Enable Cisco Nexus Features and Set Global Configurations.. 54

Cisco Nexus A and Nexus B.. 54

Create Necessary VLANs.. 55

Cisco Nexus A and Nexus B.. 55

Add Individual Port Descriptions for Troubleshooting.. 57

Cisco Nexus 9396 A.. 58

Cisco Nexus 9396 B.. 60

Create Necessary Port Channels.. 62

Cisco Nexus 9396 A.. 62

Cisco Nexus 9396 B.. 63

Add PortChannel Configurations.. 64

Cisco Nexus 9396 A and B.. 65

Configure Virtual Port Channel Domain.. 66

Cisco Nexus 9396 A.. 66

Cisco Nexus 9396 B.. 68

Enable SMB Links.. 70

Cisco Nexus 9396 A and B.. 70

Link into Existing Network Infrastructure.. 70

Configure HSRP (Optional) 71

Cisco Nexus 9396 A.. 71

Cisco Nexus 9396 B.. 72

EMC VNX5400 Deployment. 73

VNX Worksheets.. 74

Installing EMC Storage Integrator v3.1.. 80

Register the VNX for Use with the ESI PowerShell Toolkit. 83

Storage Configuration.. 84

Create Storage Pool 85

Create VNX LUNs.. 86

Configure SMB on EMC VNX5400.. 92

Configure DNS and NTP.. 92

Configure SMB Interfaces.. 94

Configure CIFS Network Service.. 96

Configure Storage.. 97

Join Active Directory Domain.. 98

Create SMB File System and Mounts.. 99

Associate Shares with CIFS Server. 102

Configure Cisco Unified Computing System Fabric Interconnects.. 103

Perform Initial Setup of the Cisco UCS 6248 Fabric Interconnects.. 103

Cisco UCS 6248 A.. 103

Cisco UCS 6248 B.. 104

Log into Cisco UCS Manager. 105

Add a Block of IP Addresses for KVM Access.. 106

Cisco UCS Manager. 106

Synchronize Cisco UCS to NTP.. 107

Cisco UCS Manager. 107

Edit the Chassis Discovery Policy.. 108

Cisco UCS Manager. 108

Enable Server and Uplink Ports.. 109

Cisco UCS Manager. 109

Configure Fibre Channel Switching Mode.. 111

Cisco UCS Manager. 111

Enable FCoE Storage Ports.. 112

Cisco UCS Manager. 113

Acknowledge the Cisco UCS Chassis.. 114

Cisco UCS Manager. 114

Create Uplink PortChannels to the Cisco Nexus 9396 Switches.. 115

Cisco UCS Manager. 115

Configure Policies, Pools, and Templates.. 117

Create an Organization (Optional) 118

Create a MAC Address Pool 119

Create WWN Pools.. 120

Create WWNN Pools.. 121

Create WWPN Pools.. 123

Create UUID Suffix Pools.. 124

Create Server Pools.. 126

Create VLANs.. 127

Create Host Firmware Package Policy.. 129

Enable Quality of Service in Cisco UCS Fabric.. 130

Create a Power Control Policy.. 133

Create a Scrub Policy.. 134

Create a Local Disk Configuration Policy.. 135

Create a Server Pool Qualification Policy.. 137

Create a Server BIOS Policy.. 139

Create vNIC/HBA Placement Policy for Virtual Machine Infrastructure Hosts.. 143

Create VMQ Connection Policy.. 144

Create vNIC Templates.. 145

Create VSANs.. 149

Assign VSANs to FCoE Storage Ports.. 151

Create vHBA Templates for Fabric A and B.. 152

Create Storage Connection Policies.. 154

Create Boot Policies.. 159

Create Service Profile Templates.. 165

Clone Template for Initial Build.. 173

Create Service Profiles.. 176

First Installation of Windows Server 2012 R2 Datacenter. 178

Present Master Boot LUN... 179

EMC Unisphere.. 179

EMC Enterprise Storage Integratore (ESI) and PowerShell 183

Install Windows Server 2012 R2.. 186

Local Configuration Tasks.. 197

Configure Management Network. 197

Run Windows Update.. 198

Install Windows Roles and Features.. 199

Configure Paging File.. 199

Configure Other Common Criteria.. 202

Update Chipset Drivers.. 204

Change Service Profile for MPIO... 207

Configure MPIO... 209

Sysprep the Image.. 212

Clone Newly Created Image.. 213

Build Infrastructure Servers.. 220

Mask LUNs to Servers.. 220

Complete Mini-Setup.. 224

Join Server to Fabric Management Domain.. 225

Rename Computer and Join to Domain.. 228

Configure Host Networks.. 228

Configure Hyper-V Virtual Switches.. 233

Install EMC PowerPath.. 235

Install EMC Unisphere Host Agent. 238

Create Fabric Management Hyper-V Cluster. 241

Add Shared Storage.. 241

Run Cluster Validation Wizard.. 248

Create Fabric Management Cluster. 249

Configure Disks.. 251

Configure Networks.. 252

Change Hyper-V Default Settings.. 255

Configure Cluster Aware Updating.. 258

Configure Constrained Delegation.. 261

(Optional) Configure Replica Broker Role.. 264

Fabric Management. 267

Fabric.. 267

Fabric Management. 268

Provisioning Fabric Management Hosts.. 268

Create Sysprepped Virtual Hard Disk. 275

Create Infrastructure Virtual Machines from Sysprepped Image.. 283

Create-VMs.ps1 PowerShell Script. 286

Configure Cluster Preferred Owners and Priority.. 292

Create Required User Accounts and Security Groups.. 293

SQL Server 2012 Failover Cluster Installation.. 309

Overview... 310

Prerequisites.. 312

Accounts.. 312

Groups.. 313

Establish the SQL Server Guest Cluster. 313

Rename and Join SQL VMs to Active Directory Domain.. 313

Assign Users and Groups to Local Administrators Group.. 317

Create and Add Shared VHDX to VM... 319

Initialize and Format VHDX.. 323

Create the SQL Cluster. 326

Assign SQL Access to SQL Share.. 330

Grant Installation Rights on VNX.. 333

Install First SQL Server Instance in Cluster. 334

Install the SQL Server Named Instances on the Guest Cluster (Additional Nodes) 350

PowerShell Deployment Toolkit SQL Server Installation.. 356

Post-Installation Tasks.. 363

Configure Windows Firewall Settings for SQL Server Named Instances.. 363

Assign Preferred Owners for SQL Server Instances in Failover Cluster Manager. 375

Virtual Machine Manager. 382

Overview... 383

Prerequisites.. 384

Accounts.. 384

Groups.. 384

Rename and Join SCVMM Servers to Active Directory Domain.. 385

Install the Windows Assessment and Deployment Kit. 387

Install the Prerequisite Windows Server Roles and Features.. 391

Install the Command-Line Utilities in SQL Server 2012 with SP2.. 396

Set Up Failover Cluster. 402

Create Shared VHDX File.. 403

Configure Shared Storage.. 407

Create the Failover Cluster. 408

Create the Virtual Machine Manager Distributed Key Management Container in Active Directory Domain Services.. 411

Installation.. 415

Install Virtual Machine Manager on First Node.. 415

Install Second Node of SCVMM Cluster. 425

Creating Virtual Machine Manager Library Share on a Failover Cluster. 433

Create a Run As Account. 448

Add Hyper-V Hosts to be Managed by SCVMM... 450

Register VNX File Share to SCVMM... 454

Configure Logical Networks.. 457

Configure Library Subdirectories (optional) 458

Configure Constrained Delegation (optional) 460

Install and Configure Cisco Nexus 1000V.. 462

Install the Virtual Supervisor Module Virtual Machine Template.. 463

Create Two Virtual Supervisor Module Virtual Machines.. 466

Add a Domain Name Service Record for the Virtual Supervisor Module VMs.. 473

Configure Virtual Servisor Modules in the VSM Virtual Machines.. 474

Configure the Cisco Nexus 1000V VSM for Use with SCVMM... 478

Connect SCVMM to VSM... 480

Configure a Logical Switch in SCVMM... 484

Create the Logical Switch on the Hyper-V Hosts.. 489

Create a VM Network. 492

Configure the SCVMM Virtual Machine Properties.. 493

Configure SCVMM Network Interfaces.. 497

Rename the New Cluster Network. 502

Configure System Center Application Virtual Machine Network Interfaces.. 506

Install Cisco UCS SCVMM Add-In.. 512

Install the SCVMM Add-In.. 513

Configure the SCVMM Add-In.. 514

Install and Configure the EMC SMI-S Provider. 515

Install the EMC SMI-S Provider. 516

Register the VNX with the Provider. 518

Create the SMI-S User for the SCVMM Run As Account. 520

Create the Run As Account within SCVMM... 522

Register the EMC SMI-S Provider with SCVMM... 524

Allocate Storage Pools to Host Groups.. 527

Create a SAN Copy Capable Template.. 529

Select the Rapid Provisioning Deployment Method.. 532

Using ODX for Virtual Machine Deployments.. 533

Operations Manager. 535

Overview... 536

Prerequisites.. 537

Accounts.. 537

Groups.. 538

Add .NET Framework 3.5.. 539

Install the SQL Server Reporting Services (Split Configuration) and Analysis Services.. 542

Install Microsoft Report Viewer 2012.. 562

Configure Operations Manager SQL Server Prerequisites.. 565

Installation.. 569

Install the Operations Manager Management Server. 569

Install the Second Operations Manager Management Server. 577

Install the Operations Manager Reporting Server. 578

Post-Installation Tasks.. 583

Register the Required Service Principal Names for the Operations Manager Management Servers   583

Deploy and Configure the Operations Manager Agent on the Virtual Machine Manager Management Server Nodes.. 585

Deploy Operations Manager Agent on Hyper-V Hosts.. 589

Install Operations Manager Console on the Virtual Machine Manager Management Server. 593

Download and Import the Required Prerequisite Management Packs in Operations Manager  597

Perform Virtual Machine Manager and Operations Manager Integration.. 600

Cisco UCS Management Pack. 604

Install the Cisco UCS Management Pack. 605

Add Firewall Exceptions.. 607

Add Cisco UCS Domains to Operations Manager. 607

Configure Run As Account. 614

Configure Fault Acknowledgement. 617

Installing and Configuring the EMC Storage Integrator Management. 620

Prepare ESI Infrastructure.. 620

Create ESI Service Account. 621

Assign Permissions to ESI Service Account. 621

Monitor ESI Management Server with SCOM... 625

Install ESI Components.. 628

Assign Monitor Role to ESI Service Account. 632

Register the VNX with the ESI Service.. 632

Install the ESI SCOM Management Packs.. 633

Import the ESI SCOM Management Packs.. 636

Create ESI Run As Account and Associate with a Profile.. 638

Setting Overrides for the EMC SI Service Discovery.. 644

Service Manager. 646

Overview... 647

Management Server. 647

Data warehouse server. 648

Self-service Portal Server. 649

Prerequisites.. 650

Accounts.. 650

Groups.. 651

Configure Service Manager Environmental Prerequisites.. 652

Add .NET Framework 3.5 on all Server Manager Servers.. 656

Install Microsoft Report Viewer 2008 SP1 Redistributable on the Management and Data Warehouse Servers.. 660

Install SQL Server 2012 Native Client on the on the Management and Data Warehouse Servers   662

Install SQL Server 2012 SP1 Analysis Management Objects.. 664

Install SQL Server Reporting Services (Split Configuration) on the Data Warehouse Server. 667

Install SharePoint Foundation 2010 SP2 on the Self-Service Portal Server. 681

Install .NET Framework 4 on the Self-Service Portal Server. 690

Request and Install an SSL Certificate on the Self-Service Portal Server. 692

Installation.. 693

Install the Management Server. 693

Install Subsequent Management Servers.. 702

Install the Data Warehouse Server. 703

Install the Service Manager Self-Service Portal Server. 722

Orchestrator. 732

Overview... 733

Prerequisites.. 733

Accounts.. 733

Groups.. 734

Add .NET Framework 3.5 and .NET Framework 4.5 with HTTP Activation.. 734

Install Silverlight. 740

Installation.. 740

Install the Full Management Server. 740

Install Second Server as Runbook Server. 753

Post-Installation Tasks.. 760

Install Microsoft Report Viewer 2012.. 760

Install the Operations Manager Console.. 764

Install the Virtual Machine Manager Console.. 767

Download and Register the Orchestrator Integration Packs.. 770

Deploy the Orchestrator Integration Packs.. 773

System Center 2012 R2 Orchestrator Integration Pack. 778

Register the Cisco UCS OIP.. 779

Deploy the Cisco UCS OIP.. 780

Configure the Cisco UCS OIP.. 781

Install Sample Runbooks.. 783

App Controller. 785

Overview... 785

Prerequisites.. 786

Accounts.. 786

Groups.. 786

Add .NET Framework 3.5.. 787

Install Silverlight. 790

Install the Virtual Machine Manager Console.. 791

Installation.. 794

Install the App Controller Portal Server. 794

Service Management Automation (SMA) 802

Overview... 803

Prerequisites.. 803

Accounts.. 803

Groups.. 804

Add Web Server Role (IIS) 804

Request and Install an SSL Certificate.. 808

Installation.. 809

Install the Web Service.. 809

Install the Runbook Worker. 815

Install the PowerShell Automation Module.. 821

System Center Service Provider Foundation.. 823

Overview... 824

Prerequisites.. 824

Accounts.. 824

Groups.. 825

Add Web Server Role (IIS) 827

Install Microsoft ASP.NET Model View Control (MVC) 4.. 831

Install WCF.. 832

Install the Virtual Machine Manager Console.. 833

Request and Install an SSL Certificate.. 837

Installation.. 839

Install System Center Service Provider Foundation 2012 R2.. 839

Service Reporting.. 846

Overview... 847

Prerequisites.. 847

Accounts.. 847

Groups.. 847

Add .NET Framework 3.5.. 847

Install SQL Server 2012 SP2.. 850

Installation.. 856

Install Service Reporting.. 856

Windows Azure Pack. 863

Overview... 863

Prerequisites.. 865

Accounts.. 865

Groups.. 865

Configure SQL Instance Permissions.. 865

Download Offline Cache.. 869

Deploy .NET 4.5 Extended with ASP.NET.. 871

Deploy IIS Recommended Configuration.. 872

Installation.. 873

Administration API 873

Tenant API 876

Tenant Public API 878

Administration Authentication Site.. 878

Tenant Authentication Site.. 879

Administration Site.. 879

Tenant Site.. 880

Database Server Setup.. 880

Post-Installation.. 884

Deploy Tenant Cluster. 885

Build Tenant Servers.. 886

Create Service Profile Template.. 886

Create Service Profiles.. 888

Clone VNX Boot LUN... 889

Associate Boot LUN to Service Profile.. 889

Complete Tenant Server Configuration.. 890

Create Tenant Hyper-V Cluster. 890

Cluster Shared Storage.. 891

Add Tenant Cluster to SCVMM... 894

Register VNX File Share to SCVMM... 896

Post-Deployment Tasks.. 899

Configure Cisco Nexus 1000V for Tenant NIC... 900

Configure Network Segment. 900

Configure Logical Switch in SCVMM... 900

Create Logical Switch on Hyper-V.. 900

Create a Virtual Machine Network. 902

About the Authors.. 903

Acknowledgments.. 903

The Microsoft Private Cloud Fast Track program is a joint effort between Microsoft and its hardware partners. The goal of the program is to help organizations develop and implement private clouds quickly while reducing both complexity and risk. The program provides a reference architecture that combines Microsoft software, consolidated guidance, and validated configurations with partner technology such as compute, network, and storage architectures, in addition to value-added software components.

The private cloud model provides much of the efficiency and agility of cloud computing, along with the increased control and customization that are achieved through dedicated private resources. With Private Cloud Fast Track, Microsoft and its hardware partners can help provide organizations with the control and the flexibility that are required to reap the potential benefits of the private cloud.

Private Cloud Fast Track utilizes the core capabilities of the Windows Server (OS), Hyper-V, and System Center to deliver a private cloud infrastructure as a service offering. These are also the key software components that are used for every reference implementation.

Private Cloud Fast Track Program Description

The Infrastructure as a Service Product Line Architecture (PLA) is focused on deploying virtualization fabric and fabric management technologies in Windows Server and System Center to support private cloud scenarios. This PLA includes reference architectures, best practices, and processes for streamlining deployment of these platforms to support private cloud scenarios.

This component of the IaaS PLA focuses on delivering core foundational virtualization fabric infrastructure guidance that aligns to the defined architectural patterns within this and other Windows Server 2012 R2 private cloud programs. The resulting Hyper-V infrastructure in Windows Server 2012 R2 can be leveraged to host advanced workloads, and subsequent releases will contain fabric management scenarios using System Center components.

Scenarios relevant to this release include:

·         Resilient infrastructure – Maximize the availability of IT infrastructure through cost-effective redundant systems that prevent downtime, whether planned or unplanned.

·         Centralized IT – Create pooled resources with a highly virtualized infrastructure that supports maintaining individual tenant rights and service levels.

·         Consolidation and migration – Remove legacy systems and move workloads to a scalable high-performance infrastructure.

·         Preparation for the cloud – Create the foundational infrastructure to begin transition to a private cloud solution.

The Fast Track program has two main solutions, as shown in Figure 1. This Cisco Validated Design will focus exclusively on the Open Solutions branch.

Figure 1        Branches of the Microsoft Private Cloud

Each branch in the Fast Track program uses a reference architecture that defines the requirements that are necessary to design, build, and deliver virtualization and private cloud solutions for small, medium, and large-size enterprise implementations.

Each reference architecture in the Fast Track program combines concise guidance with validated configurations for the compute, network, storage, and virtualization layers. Each architecture presents multiple design patterns for enabling the architecture, and each design pattern describes the minimum requirements for validating each Fast Track solution.

The Cisco and EMC Fast Track Solution presented in this document is an Open solution.  The Cisco and EMC with Microsoft Private Cloud Fast Track solution utilizes the core capabilities of Windows Server 2012 R2, Hyper-V, and System Center 2012 R2 to deliver a Private Cloud - Infrastructure as a Service offering. The key software components of every Reference Implementation are Windows Server 2012 R2, Hyper-V, and System Center 2012 R2.  The solution also includes software from Cisco and EMC to form a complete solution that is ready for your enterprise.

Business Value

The Cisco and EMC with Microsoft Private Cloud Fast Track solution provides a reference architecture for building private clouds on each organization’s unique terms. Each Fast-Track solution helps organizations implement private clouds with increased ease and confidence. Among the benefits of the Microsoft Private Cloud Fast Track Program are faster deployment, reduced risk, and a lower cost of ownership.

Reduced risk:

·         Tested, end-to-end interoperability of compute, storage, and network

·         Predefined, out-of-box solutions based on a common cloud architecture that has already been tested and validated

·         High degree of service availability through automated load balancing

Lower cost of ownership:

·         A cost-optimized, platform and software-independent solution for rack system integration

·         High performance and scalability with Windows Server 2012 R2 operating system and Hyper-V

·         Minimized backup times and fulfilled recovery time objectives for each business critical environment

Technical Benefits

The Microsoft Private Cloud Fast Track Program integrates best-in-class hardware implementations with Microsoft’s software to create a Reference Implementation. This solution has been co-developed by Cisco, EMC, and Microsoft and has gone through a validation process. As a Reference Implementation, Cisco, EMC, and Microsoft have done the work of building a private cloud that is ready to meet a customer’s needs.

Faster deployment:

·         End-to-end architectural and deployment guidance

·         Streamlined infrastructure planning due to predefined capacity

·         Enhanced functionality and automation through deep knowledge of infrastructure

·         Integrated management for virtual machine (VM) and infrastructure deployment

·         Self-service portal for rapid and simplified provisioning of resources

Program Requirements and Validation

The Microsoft Private Cloud Fast Track program is comprised of three pillars: Engineering, Marketing, and Enablement. These three pillars drive the creation of Reference Implementations, making them public and finally making them available for customers to purchase. This Reference Architecture is one step in the “Engineering” phase of the program and towards the validation of a Reference Implementation.

The Microsoft Private Cloud Fast Track program has multiple solutions; it also presents multiple design patterns that its partners can choose from to show the partners best solutions.  Table 1lists the three design patterns that Microsoft offers.

Table 1           Design Pattern Summaries

The Cisco and EMC solution is a converged solution deployed with Fibre Channel over Ethernet and SMB.

Converged Infrastructure is the sharing of network topology between network and storage network traffic. This typically implies Ethernet network devices and network controllers with particular features to provide segregation, quality of service (performance), and scalability. The result is a network fabric with less physical complexity, greater agility and lower costs than those associated with traditional Fiber-based storage networks.

In this topology, many storage designs are supported including traditional SANs, SMB3-enabled SANs, and Windows-based Scale-Out File Servers. The main point in a converged infrastructure is that all storage connectivity is network-based using a single media such as copper. SFP+ adapters are most commonly used.

Key drivers for convergence include cost savings and operational efficiency of a single common Ethernet network vs. multiple physical networks and HBAs for storage traffic. Benefits often include higher utilization levels of datacenter infrastructure with reduced equipment and management costs of the network.

The Cisco and EMC solution is based on Design Pattern 3 – Converged Infrastructure.  In Design Pattern 3, the Hyper-V hosts boot from FCoE LUNs and the fabric management VMs are hosted directly on a unified storage array through SMB.  Additionally, Pattern 3 leverages the minimal number of System Center component servers recommended in order to provide full functionality and high availability in a production environment. This document covers the steps for installing Design Pattern 3. Design Pattern 3 is outlined in Figure 2:  .

A single design pattern is introduced for Fabric Management which includes a dedicated two-to-four node Hyper-V failover cluster to host the fabric management virtual machines. This design pattern utilizes both scaled-out and highly available deployments of the System Center components to provide full functionality in a production environment.

It is recommended that the systems that comprise the Fabric Management layer be physically separated from the rest of the Fabric. Dedicated Fabric Management servers should be used to host those virtual machines which provide management for all of the resources within the cloud infrastructure. This model helps ensure that, regardless of the state of the majority of Fabric resources, management of the infrastructure and its workloads is maintained at all times.

To support this level of availability and separation, IaaS PLA cloud architectures contains a separate set of hosts, running Windows Server 2012 R2 configured as a failover cluster with the Hyper-V role enabled. It should contain a minimum two-node Fabric Management cluster (a three-node cluster is recommended for scale and availability). This Fabric Management cluster is dedicated to the virtual machines running the suite of products that provide IaaS management functionality, and it is not intended to run additional customer workloads over the Fabric infrastructure.

Furthermore, to support Fabric Management operations, these hosts contain high availability virtualized instances (virtual machines) of the management infrastructure (System Center components and their dependencies). However, for some components of the management stack, native high availability is maintained on the application level, for example, a Guest Cluster, built-in availability constructs, or a network load balanced array.

In addition to the System Center components running as virtual machines, Cisco deploys a pair of Cisco Nexus 1000V virtual machines to handle network management for the VMs.

 

Figure 2        Private Cloud Fabric Management Infrastructure

The Fabric Management cluster is configured in such a manner to help ensure maximum availability of all components of the environment.  Each Cisco UCS B200 M4 blade server is configured with sufficient memory to support the running of all the listed virtual machines illustrated above on just two physical servers.  By provisioning a fourth node, the environment retains its highly available capability even during those periods of time when a single host node is taken down for maintenance.  For example, in the above figure, if Node 3 was down for maintenance, a catastrophic failure of Node 2 would not prevent all the virtual machines from continuing to run on Node 1 and Node 4.

The Cisco and EMC architecture is highly modular. Although each customer’s components might vary in its exact configuration, after a Cisco and EMC configuration is built, it can easily be scaled as requirements and demands change. This includes both scaling up (adding additional resources within a Cisco UCS chassis and/or EMC VNX array) and scaling out (adding additional Cisco UCS chassis and/or EMC VNX array).

The Cisco UCS solution validated with Microsoft Private Cloud includes EMC VNX5400 storage, Cisco Nexus 9000 Series network switches, the Cisco Unified Computing Systems (Cisco UCS) platforms, and Microsoft virtualization software in a single package. The computing and storage can fit in one data center rack with networking residing in a separate rack or deployed according to a customer’s data center design. Due to port density, the networking components can accommodate multiple configurations.

Figure 3        Implementation Diagram

Figure 3  contains the following components:

·         One 5108 chassis each with two Cisco UCS 2208XP Fabric Extenders

·         Eight Cisco UCS B200 M4 Blade Servers

-        Dual Intel E5-2660V3 2.20 GHz processors

-        256 GB memory

-        Cisco UCS 1340 Virtual Interface Card

·         Two Cisco UCS 6248UP Fabric Interconnects

·         Two Cisco Nexus 9396PX Switches

·         10 GE and 10 G FCoE connections

·         EMC VNX5400 Unified Platform

-        75 x 300 GB SAS disks

-        Two FCoE SLIC used for booting Hyper-V hosts

-        Two SMB SLIC used for storage of all virtual hard disks used by virtual machines

-        EMC SnapView

·         Two Cisco UCS C220 M4 Rack Servers (optional)

-        Dual Intel E5-2660V3 2.20 GHz processors

-        256 GB memory

-        Cisco UCS 1325 Virtual Interface Card

·         Two Cisco Nexus 2232PP Fabric Extenders (optional)

Storage is provided by an EMC VNX5400 storage array with accompanying disk shelves. All systems and fabric links feature redundancy, providing for end-to-end high availability (HA configuration within a single chassis). For server virtualization, the deployment includes Microsoft Hyper-V. While this is the default base design, each of the components can be scaled flexibly to support the specific business requirements in question. For example, more (or different) blades and chassis could be deployed to increase compute capacity, additional disk shelves or SSDs could be deployed to improve I/O capacity and throughput, or special hardware or software features could be added to introduce new features.

The remainder of this document provides guidance through the low-level steps of deploying the base architecture, as shown in Figure 3 . This includes everything from physical cabling, to compute and storage configuration, to configuring virtualization with Microsoft Windows Server 2012 R2 Hyper-V.

Prerequisite Infrastructure

It is assumed that the necessary infrastructure components exist at the customer site and deploying these services is not in scope for the typical IaaS PLA deployment.  For the purposes of this deployment, four Hyper-V virtual machines running on two Cisco UCS C200 rack servers provide this infrastructure.

·         (2) Active Directory, DNS, and DHCP 

·         (1) NTP services synchronized to an external time source

·         (1) Microsoft’s Windows Server Update Services

Active Directory Domain Services (AD DS)

Active Directory Domain Services (AD DS) is a required foundational component. The IaaS PLA supports customer deployments for AD DS in Windows Server 2012 R2, Windows Server 2012, Windows Server 2008 R2, and Windows Server 2008.  Previous versions of the Windows operating system are not directly supported for all workflow provisioning and deprovisioning automation.  

·         Forests and domains:  The preferred approach is to integrate into an existing AD DS forest and domain, but this is not a hard requirement.  A dedicated resource forest or domain may also be employed as an additional part of the deployment.  System Center does support multiple domains or multiple forests in a trusted environment using two-way forest trusts.

·         Trusts:  System Center allows multi-domain support within a single forest in which two-way forest (Kerberos) trusts exist between all domains.  This is referred to as multi-domain or intra-forest support.

Domain Name System (DNS)

Name resolution is a required element for System Center 2012 R2 components installation and the process automation solution.  Domain Name System (DNS) integrated in AD DS is required for automated provisioning and deprovisioning components. This solution provides full support for deployments running Windows Server 2012 R2, Windows Server 2012, Windows Server 2008 R2, or Windows Server 2008 DNS.

Using non-Microsoft or non-AD DS integrated DNS solutions might be possible, but they would not provide automated creation and removal of DNS records that are related to component installation as well as virtual machine provisioning and deprovisioning processes.  Using solutions outside of AD DS integrated DNS would require manual intervention for these scenarios.  Using non-AD DS integrated DNS is not covered in this CVD.

IP Address Assignment and Management

To support dynamic provisioning and runbook automation, and to manage physical and virtual compute capacity within the IaaS infrastructure, Dynamic Host Configuration Protocol (DHCP) is used by default for all physical computers and virtual machines.  For physical hosts like the Fabric Management cluster nodes and the scale unit cluster nodes, DHCP reservations or fixed IP addresses are recommended so that physical servers and network adapters have known Internet Protocol (IP) addresses.  DHCP provides centralized management of these addresses.

Virtual Machine Manager (VMM) can provide address management both for physical computers (Hyper-V Host Servers and Scale-Out File Servers) and for virtual machines.  These IP addresses are assigned statically from IP Address Pools managed by Virtual Machine Manager.  This approach is recommended as an alternative to DHCP and also provides centralized management.

If a particular subnet or IP Address range is maintained by Virtual Machine Manager, it should not be served by DHCP. However, other subnets, for example, used by physical servers, which are not managed by Virtual Machine Manager, can still leverage DHCP.

Regardless of the IP address assignment mechanism chosen (DHCP, Virtual Machine Manager, or both), Windows Server IP Address Management (IPAM) feature can be leveraged to track in-use IP addresses for reporting and advanced automation. Optionally, both DHCP and Virtual Machine Manager features can be integrated with IPAM.  Using IPAM within Windows Server is outside the scope of this document.

Software Revisions

It is important to note the software versions used in this document. Table 2 details the software revisions used throughout this document.

Table 2           Software Revisions

PowerShell Deployment Toolkit (PDT)

Microsoft’s PowerShell Deployment Toolkit (https://gallery.technet.microsoft.com/PowerShell-Deployment-f20bb605 ) contains a script (Downloader.ps1) that can download and extract almost all the Microsoft software requirements.  Though it can be used to deploy the complete System Center environment, that exercise is beyond the scope of this document.  However, the Downloader script is an easy way to make sure you obtain the latest versions in as quick a way as possible.  Download PDT and extract all files into a single directory.

The Downloader script depends upon two files, Workflow.xml and Variable.xml.  DO NOT change any content in Workflow.xml.  It is the ‘brains’ of the process and changes to this file will likely cause scripts to fail.  Make changes only to the Variable.xml file.  For Downloader.ps1 to download all the files, you need to change two lines in the Variable.xml file.  Change the SourcePath and Download variables to reflect the location to which you wish to download the files.  It is recommended to have both these variables point to the same location.

Depending upon your network connection this download and extraction can take two or more hours.  It downloads about 6 GB of files and expands this to about 14 GB.  It does not download the Windows Server 2012 R2 and System Center 2012 R2 media.  You need to download those on your own.

The PDT scripts are not signed.  Depending upon the PowerShell execution policy in place, you might need to change it to unrestricted.  It can be reset after running the Downloader.ps1 script.

When the Downloader.ps1 script has successfully run to download and extract the files, the download subdirectory can be deleted.

Configuration Guidelines

This document provides details for configuring a fully redundant, highly-available configuration. References are made as to which component is being configured with each step whether that be A or B.  For example, Storage Processor A (SP A) and Storage Processor B (SP B) are used to identify the two EMC storage controllers that are provisioned with this document while Cisco Nexus  A and Nexus B identify the pair of Cisco Nexus switches that are configured. The Cisco UCS fabric interconnects are configured likewise. Additionally, this document details steps for provisioning multiple UCS hosts and these are identified sequentially, VMHost-Mgmt01, VMHost-Mgmt02, and so on.  Finally, when indicating that the reader should include information pertinent to their environment in a given step, this is indicated with the inclusion of <italicized text> as part of the command structure.  See the example below for the vlan create command:

The Cisco UCS PowerTool allows configuration and modification of the Cisco UCS environment by using Microsoft PowerShell.  The same conventions for entering parameters shown above are followed for entering commands, parameters, and variables within PowerShell.  One thing to note with Cisco UCS PowerTool is that many of its parameters are case sensitive, whereas parameters in PowerShell are not case sensitive.   For example, a parameter value of ‘enabled’ in PowerShell can be represented as either ‘enabled’ or ‘Enabled’ (without the single quotes).  With the Cisco UCS PowerTool cmdlets, ‘enabled’ is different from ‘Enabled’.

This document is intended to allow the reader to fully configure the customer environment. In order to do so, there are various steps which will require you to insert your own naming conventions, IP address and VLAN schemes as well as record appropriate WWPN, WWNN, or MAC addresses. The following table details the list of VLANs necessary for deployment as outlined in this guide. Note that in this document the SC-access VLAN and subnet is used for Fabric Management virtual machine access.  The Mgmt VLAN and subnet is used for management interfaces of the Hyper-V hosts.  A Layer-3 route must exist between the Mgmt and SC-access subnets.

Table 3           VLAN Names and IDs Used in this Document

Configuration Workstation

It is recommended to have a Windows 8.1 or Windows Server 2012 R2 workstation configured with certain pre-requisite software and joined to the same domain as the Hyper-V servers will be joined.  Using a properly configured workstation makes the job of installing the solution easier.  Here is the recommendation for software to be installed on the workstation.

Windows 8.1 Workstation

·         Install .NET Framework 3.5 by issuing the following command from an elevated command prompt: Enable-WindowsOptionalFeature –Online –FeatureName NetFx3 –Source D:\sources\sxs.  This assumes the drive D: is the location of your Windows distribution media.

·         Install the Remote Server Administration Tools for Windows 8.1.  This is found at http://www.microsoft.com/en-us/download/details.aspx?id=39296.  This is available in both a 32-bit and 64-bit distribution.  Make sure you select the copy to match your Windows 8.1 installation.

·         After installing the Remote Server Administration Tools, install specific management tools.

-        Hyper-V Management Tools – issue the following command from an elevated command prompt: Get-WindowsOptionalFeature -Online -FeatureName *hyper-v*all | Enable-WindowsOptionalFeature -Online

-        Failover Clustering Tools – issue the following command from an elevated command prompt:  Enable-WindowsOptionalFeature -Online -FeatureName RemoteServerAdministrationTools-Features-Clustering

Windows Server 2012 R2 Workstation

·         Install .NET Framework 3.5 by issuing the following command from an elevated command prompt: Install-WindowsFeature –Name NET-Framework-Core –Source D:\sources\sxs.  This assumes the drive D: is the location of your Windows distribution media.

·         Install the Hyper-V Management Tools by issuing this PowerShell cmdlet: Install-WindowsFeature –Name RSAT-Hyper-V-Tools

·         Install the Windows Failover Clustering Tools by issuing this PowerShell cmdlet: Install-WindowsFeature –Name RSAT-Clustering

Both Workstations

·         Cisco UCS PowerTool for UCSM, version 1.3.1. 

·         Cisco Nexus 1000V PowerShell, version 1.0. 

·         Naviseccli – Navisphere Secure Command Line Interface

·         ESI (EMC Storage Integrator) – EMC PowerShell library (only installs on Server instances)

·         Java 7 – required for running UCS Manager.  Installed from the web.

·         PuTTy – an SSH and Telnet client helpful in initial configuration of the Cisco UCS 6248UP Fabric Interconnects. This program just needs to be copied to the system.

·         PL-2303 USB-to-Serial driver – used to connect to the Cisco UCS 6248UP Fabric Interconnects through a serial cable connected to a USB port on the workstation.  The download is a .zip file.  Extract the executable from the .zip file and load it on the system.

·         If using the PDT to download installation files, you will need to install the following.

-        One of the following extraction packages

ù  7-Zip

ù  PeaZip

ù  WinRAR

-        Web Platform Installer 5.0

You can download all the software listed in the revision table to this workstation.  Some of the software, such as distribution media, can be placed into a file share for access by other systems.

Cisco PowerShell Modules

Before beginning the installation of Cisco PowerTool and the Cisco Nexus 1000V PowerShell module, make sure the following:

·         There are no open windows running Windows PowerShell

·         Uninstall all versions of Cisco UCS PowerTool that are older than Cisco UCS PowerTool, Release 0.9.1.0.

·         You have downloaded the latest versions of software

-        Cisco PowerTool version 1.3.1

-        Cisco Nexus 1000V version 1.5(2a) contains PowerShell module

Cisco UCS PowerTool

Cisco Nexus 1000V PowerShell Module

Cisco provides a PowerShell module containing cmdlets to invoke the REST APIs on the Cisco Nexus 1000V.  This can be downloaded from https://developer.cisco.com/fileMedia/download/8bf948fb-83a5-4c9e-af5c-4faac735c8d3.

Download the .zip file and expand it.  Copy the Cisco-Nexus1000V.psm1 file to the system from which the PowerShell cmdlets will be used.  Before using the cmdlets, the module must be imported.

Before issuing any other Cisco Nexus 1000V cmdlets, you need to establish a link to the virtual supervisory module with the following cmdlet.

A credentials window will open allowing for the entry of the credentials to connect to the VSM.  This cmdlet will create two global variables which must not be overridden during the PowerShell session - $VSM_IP and $Credential.

To see all cmdlets available to be used, issue the following cmdlet.

EMC ESI PowerShell

ESI components, including the PowerShell module, can only be installed on Windows Server instances.

Deployment

This document details the necessary steps to deploy base infrastructure components as well as provisioning Microsoft Private Cloud as the foundation for virtualized workloads. At the end of these deployment steps, you will be prepared to provision your applications on top of a Microsoft Private Cloud virtualized infrastructure. The outlined procedure includes:

·         Initial EMC VNX array configuration

·         Initial Cisco UCS configuration

·         Initial Cisco Nexus configuration

·         Creation of necessary VLANs for management, basic functionality, and specific to the Microsoft virtualized infrastructure

·         Creation of necessary vPCs to provide HA among devices

·         Creation of necessary service profile pools: WWPN, world-wide node name (WWNN), MAC, server, and so forth

·         Creation of necessary service profile policies: adapter, boot, and so forth

·         Creation of four service profile templates from the created pools and policies: one each for fabric A and B

·         Provisioning of four servers from the created service profiles in preparation for OS installation

·         Initial configuration of the infrastructure components residing on the EMC Controller

·         Deployment of a Microsoft Hyper-V failover cluster to handle Fabric Management roles

·         Deployment of Microsoft System Center

-        Deployment of the Cisco Plug-ins

-        Deployment of the EMC Plug-ins

·         Deployment of Microsoft Windows Azure Pack

·         Deployment of a Microsoft Hyper-V failover cluster to handle tenant workloads

The Microsoft Private Cloud Solution validated with the Cisco and EMC architecture is flexible; therefore, the exact configuration detailed in this section might vary for customer implementations depending on specific requirements. Although customer implementations might deviate from the information that follows, the best practices, features, and configurations listed in this section should still be used as a reference for building a customized Cisco and EMC with Microsoft Private Cloud solution.

Cabling Information

The following information is provided as a reference for cabling the physical equipment in a Cisco and EMC environment. The tables include both local and remote device and port locations in order to simplify cabling requirements.

The tables in this section contain details for the prescribed and supported configuration of the EMC VNX5400.

This document assumes that out-of-band management ports are plugged into an existing management infrastructure at the deployment site.

Be sure to follow the cable directions in this section. Failure to do so will result in necessary changes to the deployment procedures that follow because specific port locations are mentioned.

It is possible to order an EMC VNX5400 system in a different configuration from what is described in the tables in this section. Before starting, be sure the configuration matches what is described in the tables and diagrams in this section.  For purposes of this document, the two-port cards are located as follows:

 

Table 4           I/O Card Locations

Table 5           Cisco Nexus 9396 A Cabling Information

Table 6           Cisco Nexus 9396 B Cabling Information

Table 7           Cisco 6248 Fabric Interconnect A Cabling Information

Table 8           Cisco 6248 Fabric Interconnect B Cabling Information

The following section provides a detailed procedure for configuring the Cisco Nexus 9396 switches for use in a Cisco and EMC with Microsoft Private Cloud environment. Follow these steps precisely; failure to do so could result in an improper configuration.

Before you begin, identify the following information in Table 9 . 

Table 9           Cisco Nexus  Management Information

Set up Initial Cisco Nexus 9396 Switch

These steps provide details for the initial Cisco Nexus 9396 Switch setup.

Cisco Nexus 9396 A

On initial boot and connection to the serial or console port of the switch, the NX-OS setup should automatically start.

Cisco Nexus 9396 B

On initial boot and connection to the serial or console port of the switch, the NX-OS setup should automatically start.

Enable Cisco Nexus Features and Set Global Configurations

These steps provide details for enabling Cisco Nexus features and setting global configurations.

Cisco Nexus A and Nexus B

1.    Enter global configuration mode

2.    Enable necessary features

3.    Configure spanning tree defaults

4.    Configure timezone

5.    Save the running configuration to the start-up configuration

Create Necessary VLANs

These steps provide details for creating the necessary VLANs.

Cisco Nexus A and Nexus B

1.    Enter global configuration mode

2.    Define required VLANs

3.    Save the running configuration to the start-up configuration

4.    Validate the entry

Validate the results of the entered commands by using the show vlan command.

Add Individual Port Descriptions for Troubleshooting

These steps provide details for adding individual port descriptions for troubleshooting activity and verification.

Cisco Nexus 9396 A

1.    Place descriptions on the ports in use

2.    Copy the running configuration to the start-up configuration

3.    Validate the entry

Validate the results of the entered commands by using the show interface description command.

Cisco Nexus 9396 B

1.    Place descriptions on the ports in use

2.    Copy the running configuration to the start-up configuration

3.    Validate the entry

Validate the results of the entered commands by using the show interface description command.

 

Create Necessary Port Channels

These steps provide details for creating the necessary PortChannels between devices.

Cisco Nexus 9396 A

1.    Create peer link and add ports

2.    Create port channel link to UCS Fabric A

3.    Create port channel link to UCS Fabric B

4.    Copy the running configuration to the start-up configuration

5.    Validate the entry

Validate the results of the entered commands by using the show port-channel summary command.

Cisco Nexus 9396 B

1.    Create peer link and add ports

2.    Create port channel link to UCS Fabric B

3.    Create port channel link to UCS Fabric A

4.    Copy the running configuration to the start-up configuration

5.    Validate what was just entered

Validate the results of the entered commands by using the show port-channel summary command.

Add PortChannel Configurations

These steps provide details for adding PortChannel configurations.

Cisco Nexus 9396 A and B

1.    Make the peer link a switchport, configure the allowed VLANs, and bring it up

2.    Make the first port channel a switchport, configure the allowed VLANs, and bring it up

3.    Make the second port channel a switchport, configure the allowed VLANs, and bring it up

4.    Set MTU to 9216 to support jumbo frames

5.    Copy the running configuration to the start-up configuration

6.    Validate the entry

Validate the results of the entered commands by using the show running-config port-channel command.

Configure Virtual Port Channel Domain

These steps provide details for configuring virtual PortChannels (vPCs)

Cisco Nexus 9396 A

1.    Define the vPC domain and define keep alive with Cisco Nexus  9396 B

2.    Define Po10 as the peer link

3.    Define Po11 as vPC 11

4.    Define Po12 as vPC 12

5.    Copy the running configuration to the start-up configuration

6.    Validate the entry

Validate the results of the entered commands by using the show vpc brief command.

Cisco Nexus 9396 B

1.    Define the vPC domain and define keep alive with Nexus 9396 A

2.    Define Po10 as the peer link

3.    Define Po11 as vPC 11

4.    Define Po12 as vPC 12

5.    Copy the running configuration to the start-up configuration

6.    Validate the entry

Validate the results of the entered commands by using the show vpc brief command.

Enable SMB Links

Cisco Nexus 9396 A and B

1.    Activate ports to be used for SMB communication

2.    Allow SMB VLAN on the interface

Link into Existing Network Infrastructure

Depending on the available network infrastructure, several methods and features can be used to uplink the private cloud environment. If an existing Cisco Nexus environment is present, Cisco recommends using vPCs to uplink the Cisco Nexus 9396 switches included in the private cloud environment into the infrastructure. The previously described procedures can be used to create an uplink vPC to the existing environment.

Configure HSRP (Optional)

These steps provide examples of how to configure Hot Standby Router Protocol (HSRP).  These steps need to be repeated for each IP subnet to be routed within the Nexus switches.  If routing is provided outside the Cisco Nexus  9396, these steps are not needed.

Cisco Nexus 9396 A

1.    Select VLAN to enable

2.    Specify IPv4 address of interface

3.    Specify unique HSRP group number and virtual IPv4 address

4.    Enable DHCP relay by specifying configured DHCP server IPv4 address

5.    Repeat for all VLANs, assigning different HSRP group to each VLAN

6.    Copy running configuration to start-up configuration

7.    Validate the entry

Validate the results of the entered commands by using the show hsrp brief command.

Cisco Nexus 9396 B

1.    Select VLAN to enable

2.    Specify IPv4 address of interface

3.    Specify HSRP group and virtual IPv4 address, matching group number to share virtual IPv4 address

4.    Enable DHCP relay by specifying configured DHCP server IPv4 address

5.    Repeat for all VLANs, assigning different group to each VLAN

6.    Copy running configuration to start-up configuration

7.    Validate the entry

Validate the results of the entered commands by using the show hsrp brief command.

Initial configuration and implementation of an EMC VNX5400 is covered in detail from the EMC documentation library. This is accessible at https://mydocs.emc.com/VNX/ and select Install VNX, using the VNX5400 series as the installation type. Installation documentation covers all areas from unpacking VNX storage components, installing in rack, provisioning power requirements, and physical cabling.

When physically installed, the VNX should include the Disk Processing Enclosure (DPE) and two additional Disk Array Enclosures (DAEs), cabled as shown in Figure 4.

Figure 4        Cabling Diagram for VNX5400 with 2 DAE

To complete software setup of the VNX array, it will be necessary to configure system connectivity including the creation of an Administrative user for the VNX array.  The following worksheets (also found in the Installation documentation) list all required information, and can be used to facilitate the initial installation.

VNX Worksheets

With your network administrator, determine the IP addresses and network parameters you plan to use with the storage system, and record the information on the following worksheet. You must have this information to set up and initialize the system. The VNX5400 array is managed through a dedicated LAN port on the Control Station and each storage processor. These ports must share a subnet with the host you use to initialize the system. After initialization, any host on the same network and with a supported browser can manage the system through the management ports. This information can be recorded in Table 10 .

Table 10        IPV4 Management Port Information

 

While it is possible to implement Ipv6 settings for the VNX array, the Fast Track implementation does not require it, and it is not implemented. 

It is possible to more fully configure management IP addresses for the VNX5400 array.  Table 11 lists some of the addresses you can optionally configure.

Table 11        Optional Control Station LAN Settings

An administrative user account is required to be set for the array, and this account can be later utilized for executing NaviSecCLI commands, ass well as for the ESI PowerShell environment used to provision LUNs from storage pools, and map those LUNs to hosts. Information required is outlined in the following table.

Table 12        Login Information for the Storage System Administrator

It is also necessary at this time to install the NaviSecCLI command line interface from a supported Windows client environment. The client should have network access to the VNX5400 array for both HTTP/HTTPS access and for remote NaviSecCLI command execution.

Installation media for the NaviSecCLI utility, as well as ESI, are available by download at http://support.emc.com. The current version of the media should always be utilized. Installation of the utility is implemented through the typical application installation process for Windows-based systems.

World-wide Node Name (WWNN), World-wide Port Names (WWPN), and SP-Ports will be needed for various configuration tasks.  They can be easily retrieved from the Unisphere console.  There is a single WWNN for the VNX5400 array, but each fibre channel port used for access has its own unique WWPN.  Use the following table to record the WWNN, WWPNs, and SP-Ports for your environment.

Table 13        Optional Control Station LAN Settings

This information can be obtained through Unisphere, NaviSecCli, or ESI.

Unisphere

NaviSecCli

Below is an example for obtaining the WWPNs from the connections to the VNX5400. It may be necessary to provide additional parameters, for login, password and scope options. The example below returns configuration information for all ports configured within the array.  This includes both Fiber Channel ports, and iSCSI targets. The WWPN for any given Fiber Channel port is derived from the last half of the SP UID entry. The first half of the SP UID is the WWNN entry. As an example, the WWPN of Port 0 on SP-A Port ID 1 is 50:06:01:61:08:60:26:1B.

ESI PowerShell

When you have successfully connected to the storage system, the following commands will display the WWNN and WWPN for the ports on the selected module.  This example is showing module 3 on both SPA and SPB.

Installing EMC Storage Integrator v3.1

To use ESI features as documented with the Fast Track architecture, perform the following installation procedure. 

 

 

Register the VNX for Use with the ESI PowerShell Toolkit

Storage Configuration

The VNX5400 storage array provides two methods for configuring storage: RAID Groups and Storage Pools. Two RAID groups and a single storage pool are utilized in the Private Cloud configuration. LUNs accessed through FCoE are subsequently created within the RAID groups and storage pool to satisfy the requirements of the Management Infrastructure, the Virtual Machines, and the applications and services which run within the environment.  The following list shows how the disks are allocated.

·         RAID Group (RAID1) – LUN Cloning is used for rapid deployment of the Operating System disks for the physical servers.  Cloning requires that the LUN being cloned resides within a RAID group.  A single RAID1 group comprised of two disks is created to hold the master boot LUN that will be used for cloning.

·         RAID Group (RAID5) – Various LUNs for Fabric Management

-        Individual LUNs for each Hyper-V host’s boot volume

-        Cluster Shared Volume LUN

ù  VHDX for SQL Server Analysis Services database and log files

ù  VHDX for Virtual Machine Manager’s highly available library share

-        Disk Witness LUN for Fabric Management Cluster

-        Disk Witness LUN for Tenant Cluster

·         Storage pool for SMB storage

-        Share for virtual machine virtual hard disk and configuration storage

-        Share for SQL Server data and log file storage

The two RAID groups and the storage pool are created from the Unisphere GUI. 

Create RAID Groups

Create Storage Pool

Virtual machine hard disks and SQL Server database and log files are stored on SMB shares provided by the Data Movers on the VNX5400 storage array.  The storage requirement for the storage of the virtual machines is about 2.5 TB.  The storage requirement for the storage of the SQL Server data is about 3.5 TB.  Therefore, it is necessary to create a storage pool of at least 6 TB in size. 

A single storage pool is used for both the Fabric Management and tenant storage requirements.  You may want to configure a larger storage pool to accommodate the requirements of both environments at this.  However, the size of the pool can be expanded at any time in the future.

Create VNX LUNs

This solution requires the creation of several LUNs to be used for different purposes.  These LUNs will be configured from the RAID Groups and Storage Pool.

·         Master boot LUN from which other LUNs are cloned (50 GB) (RAID1)

·         Disk Witness LUN for Fabric Management Cluster (1 GB) (RAID5)

·         Disk Witness LUN for Tenant Cluster (1 GB) (RAID5)

·         Cluster Shared Volume is used for two specific purposes in this solution (RAID5)

-        Storage of the SQL Server Analysis Services database and log files (20 GB)

-        Highly available VMM library (300 GB)

·         SMB File System (at least 6 TB) (Storage Pool)

The private cloud environment implements a boot from SAN environment, using the concept of a Master Boot LUN. The Master Boot LUN is a storage area that will be used to maintain an image of a Windows Server 2012 R2 image to be used as a Clone source. This image should be configured as a base image to be used for subsequent installations, so all patching and custom configuration steps should be taken.  For example, maybe a desired configuration setting is to make sure that all physical servers are able to be remotely managed.  When the image is configured according to customer policy, the Microsoft sysprep utility can be run against this image to prepare it for use as a Clone.  Make sure any Microsoft hotfixes listed in the software revision table have been applied before running sysprep.

Clones created from the Master Boot LUN will be presented to the physical servers defined by Service Profiles in the Cisco UCS environment.  This style of deployment allows Service Profiles to be fully transportable between different physical blades as the boot device is external to the chassis, and also allows for multiple Master Boot images to be implemented providing support for different operating system versions or configurations which may need to be implemented over time.

Management of the boot LUN requires special consideration and needs to make sure that the LUN ID provided to the LUN as seen from the host is set to 0 (zero). The ESI (EMC Storage Integrator) PowerShell commands do not allow the manipulation of the LUN ID for devices presented to servers, and simply default to the sequential allocation of LUN IDs as implemented by the VNX array.  As a result of this behavior, the boot LUN must be the first device that is mapped to the server (Cisco UCS service profile). If this is incorrectly implemented, then the wrong target will be selected for Windows boot operations on server power-up.

Unisphere

The following steps show how to create LUNs with Unisphere. 

ESI PowerShell

As described, the ESI PowerShell commands are utilized for provisioning of the LUNs required within the environment, and assume that the storage pool creation outlined in the previous section have been completed. For this procedure, a single LUN is created, and is used to install a Windows Server 2012 R2 instance. This server instance subsequently will be processed with Windows sysprep, and be removed from the server.  All compute nodes will then use a clone of the sysprep image, and will be customized as individual server instances.

Creation of all necessary LUNs within the Private Cloud environment can be executed with the PowerShell script ProcessStorageRequests.ps1 provided Appendix B.  The defined XML configuration file is read by the PowerShell script. This XML configuration file contains five parameters. There are two classes that can be repeated multiple times.  The XML class <luns> can be repeated multiple times to define multiple LUNs for a server. The <Server> class can be repeated to create multiple server records.

For the purpose of defining and creating the Master Boot LUN, it is recommended to create a unique XML configuration file that defines only this specific device.  Later the format of the XML configuration file can be followed for creating multiple LUNs.

·         <label> - the name that will be assigned to the LUN that is created

·         <pool> - the storage pool from which the LUN will be created

·         <size> - the size of the LUN (in GB) to be created

·         <ServerName> - the name of the server that will be assigned the LUN that must match the Service Profile name in UCS Manager, including case.  This name is also used for management purposes on the VNX array

·         <IPAddress> - the management IP address of the server

In addition to the five parameters listed above that can be repeated, there are two other parameters that are defined only once.  The <Array> parameter is the name of the VNX array. The <UCSAddress> parameter is the IP address for accessing the UCS management console. An example of the contents of a configuration is shown below for a configuration file called “CFG_STORAGE_LUNS.xml”.

This configuration file is read by the following sample PowerShell script to result in a LUN named MasterBoot-2012R2 of size 60 GB being created in the storage pool called Boot-LUNs.

The execution of such a process is shown in the following figure.

Figure 5        Example Execution of Master Boot LUN Creation

Configure SMB on EMC VNX5400

Configure DNS and NTP

The SMB file servers need to be joined to the Active Directory domain.  DNS needs to be configured for the servers to find the Active Directory domain. In order to join the domain, the VNX must have a time that is within 5 minutes of the domain’s time (Kerberos requirement).  Configuring NTP to the same time service to which the domain is synchronized helps ensure time synchronicity. The following steps illustrate how to configure DNS and NTP for the data movers.

Configure SMB Interfaces

The following steps describe how to configure the network interfaces to provide access to the SMB data movers.

 

Configure CIFS Network Service

Configure Storage

 

Join Active Directory Domain

Create SMB File System and Mounts

 

Associate Shares with CIFS Server

 

The following section provides a detailed procedure for configuring the Cisco Unified Computing System for use in a private cloud environment. These steps should be followed precisely because a failure to do so could result in an improper configuration.

Before you begin, identify the following information:

Table 14        Cisco UCS Manager Configuration Information

Perform Initial Setup of the Cisco UCS 6248 Fabric Interconnects

These steps provide details for initial setup of the Cisco UCS 6248 fabric Interconnects.

Cisco UCS 6248 A

1.    Connect to the console port on the first Cisco UCS 6248 fabric interconnect.

2.    At the prompt to enter the configuration method, enter console to continue.

3.    If asked to either do a new setup or restore from backup, enter setup to continue.

4.    Enter y to continue to set up a new fabric interconnect.

5.    Enter y to enforce strong passwords.

6.    Enter the password for the admin user.

7.    Enter the same password again to confirm the password for the admin user.

8.    When asked if this fabric interconnect is part of a cluster, answer y to continue.

9.    Enter A for the switch fabric.

10.  Enter the <cluster name> for the system name.

11.  Enter the <Mgmt0 IPv4 address> address.

12.  Enter the <Mgmt0 IPv4> netmask.

13.  Enter the <default gateway Ipv4 address> of the default gateway.

14.  Enter the <cluster IPv4 address>.

15.  To configure DNS, answer y.

16.  Enter the <DNS IPv4 address>.

17.  Answer y to set up the default domain name.

18.  Enter the <default domain name>.

19.  Review the settings that were printed to the console, and if they are correct, answer yes to save the configuration.

20.  Wait for the login prompt to make sure the configuration has been saved.

Cisco UCS 6248 B

1.    Connect to the console port on the second Cisco UCS 6248 fabric interconnect.

2.    When prompted to enter the configuration method, enter console to continue.

3.    The installer detects the presence of the partner fabric interconnect and adds this fabric interconnect to the cluster. Enter y to continue the installation.

4.    Enter the admin password for the first fabric interconnect.

5.    Enter the <Mgmt0 IPv4 address>.

6.    Answer yes to save the configuration.

7.    Wait for the login prompt to confirm that the configuration has been saved.

Log into Cisco UCS Manager

These steps provide details for logging into the Cisco UCS environment.

Add a Block of IP Addresses for KVM Access

These steps provide details for creating a block of KVM IP addresses for server access in the Cisco UCS environment.

Cisco UCS Manager

Cisco UCS PowerTool

Synchronize Cisco UCS to NTP

These steps provide details for synchronizing the Cisco UCS environment to the NTP server.

Cisco UCS Manager

Cisco UCS PowerTool

Edit the Chassis Discovery Policy

These steps provide details for modifying the chassis discovery policy as the base architecture includes two uplinks from each fabric extender installed in the Cisco UCS chassis.

Cisco UCS Manager

Cisco UCS PowerTool

Enable Server and Uplink Ports

These steps provide details for enabling server, uplinks port channels, and FCoE links.

Cisco UCS Manager

Cisco UCS PowerTool

Cisco UCS PowerTool can work on both fabrics when setting up server and uplink ports.

 

Configure Fibre Channel Switching Mode

The Fibre Channel switching mode determines how the fabric interconnect behaves as a switching device between the servers and storage devices. The fabric interconnect operates in either of the following Fibre Channel switching modes:

·         End-Host Mode – End-host mode allows the fabric interconnect to act as an end host to the connected fibre channel networks, representing all server (hosts) connected to it through vHBAs.  End-host mode is synonymous with NPV mode. This is the default Fibre Channel Switching mode.

·         Switch Mode – Switch mode is the traditional Fibre Channel switching mode. Switch mode allows the fabric interconnect to connect directly to a storage device.

This solution connects directly from the Cisco UCS fabric interconnects to the EMC VNX5400 storage array.

Cisco UCS Manager

Cisco UCS PowerTool

Enable FCoE Storage Ports

The following steps demonstrate the configuration of the links between the fabric interconnect and the storage array.

Cisco UCS Manager

Cisco UCS PowerTool

Acknowledge the Cisco UCS Chassis

The connected chassis needs to be acknowledged before it can be managed by Cisco UCS Manager.

Cisco UCS Manager

Cisco UCS Manager acknowledges the chassis and the blades servers in it.  Do this for each chassis in your configuration.

Cisco UCS PowerTool

Create Uplink PortChannels to the Cisco Nexus 9396 Switches

These steps provide details for configuring the necessary PortChannels out of the Cisco UCS environment.

Cisco UCS Manager

 

 

Cisco UCS PowerTool

Configure Policies, Pools, and Templates

Every server that is provisioned in the Cisco Unified Computing System is specified by a service profile. A service profile is a software definition of a server and its LAN and SAN network connectivity In other words, a service profile defines a single server and its storage and networking characteristics. Service profiles are stored in the Cisco UCS 6200 Series Fabric Interconnects. When a service profile is deployed to a server, UCS Manager automatically configures the server, adapters, fabric extenders, and fabric interconnects to match the configuration specified in the service profile. This automation of device configuration reduces the number of manual steps required to configure servers, network interface cards (NICs), host bus adapters (HBAs), and LAN and SAN switches.

A service profile typically includes four types of information:

·         Server definition: It defines the resources (for example, a specific server or a blade inserted to a specific chassis) that are required to apply to the profile.

·         Identity information: Identity information includes the UUID, MAC address for each virtual NIC (vNIC), and WWN specifications for each HBA.

·         Firmware revision specifications: These are used when a certain tested firmware revision is required to be installed or for some other reason a specific firmware is used.

·         Connectivity definition: It is used to configure network adapters, fabric extenders, and parent interconnects, however this information is abstract as it does not include the details of how each network component is configured.

A service profile is created by the Cisco UCS server administrator. This service profile leverages configuration policies that were created by the server, network, and storage administrators. Server administrators can also create a Service profile template which can be later used to create Service profiles in an easier way. A service template can be derived from a service profile, with server and I/O interface identity information abstracted. Instead of specifying exact UUID, MAC address, and WWN values, a service template specifies where to get these values. For example, a service profile template might specify the standard network connectivity for a web server and the pool from which its interface's MAC addresses can be obtained. Service profile templates can be used to provision many servers with the same simplicity as creating a single one.

For a guide on configuring server-related policies, see:

http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide/1-0-2/b_GUI_Config_Guide/GUI_Config_Guide_chapter23.html.

Create an Organization (Optional)

These steps provide details for configuring an organization in the Cisco UCS environment. Organizations are used as a means to organize and restrict access to various groups within the IT organization, thereby enabling multi-tenancy of the compute resources.

This document assumes the use of an organization.  When the organization is implemented, you must remember the search order UCS employs when searching for components.  For example, when creating a template in a sub-organization, UCS will search first in the sub-organization to resolve a reference.  If it does not find the reference there, it will search up through its parent tree, ending at root. A sub-organization cannot resolve a reference to an item that exists in a different peer sub-organization or child sub-organization.

Cisco UCS Manager

Cisco UCS PowerTool

Create a MAC Address Pool

A MAC pool is a collection of network identities, or MAC addresses, that are unique in their Layer 2 environment and are available to be assigned to vNICs on a server. If you use MAC pools in service profiles, you do not have to manually configure the MAC addresses to be used by the server associated with the service profile.

In a system that implements multi-tenancy, you can use the organizational hierarchy to help ensure that MAC pools can only be used by specific applications or business services. Cisco UCS Manager uses the name resolution policy to assign MAC addresses from the pool.

To assign a MAC address to a server, you must include the MAC pool in a vNIC policy. The vNIC policy is then included in the service profile assigned to that server.

You can specify your own MAC addresses or use a group of MAC addresses provided by Cisco.

These steps provide details for configuring the necessary MAC address pool for the Cisco UCS environment.

Cisco UCS Manager

Cisco UCS PowerTool

Create WWN Pools

A WWN pool is a collection of World Wide Names for use by the Fibre Channel vHBAs in a Cisco UCS instance. You create separate pools for the following:

·         WW node names assigned to the server

·         WW port names assigned to the vHBA

If you use WWN pools in service profiles, you do not have to manually configure the WWNs that will be used by the server associated with the service profile. In a system that implements multi-tenancy, you can use a WWN pool to control the WWNs used by each organization.

You assign WWNs to pools in blocks. For each block or individual WWN, you can assign a boot target.

·         WWNN Pools – a WWNN pool is a WWN pool which contains only WW node names. If you include a pool of WWNNs in a service profile, the associated server is assigned a WWNN from that pool.

·         WWPN Pools – a WWPN pool is a WWN pool which contains only WW port names. If you include a pool of WWPNs in a service profile, the port on each vHBA of the associated server is assigned a WWPN from that pool.

Create WWNN Pools

These steps provide details for configuring the necessary WWNN pools for the Cisco UCS environment.

Cisco UCS Manager

Cisco UCS PowerTool

Create WWPN Pools

These steps provide details for configuring the necessary WWPN pools for the Cisco UCS environment.  Two WWPN pools are created, one for fabric A and one for Fabric B.

Cisco UCS Manager

Cisco UCS PowerTool

Create UUID Suffix Pools

A UUID suffix pool is a collection of SMBIOS UUIDs that are available to be assigned to servers. The first number of digits that constitute the prefix of the UUID are fixed. The remaining digits, the UUID suffix, is variable. A UUID suffix pool helps ensure that these variable values are unique for each server associated with a service profile which uses that particular pool to avoid conflicts.

If you use UUID suffix pools in service profiles, you do not have to manually configure the UUID of the server associated with the service profile.

These steps provide details for configuring the necessary UUID suffix pools for the Cisco UCS environment.

Cisco UCS Manager

 

Cisco UCS PowerTool

Create Server Pools

A server pool contains a set of servers. These servers typically share the same characteristics. Those characteristics can be their location in the chassis, or an attribute such as server type, amount of memory, local storage, type of CPU, or local drive configuration. You can manually assign a server to a server pool, or use server pool policies and server pool policy qualifications to automate the assignment.

If your system implements multi-tenancy through organizations, you can designate one or more server pools to be used by a specific organization. For example, a pool that includes all servers with two CPUs could be assigned to the Marketing organization, while all servers with 64 GB memory could be assigned to the Finance organization.

A server pool can include servers from any chassis in the system. A given server can belong to multiple server pools.

These steps provide details for configuring the necessary server pools for the Cisco UCS environment.

Cisco UCS Manager

 

Cisco UCS PowerTool

Create VLANs

A named VLAN creates a connection to a specific external LAN. The VLAN isolates traffic to that external LAN, including broadcast traffic.

The name that you assign to a VLAN ID adds a layer of abstraction that allows you to globally update all servers associated with service profiles that use the named VLAN. You do not need to reconfigure the servers individually to maintain communication with the external LAN.

You can create more than one named VLAN with the same VLAN ID. For example, if servers that host business services for HR and Finance need to access the same external LAN, you can create VLANs named HR and Finance with the same VLAN ID. Then, if the network is reconfigured and Finance is assigned to a different LAN, you only have to change the VLAN ID for the named VLAN for Finance.

In a cluster configuration, you can configure a named VLAN to be accessible only to one fabric interconnect or to both fabric interconnects.

These steps provide details for configuring the necessary VLANs for the Cisco UCS environment.

Cisco UCS Manager

Cisco UCS PowerTool

Create Host Firmware Package Policy

Firmware management policies allow the administrator to select the corresponding packages for a given server configuration. These often include adapter, BIOS, board controller, FC adapters, HBA option ROM, and storage controller properties.  For recommended best practices on configuring firmware policies see http://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-manager/110511-ucs-fw-mgmt-00.html.

These steps provide details for creating a firmware management policy for a given server configuration in the Cisco UCS environment. These steps show a very small sample as what is actually used in customer environments will likely vary from customer to customer.

Cisco UCS Manager

Cisco UCS PowerTool

Enable Quality of Service in Cisco UCS Fabric

QoS policies assign a system class to the outgoing traffic for a vNIC or vHBA. This system class determines the quality of service for that traffic.

You must include a QoS policy in a vNIC policy or vHBA policy and then include that policy in a service profile to configure the vNIC or vHBA.

These steps provide details for enabling the quality of service in the Cisco UCS Fabric and setting Jumbo frames.

Cisco UCS Manager

Cisco UCS PowerTool

Create a Power Control Policy

These steps provide details for creating a Power Control Policy for the Cisco UCS environment.

Cisco UCS Manager

Cisco UCS PowerTool

Create a Scrub Policy

This policy determines what happens to local data on a server during the discovery process and when the server is disassociated from a service profile. This policy can help ensure that the data on local drives is erased at those times.

These steps provide details for creating a Scrub Policy for the Cisco UCS environment.

Cisco UCS Manager

Cisco UCS PowerTool

Create a Local Disk Configuration Policy

This policy configures any optional SAS local drives that have been installed on a server through the onboard RAID controller of the local drive. This policy enables you to set a local disk mode for all servers that are associated with a service profile that includes the local disk configuration policy. The local disk modes include the following:

·         Any Configuration – For a server configuration that carries forward the local disk configuration without any changes.

·         No Local Storage – For a diskless workstation or a SAN only configuration. If you select this option, you cannot associate any service profile which uses this policy with a server that has a local disk.

·         No RAID – For a server configuration that removes the RAID and leaves the disk MBR and payload unaltered.

·         RAID Mirrored – For a 2-disk RAID 1 server configuration.

·         RAID Stripes – For a 2-disk RAID 0 server configuration.

You must include this policy in a service profile, and that service profile must be associated with a server for it to take effect.

These steps provide details for creating a local disk configuration for the Cisco UCS environment, which is necessary if the servers in question do not have a local disk.

Cisco UCS Manager

Cisco UCS PowerTool

Create a Server Pool Qualification Policy 

This policy qualifies servers based on the inventory of a server conducted during the discovery process. The qualifications are individual rules that you configure in the policy to determine whether a server meets the selection criteria. For example, you can create a rule that specifies the minimum memory capacity for servers in a data center pool.

Qualifications are used in other policies to place servers, not just by the server pool policies. For example, if a server meets the criteria in a qualification policy, it can be added to one or more server pools or have a service profile automatically associated with it.

Depending upon the implementation, you may include server pool policy qualifications in the following policies:

·         Autoconfiguration policy

·         Chassis discovery policy

·         Server discovery policy

·         Server inheritance policy

·         Server pool policy

These steps provide details for creating a server pool qualification policy for the Cisco UCS environment.

Cisco UCS Manager

Cisco UCS PowerTool

Create a Server BIOS Policy 

BIOS policy is a policy that automates the configuration of BIOS settings. You can create one or more BIOS policies which contain a specific grouping of BIOS settings that match the needs of a server or set of servers. If you do not specify a BIOS policy for a server, the default BIOS settings are applied to the server. However, these default BIOS settings can themselves be configured as per need. If a BIOS policy is specified, the policy takes precedence over the BIOS default settings.

Any change made to the default BIOS settings does not affect a server that has a BIOS policy associated with it because the BIOS policy takes precedence over the default BIOS settings. However, in the BIOS policy, if the user leaves the value as platform-default, the UCS manager refers to that particular platform’s BIOS defaults (Servers > Policies > BIOS Defaults), and uses the value specified in the BIOS defaults.

In such cases it is possible for a user to achieve the usage of both BIOS policy and BIOS defaults. This is required because some users might want to customize only a few settings in policy, and use BIOS default values for the rest of the settings.

For example, if the user creates a BIOS policy 'test-policy' and specifies these values:

·         'disabled' for Turbo boost

·         'platform-default' for Hyper Threading

And the BIOS defaults for the platform have these values:

·         'enabled ' for Turbo boost

·         'enabled' for Hyper Threading

The net effect of this is Turbo boost set as 'disabled' and Hyper Threading set as 'enabled'.

When a configuration change is made through a BIOS policy or through default BIOS settings, Cisco UCS Manager immediately pushes these changes to the CIMC buffer. These changes take effect only after the server is rebooted.

These steps provide details for creating a server BIOS policy for the Cisco UCS environment.

Cisco UCS Manager

Cisco UCS PowerTool

 

Create vNIC/HBA Placement Policy for Virtual Machine Infrastructure Hosts

Cisco UCS Manager

Cisco UCS PowerTool

Create VMQ Connection Policy

VMQ distributes the computational workload associated with virtual machine network traffic across multiple cores in the Hyper-V host. The Cisco VIC supports up to 256 queues per server and a maximum of 128 queues per eNIC. The VMQ queue value should be configured based on the expected number of synthetic NICs in all VMs that are bound to the Hyper-V switch that is bound to the adapter, plus 2. The extra 2 queues are for the default queue and a queue for the eNIC itself.

The value configured for the number of VMQ’s should be equal to or greater than the number of required queues, but should not exceed the maximum number supported by the Cisco VIC.

The following table lists the VMQ configuration for this deployment.

The number of configured interrupts should be equal to or greater than the number of logical CPU cores in the Hyper-V host.

Cisco UCS Manager

Cisco UCS PowerTool

Create vNIC Templates

These steps provide details for creating multiple vNIC templates for the Cisco UCS environment.

Cisco UCS Manager

Repeat the above steps to create a template for each of the vNIC templates shown in the following table.

 

Cisco UCS PowerTool

Create VSANs

A named VSAN creates a connection to a specific external SAN. The VSAN isolates traffic to that external SAN, including broadcast traffic. The traffic on one named VSAN knows that the traffic on another named VSAN exists, but cannot read or access that traffic.

Like a named VLAN, the name that you assign to a VSAN ID adds a layer of abstraction that allows you to globally update all servers associated with service profiles that use the named VSAN. You do not need to reconfigure the servers individually to maintain communication with the external SAN. You can create more than one named VSAN with the same VSAN ID.

In a cluster configuration, a named VSAN can be configured to be accessible only to the FC uplinks on one fabric interconnect or to the FC Uplinks on both fabric interconnects.

These steps provide details for creating named VSANs for a given server configuration in the Cisco UCS environment.

Cisco UCS Manager

 

Cisco UCS PowerTool

Assign VSANs to FCoE Storage Ports

The following steps demonstrate the assigning VSANs to specific FCoE storage ports.

Cisco UCS Manager

Cisco UCS PowerTool

Create vHBA Templates for Fabric A and B

These steps provide details for creating a vHBA template each for fabric A and fabric B for the Cisco UCS environment.

Cisco UCS Manager

 

 

Cisco UCS PowerTool

Create Storage Connection Policies

The storage connection policy contains a collection of target storage ports on storage arrays that you use to configure Cisco UCS local Fibre Channel zoning. You can create this policy underneath an organization or an initiator group. This policy is known as a Fibre Channel storage connection policy in Cisco UCS Manager.

You add vHBA initiator groups to a storage connection policy through the Fibre Channel target endpoints.

The storage arrays in these zones must be directly connected to the fabric interconnects. The target storage ports on these arrays that you include in the storage connection policy can be either Fibre Channel storage ports or FCoE storage ports. You use the WWN of a port to add it to the policy and to identify the port for the Fibre Channel zone.

These steps provide details for creating the storage connection policies for the Cisco UCS environment.  Three different policies will be created.  The first is created with a single path to the storage array to allow for the initial installation of Windows Server.  Windows Server will not install properly if it finds multiple paths to its boot LUN.  When the installation is completed and MPIO is installed within the operating system, Windows Server will recognize multiple paths to its storage.  The next two policies will define the connection for each fabric.

Cisco UCS Manager

 

Cisco UCS PowerTool

Create Boot Policies

The boot policy determines the following:

·         Configuration of the boot device

·         Location from which the server boots

·         Order in which boot devices are invoked

For example, you can choose to have associated servers boot from a local device, such as a local disk or CD-ROM (VMedia), or you can select a SAN boot or a LAN (PXE) boot.

You must include this policy in a service profile, and that service profile must be associated with a server for it to take effect. If you do not include a boot policy in a service profile, the server uses the default settings in the BIOS to determine the boot order. These  steps  provide  details  for  creating  two boot  policies  for  the Cisco UCS  environment.  The first policy will configure booting just from Fabric A to handle the initial installation of Windows Server.  The second policy will configure the primary boot to be from SPA and secondary boot to be from SPB to be used for production deployment.

Values for the WWNNs should have been recorded earlier when you were performing the initial configuration of the VNX5400.

Cisco UCS Manager

 

Cisco UCS PowerTool

Create Service Profile Templates

With a service profile template, you can quickly create several service profiles with the same basic parameters, such as the number of vNICs and vHBAs, and with identity information drawn from the same pools.  This helps ensure a consistent server configuration across all servers performing the same or similar functions.

These steps show the creation of two Service Profile Templates.  The first template is what will be used to create the service profiles for all the Hyper-V hosts.  The second template is created as a clone of the first.  The boot policy is altered in the clone to enable the non-MPIO boot required for the initial installation of Windows Server.

 

Networking

 

Storage

 

Zoning

 

vNIC/vHBA Placement

vMedia Policy

The vMedia Policy is not used in this solution.  Click Next to move to the Server Boot Order section.

Server Boot Order

 

Maintenance Policy

Server Assignment

 

Operational Policies

Clone Template for Initial Build

When the production Service Profile Template is built, it is cloned to create a template for use to build the initial installation of Windows Server.  The boot policy is altered in the template to remove the MPIO that is configured in the production template.

Cisco UCS Manager

 

Cisco UCS PowerTool

Create Service Profiles

These steps provide details for creating multiple service profiles from the previously created template.  Multiple Service Profiles will be created for the production servers and a single Service Profile will be created for the initial build server.

Cisco UCS Manager

Cisco UCS PowerTool

The following steps provide the details necessary to prepare the host for the installation of Windows Server 2012 R2 Datacenter Edition.  The previous steps configured server profiles that created the zoning for accessing the SAN and the LUN masking on the VNX5400 so that only a single path to server is available.

To speed the process of installing Windows Server 2012 R2 across all the physical hosts, a multiple step process is employed.

·         Present the Master Boot LUN to one of the servers

·         Install Windows Server 2012 R2 on a single physical server with the boot volume on the EMC VNX5400

·         Perform some initial configuration tasks that are common for all servers used in the private cloud.

-        Configure the management network

-        Update the installation with the latest patches from Microsoft Update

-        Install Windows Roles and Features

-        Present the boot LUN to both vHBAs and configure MPIO

-        Sysprep the image so other servers can be deployed more quickly

·         Clone the sysprepped image for future use

-        Remove the boot volume from the server on which it was installed.

-        Make clones of the sysprepped volume within the EMC VNX5400 so each physical server will have its own clone to boot from.

·         Complete build of other Infrastructure hosts

-        Mask the cloned LUNs for other servers.

-        Start each host and complete the mini-setup to tailor each node with things like name, IP addressing (if fixed IP addresses are used), and join to the domain).  It is possible to configure this sort of information with unattend command files.  That is beyond the scope of this document, and many shops already have such procedures in place.

Present Master Boot LUN

These steps illustrate how to present a LUN to a Cisco UCS Service Profile and server.  This first LUN is the LUN which will contain the first installation of Windows Server 2012 R2.  This image will be prepared to be used as a clone for subsequent server installations.

EMC Unisphere

EMC Enterprise Storage Integratore (ESI) and PowerShell

The following example PowerShell script utilizes both EMC Storage Integrator and the Cisco UCS PowerTool, and expects that both have been successfully installed on the configuration workstation.  After presentation of the LUN to the WWPNs defined within the Service Profile, it will be possible to proceed with Windows Server installation.

Install Windows Server 2012 R2

Local Configuration Tasks

At this point, if you have a DHCP server installed on your Management Network, the Management Network Interface should come up with an IP address.  If you do not have DHCP, use the following steps to determine which Network Interface is on the Management VLAN and configure it with a static IP with connection to the outside world.

Configure Management Network

Run Windows Update

It is highly recommended to fully patch the server at this time from Windows Update.  Depending on the patches, it might be necessary to reboot and check for updates multiple times before the server is completely patched.  This will save time by ensuring images built from this master image will not need to have as many patches applied.  It is a good practice to periodically refresh your master image when more patches are released.

Install Windows Roles and Features

All hosts in the Private Cloud require the same set of base roles and features;

·         Multi-port IO (MPIO)

·         Failvover Clustering

·         Hyper-V

You can manually add the roles and features through Server Manager > Manage > Add Roles and Features, but the following PowerShell commands will perform the same function more quickly. This script adds MPIO feature and all the Product IDs used by EMC, adds the Failover Clustering feature, and then adds the Hyper-V role.  Adding the Hyper-V role requires the system to reboot.

Configure Paging File

Microsoft has a default formula for setting and managing the paging file.  The size of the paging file is determined based upon the amount of physical memory configured on the server. The base amount of recommended memory for this solution is 256 GB.  This results in a large amount of the system disk being allocated for use by the paging file.

As most of the memory is allocated to VM usage, little of the overall memory is used by the Hyper-V host. That minimizes the actual amount of space needed for a paging file.  Secondly, as the Hyper-V host is configured to make sure the proper amount of memory, the need to swap and/or page memory to the paging file is minimized. These factors combine to allow for a smaller paging file to be configured for the environment with no impact on the performance of the virtualized environment.

The following steps show how to set a paging file to a significantly smaller size.

Configure Other Common Criteria

In order to enable complete remote management of the server by using RSAT from another system, it is necessary to enable a number of firewall rules and configure some services.  The following PowerShell script performs this function.  Additionally, it enables the server to be accessed through a Remote Desktop Connection over the Remote Desktop Protocol.

Since these settings are affecting security settings, you should check with your security team to make sure you these settings are allowed in your environment.

At this time you should implement any standard configuration items that are required by the organization to be on all servers.  This may be an anti-malware product or a management agent or other system settings.  They will vary from one organization to another.

Update Chipset Drivers

Depending upon your configuration, you might need to update the chipset drivers on your system.  These steps illustrate how to determine if you need to update the chipset drivers and then update them.

Change Service Profile for MPIO

Up until this point the service profile in use has had just a single path to the storage.  In order to configure MPIO a different service profile that is configured with multiple paths to the storage must be used.  The following steps explain how to move the LUN that is currently associated with the Initial-Installation service profile to a service profile with two paths to storage so configuration can be completed.

Configure MPIO

When you have the WWNN and WWPNs you can use EMC’s Unisphere to mask the MasterBoot-2012R2 LUN to the service profile with two paths to the storage.  The goal here is to sysprep this operating system image then clone the LUN for use by all other physical servers. This means MPIO has to be configured only once.  When the operating system image that will be used for booting the additional blades has MPIO configured, it is possible to simply mask the LUNs in Unisphere and boot.

When the zones and zonesets have been updated to reflect the multiple paths to the LUN, it is necessary to configure the EMC VNX5400 SAN to present the boot LUN to the additional paths.

Make sure the server is powered off before starting.

Sysprep the Image

When the image is configured with all the base components that should be included in each image, Microsoft’s sysprep utility is run to create an image to be used for cloning to quickly provision physical hosts needed in the private cloud environment.

Clone Newly Created Image

With the base sysprep image created, clones can be created to replicate the contents of the master LUN for other servers in the environment.

The following procedure can be performed from EMC Unisphere to create the clone relationship and copy the data from the master LUN to the boot target LUNs.

 

Build Infrastructure Servers

When a clone of the sysprepped LUN has been created for each physical server to be built, there are several steps to complete the build process:

·         Mask each cloned LUN to a different server

·         Boot the server and complete the mini-setup of the sysprepped images

·         Configure the networking and Hyper-V virtual switches

·         Install EMC PowerPath

·         Install EMC Unisphere

Mask LUNs to Servers

Mask the LUN before completing a build from the cloned, sysprepped image. Use the Unisphere management GUI as outlined previously in the “Mask Boot LUN with EMC Unisphere” section.  There are two basic steps that need to be completed.

·         Create the four host initiator records for each possible path between the server and the storage

·         Create a Storage Group for each boot LUN and add the appropriate LUN to the storage group and add the host defined by the initiator records to the appropriate storage group

Because the sysprepped image was created after MPIO was installed, each system should boot properly with multiple paths to the storage.  Following the masking operations, start each host and complete the following steps.

Create Host Initiator Records

Create Storage Groups

 

Complete Mini-Setup

When the sysprep image has been cloned and the LUNs are properly zoned and masked so the boot volumes only appear to the owning host, every server must complete its installation.  Booting from a sysprep image runs what is referred to as a ‘mini-setup’.

At this point, you will have a complete base image.  This means you will need to perform several items that will vary from site to site.

·         Activate Windows – if using Microsoft’s Key Management Service (KMS) this can happen automatically.  If using non-volume media or Microsoft’s Multiple Activation Key (MAK), manual steps are required.

·         Rename the computer and join the computer to the Active Directory Domain (sample PowerShell below)

·         Any other company-specific required tailoring

As these are not unique to the private cloud, they are not covered in this document.

Join Server to Fabric Management Domain

It is easier to join the server to the fabric management domain.  This requires configuring the management NIC in order to enable access to the domain.

 

Rename Computer and Join to Domain 

Configure Host Networks

It is recommended that you rename the network adapters from the Windows default values of “Ethernet #x” to match the vNIC name from the UCS Service Profile.  You can use the manual procedure to find the associated NIC defined earlier in the document, or you can use the sample PowerShell script that follows.  This script requires that the target machine is domain-joined and the script is run from a configuration workstation in the same domain that has the Cisco UCS PowerTool installed.  The script makes use of a variable to define the address of the Cisco UCS Manager console; modify it to reflect the customer environment.

You should make the management network, the network on which domain communications occur, the first network in the network binding order.  This procedure is run locally on each server.

This document assigns static IP addresses to all NICs, but Microsoft products will work with all systems using DHCP addresses. Only the management network should be configured with a default gateway and be registered with DNS (this was done earlier in this document).  Though it is possible to configure static IP parameters remotely, due to the variability of customer configurations, it is easily performed from the console of the server rather than providing a script that would require extensive editing.

Configure Hyper-V Virtual Switches

Hyper-V virtual switches can be configured either through the Hyper-V Manager console or by using PowerShell. The following PowerShell script assumes that you have previously renamed all the vNICs to match the names in the service profile.  It is also designed to be executed from the configuration workstation, so it asks for the system which will be targeted.

Use the Hyper-V Manager console to create virtual switches.  These instructions assume you are running the from the configuration workstation.

Install EMC PowerPath

Each system should have EMC PowerPath installed for enhanced multi-pathing capabilities.  EMC PowerPath for Windows version 6.0 or higher should be used.  This process must be performed on each host.

 

Install EMC Unisphere Host Agent

The Unisphere Host Agent allows for host specific information to be sent to management applications, like Unisphere, for ease of administration.  LUN mapping and Operating System information as well as initiator information can be forwarded from a server to the VNX through the agent.  Follow the procedure below to install the Unisphere Host Agent on either a physical Windows Server 2012 R2 server. 

Create Fabric Management Hyper-V Cluster

After you complete the build of four servers to boot from SAN in a multipath IO environment, have all the network adapters configured the same on each host, all hosts are joined to the Active Directory domain, and you have created the appropriate shares for storing the virtual hard disks, you will create the cluster on which all the System Center 2012 R2 infrastructure virtual machines will be deployed. It is recommended that the Fabric Management cluster remain a separately managed cluster and that it not be used for tenant VMs. Windows does provide enough security to isolate different VMs, so it is acceptable to use the nodes of the Fabric Management cluster for running VMs, if that is desired, but not recommended.  This document demonstrates two clusters – one for Fabric Management and the other for Tenant virtual machines.

Add Shared Storage

Microsoft Failover Clusters use SMB storage on the EMC VNX storage array for storing the VMs.  Most virtual hard disks will be stored in the SMB 3.0 shares created previously.  However, there are three FC LUNs that need to be presented to the cluster.

·         Cluster disk witness – it is recommended to utilize the disk witness instead of a file share witness, because the disk witness will always have a backup of the cluster database.

·         SQL Server Analysis Services – does not yet support storing its databases and log files on SMB shares.  A single LUN with sufficient space for the database and log files needs to be presented to the failover cluster.

·         System Center Virtual Machine Manager library – SCVMM is installed as a highly available service.  This requires its library to be stored on a file share served by a Windows Server host.  A single with sufficient space for the library needs to be presented to the failover cluster.

These LUNs were created earlier.  They have to be added to the Unisphere storage groups assigned to the four hosts that will be used to form the Fabric Management cluster.  When the same LUN is added to multiple storage groups, the VNX will display an error message cautioning about the possibility of corrupting data.  The clustering software controls access to the LUNs, so that is acceptable.

 

Before you can test and form the cluster, it is necessary to format the shared LUNs as NTFS volumes.  Perform the following steps on only one node of the cluster to format the LUN.

After the disks have been initialized and formatted, it is a good practice to go through each server that will be part of the cluster to make sure the disks can be brought online on each node.  If you cannot bring the disks online on every server that will be part of the cluster, check your zoning and masking.  After bringing the disk online, take it offline before creating the cluster.

Run Cluster Validation Wizard

Create Fabric Management Cluster

Configure Disks

The cluster automatically selects the smallest disk and assigns it the role of Disk Witness.  The other disks are added into the cluster as Available Storage, meaning they have not been assigned to any specific role or resource group within the cluster.  Additionally, the disks have generic names of Disk 1, Disk 2, Disk 3, etc. Using the Failover Cluster Manager console, the disk names can be changed to something meaningful and the data disks can be assigned as Cluster Shared Volumes for use by the virtual machines.

Configure Networks

Multiple networks have been defined for specific usage within the cluster – Mgmt, CSV, SMB, and Live Migration.  The cluster has different capabilities that can be assigned to each network, and during the creation of the cluster, the cluster assigns certain capabilities.  However, those do not always match what we want.  We need to make sure the capabilities are properly assigned.

The cluster build process assigns default names to the NICs.  For documentation and debugging purposes, it is recommended to assign meaningful names to the NICs.

There are several defaults assigned by the cluster build process.

·         The network that was assigned during the configuration is used for cluster communication and client access.  This is what we want.

·         All networks are available for Live Migration.  We want to make sure the network we defined for Live Migration is the only one configured.

·         All networks, except the one assigned during the configuration, are defined to allow cluster communication.  We do not want cluster communication on the Live Migration and SMB networks.

The desired changes can be handled through the Failover Cluster Manager console or through PowerShell.  The following sample script shows how to take care of all the above steps using a few PowerShell cmdlets.   Names and IP addresses will need to be changed to reflect the customer environment.

The following steps show how to perform the changes through the Failover Cluster Manager console.

Change Hyper-V Default Settings

There are three defaults Hyper-V settings that need to be changed.  The following PowerShell script makes the changes.  Note that the virtual hard disk path and virtual machine path use the share created earlier.

The same setting can be performed by individually selecting each host in Hyper-V Manager and altering the Enhanced Session Mode Policy, Virtual Hard Disks, and Virtual Machine settings.

It is also necessary to configure the access rights to the SMB share on which the VMs will be stored.

Configure Cluster Aware Updating

Cluster Aware Updating (CAU) allows you to update clustered servers with little or no loss in availability during the update process. During an Updating Run, CAU does the following:

·         Transparently puts each node of the cluster into node maintenance mode

·         Temporarily fails over the clustered roles off it to other nodes

·         Installs the updates and any dependent updates on the first node

·         Performs a restart if necessary

·         Brings the node back out of maintenance mode

·         Fails back the original clustered roles back onto the node

·         Proceeds to update the next node

CAU is one of the Failover Clustering tools that are installed with the Failover Clustering feature. After you have configured the cluster, all that is left is to configure CAU.  Configuration can be accomplished either through the GUI or through PowerShell.  The following steps illustrate how to configure the tool through the GUI. The last step shows the PowerShell command.

For this example, the CAU is configured to be self-updating, meaning that it will automatically update itself based on the schedule created.  Since Microsoft publishes patches on the second Tuesday of each month, this schedule is being configured to run on the third Tuesday of each month, providing time for the patches to be tested before they are automatically applied.

Configure Constrained Delegation

Delegation is a feature that allows one computer to act on behalf of another computer.  By default, many such functions are disallowed for security reasons.  If you perform actions on a given computer, it is acting on its behalf and those functions will work.

In order for the cluster nodes to perform file functions on the EMC VNX5400, it is necessary to configure the hosts to delegate certain functions to the file service of the VNX.  In addition, if you plan to continue managing the environment from a management workstation instead of moving from host to host to perform functions, it will be necessary to configure Constrained Delegation.  If you try to use a remote management tool, such as the Remote Server Administration Toolkit (RSAT), to perform the same function, you have another computer initiating an action that is performed on another computer.  Some of those functions will generate an access violation.  Constrained Delegation allows for selecting specific functions (the ‘constrained’ part) to be delegated.

In the following example, a management workstation (Mgmt) also serves as a file server to hold ISO files for software installation.  Mgmt is configured with RSAT, so it can run the Failover Cluster Manager console and the Hyper-V Manager console, among others.  Constrained delegation needs to be configured on both the Hyper-V hosts and on the Mgmt workstation to allow for the remote management.

(Optional) Configure Replica Broker Role

One of the features of Hyper-V is the ability to asynchronously replicate virtual machines to another Hyper-V server.  In order to replicate from a Failover Cluster it is necessary to configure a Hyper-V Replica Broker on the cluster.

 

The PLA patterns at a high level include the concept of a compute, storage, and network Fabric. This is logically and physically independent from components such as System Center, which provide Fabric Management.

Figure 6        Components of the Microsoft Private Cloud

Fabric

The Fabric is typically the entire compute, storage, and network infrastructure, consisting of one or more capacity clouds (sometimes referred as Fabric resource pools) that carry characteristics like delegation of access and administration, SLAs, and cost metering. The Fabric is usually implemented as Hyper-V host clusters or stand-alone hosts managed by the System Center infrastructure.

For private cloud infrastructures, a Fabric capacity cloud constitutes of one or more scale units. In a modular architecture, the concept of a scale unit refers to the point to which a module in the architecture can be consumed (for example, scale) before another module is required. A scale unit can be as small as an individual server because it provides finite capacity, and CPU and random access memory (RAM) resources can be consumed up to a certain point. However, when it is consumed up to its maximum capacity, an additional server is required to continue scaling.

Each scale unit also has an associated amount of physical installation and configuration labor. With larger scale units, like a preconfigured full rack of servers, the labor overhead can be minimized. Thus, larger scale units may be more effective from the standpoint of implementation costs. However, it is critical to know the scale limits of all components, both hardware and software, when determining the optimum scale units for the overall architecture.

Scale units allow the documentation of all the requirements (for example, space, power, heating, ventilation and air conditioning (HVAC), and connectivity) that are needed for implementation.

Fabric Management

Fabric management is the concept of treating discrete capacity clouds as a single Fabric. Fabric Management allows centralizing and automating complex management functions that can be carried out in a highly standardized, repeatable fashion to increase availability and lower operational costs.

Provisioning Fabric Management Hosts

In order to properly size Fabric Management host systems, the following table outlines the virtual machines (and their default configurations) that are deployed to compose the fabric management component architecture. These virtual machines are hosted on a dedicated four-node Hyper-V failover cluster.

These virtual machines serve as the basis for fabric management operations. The following table summarizes the fabric management virtual machine requirements by the System Center component that supports the product or operating system role.

Additional information about preferred host to run the VM for load balancing purposes, start priority of the VM when the cluster starts, and anti-affinity (make sure two VMs do not run the same node) are also included to show the settings of a sample PowerShell script that can be used to build the VMs.

Table 15        VM Configuration Summaries

One of the features of Windows Server 2012 R2 Failover Clustering is dynamic quorum.  This gives the administrator the ability to automatically manage the quorum vote assignment for a node, based on the state of the node. When a node shuts down or crashes, the node loses its quorum vote. When a node successfully rejoins the cluster, it regains its quorum vote. By dynamically adjusting the assignment of quorum votes, the cluster can increase or decrease the number of quorum votes that are required to keep running. This enables the cluster to maintain availability during sequential node failures or shutdowns.

It is critical the Fabric Management cluster be available at all times.  Therefore, this design includes a minimum of four nodes for the Fabric Management cluster.  This helps ensure that even if a node is down for maintenance, for example, applying a patch, the remaining nodes will continue to provide a highly available environment with the remaining three nodes.  Therefore, should an unexpected hardware or software failure occur while one node is down due to planned maintenance, the remaining components of the cluster will continue their operations.  The following picture shows a typical configuration of the cluster just before taking the fourth node down for maintenance.

 

Figure 7        Management Cluster

Following the steps in the previous chapter, you created a four-node Windows Server Failover Cluster using the EMC VNX5400 for shared storage.

Create Sysprepped Virtual Hard Disk

When creating a master image for the virtual machines, a timesaving process is to make copies of this image to build each of the required infrastructure servers running as virtual machines. Therefore, additional virtual machines can be created by making copies of this master image. Either the Failover Cluster Manager console or PowerShell can be used to create the additional virtual machines.

The following instructions detail how to create the first VM that will be sysprepped for use in creating all the VMs.

When the installation completes, you should follow procedures similar to those provided earlier for the physical host, including updating with latest patches, configuring for remote management, and installing any customer-specific utilities and tools. This machine does not have to be joined to the domain.  Sysprepping removes domain information.

Many of the infrastructure servers require the .NET Framework 3.5 Features.  The following steps demonstrate how to add this feature to the image before sysprepping it.  This will save time while configuring the VMs that require this feature as you will not have to install it each time it is needed.  This is not required, but it is a time-saver.

When the above preparations have been completed, eject the Windows Server installation media, and sysprep the VM.  The VM hard disk file can be copied and used as the basis for configuring the rest of the infrastructure server VMs (except for the Service Manager Portal, which requires Windows Server 2008 R2 SP1).

To sysprep the VM enter the following command in an elevated PowerShell or command window:

This will sysprep the image and shut it down so the virtual hard disk can be copied for new machines.

To minimize the time required to copy the sysprepped VHDX file, it can be optimized with the following sample PowerShell cmdlets.  This optimizes the allocation of space used by virtual hard disk files, except for fixed virtual hard disks.

Create Infrastructure Virtual Machines from Sysprepped Image

Below are abbreviated instructions, very similar to above, for creating the infrastructure virtual machines by using the Failover Cluster Manager console. Following these instructions for creating a virtual machine from the sysprepped image are the PowerShell commands that perform the same function.

Create-VMs.ps1 PowerShell Script

Instead of using the GUI to create all infrastructure VMs, PowerShell can perform the same tasks.  The following script can be used to create infrastructure VMs in a cluster.  It must be modified to reflect each customer’s environment.  The script is supplied as a sample.  Neither support nor warranty is implied.

Configure Cluster Preferred Owners and Priority

The fabric management components have some machines that back up each other.  For example, there are at least four SQL virtual machines and two SCVMM machines.  These machines also happen to be configured in failover cluster configurations for high availability.  For machines that have a relationship like this, you will want to make sure they are running on separate nodes in the cluster.  If all nodes of the SQL cluster were running on a single node of the Hyper-V cluster, and that node failed, SQL would become temporarily unavailable while the VMs are restarted on a surviving node of the cluster.  If each VM of the SQL cluster is running on different nodes in the Hyper-V cluster, the loss of a Hyper-V node will have a lesser impact on SQL’s availability because there will be a surviving SQL VMs to continue serving the SQL instances.

To implement this capability requires that you assign preferred owners to those VMs you want to keep separated.  It is not required that all VMs be configured with preferred owners, primarily those that are serving the same role. Setting preferred owners is accomplished from within the Failover Cluster Manager console.

In addition, it is possible to help ensure that some VMs are started before others by assigning a priority to those VMs.  For example, many of the other VMs use SQL.  Therefore, it makes sense that SQL be given a higher priority when it is starting.  Other applications, such as a report server, most likely will not be needed as quickly, so it can be configured with a lower priority.  Since the settings for preferred owners and priority are located in the same window, examples will be provided here for setting some priorities.

The table at the beginning of this section shows a sample configuration for preferred nodes.  The sample PowerShell script for automatically creating the VMs will automatically assign the preferred nodes.

If you did not use the sample PowerShell script to create the Fabric Management VMs, the following steps illustrate the configuration of preferred owners and priority within the Failover Cluster Manager console.

Create Required User Accounts and Security Groups

While each System Center 2012 R2 component installation section in this document outlines the individual accounts and groups required for each installation and operation, a short summary is provided in the tables below.

Active Directory Domain User Accounts

The following Active Directory user accounts are required for the Fast Track System Center 2012 R2 installation:

Table 16        Active Directory User Accounts

Active Directory Domain Security Groups

The following Active Directory security groups are required for the Fast Track System Center 2012 R2 installation:

Table 17        Active Directory Domain Security Groups

The following sample PowerShell script will add the required accounts and groups to Active Directory. The script reads an XML file that contains the users and groups and details the group memberships. The XML file follows the script file.  Modify the XML file if you wish to use a different naming convention.

FTUsersGroups.ps1

 

FTUsersGroup.xml

 

This four-node SQL Server failover cluster contains all the databases for each System Center product in discrete instances by product and function.  This separation of instances allows for division by unique requirements and scale-over time as the needs of each component scales higher.

Not all features are supported for failover cluster installations, some features cannot be combined on the same instances, and some allow configuration only during the initial installation. Specifically this applies to database engine services and analysis services. As a rule, database engine services and analysis services are hosted in separate instances within the failover cluster. SQL Server Reporting Services (SSRS) is not a cluster aware SQL Server service and if deployed within the cluster, it can only be deployed to the scope of a single node.  For this reason, SSRS will be installed on the respective System Center component server (virtual machine).  This installation is “files only”, and the SSRS configuration provisions reporting services databases to be hosted on the component’s corresponding database instance in the SQL Server failover cluster.  The exception to this is the System Center Operations Manager Analysis Services and Reporting Services configuration.  For this instance, Analysis Services and Reporting Services must be installed with the same server and with the same instance to support Virtual Machine Manager and Operations Manager integration.  Similarly, SQL Server Integration Services is also is also not a cluster-aware SQL Server service and if deployed within the cluster, it can only be deployed to the scope of a single node. For this reason the Service Reporting SQL services functionality will be installed on the Service Reporting virtual machine.

In System Center 2012 R2, the App Controller and Orchestrator components can share an instance of SQL Server with a SharePoint farm, which provides additional consolidation of the SQL Server instance requirements.  That shared instance can be considered as a general System Center instance, while other instances are dedicated per individual System Center component.

The SQL Server 2012 SP2 failover cluster installation process includes the high-level steps shown in the following figure.

 

Figure 8        SQL: Server Failover Cluster Installation Steps

Overview

There is a decision in the SQL Server architecture that must occur prior to deployment. There are multiple valid SQL Server deployment scenarios, as follows.

·         Architecture

-        Physical servers

-        Virtual machines

·         Storage

-        Shared VHDX

-        iSCSI

-        Fibre Channel

From these choices described, the standard IaaS PLA architecture recommends a minimum two-node virtualized SQL Server guest cluster that is scaled accordingly for your deployment. The subsequent sections of this document contain guidance for deploying a four-node cluster, a configuration large enough to support a large enterprise environment.

A high-level walkthrough on how to install SQL Server 2012 SP2 is provided below. The following assumptions are made prior to installation:

·         Four base virtual machines running Windows Server 2012 R2 have been provisioned for SQL Server.

·         Cluster storage has been configured

-        One VHDX file for disk witness (1 GB)

-        One VHDX file for SSAS database and log files (15 GB)

-        SMB 3.0 file share for storage of all other database and log files

As discussed in the Infrastructure-as-a-Service Fabric Management Architecture Guide, virtual machines running SQL Server are deployed as a guest failover cluster to contain all the databases for each System Center product in discrete instances by product and function. In cases that require SQL Server Reporting Services, SQL Server Reporting Services is installed on the hosting System Center component server (for example, the Operations Manager reporting server). However, this installation is “Files only,” and the SQL Server Reporting Services configuration configures remote Reporting Services databases that are hosted on the component instance on the SQL Server cluster. All instances are required to be configured with Windows Authentication. The following table outlines the options required for each instance.

Table 18        Storage Requirements

Prerequisites

The following environment prerequisites must be met before proceeding with installation.

Accounts

Verify that the following accounts have been created:

Table 19        SQL Server Accounts

Groups

Verify that the following security groups have been created:

Table 20        SQL Server Security Groups

Establish the SQL Server Guest Cluster

The SQL VMs were created with two virtual NICs – SC-database and SC-SMB. The SC-database NIC is used by any other VM accessing the services of the SQL Server cluster and as the management NIC of the cluster. It must be configured to allow access to the Active Directory domain. The SC-SMB NIC is used to access the SMB storage.  It is not configured with any default gateway. Before proceeding, finish the configuration of the SQL VMs by configuring their networks, renaming them from the default name assigned by the sysprep process, and joining them to the Active Directory domain.

Rename and Join SQL VMs to Active Directory Domain

 

 

Assign Users and Groups to Local Administrators Group

 

Create and Add Shared VHDX to VM

 

Initialize and Format VHDX

 

Create the SQL Cluster

 

Assign SQL Access to SQL Share

The VNX5400 account and the SQL Server Admins security group need full permission to the share that will be used for SQL Server. To configure proper access permissions to the SQL share, complete the following steps:

 

Grant Installation Rights on VNX

The account used for installation of SQL Server needs the SeSecurity privilege on the VNX.  By default, no Windows account has this access to the VNX.

 

Install First SQL Server Instance in Cluster

All SQL Server instances can be installed on the first node of the cluster before moving on to install on subsequent nodes.  The following table can be tailored to the customer environment to provide answers to configuration items that are needed for the installation and configuration of each SQL instance.

Table 21        Sample System Center Component Database Worksheet

 

The IaaS PLA installation requires separate instances for each System Center product. The instances associated with these products are:

·         SCSMDB (Service Manager database)

·         SCSMDW (Service Manager data warehouse)

·         SCSMAS (Service Manager SQL Analysis Services)

·         SCDB (Shared App Controller, Orchestrator, Service Provider Foundation, Services Management Automation, Service Manager self-service portal, Microsoft SharePoint® Foundation 2010 services, and WSUS database)

·         SCVMMDB (Virtual Machine Manager database and optional WSUS database)

·         SCOMDB (Operations Manager database)

·         SCOMDW (Operations Manager data warehouse)

·         WAPDB (Windows Azure Pack database)

For multi-instance failover clusters, installation of SQL Server 2012 SP2 must be performed for each instance. As such, these steps must be performed for each instance sequentially.

Install the SQL Server Named Instances on the Guest Cluster (Additional Nodes)

After the creation of all required SQL Server instances on Node 1 is complete, additional nodes (Node 2 is required and additional nodes are optional) can be added to each instance of the cluster. Follow these steps to begin the installation of additional nodes of the cluster.

PowerShell Deployment Toolkit SQL Server Installation

Earlier in this document, reference was made to the PowerShell Deployment Toolkit (PDT).  It is beyond the scope of this document to explain its capabilities and use.  However, it is a very useful tool.  As the deployment of the shared SQL Server databases required for the Microsoft Private Cloud requires eight instances across four nodes, this requires performing the SQL Server installation process 32 times.  It is possible to create Variable.xml and Workflow.xml files to be used by the Installer.ps1 script from PDT to install all instances of SQL Server on the four node cluster.

Generally, the Workflow.xml is never modified.  When PDT was originally written, it was written to use the SQL collation values for the SQL Server database installations.  Since then Microsoft has changed its recommendation to use the Windows collation values.  This change can be accomplished by replacing all occurrences of <Collation>Default</Collation> with <Collation>Locale</Collation> within the Workflow.xml file.  No other changes are made to the Workflow.xml file.

The following represents a sample Variable.xml file that has been used to successfully install all the SQL Server database instances.  Note that it must be changed to match the customer environment.  The following changes must be made:

·         Variables at the beginning of the file.

·         Security Groups in the <Components> section

·         Values in each <SQL> section.

By default, Installer.ps1 looks for the Variable.xml and Workflow.xml files to be in the same directory.  The directory structure used by the two XML files for finding installation components is the directory structure created by the Downloader.ps1 file mentioned earlier in this document.  Installer.ps1 can be run with a –ValidateOnly switch in order to check that the modified Variable.xml file is properly formatted and the proper values are set.

When you have a clean validation, you can run the Installer.ps1 file from the installation workstation or another domain member.  It cannot be run from any of the SQL Server cluster nodes.

This information is provided as sample only.  No warranty or support is implied in the use of this sample file.

Post-Installation Tasks

When the installation is complete, the following tasks must be performed to complete the installation of SQL Server.  If you used PDT to deploy the SQL Server installations, these steps should have been completed, but you may want to validate that it worked in your environment.

Configure Windows Firewall Settings for SQL Server Named Instances

To support the multi-instance cluster, you must configure each SQL Server instance to use a specific TCP/IP port for the database engine or analysis services. The default instance of the database engine uses port 1433, and named instances use dynamic ports. To configure the firewall rules to allow access to each named instance, static listening ports must be assigned.

Use the following procedure to configure the TCP/IP port. For more information, see Configure a Server to Listen on a Specific TCP Port (SQL Server Configuration Manager) [1].

 

 

Assign Preferred Owners for SQL Server Instances in Failover Cluster Manager

To support the proper distribution of SQL Server instances across the multi-instance SQL Server cluster, you must configure failover clustering in Windows to assign preferred owners for each SQL Server instance. The following steps are provided to assist with this configuration.  One node is specified as the primary and different node as the secondary preferred owner.  This does not prevent the SQL instance from failing over to a different node than those specified in the preferred owner list, but it does make sure the instances remain balanced when preferred nodes are available.

Two servers running Virtual Machine Manager Management Server role are deployed and configured in a failover cluster that uses a dedicated SQL Server instance in the virtualized SQL Server cluster.

One library share is used for the Virtual Machine Manager library.  Provisioning the Library Share on a File Server cluster instead of a stand-alone server is recommended.  Additional library servers can be added as needed.

Virtual Machine Manager and Operations Manager integration is configured during the installation process.

The installation process for System Center 2012 R2 Virtual Machine Manager includes the high-level steps shown in Figure 9.

Figure 9        Virtual Machine Manager Installation Process

 

Overview

This section provides a high-level walkthrough for deploying Virtual Machine Manager into the IaaS PLA fabric management architecture. The following assumptions are made prior to the installation:

• Two base virtual machines running Windows Server 2012 R2 have been provisioned and configured as a Windows Failover Cluster, which includes:

-        The selected operating system installation type must be Full Installation.

-        At least two shared storage LUNs or one shared storage LUN and a file share witness disk.

-        A dedicated virtual network adapter for cluster communication.

-        At least one dedicated virtual network adapter for iSCSI communications (if using iSCSI).

• The target virtual machines must have the Windows Assessment and Deployment Kit (ADK) for Windows 8 and Windows Server 2012 R2 installed.
• The target virtual machine must have the Windows Server Update Services (WSUS) 4.0 console installed (available in Windows Server 2012 R2).

-        Virtual Machine Manager can use a WSUS root server or a downstream WSUS server. Virtual Machine Manager does not support using a WSUS replica server. The WSUS server can be dedicated to Virtual Machine Manager or it can be a WSUS server that is already in use.

• A Microsoft SQL Server instance dedicated to Virtual Machine Manager as outlined in previous steps must be available.

-        The Virtual Machine Manager SQL Server instance must be case-insensitive (this is the default in SQL Server 2012).

-        The SQL Server name must not exceed 15 characters.

-        The account used to install Virtual Machine Manager must have the rights needed to connect to the remote SQL Server instance and create databases.

• The installation account must have rights to create the distributed key management container in AD DS, or this container must already exist prior to running the Virtual Machine Manager setup.

Prerequisites

The following environment prerequisites must be met before prceeding.

Accounts

Verify that the following service accounts have been created:

Table 22        Virtual Machine Manager Accounts

Groups

Verifty that the following security groups are created:

Table 23        Virtual Machine Manager Security Groups

For more information, see Creating User Roles in VMM on Microsoft TechNet.

Rename and Join SCVMM Servers to Active Directory Domain

Install the Windows Assessment and Deployment Kit

The Virtual Machine Manager installation requires that the Windows Assessment and Deployment Kit (ADK) is installed on the Virtual Machine Manager management server. To download this kit, see Windows Assessment and Deployment Kit for Windows 8 in the Microsoft Download Center.

During installation, only the Deployment Tools and the Windows Pre-installation Environment features will be selected. This installation also assumes the Virtual Machine Manager servers have Internet access. If that is not the case, an offline installation can be performed. For more information for this installation option and complete installation details, see Installing the Windows ADK.

The ADK downloads as an installer with two options.  One option assumes that the system on which the download is executed has access to the internet for downloading all the installation files.  The second option is used to create a distribution point to be used for installing the ADK on systems that do not have access to the internet for downloading the files.  If you used the PDT to copy the distribution files, you do not need to perform the following steps that perform the second option.  You can access the installation media from the PDT download directory.

The following steps outline how to install the Windows ADK on the Virtual Machine Manager management server using the extracted files.

Install the Prerequisite Windows Server Roles and Features

The Virtual Machine Manager installation requires the WSUS Administration Tools to be installed on the Virtual Machine Manager management servers. In addition, the Failover Clustering Feature must be installed. Follow the steps below to install the pre-requisite roles and features on the Virtual Machine Manager management servers.

Install the Command-Line Utilities in SQL Server 2012 with SP2

The Virtual Machine Manager installation requires that the command-line utilities and management tools in SQL Server 2012 with SP2 are installed on the Virtual Machine Manager management server. Use the following procedure to install the command-line utilities and management tools on the Virtual Machine Manager management server.

Set Up Failover Cluster

The Virtual Machine Manager Failover cluster installation requires a quorum model. That model can be a disk witness or a file share witness. This installation assumes the use of a disk witness quorum model.

Create Shared VHDX File

Configure Shared Storage

Create the Failover Cluster

During the provisioning process, two virtual machines were built to the specifications outlined in the IaaS PLA Fabric Management Architecture Guide to support a high availability Virtual Machine Manager for fabric management. After the shared storage was created, it was configured within each virtual machine to make them accessible to each candidate cluster node.

Create the Virtual Machine Manager Distributed Key Management Container in Active Directory Domain Services

The Virtual Machine Manager installation requires that an Active Directory container be created to house the distributed key information for Virtual Machine Manager.

For more information, see Configuring Distributed Key Management in VMM.

Use the following procedure to create an AD DS container to house the distributed key information. These instructions assume that a domain controller running at least Windows Server 2008 R2 functional level is in use.

 

 

Installation

Install Virtual Machine Manager on First Node

Apply the latest Update Rollup to the installation to make sure the latest fixes are applied to the installation.

Install Second Node of SCVMM Cluster

Apply the latest Update Rollup to the installation to make sure the latest fixes are applied to the installation.

Creating Virtual Machine Manager Library Share on a Failover Cluster

In a highly available installation of Virtual Machine Manager, the Virtual Machine Manager Library must reside on a Windows Server file share outside the Virtual Machine Manager cluster infrastructure; it is not a supported configuration to reside on the Virtual Machine Manager cluster or its nodes.

In addition, creating a highly available Virtual Machine Manager Library is a recommended practice given that the Virtual Machine Manager servers are highly available servers.

The Private Cloud IaaS PLA physical architecture makes no recommendations for where the Virtual Machine Manager Library resides, other than that it should have the same high availability as other aspects of the installation. Although any file server cluster will suffice, this document details the steps required to host the Virtual Machine Manager Library on the SQL Server Cluster created in earlier portions of this document as an example.

 

 

Create a Run As Account

SCVMM has implemented a role-based access capability to allow for different individuals to have different levels of access to the functions available within SCVMM. To assign different levels of access, you need to create Run As Accounts. At a minimum, you should create a Run As Account for full administration of the SCVMM installation and start using it for any further configuration of the environment.

Add Hyper-V Hosts to be Managed by SCVMM

Register VNX File Share to SCVMM

Complete the following steps to register the VNX file share to VMM.

Configure Logical Networks

Configure Library Subdirectories (optional)

Having a library as part of VMM provides a handy location for storage of many items that are used regularly in the management and maintenance of the cloud.  It can be helpful to create subdirectories within the standard SCVMM library share for storage of items, such as distribution media in the form of ISO files.

Configure Constrained Delegation (optional)

By default, when SCVMM is creating a virtual machine, and you are using an ISO file from the library for installation purposes, the ISO file is copied and made part of the virtual machine’s definition. This wastes time copying the file and it takes extra space. It also means that different versions of installation media may end up getting stored all over. Sharing ISO items across nodes requires additional configuration of the SCVMM hosts and any system that runs the SCVMM console. This is called constrained delegation which allows the SCVMM host to operate on behalf of the virtual machine being created.

This is a security change to a default installation, so it should be reviewed with your security department before deployment.

Install and Configure Cisco Nexus 1000V

The Cisco Nexus 1000V for Microsoft Hyper-V package (a zip file) is available at the download URL location provided with the software. Complete the following steps to download the Cisco Nexus 1000V for Microsoft Hyper-V package.  Extract the contents of the zip file and find the location of these components:

·         Virtual Supervisor Module (VSM) ISO located in VSM\Install subdirectory

·         Virtual Ethernet Module (VEM) MSI package located in VEM subdirectory

·         Cisco VSEM Provider MSI package located in VMM subdirectory

Cisco Nexus 1000V for Hyper-V can be downloaded at the following link:

http://software.cisco.com/download/release.html?mdfid=284786025&flowid=42792&softwareid=282088129&release=5.2(1)SM1(5.2c)&relind=AVAILABLE&rellifecycle=&reltype=latest

Install the Virtual Supervisor Module Virtual Machine Template

Create Two Virtual Supervisor Module Virtual Machines

The Cisco Nexus 1000V is deployed as a pair of highly available virtual machines.  Each machine will have its own unique management IP address, and the highly available service will have its own virtual IP address separate from the individual machines.  It is necessary to add, at a minimum, the virtual IP address to DNS. SCVMM uses the virtual IP address for communication.

Add a Domain Name Service Record for the Virtual Supervisor Module VMs

Configure Virtual Servisor Modules in the VSM Virtual Machines

Configure the Cisco Nexus 1000V VSM for Use with SCVMM

Enter the following configuration commands on the primary VSM:

 

Connect SCVMM to VSM

Configure a Logical Switch in SCVMM

Create the Logical Switch on the Hyper-V Hosts

Create a VM Network

Configure the SCVMM Virtual Machine Properties

Configure SCVMM Network Interfaces

Rename the New Cluster Network

Configure System Center Application Virtual Machine Network Interfaces

Install Cisco UCS SCVMM Add-In

The Cisco UCS Add-in for Microsoft System Center 2012 R2 Virtual Machine Manager enables management of Cisco UCS from within SCVMM.

Installation of this add-in requires that Cisco UCS PowerTool is already installed on the servers to which the UI extensions add-in will be added.  The add-in needs to be installed on any SCVMM console from which you want to use the extensions. 

The latest add-in can be downloaded from https://software.cisco.com/download/release.html?mdfid=286282669&flowid=72562&softwareid=284574016&release=1.1.1&relind=AVAILABLE&rellifecycle=&reltype=latest.

Install the SCVMM Add-In

Configure the SCVMM Add-In

Install and Configure the EMC SMI-S Provider

SCVMM storage integration requires an SMI-S provider instance to communicate with the VNX storage array.  The following sections outline the minimum requirements for configuring the SMI-S provider and VMM environment to allow for SCVMM to manage VNX storage and perform rapid virtual machine deployment.  At a high level the required steps include:

·         Installing the EMC SMI-S Provider

·         Registering the VNX with the Provider

·         Creating the SMI-S user for the SCVMM run as account

·         Creating the run as account within SCVMM

·         Registering the EMC SMI-S provider with SCVMM

·         Creating classifications and choosing storage pools for management

·         Allocating Storage Pools to Host Groups

·         Configuring the Library Server

·         Creating a San Copy Capable Template

·         Selecting the Rapid Provisioning Deployment Method

Additional information can be found in the document titled “Storage Automation with System Center 2012 and EMC Storage Systems using SMI-S” available at https://support.emc.com.

Install the EMC SMI-S Provider

Register the VNX with the Provider

Create the SMI-S User for the SCVMM Run As Account

Create the Run As Account within SCVMM

Register the EMC SMI-S Provider with SCVMM

Allocate Storage Pools to Host Groups

 

 

Create a SAN Copy Capable Template

Select the Rapid Provisioning Deployment Method

SCVMM supports both clones and snapshots for SAN Copy-based deployments.  The copy method can be changed through PowerShell or from the GUI.  The following steps detail how to change this setting using either method.

Using ODX for Virtual Machine Deployments

Starting with SCVMM 2012 R2, you can use ODX when deploying virtual machines from templates. When using the “network” transfer type, SCVMM 2012 R2 automatically attempts to use ODX to perform the virtual machine deployments. The VNX supports ODX when copies are performed across LUNs within the same storage array and within a LUN on the storage array. For the purposes of this document, only block based VNX support for ODX is discussed. The following steps can be used to verify if ODX is enabled on the VNX.

A minimum of two Operations Manager servers are deployed in a single management group that is using a dedicated SQL Server instance in the virtualized SQL Server cluster.  An Operations Manager agent is required to be installed on every management host and each scale unit cluster node to support health monitoring functionality.  Additionally, agents may be installed on every guest virtual machine to provide guest level monitoring capabilities.

Operations Manager gateway servers and additional management servers are supported for custom solutions; however, for the base reference implementation these additional roles are not implemented.  Additionally, if there is a requirement to monitor agentless devices in the solution, such as data center switches, additional management servers should be deployed to handle the additional load.  These additional management servers should be configured into an Operations manager Resource Pool dedicated to this task (http://technet.microsoft.com/library/hh230706.aspx).  Deployment of these additional servers is beyond the scope of this CVD.

The Operations Manager installation uses a dedicated SQL Server instance in the virtualized SQL Server cluster.  The installation follows a split SQL Server configuration:  SQL Server Reporting Services and Operations Manager Management Server components reside on the Operations Manager virtual machines, and the SQL Server Reporting Services and Operations Manager databases utilize a dedicated instance on the virtualized SQL Server cluster.  Note that for the IaaS PLA implementation, the Data Warehouse is sized for 90-day retention instead of the default retention period.

The Operations Manager installation process includes the high-level steps shown in Figure 10.

 

Figure 10      Operations Manager Installation Process

Overview

This section provides a high-level walkthrough for deploying Operations Manager into the fabric management architecture. The following assumptions are made:

·         Two base virtual machines running Windows Server 2012 R2 have been provisioned for Operations Manager.

·         A SQL Server 2012 SP1 cluster with dedicated instances has been established in previous steps.

-        The default SQL Server collation settings are SQL_Latin1_General_CP1_CI_AS.

-        SQL Server full text search is required.

·         The installation will follow a remote SQL Server configuration with multiple SQL Server instances:

-        SQL Server Reporting Services and SQL Server Analysis Services are installed in one SQL instance locally on the Operations Manager reporting server. The reporting services databases will run on the remote Operations Manager data warehouse SQL FCI and the Analysis Services data will reside locally on the Operations manager reporting server.

-        The Operations Manager databases on will run on a separate SQL Server instance in the Fabric Management SQL Server cluster.

Prerequisites

The following environment prerequisites must be met before proceeding.

Accounts

Verify that the following domain accounts have been created:

Table 24        Operations Manager Accounts

Groups

Verify that the following security groups have been created.

Table 25        Operations Manager Security Groups

Add .NET Framework 3.5

The Operations Manager installation requires that .NET Framework 3.5 is enabled to support installation. Use the following procedure to enable .NET Framework 3.5.

If you created your VMs from a sysprepped image that had this installed, you can skip this step.

Install the SQL Server Reporting Services (Split Configuration) and Analysis Services

The Operations Manager installation requires SQL Server Reporting Services and SQL Server Analysis Services to be installed to support the Operations Manager reporting features and integration with Virtual Machine Manager. Perform the following procedure to install SQL Server Reporting Services and SQL Server Analysis Services to support the Operations Manager reporting features.

Install Microsoft Report Viewer 2012

The Operations Manager installation requires that Microsoft Report Viewer 2012 is installed prior to installing Operations Manager. Use the following procedure to install Microsoft Report Viewer 2012.

Configure Operations Manager SQL Server Prerequisites

The following prerequisite steps must be completed prior to the installation of Operations Manager roles.

 

 

Installation

The demonstration installation assumes the Operations Manager servers are in an isolated network with no access to the internet. If the servers have internet access, some steps can be made more automatic. The PDT provides a copy routine that will automatically copy all the required software for an installation that is not connected to the internet.

Install the Operations Manager Management Server

The following steps must be completed to install and configure the Operations Manager database and server roles.

Install the Second Operations Manager Management Server

Installation of the second Operations Manager management server is almost identical to installing the first server. The following steps show which setup entries are different during installation.

Install the Operations Manager Reporting Server

The following steps must be completed to install and configure the Operations Manager reporting server role.

Post-Installation Tasks

After the installation is complete, the following tasks must be performed to complete Operations Manager and Virtual Machine Manager integration.

Register the Required Service Principal Names for the Operations Manager Management Servers

The following steps must be performed on a domain controller or on one of the Operations Manager servers by using a domain admin account or an account with permissions to create service principal names.

Deploy and Configure the Operations Manager Agent on the Virtual Machine Manager Management Server Nodes

Deploy Operations Manager Agent on Hyper-V Hosts

Install Operations Manager Console on the Virtual Machine Manager Management Server

Download and Import the Required Prerequisite Management Packs in Operations Manager

This demonstration installation assumes the Fabric Management servers are in an isolated network without access to the internet. Use of the PDT copy routine will automate the download of the required management packs into a share that is accessible by this process.

Perform Virtual Machine Manager and Operations Manager Integration

After all prerequisite configurations and installations are performed, the integration of Virtual Machine Manager and Operations Manager can be completed.

Cisco UCS Management Pack

The Cisco UCS SCOM (System Center Operations Manager) Management Pack is a plug-in for System Center Operations Manager. It is used to monitor the health of the Cisco UCS system in the data center. With this plug-in, you can monitor chassis, blades, and service profiles across multiple Cisco UCS systems. Additionally, the Cisco UCS SCOM management pack enables correlation of faults and events between the Cisco UCS infrastructure and both bare-metal and virtualized operating systems already managed by SCOM.

Install the Cisco UCS Management Pack

Add Firewall Exceptions

After you have installed the Cisco UCS management pack, you must add firewall exceptions for port 8732 on every Operations Manager management server hosting the Cisco UCS Management Service. Issue the following PowerShell commands to create the inbound and outbound rules.

Add Cisco UCS Domains to Operations Manager

There are multiple combinations of how you may want to deploy the Cisco UCS management pack when deploying within an environment with multiple Operations Manager management servers. You can just deploy on the first management server, or you can deploy on both. These instructions provide the steps to deploy to the first management server.

Configure Run As Account

Operations Manager uses Run As accounts to establish a connection to a Cisco UCS domain. The Run As account must be an administrator account.

Configure Fault Acknowledgement

Installing and Configuring the EMC Storage Integrator Management

The installation and configuration of ESI and the SCOM management pack includes several steps outlined below:

·         Prepare ESI infrastructure

·         Install the ESI components

·         Register the VNX array with the ESI Service

·         Create an ESI Service user for the SCOM Management Pack Run As Account

·         Install the ESI SCOM Management Packs

·         Import the ESI SCOM Management Packs

·         Create an ESI Run As Account and associating the account with a Profile

·         Set Overrides for the EMC SI Service Discovery

Additional information can be found in the EMC Storage Integrator online help file, specifically the “ESI Service and ESI SCOM Management Packs” section.

The ESI Service is recommended to be installed on a separate virtual machine.  This separate virtual machine was earlier used for installing the EMC SMI-S provider.

Prepare ESI Infrastructure

To prepare the environment for using ESI the following steps need to be completed.

·         Create ESI service account for SCOM management pack Run As account

·         Assign Permissions to ESI service account

·         Create and configure Active Directory objects

It is also required to assign monitoring capabilities to this created account.  It requires the ESI PowerShell module to assign that capability, so that step will be provided after ESI PowerShell is installed.

Create ESI Service Account

 

Assign Permissions to ESI Service Account

The account that monitors the ESI Service does not need administrative access to the host running the ESI Service but it must have the following access permissions:

·         Allow log on locally permission enabled – use Local Security Policy (secpol.msc) at Security Settings > Local Policies > User Rights Assignment > Allow log on locally

·         Read access to the Application Event log – use Computer Management (compmgmt.msc) to add the account to the Event Log Readers group

·         Use WMI (Windows Management Instrumentation) to query the status of the service – use the WMI Control Console (wmimgmt.msc) to add the account under the Security tab.

Alternatively, the service account can be added to the Administrators group on the ESI Host system, if this is acceptable. This will provide all the permissions necessary for the integration to work.

To configure the minimum permissions required, follow the steps outlined here:

Monitor ESI Management Server with SCOM

Install ESI Components

Assign Monitor Role to ESI Service Account

Register the VNX with the ESI Service

Install the ESI SCOM Management Packs

Import the ESI SCOM Management Packs

Create ESI Run As Account and Associate with a Profile

Setting Overrides for the EMC SI Service Discovery

 

The Service Manager Management Server is installed on a pair of virtual machines.  A third virtual machine hosts the Management Server for the Service Manager Data Warehouse and a fourth virtual machine hosts the Service Manager Self Service Portal.

The Service Manager environment will be supported by four separate SQL instances on the virtual SQL Cluster:

·         Service Manager Management Server database (CMDB);

·         Service Manager Data Warehouse databases;

·         Service Manager Data Warehouse Analysis database and

·         SharePoint foundation database (used by the Service Manager portal).

For the IaaS PLA implementation, the Change request and Service requests are sized for 90-day retention instead of the default retention period of 365 days. The following virtual machine configurations are used.

The Service Manager installation process includes the high-level steps shown in Figure 11.

 

Figure 11      Service Manager Installation Process

Overview

This section provides a high-level walkthrough for deploying Service Manager into the fabric management architecture. The following requirements are necessary to deploy the management, data warehouse, and self-service portal servers:

Management Server

·         A base virtual machine running Windows Server 2012 R2 has been provisioned for the Service Manager management server role

·         A multi-node, SQL Server 2012 SP2 cluster with dedicated Service Manager instances has been established in previous steps for Service Manager

-        SCSMDB - instance for Service Manager management database.

·         .NET Framework 3.5 SP1 is installed

·         Microsoft Report Viewer 2008 Service Pack 1 Redistributable is installed.  To install, see article 971119 in the Microsoft Knowledge Base

·         Microsoft SQL Server 2012 Native Client is installed.  To install, see SQL Server 2012 Native Client

·         The Microsoft SQL Server 2012 Analysis Management Objects are installed.  To install, see SQL Server Analysis Management Objects

Data warehouse server

·         A base virtual machine running Windows Server 2012 R2 has been provisioned for the Service Manager management server role

·         A multi-node, SQL Server 2012 SP2 cluster with a dedicated instance has been established in previous steps for Service Manager, which includes:

-        SCSMAS – instance for SQL Server 2012 SP2 Analysis Services and SQL Server Reporting Services databases

-        SCSMDW – instance for Service Manager data warehouse databases

-        .NET Framework 3.5 SP1 is installed

·         Microsoft Report Viewer 2008 Service Pack 1 Redistributable is installed.  To install, see article 971119 in the Microsoft Knowledge Base

·         Microsoft SQL Server 2012 Native Client is installed.  To download, see SQL Server 2012 Native Client

·         Microsoft SQL Server 2012 Analysis Management Objects are installed.  To install, see SQL Server Analysis Management Objects

·         Microsoft SQL Server 2012 Reporting Services (split configuration) is installed.  Microsoft SQL Server 2012 management tools are installed

Self-service Portal Server

·         A base virtual machine running Windows Server 2008 R2 (x64) has been provisioned for the Service Manager management server role

·         A multi-node, SQL Server 2012 SP2 cluster with a dedicated instance has been established in previous steps for Service Manager

·         .NET Framework 3.5 SP1 is installed

·         SharePoint Foundation 2012 SP2 is installed.  SharePoint Foundation has a number of prerequisites.  If the self-service portal server has internet connectivity, the instructions show how to automatically install them.  If the server does not have internet connectivity, the following components must be manually downloaded and installed in this order.

-        SQL Server 2008 Native Client – http://go.microsoft.com/fwlink/?LinkId=123718&clcid=0x409

-        WCF fix for Win2008 R2 (KB976462) – http://go.microsoft.com/fwlink/?LinkID=166231

-        Windows Identity Framework (KB974405) – http://go.microsoft.com/fwlink/?LinkID=166363

-        Microsoft Sync Framework Runtime v1.0 (x64) – http://go.microsoft.com/fwlink/?LinkID=141237&clcid=0x409

-        Microsoft Chart Controls for the Microsoft .NET Framework 3.5 – http://go.microsoft.com/fwlink/?LinkID=141512

-        Microsoft Filter Pack 2.0 – http://www.microsoft.com/en-us/download/details.aspx?id=17062

-        Microsoft SQL Server 2008 Analysis Services ADOMD.NET – http://go.microsoft.com/fwlink/?LinkID=160390&clcid=0x409

-        Microsoft Server Speech Platform Runtime – http://go.microsoft.com/fwlink/?LinkID=166378

-        Microsoft Server Speech Recognition Language - TELE(en-US) http://go.microsoft.com/fwlink/?LinkID=166371

-        SQL 2008 R2 Reporting Services SharePoint 2010 Add-in – http://go.microsoft.com/fwlink/?LinkID=166379

Prerequisites

The following environment prerequisites must be met before proceeding.

Accounts

Verify that the following service accounts have been created:

Table 26        Service Manager Accounts

Groups

Verify that the following security groups have been created:

Table 27        Service Manager Security Groups

Configure Service Manager Environmental Prerequisites

Complete the following procedures to install the Service Manager roles correctly.

 

 

Add .NET Framework 3.5 on all Server Manager Servers

The Service Manager installation requires that .NET Framework 3.5 be enabled to support installation. Use the following procedure to enable.NET Framework 3.5.

 

Install Microsoft Report Viewer 2008 SP1 Redistributable on the Management and Data Warehouse Servers

The Server Manager management server and the data warehouse server installations also require the Microsoft Report Viewer 2008 SP1 Redistributable. Use the following procedure to install the Microsoft Report Viewer 2008 SP1 Redistributable.

 

Install SQL Server 2012 Native Client on the on the Management and Data Warehouse Servers

The Server Manager management server and data warehouse server installations also require that SQL Server 2012 Native Client is installed prior to installation. Use the following procedure to install SQL Server 2012 Native Client.

Install SQL Server 2012 SP1 Analysis Management Objects

The Server Manager management server and data warehouse server installations also require SQL Server 2012 SP2 Analysis Management Objects. Use the following procedure to install the SQL Server 2012 SP2 Analysis Management Objects.

 

Install SQL Server Reporting Services (Split Configuration) on the Data Warehouse Server

The Service Manager data warehouse installation requires that SQL Server Reporting Services is installed to support the Service Manager reporting features. Use the following procedure to install SQL Server Reporting Services.

Install SharePoint Foundation 2010 SP2 on the Self-Service Portal Server

SharePoint Foundation 2010 Service Pack 2 (SP2) must be installed to configure SharePoint with the SQL Server 2012 SP2 installation. Use the following procedure to install SharePoint Foundation 2010 SP2 on the Service Manager self-service portal server only.

SharePoint Foundation has a number of prerequisites.  If the self-service portal server has internet connectivity, the instructions show how to automatically install them.  If the server does not have internet connectivity, the following components must be manually downloaded and installed in this order.  These are installations that require accepting the defaults, so individual instructions are not provided.

·         SQL Server 2008 Native Client – http://go.microsoft.com/fwlink/?LinkId=123718&clcid=0x409

·         WCF fix for Win2008 R2 (KB976462) – http://go.microsoft.com/fwlink/?LinkID=166231

·         Windows Identity Framework (KB974405) – http://go.microsoft.com/fwlink/?LinkID=166363

·         Microsoft Sync Framework Runtime v1.0 (x64) – http://go.microsoft.com/fwlink/?LinkID=141237&clcid=0x409

·         Microsoft Chart Controls for the Microsoft .NET Framework 3.5 – http://go.microsoft.com/fwlink/?LinkID=141512

·         Microsoft Filter Pack 2.0 – http://www.microsoft.com/en-us/download/details.aspx?id=17062

·         Microsoft SQL Server 2008 Analysis Services ADOMD.NET – http://go.microsoft.com/fwlink/?LinkID=160390&clcid=0x409

·         Microsoft Server Speech Platform Runtime – http://go.microsoft.com/fwlink/?LinkID=166378

·         Microsoft Server Speech Recognition Language - TELE(en-US) http://go.microsoft.com/fwlink/?LinkID=166371

·         SQL 2008 R2 Reporting Services SharePoint 2010 Add-in – http://go.microsoft.com/fwlink/?LinkID=166379

Install .NET Framework 4 on the Self-Service Portal Server

Additionally, the Service Manager self-service portal installation requires that the .NET Framework 4 package is installed. Use the following procedure to install .NET Framework 4 on the self-service portal.

Request and Install an SSL Certificate on the Self-Service Portal Server

Additionally, the Service Manager self-service portal installation requires a secure socket layer (SSL) certificate to enable SSL on the portal website. If you are installing the self-service portal without SSL, you can skip this section. There are several ways to request an SSL certificate. The following procedure describes how to request the certificate through the IIS Manager console when you have an in-house CA. 

Installation

Install the Management Server

Complete the following procedure to install the Service Manager management server role.

Install Subsequent Management Servers

Repeat the installation process if installing a second Service Manager Server.  The installation is shorter in subsequent installations because the Service Manager database has already been created, so this installation will make use of that database.

Install the Data Warehouse Server

The following steps must to be completed to install the Service Manager data warehouse server role.

 

Install the Service Manager Self-Service Portal Server

The following steps must to be completed to install the Service Manager self-service portal server role.

 

Two Orchestrator Runbook servers are deployed for high availability purposes. Orchestrator provides built-in failover capabilities. By default, if the primary Runbook server fails, any runbooks that were running on that server will be started from their beginning on the standby Runbook server.  In addition, the use of multiple Runbook servers supports Orchestrator scalability. By default, each Runbook server can run a maximum of 50 simultaneous runbooks. To run larger number of simultaneous runbooks, additional Runbook servers are recommended to accommodate scale environments.

Orchestrator Web service is a REST-based service that enables Orchestration Console and various custom applications, for example, System Center Service Manager, to connect to Orchestrator in order to start and stop runbooks and retrieve information about jobs.  If the Web service is unavailable, it is not possible to stop and start new runbooks.  For high availability and additional capacity there are two IIS servers with Orchestrator Web service role installed and configured for load balancing.  For the PLA, those two servers are the same as Runbook servers.

Domain accounts are used for Orchestrator services and a domain group for the Orchestrator Users group.

The Orchestrator installation process includes the high-level steps shown in Figure 12.

 

Figure 12      Orchestrator Installation Process

Overview

This section provides the procedure to set up Orchestrator in the fabric management architecture. The following requirements are necessary for the setup:

·         Base virtual machines running Windows Server 2012 R2 have been provisioned.

·         A multinode, SQL Server 2012 SP2 cluster with a dedicated instance has been established for Orchestrator in previous steps.

·         .NET Framework 3.5 is required

Prerequisites

The following environment prerequisites must be met before proceeding.

Accounts

Verify that the following service accounts have been created:

 

Table 28        Orchestrator Accounts

Groups

Verify that the following security groups have been created:

Table 29        Orchestrator Security Groups

Add .NET Framework 3.5 and .NET Framework 4.5 with HTTP Activation

The Orchestrator installation requires that .NET Framework 3.5 and HTTP Activation for .NET 4.5 are enabled. Use the following procedure to enable these features.

 

 

Install Silverlight

Installation

Install the Full Management Server

Complete the following steps to install all Orchestrator components.

 

Install Second Server as Runbook Server

Complete the following steps to install the Orchestrator Runbook components on a second server.

Post-Installation Tasks

After the installation is complete, install and configure Orchestrator Integration Packs on the target runbook servers.

Install Microsoft Report Viewer 2012

Additionally, Orchestrator requires the Operations Manager console, but prior to installing it, you must install the Microsoft Report Viewer 2012 package.

Use the following procedure to install the Microsoft Report Viewer 2012 package.

 

Install the Operations Manager Console

Install the Virtual Machine Manager Console

Download and Register the Orchestrator Integration Packs

Complete the following steps to register the Orchestrator Integration Packs.

 

Deploy the Orchestrator Integration Packs

Complete the following steps to deploy the Orchestrator Integration Packs.

System Center 2012 R2 Orchestrator Integration Pack

The Cisco UCS OIP (Orchestrator Integration Pack) is a plug-in for System Center 2012 Orchestrator. It is used to develop runbooks for automating processes that need to read and modify information within Cisco UCS Manager.

Register the Cisco UCS OIP

After downloading the Cisco UCS OIP, extract the installation file from the zip file. Then perform the following steps on all Orchestrator management servers to register the integration pack.

 

Deploy the Cisco UCS OIP

Configure the Cisco UCS OIP

On each system running the Orchestrator Runbook Designer, configure the Cisco UCS OIP.

Install Sample Runbooks

Cisco provides a small set of sample runbooks that assist in learning how to use the various activities available in their Integration Pack.  Download the zip file and extract its contents.  Perform the import of the sample runbooks on any Runbook Designer system.

 

The App Controller installation process includes the high-level steps shown in Figure 13.

Figure 13      App Controller Installation Process

Overview

This section provides a high-level walkthrough for how to set up App Controller. The following requirements are necessary for the setup:

·         A base virtual machine running Windows Server 2012 R2 has been provisioned for App Controller.

·         A SQL Server 2012 SP2 cluster with dedicated instance has been established in previous steps for App Controller.

·         The System Center Virtual Machine Manager console is installed.

·         .NET Framework 3.5 is installed.

·         Microsoft Silverlight Runtime is installed.

·         A Trusted Server Authentication (SSL) Certificate (the CN field of the certificate must match the server name) is installed.

Prerequisites

The following environment prerequisites must be met before proceeding.

Accounts

Verify that the following service accounts have been created:

Table 30        App Controller Accounts

Groups

Verify that the following security groups have been created:

Table 31        App Controller Security Groups

Add .NET Framework 3.5

The App Controller installation requires that .NET Framework 3.5 is enabled to support installation. Use the following procedure to enable.NET Framework 3.5.

Install Silverlight

Install the Virtual Machine Manager Console

Complete the following steps install the Virtual Machine Manager console on the target App Controller virtual machines.

Installation

Install the App Controller Portal Server

Complete the following steps to install the App Controller portal server.

 

Service Management Automation is included in the System Center 2012 R2 release as an add-on component of Windows Azure Pack allowing for the automation of various tasks, similar to those performed using Orchestrator runbooks.

Service Management Automation also incorporates the concept of a runbook for developing automated management sequences, but rather than use activities to piece together the tasks, Service Management Automation relies on PowerShell workflows. PowerShell workflows are based on Windows Workflow Foundation and allow for asynchronous task management of multiple devices in IT environments.

Service Management Automation is made up of three roles: the runbook worker(s), web service(s), and the Service Management Automation PowerShell module. The Web Service provides an endpoint to which Windows Azure Pack connects. It is also responsible for assigning runbook jobs to runbook workers and delegating access user rights to Service Management Automation. Runbook workers actually initiate runbook jobs and can be deployed in a distributed fashion for redundancy purposes. A Service Management Automation PowerShell module is also included which provides a set of additional cmdlets.

The Service Management Automation installation process includes the high-level steps shown in Figure 14:

 

Figure 14      Service Manager Automation Installation Process

Overview

Service Management Automation is a set of tools that is integrated as the Automation extension in Windows Azure Pack for Windows Server. IT pros and IT developers can use Automation to construct, run, and manage runbooks to integrate, orchestrate, and automate IT business processes. Automation runbooks run on the Windows PowerShell workflow engine.

Prerequisites

The following environment prerequisites must be met before proceeding.

Accounts

Verify that the following service accounts have been created:

 

Table 32        Service Manager Automation Accounts

Groups

Verify that the following security groups have been created:

Table 33        Service Manager Automation Groups

Add Web Server Role (IIS)

The Service Management Automation installation requires the Web Server Role and several additional role features. Use the following procedure to add this role and features to the server.

 

 

Request and Install an SSL Certificate

There are several ways to request an SSL certificate. The following procedure describes how to request the certificate through the IIS Manager console.  This procedure assumes that you are running a Certificate Authority within your environment.  If you are using externally requested certificates, your procedure will be different.

Installation

Install the Web Service

Complete the following steps to install the Web Service.

 

Install the Runbook Worker

Complete the following steps to install the Runbook Worker.

Install the PowerShell Automation Module

Complete the following steps to install the PowerShell Automation Module.

 

In System Center 2012 R2, Service Provider Foundation (SPF) provides web service API that integrates with Virtual Machine Manager. Its primary purpose is to provide service providers and third party vendors with the ability to develop portals that seamlessly front end the infrastructure components of System Center.

The SPF architecture allows for compute resource management through a REST API that facilities communication with a web service through the OData protocol. Claims-based authentication can be used to verify authorized tenant resources assigned by the service provider. These resources are housed in a database.

The System Center Service Provider Foundation (SPF) 2012 R2 installation process includes the high-level steps shown in Figure 15:

Figure 15      Service Provider Foundation Installation Process

Overview

Service providers can use Service Provider Foundation technology to offer infrastructure as a service (IaaS) to their clients. If a service provider has a front-end portal for clients to interact with, Service Provider Foundation makes it possible for the clients to access the resources on their hosting provider’s system without making changes to the portal.

This section provides a high-level walkthrough for how to set up Service Provider Foundation. The following requirements are necessary for the setup:

·         A base virtual machine running Windows Server 2012 R2 has been provisioned for Service Provider Foundation.

·         A SQL Server 2012 SP2 cluster has been established in previous steps with a dedicated instance for Service Provider Foundation.

·         The System Center Virtual Machine Manager console is installed.

·         A Trusted Server Authentication (SSL) Certificate (the CN field of the certificate must match the server name) is installed.

Prerequisites

The following environment prerequisites must be met before proceeding.

Accounts

Verify that the following service accounts have been created:

Table 34        Service Provider Foundation Accounts

Groups

Verify that the following security groups have been created:

Table 35        Service Provider Foundation Groups

Add Web Server Role (IIS)

The Service Provider Foundation installation requires the Web Server Role and several additional role features. Use the following procedure to add this role and features to the server.

Install Microsoft ASP.NET Model View Control (MVC) 4

Install WCF

Install the Virtual Machine Manager Console

Complete the following steps to install the Virtual Machine Manager console on the target Service Provider Foundation virtual machine.

 

Request and Install an SSL Certificate

There are several ways to request an SSL certificate. The following procedure describes how to request the certificate through the IIS Manager console.  This procedure assumes that you are running a Certificate Authority within your environment.  If you are using externally requested certificates, your procedure will be different.

 

Installation

Install System Center Service Provider Foundation 2012 R2

Complete the following steps to install Service Provider Foundation 2012 R2.

 

Introduced in System Center 2012 R2, Service Reporting offers cloud administrators the ability to view resource consumption and operating system inventory amongst tenants.  It also provides a chargeback model to report on usage expenses.

Data for Service Reporting is collected from both Operations Manager and Windows Azure Pack, and the Service Reporting component itself is configured using PowerShell.  In order for Service Reporting to obtain information from Virtual Machine Manager, Operations Manager agents must be installed on all VMM management servers, and the VMM Operations Manager Connector must be configured.  Service Provider Foundation (SPF) is required to pass data from Operations Manager to Windows Azure Pack.  Windows Azure Pack is then used to collect data from service providers and VMM Clouds.

Excel can be used to connect to SQL Server Analysis Services to analyze the collected data.  Reports are generated to show usage and capacity data from virtual machines, along with an inventory of used tenant operating systems.

The Service Reporting installation process includes the high-level steps shown in Figure 16:

Figure 16      Service Reporting Installation Process

Overview

Service Reporting in System Center 2012 R2 enables administrators at IT hosting providers to view tenant consumption of virtual machines, resources (computation, network, and storage), and operating system inventory in their infrastructure. 

This section provides a high-level walkthrough for how to set up Service Reporting. The following requirements are necessary for the setup:

·         A base virtual machine running Windows Server 2012 R2 has been provisioned for Service Reporting.

·         .NET Framework 3.5 is installed.

Prerequisites

No specific environment prerequisites must be met before proceeding.

Accounts

No specific service accounts are required for this component

Groups

No specific groups are required for this component.

Add .NET Framework 3.5

The Reporting Services installation requires that .NET Framework 3.5 is enabled to support installation. Use the following procedure to enable .NET Framework 3.5.

Install SQL Server 2012 SP2

Installation

Install Service Reporting

Complete the following steps to install Service Reporting.

 

The Windows Azure Pack installation process includes the high-level steps shown in Figure 17.

Figure 17      Windows Azure Pack Installation Process

Overview

Windows Azure Pack (WAP) for Windows Server is a collection of Windows Azure technologies, available to Microsoft customers at no additional cost for installation into your data center. It runs on top of Windows Server 2012 R2 and System Center 2012 R2 and, using the Windows Azure technologies, enables you to offer a rich, self-service, multi-tenant cloud, consistent with the public Windows Azure experience.

WAP is designed to be very scalable. It comprises seven modules:

·         Administration Site – A portal for administrators to configure and manage resource clouds, user accounts, tenant plans, quotas, and pricing. In this portal, administrators create Web Site clouds, virtual machine private clouds, create plans, and manage user subscriptions.

·         Administration Authentication Site – By default, Windows Azure Pack uses Windows authentication for the administration portal. You also have the option to use Windows Azure Active Directory Federation Services (AD FS) to authenticate users.

·         Administration API – exposes functionality to complete administrative tasks from the management portal for administrators or using Windows PowerShell cmdlets.

·         Tenant Site – A customizable self-service portal to provision, monitor, and manage services, such as Windows Azure Pack: Web Sites, Windows Azure Virtual Machines, and Windows Azure Pack: Service Bus. In this portal, users sign up for services and create services, virtual machines, and databases.

·         Tenant Authentication Site – uses an ASP.NET Membership provider to provide authentication for the management portal for tenants.

·         Tenant Public API – enables end users to manage and configure cloud services that are included in the plans to which they are subscribed. The Tenant Public API is designed to serve all the requirements of end users that subscribe to the various services that a hosting service provider provides.

·         Tenant API – enables users, or tenants, to manage and configure cloud services that are included in the plans in which they are subscribed.

All modules can be installed into a single VM, in separate VMs, or in combinations of VMs.  As need for capacity increases, additional VMs can be deployed with the same combination of modules in a network load balanced configuration.

This deployment guide show five modules deployed in pairs of load balanced VMs and two, the Administration Site and the Administration Authorization Site, deployed as single VMs.  These two administration sites generally do not require scaling.

·         Tenant Site – WAP01, WAP01b

·         Tenant Authentication Site – WAP02, WAP02b

·         Tenant Public API – WAP03, WAP03b

·         Tenant API – WAP04, WAP04b

·         Administration API – WAP05, WAP05b

·         Administration Site – WAP06

·         Administration Authentication Site – WAP07

Prerequisites

The following environment prerequisites must be met before proceeding.

Accounts

Verify that the following service accounts have been created:

Table 36        Windows Azure Pack Service Accounts

Groups

No specific groups are required for this component.

Configure SQL Instance Permissions

Download Offline Cache

WAP can be installed either through an online GUI or through command line.  These instructions assume the use of the command line.  Using the command line requires that you first create an offline cache of the components that will be installed.  A Windows Server 2012 or 2012 R2 server with internet connectivity is required for obtaining these files.  All files should be copied to the same parent directory as they share a subdirectory structure.  For easy installation, this should be a file share accessible by all WAP virtual machines.  The PDT copy routine documented at the beginning of this document will download all the components for off-line installation.

Deploy .NET 4.5 Extended with ASP.NET

Deploy IIS Recommended Configuration

Installation

You can install the components in any order, although you will not be able to open the management portal for administrators or tenants until you have installed and configured the Service Management API and authentication sites.  The recommended installation order is:

1.    Service Management APIs

2.    Authentication Sites

3.    Management Portals

Administration API

Tenant API

Tenant Public API

Administration Authentication Site

Tenant Authentication Site

Administration Site

Tenant Site

Database Server Setup

These instructions need to be followed on the following VMs:

·         Tenant API – WAP04, WAP04b

·         Tenant Public API – WAP03, WAP03b

·         Administration Authentication – WAP07

·         Tenant Authentication – WAP02, WAP02b

·         Administration Site – WAP06

·         Tenant Site – WAP01, WAP01b

After the respective WAP product has been installed, perform the following steps to configure the connection from the VM to the WAP database.

Post-Installation

Microsoft has several recommended practices that should be followed when installation of the WAP components are completed. They can be found here - http://msdn.microsoft.com/en-us/library/jj902594.aspx.

Additionally, the solution as presented in this document creates multiple VMs for WAP components. A network load balancing solution must be implemented to load balance the components. The deployment of a load balancing solution is beyond the scope of this document.

The process required to deploy a tenant cluster is very similar to the process used to deploy the Fabric Management cluster. Most of the instructions for building the Fabric Management cluster can be used for building a tenant cluster. The following steps are required.

·         Build tenant servers

-        Create tenant Service Profile Template and Server Profiles within UCSM

-        Clone the sysprepped MasterBood2012R2 LUN for each tenant host

-        Complete tenant server configuration

·         Build tenant Hyper-V cluster

-        Create cluster shared storage

-        Run cluster validation wizard

-        Create tenant cluster

-        Configure cluster aware updating

-        Configure constrained delegation

-        Change Hyper-V default settings

·         Configure Cisco Nexus  1000V for Tenant NIC

-        Configure network segment

-        Configure logical switch in SCVMM

-        Create VM network

-        Create Logical switch on Hyper-V

Only the steps that vary significantly from the previously provided instructions are presented in this guide. For all other steps, refer to the instructions presented for deploying the Fabric Management cluster substituting appropriate naming conventions for the tenant environment.

 

Build Tenant Servers

Create Service Profile Template

These steps provide details for creating a service profile template by cloning the previously created service profile template and then modifying it. The original Service Profile Template will be modified to reflect a sample network configuration for a tenant.

The following is an example. If you require more than the single NIC for tenant VMs, you will need to adjust your procedure accordingly.

Create Service Profiles

These steps provide details for creating a service profile from a template.

The Service Profiles should automatically associate with a physical blade server from the FabMgmt pool specified in the Service Profile Template.  If not, select the unassigned Service Profile and associate it with a server.  Association with a physical blade server causes the blade to power on.

Clone VNX Boot LUN

Detailed steps for cloning the boot LUN from the MasterBoot2012R2 LUN are provided in the instructions for the deployment of the Fabric Management cluster.  Follow those steps using naming convention appropriate for the tenant environment.

Associate Boot LUN to Service Profile

When the Cisco UCS service profiles have been created, the tenant blades in the environment each have a unique configuration. To proceed with the VSPEX deployment, specific information must be gathered from each Cisco UCS blade.

Table 37        Tenant HBA WWPNs for Fabric A and Fabric B

 

After the four boot LUNs have been cloned from the MasterBoot2012R2 LUN, the LUNs must be masked to the appropriate Service profile using the WWN information gathered earlier. Detailed steps for associating the boot LUN to a service profile are provided in the instructions for the deployment of the Fabric Management cluster. Follow those steps using naming convention appropriate for the tenant environment.

Complete Tenant Server Configuration

Use the UCS Manager KVM to connect to each tenant server and boot each server.  Complete the mini-setup for each server. When the mini-setup has been completed for each server, complete the configuration for the tenant servers. In general, this will require performing at least the following steps. These steps are similar to the steps performed when building the Fabric Management servers.

·         Rename and configure networks (remember this is an example installation – your network configuration may vary for VM NICs)

-        Assign fixed IP addresses on the host networks

-        Just rename, do not assign IP information, to the VM NICs

-        Make sure the binding order places the Mgmt NIC first.

·         Rename the server and join to the Active Directory domain

·         Make sure it is up-to-date on patches

·         Install and configure EMC PowerPath and Unisphere agent

Create Tenant Hyper-V Cluster

Using instructions presented in the section on creating the Fabric Management Cluster, create the Hyper-V cluster for use by the tenant.

·         Create cluster shared storage

-        Create EMC VNX SMB share for tenant VMs

-        Add to Storage Group for each server

-        Initialize and format from one server

·         Run cluster validation wizard

·         Create tenant cluster

-        Rename cluster NICs

-        Rename disk witness

·         Assign full control on the share

-        Domain administrator account

-        SCVMM service account

-        Cluster computer account

-        Individual cluster node computer accounts

·         Configure Cluster Aware Updating

·         Configure constrained delegation

·         Change Hyper-V default settings

-        Set Enhanced Session Mode

-        Set default VM virtual hard disk path to EMC VNX SMB share

-        Set default VM path to EMC VNX SMB share

Cluster Shared Storage

During the configuration performed for the Fabric Management environment, a LUN was created in advance for use as the disk witness for the tenant cluster. A storage pool was also created for use by the file systems used for SMB shares. If it was not created with enough storage to handle the requirements of both the Fabric Management and tenant environments, it will need to be expanded. The following instructions use existing space for the creation of a tenant file system share.

During the initial configuration of the EMC VNX storage, a LUN was created to be used as the disk witness for the tenant cluster.  Add that LUN to the storage group for each node in the tenant Hyper-V cluster.  From one node, initialize and format the disk.

Complete the rest of the steps listed above to complete the creation of the tenant Hyper-V cluster.

Add Tenant Cluster to SCVMM

 

Register VNX File Share to SCVMM

Post-Deployment Tasks

Following the deployment of the tenant cluster, you may want to perform additional processes similar to those performed on the Fabric Management cluster

·         Change Hyper-V default settings

·         Configure constrained delegation

·         Configure Cluster Aware Updating

·         Configure Hyper-V Replica Broker

Instructions for these steps can be found earlier in this deployment guide in the section detailing the deployment of the Fabric Management cluster.

Configure Cisco Nexus 1000V for Tenant NIC

Configure Network Segment

Connect to the primary Cisco Nexus 1000V VSM and enter the following configuration commands.  The following example is setting up subnet 192.168.100.0/24 on VLAN 1000 for use by the tenant VMs.

Configure Logical Switch in SCVMM

To configure the logical switch in SCVMM follow the steps presented in the instructions for configuring the Cisco Nexus 1000V for the Fabric Management environment.  The following differences should be noted.

·         Substitute T01-Fabric for SC-Fabric as the name of the logical switch

·         Substitute N1KV-T01-Uplink for N1KV-SC-Uplink as the uplink port profile

·         Provide a unique name when creating the Port Classification

Create Logical Switch on Hyper-V

To create the logical switch in each Hyper-V host follow the steps presented in the instructions for configuring the Cisco Nexus 1000V for the Fabric Management environment.  The following differences should be noted:

·         Substitute T01-access for SC-access as the name of the network

·         Substitute N1KV-T01-access for N1KVSC-access as the name of the logical switch

Create a Virtual Machine Network

To create a VM network in SCVMM follow the steps presented in the instructions for configuring the Cisco Nexus 1000V for the Fabric Management environment.  The following differences should be noted:

·         Substitute T01-access for SC-access as the name of the network

 

 

 

Tim Cerling, Technical Marketing Engineer, Cisco

Tim Cerling is a Technical Marketing Engineer with Cisco's Datacenter Group, focusing on delivering customer-driven solutions on Microsoft Hyper-V and System Center products. Tim has been in the IT business since 1979. He started working with Windows NT 3.5 on the DEC Alpha product line during his 19-year tenure with DEC, and he has continued working with Windows Server technologies since then with Compaq, Microsoft, and now Cisco. During his twelve years as a Windows Server specialist at Microsoft, he co-authored a book on Microsoft virtualization technologies - Mastering Microsoft Virtualization. Tim holds a BA in Computer Science from the University of Iowa.

Acknowledgments

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, we would like to thank:

Mike Mankovsky, Technical Leader Engineering, Cisco Systems, Inc.

Mike Mankovsky is a Cisco Unified Computing System architect, focusing on Microsoft solutions with extensive experience in Hyper-V, storage systems, and Microsoft Exchange Server.  He has expert product knowledge in Microsoft Windows storage technologies and data protection technologies.

David Feisthammel, Consulting Solutions Engineer, EMC

Dave is a consulting solutions engineer with EMC Corporation based in Bellevue, Washington, just blocks from the Microsoft headquarters campus. As a member of EMC's Microsoft Partner Engineering team, he focuses on Microsoft's enterprise hybrid cloud technologies, including Windows Server, Hyper-V, and System Center. Dave is an accomplished IT professional with progressive international and domestic experience in the development, implementation, and market launch of IT solutions and products. With nearly three decades of experience in Information Technology, he has presented, lectured, taught, and written on various topics related to systems management.