FlexPod Datacenter with Cisco ACI Multi-Pod with NetApp MetroCluster IP and VMware vSphere 6.7

Deployment Guide for FlexPod Datacenter with Cisco ACI Multi-Pod with NetApp MetroCluster IP and VMware vSphere 6.7

NOTE: Works with document’s Advanced Properties “First Published” property. Click File | Properties | Advanced Properties | Custom.

Last Updated: October 19, 2018

NOTE: Works with document’s Advanced Properties “Last Updated” property. Click File | Properties | Advanced Properties | Custom.

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, see:

http://www.cisco.com/go/designzone.

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS.  CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.  IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE.  USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS.  THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS.  USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS.  RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2018 Cisco Systems, Inc. All rights reserved.

 

Table of Contents

Executive Summary. 8

Solution Overview.. 9

Introduction. 9

Audience. 9

Purpose of this Document 9

What’s New in this Release?. 9

Solution Design. 11

Architecture. 11

Physical Topology. 11

Base Infrastructure. 12

Software Revisions. 12

Required VLANs. 13

vCenter and Active Directory. 14

Physical Infrastructure. 14

FlexPod Cabling. 14

Cisco ACI Leaf-Spine Cabling. 16

Inter Pod Network Cabling. 17

MetroCluster IP Nexus 3132Q-V Switch Cabling. 18

Cisco ACI Configuration. 19

Cisco Application Policy Infrastructure Controller (APIC) 19

Cisco APIC Deployment Considerations. 19

Cisco APIC Login. 20

Cisco ACI Fabric Discovery. 21

Initial ACI Fabric Setup Verification. 22

Software Upgrade for Devices in the First Site (if required) 22

Time Zone and NTP Server 23

Domain Name Servers. 24

Fabric Wide Enforce Subnet Check for IP and MAC Learning. 25

QoS Preservation Setting. 25

Cisco ACI Inter-Pod Deployment 26

IPN Device Configurations. 26

Spine Configurations for Pod 1. 33

Multi-Pod Setup Configuration. 38

Second Site - Spine Discovery. 42

Second Site - Spine Configuration. 43

Second Site Leaf Discovery. 45

ACI Fabric Setup. 45

Software Upgrade for 40GbE Site devices (if required) 45

Setting Up Out-of-Band Management IP Addresses for New ACI Switches. 45

BGP Route Reflectors. 46

Fabric Access Policy Setup. 49

Create Link Level Policies. 49

Create CDP Policy. 50

Create LLDP Interface Policies. 50

Create Port Channel Policy. 51

Create BPDU Filter/Guard Policies. 53

Create VLAN Scope Policy. 53

Create Firewall Policy. 54

Create Virtual Port Channels (VPCs) 55

VPC – UCS Fabric Interconnects. 55

VPC – NetApp AFF Cluster 60

Configuring Common Tenant for In-Band Management Access. 64

Create VRFs. 64

Create Bridge Domain. 65

Create Application Profile. 67

Create EPG. 67

Create Security Filters in Tenant Common. 70

Deploy FPV-Foundation Tenant 72

Create Application Profile for Host Connectivity and MetroCluster 72

ACI Shared Layer 3 Out Setup. 74

Configuring the Nexus 7000s for ACI Connectivity (Sample) 75

Configuring ACI Shared Layer 3 Out 78

Storage Configuration. 88

Site Preparation. 88

Hardware Universe. 88

Interoperability Matrix Tool 88

Cluster Interconnect Switch Configuration. 88

Download the RCF. 88

Initialize the Switches. 88

Upgrade NX-OS on Switches. 89

ONTAP 9.4 Deployment 90

Disk Assignment 90

Cluster Setup. 91

Verify HA Config State of Controllers and Chassis. 94

ONTAP Base Configuration. 95

ONTAP Network Configuration. 97

MetroCluster Base Configuration. 99

ONTAP Logical Configuration. 103

Server Configuration. 111

Cisco UCS Base Configuration. 111

Perform Initial Setup. 111

Cisco UCS Setup. 113

Log in to Cisco UCS Manager 113

Upgrade Cisco UCS Manager Software to Version 3.2(3d) 113

Anonymous Reporting. 113

Configure Cisco UCS Call Home. 114

Add Block of IP Addresses for KVM Access. 114

Synchronize Cisco UCS to NTP. 115

Edit Policy to Automatically Discover Server Ports. 116

Edit Chassis Discovery Policy. 116

Verify Server and Enable Uplink Ports. 117

Acknowledge Cisco UCS Chassis and FEX. 118

Create Uplink Port Channels to Cisco Nexus 93180 Switches. 118

Create an Organization for this FlexPod. 119

Create an IQN Pool for iSCSI Boot 120

Create iSCSI Boot IP Address Pools. 121

Create MAC Address Pools. 123

Create UUID Suffix Pool 125

Create Server Pool 126

Create VLANs. 126

Modify Default Host Firmware Package. 129

Set Jumbo Frames in Cisco UCS Fabric. 130

Create Local Disk Configuration Policy (Optional) 131

Create Network Control Policy for Cisco Discovery Protocol (CDP) and Link Layer Discovery Protocol (LLDP) 132

Create Power Control Policy. 133

Create Server BIOS Policy. 134

Create High Traffic VMware Adapter Policy. 136

Update the Default Maintenance Policy. 137

Create vNIC Templates. 138

Create LAN Connectivity Policy for iSCSI Boot 144

Create iSCSI Boot Policies. 147

Create iSCSI Boot Service Profile Template. 148

Create Service Profiles. 157

Cisco UCS Configuration on 10GbE Site. 158

Storage Configuration – SAN Boot 159

Gather Necessary Information. 159

NetApp ONTAP Boot Storage Setup. 160

Create igroups. 160

Map Boot LUNs to igroups. 160

VMware vSphere 6.7 Setup. 161

VMware ESXi 6.7. 161

Download Cisco Custom Image for ESXi 6.7. 161

Log in to Cisco UCS Fabric Interconnect 161

Set Up VMware ESXi Installation. 162

Install ESXi 162

Set Up Management Networking for ESXi Hosts. 163

Log in to VMware ESXi Hosts by Using VMware Host Client 165

Set Up VMkernel Ports and Virtual Switch. 165

Setup iSCSI Multipathing. 168

Mount Required Datastores. 169

Configure NTP on ESXi Hosts. 171

Install VMware ESXi Patches and VAAI plugin. 172

VMware vCenter 6.7. 173

Building the VMware vCenter Server Appliance. 173

Setting Up VMware vCenter Server 183

ESXi Dump Collector setup for iSCSI-Booted Hosts. 186

Add APIC-Integrated vSphere Distributed Switch (vDS) 186

Create Virtual Machine Manager (VMM) Domain in APIC. 186

Attaching the VMM Domain to the IB-Mgmt EPG. 189

Add ESXi Hosts to APIC-Integrated vDS. 190

Adding Application Tenant 192

FPV-App-A Tenant Setup. 192

Adding 3-Tier-Application Profile. 193

Design Validation. 203

Data Center Failure and Recovery. 205

Validation Environment Before Site Failure. 205

Primary Site Failure. 205

Validation Environment after Site Failure. 207

Restoring the Primary Site. 208

Validation Environment after Failback. 210

Summary. 211

References. 212

Products and Solutions. 212

Interoperability Matrixes. 213

About the Authors. 214

Acknowledgements. 214

 

 

Cisco Validated Designs deliver systems and solutions that are designed, tested, and documented to facilitate and improve customer deployments. These designs incorporate a wide range of technologies and products into a portfolio of solutions that have been developed to address the business needs of the customers and to guide them from design to deployment.

Customers looking to deploy applications using a shared data center infrastructure face a number of challenges. A recurrent infrastructure challenge is to achieve the required levels of IT agility and efficiency that can effectively meet the company’s business objectives. Addressing these challenges requires having an optimal solution with the following key characteristics:

·         Availability: Help ensure applications and services availability at all times with no single point of failure

·         Flexibility: Ability to support new services without requiring underlying infrastructure modifications

·         Efficiency: Facilitate efficient operation of the infrastructure through re-usable policies

·         Manageability: Ease of deployment and ongoing management to minimize operating costs

·         Scalability: Ability to expand and grow with significant investment protection

·         Compatibility: Minimize risk by ensuring compatibility of integrated components

Cisco and NetApp have partnered to deliver a series of FlexPod solutions that enable strategic data center platforms with the above characteristics. FlexPod solution delivers an integrated architecture that incorporates compute, storage, and network design best practices thereby minimizing IT risks by validating the integrated architecture to ensure compatibility between various components. The solution also addresses IT pain points by providing documented design guidance, deployment guidance and support that can be used in various stages (planning, designing and implementation) of a deployment.

FlexPod Datacenter with Cisco ACI Multi-Pod, NetApp MetroCluster IP CVD delivers a FlexPod Datacenter solution to provide a highly available multi-datacenter environment. The multi-datacenter architecture offers the ability to balance workloads between two datacenters utilizing non-disruptive workload mobility thereby enabling migration of services between sites without the need for sustaining an outage.

The FlexPod with ACI Multi-Pod and NetApp MetroCluster IP solution showcases:

·         Seamless workload mobility across data centers

·         Consistent policies across the sites

·         Layer-2 extension across geographically dispersed DCs

·         Enhanced downtime avoidance during maintenance

·         Disaster avoidance and recovery

Introduction

FlexPod solution is a pre-designed, integrated and validated architecture for data center that combines Cisco UCS servers, Cisco Nexus family of switches and NetApp Storage Arrays into a single, flexible architecture. FlexPod is designed for high availability, with no single points of failure, while maintaining cost-effectiveness and flexibility in the design to support a wide variety of workloads.

In the FlexPod datacenter with Cisco ACI Multi-Pod and NetApp MetroCluster IP solution, Cisco ACI Multi-Pod solution allows interconnecting and centrally managing two or more ACI fabrics deployed in separate, geographically dispersed datacenters. NetApp MetroCluster IP provides a synchronous replication solution between two NetApp controllers providing storage high availability and disaster recovery in a campus or metropolitan area. This validated design enables customers to quickly and reliably deploy VMware vSphere based private cloud on a distributed integrated infrastructure thereby delivering a unified solution which enables multiple sites to behave in much the same way as a single site. 

This document covers the deployment and validation of the FlexPod datacenter with Cisco ACI Multi-Pod and NetApp MetroCluster IP solution.  The deployment is a detailed walk through of the solution buildout for two geographically distributed data centers at a distance of 75 km from each other. The FlexPod deployment discussed in this document has been validated for resiliency (under fair load) and fault tolerance during system upgrades, component failures, and partial as well as total power loss scenarios.

Audience

The audience for this document includes, but is not limited to, sales engineers, field consultants, professional services, IT managers, partner engineers, and customers who want to take advantage of an infrastructure built to deliver IT efficiency and enable IT innovation.

Purpose of this Document

This document provides a step-by-step configuration and implementation guidelines for the FlexPod Datacenter extending across multiple data centers.  These data centers are implemented with Cisco UCS Fabric Interconnects, NetApp AFF, Cisco ACI Multi-Pod solution, and NetApp MetroCluster IP. This document covers deployment of VMware vSphere 6.7 on FlexPod Datacenter using iSCSI and NFS storage protocols across the configured data centers.

What’s New in this Release?

The following design elements distinguish this version of FlexPod from previous FlexPod models:

·         Integration of Cisco ACI 3.2 Multi-Pod with FlexPod Datacenter (DC) for seamlessly supporting multiple sites.

·         Integration of NetApp MetroCluster IP for synchronous data replication across the two DCs

·         Support for vSphere 6.7

·         Setting up, validating and highlighting operational aspects of this new multi-DC FlexPod design

·         Deployment guidance for setting up and connecting two DCs using a Nexus 7000 based Inter-Pod Network.

For detailed information about the FlexPod Datacenter with Cisco ACI Multi-Pod and NetApp MetroCluster IP solution design elements, see: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_esxi67_n9k_aci_metrocluster_design.html

Architecture

The FlexPod Datacenter with Cisco ACI Multi-Pod and NetApp MetroCluster IP solution couples Cisco ACI multi-DC functionality with NetApp multi-site software features.  This solution combines array-based clustering with synchronous replication to deliver continuous availability and zero data loss. Cisco ACI Multi-Pod allows layer-2 extensions across both sites and allows layer-3 connectivity out of the ACI fabric utilizing the local gateways at each datacenter for optimal routing configuration. At the virtualization layer, VMware vSphere High Availability (HA) is enabled for all the ESXi hosts and a single vCenter instance manages this HA cluster. Two NetApp AFF A700 systems configured for MetroCluster IP provide a seamless, fully replicated storage solution for workloads deployed on these ESXi hosts in either datacenter. The validated solution achieves the following core design goals:

·         Campus wide and metro wide protection and provide WAN based disaster recovery

·         Design supporting active/active deployment use case

·         Common management layer across multiple (two) datacenters for deterministic deployment

·         Consistent policy and seamless workload migration across the sites

·         IP based storage access and synchronous replication

Physical Topology

FlexPod Datacenter with Cisco ACI Multi-Pod, NetApp MetroCluster IP and VMware vSphere 6.7 solution was validated for two very similarly built datacenters separated by a distance of 75KM. The high-level physical topology is shown in Figure 1:

Figure 1       High-level Physical Topology

Each datacenter contains all the components highlighted in a FlexPod datacenter for Cisco ACI; Cisco UCS, Cisco Nexus 9000 spine and leaf switches, a multi-node cluster of Cisco APICs and NetApp AFF A700 storage controllers. These two datacenters are then connected over two 75km long fiber links using a pair of Cisco Nexus 7004 switches at each site as shown in Figure 1. Cisco ACI Multi-Pod configuration is enabled to manage the network at both of the datacenters as a single entity such that the single cluster of APIC controllers is utilized to manage both ACI fabrics. These FlexPod datacenter components are connected and configured according to the best practices of both Cisco and NetApp and provide an ideal platform for running a variety of workloads with confidence.

Base Infrastructure

The reference architecture described in this document leverages the components detailed in the FlexPod Datacenter with VMware 6.5 Update1 and Cisco ACI 3.1 Design Guide. The FlexPod with ACI design at each datacenter was built using the core design principles and configurations outlined in the FlexPod Datacenter with VMware 6.5 Update1 and Cisco ACI 3.1 Design Guide. However, some software and hardware upgrades such as the new vSphere version 6.7, ACI version 3.2, and NetApp data ONTAP 9.4 running on NetApp AFF A700 controllers are new additions to this solution.  

Software Revisions

Table 1 lists the software versions for hardware and virtual components used in this solution.  Each version used has been certified within interoperability matrixes supported by Cisco, NetApp, and VMware.  For more information about supported versions, consult the following sources:

·         Cisco UCS Hardware and Software Interoperability Tool

·         NetApp Interoperability Matrix

·         VMware Compatibility Guide

·         Cisco ACI Recommended Release

·         ACI Virtualization Compatibility Matrix

Table 1    Software Revisions

Required VLANs

Table 2 outlines the VLANs necessary for solution deployment. In this table, ACI-VMM range indicates dynamically assigned VLANs from the APIC-Controlled VMware Virtual Distributed Switch.

Table 2    Required VLANs

vCenter and Active Directory

The vCenter and AD servers used in this validation were installed within the FlexPod environment however their installation is not covered in this document. Table 3 lists the required VMs:

Table 3    Infrastructure Virtual Machines

Physical Infrastructure

FlexPod Cabling

The information in this section is provided as a reference for cabling the physical equipment in a FlexPod environment. The tables in this section contain details for the prescribed and supported configuration of the NetApp AFF A700 running NetApp ONTAP® 9.4.

This document assumes that out-of-band management ports are plugged into an existing management infrastructure at the deployment sites. These interfaces will be used in various configuration steps. Make sure to use the cabling directions in this section as a guide.

The NetApp storage controller and disk shelves should be connected according to best practices for the specific storage controller and disk shelves. For disk shelf cabling, refer to the Universal SAS and ACP Cabling Guide: https://library.netapp.com/ecm/ecm_get_file/ECMM1280392.

Figure 2 details the cable connections used in the validation environment for one of the sites based on the Cisco UCS 6332-16UP Fabric Interconnect.

An additional 1Gb management connection is required for an out-of-band network switch apart from the FlexPod infrastructure. Cisco UCS fabric interconnects and Cisco Nexus switches are connected to the out-of-band network switch, and each NetApp AFF controller has two connections to the out-of-band network switch.

Figure 2       FlexPod Cabling with Cisco UCS 6332-16UP Fabric Interconnect

Figure 3 details the cable connections used in the validation lab for the 10GbE site based on the Cisco UCS 6248UP Fabric Interconnect.

Figure 3       FlexPod Cabling with Cisco UCS 6248UP Fabric Interconnect

Cisco ACI Leaf-Spine Cabling

Figure 4 and Figure 5 show the ports and connections between the leaf and spine switches at two sites. Each leaf switch is connected to both of the spine switches in its respective site using 40G links. Customers can increase the number of links from each leaf switch to the spine switch based on their bandwidth requirements.

Figure 4       Leaf-Spine Connectivity for the 40GbE Site

Figure 5       Leaf-Spine Connectivity for 10GbE Site

Inter Pod Network Cabling

The Inter Pod Network (IPN) consists of four Nexus 7004 switches connected to ACI spine switched as shown in Figure 6:

Figure 6       Inter Pod Network Physical Design

At each site, both of the Cisco Nexus 9504 Spine switches are connected to both the Cisco Nexus 7004 devices for high availability using 10GbE connections. Cisco Nexus 9504 Spine switches with N9K-X9732C-EX line-card only support 40G ports therefore to connect the 40G port on Cisco Nexus 9504 to a 10G port on the Cisco Nexus 7004, the CVR-QSFP-SFP10G adapter is utilized. The two long distance connections between the Cisco Nexus 7004 devices are achieved by using SFP-10G-ZR optics connected to 75KM long fiber. As shown in Figure 6, all the network connectivity is fully redundant and failure of one or in some cases more than one link (or device) will keep the connectivity between the sites intact.

MetroCluster IP Nexus 3132Q-V Switch Cabling

The Cisco Nexus 3132Q-V intercluster switch is used for the cluster HA interconnect, MetroCluster IP iSCSI traffic, and IP HA and DR replication across the IP fabric. This switch is dedicated for back-end connectivity only and can be configured quickly with Remote Configuration File (RCF) files, downloadable from the NetApp Support site. The physical connectivity from the MetroCluster IP nodes to the intercluster switch is shown in Figure 7:

Figure 7       MetroCluster IP Physical Connectivity

In the validation setup, all the links shown in Figure 7 including the cross-site links are 40GbE links. If the workload does not require very high throughputs, 10GbE can be utilized for connecting the two sites.

This section provides a detailed procedure for configuring the Cisco ACI fabric for use in a FlexPod datacenter with Cisco ACI Multi-Pod and NetApp MetroCluster IP solution environment. In order to deploy an ACI Multi-Pod network, customer can have a single site ACI configuration already in place which can be easily extended to incorporate the Inter-Pod Network (IPN) configuration and a second site configuration. This deployment guide describes the Greenfield deployment scenario where two new sites are being brought up at the same time. When setting up Cisco ACI Multi-Pod for these two new datacenters, configuration steps are performed in the following order:

1.    Setup base Cisco ACI configuration for the first site including leaf and spine switch discovery.

2.    IPN device configurations and connectivity in both 10GbE site and 40GbE site.

3.    Spine switch configuration for IPN connectivity on 10GbE site (configured first).

4.    Multi-Pod configuration to connect the two sites together.

5.    Spine and Leaf switch discovery on 40GbE site (configured second).

6.    Setting up Spine switch configuration on 40GbE site to complete the Multi-Pod setup.

7.    Setting up network, compute and storage configuration on both sites.

While not detailed in this deployment guide, if you are trying to incorporate multi-pod configuration to an existing datacenter with pre-configured ACI fabric, the following modified steps need to be followed:

1.    IPN device configurations and connectivity in both sites.

2.    Spine switch configuration for IPN connectivity on first site (10GbE site in this deployment).

3.    Multi-Pod configuration to connect the two sites together.

4.    Spine and Leaf switch discovery on second site (40GbE site in this deployment).

5.    Setting up Spine switch configuration on second site to complete the Multi-Pod setup.

6.    Setting up network, compute and storage configuration on the new site

Cisco Application Policy Infrastructure Controller (APIC)

Cisco APIC Deployment Considerations

Cisco APIC design considerations are covered in the FlexPod datacenter with Cisco ACI Multi-Pod and NetApp MetroCluster Design guide:

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_esxi67_n9k_aci_metrocluster_design.html#_Toc523896846

In the following steps, the configuration information from a single (primary) APIC deployed in the first site is covered. Cisco recommends a cluster of at least 3 or more APICs controlling a single site ACI Fabric and to follow the recommendations outlined above for Multi-Pod setup.

Cisco APIC Login

To verify the device setup using Cisco APIC, complete the following steps: 

1.    Log into the APIC GUI using a web browser, by browsing to the out of band IP address configured for APIC. Login with the admin user id and password.

2.    Take the appropriate action to close any warning or information screens.

Cisco ACI Fabric Discovery

This section details the steps for adding the two Nexus 93180LC-EX leaf switches to the Fabric. This procedure assumes a green field deployment where new leaf switches are being added to the fabric.  If the fabric has already been configured for the first site, skip this section and move on to Multi-Pod configuration sections.

These switches are automatically discovered in the ACI Fabric and available in the APIC for assigning a node name and node ID. To add Nexus 93180LC-EX leaf switches to the Fabric, complete the following steps:

1.    At the top in the APIC home page, select the Fabric tab and make sure Inventory under Fabric is selected.

2.    In the left pane, select Fabric Membership.

3.    The two 93180 Leaf Switches will be listed on the Fabric Membership page with Node ID 0

4.    Connect to the two Nexus 93180 leaf switches using serial consoles and login in as admin with no password (press enter).  Use show inventory to get the leaf’s serial number.

5.    Match the serial numbers from the leaf listing to determine the A and B switches under Fabric Membership. 

6.    In the APIC GUI, under Fabric Membership, double-click the A leaf in the list.  Enter a Node ID and a Node Name for the Leaf switch and click Update.

7.    Repeat step 6-8 for the B leaf switch as well as any additional leaf switches that are discovered.

8.    Click Topology in the left pane.  The discovered ACI Fabric topology will appear.  It may take a few minutes for the Nexus 93180 Leaf switches to appear and you will need to click the refresh button for the complete topology to appear.

Initial ACI Fabric Setup Verification

This section details the steps for the initial setup of the Cisco ACI Fabric, where

·         The software release is updated (if required)

·         NTP and DNS servers are setup

·         BGP route reflectors are configured

·         Fabric wide subnet enforcement

·         Update QoS preservation setting

Software Upgrade for Devices in the First Site (if required)

To verify and upgrade the Cisco ACI software, complete the following steps:

1.    In the APIC GUI, at the top select Admin > Firmware.

2.    This document was validated with ACI software release 3.2(2l). Select Fabric Node Firmware in the left pane under Firmware Management. All switches should show the same firmware release and the release version should be at minimum n9000-13.2(2l). The switch software version should also correlate with the APIC version.

3.    Click Admin > Firmware > Controller Firmware. If all the APICs are not at the same release level i.e. a minimum of 3.2(2l), follow the Cisco APIC Management, Installation, Upgrade, and Downgrade Guide to upgrade both the APICs and switches if the APICs are not at a minimum release of 3.2(2l) and the switches are not at n9000-13.2(2l). 

Time Zone and NTP Server

This procedure will allow customers to verify and add setup of an NTP server for synchronizing the fabric time. To verify the time zone and NTP server set up, complete the following steps:

1.    To verify a previously added NTP setup in the fabric, select and expand Fabric > Fabric Policies

2.    From the left pane, select Policies > Pod > Date and Time.

3.    Select default. In the Datetime Format - default pane, verify the correct Time Zone is selected and that Offset State is enabled. Adjust as necessary and click Submit and Submit Changes in the resulting pop-up window.

4.    On the left, expand Policy default. Verify that at least one NTP Server is listed.

5.    If desired, select enabled for Server State to enable the ACI fabric switches as NTP servers. Click Submit.

6.    If an NTP server had not been previously defined, on the right pane use the + sign next to NTP servers to add NTP servers accessible on the out of band management subnet. Enter an IP address and select the default (Out-of-Band) Management EPG. Click Submit to add the NTP server. Repeat this process to add all NTP servers.

Domain Name Servers

To verify and add DNS server in the ACI fabric, complete the following steps:

1.    Select and expand Fabric > Fabric Policies

2.    In the left pane, expand Policies > Global > DNS Profiles and select default.

3.    Verify the DNS Providers and DNS Domains.

4.    If DNS server has not been previously defined, in the Management EPG drop-down, select the default (Out-of-Band) Management EPG. Use the + sign to the right of DNS Providers followed by the + sign to the right of DNS Domains to add DNS servers and the DNS domain name.

5.    Click Submit to complete the DNS configuration.

Fabric Wide Enforce Subnet Check for IP and MAC Learning

In this ACI deployment, Enforce Subnet Check for IP & MAC Learning is enabled. To verify this setting, complete the following steps:

1.    Select and expand System > System Settings > Fabric Wide Setting.

2.    Make sure that Enforce Subnet Check is selected.  If Enforce Subnet Check is not selected, select it and click Submit.

QoS Preservation Setting

In this FlexPod with ACI deployment, QoS Preservation should be turned off, complete the following steps to verify and set this setting:

1.    Select and expand Fabric > Access Policies > Policies > Global > QOS Class.

2.    Make sure that Preserve QoS is not selected. 

Cisco ACI Inter-Pod Deployment

In the previous steps, both leaf and spine switches in first site, POD 1, have already been added to the ACI environment. The steps below will walk you through setting up:

1.    IPN device configurations in both sites.

2.    Spines configuration on first site (10GbE site).

3.    Multi-Pod configuration.

4.    Spines and Leaves discovery on second site (40GbE site).

5.    Setting up Spines configuration on the second site to complete the Multi-Pod setup.

Refer to Figure 8 for details about various device links and associated IP addresses and subnets used in the configurations below. The second site being added is assigned POD ID 11.

Figure 8       IPN and Spine Connectivity

IPN Device Configurations

This section details the relevant configurations of the four IPN devices. The configuration enables multicast, OSPF, jumbo MTU and DHCP relay on all the IPN devices. The IPN devices are connected using two dedicated 10Gbps links. The configuration below uses a single IPN device as RP for PIM bidir traffic. In a production network, deploying a phantom RP for high availability is recommended: https://www.cisco.com/c/dam/en/us/products/collateral/ios-nx-os-software/multicast-enterprise/prod_white_paper0900aecd80310db2.pdf.

Pod 1 7004-1

feature ospf

feature pim

feature lacp

feature dhcp

feature lldp

!

! Define the multicast groups and associated RP addresses

!

ip pim rp-address 10.241.255.1 group-list 225.0.0.0/8 bidir

ip pim rp-address 10.241.255.1 group-list 239.0.0.0/8 bidir

ip pim ssm range 232.0.0.0/8

!

service dhcp

ip dhcp relay

!

interface port-channel1

  description To Pod 1 7004-2

  mtu 9216

  ip address 10.242.252.1/30

  ip ospf network point-to-point

  ip ospf mtu-ignore

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

 

!

interface Ethernet3/21

  mtu 9216

  channel-group 1 mode active

  no shutdown

 

interface Ethernet3/22

  mtu 9216

  channel-group 1 mode active

  no shutdown

!

interface Ethernet3/13

  description to Pod1 Spine-1 interface E4/29

  mtu 9216

  no shutdown

!

interface Ethernet3/13.4

  mtu 9216

  encapsulation dot1q 4

  ip address 10.242.241.2/30

  ip ospf network point-to-point

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  !

  ! DHCP relay for forwarding DHCP queries to APIC’s in-band IP address

  !

  ip dhcp relay address 10.12.0.1

  ip dhcp relay address [APIC 2’s IP]

  ip dhcp relay address [APIC 3’s IP]

  no shutdown

!

interface Ethernet3/14

  description Pod 11 Spine-2 interface E4/29

  mtu 9216

  no shutdown

!

interface Ethernet3/14.4

  mtu 9216

  encapsulation dot1q 4

  ip address 10.242.243.2/30

  ip ospf network point-to-point

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  ip dhcp relay address 10.12.0.1

  ip dhcp relay address [APIC 2’s IP]

  ip dhcp relay address [APIC 3’s IP]

  no shutdown

!

interface Ethernet3/24

  description Link to Pod 11 7004-1 E3/12

  mtu 9216

  ip address 10.241.253.2/30

  ip ospf network point-to-point

  ip ospf mtu-ignore

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  no shutdown

!

interface loopback0

  description Loopback to be used as Router-ID

  ip address 10.242.255.31/32

  ip router ospf 10 area 0.0.0.0

!

interface loopback1

  description PIM RP Address

  ip address 10.241.255.1/30

  ip ospf network point-to-point

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

!

router ospf 10

  router-id 10.242.255.31

  log-adjacency-changes

!

 

Pod 1 7004 – 2

feature ospf

feature pim

feature lacp

feature dhcp

feature lldp

!

! Define the multicast groups and associated RP addresses

!

ip pim rp-address 10.241.255.1 group-list 225.0.0.0/8 bidir

ip pim rp-address 10.241.255.1 group-list 239.0.0.0/8 bidir

ip pim ssm range 232.0.0.0/8

!

service dhcp

ip dhcp relay

!

interface port-channel1

  description To Pod 1 7004-1

  mtu 9216

  ip address 10.242.252.2/30

  ip ospf network point-to-point

  ip ospf mtu-ignore

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

!

interface Ethernet3/21

  mtu 9216

  channel-group 1 mode active

  no shutdown

 

interface Ethernet3/22

  mtu 9216

  channel-group 1 mode active

  no shutdown

!

interface Ethernet3/13

  description to Pod1 Spine-1 interface E4/30

  mtu 9216

  no shutdown

!

interface Ethernet3/13.4

  mtu 9216

  encapsulation dot1q 4

  ip address 10.242.242.2/30

  ip ospf network point-to-point

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  !

  ! DHCP relay to forward DHCP queries to APIC’s in-band IP address

  !

  ip dhcp relay address 10.12.0.1

  ip dhcp relay address [APIC 2’s IP]

  ip dhcp relay address [APIC 3’s IP]

  no shutdown

!

interface Ethernet3/14

  description to Pod1 Spine-2 interface E4/30

  mtu 9216

  no shutdown

!

interface Ethernet3/14.4

  mtu 9216

  encapsulation dot1q 4

  ip address 10.242.244.2/30

  ip ospf network point-to-point

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  ip dhcp relay address 10.12.0.1

  ip dhcp relay address [APIC 2’s IP]

  ip dhcp relay address [APIC 3’s IP]

  no shutdown

!

interface Ethernet3/24

  description Link to Pod 1 7004-2 E3/12

  mtu 9216

  ip address 10.241.254.2/30

  ip ospf network point-to-point

  ip ospf mtu-ignore

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  no shutdown

!

interface loopback0

  description Loopback to be used as Router-ID

  ip address 10.242.255.32/32

  ip router ospf 10 area 0.0.0.0

!

router ospf 10

  router-id 10.241.254.2

  log-adjacency-changes

!

 

Pod 11 7004-1

feature ospf

feature pim

feature lacp

feature dhcp

feature lldp

!

! Define the multicast groups and associated RP addresses

!

ip pim rp-address 10.241.255.1 group-list 225.0.0.0/8 bidir

ip pim rp-address 10.241.255.1 group-list 239.0.0.0/8 bidir

ip pim ssm range 232.0.0.0/8

!

service dhcp

ip dhcp relay

!

interface port-channel1

  description To Pod 11 7004-2

  mtu 9216

  ip address 10.241.252.1/30

  ip ospf network point-to-point

  ip ospf mtu-ignore

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

!

interface Ethernet3/9

  mtu 9216

  channel-group 1 mode active

  no shutdown

 

interface Ethernet3/10

  mtu 9216

  channel-group 1 mode active

  no shutdown

!

 

interface Ethernet3/1

  description to Pod11 Spine-1 interface E4/29

  mtu 9216

  no shutdown

!

interface Ethernet3/1.4

  mtu 9216

  encapsulation dot1q 4

  ip address 10.241.241.2/30

  ip ospf network point-to-point

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  !

  ! DHCP relay for forwarding DHCP queries to APIC’s in-band IP address

  !

  ip dhcp relay address 10.12.0.1

  ip dhcp relay address [APIC 2’s IP]

  ip dhcp relay address [APIC 3’s IP]

  no shutdown

!

interface Ethernet3/2

  description Pod 11 Spine-2 interface E4/29

  mtu 9216

  no shutdown

!

interface Ethernet3/2.4

  mtu 9216

  encapsulation dot1q 4

  ip address 10.241.243.2/30

  ip ospf network point-to-point

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  ip dhcp relay address 10.12.0.1

  ip dhcp relay address [APIC 2’s IP]

  ip dhcp relay address [APIC 3’s IP]

  no shutdown

!

 

interface Ethernet3/12

  description Link to Pod 1 7004-1

  mtu 9216

  ip address 10.241.253.1/30

  ip ospf network point-to-point

  ip ospf mtu-ignore

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  no shutdown

!

interface loopback0

  description Loopback to be used as Router-ID

  ip address 10.241.255.31/32

  ip router ospf 10 area 0.0.0.0

 

router ospf 10

  router-id 10.241.255.31

  log-adjacency-changes

!

 

Pod 11 7004-2

feature ospf

feature pim

feature lacp

feature dhcp

feature lldp

!

! Define the multicast groups and associated RP addresses

!

ip pim rp-address 10.241.255.1 group-list 225.0.0.0/8 bidir

ip pim rp-address 10.241.255.1 group-list 239.0.0.0/8 bidir

ip pim ssm range 232.0.0.0/8

!

service dhcp

ip dhcp relay

!

interface port-channel1

  description To POD 11 7004-1

  mtu 9216

  ip address 10.241.252.2/30

  ip ospf network point-to-point

  ip ospf mtu-ignore

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

!

interface Ethernet3/9

  mtu 9216

  channel-group 1 mode active

  no shutdown

!

interface Ethernet3/10

  mtu 9216

  channel-group 1 mode active

  no shutdown

!

interface Ethernet3/1

  description Pod 11 Spine-1 E4/30

  mtu 9216

  no shutdown

 

interface Ethernet3/1.4

  mtu 9216

  encapsulation dot1q 4

  ip address 10.241.242.2/30

  ip ospf network point-to-point

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  ip dhcp relay address 10.12.0.1

  ip dhcp relay address [APIC 2’s IP]

  ip dhcp relay address [APIC 3’s IP]

  no shutdown

!

interface Ethernet3/2

  description To Pod 11 Spine-2 E4/30

  mtu 9216

  no shutdown

!

interface Ethernet3/2.4

  mtu 9216

  encapsulation dot1q 4

  ip address 10.241.244.2/30

  ip ospf network point-to-point

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  ip dhcp relay address 10.12.0.1

  ip dhcp relay address [APIC 2’s IP]

  ip dhcp relay address [APIC 3’s IP]

  no shutdown

!

interface Ethernet3/12

  description Link to Pod 1 7004-2

  mtu 9216

  ip address 10.241.254.1/30

  ip ospf network point-to-point

  ip ospf mtu-ignore

  ip router ospf 10 area 0.0.0.0

  ip pim sparse-mode

  no shutdown

!

interface loopback0

  description Loopback to be used as Router-ID

  ip address 10.241.255.32/32

  ip router ospf 10 area 0.0.0.0

!

router ospf 10

  router-id 10.241.255.32

  log-adjacency-changes

!

Spine Configurations for Pod 1

This section describes the relevant configurations of the Spine switches to enable Multi-Pod configuration. As previously stated, both sites are configured with different Tunnel Endpoint (TEP) pools, 10.11.0.0/16 (Pod11) and 10.12.0.0/16 (Pod1).

Create VLAN Pool

1.    Log into the APIC GUI and follow Fabric >Access Policies.

2.    In the left pane, expand Pools and right-click VLAN and select Create VLAN Pool.

3.    Provide a Name for the VLAN Pool (MultiPod-vlans), select static Allocation and click + to add a VLAN range.

4.    In the VLAN range, enter a single VLAN 4, select Static Allocation and click OK.

 

5.    Click Submit to finish creating VLAN pool

Create AEP for Spine Connectivity

1.    Log into the APIC GUI and follow Fabric > Access Policies.

2.    In the left pane, expand Global Policies, right-click Attachable Access Entity Profile and select Create Attachable Access Entity Profile.

3.    Provide a Name for the AEP (MultiPod-aep) and click Next.

 

4.    Click Finish to complete the AEP creation without adding any interfaces.

Create External Routed Domain for Spine Connectivity

1.    Log into the APIC GUI and follow Fabric > Access Policies.

2.    In the left pane, expand Physical and External Domains, right-click External Routed Domains and select Create Layer 3 Domain.

3.    Provide a Name for the Layer 3 domain (MultiPod-L3).

4.    From the Associated Attachable Entity Profile drop-down list, select the recently created AEP (MultiPod-aep).

5.    From the VLAN Pool drop-down list, select the recently created VLAN Pool (MultiPod-vlans).

6.    Click Submit to finish creating the Layer 3 domain.

Create Link Level Interface Policy for Spine Connectivity

1.    Log into the APIC GUI and follow Fabric > Access Policies.

2.    In the left pane, expand Interface Policies > Policies > Link Level.

3.    Right-click Link Level and select Create Link Level Policy.

4.    Provide a name for the Link Level Policy (MultiPod-inherit) and make sure Auto Negotiation is set to on and Speed is set to inherit.

5.    Click Submit to create the policy.

Create Spine Policy Group

1.    Log into the APIC GUI and follow Fabric > Access Policies.

2.    In the left pane, expand Interface Policies > Policy Groups.

3.    Right-click Spine Policy Group and select Create Spine Access Port Policy Group.

4.    Provide a Name for the Spine Access Port Policy Group (MultiPod-PolGrp).

5.    From the Link Level Policy drop-down list, select recently created policy (MultiPod-Inherit).

6.    From the CDP Policy drop-down list, select the previously created policy to enable CDP (CDP-Enabled).

7.    From the Attached Entity Profile drop-down list, select the recently created AEP (MultiPod-aep).

8.    Click Submit.

Create Spine Interface Profile

1.    Log into the APIC GUI and follow Fabric > Access Policies.

2.    In the left pane, expand Interface Policies > Profiles.

3.    Right-click Spine Profiles and select Create Spine Interface Profile.

4.    Provide a Name for the Spine Interface Profile (MultiPod-Spine-IntProf).

5.    Click + to add Interface Selectors.

6.    Provide a Name for the Spine Access Port Selector (Spine-Intf).

7.    For Interface IDs, add interfaces that connect to the two IPN devices (4/29-4/30)

8.    From the Interface Policy Group drop-down list, select the recently created Policy Group (MultiPod-PolGrp).

9.    Click OK to finish creating Access Port Selector.

10.  Click Submit to finish creation Spine Interface Profile.

Create Spine Profile

1.    Log into the APIC GUI and follow Fabric > Access Policies.

2.    In the left pane, expand Switch Policies > Profiles.

3.    Right-click Spine Profiles and select Create Spine Profile.

4.    Provide a Name for the Spine Profile (Spine-Prof).

5.    Click + to add Spine Selectors.

6.    Provide a Name for the Spine Selector (Pod1-Spines) and from the drop-down list under Blocks, select the spine switch IDs (211-212).

7.    Click Update and then Next

8.    For the Interface Selector Profiles, select recently created interface selector profile (MultiPod-Spine-intProf).

9.    Click Finish to complete creating Spine Profile.

Multi-Pod Setup Configuration

When the original FlexPod with ACI setup was completed, a Pod (1) and the TEP Pool (10.12.0.0/16) was created as part of the setup. In this configuration step, Pod and TEP addresses are defined for the second Site and Multi-Pod is configuration is completed on the APIC. The Pod ID used for second Site is 11 and the TEP Pool used is 10.11.0.0/16.

Setup Pod and TEP Pool for 40GbE Site

1.    Log into the APIC GUI and follow Fabric > Inventory.

2.    In the left pane, right-click Pod Fabric Setup Policy and select Setup Pods.

3.    Enter the Pod ID and TEP Pool for 40GbE Site.

4.    Click Submit.

Create Multi-Pod

1.    Log into the APIC GUI and follow Fabric > Inventory.

2.    In the left pane, right-click Pod Fabric Setup Policy and select Create Multi-Pod.

3.    Provide a Community string (extended:as2-nn4:5:16).

4.    Select Enable Atomic Counters for Multi-pod Mode.

5.    Select Peering Type as Full Mesh (since there are only two sites).

6.    Click + to add a Pod Connection Profile.

7.    Add Pod 1 and provide Dataplane TEP or ETEP shared by Spines at first site as outlined in TEP Interfaces and click Update.

8.    Add Pod 11 and provide Dataplane TEP or ETEP shared by Spines at second site as outlined in TEP Interfaces and click Update.

 

9.    Click + to add Fabric External Routing Profile.

10.  Provide a name (FabExtRoutingProf) and define the subnets used for defining point-to-point connections between Spines and IPN devices. In this guide, all the point-to-point connections and within following two subnets: 10.241.0.0/16 and 10.242.0.0/16. Click Update.

11.  Click Submit to complete the Multi-Pod configuration.

Create Routed Outside for Multi-Pod

1.    Log into the APIC GUI and follow Fabric > Inventory.

2.    In the left pane, right-click Pod Fabric Setup Policy and select Create Routed Outside for Multipod.

3.    Provide the OSPF Area ID as configured on the IPN devices (0.0.0.0).

4.    Select the OSPF Area Type (Regular area).

5.    Click Next.

6.    Click + to add first Spine.

7.    Select the first Spine (Pod-1/Node-211) and add Router ID (Loopback) as shown in Figure 8 and click Update.

8.    Click + to add second Spine.

9.    Select the second Spine (Pod-1/Node-212) and add Router ID (Loopback) as shown in Figure 8 and click Update.

10.  From the drop-down list for OSPF Profile For Sub-Interfaces, select Create OSPF Interface Policy.

11.  Provide a Name for the Policy (P2P).

12.  Select Network Type Point-to-Point.

13.  Check Advertise Subnet and MTU ignore.

14.  Click Submit.

15.  Click + to add Routed Sub-Interfaces.

16.  Add all four interfaces (Path) and their respective IP addresses connecting the first Site’s spines switches to IPN devices (Figure 8).

 

17.  Click Finish.

18.  Browse to Tenants > Infra.

19.  In the left pane, expand the Networking > External Routed Networks and click multipod.

20.  From the External Routed Domain drop-down list on the main page, select MultiPod-L3.

 

21.  Click Submit.

The 10GbE Site spine configuration is now complete. Log into the IPN devices to verify OSPF routing and neighbor relationship

Second Site - Spine Discovery

With the Multi-Pod network configured on first site, the spine switches on the second site should now be visible under the Fabric > Inventory > Fabric Membership.

Setup Pod and Node IDs for the 40GbE Site Spines

1.    Log into the APIC GUI and follow Fabric > Inventory > Fabric Membership.

2.    In the main window, double click and update Pod ID (11), Node ID (1111 and 1112) and Node Names for both the new Spines.

 

Second Site - Spine Configuration

1.    After adding the Spines to the Fabric, the next step is to configure the Spines on the second Site to correctly communicate with the IPN devices on the same Site. The leaf switches from this site will not be visible unless this step is complete.

2.    Log into the APIC GUI and follow Fabric > Access Policies.

3.    From the left pane, expand Switch Policies > Profiles > Spine Profiles.

4.    Select the previously created Spine Profile (Spine-Prof).

5.    In the main window, click + to add additional (second site) spines.

6.    Provide a Name and select the Node IDs for the two spines (1111-1112).

7.    Click Update.

Create Routed Outside for Spines at the Second Site

1.    Log into the APIC GUI and follow Fabric > Inventory.

2.    In the left pane, right-click Pod Fabric Setup Policy and select Create Routed Outside for A Pod.

3.    Click + to add first Spine.

4.    Select the first Spine (Pod-11/Node-1111) and add Router ID (Loopback) as shown in Figure 8 and click Update.

5.    Click + to add second Spine.

6.    Select the second Spine (Pod-11/Node-1112) and add Router ID (Loopback) as shown in Figure 8 and click Update.

7.    Click + to add Routed Sub-Interfaces.

8.    Add all four interfaces (Path) and their respective IP addresses connecting the Spines to IPN devices (Figure 8).

 

9.    Click Submit.

The second site spine configuration is now complete. Log into the IPN devices to verify OSPF routing and neighbor relationship.

Second Site Leaf Discovery

With the Multi-Pod network configured on the second Site, all the Leaf switches on the site should now be visible under the Fabric > Inventory > Fabric Membership. In the main window, double-click and update Pod ID (11), Node ID, and Node Names for the leaf devices.

With the discovery and addition of all the ACI devices, the Multi-Pod portion of the configuration is now complete.

ACI Fabric Setup

This section details the additional steps required during the initial setup of the Cisco ACI Fabric, where

·         Optional Software upgrade of the 40GbE Site devices.

·         Out of band management IPs are assigned to the new devices.

·         BGP route reflectors are configured.

Software Upgrade for 40GbE Site devices (if required)

Repeat the software upgrade process outlined in the document above to upgrade the Leaf and Spine devices in the recently added Site.

Setting Up Out-of-Band Management IP Addresses for New ACI Switches

To set up out-of-band management IP addresses, complete the following steps:

1.    To add out-of-band management interfaces for all the switches in the ACI Fabric, select Tenants > mgmt.

2.    Expand Tenant mgmt on the left. Right-click Node Management Addresses and select Create Static Node Management Addresses.

3.    Enter the node number (or range) for the new switches as defined in the previous steps.

4.    Select the checkbox for Out-of-Band Addresses.

5.    Select default for Out-of-Band Management EPG.

6.    Considering that the IPs will be applied in a consecutive range, enter a starting IP address and netmask in the Out-Of-Band IPV4 Address field.

7.    Enter the out of band management gateway address in the Gateway field.

8.    Click SUBMIT, then click YES.

9.    On the left, expand Node Management Addresses and select Static Node Management Addresses. Verify the mapping of IPs to switching nodes.

10.  On successful completion of this step, direct out-of-band access to the switches is now available using SSH.

BGP Route Reflectors

In this ACI deployment, all the spine switches are set up as BGP route-reflectors to distribute the leaf routes throughout the fabric. To verify and set the BGP Route Reflector, complete the following steps:

1.    Select and expand System > System Settings > BGP Route Reflector.

2.    Verify that a unique Autonomous System Number has been selected for this ACI fabric. If not already defined, use the + sign on the right to add the spine nodes to the list of Route Reflector Nodes. Click Submit to complete configuring the BGP Route Reflector.

3.    To verify or enable the BGP Route Reflector, select and expand Fabric > Fabric Policies > Pods > Policy Groups. Under Policy Groups make sure a policy group has been created and selected.  The BGP Route Reflector Policy field should show “default.”

4.    If a Policy Group has not been created, on the left, right-click Policy Groups under Pods and select Create Pod Policy Group. In the Create Pod Policy Group window, name the Policy Group ppg-Pod1. Select the default BGP Route Reflector Policy.

5.    Click Submit to complete creating the Policy Group.

6.    On the left expand Profiles under Pods and select Pod Profile default > default.

7.    Verify that the ppg-Pod1 or the Fabric Policy Group identified above is selected. If the Fabric Policy Group is not selected, view the drop-down list to select it and click Submit.

Fabric Access Policy Setup

This section details the steps to create various access policies creating parameters for CDP, LLDP, LACP, etc. These policies are used during vPC and VM domain creation.  In an existing fabric, these policies may already exist. To define fabric access policies, complete the following steps:

1.    Log into the APIC AGUI.

2.    In the APIC UI, select and expand Fabric > Access Policies > Policies > Interfaces.

Create Link Level Policies

This procedure will create link level policies for setting up the 1Gbps, 10Gbps, and 40Gbps link speeds. To create the link level policies, complete the following steps:

1.    In the left pane, right-click Link Level and select Create Link Level Policy.

2.    Name the policy as “1Gbps-Auto” and select the 1Gbps Speed.

3.    Click Submit to complete creating the policy.

4.    In the left pane, right-click Link Level and select Create Link Level Policy.

5.    Name the policy “10Gbps-Auto” and select the 10Gbps Speed.

6.    Click Submit to complete creating the policy.

7.    In the left pane, right-click Link Level and select Create Link Level Policy.

8.    Name the policy “40Gbps-Auto” and select the 40Gbps Speed.

9.    Click Submit to complete creating the policy.

Create CDP Policy

This procedure creates policies to enable or disable CDP on a link. To create a CDP policy, complete the following steps:

1.    In the left pane, right-click CDP interface and select Create CDP Interface Policy.

2.    Name the policy as “CDP-Enabled” and enable the Admin State.

3.    Click Submit to complete creating the policy.

4.    In the left pane, right-click the CDP Interface and select Create CDP Interface Policy.

5.    Name the policy “CDP-Disabled” and disable the Admin State.

6.    Click Submit to complete creating the policy.

Create LLDP Interface Policies

This procedure will create policies to enable or disable LLDP on a link. To create an LLDP Interface policy, complete the following steps:

1.    In the left pane, right-click LLDP interface and select Create LLDP interface Policy.

2.    Name the policy as “LLDP-Enabled” and enable both Transmit State and Receive State.

3.    Click Submit to complete creating the policy.

4.    In the left, right-click the LLDP interface and select Create LLDP Interface Policy.

5.    Name the policy as “LLDP-Disabled” and disable both the Transmit State and Receive State.

6.    Click Submit to complete creating the policy.

Create Port Channel Policy

This procedure will create policies to set LACP active mode configuration and the MAC-Pinning mode configuration. To create the Port Channel policy, complete the following steps:

1.    In the left pane, right-click Port Channel and select Create Port Channel Policy.

2.    Name the policy as “LACP-Active” and select LACP Active for the Mode.  Do not change any of the other values.

3.    Click Submit to complete creating the policy.

4.    In the left pane, right-click Port Channel and select Create Port Channel Policy.

5.    Name the policy as “MAC-Pinning” and select MAC Pinning-Physical-NIC-load for the Mode.  Do not change any of the other values.

6.    Click Submit to complete creating the policy.

Create BPDU Filter/Guard Policies

This procedure will create policies to enable or disable BPDU filter and guard. To create a BPDU filter/Guard policy, complete the following steps:

1.    In the left pane, right-click Spanning Tree Interface and select Create Spanning Tree Interface Policy.

2.    Name the policy as “BPDU-FG-Enabled” and select both the BPDU filter enabled and BPDU Guard enabled Interface Controls. 

3.    Click Submit to complete creating the policy.

4.    In the left pane, right-click Spanning Tree Interface and select Create Spanning Tree Interface Policy.

5.    Name the policy as “BPDU-FG-Disabled” and make sure both the BPDU filter enabled and BPDU Guard enabled Interface Controls are cleared. 

6.    Click Submit to complete creating the policy.

Create VLAN Scope Policy

To create policies to enable port local scope for all the VLANs, complete the following steps:

1.    In the left pane, right-click the L2 Interface and select Create L2 Interface Policy.

2.    Name the policy as VLAN-Scope-Port-Local and make sure Port Local scope is selected for VLAN Scope. Do not change any of the other values.

3.    Click Submit to complete creating the policy.

4.    Repeat above steps to create a VLAN-Scope-Global Policy and make sure Global scope is selected for VLAN Scope. Do not change any of the other values. See below.

Create Firewall Policy

When using the unified configuration wizard “Creating Interface and Switch Profiles” and a vCenter Domain Profile Using the GUI, Cisco APIC applies the firewall policy in the mode you chose: Learning, Enabled, or Disabled. Since this CVD does not cover distributed firewall configuration, a firewall policy is added to disable the distributed firewall. Complete the following steps:

1.    In the left pane, right-click Firewall and select Create Firewall Policy.

2.    Name the policy “Firewall-Disabled” and select Disabled for Mode. Do not change any of the other values.

3.    Click Submit to complete creating the policy.

Create Virtual Port Channels (VPCs)

This section details the steps to setup VPCs for connectivity to the Cisco UCS and the NetApp Storage.

VPC – UCS Fabric Interconnects

To setup VPCs for connectivity to the UCS Fabric Interconnects at the two sites, complete the following steps:

Figure 9       Cisco UCS Connectivity – POD 11

Table 4    VLANs for Cisco UCS Hosts

1.    In the APIC GUI, select Fabric > Access Policies > Quick Start.

2.    In the right pane, select Configure and interface, PC and VPC.

3.    In the configuration window, configure a VPC domain between the 93180 leaf switches by clicking + under VPC Switch Pairs.

4.    Enter a VPC Domain ID (21 in this example).

5.    From the drop-down list, select 93180 Switch A and 93180 Switch B IDs to select the two leaf switches.

6.    Click Save.

7.    Click the + under Configured Switch Interfaces.

8.    Select the two Nexus 93180 switches under the Switches drop-down list.

9.    Click  to add switch interfaces.

10.  Configure various fields by utilizing the policies created earlier as shown in the figure below. In this screenshot, port 1/21 on both leaf switches is connected to UCS Fabric Interconnect A using 40Gbps links.

11.  Click Save.

12.  Click Save again to finish the configuring switch interfaces.

13.  Click Submit.

14.  From the right pane, select Configure and interface, PC and VPC.

15.  Select the switches configured in the last step under Configured Switch Interfaces.

16.  Click  on the right to add switch interfaces.

17.  Configure various fields as shown in the screenshot. In this screenshot, port 1/24 on both leaf switches is connected to UCS Fabric Interconnect B using 40Gbps links. Instead of creating a new domain, the External Bridged Domain created in the last step (UCS) is also used for FI-B as shown below.

18.  Click Save.

19.  Click Save again to finish the configuring switch interfaces.

20.  Click Submit.

21.  Repeat this procedure to configure the UCS domain in POD 1. For a uniform configuration, the same External Bridge Domain (UCS) will be utilized for all the Fabric Interconnects. Use the switch and port information in Figure 10 as a reference.

Figure 10     Cisco UCS Connectivity – POD 1

VPC – NetApp AFF Cluster

Complete the following steps to setup VPCs for connectivity to the NetApp AFF storage controllers. The VLANs configured for NetApp are shown in Table 5.

Figure 11     NetApp AFF A700 Connectivity – POD 1

Table 5    VLANs for Storage

1.    In the APIC GUI, select Fabric > Access Policies > Quick Start.

2.    In the right pane, click to Configure and interface, PC and VPC.

3.    Under Configured Switch Interfaces, select the paired Nexus 93180 switches configured in the previous step.

4.    Click  on the right to add switch interfaces.

5.    Configure various fields as shown in the screenshot below. In this screen shot, port 1/23 on both leaf switches is connected to Storage Controller 1 using 40Gbps links.

6.    Click Save.

7.    Click Save again to finish the configuring switch interfaces.

8.    Click Submit.

9.    From the right pane, select Configure and interface, PC and VPC.

10.  Select the previously configured paired Nexus 93180 switches.

11.  Click  to add switch interfaces.

12.  Configure various fields as shown in the screenshot below. In this screenshot, port 1/24 on both leaf switches is connected to Storage Controller 2 using 40Gbps links. Instead of creating a new domain, the Bare Metal Device created in the previous step (NetApp-AFF) is attached to the storage controller 2 as shown below.

13.  Click Save.

14.  Click Save again to finish the configuring switch interfaces.

15.  Click Submit.

16.  Repeat this procedure to configure the NetApp AFF storage controller on second site (POD 1). For a uniform configuration, the same Bare Metal Domain (NetApp-AFF) will be utilized for all the Storage Controllers. Use the switch and port information in Figure 12 as a reference.

Figure 12     NetApp AFF 700 Connectivity – POD 1

Configuring Common Tenant for In-Band Management Access

This section details the steps to setup in-band management access in the Tenant common. This design will allow all the other tenant EPGs to access the common management segment for Core Services VMs such as AD/DNS.

1.    In the APIC GUI, select Tenants > common.

2.    In the left pane, expand Tenant common and Networking.

Create VRFs

To create VRFs, complete the following steps:

1.    Right-click VRFs and select Create VRF.

2.    Enter “FPV-Common-IB” as the name of the VRF.

3.    Uncheck Create A Bridge Domain.

4.    Click Finish.

Create Bridge Domain

To create the incoming IB-MGMT Bridge domain for Core-Services, complete the following steps:

1.    In the left pane, expand Tenant common and Networking.

2.    Right-click Bridge Domains and select Create Bridge Domain.

3.    Name the Bridge Domain as “FPV-Common-IB”.

4.    Select “FPV-Common-IB” from the VRF drop-down list.

5.    Select Custom under Forwarding and enable the flooding as shown in the screenshot below.

6.    Click Next.

7.    Under L3 Configurations, make sure Limit IP Learning to Subnet is selected.  Click Next.

8.    No changes are needed for Advanced/Troubleshooting. Click Finish.

Create Application Profile

To create an Application profile, complete the following steps:

1.    In the left pane, expand Tenant common and Application Profiles.

2.    Right-click the Application Profiles and select Create Application Profiles.

3.    Enter “FPV-Common-IB-Mgmt” as the name of the application profile.

4.    Click Submit.

Create EPG

To create the FPV-Common-Services EPG, complete the following steps:

1.    Expand the “FPV-Common-IB-Mgmt” Application Profile and right-click Application EPGs.

2.    Select Create Application EPG.

3.    Enter “FPV-Common-Services” as the name of the EPG.

4.    Select “FPV-Common-IB” from the drop-down list for Bridge Domain.

5.     Click Finish.

Set Domains for the EPG

To set Domains, complete the following steps:

1.    Expand the newly create EPG and click Domains.

2.    Right-click Domains and select Add L2 External Domain Association.

3.    Select the “FPV-UCS-L2” as the L2 External Domain Profile. This domain was created when configuring the VPC for Cisco UCS.

4.     Click Submit.

5.    Right-click Domains and select Add Physical Domain Association.

6.    Select the NetApp as the Physical Domain Profile. This domain was created when configuring a VPC for NetApp controllers.

7.    Click Submit.

Set Static Ports for the EPG

To set Static Ports, complete the following steps:

1.    In the left pane, right-click Static Ports.

2.    Select Deploy Static EPG on PC, VPC, or Interface.

3.    In the next screen, for the Path Type, select Virtual Port Channel and from the Path drop-down list, select the VPC for UCS Fabric Interconnect A configured earlier.

4.    Enter the Common-Services VLAN under Port Encap.

5.    Change Deployment Immediacy to Immediate.

6.    Leave the Mode set to Trunk.

7.    Click Submit.

8.    In the left pane, right-click Static Ports.

9.    Select Deploy Static EPG on PC, VPC, or Interface.

10.  In the next screen, for the Path Type, select Virtual Port Channel and from the Path drop-down list, select the VPC for UCS Fabric Interconnect B configured earlier.

11.  Enter the Common-Services VLAN under Port Encap.

12.  Change Deployment Immediacy to Immediate.

13.  Leave the Mode set to Trunk.

14.  Click Submit.

15.  In the left pane, right-click Static Ports.

16.  Select Deploy Static EPG on PC, VPC, or Interface.

17.  In the next screen, for the Path Type, select Virtual Port Channel and from the Path drop-down list, select the first VPC for the AFF A700 configured earlier.

18.  Enter the Common-Services VLAN under Port Encap.

19.  Change Deployment Immediacy to Immediate.

20.  Leave the Mode set to Trunk.

21.  Click Submit.

22.  In the left pane, right-click Static Ports.

23.  Select Deploy Static EPG on PC, VPC, or Interface.

24.  In the next screen, for the Path Type, select Virtual Port Channel and from the Path drop-down list, select the second VPC for the AFF A700configured earlier.

25.  Enter the Common-Services VLAN under Port Encap.

26.  Change Deployment Immediacy to Immediate.

27.  Leave the Mode set to Trunk.

28.  Click Submit.

Create EPG Subnet

A subnet gateway for this Common Services EPG provides Layer 3 connectivity to Tenant subnets. To create an EPG Subnet, complete the following steps:

1.    In the left pane, right-click Subnets and select Create EPG Subnet.

2.    In CIDR notation, enter an IP address and subnet mask to serve as the gateway within the ACI fabric for routing between the Common Services subnet and Tenant subnets. In this lab validation, 10.2.156.254/24 will be used for the EPG subnet gateway.  Set the Scope of the subnet to Shared between VRFs, and Advertised Externally.

3.    Click Submit to create the Subnet.

Create Provided Contract

To create Provided Contract, complete the following steps:

1.    In the left pane, within the FPV Common-Services EPG, right-click Contracts and select Add Provided Contract.

2.    In the Add Provided Contract window, select Create Contract from the drop-down list.

3.    Name the Contract FPV-Allow-Common-Services.

4.    Set the Scope to Global.

5.    Click + to add a Subject to the Contract.

6.    Name the subject “Allow-All-Traffic”.

7.    Click + under Filter Chain to add a Filter.

8.    From the drop-down Name list, select common/default.

9.    In the Create Contract Subject window, click Update to add the Filter Chain to the Contract Subject.

10.  Click OK to add the Contract Subject.

11.  Click Submit to finish creating the Contract.

12.  Click Submit to finish adding a Provided Contract.

Create Security Filters in Tenant Common

Security filters defined in the “common” tenant can be used by contracts in all the tenants of the ACI fabric. To create Security Filters for NFSv3 with NetApp Storage and for iSCSI, complete the following steps.

This section can also be used to set up other filters necessary in any environment. Customers can choose to define these filters within the appropriate tenants if they do not want to make these filters available system wide.

1.    In the APIC GUI, at the top select Tenants > common.

2.    On the left, expand Tenant common > Contracts > Filters.

3.    Right-click Filters and select Create Filter.

4.    Name the filter “Allow-All”.

5.    Click the + sign to add an Entry to the Filter.

6.    Name the Entry “Allow-All” and select EtherType IP.

7.    Leave the IP Protocol set at Unspecified.

8.    Click Update to add the Entry.

9.    Click Submit to complete adding the Filter.

10.  Right-click Filters and select Create Filter.

11.  Name the filter “NFS”.

12.  Click the + sign to add an Entry to the Filter.

13.  Name the Entry “tcp-111” and select EtherType IP.

14.  Select the tcp IP Protocol and enter 111 for From and To under the Destination Port / Range by backspacing over Unspecified and entering the number.

15.  Click Update to add the Entry.

16.  Click the + sign to add another Entry to the Filter.

17.  Name the Entry “tcp-635” and select EtherType IP.

18.  Select the tcp IP Protocol and enter 635 for From and To under the Destination Port / Range by backspacing over Unspecified and entering the number.

19.  Click Update to add the Entry.

20.  Click the + sign to add another Entry to the Filter.

21.  Name the Entry “tcp-2049” and select EtherType IP.

22.  Select the tcp IP Protocol and enter 2049 for From and To under the Destination Port / Range by backspacing over Unspecified and entering the number.

23.  Click Update to add the Entry.

24.   Click Submit to complete adding the Filter.

25.  Right-click Filters and select Create Filter.

26.  Name the filter “iSCSI”.

27.  Click the + sign to add an Entry to the Filter.

28.  Name the Entry “iSCSI” and select EtherType IP.

29.  Select the tcp IP Protocol and enter 3260 for From and To under the Destination Port / Range by backspacing over Unspecified and entering the number.

30.  Click Update to add the Entry.

31.  Click Submit to complete adding the Filter.

Deploy FPV-Foundation Tenant

This section details the steps for creating the FPV-Foundation Tenant in the ACI Fabric.  This tenant will host infrastructure connectivity for the compute (VMware ESXi on UCS nodes) and the storage environments. To deploy the FPV-Foundation Tenant, complete the following steps:

1.    In the APIC GUI, select Tenants > Add Tenant.

2.    Name the Tenant “FPV-Foundation”.

3.    For the VRF Name, enter “FPV-Foundation”.  Keep the check box “Take me to this tenant when I click finish” checked.

4.     Click Submit to finish creating the Tenant.

Create Application Profile for Host Connectivity and MetroCluster

To create an application profile for host connectivity, complete the following steps:

1.    In the left pane, under the Tenant FPV-Foundation, right-click Application Profiles and select Create Application Profile.

2.    Name the Profile “Host-Conn” and click Submit to complete adding the Application Profile.

3.    Repeat these two steps to create an Application Profile named “MetroCluster”.

The following EPGs and the corresponding mappings will be created under these application profiles.

Table 6    EPGs and Mappings for Application Profile Host-Connectivity

 

Table 7    EPG and Mappings for Application Profile MetroCluster

Create Bridge Domains and EPGs

To create bridge domains and EPGs, complete the following steps:

1.    For each row in the tables above, in the left pane, under Tenant FPV-Foundation, expand Networking > Bridge Domains.

2.    Right-click Bridge Domains and select Create Bridge Domain.

3.    Name the Bridge Domain {bd-FPV-Foundation}.

4.    Select FPV-Foundation from the VRF drop-down list.

5.    Select Custom under Forwarding and enable flooding.

6.    Click Next.

7.    Under L3 Configurations, make sure Limit IP Learning to Subnet is selected and select EP Move Detection Mode – GARP based detection.  Select Next.

8.    No changes are needed for Advanced/Troubleshooting.

9.    Click Finish to finish creating the Bridge Domain.

10.  Repeat the above steps to add all Bridge Domains listed in Table 6 and Table 7.

11.  In the left pane, expand Application Profiles > Host-Conn. Right-click Application EPGs and select Create Application EPG.

12.  Name the EPG {vMotion}.

13.  From the Bridge Domain drop-down list, select the Bridge Domain from the table.

14.  Click Finish to complete creating the EPG.

15.  In the left pane, expand the Application EPGs and EPG {vMotion}.

16.  Right-click Domains and select Add L2 External Domain Association.

17.  From the drop-down list, select the previously defined {FPV-UCS-L2} L2 External Domain Profile.

18.   Click Submit to complete the L2 External Domain Association.

19.  Repeat the Domain Association steps (6-9) to add appropriate EPG specific domains from Table 6 and Table 7.

20.  Right-click Static Ports and select Deploy EPG on PC, VPC, or Interface.

21.  In the Deploy Static EPG on PC, VPC, Or Interface Window, select the Virtual Port Channel Path Type.

22.  From the drop-down list, select the appropriate VPCs.

23.  Enter VLAN from Table 3. {3000} for Port Encap.

24.  Select Immediate for Deployment Immediacy and for Mode select Trunk.

25.   Click Submit to complete adding the Static Path Mapping.

26.  Repeat steps these steps for each EPG listed in Table 6 and Table 7.

ACI Shared Layer 3 Out Setup

This section describes the procedure for deploying the ACI Shared Layer 3 Out.  The FlexPod Datacenter with Cisco ACI Multi-Pod, NetApp MetroCluster IP and VMware vSphere 6.7 solution utilizes two separate geographically dispersed datacenters where each datacenter leverages its own connection to the external network domain for optimal routing. The connection is defined in the tenant common and provides a contract to be consumed by all the application EPGs that require access to non-ACI networks. Tenant network routes can be shared with the Nexus 7004 switches using OSPF and external routes from the Nexus 7000s are shared with the tenant.

This section provides a detailed procedure for setting up the Shared Layer 3 Out in Tenant common to existing customer Nexus 7000 routers using sub-interfaces and VRF aware OSPF.  Some highlights of this connectivity are:

·         A new bridge domain and associated VRF is configured in Tenant common for external connectivity.

·         The shared Layer 3 Out created in Tenant common “provides” an external connectivity contract that can be “consumed” from any tenant.

·         Routes to tenant EPG subnets connected by contract are shared across VRFs with the Nexus 7000 core routers using OSPF.

·         The Nexus 7000s’ default gateway is shared with the ACI fabric using OSPF.

·         Each of the two Nexus 7000s is connected to each of the two Nexus 9000 leaf switches.

·         Sub-interfaces are configured and used for external connectivity.

·         The Nexus 7000s are configured to originate and send a default route to the Nexus 9000 leaf switches.

Configuring the Nexus 7000s for ACI Connectivity (Sample)

The following configuration is a sample from the virtual device contexts (VDCs) from four Nexus 7004s.  Interfaces and a default route from the four Nexus 7000s also need to be set up, but is not shown here because that configuration is customer specific.

Figure 13     POD11 - ACI Shared Layer 3 Out Connectivity Details

Nexus POD11-7004-1

Nexus POD11-7004-2

 

Figure 14     POD1 - ACI Shared Layer 3 Out Connectivity Details

Nexus POD1-7004-1

Nexus POD1-7004-2

Configuring ACI Shared Layer 3 Out

1.    Log into Cisco APIC. At the top, select Fabric > Access Policies.

2.    On the left, expand Physical and External Domains.

3.    Right-click External Routed Domains and select Create Layer 3 Domain.

4.    Name the Domain SharedL3Out.

5.    Use the Associated Attachable Entity Profile drop-down list to select Create Attachable Entity Profile.

6.    Name the Profile aep-Shared-L3Out and click Next.

7.    Click Finish to continue without specifying interfaces.

8.    Back in the Create Layer 3 Domain window, use the VLAN Pool drop-down list to select Create VLAN Pool.

9.    Name the VLAN Pool L3Out-vlans and select Static Allocation.

10.  Click the + sign to add and Encap Block.

11.  In the Create Ranges window, enter the From and To VLAN IDs for the Shared-L3-Out VLAN range (301-308).  Select Static Allocation.

12.  Click OK to complete adding the VLAN range.

13.  Click Submit to complete creating the VLAN Pool.

14.  Click Submit to complete creating the Layer 3 Domain.

15.  At the top, select Fabric > Access Policies.

16.  On the left, select Quick Start.  Under Steps, select Configure an interface, PC, or VPC.

17.  In the center pane, click the green plus sign to select switches.

18.  Using the Switches pull-down, select the first leaf switch connected to the Nexus 7000.

19.  Click the green plus sign to configure switch interfaces.

20.  Next to interfaces, enter the port identifiers for the port connected to the Nexus 7000 and used for Shared-L3-Out.  Fill in the policies, Attached Device Type, and External Route Domain as covered below.

21.  On the lower right, click Save. Click Save again and then click Submit.

22.  At the top, select Tenants > common.

23.  On the left, expand Tenant common and Networking.

24.  Right-click VRFs and select Create VRF.

25.  Name the VRF “vrf-common-outside”.  Select default for both the End Point Retention Policy and Monitoring Policy.  Un-select the Create A Bridge Domain check box.

26.  Click Finish to complete creating the VRF.

27.  On the left, right-click External Routed Networks and select Create Routed Outside.

28.  Name the Routed Outside N7K-SharedL3Out.

29.  Select the checkbox next to OSPF.

30.  Enter 0.0.0.10 (configured in the Nexus 7000s) as the OSPF Area ID.

31.  Using the VRF drop-down list, select vrf-common-outside.

32.  Using the External Routed Domain drop-down list, select SharedL3Out.

33.  Click the + sign to the right of Nodes and Interfaces Protocol Profiles to add a Node Profile.

34.  Name the Node Profile Nodes-1101-1102 for the Nexus 9000 Leaf Switches.

35.  Click the + sign to the right of Nodes to add a Node.

36.  In the select Node window, select Leaf switch 1101.

37.  Provide a Router ID IP address that will also be used as the Loopback Address (10.251.255.1).

38.  Click OK to complete selecting the Node.

39.  Click the + sign to the right of Nodes to add a Node.

40.  In the select Node window, select Leaf switch 1102.

41.  Provide a Router ID IP address that will also be used as the Loopback Address (10.251.255.2).

42.  Click OK to complete selecting the Node.

43.  Click the + sign to the right of OSPF Interface Profiles to create an OSPF Interface Profile.

44.  Name the profile “Node-1101-1102-IntfPol”.

45.  Click Next.

46.  Using the OSPF Policy drop-down list, select Create OSPF Interface Policy.

47.  Name the policy “osfp-Nexus-7K”.

48.  Select the Point-to-Point Network Type.

49.  Select the Advertise subnet and MTU ignore Interface Controls.

50.  Click SUBMIT to complete creating the policy.

51.  Click Next.

52.  Select Routed Sub-Interface under Interfaces.

53.  Click the + sign to the right of Routed Sub-Interfaces to add a routed sub-interface.

54.  In the Select Routed Sub-Interface window, select the interface on Node 1101 that is connected to Nexus 7000 as shown in Figure 13.

55.  Enter VLAN <305> for Encap.

56.  Enter the IPv4 Primary Address (10.251.231.1/30)

57.  Leave the MTU set to inherit.

58.  Click OK to complete creating the routed sub-interface.

59.  Repeat steps to add all the remaining 3 sub interfaces as shown in Figure 13.

60.  Click OK to complete creating the Node Interface Profile.

61.  Click OK to complete creating the Node Profile.

62.  Click + under Nodes and Interface Protocol Profiles and repeat steps 33-61 to add the Nodes 201 and 202 at POD1. Use Figure 14 as a reference for interface, VLAN and IP address information.

63.  Click Next.

64.  Click the + sign under External EPG Networks to create and External EPG Network.

65.  Name the External Network Default-Route.

66.  Click the + sign to add a Subnet.

67.  Enter 0.0.0.0/0 as the IP Address.  Select the checkboxes for External Subnets for the External EPG, Shared Route Control Subnet, and Shared Security Import Subnet.

68.  Click OK to complete creating the subnet.

69.  Click OK to complete creating the external network.

70.  Click Finish to complete creating the N7K-SharedL3Out.

71.  On the left, right-click Contracts and select Create Contract.

72.  Name the contract “Allow-Shared-L3Out”.

73.  Select the Global Scope to allow the contract to be consumed from all tenants.

74.  Click the + sign to the right of Subjects to add a contract subject.

75.  Name the subject “Allow-All”.

76.  Click the + sign to the right of Filters to add a filter.

77.  Use the drop-down list to select the Allow-All filter from Tenant common.

78.  Click Update.

79.  Click OK to complete creating the contract subject.

80.  Click Submit to complete creating the contract.

81.  On the left, expand Tenant common, Networking, External Routed Networks, Shared-L3-Out, and Networks.  Select Default-Route.

82.  On the right, under Policy, select Contracts.

83.  Click the + sign to the right of Provided Contracts to add a Provided Contract.

84.  Select the common/Allow-Shared-L3-Out contract and click Update.

Site Preparation

The Installation and Setup Instructions for the NetApp AFF A700 systems are available here: https://library.netapp.com/ecm/ecm_download_file/ECMP12522110

The Disk Shelf Documentation to plan the Disk Shelf connectivity and initial configuration is available here: https://mysupport.netapp.com/documentation/productlibrary/index.html?productID=30147

Hardware Universe

Refer to the NetApp Hardware Universe to plan the physical location of the Storage Arrays in both data centers, available here:

·         https://hwu.netapp.com/

The NetApp Hardware Universe (HWU) application provides supported hardware and software components for any specific ONTAP version. It provides configuration information for all the NetApp storage appliances currently supported by ONTAP software. It also provides a table of component compatibilities.  

Interoperability Matrix Tool

Refer to NetApp IMT to get a list of hardware, software, and firmware that are interoperable in a MetroCluster solution, available here:

·         http://mysupport.netapp.com/matrix/

It is recommended to check IMT prior to first time deployment and subsequent upgrades and refreshes.

Cluster Interconnect Switch Configuration

In a MetroCluster IP configuration the network switches will be configured to function as the Cluster Interconnects and also as the backend MetroCluster IP network between the two sites.

Download the RCF

To download the Reference Configuration Files for the IP network switches, complete the following steps:

1.    Browse to the URL: https://mysupport.netapp.com/NOW/download/software/metrocluster_ip/rcfs/download.shtml

2.    Download the RCF zip file that corresponds to the switch model that you intend to use, Cisco Nexus 3132Q-V in this case.

Initialize the Switches

To Initialize the IP Network Switches in both the sites, complete the following steps:

1.    Establish a console connection to the switch.

2.    Erase the existing configuration:

3.    Reload the switch:

4.    Setup the switch once it enters the configuration wizard:

Upgrade NX-OS on Switches

To upgrade the NX-OS to the latest version on the switches in both the sites, refer to the Cluster Network Compatibility Matrix: https://mysupport.netapp.com/NOW/download/software/cm_switches/

Table 8    Switch to RCF File Mapping

1.    Download the supported NX-OS software from https://software.cisco.com/download/home/283734368

2.    Copy the NX-OS software to the switch using any supported Transfer Protocol, example:

3.    Refer to Table 1  and copy the appropriate RCF file to the switch, example:

4.    Verify that the RCF and NX-OS files are present on the switch’s bootflash directory:

5.    Install the switch software:

6.    Merge the RCF with the running-config on the switch:

7.    Save the running config on the switch:

ONTAP 9.4 Deployment

Disk Assignment

With ONTAP 9.4, MetroCluster IP configurations support new installations with AFF systems using Advanced Disk Partitioning (ADP). It is recommended to have a minimum of eight SAS disk shelves (four shelves at each site) to allow automatic disk ownership on a per-shelf basis. If the recommendation cannot be met, a minimum of four disk shelves are required (two shelves at each site) for an initial deployment and the ownership must be set manually.

For new systems, the disk and partition assignments are configured in the factory. For existing systems that are being repurposed for MetroCluster IP, the disk and partition auto-assign can be configured in systems that meet the minimum recommendation of four shelves per site or total numbers of shelves need to be a multiple of four.

If the disk shelf count does not meet either criteria, a manual assignment should be performed.

To configure the partitions and ownership manually, complete the following steps:

1.    Delete all the existing partitions on the Nodes and remove ownership of all the drives.

2.    Run “set-defaults” on the LOADER prompt.

3.    From the Boot Menu select option 4.

4.    From the Maintenance Mode, delete the root aggregate and remove ownership of all the partitions.

5.    From the Maintenance Mode Assign Disk and Partition ownership to the Nodes.

6.    At the LOADER prompt set the root aggregate disk limit using the argument “setenv root-configuration "-d # -p # -s #"

7.    From the Boot Menu select option 4.

For more information, see: MetroCluster IP Installation and Configuration Guide.

Cluster Setup

Cluster Create

Perform the following operation on both clusters of the MetroCluster configuration.

1.    Establish a console session to the Storage Node and complete the following steps:

2.    Enter the cluster name.

3.    Enter the cluster base license key.

4.    Provide any additional license keys that will be required for this solution.

5.    Configure the cluster management interface.

6.    Configure the DNS.

7.    Configure storage failover.

8.    Enter the controller’s physical location.

Cluster Join

Perform this operation on both clusters in the MetroCluster configuration.

1.    Establish a console session to the storage node and complete the following steps.

2.    Initiate the Cluster Join process.

3.    Join the existing cluster.

Verify HA Config State of Controllers and Chassis

To make sure that the HA state of the controllers and chassis is configured for MetroCluster, complete the following steps:

1.    From the Loader prompt, enter the maintenance mode.

2.    Verify the HA config settings from the maintenance mode.

3.    If the displayed system state of the controller is not mccip, set the HA state for the controller.

4.    If the displayed system state of the chassis is not mccip, set the HA state for the chassis.

ONTAP Base Configuration

Disable Automatic Drive Assignment

Run the following command on all the nodes in the MetroCluster, only if your configuration has less than four storage shelves per site or if you need to manually assign drives:

Verify Disk Assignment of Pool 0 Drives

Make sure that the local disks on each site are visible and assigned correctly:

Storage Failover HA Configuration

To configure HA on each storage node in the MetroCluster, complete the following steps:

1.    Establish an SSH session to the cluster management IP <cluster_mgmt>.

2.    Query the status of HA.

3.    Configure HA, if it not done already.

Service Processor Configuration

To configure the service processors on each storage node in the MetroCluster, complete the following steps.

1.    Establish an SSH session to the cluster management IP <cluster_mgmt>.

2.    List the status of the service processors in the cluster.

3.    Configure the service processor IP address.

Zero All Spare Disks

Zero all the spare disks in the cluster by running the following command in both the clusters:

Set Auto-Revert on Cluster Management

To set the auto-revert parameter on the cluster management interface, complete the following step on both clusters:

Configure Network Time Protocol

Configure the time synchronization on the MetroCluster by completing the following steps on both clusters.

1.    Set the time zone for the cluster.

2.    Set the date for the cluster.

3.    Configure the Network Time Protocol (NTP) servers for the cluster.

Configure Simple Network Management Protocol

To configure the Simple Network Management Protocol (SNMP), complete the following steps in both the clusters:

1.    Configure basic SNMP information, such as the location and contact. When polled, this information is visible as the sysLocation and sysContact variables in SNMP.

2.    Configure SNMP traps to send to remote hosts such as a DFM server or another fault management system.

Configure SNMPv1 Access

To configure SNMPv1 access, set the shared, secret plaintext password (called a community):

ONTAP Network Configuration

Enable Cisco Discovery Protocol

To enable the Cisco Discovery Protocol (CDP) on the NetApp storage controllers, run the following command to enable CDP on all nodes in the MetroCluster:

Disable Flow Control

NetApp recommends disabling flow control on all 40GbE and UTA2 ports that are connected to external devices. To disable flow control, run the following commands on each node in the MetroCluster.

Modify Port Speed to 40Gbps

Set the administrative speed of the cluster and data ports on the storage nodes to 40Gbp.

Remove Required Ports from Default Broadcast Domain

All interfaces of the storage nodes are by default placed in the Default broadcast domain. To use these interfaces, they need to be removed from the Default domain.

Create Jumbo Frame MTU Broadcast Domains

If Jumbo Frames are required for the solution, run the following commands to configure them in the broadcast domains in both clusters.

Create Broadcast Domain for In-Band Management Traffic

Create a broadcast domain for the in-band management traffic and set the MTU to 1500.

Create Interface Groups

Create LACP interface groups for the 40GbE data interfaces on all the nodes in the MetroCluster.

To configure the interface groups on both nodes of the clusters, run the following commands:

Configure Jumbo Frames on the Interface Groups

To configure jumbo frames on the interface groups on both nodes of the clusters, run the following commands:

Create VLANs

Create the VLANs that are required for this solution and add the VLAN interfaces to the respective broadcast domains.

IB-MGMT VLAN

 

Inter-Cluster VLAN

 

NFS VLAN

 

iSCSI-A VLAN

 

 

iSCSI-B VLAN

MetroCluster Base Configuration

Create Intercluster LIFs

To create the intercluster LIFs on both the nodes in both the clusters, run the following commands:

Record the Intercluster LIF IP Addresses

Make a note of the Intercluster LIF IP addresses from both the clusters, which will be used to set up the cluster-peer relationship.

Disable Automatic Drive Assignment

If the deployment has fewer than four storage shelves per site, disable automatic drive assignment on all the nodes.

Create Cluster Peer Relationship

To set up the cluster-peer relationship between the two clusters in the MetroCluster configuration, run the following command:

40GbE Site

 

10GbE Site

Verify whether the cluster-peer relationship is created successfully by running the following commands on either site:

Create DR Group

To create the Disaster Recovery (DR) group relationships between the clusters, complete the following steps.

1.    Verify that the nodes in both clusters are ready for creation of the DR group.

2.    Create the DR group. You can run the following command from either cluster in the MetroCluster configuration.

Create the MetroCluster IP Interfaces

Complete the following steps to configure the MetroCluster IP interfaces for replication of node storage and nonvolatile cache.

Use the following table to record the IP addresses that you plan to use.

1.    If there are at least two shelves connected to each node, ensure that each node has disk auto assignment enabled.

2.    Check whether the nodes are ready for interface creation.

3.    Create two interfaces on each node using the ports e5a and e5b.

 

4.    Verify that the interfaces have been configured.

Establish the MetroCluster Connection

To establish the connection between the two sites using the MetroCluster IP interfaces created previously, complete the following steps:

1.    Verify that the nodes are ready to connect.

2.    Establish the connection.

3.    Verify that the connection is established.

4.    Verify that the iSCSI connections are established.

Verify and Perform Pool 1 Drive Assignment

If drives are being auto assigned, verify pool 1 assignment. If you are using manual assignment, assign drives to pool 1.

For more information, see the MetroCluster IP Installation and Configuration Guide.

Enable Automatic Drive Assignment

If automatic drive assignment was disabled as directed previously, re-enable it on all nodes.

Mirror the Root Aggregates

Mirror the root aggregates on each node in the MetroCluster configuration to provide data protection.

Create Mirrored Data Aggregates

Create mirrored data aggregates in Node_01 and Node_02 at both sites.

Implement the MetroCluster Configuration

Make sure that the following are met before configuring the MetroCluster:

·         At least two non-root mirrored data aggregates on each cluster

·         The ha-config state of the controllers and chassis must be mccip

1.    Run the following command from any one node in the MetroCluster deployment.

2.    Verify the networking status on both sites.

3.    Verify the MetroCluster configuration from both sites.

Check the MetroCluster Configuration

1.    Run the following command from either cluster to check whether the components and relationships are working correctly.

2.    Run the following commands for more detailed results from the most recent metrocluster check run.

Verify Switchover, Healing, and Switchback

Before moving the MetroCluster deployment to production, it is recommended to verify the switchover, healing, and switchback operations.

For detailed operational procedures, see the MetroCluster Management and Disaster Recovery Guide.

ONTAP Logical Configuration

Create Storage Virtual Machines (SVM)

To create an SVM on both the sites, complete the following steps:

1.    Create the SVM.

2.    Remove unused protocols from the SVM.

3.    Add the data aggregates to the aggregate list.

Create NFS Service

To create the NFS service on both sites, complete the following steps:

1.    Enable and run the NFS protocol in the SVM.

2.    Enable the SVM vstorage parameter for the NetApp NFS VAAI plug-in.

Create iSCSI Service

Create the iSCSI service on the SVMs in both sites.

Create Load-Sharing Mirrors of SVM Root Volume

To create load-sharing mirrors of the SVM root volumes in both sites, complete the following steps:

1.    Create a volume on each node to be the load-sharing mirror of the SVM root volume.

40GbE Site

10GbE Site

2.    Create a job schedule at both the sites to update the root volume mirror relationships every 15 minutes.

3.    Create the mirroring relationships.

40GbE Site

 

10GbE Site

4.    Initialize the mirroring relationship on both the sites.

Configure HTTP Access

To configure secure access to the storage controllers, complete the following steps at both sites:

1.    Increase the privilege level to access the certificate commands.

2.    Generally, a self-signed certificate is already in place. Verify the certificate and obtain parameters (for example, <serial- number>) by running the following command:

3.    For each SVM shown, the certificate common name should match the DNS FQDN of the SVM. Delete the default certificates and replace them with either self-signed certificates or certificates from a certificate authority (CA). To delete the default certificates, run the following command:

4.    To generate and install self-signed certificates, run the following commands as one-time commands. Generate a server certificate for the Infra_<site_id> and the cluster SVMs. Use TAB completion to aid in the completion of these commands.

5.    Enable each certificate that was created by using the server-enabled true and client-enabled false parameters. Use TAB completion to aid in the completion of these commands.

6.    Disable HTTP cluster management access.

7.    Revert to the normal admin privilege level and set up the system to allow SVM logs to be available by web.

Configure NFSv3

To configure NFSv3 on the SVM, complete the following steps on both sites:

1.    Create a rule for the infrastructure NFS subnet in the default export policy.

2.    Assign the FlexPod export policy to the infrastructure SVM root volume.

Create Volumes for Infrastructure Data and ESXi Boot

To create volumes, run the following commands on both sites.

40GbE Site

10GbE Site

Create NFS Volumes for Datastore Heartbeats

To create NFS volumes, run the following command on both sites:

40GbE Site

 

10GbE Site

Create Volumes for LUN Heartbeats

To create volumes to host LUNs for datastore heartbeats, run the following commands on both sites:

40GbE Site

10GbE Site

Create ESXi Boot LUNs

To create ESXi boot LUNs, run the following command on both sites.

40GbE Site

10GbE Site

Create LUNs for Datastore Heartbeats

To create LUNs for datastore heartbeats, run the following command on both sites:

40GbE Site

10GbE Site

Schedule Deduplication

On NetApp AFF systems, deduplication is enabled by default. To schedule deduplication, complete the following steps on both sites:

1.    Enable deduplication on the ESXi boot volumes.

2.    Assign a deduplication schedule to run during the weekends.

3.    Create a cron schedule.

4.    Create a policy to always run deduplication.

5.    Enable deduplication on the infrastructure data volumes.

6.    Assign the policy to always run deduplication.

Create iSCSI Data LIFs

Refer to the following table and run the following commands to create the iSCSI Data LIFs on both the nodes at each site.

40GbE Site

10 GbE Site

Create NFS Data LIFs

Refer to the table below and run the following commands to create the NFS data LIFs on both nodes in each site.

40 GbE Site

10 GbE Site

Add Infrastructure SVM Administrator

To add the infrastructure SVM administrator and SVM administration LIF in the out-of-band management network, complete the following steps on both the sites:

1.    Create a LIF for SVM Management

2.    Create a default route to allow the SVM management interface to reach the outside world.

3.    Set a password for the SVM vsadmin user and unlock the user.

Cisco UCS Base Configuration

This section describes the configuration steps for the Cisco UCS 6332-16UP Fabric Interconnects (FI) in a design that will support iSCSI boot to the NetApp AFF through the Cisco ACI Fabric.  Fibre Channel (FC) boot is not covered in this design, as FC traffic would not be supported over the InterPod Network.

Table 9    Lab Validation Infrastructure (FPV-Foundation) Tenant Configuration

Perform Initial Setup

This section provides detailed steps to configure the Cisco Unified Computing System (Cisco UCS) for use in a FlexPod environment. The steps are necessary to provision the Cisco UCS B-Series and C-Series servers and should be followed precisely to avoid improper configuration.

Cisco UCS Fabric Interconnect A

To configure the Cisco UCS for use in a FlexPod environment, complete the following steps:

1.    Connect to the console port on the first Cisco UCS fabric interconnect.

Enter the configuration method: gui

Physical switch Mgmt0 IP address: <ucsa-mgmt-ip>

Physical switch Mgmt0 IPv4 netmask: <ucsa-mgmt-mask>

IPv4 address of the default gateway: <ucsa-mgmt-gateway>

2.    Using a supported web browser, connect to http://<ucsa-mgmt-ip>, accept the security prompts, and click the ‘Express Setup’ link.

3.    Select Initial Setup and click Submit.

4.    Select Enable clustering, Fabric A, and IPv4.

5.    Fill in the Virtual IP Address with the UCS cluster IP.

6.    Completely fill in the System setup section.  For system name, use the overall UCS system name. For the Mgmt IP Address, use <ucsa-mgmt-ip>.

7.    Click Submit.

Cisco UCS Fabric Interconnect B

To configure the second Fabric Interconnect, complete the following steps:

1.    Connect to the console port on the second Cisco UCS fabric interconnect.

Enter the configuration method: gui

Physical switch Mgmt0 IP address: <ucsb-mgmt-ip>

Physical switch Mgmt0 IPv4 netmask: <ucsb-mgmt-mask>

IPv4 address of the default gateway: <ucsb-mgmt-gateway>

2.    Using a supported web browser, connect to http://<ucsb-mgmt-ip>, accept the security prompts, and click the ‘Express Setup’ link.

3.    Under System setup, enter the Admin Password entered above and click Submit.

4.    Enter <ucsb-mgmt-ip> for the Mgmt IP Address and click Submit.

Cisco UCS Setup

Log in to Cisco UCS Manager

To log in to the Cisco Unified Computing System (UCS) environment, complete the following steps:

1.    Open a web browser and navigate to the Cisco UCS fabric interconnect cluster address.

2.    Click the Launch UCS Manager link under HTML to launch Cisco UCS Manager.

3.    If prompted to accept security certificates, accept as necessary.

4.    When prompted, enter admin as the user name and enter the administrative password.

5.    Click Login to log in to Cisco UCS Manager.

Upgrade Cisco UCS Manager Software to Version 3.2(3d)

This document assumes the use of Cisco UCS 3.2(3d) release. To upgrade the Cisco UCS Manager software and the Cisco UCS Fabric Interconnect software to version 3.2(3d), refer to Cisco UCS Manager Install and Upgrade Guides.

Anonymous Reporting

To create anonymous reporting, complete the following step:

1.    In the Anonymous Reporting window, select whether to send anonymous data to Cisco for improving future products. If you select Yes, enter the IP address of your SMTP Server. Click OK.

Configure Cisco UCS Call Home

It is highly recommended by Cisco to configure Call Home in Cisco UCS Manager.  Configuring Call Home will accelerate resolution of support cases. To configure Call Home, complete the following steps:

1.    In Cisco UCS Manager, click the Admin icon on the left.

2.    Select All > Communication Management > Call Home.

3.    Change the State to On.

4.    Fill in all the fields according to your Management preferences and click Save Changes and OK to complete configuring Call Home.

Add Block of IP Addresses for KVM Access

To create a block of IP addresses for in band server Keyboard, Video, Mouse (KVM) access in the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click the LAN icon on the left.

2.    Expand Pools > root > IP Pools.

3.    Right-click IP Pool ext-mgmt and select Create Block of IPv4 Addresses.

4.    Enter the starting IP address of the block, number of IP addresses required, and the subnet mask and gateway information.

5.    Click OK to create the block.

6.    Click OK in the confirmation message.

Synchronize Cisco UCS to NTP

To synchronize the Cisco UCS environment to the NTP servers in the Nexus switches, complete the following steps:

1.    In Cisco UCS Manager, click the Admin icon on the left.

2.    Expand All > Time Zone Management.

3.    Select Timezone.

4.    In the Properties pane, select the appropriate time zone in the Timezone menu.

5.    Click Save Changes, and then click OK.

6.    Click Add NTP Server.

7.    Enter <oob-ntp-ip> and click OK. Click OK on the confirmation.

8.    Repeat if there is a second NTP server to be added.

Edit Policy to Automatically Discover Server Ports

If the UCS Port Auto-Discovery Policy is enabled, server ports will be discovered automatically. To enable the Port Auto-Discovery Policy, complete the following steps:

1.    In Cisco UCS Manager, click the Equipment icon on the left and select Equipment in the second list.

2.    In the right pane, click the Policies tab.

3.    Under Policies, select the Port Auto-Discovery Policy tab.

4.    Under Properties, set Auto Configure Server Port to Enabled.

5.    Click Save Changes.

6.    Click OK.

Edit Chassis Discovery Policy

Setting the discovery policy simplifies the addition of Cisco UCS B-Series chassis and of additional fabric extenders for further Cisco UCS C-Series connectivity. To modify the chassis discovery policy, complete the following steps:

1.    In Cisco UCS Manager, click the Equipment icon on the left and select Equipment in the second list.

2.    In the right pane, click the Policies tab.

3.    Under Global Policies, set the Chassis/FEX Discovery Policy to match the minimum number of uplink ports that are cabled between the chassis or fabric extenders (FEXes) and the fabric interconnects.

4.    Set the Link Grouping Preference to Port Channel. If Backplane Speed Preference appears, leave it set at 40G.  For Backplane Speed Preference, select 40G if you have servers with VIC1340s and Port Expander cards. Otherwise, select 4x10G.

5.    Click Save Changes.

6.    Click OK.

Verify Server and Enable Uplink Ports

To enable server and uplink ports, complete the following steps:

1.    In Cisco UCS Manager, click the Equipment icon on the left.

2.    Expand Equipment > Fabric Interconnects > Fabric Interconnect A (primary) > Fixed Module.

3.    Expand and select Ethernet Ports.

4.    On the right, verify that the ports that are connected to the chassis, Cisco FEX, and direct connect UCS C-Series servers are configured as Server ports. If any Server ports are not configured correctly, right-click them, and select “Configure as Server Port.” Click Yes to confirm server ports and click OK.

5.    Select the ports that are connected to the Cisco Nexus 93180 switches, right-click them, and select Configure as Uplink Port.

6.    Click Yes to confirm uplink ports and click OK.

7.    Select Equipment > Fabric Interconnects > Fabric Interconnect B (subordinate) > Fixed Module.

8.    Expand and select Ethernet Ports.

9.    Repeat the steps above to configure server and uplink ports

Acknowledge Cisco UCS Chassis and FEX

To acknowledge all Cisco UCS chassis and any external 2232 FEX modules, complete the following steps:

1.    In Cisco UCS Manager, click the Equipment icon on the left.

2.    Expand Chassis and select each chassis that is listed.

3.    Right-click each chassis and select Acknowledge Chassis.

4.    Click Yes and then click OK to complete acknowledging the chassis.

5.    If Nexus FEX are part of the configuration, expand Rack Mounts and FEX.

6.    Right-click each FEX that is listed and select Acknowledge FEX.

7.    Click Yes and then click OK to complete acknowledging the FEX.

Create Uplink Port Channels to Cisco Nexus 93180 Switches

To configure the necessary port channels out of the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click the LAN icon on the left.

2.    Under LAN > LAN Cloud, expand the Fabric A tree.

3.    Right-click Port Channels.

4.    Select Create Port Channel.

5.    Enter 119 as the unique ID of the port channel.

6.    Enter Po119-ACI as the name of the port channel.

7.    Click Next.

8.    Select the ports connected to the Nexus switches to be added to the port channel:

a.     Click >> to add the ports to the port channel.

b.    Click Finish to create the port channel.

c.     Click OK.

9.    Expand Port Channels and select Port-Channel 119. Since the vPC has already been configured in the ACI fabric, this port channel should come up. Note that it may take a few minutes for the port channel to come up.

10.  In the navigation pane, under LAN > LAN Cloud, expand the fabric B tree.

11.  Right-click Port Channels.

12.  Select Create Port Channel.

13.  Enter 120 as the unique ID of the port channel.

14.  Enter Po120-ACI as the name of the port channel.

15.  Click Next.

16.  Select the ports connected to the Nexus switches to be added to the port channel:

a.     Click >> to add the ports to the port channel.

b.    Click Finish to create the port channel.

c.     Click OK.

17.  Expand Port Channels and select Port-Channel 120. Since the vPC has already been configured in the ACI fabric, this port channel should come up. Note that it may take a few minutes for the port channel to come up.

Create an Organization for this FlexPod

To create a UCS Organization to contain unique parameters for this particular FlexPod, complete the following steps on Cisco UCS Manager.

1.    Select the Servers icon on the left.

2.    Under Servers > Service-Profiles > root, right-click Sub-Organizations and select Create Organization.

3.    Name the Organization “FPV-FlexPod”, enter an optional Description, and click OK.

4.    Click OK for the confirmation.

Create an IQN Pool for iSCSI Boot

To configure the necessary IQN pool for the Cisco UCS environment, complete the following steps on Cisco UCS Manager.

1.    Select the SAN icon on the left.

2.    Select Pools > root.

3.    Right-click IQN Pools under the root organization.

4.    Select Create IQN Suffix Pool to create the IQN pool.

5.    Enter IQN-Pool for the name of the IQN pool.

6.    Optional: Enter a description for the IQN pool.

7.    Enter iqn.2010-11.com.flexpod for the Prefix.

8.    Select Sequential for Assignment Order.

9.    Click Next.

10.  Click Add.

11.  Enter a name to identify the individual UCS host for the Suffix.

12.  Enter 1 for the From field.

13.  Specify a size of the IQN block sufficient to support the available server resources.

14.  Click OK.

15.  Click Finish and OK to complete creating the IQN pool.

Create iSCSI Boot IP Address Pools

To configure the necessary iSCSI IP Address pools for the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click the LAN icon on the left.

2.    Select and expand Pools > root > Sub-Organizations > FPV-FlexPod.

3.    Right-click IP Pools under the FPV-FlexPod organization.

4.    Select Create IP Pool to create the IP pool.

5.    Enter iSCSI-IP-Pool-A as the name of the first IP pool.

6.    Optional: Enter a description for the IP pool.

7.    Select Sequential for Assignment Order.

8.    Click Next.

9.    Click Add to add a Block of IPs to the pool.

10.  Specify a starting IP address and subnet mask in the subnet <192.168.10.0/24> for iSCSI boot on Fabric A. It is not necessary to specify the Default Gateway or DNS server addresses.

11.  Specify a size for the IP pool that is sufficient to support the available blade or server resources.

12.  Click OK.

13.  Click Next.

14.  Click Finish.

15.  In the confirmation message, click OK.

16.  Right-click IP Pools under the FPV-FlexPod organization.

17.  Select Create IP Pool to create the IP pool.

18.  Enter iSCSI-IP-Pool-B as the name of the second IP pool.

19.  Optional: Enter a description for the IP pool.

20.  Select Sequential for Assignment Order

21.  Click Next.

22.  Click Add to add a Block of IPs to the pool.

23.  Specify a starting IP address and subnet mask in the subnet <192.168.20.0/24> for iSCSI boot on Fabric B. It is not necessary to specify the Default Gateway or DNS server addresses.

24.  Specify a size for the IP pool that is sufficient to support the available blade or server resources.

25.  Click OK.

26.  Click Next.

27.  Click Finish.

28.  In the confirmation message, click OK.

Create MAC Address Pools

To configure the necessary MAC address pools for the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click the LAN icon on the left.

2.    Select Pools > root.

3.    Right-click MAC Pools under the root organization.

4.    Select Create MAC Pool to create the MAC address pool.

5.    Enter MAC-Pool-A as the name of the MAC pool.

6.    Optional: Enter a description for the MAC pool.

7.    Select Sequential as the option for Assignment Order.

8.    Click Next.

9.    Click Add.

10.  Specify a starting MAC address.

11.  Specify a size for the MAC address pool that is sufficient to support the available blade or server resources assuming that multiple vNICs can be configured on each server.

12.  Click OK.

13.  Click Finish.

14.  In the confirmation message, click OK.

15.  Right-click MAC Pools under the root organization.

16.  Select Create MAC Pool to create the MAC address pool.

17.  Enter MAC-Pool-B as the name of the MAC pool.

18.  Optional: Enter a description for the MAC pool.

19.  Select Sequential as the option for Assignment Order.

20.  Click Next.

21.  Click Add.

22.  Specify a starting MAC address.

23.  Specify a size for the MAC address pool that is sufficient to support the available blade or server resources.

24.  Click OK.

25.  Click Finish.

26.  In the confirmation message, click OK.

Create UUID Suffix Pool

To configure the necessary universally unique identifier (UUID) suffix pool for the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click the Servers icon on the left.

2.    Select Pools > root.

3.    Right-click UUID Suffix Pools.

4.    Select Create UUID Suffix Pool.

5.    Enter UUID-Pool as the name of the UUID suffix pool.

6.    Optional: Enter a description for the UUID suffix pool.

7.    Keep the prefix at the Derived option.

8.    Select Sequential for the Assignment Order.

9.    Click Next.

10.  Click Add to add a block of UUIDs.

11.  Keep the From field at the default setting. Optionally, specify identifiers such as UCS location.

12.  Specify a size for the UUID block that is sufficient to support the available blade or server resources.

13.  Click OK.

14.  Click Finish.

15.  Click OK.

Create Server Pool

To configure the necessary server pool for the VMware management environment, complete the following steps:

1.    In Cisco UCS Manager, click the Servers icon on the left.

2.    Expand Pools > root > Sub-Organizations > FPV-FlexPod.

3.    Right-click Server Pools under the FPV-FlexPod Organization.

4.    Select Create Server Pool.

5.    Enter FPV-MGMT-Pool as the name of the server pool.

6.    Optional: Enter a description for the server pool.

7.    Click Next.

8.    Select two (or more) servers to be used for the VMware management cluster and click >> to add them to the FPV-MGMT-Pool server pool.

9.    Click Finish.

10.  Click OK.

Create VLANs

To configure the necessary virtual local area networks (VLANs) for the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click the LAN icon on the left.

2.    Select LAN > LAN Cloud.

3.    Right-click VLANs.

4.    Select Create VLANs.

5.    Enter “Native-VLAN” as the name of the VLAN to be used as the native VLAN.

6.    Keep the Common/Global option selected for the scope of the VLAN.

7.    Enter the native VLAN ID <2>.

8.    Keep the Sharing Type as None.

9.    Click OK and then click OK again.

10.  Expand the list of VLANs in the navigation pane, right-click the newly created “Native-VLAN” and select Set as Native VLAN.

11.  Click Yes and then click OK.

12.  Right-click VLANs.

13.  Select Create VLANs.

14.  Enter “IB-MGMT” as the name of the VLAN to be used for management traffic.

15.  Keep the Common/Global option selected for the scope of the VLAN.

16.  Enter the UCS In-Band management VLAN ID <213>.

17.  Keep the Sharing Type as None.

18.  Click OK, and then click OK again.

19.  Right-click VLANs.

20.  Select Create VLANs.

21.  Enter “NFS” as the name of the VLAN to be used for infrastructure NFS.

22.  Keep the Common/Global option selected for the scope of the VLAN.

23.  Enter the UCS Infrastructure NFS VLAN ID <3150>.

24.  Keep the Sharing Type as None.

25.  Click OK, and then click OK again.

26.  Right-click VLANs.

27.  Select Create VLANs.

28.  Enter “iSCSI-A” as the name of the VLAN to be used for UCS Fabric A iSCSI boot.

29.  Keep the Common/Global option selected for the scope of the VLAN.

30.  Enter the UCS Fabric A iSCSI boot VLAN ID <3110>.

31.  Keep the Sharing Type as None.

32.  Click OK, and then click OK again.

33.  Right-click VLANs.

34.  Select Create VLANs.

35.  Enter “iSCSI-B” as the name of the VLAN to be used for UCS Fabric B iSCSI boot.

36.  Keep the Common/Global option selected for the scope of the VLAN.

37.  Enter the UCS Fabric B iSCSI boot VLAN ID <3120>.

38.  Keep the Sharing Type as None.

39.  Click OK and then click OK again.

40.  Right-click VLANs.

41.  Select Create VLANs.

42.  Enter “vMotion” as the name of the VLAN to be used for VMware vMotion.

43.  Keep the Common/Global option selected for the scope of the VLAN.

44.  Enter the vMotion VLAN ID <3000>.

45.  Keep the Sharing Type as None.

46.  Click OK and then click OK again.

47.  Right-click VLANs.

48.  Select Create VLANs.

49.  Enter “FPV-vSwitch-Pool” as the prefix for this VLAN pool.

50.  Keep the Common/Global option selected for the scope of the VLAN.

51.  Enter a range of 50 VLANs for VLAN ID. <1101-1150> was used in this validation.

52.  Keep the Sharing Type as None.

53.  Click OK and then click OK again.

Modify Default Host Firmware Package

Firmware management policies allow the administrator to select the corresponding packages for a given server configuration. These policies often include packages for adapter, BIOS, board controller, FC adapters, host bus adapter (HBA) option ROM, and storage controller properties.

To specify the UCS 3.2(3d) release for the Default firmware management policy for a given server configuration in the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click the Servers icon on the left.

2.    Select Policies > root.

3.    Expand Host Firmware Packages.

4.    Select default.

5.    In the Actions pane, select Modify Package Versions.

6.    Select the version 3.2(3d)B for the Blade Package, and 3.2(3d)C (optional) for the Rack Package.

7.    Leave Excluded Components with only Local Disk selected.

8.    Click OK then click OK again to modify the host firmware package.

Set Jumbo Frames in Cisco UCS Fabric

To configure jumbo frames and enable the base quality of service in the Cisco UCS fabric, complete the following steps:

1.    In Cisco UCS Manager, click the LAN icon on the left.

2.    Select LAN > LAN Cloud > QoS System Class.

3.    In the right pane, click the General tab.

4.    On the Best Effort row, enter 9216 in the box under the MTU column.

5.    Click Save Changes in the bottom of the window.

6.    Click OK.

Create Local Disk Configuration Policy (Optional)

A local disk configuration for the Cisco UCS environment is necessary if the servers in the environment do not have a local disk.

To create a local disk configuration policy, complete the following steps:

1.    In Cisco UCS Manager, click the Servers icon on the left.

2.    Select Policies > root.

3.    Right-click Local Disk Config Policies.

4.    Select Create Local Disk Configuration Policy.

5.    Enter SAN-Boot as the local disk configuration policy name.

6.    Change the mode to No Local Storage.

7.    Click OK to create the local disk configuration policy.

8.    Click OK.

Create Network Control Policy for Cisco Discovery Protocol (CDP) and Link Layer Discovery Protocol (LLDP)

To create a network control policy that enables CDP and LLDP on virtual network ports, complete the following steps:

1.    In Cisco UCS Manager, click the LAN icon on the left.

2.    Select Policies > root.

3.    Right-click Network Control Policies.

4.    Select Create Network Control Policy.

5.    Enter Enable-CDP-LLDP as the policy name.

6.    For CDP, select the Enabled option.

7.    For LLDP, scroll down and select Enabled for both Transmit and Receive.

8.    Click OK to create the network control policy.

 

9.    Click OK.

Create Power Control Policy

To create a power control policy for the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click the Servers icon on the left.

2.    Select Policies > root.

3.    Right-click Power Control Policies.

4.    Select Create Power Control Policy.

5.    Enter No-Power-Cap as the power control policy name.

6.    Change the power capping setting to No Cap.

7.    Click OK to create the power control policy.

8.    Click OK.

Create Server BIOS Policy

To create a server BIOS policy for the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click the Servers icon on the left.

2.    Select Policies > root.

3.    Right-click BIOS Policies.

4.    Select Create BIOS Policy.

5.    Enter Virtual-Host as the BIOS policy name.

6.    Click OK then OK again.

7.    Expand BIOS Policies and select Virtual-Host.

8.    Set the following within the Main tab:

a.     CDN Control  > Enabled

b.    Quiet Boot  > Disabled

9.    Click Save Changes and OK.

10.  Click the Advanced tab and then select the Processor sub-tab.

11.  Set the following within the Processor sub-tab:

a.     DRAM Clock Throttling  > Performance

b.    Frequency Floor Override  > Enabled

c.     Processor C State  > Disabled

d.    Processor C1E  > Disabled

e.    Processor C3 Report  > Disabled

f.      Processor C7 Report  > Disabled

g.    Energy Performance  > Performance

12.  Click Save Changes and OK.

13.  Click the RAS Memory sub-tab and select:

a.     LV DDR Mode > Performance-Mode

14.  Click Save Changes and OK.

Create High Traffic VMware Adapter Policy

To create the option VMware-High-Traffic Ethernet Adapter policy to provide higher vNIC performance, complete the following steps:

1.    In Cisco UCS Manager, click the Servers icon on the left.

2.    Select Policies > root.

3.    Right-click Adapter Policies and select Create Ethernet Adapter Policy.

4.    Name the policy VMware-HighTrf.

5.    Expand Resources and set the values as shown below.

6.    Expand Options and select Enabled for Receive Side Scaling (RSS).

7.    Click OK, then OK again to complete creating the Ethernet Adapter Policy.

Update the Default Maintenance Policy

To update the default Maintenance Policy, complete the following steps:

1.    In Cisco UCS Manager, click the Servers icon on the left.

2.    Select Policies > root.

3.    Select Maintenance Policies > default.

4.    Change the Reboot Policy to User Ack.

5.    Select “On Next Boot” to delegate maintenance windows to server administrators.

6.    Click OK to save changes.

7.    Click OK to accept the change.

Create vNIC Templates

To create multiple virtual network interface card (vNIC) templates for the Cisco UCS environment, complete the following steps. A total of 6 vNIC Templates will be created.

Create Infrastructure vNICs Templates

1.    In Cisco UCS Manager, click the LAN icon on the left.

2.    Expand Policies > root > Sub-Organizations > FPV-FlexPod.

3.    Right-click vNIC Templates under FPV-FlexPod.

4.    Select Create vNIC Template.

5.    Enter Infra-A as the vNIC template name.

6.    Keep Fabric A selected.

7.    Do not select the Enable Failover checkbox.

8.    Select Primary Template for Redundancy Type.

9.    Leave the Peer Redundancy Template set to <not set>.

10.  Under Target, make sure that only the Adapter checkbox is selected.

11.  Select Updating Template as the Template Type.

12.  Under VLANs, select the checkboxes for the IB-MGMT, Native, NFS, and vMotion VLANs.

13.  Set Native-VLAN as the native VLAN.

14.  Select vNIC Name for the CDN Source.

15.  For MTU, enter 9000.

16.  In the MAC Pool list, select MAC-Pool-A.

17.  In the Network Control Policy list, select Enable-CDP-LLDP.

18.  Click OK to create the vNIC template.

19.  Click OK.

Create the Secondary Redundancy Template Infra-B

1.    Select the LAN icon on the left.

2.    Expand Policies > root > Sub-Organizations > FPV-FlexPod.

3.    Right-click vNIC Templates under FPV-FlexPod.

4.    Select Create vNIC Template.

5.    Enter Infra-B as the vNIC template name.

6.    Select Fabric B.

7.    Do not elect the Enable Failover checkbox.

8.    Set Redundancy Type to Secondary Template.

9.    Select Infra-A for the Peer Redundancy Template.

10.  In the MAC Pool list, select MAC-Pool-B. The MAC Pool is all that needs to be selected for the Secondary Template.

11.  Click OK to create the vNIC template.

12.  Click OK.

Create iSCSI Boot vNICs

To create iSCSI Boot vNICs, complete the following steps:

1.    In Cisco UCS Manager, click the LAN icon on the left.

2.    Expand Policies > root > Sub-Organizations > FPV-FlexPod.

3.    Right-click vNIC Templates under FPV-FlexPod.

4.    Select Create vNIC Template.

5.    Enter iSCSI-A as the vNIC template name.

6.    Keep Fabric A selected.

7.    Do not select the Enable Failover checkbox.

8.    Select No Redundancy for Redundancy Type.

9.    Under Target, make sure that only the Adapter checkbox is selected.

10.  Select Updating Template as the Template Type.

11.  Under VLANs, select the checkbox for iSCSI-A.

12.  Set iSCSI-A as the native VLAN.

13.  Select vNIC Name for the CDN Source.

14.  For MTU, enter 9000.

15.  In the MAC Pool list, select MAC-Pool-A.

16.  In the Network Control Policy list, select Enable-CDP-LLDP.

17.  Click OK to create the vNIC template.

18.  Click OK.

Create the Infra-iSCSI-B Template

1.    Select the LAN icon on the left.

2.    Expand Policies > root > Sub-Organizations > FPV-FlexPod.

3.    Right-click vNIC Templates under FPV-FlexPod.

4.    Select Create vNIC Template.

5.    Enter iSCSI-B as the vNIC template name.

6.    Select Fabric B.

7.    Do not elect the Enable Failover checkbox.

8.    Select No Redundancy for Redundancy Type.

9.    Under Target, make sure that only the Adapter checkbox is selected.

10.  Select Updating Template as the Template Type.

11.  Under VLANs, select the checkbox for iSCSI-B.

12.  Set iSCSI-B as the native VLAN.

13.  Select vNIC Name for the CDN Source.

14.  For MTU, enter 9000.

15.  In the MAC Pool list, select MAC-Pool-B.

16.  In the Network Control Policy list, select Enable-CDP-LLDP.

17.  Click OK to create the vNIC template.

18.  Click OK.

Create vNIC Templates for APIC-Implemented vDS

To create vNIC templates for APIC-controlled vDS, complete the following steps:

1.    In Cisco UCS Manager, click the LAN icon on the left.

2.    Expand Policies > root > Sub-Organizations > FPV-FlexPod.

3.    Right-click vNIC Templates under FPV-FlexPod.

4.    Select Create vNIC Template.

5.    Enter APIC-vDS-A as the vNIC template name.

6.    Keep Fabric A selected.

7.    Do not select the Enable Failover checkbox.

8.    Select Primary Template for Redundancy Type.

9.    Leave the Peer Redundancy Template set to <not set>.

10.  Under Target, make sure that only the Adapter checkbox is selected.

11.  Select Updating Template as the Template Type.

12.  Under VLANs, select the checkboxes for the 50 Virtual-Switch-Pool VLANs.

13.  Do not set a native VLAN.

14.  Select vNIC Name for the CDN Source.

15.  For MTU, enter 9000.

16.  In the MAC Pool list, select MAC-Pool-A.

17.  In the Network Control Policy list, select Enable-CDP-LLDP.

18.  Click OK to create the vNIC template.

19.  Click OK.

Create the Secondary Redundancy Template APIC-VS-B

1.    Select the LAN icon on the left.

2.    Expand Policies > root > Sub-Organizations > FPV-FlexPod.

3.    Right-click vNIC Templates under FPV-FlexPod.

4.    Select Create vNIC Template:

a.     Enter APIC-vDS-B as the vNIC template name.

b.    Select Fabric B.

c.     Do not elect the Enable Failover checkbox.

d.    Set Redundancy Type to Secondary Template.

e.    Select APIC-vDS-A for the Peer Redundancy Template.

f.      In the MAC Pool list, select MAC-Pool-B. The MAC Pool is all that needs to be selected for the Secondary Template.

g.    Click OK to create the vNIC template.

5.    Click OK.

Create LAN Connectivity Policy for iSCSI Boot

To configure the necessary Infrastructure LAN Connectivity Policy, complete the following steps:

1.    In Cisco UCS Manager, click the LAN icon on the left.

2.    Expand Policies > root > Sub-Organizations > FPV-FlexPod.

3.    Right-click LAN Connectivity Policies under FPV-FlexPod.

4.    Select Create LAN Connectivity Policy.

5.    Enter iSCSI-Boot as the name of the policy.

6.    Click the upper Add button to add a vNIC.

7.    In the Create vNIC dialog box, enter 00-Infra-A as the name of the vNIC.

8.    Select the Use vNIC Template checkbox.

9.    In the vNIC Template list, select Infra-A.

10.  In the Adapter Policy list, select VMware.

11.  Click OK to add this vNIC to the policy.

12.  Click the upper Add button to add another vNIC to the policy.

13.  In the Create vNIC box, enter 01-Infra-B as the name of the vNIC.

14.  Select the Use vNIC Template checkbox.

15.  In the vNIC Template list, select Infra-B.

16.  In the Adapter Policy list, select VMware.

17.  Click OK to add the vNIC to the policy.

18.  Click the upper Add button to add another vNIC to the policy.

19.  In the Create vNIC box, enter 02-iSCSI-A as the name of the vNIC.

20.  Select the Use vNIC Template checkbox.

21.  In the vNIC Template list, select iSCSI-A.

22.  In the Adapter Policy list, select VMware.

23.  Click OK to add the vNIC to the policy.

24.  Click the upper Add button to add another vNIC to the policy.

25.  In the Create vNIC box, enter 03-iSCSI-B as the name of the vNIC.

26.  Select the Use vNIC Template checkbox.

27.  In the vNIC Template list, select iSCSI-B.

28.  In the Adapter Policy list, select VMware.

29.  Click OK to add the vNIC to the policy.

30.  Click the upper Add button to add another vNIC to the policy.

31.  In the Create vNIC box, enter 04-APIC-vDS-A as the name of the vNIC.

32.  Select the Use vNIC Template checkbox.

33.  In the vNIC Template list, select APIC-vDS-A.

34.  In the Adapter Policy list, select VMware. Optionally, select the VMware-HighTrf Adapter Policy.

35.  Click OK to add the vNIC to the policy.

36.  Click the upper Add button to add another vNIC to the policy.

37.  In the Create vNIC box, enter 05-APIC-vDS-B as the name of the vNIC.

38.  Select the Use vNIC Template checkbox.

39.  In the vNIC Template list, select APIC-vDS-B.

40.  In the Adapter Policy list, select VMware. Optionally, select the VMware-HighTrf Adapter Policy.

41.  Click OK to add the vNIC to the policy.

42.  Expand the Add iSCSI vNICs section.

43.  Click the lower Add button to add an iSCSI boot vNIC to the policy.

44.  In the Create iSCSI vNIC box, enter iSCSI-Boot-A as the name of the vNIC.

45.  Select 02-iSCSI-A for the Overlay vNIC.

46.  Select the default iSCSI Adapter Policy.

47.  FPV-Foundation-iSCSI-A (native) should be selected as the VLAN.

48.  Do not select anything for MAC Address Assignment.

49.  Click OK to add the vNIC to the policy.

50.  Click the lower Add button to add an iSCSI boot vNIC to the policy.

51.  In the Create iSCSI vNIC box, enter iSCSI-Boot-B as the name of the vNIC.

52.  Select 03-iSCSI-B for the Overlay vNIC.

53.  Select the default iSCSI Adapter Policy.

54.  FPV-Foundation-iSCSI-B (native) should be selected as the VLAN.

55.  Do not select anything for MAC Address Assignment.

56.  Click OK to add the vNIC to the policy.

57.  Click OK, then OK again to create the LAN Connectivity Policy.

Create iSCSI Boot Policies

This procedure applies to a Cisco UCS environment in which two iSCSI logical interfaces (LIFs) are on storage cluster node 1 (iscsi_lif01a and iscsi_lif01b) and two iSCSI LIFs are on storage cluster node 2 (iscsi_lif02a and iscsi_lif02b).

This boot policy configures the primary target to be iscsi_lif01a with four SAN paths.

To create a boot policy for the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click the Servers icon on the left.

2.    Expand Policies > root > Sub-Organizations > FPV-FlexPod.

3.    Right-click Boot Policies under FPV-FlexPod.

4.    Select Create Boot Policy.

5.    Enter iSCSI-Boot as the name of the boot policy.

6.    Optional: Enter a description for the boot policy.

7.    Keep the Reboot on Boot Order Change option cleared.

8.    Expand the Local Devices drop-down list and select Add Remote CD/DVD.

9.    Expand the iSCSI vNICs drop-down list and select Add iSCSI Boot.

10.  Enter iSCSI-Boot-A in the iSCSI vNIC field.

11.  Click OK.

12.  From the iSCSI vNICs drop-down list, select Add iSCSI Boot.

13.  Enter iSCSI-Boot-B in the iSCSI vNIC field.

14.  Click OK.

15.  Click OK, then click OK again to create the boot policy.

Create iSCSI Boot Service Profile Template

In this procedure, one service profile template is created for Fabric A boot.

To create the service profile template, complete the following steps:

1.    In Cisco UCS Manager, click the Servers icon on the left.

2.    Expand Service Profile Templates > root > Sub-Organizations > FPV-FlexPod.

3.    Select and right-click FPV-FlexPod.

4.    Select Create Service Profile Template to open the Create Service Profile Template wizard.

5.    Enter iSCSI-Boot-A as the name of the service profile template. This service profile template is configured to boot from storage node 1 on fabric A.

6.    Select the “Updating Template” option.

7.    Under UUID, select UUID-Pool as the UUID pool.

8.      Click Next.

Configure Storage Provisioning

1.    If you have servers with no physical disks, click the Local Disk Configuration Policy and select the SAN-Boot Local Storage Policy. Otherwise, select the default Local Storage Policy.

2.    Click Next.

Configure Networking Options

1.    Keep the default setting for Dynamic vNIC Connection Policy.

2.    Select the “Use Connectivity Policy” option to configure the LAN connectivity.

3.    Select iSCSI-Boot from the LAN Connectivity Policy drop-down list.

4.    Select IQN-Pool from the 192 Name Assignment drop-down list.

5.    Click Next.

Configure SAN Connectivity Options

1.    Select the No vHBAs option for the “How would you like to configure SAN connectivity?” field.

2.    Click Next.

Configure Zoning Options

1.    Configure no Zoning Options and click Next.

Configure vNIC/HBA Placement

1.    In the “Select Placement” list, leave the placement policy as “Let System Perform Placement.”

2.    Click Next.

Configure vMedia Policy

1.    Do not select a vMedia Policy.

2.    Click Next.

Configure Server Boot Order

1.    Select iSCSI-Boot for Boot Policy.

2.    Under Boot Order, expand Boot Order and select the iSCSI-Boot-A row.

3.    Select the Set iSCSI Boot Parameters button.

4.    Select iSCSI-IP-Pool-A for the Initiator IP Address Policy.

5.    Scroll to the bottom of the window and click Add.

6.    Enter the IQN (Target Name) from the Infra-A iSCSI Target Name. 

7.    For IPv4 address, enter the IP address of iscsi_a_lif02a from the Infra-A. 

8.    Click OK to complete configuring the iSCSI target.

9.    Click Add to add a second target.

10.  Enter the previously captured IQN (Target Name) again. 

11.  For IPv4 address, enter the IP address of iscsi_a_lif01a from the Infra-A SVM. 

12.  Click OK to complete configuring the iSCSI target.

13.  Click OK to complete setting the iSCSI Boot Parameters for Fabric A Boot.

14.  Under Boot Order, select the iSCSI-Boot-B row.

15.  Select the Set iSCSI Boot Parameters button.

16.  Select iSCSI-IP-Pool-B for the Initiator IP Address Policy.

17.  Scroll to the bottom of the window and click Add.

18.  Enter the previously captured IQN (Target Name)

19.  For IPv4 address, enter the IP address of iscsi_lif02b from the Infra-A SVM. 

20.  Click OK to complete configuring the iSCSI target.

21.  Click Add to add a second target.

22.  Enter the previously captured IQN (Target Name. 

23.  For IPv4 address, enter the IP address of iscsi_lif01b from the Infra-A SVM. 

24.  Click OK to complete configuring the iSCSI target.

25.  Click OK to complete setting the iSCSI Boot Parameters for Fabric A Boot.

26.  Click Next.

Configure Maintenance Policy

To configure the Maintenance Policy, complete the following steps:

27.  Change the Maintenance Policy to default.

28.  Click Next.

Configure Server Assignment

To configure server assignment, complete the following steps:

1.    In the Pool Assignment list, select FPV-MGMT-Pool.

2.    Select Down as the power state to be applied when the profile is associated with the server.

3.    Expand Firmware Management at the bottom of the page and select the default policy.

4.    Click Next.

Configure Operational Policies

To configure the operational policies, complete the following steps:

1.    In the BIOS Policy list, select Virtual-Host.

2.    Expand Power Control Policy Configuration and select No-Power-Cap in the Power Control Policy list.

3.    Click Finish to create the service profile template.

4.    Click OK in the confirmation message.

Create Service Profiles

To create service profiles from the service profile template, complete the following steps:

1.    Connect to UCS Manager and click the Servers icon on the left.

2.    Select Service Profile Templates > root > Sub-Organizations > FPV-FlexPod > Service Template iSCSI-Boot-A.

3.    Right-click iSCSI-Boot-A and select Create Service Profiles from Template.

4.    Enter fpv-esxi-0 as the service profile prefix.

5.    Enter 1 as “Name Suffix Starting Number.”

6.    Enter 2 as the “Number of Instances.”

7.    Click OK to create the service profiles.

 

8.    Click OK in the confirmation message.

Cisco UCS Configuration on 10GbE Site

All the steps in this section will be repeated for configuring Cisco UCS 6248 Fabric Interconnect. Customers should make sure unique values are used for various configuration parameters including:

·         Management IP address Pools

·         UUID Pool

·         IQN range in the IQN Pool

·         iSCSI IP Pools

·         MAC Address Pool

·         UUID Pool

When UCS configuration is complete, the next step is to configure boot from SAN.

Gather Necessary Information

After the Cisco UCS service profiles have been created, each infrastructure server in the environment will have a unique configuration. To proceed with the UCS boot from SAN deployment, specific information must be gathered from each Cisco UCS server and from the NetApp controllers. Insert the required information into Table 10 and Table 11.

In the tables below, Infra-A SVM is defined on the NetApp A700 system in POD11 (40GbE site) while Infra-B SVM is defined on the NetApp A700 system in POD1 (10GbE site). The boot LUNs for first two ESXi server, fpv-esxi-01 and fpv-esxi-02 are created in the Infra-A SVM while the SVMs for other two ESXi servers are created in

Table 10      iSCSI LIFs for iSCSI IQN

Table 11      vNIC iSCSI IQNs for Fabric A and Fabric B

NetApp ONTAP Boot Storage Setup

Create igroups

1.    To create igroups for the first two ESXi servers in 40 GbE site, run the following commands on the NetApp A700 system:

2.    To create igroups for the second two ESXi servers in 10GbE site, run the following commands on the NetApp A700 system:

Map Boot LUNs to igroups

1.    To map LUNs to igroups, run the following commands on 40GbE site:

2.    To map LUNs to igroups, run the following commands on 10GbE site:

VMware ESXi 6.7

This section provides detailed instructions for installing VMware ESXi 6.7 in a FlexPod environment. After the procedures are completed, two ESXi hosts will be provisioned in each site.

Several methods exist for installing ESXi in a VMware environment. These procedures focus on how to use the built-in keyboard, video, mouse (KVM) console and mapped CD/DVD in Cisco UCS Manager to map remote installation media to individual servers and connect to their boot logical unit numbers (LUNs).

Download Cisco Custom Image for ESXi 6.7

If the VMware ESXi custom image has not been downloaded, complete the following steps to download:

1.    Click the following link: VMware vSphere Hypervisor (ESXi) 6.7.

2.    You will need a user id and password on vmware.com to download this software.

3.    Download the .iso file.

Log in to Cisco UCS Fabric Interconnect

Cisco UCS Manager

The Cisco UCS IP KVM enables the administrator to begin the installation of the operating system (OS) through remote media. It is necessary to log in to the Cisco UCS environment to run the IP KVM.

To log in to the Cisco UCS environment, complete the following steps:

1.    Open a web browser and enter the IP address for the Cisco UCS cluster address. This step launches the Cisco UCS Manager application.

2.    Click the Launch UCS Manager link under HTML to launch the HTML 5 UCS Manager GUI.

3.    If prompted to accept security certificates, accept as necessary.

4.    When prompted, enter admin as the user name and enter the administrative password.

5.    To log in to Cisco UCS Manager, click Login.

6.    From the main menu, click Servers on the left.

7.    Select the fpv-esxi-xx Service Profile (xx represents the servers 01-04)

8.    On the right, under the General tab, click the >> to the right of KVM Console.

9.    Follow the prompts to launch the KVM console.

Set Up VMware ESXi Installation

To prepare the server for the OS installation, complete the following steps on each ESXi host:

1.    In the KVM window, click Virtual Media in the top right corner.

2.    Click Activate Virtual Devices.

3.    If prompted to accept an Unencrypted KVM session, accept as necessary.

4.    Click Virtual Media and select Map CD/DVD.

5.    Browse to the ESXi installer ISO image file and click Open.

6.    Click Map Device.

7.    Click the KVM tab to monitor the server boot.

Install ESXi

To install VMware ESXi to the iSCSI-bootable LUN of the hosts, complete the following steps on all the hosts:

1.    Boot the server by selecting Boot Server and clicking OK, then click OK two more times.

2.    On reboot, the machine detects the presence of the ESXi installation media. Select the ESXi installer from the boot menu that is displayed.

3.    After the installer is finished loading, press Enter to continue with the installation.

4.    Read and accept the end-user license agreement (EULA). Press F11 to accept and continue.

5.    Select the LUN that was previously set up as the installation disk for ESXi and press Enter to continue with the installation.

6.    Select the appropriate keyboard layout and press Enter.

7.    Enter and confirm the root password and press Enter.

8.    The installer issues a warning that the selected disk will be repartitioned. Press F11 to continue with the installation.

9.    After the installation is complete, press Enter to reboot the server. The mapped iso will be automatically unmapped.

Set Up Management Networking for ESXi Hosts

Adding a management network for each VMware host is necessary for managing the host. To add a management network for the VMware hosts, complete the steps in the following subsections.

To configure each ESXi host with access to the management network, complete the following steps:

1.    After the server has finished rebooting, press F2 to customize the system.

2.    Log in as root, enter the password set in the last step, and press Enter to log in.

3.    Select Troubleshooting Options and press Enter.

4.    Select Enable ESXi Shell and press Enter.

5.    Select Enable SSH and press Enter.

6.    Press Esc to exit the Troubleshooting Options menu.

7.    Select the Configure Management Network option and press Enter.

8.    Select Network Adapters and press Enter.

9.    Verify that the numbers in the Hardware Label field match the numbers in the Device Name field.

10.  Use the arrow keys and spacebar to highlight and select vmnic1.

11.  Press Enter.

12.  Select the VLAN (Optional) option and press Enter.

13.  Enter the UCS <ib-mgmt-vlan-id> <213> and press Enter.

14.  Select IPv4 Configuration and press Enter.

15.  Select the Set static IPv4 address and network configuration option by using the space bar.

16.  Enter the IP address for managing the ESXi host.

17.  Enter the subnet mask for the ESXi host.

18.  Enter the default gateway for the ESXi host.

19.  Press Enter to accept the changes to the IP configuration.

20.  Select the DNS Configuration option and press Enter.

21.  Enter the IP address of the primary DNS server.

22.  Optional: Enter the IP address of the secondary DNS server.

23.  Enter the fully qualified domain name (FQDN) for the ESXi host.

24.  Press Enter to accept the changes to the DNS configuration.

25.  Press Esc to exit the Configure Management Network menu. Enter Y to Apply changes and restart management network.

26.  Select Test Management Network to verify that the management network is set up correctly and press Enter.

27.  Press Enter to run the test, press Enter again once the test has completed, review environment if there is a failure.

28.  Re-select the Configure Management Network and press Enter.

29.  Select the IPv6 Configuration option and press Enter.

30.  Using the spacebar, select Disable IPv6 (restart required) and press Enter.

31.  Press Esc to exit the Configure Management Network submenu.

32.  Press Y to confirm the changes and reboot the ESXi host.

Log in to VMware ESXi Hosts by Using VMware Host Client

To log in to the fpv-esxi-xx (xx is server number 01-04) ESXi host by using the VMware Host Client, complete the following steps:

1.    Open a web browser on the management workstation and navigate to the fpv-esxi-xx management IP address. Respond to any security prompts.

2.    Enter root for the user name.

3.    Enter the root password.

4.    Click Login to connect.

5.    Repeat this process to log into all the ESXi hosts in a separate browser tabs or windows.

Set Up VMkernel Ports and Virtual Switch

To set up the VMkernel ports and the virtual switches on all the ESXi hosts, complete the following steps:

1.    From the Host Client, select Networking on the left.

2.    In the center pane, select the Virtual switches tab.

3.    Highlight vSwitch0.

4.    Select Edit settings.

5.    Change the MTU to 9000.

6.    Expand NIC teaming and highlight vmnic1. Select Mark active.

7.    Click Save.

8.    Select Networking on the left.

9.    In the center pane, select the Virtual switches tab.

10.  Highlight iScsiBootvSwitch.

11.  Select Edit settings.

12.  Change the MTU to 9000

13.  Click Save.

14.  Select Add standard virtual switch.

15.  Name the vSwitch iScsiBootvSwitch-B.

16.  Set the MTU to 9000.

17.  Select vmnic3 for the Uplink.

18.  Click Add.

19.  Select the VMkernel NICs tab.

20.  Highlight vmk1 iScsiBootPG.

21.  Select Edit settings.

22.  Change the MTU to 9000.

23.  Expand IPv4 settings and change the IP address to an address outside of the UCS iSCSI-IP-Pool-A.

24.  Click Save.

25.  Select Add VMkernel NIC

26.  Specify a New port group name of iScsiBootPG-B.

27.  Select iScsciBootvSwitch-B for Virtual switch.

28.  Set the MTU to 9000. Do not enter a VLAN ID since the iSCSI-B VLAN is also the native VLAN on this vNIC.

29.  Select Static for the IPv4 settings and expand the option to provide the Address and Subnet Mask within the Configuration.

30.  Click Create.

31.  Select Add VMkernel NIC.

32.  Specify a New port group name of VMkernel-Infra-NFS.

33.  Select vSwitch0 for Virtual switch.

34.  Enter the UCS Foundation Tenant NFS VLAN id <3050>.

35.  Set the MTU to 9000.

36.  Select Static for the IPv4 settings and expand the option to provide the Address and Subnet Mask in the Foundation Tenant NFS subnet <192.168.50.0/24>.

37.  Click Create.

38.  Select Add VMkernel NIC

39.  Specify a New port group name of VMkernel-vMotion.

40.  Select vSwitch0 for Virtual switch.

41.  Enter the UCS vMotion VLAN id <3000>.

42.  Set the MTU to 9000.

43.  Select Static for the IPv4 settings and expand the option to provide the Address and Subnet Mask in the vMotion subnet <192.168.100.0/24>.

44.  Select the vMotion stack for the TCP/IP stack.

45.  Click Create.

46.  Optionally, if you have 40GE vNICs in this FlexPod, create two more vMotion VMkernel ports in the same subnet and VLAN. These will need to be in new port groups.

47.  On the left, select Networking, then select the Port groups tab.

48.  In the center pane, right-click VM Network and select Remove.

49.  Click Remove to complete removing the port group.

50.  In the center pane, select Add port group.

51.  Name the port group IB-MGMT Network, enter <ib-mgmt-vlan-id> <213> in the VLAN ID field, and make sure Virtual switch vSwitch0 is selected.

52.  Click Add to finalize the edits for the IB-MGMT Network.

53.  Highlight the VMkernel-vMotion Port group and select Edit settings.

54.  Expand NIC teaming and select the radio button next to Override failover order.

55.  Select vmnic0 and then select Mark standby to pin vMotion traffic to UCS Fabric Interconnect B (vmnic1) with failover.

56.  Click Save.

57.  Select the Virtual Switches tab, then vSwitch0. The properties for vSwitch0 VMkernel NICs should be similar to the following example:

58.  Select the VMkernel NICs tab to confirm configured virtual adapters. The adapters listed should be similar to the following example:

Setup iSCSI Multipathing

To setup the iSCSI multipathing on all the ESXi hosts, complete the following steps:

1.    From each Host Client, select Storage on the left.

2.    In the center pane, click Adapters.

3.    Select the iSCSI software adapter and click Configure iSCSI.

4.    Under Dynamic targets, click Add dynamic target.

5.    Enter the IP Address of NetApp storage iscsi_a_lif01a and press Enter.

6.    Repeat putting the ip address of iscsi_a_lif01b, iscsi_a_lif02a, iscsi_a_lif02b.

7.    Click Save configuration.

 

Mount Required Datastores

To mount the required datastores, complete the following steps on each ESXi host by entering the appropriate IP addresses from the NetApp controllers at both the sites:

1.    From the Host Client, select Storage on the left.

2.    In the center pane, select the Datastores tab.

3.    In the center pane, select New Datastore to add a new datastore.

4.    In the New datastore popup, select Mount NFS datastore and click Next.

 

5.    Enter “infra_a_datastore_1” for the datastore name

6.    Enter the IP address for the nfs_lif01 LIF for the NFS server from A700 in 40GbE site. 

7.    Enter “/infra_a_datastore_1” for the NFS share.  Leave the NFS version set at NFS 3. 

8.    Click Next.

9.    Repeat this process till all four NFS datastores i.e. infra_a_datastore_1, infra_a_datastore_2, infra_b_datastore_1, infra_b_datastore_2 are added to the ESXi server. Make sure the correct IP address from appropriate storage controller is used.

10.  Mount these datastores on all the ESXi hosts at both sites.

Configure NTP on ESXi Hosts

To configure Network Time Protocol (NTP) on the ESXi hosts, complete the following steps on each host:

1.    From the Host Client, select Manage on the left.

2.    In the center pane, select the Time & Date tab.

3.    Click Edit settings.

4.    Make sure Use Network Time Protocol (enable NTP client) is selected.

5.    Use the drop-down list to select Start and stop with host.

6.    Enter the appropriate NTP address(es) in the NTP servers box separated by a comma.

7.    Click Save to save the configuration changes.

8.    Select Actions > NTP service > Start.

9.    Verify that NTP service is now running and the clock is now set to approximately the correct time.

Install VMware ESXi Patches and VAAI plugin

To install necessary ESXi patches and updated device drivers, complete the following steps on each host:

1.    Download VMware ESXi patch ESXi670-201806001 from https://my.vmware.com/group/vmware/patch#search.

2.    Download the NetApp NFS Plug-in for VMware VAAI 1.1.2 offline bundle from https://mysupport.netapp.com/NOW/download/software/nfs_plugin_vaai_esxi5.5/1.1.2/.

3.    From the Host Client, select Storage on the left.

4.    In the center pane, select infra_a_datastore_1 and then select Datastore browser.

5.    In the Datastore browser, select Create directory and create a Drivers folder in the datastore.

6.    Select the Drivers folder.

7.    Use Upload to upload the two downloaded items above to the Drivers folder on infra_a_datastore_1. Since infra_a_datastore_1 is accessible to all the ESXi hosts, this upload only needs to be done on the first host.

8.    Use ssh to connect to each ESXi host as the root user.

9.    Enter the following commands on each host.

VMware vCenter 6.7

The procedures in the following subsections provide detailed instructions for installing the VMware vCenter 6.7 Server Appliance in an environment. After the procedures are completed, a VMware vCenter Server will be configured.

Building the VMware vCenter Server Appliance

The VCSA deployment consists of 2 stages: install and configuration. To build the VMware vCenter virtual machine, complete the following steps:

1.    Locate and copy the VMware-VCSA-all-6.7.0-8832884.iso file to the desktop of the management workstation.  This ISO is for the VMware vSphere 6.5 vCenter Server Appliance.

2.    Using ISO mounting software, mount the ISO image as a disk on the management workstation. (For example, with the Mount command in Windows Server 2012).

3.    In the mounted disk directory, navigate to the vcsa-ui-installer > win32 directory and double-click installer.exe. The vCenter Server Appliance Installer wizard appears.

4.    Click Install to start the vCenter Server Appliance deployment wizard.

5.    Click Next in the Introduction section.

6.    Read and accept the license agreement and click Next.

7.    In the “Select deployment type” section, select vCenter Server with an Embedded Platform Services Controller and click Next.

8.    In the “Appliance deployment target”, enter the ESXi host name or IP address for fpv-esxi-01, User name and Password. Click Next.

  

9.    Click Yes to accept the certificate.

10.  Enter the Appliance name and password details in the “Set up appliance VM” section. Click Next.

   

11.  In the “Select deployment size” section, select the deployment size and Storage size. For example, “Small” Deployment size was selected in this CVD.

12.  Click Next.

13.  Select preferred datastore e.g. the “infra_a_datastore_2” was selected in the CVD. Click Next.

14.  In the “Network Settings” section, configure the following settings:

a.     Choose a Network: Core-Services Network

b.    IP version: IPV4

c.     IP assignment: static

d.    System name: <vcenter-fqdn>

e.    IP address: <vcenter-ip>

f.      Subnet mask or prefix length: <vcenter-subnet-mask>

g.    Default gateway: <vcenter-gateway>

h.     DNS Servers: <dns-server>

    

15.  Click Next.

16.  Review all values and click Finish to complete the installation.

17.  The vCenter appliance installation will take a few minutes to complete.

18.  Click Continue to proceed with stage 2 configuration.

19.  Click Next.

20.  In the Appliance Configuration, configure the below settings:

a.     Time Synchronization Mode: Synchronize time with the ESXi host.

b.    SSH access: Enabled.

21.  Click Next.

22.  Complete the SSO configuration as shown below.

23.  Click Next.

24.  If preferred, select Join the VMware’s Customer Experience Improvement Program (CEIP).

25.  Click Next.

26.  Review the configuration and click Finish.

27.  Click OK.

28.  Make note of the access URL shown in the completion screen.

29.  Click Close.

Setting Up VMware vCenter Server

To set up the VMware vCenter Server, complete the following steps:

1.    Using a web browser, navigate to https://<vcenter-ip>/vsphere-client.

2.    Log in using the Single Sign-On username (Administrator@flexpod.local) and password created during the vCenter installation.

3.    Click “Create Datacenter” in the center pane.

4.    Type a name for the FlexPod Datacenter <FPV-FlexPod-DC> in the Datacenter name field.

5.    Click OK.

  

6.    Right-click the data center just created and select New Cluster.

7.    Name the cluster FPV-Foundation.

8.    Check the box to turn on DRS. Leave the default values.

9.    Check the box to turn on vSphere HA. Leave the default values.

  

10.  Click OK to create the new cluster.

11.  On the left pane, expand the Datacenter.

12.  Right-click the FPV-Foundation cluster and select Add Host.

13.  In the Host field, enter either the IP address or the FQDN name of one of the VMware ESXi hosts. Click Next.

14.  Type root as the user name and the root password. Click Next to continue.

15.  Click Yes to accept the certificate.

16.  Review the host details and click Next to continue.

17.  Assign a license or leave in evaluation mode and click Next to continue.

18.  Click Next to continue.

19.  Click Next to continue.

20.  Review the configuration parameters. Click Finish to add the host.

21.  Repeat the steps 12 to 20 to add the remaining VMware ESXi hosts from both the sites to the cluster.

ESXi Dump Collector setup for iSCSI-Booted Hosts

ESXi hosts booted with iSCSI using the VMware iSCSI software initiator need to be configured to do core dumps to the ESXi Dump Collector that is part of vCenter. The Dump Collector is not enabled by default on the vCenter Appliance. To setup the ESXi Dump Collector, complete the following steps:

1.    Log into the vSphere web client as Administrator@flexpod.local.

2.    In the vSphere web client, select Home.

3.    In the center pane, click System Configuration.

4.    In the left pane, select Services.

5.    Under services, click VMware vSphere ESXi Dump Collector.

6.    In the center pane, click the green start icon to start the service.

7.    In the Actions menu, click Edit Startup Type.

8.    Select Automatic.

9.    Click OK.

10.  Connect to each ESXi host via ssh as root

11.  Run the following commands:

Add APIC-Integrated vSphere Distributed Switch (vDS)

The APIC-Integrated vDS is an integration between the Cisco ACI fabric and VMware allowing EPGs to be created in the ACI fabric and pushed into the vDS as port groups. The Virtual Machine Manager (VMM) domain in the APIC is configured with a pool of VLANs (50 in this validation) that are used as EPGs are assigned as port groups by associating the VMM domain with the EPG.  These VLANs are already assigned to the UCS server vNICs and piped through Cisco UCS.

Create Virtual Machine Manager (VMM) Domain in APIC

1.    In the APIC GUI, select Virtual Networking > Inventory.

2.    On the left, expand VMM Domains > VMware.

3.    Right-click VMware and select Create vCenter Domain.

4.    Name the Virtual Switch fpv-vc-vDS.  Leave VMware vSphere Distributed Switch selected.

5.    Select the “FPV-UCS-L2_AttEntityP” Associated Attachable Entity Profile.

6.    Under VLAN Pool, select Create VLAN Pool.

7.    Name the VLAN Pool FPV-VC-DS.  Leave Dynamic Allocation selected.

8.    Click the “+” to add a block of VLANs to the pool.

9.    Enter the VLAN range <1100-1150> and click OK.

10.  Click Submit to complete creating the VLAN Pool.

11.  Click the “+” to the right of vCenter Credentials to add credentials for the vCenter.

12.  For name, enter the vCenter hostname <fpv-vc>.  For the username and password, the vCenter Admin, or AD established account with the appropriate privileges.

13.  Click OK to complete creating the vCenter credentials.

14.  Click the “+” to the right of vCenter to add the vCenter linkage.

15.  Enter the vCenter hostname for Name.  Enter the vCenter FQDN or IP address.

16.  Leave DVS Version as vCenter Default. 

17.  Enable Stats Collection.

18.  For Datacenter, enter the exact Datacenter name specified in vCenter.

19.  Do not select a Management EPG.

20.  For Associated Credential, select the vCenter credentials entered in step 13.

21.  Click OK to complete the vCenter linkage.

22.  For Port Channel Mode, select MAC Pinning-Physical-NIC-load.

23.  For vSwitch Policy, select LLDP.

24.  Click Submit to complete creating the vCenter Domain.

Attaching the VMM Domain to the IB-Mgmt EPG

1.    In the APIC GUI, select Tenants > common.

2.    Under Tenant common, expand Application Profiles > FPV-Common-IB-Mgmt > Application EPGs > FPV-Common-Services.

3.    Under the FPV-Common-Services EPG, right-click Domains and select Add VMM Domain Association.

4.    Use the drop-down list to select the fpv-vc-vDS VMM Domain Profile.  Select On Demand Deploy Immediacy and Pre-provision Resolution Immediacy.

5.    Click Submit to create the FPV-Common-Services port group in the vDS.

 

Add ESXi Hosts to APIC-Integrated vDS

1.    Log into the vCenter vSphere Web Client with the Admin user.

2.    Under the Navigator on the left, select the Networking icon.

3.    Expand the vCenter, Datacenter, and vDS folder. Right-click the vDS and select Add and Manage Hosts.

4.    Select Add hosts and click Next.

5.    Click the to add New hosts.

6.    Select all four hosts and click OK.

7.    Click Next.

8.    Select only Manage physical adapters and click Next.

9.    On both hosts, assign vmnic4 as uplink1 and vmnic 5 as uplink2.  Click Next.

10.  Click Next and Finish to complete adding all the ESXi hosts to the vDS.  VMs can now be assigned to the FPV-Common-Services port group in the vDS.

The following table lists the VLANs, Subnets, and Bridge Domains for the sample application tenant called FPV-App-A set up as part of this lab validation:

Table 12      Tenant FPV-App-A Configuration

FPV-App-A Tenant Setup

This tenant will host the three application tiers of the sample three-tier application. This tenant will be hosted on the existing ESXi servers and will utilize the infrastructure datastore.

To deploy the FPV-App-A Tenant, complete the following steps:

1.    In the APIC GUI, at the top select Tenants > Add Tenant.

2.    Name the Tenant FPV-App-A. Select the default Monitoring Policy.

3.    For the VRF Name, also enter FPV-App-A.  Leave the Take me to this tenant when I click finish checkbox checked.

4.    Click Submit to finish creating the Tenant.

Adding 3-Tier-Application Profile

The Application Profile described below provides Web, App and DB EPGs to deploy a sample 3-tier application.

1.    On the left, right-click Application Profiles and select Create Application Profile.

2.    Name the Application Profile 3-Tier-App and select the default Monitoring Policy.

3.    Click Submit to complete creating the Application Profile.

4.    Expand 3-Tier-App, right-click Application EPGs under 3-Tier-App and select Create Application EPG.

5.    Name the EPG Web and leave Intra EPG Isolation set at Unenforced.

6.    Use the Bridge Domain drop-down list to select create Bridge Domain.

7.    Name the Bridge Domain BD-Internal and select the FPV-App-A VRF.

8.    Change Forwarding to Custom, leave Hardware Proxy for L2 Unknown Unicast, and leave Flood for L3 Unknown Multicast Flooding.

9.    Click Next.

10.  Leave L3 Configurations settings at their defaults, and click Next.

11.  Select the default Monitoring Policy and click Finish to complete creating the Bridge Domain.

12.  Click Finish to complete creating the EPG.

13.  On the left expand 3-Tier-App, Application EPGs, and EPG Web.

14.  Under EPG Web, right-click Domains and select Add VMM Domain Association.

15.  Select the fpv-vc-vDS VMM Domain Profile.

16.  Select Immediate for both the Deploy Immediacy and the Resolution Immediacy. Leave VLAN Mode set at Dynamic.

17.  Click Submit to compete adding the VMM Domain Association.

18.  On the left under EPG Web, right-click Contracts and select Add Provided Contract.

19.  In the Add Provided Contract window, use the Contract drop-down list to select Create Contract.

20.  Name the Contract Allow-Web-App. Select the Application Profile Scope.

21.  Click the “+” sign to add a Contract Subject.

22.  Name the subject Allow-All.

23.  Click the “+” sign to add a Contract filter.

24.  Use the drop-down list to select the Allow-All filter from Tenant common.  Click Update.

25.  Click OK to complete creating the Contract Subject.

26.  Click Submit to complete creating the Contract.

27.  Click Submit to complete adding the Provided Contract.

28.  Right-click Contracts and select Add Consumed Contract.

29.  In the Add Consumed Contract window, use the Contract drop-down list to select the common/Allow-Shared-L3Out contract.

30.  Click Submit to complete adding the Consumed Contract.

31.  Optionally, repeat steps 169-171 to add the common/FPV-Allow-Common-Services Consumed Contract.

32.  On the left under EPG Web, right-click Subnets and select Create EPG Subnet.

33.  For the Default Gateway IP, enter a gateway IP address and mask from a subnet in the Supernet (172.16.0.0/16) that was set up for assigning Tenant IP addresses.

34.  For scope, select Advertised Externally and Shared between VRFs.

35.  Click Submit to complete creating the EPG Subnet.

36.  Right-click Application EPGs under 3-Tier-App and select Create Application EPG.

37.  Name the EPG App and leave Intra EPG Isolation set at Unenforced.

38.  Use the Bridge Domain drop-down list to select BD-Internal within the current Tenant. Select the default Monitoring Policy.

39.  Click Finish to complete creating the EPG.

40.  On the left expand 3-Tier-App, Application EPGs, and EPG App.

41.  Under EPG App, right-click Domains and select Add VMM Domain Association.

42.  Select the fpv-vc-vDS VMM Domain Profile.

43.  Select Immediate for both the Deploy Immediacy and the Resolution Immediacy. Select the Dynamic VLAN Mode.

44.  Click Submit to compete adding the VMM Domain Association.

45.  On the left under EPG App, right-click Contracts and select Add Provided Contract.

46.  In the Add Provided Contract window, use the Contract drop-down list to select Create Contract.

47.  Name the Contract Allow-App-DB. Select the Application Profile Scope.

48.  Click the “+” sign to add a Contract Subject.

49.  Name the subject Allow-All.

50.  Click the “+” sign to add a Contract filter.

51.  Use the drop-down list to select the Allow-All filter from Tenant common.  Click Update.

52.  Click OK to complete creating the Contract Subject.

53.  Click Submit to complete creating the Contract.

54.  Click Submit to complete adding the Provided Contract.

55.  Right-click Contracts and select Add Consumed Contract.

56.  In the Add Consumed Contract window, use the Contract drop-down list to select the Allow-Web-App contract in the current tenant.

57.  Click Submit to complete adding the Consumed Contract.

58.  Optionally, repeat steps 196-198 to add the common/FPV-Allow-Common-Services Consumed Contract.

59.  On the left under EPG App, right-click Subnets and select Create EPG Subnet.

60.  For the Default Gateway IP, enter a gateway IP address and mask from a subnet in the Supernet (172.16.0.0/16) that was set up for assigning Tenant IP addresses.

61.  If this EPG was connected to FPV-Allow-Common-Services by contract, select only the Shared between VRFs scope.  Otherwise, if the tenant SVM management interface will only be accessed from EPGs within the tenant, leave only the Private to VRF Scope selected.

62.  Click Submit to complete creating the EPG Subnet.

63.  Right-click Application EPGs under 3-Tier-App and select Create Application EPG.

64.  Name the EPG DB and leave Intra EPG Isolation set at Unenforced.

65.  Use the Bridge Domain drop-down list to select BD-Internal in the current tenant. Select the default Monitoring Policy.

66.  Click Finish to complete creating the EPG.

67.  On the left expand 3-Tier-App, Application EPGs, and EPG DB.

68.  Under EPG DB, right-click Domains and select Add VMM Domain Association.

69.  Select the fpv-vc-vDS VMM Domain Profile.

70.  Select Immediate for both the Deploy Immediacy and the Resolution Immediacy. Select the Dynamic VLAN Mode.

71.  Click Submit to compete adding the VMM Domain Association.

72.  On the left under EPG DB, right-click Contracts and select Add Consumed Contract.

73.  In the Add Consumed Contract window, use the Contract drop-down list to select the Allow-App-DB contract in the current tenant.

74.  Click Submit to complete adding the Consumed Contract.

75.  On the left under EPG DB, right-click Subnets and select Create EPG Subnet.

76.  For the Default Gateway IP, enter a gateway IP address and mask from a subnet in the Supernet (172.16.0.0/16) that was set up for assigning Tenant IP addresses.

77.  If the FPV-Allow-Common-Services contract was consumed, select only the Shared between VRFs scope. Otherwise, select only the Private to VRF scope.

78.  Click Submit to complete creating the EPG Subnet.

All three EPGs configured for hosting the 3-Tier application are attached to the Virtual Machine Manager domain. At the end of these configuration steps, three port-groups will be available on the APIC controlled vDS. Customers can deploy VMs in the appropriate EPGs by attaching the VMs in VMware environment with the correct port-groups.

The validation of the environment was centered on resiliency tests of the Multi-Pod environment.  Storage, compute and network failover simulations were run under fair load to verify application continuity when possible. The current FlexPod datacenter system, like the previous FlexPod datacenter solutions is highly redundant and system survived with little or no impact to applications under most of the testing scenarios outline in Table 13.  Workload was generated using IOmeter while the failures were introduced via cable pulls, interface shutdowns, and the powering off the equipment.  Various failure test scenarios covered during validation are shown below. Functionality verification test cases are not covered in the Table 13:

Table 13      Validation Test Scenarios

Data Center Failure and Recovery

One of the major failure scenarios covered during the FlexPod Datacenter with Cisco ACI Multi-Pod and NetApp MetroCluster IP CVD validation was simulating a power outage in one of the datacenters resulting in failing the applications over to the second site. Since this failure scenario covers network, compute, virtualization and storage failure, various observations are outlined in this section for a better customer understanding.

Validation Environment Before Site Failure

Before any system failure occurs, the validation environment is in the following state:

·         All physical components were up and working as expected.

·         POD 11 was considered the primary site for the application being tested and this site would be powered off

·         POD 1 was considered the secondary site and this site would be used to restore the application VMs from primary site

·         The vCenter and the backup AD was hosted in the secondary site

·         Primary AD and the application VMs (Linux based) were hosted in the primary site.

Primary Site Failure

In the primary site, power was shut down for all the devices.

View from vCenter

From the vCenter hosted in the secondary site, the hosts, VMs and datastores in the primary site are no longer accessible as shown in Figure 15 and Figure 16:

Figure 15     Primary Site Hosts and VMs Inaccessible

Figure 16     Primary Site Datastores Inaccessible

View from NetApp AFF A700

From the AFF A700 in the secondary site, the primary site is not reachable:

Storage Failover Initiation

To restore the application, the datastores from primary site need to be brought up at the secondary site. To perform the storage system recovery, the MetroCluster failover has to be initiated manually. The command to initiate the switchover is entered on the secondary site AFF A700.

VMware HA

When the datastores are available at the secondary site, vSphere HA kicks in and the application VMs and the primary AD VM is brought up on secondary site hosts.

Validation Environment after Site Failure

After the application running in the primary site is recovered and is running in the secondary site, the validation environment is in the following state:

·         All physical components in POD11 (40GbE site) are down.

·         All the VMs including vCenter and AD are hosted in the secondary site.

·         ESXi hosts and datastores from the primary site (POD 11) are inaccessible.

Restoring the Primary Site

With the application and storage from the primary site running on the secondary site, the primary site is powered back up. When the communication is restored, devices from two datacenters start communication with each other. On initial powerup of the primary site AFF A700, the MetroCluster IP state is changed to “waiting-for-switchback”:

Within the primary site, the AFF A700 will be ready after a few minutes of internal checks and boot up processes:

Giveback from the secondary site can be sped-up with the following commands:

View from NetApp AFF A700

With the primary site AFF A700 ready, as switchback recovery will need to be initiated:

Switchback of primary site datastore back from the secondary site:

View from vCenter

The switchback brings a full return to functionality.  The VMs will need to vMotion back over to 40 GbE site when the datastore services via NFS have returned to normal.

Validation Environment after Failback

After the environment is fully restored and VMs are migrated back to their original sites, the validation environment is back to the same state where:

·         All physical components were up and working as expected.

·         The vCenter and the backup AD are hosted in the secondary site

·         Primary AD and the application VMs (Linux based) are hosted in the primary site.

In this failure scenario, various components of the solutions were successfully validate including:

·         Multi-Pod configuration and design

·         Routing inside and outside ACI fabric

·         Layer-2 extension between the two datacenters

·         MetroCluster failover and recovery

·         VMware HA

·         Storage and VM vMotion

FlexPod Datacenter with Cisco ACI Multi-Pod, NetApp MetroCluster IP, and VMware vSphere 6.7 solution allows interconnecting and centrally managing two or more ACI fabrics deployed in separate, geographically dispersed datacenters. NetApp MetroCluster IP provides a synchronous replication solution between two NetApp controllers providing storage high availability and disaster recovery in a campus or metropolitan area. This validated design enables customers to quickly and reliably deploy VMware vSphere based private cloud on a distributed integrated infrastructure thereby delivering a unified solution which enables multiple sites to behave in much the same way as a single site.

The validated solution achieves the following core design goals:

·         Campus-wide and metro-wide protection and provide WAN based disaster recovery

·         Design supporting active/active deployment use case

·         Common management layer across multiple (two) datacenters for deterministic deployment

·         Consistent policy and seamless workload migration across the sites

Products and Solutions

Cisco Unified Computing System:

http://www.cisco.com/en/US/products/ps10265/index.html

Cisco UCS 6200 Series Fabric Interconnects:

http://www.cisco.com/en/US/products/ps11544/index.html

Cisco UCS 6200 Series Fabric Interconnects:

https://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-6300-series-fabric-interconnects/index.html

Cisco UCS 5100 Series Blade Server Chassis:

http://www.cisco.com/en/US/products/ps10279/index.html

Cisco UCS B-Series Blade Servers:

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-b-series-blade-servers/index.html

Cisco UCS C-Series Rack Servers:

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-c-series-rack-servers/index.html

Cisco UCS Adapters:

http://www.cisco.com/en/US/products/ps10277/prod_module_series_home.html

Cisco UCS Manager:

http://www.cisco.com/en/US/products/ps10281/index.html

Cisco Nexus 9000 Series Switches:

http://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/tsd-products-support-series-home.html

Cisco Application Centric Infrastructure:

http://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/index.html

VMware vCenter Server:

http://www.vmware.com/products/vcenter-server/overview.html

NetApp Data ONTAP:

http://www.netapp.com/us/products/platform-os/ontap/index.aspx

 NetApp AFF A700:

https://www.netapp.com/us/products/storage-systems/all-flash-array/aff-a-series.aspx

NetApp MetroCluster IP:

https://www.netapp.com/us/products/backup-recovery/metrocluster-bcdr.aspx

Interoperability Matrixes

Cisco UCS Hardware Compatibility Matrix:

https://ucshcltool.cloudapps.cisco.com/public/

VMware Compatibility Guide:

http://www.vmware.com/resources/compatibility

NetApp Interoperability Matric Tool:

http://mysupport.netapp.com/matrix/

Haseeb Niazi, Technical Marketing Engineer, Cisco Systems, Inc.

Haseeb Niazi has over 19 years of experience at Cisco in the Data Center, Enterprise and Service Provider Solutions and Technologies. As a member of various solution teams and Advanced Services, Haseeb has helped many enterprise and service provider customers evaluate and deploy a wide range of Cisco solutions. As a technical marking engineer at Cisco UCS Solutions group, Haseeb focuses on network, compute, virtualization, storage and orchestration aspects of various Compute Stacks. Haseeb holds a master's degree in Computer Engineering from the University of Southern California and is a Cisco Certified Internetwork Expert (CCIE 7848).

Ramesh Issac, Technical Marketing Engineer, Cisco Systems, Inc.

Ramesh Isaac is a Technical Marketing Engineer in the Cisco UCS Data Center Solutions Group. Ramesh has worked in data center and mixed-use lab settings since 1995. He started in information technology supporting Unix environments and focused on designing and implementing multi-tenant virtualization solutions in Cisco labs before entering Technical Marketing.  Ramesh holds certifications from Cisco, VMware, and Red Hat.

Arvind Ramakrishnan, Solutions Architect, NetApp, Inc.

Arvind Ramakrishnan works for the NetApp Infrastructure and Cloud Engineering team. He is focused on the development, validation and implementation of Cloud Infrastructure solutions that include NetApp products. Arvind has more than 10 years of experience in the IT industry specializing in Data Management, Security and Data Center technologies. Arvind holds a bachelor’s degree in Electronics and Communication.

Acknowledgements

·         John George, Technical Marketing Engineer, Cisco

·         Archana Sharma, Technical Marketing Engineer, Cisco

·         Aaron Kirk, Product Manager, NetApp

NOTE: Available paragraph styles are listed in the Quick Styles Gallery in the Styles group on the Home tab. Alternatively, they can be accessed via the Styles window (press Alt + Ctrl + Shift + S).