Cisco HyperFlex Virtual Server Infrastructure 3.0 with Cisco ACI 3.2 and VMware vSphere 6.5
Available Languages
Cisco HyperFlex Virtual Server Infrastructure 3.0 with Cisco ACI 3.2 and VMware vSphere 6.5
Design and Deployment Guide for Cisco HyperFlex 3.0 with VMware vSphere 6.5U2, Cisco UCS Manager 3.2, Cisco ACI 3.2, and Cisco UCS 6300 Series Fabric Interconnects
Last Updated: July 25, 2019
About the Cisco Validated Design Program
The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to:
http://www.cisco.com/go/designzone.
ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.
CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.
All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)
© 2019 Cisco Systems, Inc. All rights reserved.
Table of Contents
Technology and Components Overview
Cisco Unified Computing System
Cisco UCS Fabric Interconnects
Cisco UCS 1227 and 1387 Virtual Interface Cards
Cisco Application Centric Infrastructure and Nexus Switching
Cisco ACI Fabric Design Constructs
Cisco HyperFlex Networking Design
HyperFlex Design Guidelines and Best Practices
HyperFlex Installer Requirements
ACI Fabric – Core Connectivity
ACI Fabric – Edge Connectivity
ACI Fabric Design – Access Policies
ACI Fabric Design – EPGs, Contracts and Application Profiles
ACI Fabric Design – Multi-tenancy
ACI Fabric – Recommended Fabric Settings
ACI Fabric Design – Enabling Access to Cisco HyperFlex Installer and Existing Infrastructure
ACI Fabric Design – Enabling Access to Cisco UCS Domain and HyperFlex Nodes
ACI Fabric Design – Enabling Foundational Infrastructure for Cisco HyperFlex System
ACI Fabric Design - Integration with Virtual Machine Manager
ACI Fabric Design – Onboarding Multi-Tier Applications
Validated Software and Hardware Components
Solution Deployment – Data Center Network Fabric
Review Existing ACI Fabric Topology
Verify Time Zone and NTP Setup
Verify BGP Route Reflector Setup
Enable IP Aging (System Wide Setting)
COS Preservation (Fabric Wide Setting)
ACI Fabric Access – Pre-configure Fabric Access Policies
Deploy New Leaf Switches for HyperFlex Environment
ACI Fabric Discovery of Leaf Switches
Add Nexus 9000 Series Leaf Switches to the ACI Fabric
Setup Out-of-Band Management for New Leaf Switches
Enabling Access to Cisco HyperFlex Installer and Existing Infrastructure
ACI Fabric Deployment – Layer 3 Routed Connectivity to Outside Networks
Enable Access to Cisco UCS Domain and HyperFlex Nodes
Create Virtual Port Channels to Cisco UCS Domain for Cisco HyperFlex Deployment
Enabling Foundational Infrastructure for Cisco HyperFlex System
Create Foundation Tenant and VRF
Configure ACI Fabric for HyperFlex In-Band Management
Configure ACI Fabric for HyperFlex Storage Data Traffic
Configure ACI Fabric for HyperFlex vMotion Traffic
Solution Deployment – Cisco HyperFlex and Cisco UCS
Bring up UCS Domain with Fabric Interconnects
Deploy HX Installer OVA in an Existing Virtualization Environment
Auto-Support and Notifications
Deploy VMware vCenter Plugin to Manage HyperFlex (Optional)
Create Datastores for Virtual Machines
Cisco Intersight Cloud-Based Management
Solution Deployment – Virtual Networking
Deploying APIC-controlled VMware vSphere vDS for HyperFlex Application Traffic
Add HyperFlex ESXi Hosts to VMware vSphere vDS
Deploying Cisco ACI Plug-in for Managing ACI from VMware vCenter
Prerequisites for Cisco ACI Plug-in
Verify the ACI Plug-in Installation from VMware vCenter
Connecting ACI Plug-in to ACI Fabric
Deploying APIC-controlled Cisco AVE for HyperFlex Application Traffic
Add HyperFlex ESXi Hosts to Cisco AVE
Download Cisco AVE OVF file to VMware vCenter
Setup Management Networking for Cisco AVE Virtual Machines
Deploy Cisco AVE VM on the ESXi Hosts Using the Cisco ACI Plug-In
Solution Deployment – Onboarding Multi-Tier Applications
Create Tenant and VRF for Application
Verify Virtual Networking for the Application EPGs
Web-Tier to Shared L3 Out Contract
Nexus 7000 – Sample Configuration
Datacenter Network Fabric – Cisco ACI
Cisco HyperFlex™ delivers a hyper-converged infrastructure solution with software-defined computing and networking based on Cisco Unified Computing Systems ™ (Cisco UCS ®) and software-defined storage based on Cisco HyperFlex HX data Platform software. Cisco HyperFlex HX Data Platform is a purpose-built, high-performance, distributed scale-out file system that delivers complete hyperconvergence with enterprise storage features. Cisco HyperFlex systems deliver agility, scalability, and pay-as-you-grow economics of the cloud with the benefits of on-premises infrastructure.
Cisco ACI is an industry leading software defined networking (SDN) solution that delivers application agility and data center automation for building multi-cloud networks. Cisco ACI delivers a transformational operating model for next-generation data centers with an innovative architecture that radically simplifies, optimizes and accelerates the entire application deployment lifecycle. Cisco ACI enables an automated, policy-based network deployment that secures your applications within secure, isolated containers. By attaching a Cisco HyperFlex cluster to Cisco ACI and integrating with Virtual Machine Managers such as VMware vCenter, Cisco ACI can orchestrate the virtual networking for the virtual machines hosted on Cisco HyperFlex and extend the Cisco ACI policies and security to the virtualization layer.
This Cisco Validated Design (CVD) details how to attach a Cisco HyperFlex cluster to an existing Cisco ACI fabric. Cisco Validated Designs are systems and solutions that are designed, tested, and documented to accelerate customer deployments at reduced risk. Cisco Validated designs are developed to addressed the needs of a business and incorporate a wide range of technologies, products and best practices.
The solution presented in this document is based on Cisco HyperFlex 3.0, Cisco UCS Manager 3.2, VMware vSphere 6.5, and Cisco ACI 3.2.
Introduction
Industry trends indicate a transformation towards a hyperconvergence and software-defined architectures in the Enterprise datacenters. Businesses require application agility where applications need to be provisioned quickly with the ability to quickly scale resources up (or down) in minutes.
Cisco HyperFlex with Cisco ACI is a hyperconverged data center solution for VMware vSphere environments based on software-defined computing, storage and network. This architecture is built using Cisco HyperFlex, Cisco Unified Computing System (UCS), Cisco ACI running on Cisco Nexus family of switches and VMware vSphere. This solution is designed to facilitate and accelerate the evolution to a hyperconverged data center infrastructure based on Cisco ACI’s application driven policy model with centralized management and automation.
Audience
The audience for this document includes, but is not limited to; sales engineers, field consultants, professional services, IT managers, partner engineers, and customers that are interested in leveraging industry trends towards hyperconvergence and software-defined networking to build agile infrastructures that can be deployed in minutes and keep up with business demands.
Purpose of this Document
This document provides a reference architecture for creating a Virtual Server Infrastructure solution for a VMware vSphere environment using hyperconvergence and software-defined networking. The design uses a Cisco ACI based data center fabric and Cisco Hyperflex for the hyperconverged infrastructure. The guide covers the end-to-end design as well as the implementation steps for deploying a Cisco HyperFlex system connected to a Cisco ACI fabric. The design and deployment is based on product and technology level best practices and leverages the latest software, hardware and firmware revisions available as of the writing of this document. Recommendations and best practices may be amended in later revisions.
What’s New in this Release?
This release of Cisco HyperFlex Virtual Server Infrastructure solution introduces Cisco ACI 3.2 for delivering a comprehensive, application centric data center architecture with centralized, policy-driven and application-centric management and orchestration of the data center network fabric to provide network connectivity to the Cisco HyperFlex system. The Cisco HyperFlex system validated in this design uses HyperFlex All-Flash nodes with Intel Xeon Scalable M5 processors, running Cisco HyperFlex software release 3.0. The HyperFlex cluster connects to the Cisco ACI fabric through a pair of Cisco UCS 6300 series Fabric interconnects running Cisco UCS Manager 3.2. The design uses 40Gb Ethernet for end-to-end connectivity.
Solution Summary
The Cisco HyperFlex Virtual Server Infrastructure (VSI) provides is a validated reference architecture, with a fully contained virtual server platform with compute and memory resources, integrated networking connectivity, a distributed high-performance log based filesystem for VM storage, and the hypervisor software for running the virtualized servers, all within a single Cisco UCS management domain. The solution is built to deliver a VMware vSphere based environment for Enterprise data centers, leveraging Cisco HyperFlex, Cisco Unified Computing System (Cisco UCS) and Cisco ACI implemented on Cisco Nexus 9000 series switches as shown in Figure 1.
Figure 1 Cisco HyperFlex with Cisco ACI Solution
This solution consists of the following family of components in the compute, storage and networking layers of the design:
· Cisco HyperFlex HX-series nodes
· Cisco Unified Computing System (Cisco UCS)
· Cisco ACI fabric built using Nexus 9000 family of switches
For a given reference architecture, specific models from the above component families are used to validate the reference architecture.
The components available from the above component families for this reference architecture are:
· One pair of Cisco UCS Fabric Interconnects, choose from models:
- Cisco UCS 6248UP Fabric Interconnect
- Cisco UCS 6296UP Fabric Interconnect
- Cisco UCS 6332 Fabric Interconnect
- Cisco UCS 6332-16UP Fabric Interconnect
· Three to Thirty-Two Cisco HyperFlex HX-Series Rack-Mount Servers, choose from models:
- Cisco HyperFlex HX220c-M5SX Rack-Mount Servers
- Cisco HyperFlex HX240c-M5SX Rack-Mount Servers
- Cisco HyperFlex HX240c-M5L Rack-Mount Servers
- Cisco HyperFlex HXAF220c-M5SX All-Flash Rack-Mount Servers
- Cisco HyperFlex HXAF240c-M5SX All-Flash Rack-Mount Servers
- Cisco HyperFlex HX220c-M4S Rack-Mount Servers
- Cisco HyperFlex HX240c-M4SX Rack-Mount Servers
- Cisco HyperFlex HXAF220c-M4S All-Flash Rack-Mount Servers
- Cisco HyperFlex HXAF240c-M4SX All-Flash Rack-Mount Servers
· Cisco HyperFlex Data Platform Software
· VMware vSphere ESXi Hypervisor
· VMware vCenter Server (end-user supplied)
Optional components for additional compute-only resources are:
· Cisco UCS 5108 Chassis
· Cisco UCS 2204XP, 2208XP or 2304 model Fabric Extenders
· Cisco UCS B200-M3, B200-M4, B200-M5, B260-M4, B420-M4, B460-M4 or B480-M5 blade servers
· Cisco UCS C220-M3, C220-M4, C220-M5, C240-M3, C240-M4, C240-M5, C460-M4 or C480-M5 rack-mount servers
For the specific models used to validate the reference architecture covered in this document. See the Solution Validation section of this document for more details.
When selecting specific models for a deployment, confirm support for the component using Cisco’s Hardware Compatibility List (HCL).
This section provides a technical overview of the compute, network, storage and management components used in this data center architecture.
Cisco Unified Computing System
Cisco Unified Computing System™ (Cisco UCS) is a next-generation data center platform that integrates computing, networking, storage access, and virtualization resources into a cohesive system designed to reduce total cost of ownership and increase business agility. The system integrates a low-latency, lossless 10 or 40 Gigabit Ethernet unified network fabric with enterprise-class, x86-architecture servers. The system is an integrated, scalable, multi-chassis platform with a unified management domain for managing all resources.
Cisco Unified Computing System consists of the following subsystems:
· Compute - The compute piece of the system incorporates servers based on latest Intel’s x86 processors. Servers are available in blade and rack form factor, managed by Cisco UCS Manager.
· Network - The integrated network fabric in the system provides a low-latency, lossless, 10/40 Gbps Ethernet fabric. Networks for LAN, SAN and management access are consolidated within the fabric. The unified fabric uses the innovative Single Connect technology to lowers costs by reducing the number of network adapters, switches, and cables. This in turn lowers the power and cooling needs of the system.
· Virtualization - The system unleashes the full potential of virtualization by enhancing the scalability, performance, and operational control of virtual environments. Cisco security, policy enforcement, and diagnostic features are now extended into virtual environments to support evolving business needs.
· Storage access – Cisco UCS system provides consolidated access to both SAN storage and Network Attached Storage over the unified fabric. This provides customers with storage choices and investment protection. Also, the server administrators can pre-assign storage-access policies to storage resources, for simplified storage connectivity and management leading to increased productivity.
· Management: The system uniquely integrates compute, network and storage access subsystems, enabling it to be managed as a single entity through Cisco UCS Manager software. Cisco UCS Manager increases IT staff productivity by enabling storage, network, and server administrators to collaborate on Service Profiles that define the desired physical configurations and infrastructure policies for applications. Service Profiles increase business agility by enabling IT to automate and provision resources in minutes instead of days.
Cisco UCS Differentiators
Cisco Unified Computing System has revolutionized the way servers are managed in data-center and provides a number of unique differentiators that are outlined below:
· Embedded Management — Servers in Cisco UCS are managed by embedded software in the Fabric Interconnects, eliminating need for any external physical or virtual devices to manage the servers.
· Unified Fabric — Cisco UCS uses a wire-once architecture, where a single Ethernet cable is used from the FI from the server chassis for LAN, SAN and management traffic. Adding compute capacity does not require additional connections. This converged I/O reduces overall capital and operational expenses.
· Auto Discovery — By simply inserting a blade server into the chassis or a rack server to the fabric interconnect, discovery of the compute resource occurs automatically without any intervention.
· Policy Based Resource Classification — Once a compute resource is discovered, it can be automatically classified to a resource pool based on policies defined which is particularly useful in cloud computing.
· Combined Rack and Blade Server Management — Cisco UCS Manager is hardware form factor agnostic and can manage both blade and rack servers under the same management domain.
· Model based Management Architecture — Cisco UCS Manager architecture and management database is model based and data driven. An open XML API is provided to operate on the management model which enables easy and scalable integration of Cisco UCS Manager with other management systems.
· Policies, Pools, and Templates — The management approach in Cisco UCS Manager is based on defining policies, pools and templates, instead of cluttered configuration, which enables a simple, loosely coupled, data driven approach in managing compute, network and storage resources.
· Policy Resolution — In Cisco UCS Manager, a tree structure of organizational unit hierarchy can be created that mimics the real-life tenants and/or organization relationships. Various policies, pools and templates can be defined at different levels of organization hierarchy.
· Service Profiles and Stateless Computing — A service profile is a logical representation of a server, carrying its various identities and policies. This logical server can be assigned to any physical compute resource as far as it meets the resource requirements. Stateless computing enables procurement of a server within minutes, which used to take days in legacy server management systems.
· Built-in Multi-Tenancy Support — The combination of a profiles-based approach using policies, pools and templates and policy resolution with organizational hierarchy to manage compute resources makes Cisco UCS Manager inherently suitable for multi-tenant environments, in both private and public clouds.
Cisco UCS Manager
Cisco UCS Manager (UCSM) provides unified, integrated management for all software and hardware components in Cisco UCS. Using Cisco Single Connect technology, it manages, controls, and administers multiple chassis for thousands of virtual machines. Administrators use the software to manage the entire Cisco Unified Computing System as a single logical entity through an intuitive graphical user interface (GUI), a command-line interface (CLI), or a through a robust application programming interface (API).
Cisco UCS Manager is embedded into the Cisco UCS Fabric Interconnects and provides a unified management interface that integrates server, network, and storage. Cisco UCS Manager performs auto-discovery to detect inventory, manage, and provision system components that are added or changed. It offers comprehensive set of XML API for third party integration, exposes thousands of integration points and facilitates custom development for automation, orchestration, and to achieve new levels of system visibility and control.
Cisco UCS Fabric Interconnects
The Cisco UCS Fabric interconnects (FIs) provide a single point for network connectivity and management for the entire Cisco UCS system. Typically deployed as an active-active pair, the Cisco UCS fabric interconnects integrate all components into a single, highly-available management domain controlled by the Cisco UCS Manager. All models of Cisco UCS servers and chassis, Cisco UCS C-Series, S-Series and HX-Series Rack-Mount Servers, Cisco UCS B-Series Blade Servers and Cisco UCS 5100 Series Blade Server Chassis attach to the Cisco UCS Fabric Interconnects to become a single unified fabric, with low-latency, lossless, cut-through switching for all servers within its domain. Cisco UCS Fabric Interconnects provide each server or chassis with LAN, SAN and management connectivity using a single set of cables.
Depending on the model chosen, the Cisco UCS Fabric Interconnect offers line-rate, low-latency, lossless 10 Gigabit or 40 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE) and Fibre Channel connectivity. The Cisco UCS 6200 Series Fabric Interconnects deliver lossless, line rate 10 Gigabit Ethernet on all ports with up to 1.92 Tbps switching capacity and 160 Gbps bandwidth per chassis, regardless of packet size and enabled services. The Cisco UCS 6300 Series Fabric Interconnects offers the same features while supporting even higher performance, low latency, lossless, line rate 40 Gigabit Ethernet, with up to 2.56 Tbps of switching capacity. Backward compatibility and scalability is assured by enabling 40 Gbps quad SFP (QSFP) ports to be configured as breakout ports with 4x10GbE breakout cables that can connect to Cisco UCS servers with 10GbE interfaces.
Cisco HyperFlex nodes must use a 40GbE VIC adapter to connect to Cisco UCS 6300 Series Fabric Interconnects.
Cisco UCS 6200 Series Fabric Interconnects
Cisco UCS 6200 Series FIs are a family of line-rate, low-latency, lossless switches that can support 1/10 Gigabit Ethernet and Fibre Channel over Ethernet (FCoE), or native (4/2/1 or 8/4/2) Fibre Channel (FC) connectivity.
Cisco UCS 6300 Series Fabric Interconnects
Cisco UCS 6300 Series Fabric Interconnects provide a 40Gbps unified fabric with higher workload density and double the switching capacity of Cisco UCS 6200 FIs. When combined with the newer 40Gbps Cisco UCS 2300 Series Fabric Extenders, they provide 40GbE / FCoE port connectivity to enable end-to-end 40GbE / FCoE solution. The unified ports also support 16G FC for high speed FC connectivity to SAN.
The two 6300 Fabric Interconnect models currently available are:
· Cisco UCS 6332 Fabric Interconnect is a 1RU 40GbE and FCoE switch offering up to 2.56 Tbps of throughput. The switch has a density of 32 x 40Gbps Ethernet, and FCoE ports. This model is aimed at IP storage deployments requiring high-performance 40Gbps FCoE connectivity to Cisco MDS switches.
· Cisco UCS 6332-16UP Fabric Interconnect is a 1RU 40GbE/FCoE switch and 1/10 Gigabit Ethernet, FCoE and FC switch offering up to 2.24 Tbps throughput. The switch has 24x40Gbps fixed Ethernet/FCoE ports and 16x1/10Gbps Ethernet/FCoE or 4/8/16Gbps FC ports. This model is aimed at FC storage deployments requiring high performance 16Gbps FC connectivity to Cisco MDS switches.
Comparison of Cisco UCS Fabric Interconnect Models
Table 1 provides a comparison of the port capabilities of the different Fabric Interconnect models.
Table 1 Cisco UCS 6200 and 6300 Series Fabric Interconnects
| Features |
6248 |
6296 |
6332 |
6332-16UP |
| Max 10G ports |
48 |
96 |
96* + 2** |
72* + 16 |
| Max 40G ports |
- |
- |
32 |
24 |
| Max unified ports |
48 |
96 |
- |
16 |
| Max FC ports |
48 x 2/4/8G FC |
96 x 2/4/8G FC |
- |
16 x 4/8/16G FC |
* Using 40G to 4x10G breakout cables ** Requires QSA module
In a Cisco HyperFlex deployment, due to mandatory QoS settings in the configuration, the 6332 and 6332-16UP will be limited to a maximum of four 4x10Gbps breakout ports, which can be used for other non-HyperFlex servers.
For additional information on Cisco UCS Fabric Interconnects, see the Solution References section at the end of this document.
Cisco HyperFlex
Cisco HyperFlex expands on the innovations in the Cisco UCS platform to deliver a hyperconverged system with the agility, scalability and pay-as-you-grow economics of the cloud but with the benefits of on-premise infrastructure. Cisco HyperFlex is part of Cisco’s data center vision and strategy to support common policies and management across a range of traditional, converged and hyperconverged systems. Similar to Cisco UCS, Cisco HyperFlex provides unified compute, storage, networking and management by connecting through a pair of Cisco UCS Fabric Interconnects. The single point of connectivity integrates all Cisco HyperFlex HX-series nodes into a single unified cluster that can be managed from the embedded Cisco UCS manager running on the Fabric Interconnects and the HTML5-based Cisco HyperFlex Connect management tool or from the cloud using Cisco Intersight.
Cisco HyperFlex is designed to be a pre-integrated end-to-end software-defined infrastructure solution that combines:
· Software-defined computing using Cisco UCS servers with the latest generation Intel Xeon ™ Scalable processors
· Software-defined storage with Cisco HyperFlex HX Data Platform software
· Software-defined networking with Cisco’s unified fabric and integration with Cisco’s Application Centric Infrastructure
Cisco HyperFlex Architecture
Cisco HyperFlex delivers a common platform to support a wide range of applications and use cases, including virtualized and containerized applications, virtual desktops, big data environments, enterprise applications, databases, and Internet infrastructure applications, along with test and development environments. Cisco HyperFlex systems uses Hyperflex HX Data Platform software that is a purpose-built to deliver high-performance, distributed scale-out file system with enterprise-class storage features and services. The HX Data Platform is designed to support multiple hypervisors, containers, and bare metal to power existing and new cloud-native workloads.
Figure 2 Cisco HyperFlex Architecture
Cisco HyperFlex Data Platform
Cisco HyperFlex systems run on Cisco HyperFlex HX Data Platform (HXDP) software that combines the cluster’s solid-state disk (SSD) drives and hard-disk drives (HDDs) into a single distributed, multitier, object-based data store. The HX Data Platform uses a self-healing architecture that replicates data for high availability and can remediate around hardware failures. Hyperflex HX Data Platform is a purpose-built, high-performance, extensible distributed scale-out file system that supports multiple hypervisors with following enterprise-grade storage features and data management services.
· Data Management Features include deduplication, compression, thin-provisioning, fast space-efficient cloning, snapshots, native replication, and integration with leading backup solution vendors
· Continuous Data Optimization with data deduplication and compression
· Securely encrypted storage that optionally encrypts at both the caching and persistent layers of storage
· Dynamic Data placement that optimizes performance and resilience with all cluster resources participating in I/O.
· Cluster-wide parallel data distribution and replication for high availability and performance, accelerated by the low latency and high bandwidth of the Cisco UCS network fabric.
· Stretch clusters that can span geographic distances with no data loss and continuous operation in the event of a data center failure.
· Logical availability zones increase scalability and availability by automatically protecting your data against multiple component and node failures.
· Native replication creates remote copies of individual virtual machines for disaster recovery purposes. With all nodes in the local and remote clusters participating in the replication activity, the overhead on an individual node is minimal.
· Linear and incremental scaling provides faster, easier scalability compared to enterprise shared-storage systems, in which controller resources can become a bottleneck requiring upgrades to the storage system. However, in Cisco HyperFlex Systems, when you add an increment of storage, you also increment the data platform processing capacity. You can also independently scale compute, caching, and capacity, giving you full flexibility to scale your hyperconverged environment based on evolving business needs.
· API-based data platform architecture provides data virtualization flexibility to support existing and new cloud-native data types. Data protection API integration enables allows enterprise backup solutions to protect your data.
· Simplified approach to storage administration eliminates the need to configure LUNs or to require a storage administrator to configure SANs. Storage and data services are managed completely through hypervisor tools such as VMware vCenter.
Cisco HyperFlex Models
Cisco HyperFlex Platform includes both hybrid and all-flash models with powerful data optimization features that bring the full potential of hyperconvergence to a wide range of workloads and use cases.
A HyperFlex cluster requires a minimum of three HX-Series converged nodes (with disk storage). Data is replicated across at least two of these nodes, and a third node is required for continuous operation in the event of a single-node failure. Each node with disk storage is equipped with at least one high-performance SSD drive for data caching and rapid acknowledgment of write requests. Each node also is equipped with additional disks, up to the platform’s physical limit, for long term storage and capacity. HyperFlex cluster can also support compute-only nodes.
· Hybrid nodes use SSD drives for caching and HDDs for the capacity layer.
· All-flash nodes use SSD drives or NVMe storage for caching, and SSD drives for the capacity layer.
Hybrid and all-flash options are available in three types of nodes:
· Cisco HyperFlex HX220c M4/M5 Hybrid and All-Flash Nodes: These balance up to 6 or 8 drives of disk capacity in a 2-socket, 1-rackunit (1RU) package ideal for small-footprint environments.
· Cisco HyperFlex HX240c M4/M5 Hybrid and All-Flash Nodes: These balance high disk capacity (up to 23 drives and 12 on HX240c-M5L model) in a 2-socket, 2RU package ideal for storage-intensive applications.
· Cisco HyperFlex HX220c M4/M5 Edge Nodes: These nodes are designed to work in simplified three-node clusters using existing Gigabit Ethernet switches that provides the necessary flexibility for deployment in remote and branch-office environments. Cisco HyperFlex Edge systems offer the same easy deployment and management as all Cisco HyperFlex systems.
Table 2 Cisco HyperFlex HX-Series Nodes – All Flash Models
| All-Flash Models |
Boot Drive |
Data Logging or Housekeeping Drive |
Write-Log or Caching Drive |
Capacity Drives |
| HXAF220c-M5SX |
240GB M.2 form factor SSD |
240GB SSD |
1.6TB NVMe SSD -or- |
6-8 |
| HXAF240c-M5SX |
240GB M.2 form factor SSD |
240GB SSD |
375GB Optane NVMe -or- 1.6TB NVMe SSD -or- |
6-23 |
| HXAF220c-M4S |
2 Cisco FlexFlash SD cards |
120GB |
400GB NVMe SSD |
6 |
| HXAF240c-M4SX |
2 Cisco FlexFlash SD cards |
120GB |
400GB NVMe SSD |
6-23 |
Some important points to note about HyperFlex All-Flash nodes are summarized below:
· A minimum of three nodes and a maximum of sixteen nodes can be configured in one HX All-Flash cluster.
· For configurations requiring self-encrypting drives on All-Flash nodes, the caching SSD is replaced with an 800 GB SAS SED SSD, and the capacity disks are also replaced with either 800 GB, 960 GB or 3.8 TB SED SSDs on All-Flash nodes and 1.2TB SAS SED HDDs on Hybrid nodes.
· In M4 generation server All-Flash nodes, either a 400 GB or 800 GB caching SAS SSD may be chosen. This option is provided to allow flexibility in ordering based on product availability, pricing and lead times. There is no performance, capacity, or scalability benefit in choosing the larger disk.
· In M5 generation server All-Flash nodes, either a 375GB Optane NVMe SSD or 400 GB SAS SSD or 1.6 TB NVMe SSD caching drive may be chosen. While the NVMe option can provide a higher level of performance, the partitioning of the three disks is the same, therefore the amount of cache available on the system is the same regardless of the model.
Table 3 Cisco HyperFlex HX-Series Nodes – Hybrid Models
| Hybrid Models |
Boot Drive |
Data Logging or Housekeeping Drive |
Write-Log or Caching Drive |
Capacity Drives |
| HX220c-M5SX |
240GB M.2 form factor SSD |
240GB SSD |
480GB SSD -or- |
6-8 |
| HX240c-M5SX |
240GB M.2 form factor SSD |
240GB SSD |
1.6TB SSD |
6-23 |
| HX240c-M5L |
240GB M.2 form factor SSD |
240GB SSD |
3.2TB SSD |
6-12 |
| HX220c-M4S |
2 Cisco FlexFlash SD cards |
120GB -or- |
480GB SAS SSD |
6 |
| HXAF240c-M4SX |
2 Cisco FlexFlash SD cards |
120GB -or- |
1.6TB SAS SSD |
6-23 |
Some important points to note about HyperFlex Hybrid nodes are summarized below:
· A minimum of three nodes and a maximum of eight nodes can be configured in one HX Hybrid cluster.
· In M5 generation server Hybrid nodes, either a 400 GB or 800 GB caching SAS SSD may be chosen. This option is provided to allow flexibility in ordering based on product availability, pricing and lead times. There is no performance, capacity, or scalability benefit in choosing the larger disk.
· For configurations requiring self-encrypting drives on Hybrid nodes, the models that use 480 or 800GB caching SSDs are replaced with an 800 GB SAS SED SSD, and the capacity disks are replaced with 1.2TB SAS SED HDDs. On models that use the 1.6TB caching SSDs, they are replaced with 1.6TB SAS SED SSD and the capacity disks are replaces with 1.2TB SAS SED HDDs. HX240c-M5L does not support self-encrypting drives at this time.
Cisco HyperFlex Edge Models
Currently supported Edge node models are:
· M5 Models: HX220c M5SX All-Flash and HX220c M5SX Hybrid
· M4 Models: HX220c M4S All-Flash and HX220c M4S Hybrid
Table 4 Differences between Cisco HyperFlex (All-Flash/Hybrid) Models and Edge Models
Cisco HyperFlex Compute-Only Nodes
HyperFlex provides a number of compute-only options using standard models of Cisco UCS Rack-Mount and Blade Servers. All current models of Cisco UCS M4 and M5 generation servers, except the C880 M4 and M5, may be used as compute-only nodes connected to a Cisco HyperFlex cluster, along with a limited number of previous M3 generation servers. Any valid CPU and memory configuration is allowed in the compute-only nodes, and the servers can be configured to boot from SAN, local disks, or internal SD cards. The following servers maybe used as compute-only nodes in an existing HX-cluster (not supported in an Edge cluster):
· M5 Blade Server Models: Cisco UCS B200 M5 and B480 M5 Blade Servers
· M5 Rack-Mount Server Models: Cisco UCS C220 M5, C240 M5 and C480 M5 Rack-Mount Servers
· M4 Blade Server Models: Cisco UCS B200 M4, B260 M4, B420 M4, and B460 M4 Blade Servers
· M4 Rack-Mount Server Models: Cisco UCS C220 M4, C240 M4 and C460 M4 Rack-Mount Servers
· M3 Models: Cisco UCS B200 M3 Blade Server, Cisco UCS C220 M3 and C240 M3 Rack-Mount Servers
Cisco HyperFlex HX Data Platform Controller
A Cisco HyperFlex HX Data Platform controller resides on each node and implements the distributed file system. The controller runs as software in user space within a virtual machine, and intercepts and handles all I/O from the guest virtual machines. The Storage Controller Virtual Machine (SCVM) uses the VMDirectPath I/O feature to provide PCI passthrough control of the physical server’s SAS disk controller. This method gives the controller VM full control of the physical disk resources, utilizing the SSD drives as a read/write caching layer, and the HDDs or SDDs as a capacity layer for distributed storage. The controller integrates the data platform into the VMware vSphere cluster through the use of three preinstalled VMware ESXi vSphere Installation Bundles (VIBs) on each node:
· IO Visor: This VIB provides a network file system (NFS) mount point so that the ESXi hypervisor can access the virtual disks that are attached to individual virtual machines. From the hypervisor’s perspective, it is simply attached to a network file system. The IO Visor intercepts guest VM IO traffic, and intelligently redirects it to the HyperFlex SCVMs.
· VMware API for Array Integration (VAAI): This storage offload API allows vSphere to request advanced file system operations such as snapshots and cloning. The controller implements these operations by manipulating the filesystem metadata rather than actual data copying, providing rapid response, and thus rapid deployment of new environments.
· stHypervisorSvc: This VIB adds enhancements and features needed for HyperFlex data protection and VM replication.
Cisco UCS Network Adapters
Cisco UCS supports converged network adapters (CNAs) to provide connectivity to Cisco UCS HX-Series rack servers. CNAs eliminate the need for multiple network interface cards (NICs) and host bus adapters (HBAs) by converging LAN, SAN and Management traffic to a single interface. While Cisco UCS supports a wide variety of interface cards, the adapters currently available for Cisco UCS HX-series rack-servers are outlined below.
Cisco UCS 1227 and 1387 Virtual Interface Cards
The Cisco UCS Virtual Interface Card (VIC) 1227 is a dual-port Enhanced Small Form-Factor Pluggable (SFP+) 10-Gbps Ethernet and Fibre Channel over Ethernet (FCoE)-capable PCI Express (PCIe) modular LAN-on-motherboard (mLOM) adapter installed in the Cisco UCS HX-Series Rack Servers. The VIC 1227 is used in conjunction with the Cisco UCS 6248UP or 6296UP model Fabric Interconnects.
The Cisco UCS VIC 1387 Card is a dual-port Enhanced Quad Small Form-Factor Pluggable (QSFP+) 40-Gbps Ethernet and Fibre Channel over Ethernet (FCoE)-capable PCI Express (PCIe) modular LAN-on-motherboard (mLOM) adapter installed in the Cisco UCS HX-Series Rack Servers. The VIC 1387 is used in conjunction with the Cisco UCS 6332 or 6332-16UP model Fabric Interconnects.
The mLOM slot can be used to install a Cisco VIC without consuming a PCIe slot, which provides greater I/O expandability. It incorporates next-generation CNA technology from Cisco, providing investment protection for future feature releases. The card enables a policy-based, stateless, agile server infrastructure that can present up to 256 PCIe standards-compliant interfaces to the host, each dynamically configured as either a network interface card (NICs) or host bus adapter (HBA). The personality of the interfaces is set programmatically using the service profile associated with the server. The number, type (NIC or HBA), identity (MAC address and World Wide Name [WWN]), failover policy, adapter settings, bandwidth, and quality-of-service (QoS) policies of the PCIe interfaces are all specified using the service profile.
Table 5 Cisco UCS VIC 1227 and 1387 Interface Cards
| Cisco VIC 1227 mLOM Card |
Cisco VIC 1387 mLOM Card |
|
|
|
Hardware revision V03 or later of the Cisco VIC 1387 card is required for the Cisco HyperFlex HX-series servers.
For more information about Cisco UCS Network Adapters, see Solution References.
Cisco Application Centric Infrastructure and Nexus Switching
Cisco ACI is a data center architecture designed to address the requirements of today’s traditional networks, as well as to meet emerging demands that new application trends and business factors are placing on the network. Software-defined networking (SDN) has garnered much attention in the networking industry over the past few years due to its promise of a more agile and programmable network infrastructure. Cisco ACI not only delivers network agility and programmability but goes beyond SDN’s initial vision of operational efficiency by providing solutions and innovations that SDN does not.
Cisco ACI leverages a network fabric that uses industry proven protocols coupled with innovative technologies to create a flexible, scalable, and highly available architecture of low-latency, high-bandwidth links. The ACI fabric delivers industry leading innovations in management automation, programmatic policies, and dynamic workload provisioning. The ACI fabric accomplishes this with a combination of hardware, policy-based control systems, and closely coupled software to provide advantages not possible in other vendor solutions. Cisco ACI delivers a resilient fabric that meets the needs of today's dynamic applications.
Cisco ACI takes a policy-based, systems approach to operationalizing the data center network. The policy is centered on the needs (reachability, access to services, security policies) of the applications. The ACI fabric enables application instantiations by encapsulating application requirements and requisite characteristics in group-based policies and profiles that enable end-to-end connectivity and services.
Cisco ACI Architecture
The Cisco ACI fabric is a leaf-and-spine architecture where every leaf connects to every spine using high-speed 40/100-Gbps Ethernet links, with no direct connections between the spine nodes or the leaf nodes. The ACI fabric is based on a routed underlay network with a switched VXLAN overlay, where every leaf in the fabric is VXLAN Tunnel Endpoint (VTEP). The Cisco ACI fabric is built on a network of individual components that are provisioned and managed as a single entity. The Cisco ACI fabric supports both Layer 2 (L2) and Layer 3 (L3) forwarding across this fabric infrastructure.
Figure 3 Cisco ACI – High-Level Architecture
Architectural Building Blocks
At a high-level, the key architectural buildings blocks of the Cisco ACI fabric are:
· Application Policy Infrastructure Controller (APIC) - Cisco APIC is the unifying point of control in Cisco ACI for automating and managing the end-to-end data center fabric. The APIC is a physical appliance that serves as a software controller for the overall fabric. APIC is based on Cisco UCS C-series rack-mount servers with 2x10Gbps links for dual-homed connectivity to a pair of leaf switches and 1Gbps interfaces for out-of-band management. It is typically deployed as a cluster of 3 appliances for high availability and enhanced performance.
APIC is responsible for all tasks enabling traffic transport including fabric activation, switch firmware management and network policy configuration and instantiation. Though the APIC acts as the centralized point of configuration for policy and network connectivity, it is never in-line with the data path or the forwarding topology and the fabric can still forward traffic even when communication with the APIC is disrupted. Cisco APIC optimizes the application lifecycle for scale and performance, and supports flexible application provisioning across physical and virtual resources. The Cisco APIC exposes northbound APIs through XML and JSON and provides both a command-line interface (CLI) and GUI that manage the fabric through the same APIs available programmatically. For control plane resiliency, a cluster of three APICs, dual-homed to a pair of leaf switches in the fabric are typically deployed.
· Cisco Nexus 9000 Series Switches – The Cisco ACI fabric is built on a network of Cisco Nexus 9000 series switches that provide low-latency, high-bandwidth connectivity with industry proven protocols and innovative technologies to create a flexible, scalable, and highly available architecture. ACI is supported on several models of Nexus 9000 series switches and line cards. The selection of a Nexus 9000 series switch as an ACI spine or leaf switch will depend on a number of factors such as physical layer connectivity (1/10/25/40/50/100-Gbps), FEX aggregation support, analytics support in hardware (Cloud ASICs), FCoE support, link-level encryption, support for Multi-Pod, Multi-Site etc.
· Spine Nodes – The spines provide high-speed (40/100-Gbps) connectivity between leaf nodes. The ACI fabric forwards traffic by doing a host lookup in a mapping database that contains information about the leaf node where an endpoint (IP, Mac) resides. All known endpoints are maintained in a hardware database on the spine switches. The number of endpoints or the size of the database is a key factor in the choice of a Nexus 9000 model as a spine switch. Leaf switches also maintain a database but only for those hosts that send/receive traffic through it.
· Leaf Nodes – Leaf switches are essentially Top-of-Rack (ToR) switches that end devices connect into. They provide Ethernet connectivity to devices such as servers, firewalls, storage and other network elements. Leaf switches provide access layer functions such as traffic classification, policy enforcement, L2/L3 forwarding of edge traffic etc. The criteria for selecting a specific Nexus 9000 model as a leaf switch will be different from that of a spine switch.
· Cisco ACI Virtual Edge (AVE) – Cisco AVE is the next generation of the Cisco Application Virtual Switch (AVS). Cisco AVE is designed to hypervisor agnostic and brings Cisco ACI policy model, security and visibility to the virtual domain. Cisco AVE runs as virtual machine on VMware ESXi Hypervisor release 6.0 and later.
There are currently two generations of ACI-capable Nexus 9000 switches. A mixed deployment is supported but it can impact the design and features available in the ACI fabric.
Cisco ACI Fabric Design Constructs
Cisco ACI architecture uses a number of design constructs that are critical to a Cisco ACI based design. The key design elements in ACI are summarized below and Figure 4 illustrates the relationship between them.
· Tenant – A tenant is a logical container which can represent an actual tenant, organization, application or a construct for grouping. From a policy perspective, a tenant represents a unit of isolation. All application configurations in Cisco ACI are part of a tenant. Within a tenant, multiple VRF contexts, bridge domains, and EPGs can be defined according to application requirements.
· VRF – Tenants can be further divided into Virtual Routing and Forwarding (VRF) instances (separate IP spaces) to further separate the organizational and forwarding requirements for a given tenant. A Tenant can have multiple VRFs. IP addressing can be duplicated across VRFs for multitenancy.
· Bridge Domain – A bridge domain (BD) is a L2 forwarding construct that represents a broadcast domain within the fabric. A bridge domain is associated with a single tenant VRF but a VRF can have multiple bridge domains and endpoints. The endpoints in a BD can be anywhere in the ACI fabric, distributed across multiple leaf switches. To minimize flooding across the fabric, ACI provides several features such as learning of endpoint addresses (Mac/IP/Both), forwarding of ARP Requests directly to the destination leaf node, maintaining a mapping database of active remote conversations, local forwarding, and probing of endpoints before they expire. Subnet(s) can be associated with a BD to provide an L3 gateway to the endpoints in the BD.
· End Point Group – An End Point Group (EPG) is a collection of physical and/or virtual end points that require common services and policies, independent of their location. Endpoints could be physical servers, VMs, storage arrays, etc. For example, a Management EPG could be a collection of endpoints that connect to a common segment for management. Each EPG is associated with a single bridge domain but a bridge domain can have multiple EPGs mapped to it.
· Application Profile – An application profile (AP) models application requirements and contains one or more EPGs as necessary to provide the application capabilities. A Tenant can contain one or more application profiles and an application profile can contain one or more EPGs.
· Contracts – Contracts are rules and policies that define the interaction between EPGs. Contracts determine how applications use the network. Contracts are defined using provider-consumer relationships; one EPG provides a contract and another EPG(s) consumes that contract. Contracts utilize inbound/outbound filters to limit the traffic between EPGs or applications based EtherType, IP protocols, TCP/UDP port numbers and can specify QoS and L4-L7 redirect policies.
Figure 4 ACI Design Constructs - Relationship Between Major Components
In Cisco ACI, by default, the endpoints in different EPGs cannot communicate with each other unless explicitly allowed through a contract. Endpoints within an EPG are allowed to communicate free and no special configuration is necessary to enable this. Also, note that each tenant in ACI can have multiple VRFs and bridge domains and each bride domain can be associated with multiple application profiles and EPGs.
Overview
The Cisco HyperFlex Virtual Server Infrastructure with Cisco ACI is a flexible, scalable and resilient architecture for Enterprise data centers that deliver business agility by enabling the rapid roll-out infrastructure to host applications and related services. The solution leverages a Cisco ACI fabric that takes a policy-based, application centric-view of the network to provide end-to-end connectivity and services to the applications that connect to it. This design assumes that the customer has an existing Cisco ACI fabric and therefore the focus is on attaching a Cisco HyperFlex environment to the existing ACI fabric.
The Cisco HyperFlex VSI with Cisco ACI solution addresses a number of key design goals and functional requirements:
· Resilient design across all layers of the infrastructure with no single point of failure
· Scalable design with the flexibility to independently scale computing, storage, and network bandwidth as needed
· Modular design where resources can be scaled as needed to meet business demands
· Flexible design with options for different aspects of the design such as the choice of individual components, type of storage access, connectivity options etc.
· Ability to simplify and automate, including integrating with external automation and orchestration tools
· Interoperable, integrated and verified design ready for deployment
Some of the high-level functionality provided in this design are as follows:
· The solution provides a hyperconverged solution based on Cisco HyperFlex to provide the compute, storage and server networking for the physical servers used in the system.
· The solution provides a software-defined solution based on Cisco ACI that brings innovations that go beyond what traditional SDN solutions provide.
· The solution provides Layer-3 connectivity to existing networks within the Enterprise from the Cisco ACI fabric. The existing networks could be a non-ACI data center network, campus or WAN networks, external networks to the Internet, etc. In this design, this connectivity is used to access existing application, network and infrastructure services necessary for installing Cisco HyperFlex on HX nodes attached to the ACI Fabric. HyperFlex installation requires access to the following services. See HyperFlex Installation Pre-Requisites section for more details.
- Existing VMware vCenter to manage the HyperFlex cluster being installed
- HyperFlex Installer Virtual Machine for installing the HX cluster
- NTP and DNS services
- Access for Cloud services such as Cisco Intersight (optional) and Cisco Umbrella (optional).
- Enabling shared services within the ACI fabric - application and network layer services
- Access to existing shared services outside the fabric – application and network services
- Enabling external connectivity to networks outside the ACI fabric – shared access
- Enabling firewall services for applications
- Multi-tenancy to support the organizational structure of the business
- Deploying virtualized applications that utilize all of the above – multi-tier applications
The design also incorporates and aligns with the technology and product best practices of the software and hardware components used in the solution and ensures that there is compatibility between the components used. The design was then built and verified in the Cisco labs to validate the end-to-end design – see Topology in Figure 5. See the Solution Validation section of this document for hardware and software components used to validate this design in the Cisco labs.
Topology
A high-level topology of the Cisco HyperFlex VSI with Cisco ACI solution is shown in Figure 5.
Figure 5 Cisco HyperFlex with Cisco ACI Topology
Cisco Umbrella
For information on integrating cloud-based security using Cisco Umbrella into Cisco HyperFlex solutions, refer to the white paper listed into the Solution References section of this document.
Cisco HyperFlex System Design
In this section, we take a closer at the design of the HyperFlex system used in this solution.
HyperFlex Cluster
The Cisco HyperFlex system consists of cluster of HX-series rack-mount servers connected to a pair of Cisco UCS Fabric Interconnects. A single pair of Cisco UCS Fabric Interconnects can support up to 8 HyperFlex clusters. As of HyperFlex software release 3.0, a single HyperFlex cluster can also have anywhere from 3-32 HX-Series nodes in a Standard HyperFlex Cluster Topology and from 3-64 servers in an Extended HyperFlex Cluster Topology by adding up to 32 compute-only servers. The compute-only servers are HX supported models of Cisco UCS rack-mount servers and/or Cisco UCS 5108 Blade chassis with Cisco UCS blade servers.
Figure 6 Standard HyperFlex Cluster
Figure 7 Extended HyperFlex Cluster
HyperFlex Stretched Cluster
Stretched clusters physically locate half of the cluster nodes in one location, while the remaining half are located in a distant secondary location. The data written to the stretched HyperFlex cluster is stored concurrently in both physical locations, therefore this system design provides for additional data protection and availability because the data in the cluster remains available and the cluster remains online, even if an entire site experiences a failure. Since all data written to the cluster is written simultaneously to the two sites, this design can be considered an active/active disaster recovery design.
The recovery point objective (RPO, or the maximum amount of data that can potentially be lost prior to a failure) is essentially zero, due to all data being immediately and continuously available in both sites. The recovery time objective (RTO, or amount of time needed to return services to their functional state after a failure) is also near zero from an architectural standpoint, since all data and services are immediately available in the site which remains online, however actual recovery involves restarting the guest virtual machines which takes some time.
Stretched Clusters are supported as of HyperFlex software release 3.0. This design does not use stretched clustering. For a more detailed discussion on HyperFlex Stretched Clusters – see Cisco HyperFlex 3.0 for Virtual Server Infrastructure with VMware ESXi design guide listed in Solution References.
HyperFlex Server Options
Cisco HX-Series Rack-Mount Servers
HyperFlex clusters are built using Cisco HX-series converged nodes. A single cluster can support anywhere from 3-32 nodes. The HX-Series converged servers are connected directly to the Cisco UCS Fabric Interconnects in Direct Connect mode. This option enables the Cisco UCS Manager running on the Fabric Interconnects to manage the HX-Series rack-mount Servers using a single cable for both management traffic and data traffic. The HyperFlex M4 and M5 generation servers can be configured and connected as summarized below.
· Cisco HyperFlex M4 generation servers can be deployed using either Cisco VIC 1227 or Cisco VIC 1387 network interface card (NIC) installed in a modular LAN on motherboard (MLOM) slot. VIC1227 and VIC1387 provides dual 10GbE and 40GbE ports respectively for uplink connectivity.
· Cisco HyperFlex M5 generation servers can be deployed only with the Cisco VIC 1387 card.
· The standard and redundant connection practice is to connect port 1 of the VIC card (the right-hand port) to a port on FI-A, and port 2 of the VIC card (the left-hand port) to a port on FI-B. The HyperFlex installer checks for this connectivity on all servers in the HX-cluster.
Failure to follow this cabling practice can lead to errors, discovery failures, and loss of redundant connectivity.
· Multiple uplink options are available for connectivity between the Cisco HX-series servers and the Fabric Interconnects, but only specific combinations are supported. For example, use of the Cisco QSA module to convert a 40GbE QSFP+ port into a 10 GbE SFP+ port is not allowed on M4 generation servers, but is allowed for M5 generation servers in order to configure a mixed cluster of M4 and M5 generation servers to connect to Cisco UCS 6200 series Fabric Interconnects. Table 6 provides a complete list of supported options.
Table 6 Uplink Interface Options on HX-Series Servers
Cisco UCS B-Series Blade Servers
In HyperFlex Extended Clusters, up to 16 blade servers in a Cisco UCS 5108 Blade server chassis can be added for additional compute capacity. The blade chassis comes populated with 1-4 power supplies, and 8 modular cooling fans. In the rear of the chassis are two bays for installation of Cisco Fabric Extenders. The Fabric Extenders (also referred to as FEX, IO Modules, or IOMs) connect the chassis to the Fabric Interconnects. Internally, the Fabric Extenders connect to the Cisco VIC card installed in each blade server through the chassis backplane.
A blade server chassis have can have up to two fabric extenders and operate in an active-active mode for data forwarding and active-passive mode for management functions. Fabric Extenders serve as a consolidation point for all blade server I/O traffic and provide 10 GbE and 40GbE uplink connectivity from the chassis to the Fabric Interconnects.
Table 7 lists the FEX models currently available for the blade server chassis. The FEX 2200 and 2300 series modules are used to connect to 6200 and 6300 series Fabric Interconnects respectively.
Table 7 Cisco UCS 2200 and 2300 Series Fabric Extenders
| Fabric Extender Model |
External Links to FI |
Internal Links to Servers |
Total Uplink I/O to the chassis |
| Cisco UCS 2204XP |
4 x 10GbE/FCoE (SFP+) |
16 x 10GbE |
Up to 80Gbps |
| Cisco UCS 2208XP |
8 x 10GbE/FCoE (SFP+) |
32 x 10GbE |
Up to 160Gbps |
| Cisco UCS 2304XP |
4 x 40GbE/FCoE (QSFP+) |
8 x 40GbE |
Up to 320Gbps |
The standard practice is to connect 1-8 10GbE links, or 1-4 40GbE links from the left-side IOM, or IOM 1, to FI-A, and to connect the same number of 10GbE or 40GbE links from the right-side IOM, or IOM 2, to FI-B. These links can be deployed as independent links (Discrete Mode) or grouped together using link aggregation (Port Channel Mode).
Figure 8 Fabric Extender to Fabric Interconnect Connectivity Options
In discrete mode, each server is pinned to a FEX link going to a port on the fabric interconnect and if the link goes down, the server’s connection also goes down through the FEX link. In port channel mode, the flows from the server will be redistributed across the remaining port channel members. This is less disruptive overall and therefore port channel mode is recommended for this design.
Cisco UCS C-Series Rack-Mount Servers
In HyperFlex Extended Clusters, up 32 Cisco UCS Rack-Mount Servers for additional compute capacity. Similar to the Cisco HX-series servers, Cisco UCS C-Series Rack-Mount Servers should be directly to the Cisco UCS Fabric Interconnects in Direct Connect mode. The Cisco UCS C-Series servers can use either Cisco VIC 1227 or Cisco VIC 1387 network interface card (NIC) installed in a modular LAN on motherboard (MLOM) slot to enable dual 10 GbE or 40GbE connectivity to the Fabric Interconnects. The standard and redundant connection practice is to connect port 1 of the VIC card to a port on FI-A, and port 2 of the VIC card to a port on FI-B. Failure to follow this cabling practice can lead to errors, discovery failures, and loss of redundant connectivity.
Cisco HyperFlex Networking Design
The Cisco HyperFlex system requires multiple VLANs and IP subnets to enable connectivity between the different sub-components within the HyperFlex system. The networking required in a HyperFlex system are covered in this section.
HyperFlex VLANs and Subnets
The networking required within a HyperFlex system can be categorized into the following groups:
· Management: The HyperFlex system requires network connections to manage the ESXi hosts and the storage platform controller virtual machines (SCVM) in a given HX cluster. The SCVM management interfaces required for a HX cluster includes following interfaces:
- SCVM Management Interfaces
- SCVM Replication Interfaces
- Roaming HX Cluster Management IP – one per HX cluster
· Storage: The HyperFlex system requires network connections to manage the storage sub-system in a given HX cluster. These connections are used by the Cisco Data Platform software (HXDP) to manage the HX Distributed File System. The storage interfaces required for a HX cluster are:
- SCVM Storage Interface
- Roaming HX Cluster Storage IP – one per HX cluster
- VMkernel interface for storage access on each ESXi host in the HX cluster
· vMotion: To enable vMotion for guest VMs on ESXi hosts in the HX cluster, a VMkernel interface for each ESXi host in the HX cluster is required.
· Virtual Machines: The HX system will need to provide connectivity to the guest virtual machines deployed on hosts in the HX cluster so that they can be accessed by other VMs and users in the organization.
The HyperFlex networking for Management, Storage and vMotion at the HX node and cluster level are shown in Figure 9.
Figure 9 HyperFlex Network – Node and Cluster Level
HyperFlex Networking Guidelines and Best Practices
The design guidelines and best practices for HyperFlex networking and their use in this design are as follows.
· In-Band Management Network for a HyperFlex cluster requires an IP address for each node in the cluster and additional IP as Cluster Management IP
· Storage Data Network for a HyperFlex cluster requires an IP address for Storage Controller VM (one per node) in the cluster and additional IP as Cluster Storage IP
· The HyperFlex storage data must be on a dedicated network and therefore must be in a separate VLAN from other VLANs in the HyperFlex cluster.
· The HyperFlex storage data VLAN should be unique to each HyperFlex cluster.
· The HyperFlex installer, during installation, requires reachability to the components in the UCS domain, specifically the Fabric Interconnect’s cluster management IP address and the external management (ext-mgmt) IP addresses of the UCS servers (HX nodes). Cluster management IP is a roaming IP that is assigned to the primary Fabric Interconnect and both the cluster IP and the external management IP of the HX nodes are reachable via the dedicated management ports on each FI.
· All storage and vMotion traffic in a HyperFlex system is configured to use jumbo frames by default. Jumbo frames enable IP traffic to use a Maximum Transmission Unit (MTU) size of 9000 bytes. Larger MTU value enables each IP packet to carry a larger payload, therefore transmitting more data per packet, and consequently sending and receiving data faster. The HyperFlex installer will configure the uplinks (vNICs) on all servers in the HX cluster to use a jumbo frame MTU for storage and vMotion. This means that the uplinks from the UCS domain (FI uplinks) and the Nexus interfaces that they connect to, must also be configured for jumbo frames.
· Replication Networking is setup after the initial install by the installer or Cisco Intersight. This design was validated with a single HyperFlex cluster – therefore, replication was not validated in this design. For a detailed discussion on HyperFlex Replication – see the Cisco HyperFlex 3.0 for Virtual Server Infrastructure with VMware ESXi design guide listed in Solution References.
Validated Design – HyperFlex VLANs and Subnets
The tables below show the required VLANs and IP addresses for a HyperFlex cluster; they are provided as input to the HyperFlex installer. The specific VLAN IDs and IP addresses shown in the tables are the values used for validating this solution in the Cisco Labs.
Table 8 shows the VLANs that the HyperFlex Installer (HX Installer VM or Cisco Intersight) uses to provision the HyperFlex system, UCS domain and the vSphere environment. For each network type listed, HX installer will create a corresponding VMware virtual switch (vSwitch) on each ESXi host in the HX cluster. The installer will also provision the virtual switches with port-groups for each of the VLANs listed. The HyperFlex Installer also configures these VLANs on UCS Fabric Interconnects, on both uplinks to the ACI fabric and on the vNICs to the HX nodes.
If replication is enabled (to a second cluster), a VLAN will need to be allocated for this. The replication VLAN will map to a port-group on the inband management vSwitch. Replication networking is not part of the initial automated install of the HX cluster. Replication was not validated in this design.
The HX Installer uses the VM Network VLAN pool to create VLANs in the UCS Fabric and port-groups in vSphere. The VLAN names in the UCS Fabric will be: hxv-vm-network2018 - hxv-vm-network2028 and the port-groups in vSphere will be: hxv-vm-network-2018 - hxv-vm-network-2028. Additional VLANs can be added for VM network as needed. The VM network VLANs are initially mapped to port-groups on a VMware virtual switch but they can be migrated to an APIC-controlled VMware virtual distributed switch (vDS), Cisco AVS, or Cisco ACI Virtual Edge after the initial install.
The infrastructure (4093) required for VxLAN tunneling to the APIC-controlled Cisco ACI Virtual Edge used in this design, is provisioned directly from Cisco UCS Manager, after the initial HX install. Cisco AVE used
Table 9 shows the IP addresses that the HX installer uses to do the initial setup of the HyperFlex system, UCS domain, and the vSphere environment. It includes IP addresses for ESXi VMkernel interfaces for management, vMotion and storage for each node in the cluster as well as two roaming IP addresses for the HX cluster (storage and management). The address pool for VM network is allocated for guest VMs but they are not used in any provisioning done by the HX installer. If replication is enabled, additional addressing is required for the SCVM Replication interface on each host and for the replication cluster IP. All routed networks have a gateway configured in the ACI network (except for out-of-band ext-mgmt). The storage data network is limited to nodes in a single HyperFlex cluster and should be assigned a unique subnet for every HyperFlex cluster deployed.
Cisco UCS Networking Design
Every server in a Cisco HyperFlex cluster connects to both Fabric Interconnects in the Cisco UCS domain. This section details the network design within the UCS domain.
Unified Fabric – Cisco UCS Fabric Interconnects
Cisco UCS Fabric Interconnects (FI) are an integral part of the Cisco Unified Computing System, providing a unified fabric for integrated LAN, SAN and management connectivity for all servers that connect to the Fabric Interconnects. A single pair of Fabric Interconnects and the servers that connect to it is typically referred to as a Cisco UCS domain. Fabric Interconnects provide a lossless and deterministic switching fabric, capable of handling I/O traffic from hundreds of servers.
This design supports the following two models of Fabric Interconnects:
· Cisco UCS 6200 series fabric interconnects provide a 10GbE unified fabric with 10GbE uplinks for northbound connectivity to the datacenter network.
· Cisco UCS 6300 series fabric interconnects provide a 40GbE unified fabric with 40GbE uplinks for northbound connectivity to the datacenter network.
Cisco UCS Fabric Interconnects are typically deployed in pairs to form a single management cluster but with two separate network fabrics, referred to as the FI-A or Fabric A or A-side and FI-B or Fabric B or B-side fabrics. Cisco UCS Manager that manages the UCS domain, runs on the Fabric Interconnects to provide Management services, where one FI is the primary, and the other is the secondary. Each FI has its own IP address and a third roaming IP that serves as the cluster IP address for management. This primary/secondary relationship is only for the management cluster, and has no effect on data transmission. The network fabric on both Fabric Interconnects are active at all times, forwarding data on both network fabrics while providing redundancy in the event of a failure.
Uplink Connectivity to Data Center Network Fabric
The Cisco UCS Fabric Interconnects provide uplink or northbound connectivity to other parts of the Enterprise and typically connects into Nexus switches in data center network fabric. In this design, the Fabric Interconnects connect to Nexus 9000 series leaf switches in the Cisco ACI fabric.
For redundancy and bandwidth, multiple links from each FI are used for uplink connectivity to data center fabric. Cisco UCS FI supports 802.3ad standards for aggregating links into a port-channel (PC) using Link Aggregation Protocol (LACP). Multiple links on each FI are bundled together in a port-channel and connected to upstream switches in the data center network. The port-channel provides link-level redundancy and higher aggregate bandwidth for LAN, SAN and Management traffic to/from the UCS domain. The switches in the data center fabric that connect to single FI are bundled into a virtual Port Channel (vPC). vPC enables links that are physically connected to two different switches to be bundled such that it appears as a “single logical" port channel to a third device (in this case, FI). This PC/vPC based design has many benefits such as:
· Higher resiliency - both link and node-level redundancy
· Higher uplink bandwidth by bundling links
· Flexibility to increase the uplink bandwidth as needed by adding more links to the bundle.
All uplinks on the Cisco UCS FIs operate as trunks, carrying multiple 802.1Q VLAN IDs across the uplinks. And all VLAN IDs defined on Cisco UCS should be trunked across all available uplinks. This is important as traffic may need to be forwarded between servers in the UCS domain but use different fabric (FI-A, FI-B) as its primary data path. There are also failure scenarios where a VIC or an internal fabric level port or link failure results in traffic that normally does not leave the Cisco UCS domain, to now be forced over the Cisco UCS uplinks for intra-domain connectivity. Reachability through the second fabric may also be needed for maintenance events such as FI firmware upgrade that may require a fabric to be rebooted.
Connectivity for HyperFlex Installation
To deploy a HyperFlex system using either the HyperFlex Installer VM or using Cisco Intersight, requires the installer to have connectivity to the following components and subnets in the UCS domain where the servers reside.
· IP connectivity to the management interfaces of both Fabric Interconnects – this is typically an out-of-band network dedicated for management. The out-of-band network in this design is not part of the ACI fabric.
· IP connectivity to the external management IP address of each server in the HX cluster. This IP address comes from an IP Pool (ext-mgmt) defined as a part of the service profile template for configuring the servers in the cluster. The IP address is assigned to the CIMC interface on each server which is reachable through the out-of-band management network of the Fabric Interconnects.
Other Considerations – Jumbo Frames
All storage and vMotion traffic in a HyperFlex system is configured to use jumbo frames by default. Jumbo frames enable IP traffic to use a Maximum Transmission Unit (MTU) size of 9000 bytes. Using a larger MTU value means that each IP packet sent carries a larger payload, therefore transmitting more data per packet, and consequently sending and receiving data faster. The HyperFlex installer will configure the uplinks (vNICs) on all servers in the HX cluster to use a jumbo frame MTU for storage and vMotion. This means that the uplinks from the UCS domain (FI uplinks) and the Nexus interfaces that they connect to, must also be configured for jumbo frames. Failure to configure the Fabric Interconnect uplinks or the Nexus switches for jumbo frames can lead to service interruptions during some failure scenarios, including Cisco UCS firmware upgrades, or when a cable or port failure would cause storage traffic to traverse the northbound Cisco UCS uplink switches.
Validated Design – Cisco HX Cluster and UCS Domain
To validate this design, a HyperFlex standard cluster consisting of 4 Cisco HX Hybrid M5 rack-mount servers are connected through a pair of Cisco UCS 6332-16UP Fabric Interconnects as shown in Figure 10.
Figure 10 Cisco HyperFlex System and UCS Domain - Validated Topology
Each server in the cluster uses a VIC 1387 adapter with two 40Gbps uplink ports to connect to each FI, forming a path through each fabric (FI-A, FI-B). The two uplink ports are bundled in a port-channel to provide 2x40Gbps of uplink bandwidth from each server and redundancy in the event of a failure.
To connect to the upstream data center network, the Cisco UCS 6300 Fabric Interconnects are connected to a pair of upstream Nexus 9000 ACI leaf switches as follows:
· 2 x 40GbE links from FI-A to Nexus leaf switches, one to each Nexus switch
· 2 x 40GbE links from FI-B to Nexus leaf switches, one to each Nexus switch
The FI side ports are configured to be a port-channel, with vPC configuration on the Nexus leaf switches. The two links from separate Nexus 9000 leaf switches in the ACI fabric that connect to a specific FI is configured to be part of the same vPC.
The above connectivity provides the UCS domain with redundant paths and 160Gbps (40Gbps per link x 2 uplinks per FI x 2 FI) of aggregate uplink bandwidth to/from the ACI fabric. The uplink bandwidth can be increased as needed by adding additional connections to the port-channel.
The VLANs listed in the Validated Design – HyperFlex VLANs and Subnets section are enabled on the uplinks from both Fabric Interconnects to Nexus ACI Leaf switches. The VLANs are also enabled on the individual vNIC templates going to each server in the HX cluster.
Virtual Networking Design
The HyperFlex installer deploys the Cisco HyperFlex system with a pre-defined virtual networking design on the ESXi hosts in the cluster. The design segregates the different types of traffic through the HyperFlex system using different VMware virtual switches (vSwitch). Four virtual switches are created by the HyperFlex installer as summarized below. Each vSwitch is assigned two uplinks – the uplink adapters seen by the host at the hypervisor level are virtual NICs (vNICs) created on the VIC converged network adapter installed on the HX server. The vNICs for each server are created in Cisco UCS Manager using service profiles. Installer creates the vNICs as well.
The virtual Switches created by the installer are:
vswitch-hx-inband-mgmt: This is the default ESXi vSwitch0 which is renamed by the ESXi kickstart file as part of the automated installation. The switch has two uplinks, active on fabric A and standby on fabric B – jumbo frames are not enabled on these uplinks. The following port groups are created on this switch:
· Port group for the standard ESXi Management Network. The default ESXi VMkernel port: vmk0, is configured as a part of this group on each ESXi HX node.
· Port Group for the HyperFlex Storage Platform Controller Management Network. The SCVM management interfaces is configured as a part of this group on each HX node.
· If replication is enabled across two HX clusters, a third port group should be deployed for VM snapshot replication traffic.
The VLANs associated with the above port-groups are all tagged VLANs (not native VLANs) in Cisco UCS vNIC templates. Therefore, the VLANs are also explicitly configured in ESXi/vSphere.
vswitch-hx-storage-data: This vSwitch is created as part of the automated installation. The switch has two uplinks, active on fabric B and standby on fabric A – jumbo frames are enabled on these uplinks (recommended):
· Port group for the ESXi Storage Data Network. The ESXi VMkernel port:vmk1 is configured as a part of this group on each HX node.
· Port group for the Storage Platform Controller VMs. The SCVM storage interfaces is configured as a part of this group on each HX node.
The VLANs associated with the above port-groups are all tagged VLANs (not native VLANs) in Cisco UCS vNIC templates. Therefore, the VLANs are also explicitly configured in ESXi/vSphere.
vswitch-hx-vm-network: This vSwitch is created as part of the automated installation. The switch has two uplinks, active on both fabrics A and B – jumbo frames are not enabled on these uplinks. The VLANs associated with the above port-groups are all tagged VLANs (not native VLANs) in Cisco UCS vNIC templates. Therefore, the VLANs are also explicitly configured in ESXi/vSphere.
vmotion: This vSwitch is created as part of the automated installation. The switch has two uplinks, active on fabric A and standby on fabric B – jumbo frames are enabled on these uplinks (recommended). The IP addresses of the VMkernel ports (vmk2) are configured by using post_install script. The VLANs associated with the above port-groups are all tagged VLANs (not native VLANs) in Cisco UCS vNIC templates. Therefore, the VLANs are also explicitly configured in ESXi/vSphere.
Virtual Networking Using VMware vSphere vSwitch
Cisco HyperFlex Installer deploys the following default virtual network design on each ESXi host in the HyperFlex cluster, this is the default setup. To support this multi-vSwitch environment, the HX Installer will user service profiles on Cisco UCS Manager to configure each HyperFlex host with multiple virtual NIC (vNIC) interfaces which are then used as uplinks for each vSphere vSwitch. The Cisco UCS Manager then enables the VLANs for management, storage, vMotion, and application traffic on the appropriate vNIC interfaces.
Figure 11 shows the default virtual networking deployed by the HX Installer on ESXi hosts.
Figure 11 Virtual Networking on HyperFlex Nodes – Default Setup
Virtual Networking Using VMware vSphere vDS or Cisco AVE
Cisco ACI can also manage the virtual networking in a Cisco HyperFlex environment using a APIC controlled VMware vSphere Distributed Switch (vDS) or Cisco ACI Virtual Edge (AVE). The default virtual networking deployed on Cisco HyperFlex systems can be converted to use either a VMware vDS or Cisco AVE instead of the default VMware virtual switches deployed by the HyperFlex installer.
Figure 12 Virtual Networking on HyperFlex Nodes – using VMware vDS
Figure 13 Virtual Networking on HyperFlex Nodes – using Cisco AVE
This design uses Cisco ACE for the guest VM (application or services) networking but all infrastructure connectivity remains on VMware vSwitch(s). The infrastructure access is maintained on VMware vSwitch(s) to ensure access to management, storage and vMotion on the host to maximize availability under failure scenarios that result in connectivity issues between host and VMware vCenter.
Cisco ACI Virtual Edge – for Application Tenant traffic
Cisco ACI Virtual Edge (AVE) is the next generation of application virtual switches for Cisco ACI environments. Unlike, AVS, Cisco AVE is hypervisor-independent, runs in the user-space, and leverages the native VMware vSphere Distributed Switch (VDS) to operate. However, it is similar to Cisco AVS since it is purpose-built for Cisco ACI and operates as a virtual leaf (vLeaf) switch, managed by Cisco APIC. Cisco AVE is supported as of Cisco APIC release 3.1(1i) and VMware vCenter Server 6.0. Cisco AVE
Cisco AVE is a distributed services VM that leverages the hypervisor-resident VMware vDS and uses OpFlex protocol for control plane communication. Cisco AVE supports two modes of operation for data forwarding: Local Switching and No Local Switching.
Local Switching Mode: In this mode, Cisco AVE forwards all intra-EPG traffic locally but all inter-EPG traffic is forwarded to the leaf switch. Also, this mode supports both VLAN and VXLAN encapsulation for forwarding traffic to the leaf. The encapsulation type is specified during VMM integration, when Cisco AVE VMM domain is created. You can also specify that both encapsulations be used in a given VMM domain.
No Local Switching Mode: In this mode, Cisco AVE forwards all traffic (intra-EPG and inter-EPG traffic) to the leaf switch. Only VXLAN encapsulation type is supported in this mode.
Figure 14 Cisco AVE – Local vs. No Local Switching
If Local switching mode is used, either a range of VLANs or a single infra-VLAN must be specified when using VLAN and VXLAN encapsulations respectively. The specified VLAN(s) have local scope as they are only significant to the layer 2 segment between Cisco AVE and ACI Leaf switch.
As stated earlier, Cisco AVE leverages the VMware vDS to operate and the VDS is configured for private VLAN (PVLAN) mode. When a Cisco AVE based VMM domain is created on Cisco APIC, they must associate the domain with a range of VLANs to be used for PVLAN pair association of port groups on the DVS. Server administrators do not need to associate PVLANs to port groups on vCenter because Cisco APIC automatically associates PVLAN pairs with the corresponding ACI EPGs.
If there are multiple vCenter servers connected to the Cisco ACI fabric, care should be taken to ensure that the MAC address allocation schema does not result in overlapping mac-addresses across the different vCenter servers. If the default OUI allocation is used, it can create overlaps and result in duplicate mac-addresses being generated.
The following are the design guidelines and best practices:
· Cisco AVE VM can be deployed using local or remote storage – local storage is recommended.
· Cisco AVE should be deployed on a server with Intel Nehalem CPU or later – EVC should reflect this.
· Only one Cisco AVE VM per host
· vMotion is not support for Cisco AVE VM
· Configuration of the Cisco AVE VM can be lost if the ESXi host or Cisco AVE VM is removed or moved from vCenter.
Validated Design – Virtual Networking on HyperFlex Nodes
The virtual networking on Cisco HyperFlex systems attached to a Cisco ACI Fabric can be converted to use either a VMware vDS or Cisco AVE instead of the default VMware virtual switches deployed by the HyperFlex installer. In this design, the tenant applications are migrated from VM Network vSwitch to either a VMware vDS or an APIC-controlled distributed switch (Cisco AVE). However, the default HyperFlex infrastructure networks, management, storage data and vMotion will remain on the default virtual switches created by the HyperFlex installer. The resulting ESXi networking design is therefore a combination of 3 virtual switches and a fourth virtual switch for application traffic that is migrated to either a VMware vDS or an APIC-Controlled Cisco AVE switch. Both options are covered in this document.
Table 10 lists the ESXi networking deployed by the HX Installer on the HX nodes used in the design.
Table 10 Virtual Switching on HyperFlex Nodes
The port-groups for application virtual machines on the vswitch-hx-vm-network vSwitch and the associated uplinks are migrated to Cisco AVE or VMware vDS when the ACI fabric integration with VMM is in place. Cisco AVE is deployed in local switching mode with VxLAN tunneling. The infrastructure VLAN (v4093) is required for VxLAN tunneling and must be added to the ACI Leaf switches and to the UCS Fabric (uplinks and vNICs to the HX servers) before the APIC can create the Cisco AVE switch in the vSphere environment.
HyperFlex Host Design
Storage Controller Virtual Machines
A key component of the Cisco HyperFlex system is the Storage Controller Virtual Machine running on each of the nodes in the HyperFlex cluster. The controller virtual machines cooperate to form and coordinate the Cisco HX Distributed Filesystem, and service all the guest virtual machine IO requests. The controller virtual machines are deployed as a vSphere ESXi agent, which is similar in concept to that of a Linux or Windows service. ESXi agents are tied to a specific host, they start and stop along with the ESXi hypervisor, and the system is not considered to be online and ready until both the hypervisor and the agents have started. Each ESXi hypervisor host has a single ESXi agent deployed, which is the controller virtual machine for that node, and it cannot be moved or migrated to another host, nor should its settings be manually modified in any way. The collective ESXi agents are managed via an ESXi agency in the vSphere cluster.
The storage controller virtual machine runs custom software and services that manage and maintain the Cisco HX Distributed Filesystem. The services and processes that run within the controller virtual machines are not exposed as part of the ESXi agents to the agency, therefore the ESXi hypervisors nor vCenter server have any direct knowledge of the storage services provided by the controller virtual machines. Management and visibility into the function of the controller virtual machines, and the Cisco HX Distributed Filesystem is done via the HyperFlex Connect HTML management webpage, or a plugin installed to the vCenter server or appliance managing the vSphere cluster. The plugin communicates directly with the controller virtual machines to display the information requested, or make the configuration changes directed, all while operating within the same web-based interface of the vSphere Web Client. The deployment of the controller virtual machines, agents, agency, and vCenter plugin are all done by the Cisco HyperFlex installer, and requires no manual steps.
VMDirectPath I/O Passthrough
VMDirectPath I/O allows a guest virtual machine to directly access PCI and PCIe devices in an ESXi host as though they were physical devices belonging to the virtual machine itself, also referred to as PCI passthrough. With the appropriate driver for the hardware device, the guest virtual machine sends all I/O requests directly to the physical device, bypassing the hypervisor. In the Cisco HyperFlex system, the Storage Platform Controller virtual machines use this feature to gain full control of the Cisco 12Gbps SAS HBA cards in the Cisco HX-series rack-mount servers. This gives the controller virtual machines direct hardware level access to the physical disks installed in the servers, which they consume to construct the Cisco HX Distributed Filesystem. Only the disks connected directly to the Cisco SAS HBA are controlled by the controller virtual machines. Other disks, connected to different controllers, such as the SD cards, remain under the control of the ESXi hypervisor. The configuration of the VMDirectPath I/O feature is done by the Cisco HyperFlex installer and requires no manual steps.
Storage Controller Virtual Machine Locations
The physical storage location of the storage controller virtual machines differs among the different models of Cisco HX-Series rack servers, due to differences in the physical disk location and connections on those server models. The storage controller virtual machine is operationally no different from any other virtual machine in an ESXi environment. However, the SCVM must have a virtual disk with the bootable root filesystem available in a location separate from the SAS HBA that the virtual machine is controlling via VMDirectPath I/O. The configuration details of the models are as follows:
HX220c M5, HXAF220c M5, HX240c M5 and HXAF240c M5: The server boots the ESXi hypervisor from the internal M.2 form factor SSD. The M.2 SSD is partitioned by the ESXi installer, and the remaining 216 GB of space is used as a VMFS datastore. The SCVM’s root filesystem is stored on a 2.5 GB virtual disk, /dev/sda, which is placed on this VMFS datastore. The controller virtual machine has full control of all the front and rear facing hot-swappable disks via PCI passthrough control of the SAS HBA. SCVM’s operating system sees the 240 GB SSD, also known as the “housekeeping disk”, as /dev/sdb, and places HyperFlex binaries and logs on this disk. The remaining disks seen by the controller virtual machine OS are used by the HX Distributed filesystem for caching and capacity layers.
HX220c M4 and HXAF220c M4: The server boots the ESXi hypervisor from the internal mirrored SD cards. The SD card is partitioned by the ESXi installer, and the remaining space is used as a VMFS datastore. The SCVM’s root filesystem is stored on a 2.5 GB virtual disk, /dev/sda, which is placed on a VMFS datastore. The controller virtual machine has full control of all the front facing hot-swappable disks via PCI passthrough control of the SAS HBA. SCVM’s operating system sees the 120 GB or 240 GB SSD, also known as the “housekeeping disk”, as /dev/sdb, and places HyperFlex binaries and logs on this disk. The remaining disks seen by the controller virtual machine OS are used by the HX Distributed filesystem for caching and capacity layers.
HX240c M4 and HXAF240c M4: The server boots the ESXi hypervisor from the internal mirrored SD cards. The HX240c-M4SX and HXAF240c-M4SX server has a built-in SATA controller provided by the Intel Wellsburg Platform Controller Hub (PCH) chip, and the 120 GB or 240 GB housekeeping disk is connected to it, placed in an internal drive carrier. Since this model does not connect the housekeeping disk to the SAS HBA, the ESXi hypervisor remains in control of this disk, and a VMFS datastore is provisioned there, using the entire disk. On this VMFS datastore, a 2.2 GB virtual disk is created and used by the controller virtual machine as /dev/sda for the root filesystem, and an 87 GB virtual disk is created and used by the controller virtual machine as /dev/sdb, placing the HyperFlex binaries and logs on this disk. The front-facing hot swappable disks, seen by the controller virtual machine OS via PCI passthrough control of the SAS HBA, are used by the HX Distributed filesystem for caching and capacity layers.
On the HX240c M4 and HXAF240c M4 model servers, when configured with SEDs, the housekeeping disk is moved to a front disk slot. Since this disk is physically controlled by the SAS HBA in PCI passthrough mode, the configuration of the SCVM virtual disks changes to be the same as that of the HX220c and HXAF220c servers.
The following figures show the SCVM placement on the HX220c M5 and HX240 M5 ESXi Nodes:
Figure 15 HX220c M5 ESXi Node – SCVM Location
Figure 16 HX240c M5 ESXi Node – SCVM Location
HyperFlex Datastores
A new HyperFlex cluster has no default datastores configured for virtual machine storage, therefore the datastores must be created using the vCenter Web Client plugin or the HyperFlex Connect GUI. It is important to recognize that all HyperFlex datastores are thinly provisioned, meaning that their configured size can far exceed the actual space available in the HyperFlex cluster. Alerts will be raised by the HyperFlex system in HyperFlex Connect or the vCenter plugin when actual space consumption results in low amounts of free space, and alerts will be sent via auto support email alerts. Overall space consumption in the HyperFlex clustered filesystem is optimized by the default deduplication and compression features.
CPU and Memory Resource Reservations
Since the storage controller virtual machines provide critical functionality of the Cisco HX Distributed Data Platform, the HyperFlex installer will configure CPU and memory resource reservations for the controller virtual machines. This reservation guarantees that the controller virtual machines will have CPU and memory resources at a minimum level, in situations where the physical CPU or memory resources of the ESXi hypervisor host are being heavily consumed by the guest virtual machines. This is a soft guarantee, meaning in most situations the SCVMs are not using all of the resources reserved, therefore allowing the guest virtual machines to use them. Table 11 and Table 11 list the CPU and memory resource reservations for the SCVMs:
Table 11 SCVM CPU Reservations
Table 12 SCVM Memory Reservations
The compute-only nodes have a lightweight storage controller virtual machine, it is configured with only
1 vCPU of 1024MHz and 512 MB of memory reservation.
HyperFlex Scale
Cisco HyperFlex standard clusters currently scale from a minimum of 3 to a maximum of 32 Cisco HX-series converged nodes with small form factor (SFF) disks per cluster. A converged node is a member of the cluster which provides storage resources to the HX Distributed Filesystem. For the compute intensive extended cluster design, a configuration with 3 to 32 Cisco HX-series converged nodes can be combined with up to 32 compute nodes. It is required that the number of compute-only nodes should always be less than or equal to number of converged nodes.
Once the maximum size of a single cluster has been reached, it can still be scaled out by adding new HX cluster to same Cisco UCS domain, and managed via the same vCenter server. A maximum of 8 clusters can be created in a single UCS domain, and up to 100 HyperFlex clusters can be managed by a single vCenter server. When using Cisco Intersight for management and monitoring of Cisco HyperFlex clusters, there are no limits to the number of clusters being managed.
Cisco HyperFlex HX240c-M5L model servers with large form factor (LFF) disks are limited to a maximum of eight nodes per cluster, and cannot be mixed within the same cluster as models with small form factor (SFF) disks.
HyperFlex Design Guidelines and Best Practices
This section provides the recommended best practices and guidelines for managing and using a Cisco HyperFlex system.
Adding HX to an Existing Cisco UCS Domain
For a scenario where HX nodes are added to an existing Cisco UCS domain, caution is advised. A Cisco UCS firmware upgrade or changes to the configuration on the upstream switches may be required as part of the installation. All of these changes could be disruptive to the existing systems and workloads, and need to be carefully planned and implemented within a maintenance window. It is recommended that you contact Cisco TAC, or your Cisco sales engineer support team for assistance when you need to connect HX nodes to an existing Cisco UCS Domain.
ReadyClones
For the best possible performance and functionality of the virtual machines created using the HyperFlex ReadyClone feature, the following guidelines should be followed when preparing the base virtual machines for cloning:
· Base virtual machines must be stored in a HyperFlex datastore.
· All virtual disks of the base virtual machine must be stored in the same HyperFlex datastore.
· Base virtual machines can only have HyperFlex native snapshots, no VMware redo-log based snapshots can be present.
· For very high IO workloads with many clone virtual machines leveraging the same base image, it might be necessary to use multiple copies of the same base image for groups of clones. Doing so prevents referencing the same blocks across all clones and could yield an increase in performance. This step is typically not required for most uses cases and workload types.
Snapshots
HyperFlex native snapshots are high performance snapshots that are space-efficient, crash-consistent, and application consistent. These snapshots are taken by the HyperFlex Distributed Filesystem, rather than using VMware redo-log based snapshots. For the best possible performance and functionality of HyperFlex native snapshots, the following guidelines should be followed:
· Make sure that the first snapshot taken of a guest virtual machine is a HyperFlex native snapshot, by using the Schedule Snapshot. Failure to do so reverts to VMware redo-log based snapshots.
· To take the first snapshot using HX native snapshot, navigate to Hosts and Clusters in VMware vCenter. Select the virtual machine. Right-click and select Cisco HX Data Platform > Snapshot Now.
· A Sentinel snapshot becomes a base snapshot that all future snapshots are added to, and prevents the virtual machine from reverting to VMware redo-log based snapshots. Failure to do so can cause performance degradation when taking snapshots later, while the virtual machine is performing large amounts of storage IO.
· Additional snapshots can be taken from the Cisco HX Data Platform menu or the standard vSphere client snapshot menu. As long as the initial snapshot was a HyperFlex native snapshot, each additional snapshot is also considered to be a HyperFlex native snapshot.
· Do not delete the Sentinel snapshot unless you are deleting all the snapshots entirely.
· Do not revert the virtual machine to the Sentinel snapshot.
· If large numbers of scheduled snapshots need to be taken, distribute the time of the snapshots taken by placing the virtual machines into multiple folders or resource pools. For example, schedule two resource groups, each with several virtual machines, to take snapshots separated by 15 minute intervals in the scheduler window. Snapshots will be processed in batches of 8 at a time, until the scheduled task is completed.
Storage vMotion
The Cisco HyperFlex Distributed Filesystem can create multiple datastores for storing virtual machines. Though there can be multiple datastores for logical separation, all of the files are located within a single distributed filesystem. As such, performing storage vMotion of virtual machine disk files between datastores in a single cluster has little value in the HyperFlex system. Furthermore, storage vMotion can create additional filesystem consumption and generate additional unnecessary metadata within the filesystem, which must later be cleaned up via the filesystems internal cleaner process.
It is recommended to not perform a storage vMotion of a guest virtual machine between datastores within the same HyperFlex cluster. Storage vMotion between different HyperFlex clusters, or between HyperFlex and non-HyperFlex datastores are permitted.
Virtual Disk Placement
HyperFlex clusters can create multiple datastores for logical separation of virtual machine storage, yet the files are all stored in the same underlying distributed filesystem. The only difference between one datastore and another are their names and their configured sizes. Due to this, there is no compelling reason for a virtual machine’s virtual disk files to be stored on a particular datastore versus another.
All of the virtual disks that make up a single virtual machine must be placed in the same datastore. Spreading the virtual disks across multiple datastores provides no benefit, can cause ReadyClone and Snapshot errors, and lead to degraded performance in stretched clusters.
Maintenance Mode
In HyperFlex Connect, from the System Information screen, in the Nodes view, the individual nodes can be placed into HX Maintenance Mode. Also, within the vCenter Web Client, a specific menu entry for HX Maintenance Mode has been installed by the HX Plugin. This options directs the storage platform controller on the node to shutdown gracefully, redistributing storage IO to the other nodes with minimal impact. Using the standard Maintenance Mode menu in the vSphere Web Client, or the vSphere (thick) Client can be used, but graceful failover of storage IO and shutdown of the controller virtual machine is not guaranteed.
In order to minimize the performance impact of placing a HyperFlex converged storage node into maintenance mode, it is recommended to use the HX Maintenance Mode menu selection to enter or exit maintenance mode whenever possible.
Encryption
HyperFlex Encryption provides protection of data in environments where high security is desired, or compliance with industry security standards is required, such as healthcare providers (HIPAA), financial accounting systems (SOX), credit card transactions (PCI), and more. This data-at-rest encryption capability was added in HyperFlex 2.5 with other new data protection features.
HyperFlex Encryption requires the self-encrypting disks (SED) that can encrypt all the data stored on them. To enable a given HyperFlex cluster for encryption, all the disks on all of the nodes of the cluster must be SEDs.
A cluster using SEDs will store all of its data in an encrypted format, and the disks themselves perform the encryption and decryption functions. Since the hardware handles all the encryption and decryption functions, no additional load is placed on the CPUs of the HyperFlex nodes. Storing the data in an encrypted format prevents data loss and data theft, by making the data on the disk unreadable if it is removed from the system.
Each SED contains a factory generated data encryption key (DEK) which is stored on the drive in a secured manner, and is used by the internal encryption circuitry to perform the encryption of the data. In truth, an SED always encrypts the data, but the default operation mode is known as the unlocked mode, wherein the drive can be placed into any system and the data can be read from it. To provide complete security, the SED must be locked, and reconfigured into what is called auto-unlock mode. This is accomplished via software, using another encryption key, called the authentication key (AK). The authentication key is generated externally from the SED and used to encrypt the DEK. When an SED operates in auto-unlock mode, its DEK is encrypted and therefore when SED is powered on, the AK must be provided by the system, via the disk controller, to decrypt the DEK, which then allows the data to be read. Once unlocked, the SED will continue to operate normally until it loses power, when it will automatically lock itself. If a locked SED is removed from the system, then there is no method for providing the correct AK to unlock the disk, and the data on the disk will remain encrypted and unreadable.
The authentication keys which are used to encrypt the data encryption keys on the disks must be supplied by the HyperFlex cluster. The authentication keys can be provided in one of three ways:
· Local keys in Cisco UCS Manager derived from an encryption passphrase. Local keys are simpler to configure, and are intended for use in testing, proof-of-concept builds, or environments where an external Key Management System (KMS) is not available. Local key configurations create a single authentication key (AK) which is used to encrypt all the disks on all the nodes of the cluster.
· Remote keys, where Cisco UCS Manager retrieves the keys via Key Management Interoperability Protocol (KMIP) from a remote KMS. The client/server communications between the HX nodes and the KMIP server are secured using trusted certificate authority (CA) signed keys, created from certificate signing requests (CSR). Remote key configurations create a unique authentication key for each node, and that AK is used for all disks on that node, providing an even higher level of security.
· Remote keys, where Cisco UCS Manager retrieves the keys via Key Management Interoperability Protocol (KMIP) from a remote KMS, but the client/server communications between the HX nodes and the KMIP server are secured using self-signed certificates.
Cisco has tested remote and self-signed keys using KMS systems, including Gemalto SafeNet KeySecure, and Vormetric DSM. A large number of steps are required to perform the configuration of a certificate authority (CA), root certificates, and signing certificates. Note that the configuration steps are significantly different depending on the KMS being used.
The HyperFlex Connect encryption menu and configuration options are only available when the cluster contains encryption capable hardware on all of the nodes.
The HyperFlex cluster used in validating this CVD did not use SED and encryption was not enabled in this design.
Replication
Replication can be used to migrate a single virtual machine to a secondary HX cluster and a single virtual machine or groups of virtual machines can be coordinated and recovered, or all virtual machines can be recovered as part of a disaster recovery scenario. This capability, snapshot-based virtual machine level replication, was introduced in HyperFlex 2.5 along with other data protection features.
In order to start using replication, two HyperFlex clusters must be installed and have network connectivity between them. The clusters can be standard or extended clusters, and it is possible to replicate between hybrid and all-flash clusters. The clusters are allowed to use self-encrypting disks or standard disks in either location, both locations or neither locations - there is no restriction with respect to this. It is recommended that the two HyperFlex clusters be managed by two different VMware vCenter servers in order to avoid complications with duplicate virtual machine IDs.
Virtual Machines can be replicated in intervals as often as once per 5 minutes, up to once per 24 hours, depending on the Recovery Point Objective (RPO). Care must be taken to make sure that the two clusters have enough storage capacity to store the replicated snapshots of the remote cluster’s virtual machines, and also have enough CPU and RAM resources to run those virtual machines in case they must be recovered. HyperFlex Connect can be used to monitor the status of the protected virtual machines, and to initiate migrations, test recoveries, or failovers of the virtual machines.
This design was validated using a single HyperFlex cluster and as such Replication was not enabled or validated for this CVD. For more details on HyperFlex Replication, see the Cisco HyperFlex 3.0 for Virtual Server Infrastructure with VMware ESXi design guide listed in Solution References.
Logical Availability Zones
Larger scale HyperFlex clusters are subject to higher failure risks, simply due to the number of nodes in the cluster. With clusters up to 64 nodes in size, there is a higher probability that a single node could fail, compared to a cluster with fewer nodes. To mitigate these risks in larger scale clusters, a HyperFlex cluster of eight nodes or more, can be configured into different Logical Availability Zones (LAZ). The Logical Availability Zones feature groups 2 or more HyperFlex nodes together into a logically defined zone, a minimum of 3 zones are created, and the data in the cluster is distributed in such a way that no blocks are written to the nodes within a single zone more than once. Due to this enhanced distribution pattern of data across zones, wherein each zone has multiple servers, clusters with LAZ enabled can typically withstand more failures than clusters that operate without this feature enabled. The number of failures that can tolerated varies depending on the number of zones in the cluster, and the number of servers in each of the zones. Generally speaking, multiple node failures across one or two zones will be tolerated better, and with less risk than multiple nodes failing across three or more zones.
HyperFlex Installer Requirements
Cisco HyperFlex systems are delivered to customers after it has gone through a factory pre-installation process. This ensures that HyperFlex servers will be delivered with the proper firmware revisions, VMware ESXi hypervisor software pre-installed, and some components of the Cisco HyperFlex software already installed. Once on site, the final steps to be performed are dramatically reduced and simplified to the factory pre-install work.
In the Solution Deployment section of this document, the setup process is described as though this factory pre-installation work was done. The Cisco HyperFlex system can be installed using a HyperFlex installer virtual machine, available for download at cisco.com as an OVA file. The installation can also be done using Cisco Intersight, a cloud management tool but will require access to system being deployed and connectivity to devices and networks from Cisco Intersight. The installer (Cisco Installer virtual machine or Cisco Intersight) performs the Cisco UCS configuration work, the configuration of ESXi on the HyperFlex hosts, the installation of the HyperFlex HX Data Platform software and creation of the HyperFlex cluster.
Because this simplified installation method has been developed by Cisco, this CVD will not give the manual steps for configuring the elements that are handled by the installer. However, the pre-requisites for HX installation will be covered in this section. The Solution Deployment section will step through using the installer to deploy a HyperFlex system using the Installer virtual machine.
Prerequisites
This section details the prerequisites for the automated install of HyperFlex using Installer virtual machine or Cisco Intersight.
HyperFlex Installer Virtual Machine: If the Installer virtual machine is used for installing HyperFlex, the installer virtual machine must be deployed on a separate virtualization host and not on the HyperFlex cluster being deployed. The HyperFlex installer will also requires an IP address and reachability to networks and components discussed in this section.
Cisco Intersight: If Cisco Intersight is used for installing HyperFlex, see Cisco Intersight section under Solution Deployment before starting the install. It is also important to verify reachability between Cisco Intersight (installer) and the different networks and components discussed in this section and that firewalls or web proxies in the customer’s network is not blocking the necessary connectivity.
IP Addressing
Both installation methods require reachability to Cisco UCS Manager, ESXi management IP addresses on the HX hosts, and the vCenter IP addresses where the HyperFlex cluster will be managed. Additional IP addresses for the Cisco HyperFlex system need to be allocated from the appropriate subnets and VLANs to be used. IP addresses that are used by the system fall into the following groups:
Cisco UCS Manager: Three IP addresses are required by Cisco UCS Manager running on Cisco Fabric Interconnects. The IP addresses are assigned to each Fabric Interconnect’s management interface for out-of-band management and a third IP address is used for the FI cluster. The cluster IP is a roaming IP that is assigned to the primary FI in the cluster.
HyperFlex Servers: Each server in a HX cluster requires an IP address for out-of-band management. This address is assigned to the CIMC interface of the physical servers that is reachable through the Fabric Interconnect’s management interfaces. The IP addresses will be configured as a pool in Cisco UCS Manager by the installer. The pool (ext-mgmt) must be contiguous and in the same subnet.
HyperFlex and ESXi Management: Each HX node in a cluster requires an IP address for ESXi and HyperFlex management. The two addresses are assigned to ESXi host’s VMkernel interface (vmk0), and to the HyperFlex Storage Controller virtual machine for management. The two addresses must be in the same subnet across all nodes in the cluster.
HyperFlex Cluster Management: A HyperFlex cluster requires one cluster IP address (roaming) for managing the cluster. Cluster IP must be in the same subnet as the per-node HyperFlex and ESXi management IP.
The HyperFlex and ESXi management IP addresses (cluster and per-node) can be in the same subnet as ext-mgmt addresses if necessary.
HyperFlex and ESXi Storage Data Network: Each HX node in a cluster requires an IP address for the HyperFlex Storage Controller virtual machine, and second one for the ESXi host’s VMkernel interface (vmk1) for accessing the storage system. The ESXi host uses the storage interface (vmk1) to send and receive I/O data to/from the HX Distributed Data Platform Filesystem. The two addresses must be in the same subnet across all nodes in the cluster.
HyperFlex Cluster Storage Data Interface: A HyperFlex cluster requires one cluster IP address (roaming) for its cluster storage interface. Cluster IP must be in the same subnet as the per-node HyperFlex and ESXi storage data IP.
Each HyperFlex cluster should be assigned a different subnet for storage data. A non-routable subnet is also recommended.
vMotion Network: Each HX node in a cluster requires an IP address for vMotion. This IP address is assigned to the ESXi host’s VMkernel interface (vmk2) for vMotion.
HyperFlex Replication Network: Replication networking setup is done post-installation, and are not needed to complete the initial installation of a HyperFlex cluster. One IP address per HX node is assigned SCVM’s replication interface, plus one additional IP address as a roaming clustered replication interface. The addresses can be from the same subnet as the HyperFlex and ESXi management addresses, but it is recommended that the VLAN ID and subnet be unique.
DHCP versus Static IP
By default, the HX installation will assign a static IP address to the management interface of the ESXi servers. Using Dynamic Host Configuration Protocol (DHCP) for automatic IP address assignment in not recommended.
DNS
DNS servers are highly recommended to be configured for querying Fully Qualified Domain Names (FQDN) in the HyperFlex and ESXi Management group. DNS records need to be created prior to beginning the installation. At a minimum, it is highly recommended to create A records and reverse PTR records for the ESXi hosts’ management interfaces. Additional A records can be created for the Storage Controller Management interfaces, ESXi Hypervisor Storage interfaces, and the Storage Controller Storage interfaces if desired.
NTP
Consistent time clock synchronization is required across the components of the HyperFlex system, provided by reliable NTP servers, accessible for querying in the Cisco UCS Management network group, and the HyperFlex and ESXi Management group. NTP is used by Cisco UCS Manager, vCenter, the ESXi hypervisor hosts, and the HyperFlex Storage Platform Controller virtual machines. The use of public NTP servers is highly discouraged, instead a reliable internal NTP server should be used.
VLANs
Prior to the installation, the required networking to enable forwarding HyperFlex VLAN traffic must be setup. At a minimum, the networking for at least 4 VLANs needs to be setup so that traffic can be send and received on these VLANs from Cisco UCS Fabric Interconnects to the rest of the customer’s network. The VLAN networks are: HyperFlex and ESXi Management network, HyperFlex Storage Data network, vMotion network, and at least one VLAN for the application/guest virtual machine traffic. If HyperFlex Replication is to be used, another VLAN must be created and trunked for the replication traffic. These VLAN IDs will be provided as input to the HyperFlex installer – the vlan names can be modified if needed.
Usernames and Passwords
Several usernames and passwords need to be defined and provided as input to the HyperFlex installer. Usernames and passwords need to be defined for the following components:
Table 13 HyperFlex Installer - Usernames and Passwords
Cisco ACI Fabric Design
The Cisco ACI fabric consists of discrete components that operate as routers and switches but are provisioned and monitored as a single entity. These components and the integrated management enable ACI to build programmable data centers that provide connectivity as well as advanced traffic optimization, security, and telemetry functions for both virtual and physical workloads.
Cisco ACI fabric is fundamentally different from other data center network designs. A basic understanding of ACI from a technology standpoint is therefore important for understanding an ACI-based design. To learn more about ACI, see the Technology and Components Overview section for ACI and the ACI resources listed in Solution References.
Nexus 9000 Hardware
The Cisco Nexus 9000 family of switches supports two modes of operation: NX-OS standalone mode and Application Centric Infrastructure (ACI) fabric mode. In standalone mode, the Nexus switch provides traditional data center switching functionality with increased port density, low latency and 10/25/40/50/100Gbps connectivity. In this design, Nexus 9000 series switches are in ACI fabric mode.
The ACI fabric is based on a spine-leaf architecture, built using Nexus 9000 series switches where each leaf switch connects to every spine switch, using high speed links and with no direct connectivity between leaf nodes or between spine nodes. Multiple models of Nexus 9000 series switches that support ACI spine and leaf functionality are available and supported in this design.
In ACI, spine switches form the core of the ACI fabric and provide high-speed (40/100GbE) connectivity between leaf switches. A spine can be a:
· Modular Cisco Nexus 9500 series switch equipped with 40/100GbE capable line cards such as N9K-X9736PQ, N9K-X9736C-FX, N9K-X9732C-EX etc.
· Fixed form-factor Cisco Nexus 9300 series switch with 40/100GbE ports (for example, N9K-C9336PQ, N9K-C9364C)
The edge of the ACI fabric are the leaf switches. Leaf switches are top-of-rack (ToR), fixed form factor Nexus switches such as N9K-C9332PQ, N9K-C93180LC-EX, N9K-C93180YC-EX/FX, Nexus 9300 PX series switches. These switches will typically have 40/100GbE uplink ports for high-speed connectivity to spine switches and access ports that support a range of speeds (1/10/25/40GbE) for connecting to servers, storage and other network devices.
Leaf switches provide access layer functions such as traffic classification, policy enforcement, traffic forwarding and serve as an attachment point to the ACI fabric. Nexus leaf switches also provide several advanced capabilities such as support for analytics in hardware, advanced traffic management, encryption, traffic redirection for L4-L7 services, etc.
ACI Fabric – Core Connectivity
The Cisco ACI fabric is a Layer 3, routed fabric with a VXLAN overlay network for enabling L2, L3 and multicast forwarding across the fabric. VXLAN overlays provide a high degree of scalability in the number of Layer 2 segments it can support as well as the ability to extend these Layer 2 segments across a Layer 3 network. The ACI fabric provides connectivity to both physical and virtual workloads, and the compute, storage and network resources required to host these workloads in the data center.
This design assumes that the customer already has an ACI fabric in place with spine switches and APICs deployed and connected through a pair of leaf switches. The ACI fabric can support many models of Nexus 9000 series switches as spine and leaf switches. Customers can select the models that match the interface types, speeds and other capabilities that the deployment requires the design of the existing ACI core is outside the scope of this document. The design guidance in this document therefore focusses on attaching the UCS domain to an existing fabric and the connectivity and services required for enabling an end-to-end converged datacenter infrastructure based on Cisco HyperFlex, Cisco UCS, and Cisco ACI.
To validate this design, the existing ACI Fabric core is built as shown in the figure below. It consists of a pair of Nexus 9504 series spine switches, a 3-node APIC cluster and a pair of Nexus 9000 series leaf switches that the Cisco APICs connect into. The APICs currently only support 10GbE uplinks. The core provides 40GbE connectivity between leaf and spine switches.
Figure 17 Cisco ACI Fabric - Core Connectivity
ACI Fabric – Edge Connectivity
Leaf switches typically provide the edge connectivity in an ACI fabric. Some of the devices and networks that connect to ACI Leaf switches include:
· Cisco APICs that manage the ACI Fabric
· Servers
· Storage
· Outside Networks – These are networks within the customer’s network but outside the ACI fabric. These include management, campus, WAN and other data center and services networks.
· External Networks – These are external networks outside the Enterprise that are reachable directly from the ACI fabric from one or several ACI tenants. This connectivity is used for direct access to cloud services and applications or other cloud resources.
Cisco ACI also uses virtual Port-Channel or vPC technology to increase throughput and resilience on its edge connections from ACI Leaf switches. Virtual Port channels play an important role on leaf switches by allowing the edge devices to use 802.3ad LACP-based port-channeling to bundle links going to ACI leaf switches. Unlike the vPC feature in traditional NX-OS standalone mode, the ACI vPC does not require a vPC peer-link to be connected or configured between the peer leaf switches. Instead, the peer communication occurs through the spine switches, using the uplinks to the spines. Virtual Port channels can therefore be created in ACI, strictly through configuration, without having to do any vPC specific cabling between leaf switches.
When creating a vPC between two leaf switches, the switches must be of the same hardware generation. Generation 2 models have -EX or -FX or -FX2 in the name while Generation 1 does not.
In this design, the following edge components attach to leaf switches:
· Cisco APICs that manage the ACI Fabric (3-node cluster)
· Cisco UCS Compute Domain (Pair of Cisco UCS Fabric Interconnects) with HyperFlex Servers
· Routers that serve as a gateway to campus, WAN and other parts of a customer’s existing network (Outside Network) including connectivity to existing Infrastructure where NTP, DNS, HyperFlex Installer virtual machine and VMware vCenter reside. This infrastructure is used to do the initial install of a HyperFlex cluster in the ACI fabric. Connectivity to external networks for Cloud services and other Internet locations will also be through this connection.
Figure 18 shows the edge connectivity in this design.
Figure 18 Cisco ACI Fabric – Edge Connectivity
Cisco APIC that manages the ACI fabric is redundantly connected to a pair of ACI leaf switches using 2x10GbE links. For high-availability, an APIC cluster 3 APICs are used in this design. The APIC cluster connects to an existing pair of ACI leaf switches in the ACI fabric.
A leaf switch pair in the existing ACI fabric is also used to connect the ACI fabric to customer’s existing networks by connecting the leaf switch pair to Layer 3 gateways outside the ACI fabric. OSPF is used to establish a L3 routed connection between two leaf switches in the ACI fabric and two Nexus 7000 series gateways in the non-ACI portion of the customer’s network. Routes can now be exchanged between non-ACI and ACI portions of the network to enable reachability between endpoints.
A separate pair of Nexus 93180YC-EX leaf switches are added to the existing ACI fabric to provide connectivity to downstream Cisco UCS and HyperFlex system. The leaf switches use 40GbE links for upstream connectivity to spine switches (Nexus 9504) and for downstream connectivity to Cisco UCS domain with HyperFlex cluster. Two vPCs are used to connect to the Cisco UCS Domain; one to each Fabric Interconnect.
ACI Fabric Design – Access Policies
Access Policies or Fabric Access Policies are an important aspect of the Cisco ACI architecture. Fabric Access Policies are defined by the Fabric Administrator and includes all the configuration and policies that are required to connect access layer devices to the ACI fabric. This must be in place before applications and services can be deployed or access the resources provided by edge devices. The Fabric Access Policies defines the Access layer design in an ACI fabric.
The access layer connectivity at the fabric edge could be to the following types of devices:
· Physical Servers (Cisco UCS Rackmount servers)
· Layer 2 Bridged Devices (Switches, Cisco UCS Fabric Interconnects)
· Layer 3 Gateway Devices (Routers)
· Hypervisors (ESXi) and Virtual Machine Managers (VMware vCenter)
Access Policies include configuration and policies that are applied to leaf switch interfaces that connect to edge devices. They include:
· Enabling bundling on links such as configuring port channels or virtual port channels.
· Configuring Interface Policies (for example, VLAN scope global, VLAN scope port-local, LACP, LLDP, CDP)
· Interface Profiles that include all policies for an interface or set of interfaces on the leaf
· VLAN pools that define the range of VLANs to be enabled or used on the access interfaces
· Domain Profiles
· Switch Profiles
· Global policies such as AEP (detailed in a following section)
Fabric Access Policies are designed to be re-used to enable quick deployment of access layer devices to leaf switches in the ACI fabric. The Fabric Access Policies is the access design for a given edge device.
In this design, the leaf switches at the fabric edge provide access layer connectivity to:
· A cluster of 3 APICs to manage the ACI fabric – this is outside the scope of this document as this design assumes that an ACI fabric is already up and running.
· Pair of Nexus 7000 Layer 3 Gateway Routers in the customer’s existing network, outside the ACI fabric
· Pair of Cisco UCS Fabric Interconnects that provide Layer 2 bridging/switching connectivity to Cisco HyperFlex systems and servers in the UCS domain
· Virtual Machine Manager (VMM) Domain or VMware vCenter domain to manage the virtual networking of virtual machines deployed in the VMM domain. The physical access layer connection (mentioned above) is used for this connectivity.
Fabric Access Policies – Overview
A high-level overview of the Fabric Access Policies in the ACI architecture is shown in the figure below. Note that the Policies, Pools and Domains defined are tied to the Policy-Groups and Profiles associated with the physical components (Switches, Modules, and Interfaces) to define the access layer design in an ACI fabric.
Figure 19 Fabric Access Policies – High-Level Overview
Fabric Access Policies – Workflow
The workflow for deploying access layer devices in ACI involves defining policies and then applying the policies to leaf switch interfaces that connect to edge devices. Figure 20 illustrates a high-level workflow.
Figure 20 Fabric Access Policies – Workflow
The fabric access policies and profiles (for example, interface) can be reused for deploying new access layer devices in ACI.
To simplify the implementation of the above workflow, Cisco provides a configuration wizard (Create an Interface, PC, and vPC). However, it is important to understand the workflow in order to understand the access layer design, to enable policy-reuse and make policy-changes as well as for general troubleshooting.
In the following sub-sections, we take a closer look at the sub-components of the above workflow for a detailed understanding of access layer design.
Fabric Access Design – VLAN Design and VLAN Pools
Cisco ACI uses a VXLAN overlay network for communication across the fabric but use VLANs (or VXLAN) to communicate with devices outside the fabric. Unlike other data center architectures, VLANs in ACI are used to classify incoming traffic based on their VLAN tag but are not used to make forwarding decisions within the fabric. The traffic received on a VLAN are classified and mapped to an Endpoint group (EPG). EPG is a fundamental construct in ACI that are used to group endpoints with similar policies and forwarding requirements through the fabric. The EPG policies will determine the forwarding policies, and the VXLAN tunnels will transport the EPG traffic across the ACI fabric. For more details, see Cisco ACI Fundamentals listed in Solution References section.
In ACI, the VLANs defined and used at the edge are tightly coupled to the EPGs. To enable any forwarding within the fabric, a Tenant Administrator must first define an EPG and associate with the underlying physical infrastructure. EPGs are typically associated with physical ports that connect to end devices that make up an EPG. The actual connectivity to a given EPG endpoint is typically through a VLAN defined on the access port. An untagged port or VXLAN can also be used though they are not used in this design. A port can provide connectivity to multiple EPG endpoints by defining multiple VLANs, one for each EPG. In this design, the vPCs to UCS domain and outside networks all carry multiple VLANs, each mapping to different EPGs.
The VLANs used at the edge of the ACI fabric is part of the fabric access policies. The VLAN pools specify the allowed range of VLANs for a given access layer device. The VLAN pools are associated with domains that determine the scope of the vlan pool. The access layer domains can be physical (for example, rackmount server, storage array), virtual (for example, virtual machine manager or VMM) or external (for example, L2 or L3 networks outside the ACI fabric). The domain associated with an EPG will determine the allowed range of VLANs on the EPG ports that connect to that domain. Multiple domains can be associated with a single EPG. Therefore, an EPG can have multiple VLANs in multiple domains to connect to all the endpoints in the EPG. The endpoints could be spread across different leaf switches or on different ports of the same switch using same or different VLAN.
When an EPG is defined, VLANs are allocated to provide connectivity to endpoints. The allocation can be static or dynamic as outlined below:
· Static Allocation is typically used on connections to a physical device in ACI. In this design, static binding is used when connecting to any physical infrastructure. This includes connections to Fabric Interconnects in the Cisco UCS domain and networks outside the ACI fabric.
· Dynamic Allocation is typically used in ACI for virtual machines managed by a Virtual Machine Manager (VMM). In this design, dynamic allocation is used when connecting to a virtual domain. APIC integrates with VMware vCenter managing the virtual domain to allocate VLANs as needed from a pre-defined VLAN pool. To deploy virtual machines in an EPG, APIC will allocate a VLAN from the pool to map to the newly created EPG. APIC will also trigger the creation of a corresponding port-group in the VMM domain using that VLAN. Virtual machines can now be deployed to the port-group and all traffic received from the virtual machines will be tagged using that VLAN. The ACI fabric will map all traffic received on that VLAN to the new EPG.
The scope of an allocated VLAN on an interface can impact the overall VLAN design. If the VLAN scope is:
· Global, then the VLAN ID can only be associated with one EPG on a given leaf switch. The same VLAN ID can be mapped to a second VLAN ID but only on a different leaf switch.
If the EPGs on different leaf switches belong to the same bridge domain, they will be part of the same BPDU flooding domain and broadcast domain.
· Port-local, then the same VLAN ID can be mapped to multiple EPGs on the same leaf switch, if the EPGs are associated with different bridge domains. Also, the EPGs must be deployed on different ports. When the scope is port-local, the leaf switch will maintain translation entries (to/from fabric) for that VLAN on a per-port basis. Maintaining additional information does use up more hardware resources on that leaf switch. This design uses port-local scope on all access ports since it provides more flexibility in VLAN allocation and usage.
An EPG is always associated with a bridge domain. See Cisco ACI fundamentals listed in the Solution References section for more details on how EPGs related to Bridge Domains. Bridge domains in this design will be cover in a later section.
VLAN Allocation Guidelines and Best Practices
VLAN scalability (4096 VLANs) can be a limitation in traditional data center networks but since VLAN are only used at the edge to communicate with devices outside the fabric. The ACI guidelines and best practices that impact the VLAN design are summarized below. Some of the guidelines are dependent on other ACI design constructs that have not been covered yet but will be in the following subsections.
· Overlapping VLAN IDs can used on a single leaf switch or across leaf switches if the connecting devices are part of the same EPG.
· Overlapping VLAN IDs can be used on different EPGs of the same leaf switch if the different EPG devices are on separate ports, belong to different bridge domains and the vlan scope is port-local.
· Overlapping VLAN IDs can be used across different leaf switches and EPGs if the vlan scope is global. The EPGs can belong to the same bridge domain in this case but will be in the same broadcast domain.
· A VLAN pool can be shared by multiple ACI domains but a domain can only be mapped to one VLAN pool. VLAN pool and domains determine the range of VLANs allowed on the interfaces connecting to that domain and are part of the access policies established by the Fabric Administrator.
· When deploying an EPG with static binding to a physical interface, the VLAN ID specified must be from the allowed range of VLANs for that interface. The domain association for the EPG maps to a VLAN pool and this domain must also be associated with physical interface. The domain and the associated VLAN pool is mapped to the physical interface through the Access Entity Profile (AEP). AEP defines the scope of the VLAN (VLAN pool) on the physical infrastructure (port, PC or vPC). This ensures that the EPG VLAN deployed on the physical infrastructure is within the range of VLANs allowed on that infrastructure.
Validated Design – VLAN Pools
To validate this design, VLAN Pools are assigned for endpoints reachable through the access layer devices that connect into ACI Fabric leaf switches.
The VLAN pool assigned to HyperFlex cluster endpoints reachable through Fabric Interconnects in the UCS domain are shown in Table 14 . The same VLAN pool is used on both virtual port-channels connecting leaf switches to Fabric Interconnects (FI-A, FI-B) in the UCS domain.
Table 14 VLAN Pool – EPG VLANs to Cisco UCS Domain and HyperFlex Cluster
The VLANs used to connect to outside networks through external L3 gateways are shown in Table 15 . The VLANs in this case enable connectivity to the gateway routers to establish OSPF neighbor relationship, exchange routes and forward data.
Table 15 VLAN Pool – EPG VLANs to Outside Network (L3 Routed)
The VLAN pool assigned to the VMM domain for virtual machines is shown in Table 16 . The access layer connectivity to the VMM domain is through the vPCs to the UCS domain where the HyperFlex cluster hosting the virtual environment resides. APIC integration with VMware vCenter enables port-groups corresponding to the EPGs to be deployed in the VMM domain. To communicate with a virtual machine endpoint in an EPG, VLANs are dynamically assigned that provides connectivity from virtual machine to the ACI fabric through the UCS domain. APIC allocated a VLAN from the VLAN pool when an EPG is created for the virtual machines.
Table 16 VLAN Pool – EPG VLANs to VMM Domain Hosted on HyperFlex Cluster
The VLAN pool (Table 16 ) is used for the VMM Domain that uses the VMware virtual distributed switch (vDS or DVS). The pools used for a Cisco AVE based VMM domain is not shown in this table. See the Cisco AVE Deployment section for this information.
Fabric Access Design – Domains
Domains in ACI are used to define how different entities (for example, servers, network devices, storage) connect into the fabric and specify the scope of a defined VLAN pool. ACI defines four domain types based on the type of devices that connect to the leaf switch. They are:
· Physical domains are generally used for connections to bare metal servers or servers where hypervisor integration is not an option.
· External bridged domains are used to connect to Layer 2 devices outside the ACI fabric. These are referred to as Layer 2 Outside or L2OUT connections in ACI.
· External routed domains in ACI are used to connect to Layer 3 devices outside the ACI fabric. These are referred to as Layer 2 Outside or L2OUT connections in ACI.
Domains serve as a link between the fabric access policies defined by the Fabric Administrator and the ACI policy model and endpoint groups defined by the Tenant Administrator. Tenant Administrator will deploy EPGs and associate them with domains created by the Fabric Administrator. For EPGs that are statically mapped to a VLAN, the specified VLAN must be part of the vlan pool associated with that domain. For EPGs that are dynamically mapped, the VLANs are always assigned from the pool associated with that domain.
Domain Design Guidelines and Best Practices
The design guidelines and best practices for ACI domains and their use in this design are as follows:
· In ACI, a VLAN pool can be associated with multiple domains but a domain can only be associated with one VLAN pool. In this design, a VLAN pool is created for each domain.
· When an EPG is deployed on a vPC interface, the domain (and the VLAN pool) associated with the EPG must be the same as the domain associated with the vPC interfaces on the leaf switch ports. When deploying an EPG on an interface, a VLAN in the domain that the interface connects to must be specified. The EPG must also be associated with a domain and VLAN pool and the VLAN specified on the interface must be part of that VLAN pool for the EPG to be operational.
· In addition to VLAN pools, ACI domains are also associated to Attachable Entity Profiles (AEP) – AEP will be covered later in this section. Multiple domains can be associated with a AEP. Domains link VLAN pools to AEP. Without defining a VLAN pool in an AEP, a VLAN is not enabled on the leaf port even if an EPG is provisioned.
Validated Design – ACI Domains
Table 17 lists the domains defined for the different access layer connections in the solution.
Table 17 Fabric Access Policies – Domains
Fabric Access Design – Attachable Access Entity Profile
Access Entity Profile (AEP) is an ACI construct for grouping access layer devices with common interface policies. AEP is also known as an Attachable Access Entity Profile (AAEP). This includes linking domains (and VLAN pools) to the physical infrastructure (Interface, Switch).
ACI provides attachment points for connecting access layer devices to the ACI fabric. Interface Selector Profiles represents the configuration of those attachment points. Interface Selector Profiles are the consolidation of a group of interface policies (for example, LACP, LLDP, and CDP) and the interfaces or ports the policies apply to. As with policies, Interface Profiles and AEPs are designed for re-use and can be applied to multiple leaf switches if the policies and ports are the same.
Figure 21 AEP Mapping to Interface and Switch Policies for UCS Domain
AEP links domains (and VLAN pools) to the physical infrastructure through Interface and Switch Selector Profiles, thereby specifying the scope of the VLAN pool to the selected interfaces and switches in the profiles.
Figure 22 AEP Mapping to VLAN Pools and Domains for UCS Domain
For VMM Domains, the associated AEP provides the interface profiles and policies (CDP, LACP) for the virtual environment.
When deploying an EPG on a leaf port, the VLAN specified must be part of a VLAN pool for the AEP associated with that port in order for the VLAN to be operational. In summary, an AEP must be mapped to a VLAN pool in order to enable the EPG VLAN on that interface.
AEP Design Guidelines and Best Practices
The design guidelines and best practices for AEPs in ACI and their use in this design are as follows:
· Multiple domains can be associated with a single AEP. In this design, the AEP associated with leaf interfaces going to the UCS domain are linked to two domains; the physical UCS domain and the VMM domain.
· Multiple AEPs are required to support overlapping VLAN pools, to enable or disable infrastructure VLAN, to have different scopes for the VLANs or for supporting different virtual switch policies within the same VMM domain. This design uses multiple AEPs – one AEP per domain (and VLAN pool) but more for design flexibility rather than for the reasons listed above.
· AEPs can be configured using the provisioning wizard (Create an interface, PC, and vPC). However, understanding the ACI constructs and the inter-dependencies are important in order to design for policy-reuse across the ACI fabric, for troubleshooting purposes and to fully realize the benefits of policy-driven architecture that ACI provides.
Validated Design – AEP
Table 18 lists the AEPs used in this solution.
Table 18 Fabric Access Policies – AEP to Domain Mapping
ACI Fabric Design – EPGs, Contracts and Application Profiles
End Point Groups
An End Point Group (EPG) is an ACI construct that represents a collection of endpoints. Endpoints in ACI can be either physical or virtual, directly or indirectly connected to the ACI fabric. They have an address and a location. The endpoints grouped into an EPG are not tied to the topology; endpoints can be located anywhere in the fabric. Endpoint Grouping in an EPG could be based on providing a common set of services, or on a common set of policy requirements or some other factor. ACI manages endpoints in an EPG as a group rather than individually. Therefore policies are specified at the EPG level and apply to all endpoints in the EPG.
In this design, the IB-MGMT EPG represents a grouping for in-band management connectivity. The endpoints in this EPG are ESXi management interfaces on servers and HyperFlex Storage Controller VM management interfaces on HX nodes.
EPGs can be statically or dynamically mapped. If static binding is used, the Administrator will manually map the EPG to a specific path and VLAN. This in turn will program the leaf switches to classify the traffic received on that path and VLAN into an EPG. ACI will then apply policies and forward the traffic through the fabric based on the EPG classification. If dynamic binding is used, the EPG is mapped to a path and VLAN dynamically. This requires integration with a Virtual Machine Manager (VMM) such as VMware vCenter. The virtual switch is APIC deployed and managed, and can be either a VMware vDS or Cisco AVE. With the VMM integration in place, Application EPGs deployed from the APIC will now result in port-groups being dynamically created on the distributed virtual switch through VMware vCenter. There is a one-to-one mapping between a port-group in VMware and EPG in ACI. ACI will also dynamically allocate a VLAN for that EPG/port-group from a pre-defined pool of VLANs. Application virtual machines can now be deployed on that port-group, and traffic will get mapped to the corresponding EPG and be forwarded through the ACI fabric based on the EPG policies.
Validated Design – EPG
Table 19 lists some of the statically mapped EPGs used in this solution. Application EPGs are not listed in this table. Application EPGs are dynamically mapped EPGs that are deployed as needed.
Table 19 ACI Fabric Design - EPGs
Application Profiles
An application profile is an ACI construct used to model the requirements of applications and contains one or more EPGs as necessary to enable multi-tier applications and services. Multiple Application profiles are used in this design – see table below.
Contracts
Contracts define inbound and outbound traffic filter, QoS rules and Layer 4 to Layer 7 redirect policies. Contracts define the way an EPG can communicate with another EPG(s) depending on the application requirements. Contracts are defined using provider-consumer relationships; one EPG provides a contract and another EPG(s) consumes that contract. Contracts utilize filters to limit the traffic between the applications to certain ports and protocols.
Validated Design – Application Profiles and Contracts
Table 20 some of the Application Profiles used in this design and the contracts they either consume or provide. Application Profiles for application EPGs are not shown in this table.
Table 20 ACI Fabric Design – Application Profiles and Contracts
ACI Fabric Design – Multi-tenancy
The ACI architecture is designed for multi-tenancy. Multi-tenancy enables the administrator to partition the fabric along organizational or functional lines to form different tenants. Tenants enable domain-specific management by defining a Tenant Administrator.
Tenant is a logical container for grouping applications and their networking and security policies. A tenant represents a unit of isolation from a policy perspective. This container can represent an actual tenant, an organization, an application or a group based on some other criteria. All application configurations in ACI are all done within the context of a tenant. Tenants will typically have multiple Application Profiles, EPGs and associated tenant networking that includes VRFs, Bridge Domains and External Bridged or Routed Networks.
Tenants can be system-defined or user-defined. The three system-defined tenants on the ACI fabric are mgmt, infra, common tenants. The common Tenant in ACI is designed to be used for shared services that all tenants require. Shared services could Microsoft Active Directory (AD), Domain Name System (DNS), etc.
The user tenants are defined, as needed by the administrator, to meet the needs of the organization. The user tenants in this design include a foundational tenant (HXV-Foundation) for foundational infrastructure and services, and an application tenant (HXV-App-A). The foundational tenant in this design provides:
· access to in-band management networks
· access to storage data network
· access to vMotion network
One Application tenant is defined (HXV-App-A) for validation but additional application or user tenants can be defined as needed.
VRFs
A virtual routing and forwarding (VRF) instance in ACI is a tenant network. VRF is a unique Layer 3 forwarding domain. A tenant can have multiple VRFs and a VRF can have multiple bridge domains. This design uses one VRF per tenant.
Bridge Domains
A bridge domain represents a Layer 2 forwarding construct within the fabric. The bridge domain can be further segmented into EPGs. Like EPGs, bridge domains can also span multiple switches. A bridge domain can contain multiple subnets, but a subnet is contained within a single bridge domain. One or more bridge domains together form a tenant network. A bridge domain represents the broadcast domain and has global scope. EPGs must be associated with bridge domain.
Bridge Domains are an important consideration in this design. When a bridge domain contains endpoints belonging to different VLANs (outside the ACI fabric), a unique MAC address is required for every unique endpoint.
To overcome potential issues caused by overlapping MAC addresses, multiple bridge domains need to be deployed for correct storage connectivity.
Validated Design – Tenants, VRFs, Bridge Domains and Contracts
Table 21 lists some of the Tenants used in this design and the associated tenant networking. Application Tenants are not shown in this table.
Table 21 ACI Fabric Design – Tenant Networking
ACI Fabric – Recommended Fabric Settings
Enabling the following ACI Fabric settings are recommended in Cisco HyperFlex deployments.
· COS Preservation (Fabric Wide)
· Endpoint Learning (Enabled by default, Bridge Domain Level)
· Layer 2 Unknown Unicast (Bridge Domain Level)
· Clear Remote MAC Entries (Bridge Domain Level)
· ARP Flooding (Bridge Domain Level)
· Unicast Routing (Bridge Domain Level)
· GARP Based Detection for EP Move Detection Mode (Bridge Domain Level)
· Endpoint Dataplane Learning (Bridge Domain Level)
· Enforce Subnet Check (Optional, Fabric Wide)
· Limit IP Learning to Subnet (Optional, per Bridge Domain)
· Jumbo Frames and MTU
The implementation of the above features can vary depending on the generation of ACI leaf switches used in the deployment. Some examples of first and second-generation Cisco ACI leaf switches are provided below - see the Cisco Product documentation for a complete list.
· First-generation Cisco ACI leaf switches models: Nexus 9332PQ, Nexus 9372 (PX, PX-E, TX, TX-E), Nexus 9396 (PX, TX), 93120TX, 93128TX switches
· Second-generation Cisco ACI leaf switches models: Nexus 9300-EX and 9300-FX Series, Nexus 9348GC-FXP, Nexus 9336C-FX2, Nexus 93240YC-FX2 switches.
COS Preservation
The Cisco HyperFlex Installer deploys the HyperFlex system with the following QoS Classes enabled on the Cisco UCS fabric. The invidual nodes in the HyperFlex cluster marks any traffic it sends to the UCS fabric using COS values of ‘5’, ‘4’, ‘2’ or ‘1’ based on the type of traffic (Storage Data, VM Network, Management, vMotion). The UCS then classifies the traffic into the following QoS classes for queueing and forwarding through the UCS fabric.
Figure 23 Cisco UCS: QoS Classes for HyperFlex
When ACI fabric receives traffic with non-zero COS values, the default behaviour is to re-mark the traffic to a COS of ‘0’. As a result, this traffic will be classified and queued as best-effort traffic through the ACI fabric. The traffic will also leave the ACI fabric with the COS 0 marking and this COS value will determine the QOS it receives in the receiving domain. For HyperFlex, this means that any traffic forwarded through the ACI fabric will be re-marked as COS 0 and if the traffic comes back into the UCS domain, it will be classified and queued as best-effort traffic. For HyperFlex storage data and vMotion traffic that use jumbo frames, queueing traffic into the best-effort class that uses a 1500B MTU will result in packets drops. To prevent this, ACI provides a Preserve COS option that can preserve the COS for any traffic forwarded through the ACI fabric. This feature should be enabled for HyperFlex deployments with ACI. It is a fabric-wide feature so it will impact all traffic traversing the ACI fabric.
The Preserve COS is enabled in this design as shown in the figure below.
Figure 24 Cisco ACI Fabric: Preserve COS Enabled
Endpoint Learning
Endpoint learning in ACI is primarily done in hardware from data-plane traffic by examining the incoming traffic, specifically the source MAC and IP address fields in the received traffic. ACI can learn the address and location of any endpoint that sends traffic to the fabric. ACI provides several configuration settings (mostly at the bridge-domain level) that impact endpoint learning behavior.
By default, ACI learns the MAC address of all endpoints but for any “IP” learning to occur, Unicast Routing must be enabled at the bridge-domain level. Unicast Routing enables both Layer 3 forwarding and IP learning in an ACI fabric.
ACI typically learns from data-plane traffic but for silent endpoints that do not send any traffic to the fabric, ACI can also use control plane protocols such as ARP and GARP to do endpoint learning.
For bridge-domains doing Layer 2 forwarding (Unicast Routing disabled), ARP flooding can be used to learn the location of silent endpoints. ARP Flooding enables ACI to learn from the data-plane ARP traffic exchanged between the endpoints. In this scenario, the L2 Unknown Unicast option should also be set to “Flood” to prevent ACI from dropping unicast traffic destined to endpoints that it hasn’t learned of yet.
APIC GUI automatically enables ARP Flooding if L2 Unknown Unicast is set to “Flood”. However, regardless of the GUI setting, APR Flooding is always enabled in hardware when Unicast Routing is disabled.
For bridge-domains doing Layer 3 forwarding (Unicast Routing enabled), ACI can learn the location of silent or unknown hosts either by generating an ARP request or from data-plane ARP traffic. If IP subnet(s) are configured for the bridge-domain, ACI can generate an ARP request and learn the location of the unknown endpoint from its ARP response (also known as ARP gleaning). If Unicast Routing is enabled without configuring bridge-domain subnets (not recommended), ACI cannot initiate ARP requests. However, ACI can still learn their location from the data-plane ARP traffic. Though ARP Flooding is not necessary in first scenario, it should be enabled so that if the endpoint moves, ACI can learn the new location quickly rather than waiting for ACI to age out the entry for the endpoint. ACI can also detect endpoint moves using GARP by enabling the GARP-based endpoint move detection feature.
ARP Flooding must be enabled for GARP-based endpoint move detection feature.
Endpoint learning in ACI also depends on whether the endpoints are local or remote endpoints. For a given leaf switch, local endpoints are local to that leaf switch while remote endpoints connect to other leaf switches. Local and remote endpoints are also learned from data-plane traffic. However, unlike local endpoints, ACI typically learns either the MAC or IP address of remote endpoints but not both. The local endpoints information is sent to the Spine switches that maintain the endpoint database but remote endpoints are maintained on the leaf switches. Remote entries are also aged out sooner than local endpoints by default.
As stated earlier, ACI provides several options that impact endpoint learning. These settings are covered in more detail in the upcoming sections.
L2 Unknown Unicast
The L2 Unknown Unicast field for a bridge-domain specifies how unknown Layer 2 unicast frames should be forwarded within the fabric. This field can be set to “Flood” or “Hardware Proxy” mode. In “Flood mode”, the unknown Layer 2 unicast frames are flooded across all ports in the bridge-domain using the bridge-domain specific multicast tree. In “Hardware Proxy” mode, the unknown unicast frames are sent to the spine switch to do a lookup in the endpoint mapping database. However, if the spine has not learned the address of that endpoint, the unicast traffic will be dropped by the fabric. For this reason, if a Layer 2 bridge-domain has silent endpoints, the L2 Unknown Unicast field should always be set to “Flood”.
The default setting for L2 Unknown Unicast is “Hardware-Proxy” but in this design, this field is set to “Flood” as shown in the figure below.
This feature requires ARP Flooding to be enabled on the bridge-domain. Customers may also want to enable the Clear Remote MAC Entries setting. See upcoming sections for additional information on these two settings.
Clear Remote MAC Entries
This is a bridge-domain level setting that clears the remote Layer 2 MAC addresses on other switches when the corresponding MAC addresses (learnt on a vPC) are deleted from a local switch. The entries are cleared on all remote switches if it is deleted on a local switch. The setting is visible in the GUI when L2 Unknown Unicast is set to “Flood”.
ARP Flooding
ARP Flooding is used for both Layer 2 (Unicast Routing disabled) and Layer 3 bridge-domains (Unicast Routing enabled). By default, with Unicast Routing enabled, the ACI fabric will treat ARP requests as unicast packets and forward them using the target IP address in the ARP packets. It will not flood the ARP traffic to all the leaf nodes in the bridge domain. However, the ARP Flooding setting provides the ability to change this default behavior and flood the ARP traffic across the fabric to all the leaf nodes in a given bridge domain. See Endpoint Learning section above for other scenarios that require ARP Flooding.
ARP Flooding is also required in environments that use Gratuitous ARP (GARP) to indicate an endpoint move. If an endpoint move occurs on the same EPG interface, GARP feature must be enabled in ACI to detect the endpoint move – see GARP based Detection section for more details.
This feature is disabled by default. This feature can be enabled for HyperFlex deployments as shown in the figure below.
Figure 25 Cisco ACI Fabric Settings: ARP Flooding
Unicast Routing
Unicast Routing setting on the bridge-domain enables both Layer 3 forwarding and “IP” learning in an ACI fabric. The IP endpoint learning is primarily done from the data plane traffic but ACI can also initiate ARP requests to do endpoint learning in the control plane. ACI can originate ARP requests for unknown endpoints if both Unicast Routing and bridge-domain subnet is configured. However, ACI cannot generate ARP requests if a subnet is not configured for the bridge-domain, but it can still learn their location from the data-plane ARP traffic if ARP Flooding is enabled.
GARP-based Detection
Gratuitous ARP (GARP) based detection setting enables ACI to detect an endpoint IP move from one MAC address to another when the new MAC is on the same EPG interface as the old MAC. ACI can detect all other endpoint IP address moves such as moves between ports, switches, EPGs or bridge-domains but not when it occurs on the same EPG interface. With this feature, ACI can use GARP to learn of an endpoint IP move on the same EPG interface.
This is a bridge-domain level setting that can be enabled as shown in the figure below.
Figure 26 Cisco ACI Fabric Settings: GARP-based Detection
Note that ARP Flooding must be enabled to use this feature. GARP-based detection setting will not be visible on the GUI until ARP Flooding is enabled on the bridge domain.
Endpoint Dataplane Learning
Endpoint Dataplane Learning is bridge-domain level setting that enables/disables “IP” learning in the data-plane.
This feature is available as of APIC release 2.0(1m) and it is enabled by default as shown in the figure below.
Figure 27 Cisco ACI Fabric Settings: Endpoint Dataplane Learning
Enforce Subnet Check for Endpoint Learning (Optional)
This feature changes the default endpoint learning behavior of the ACI fabric. Enabling this feature will disable endpoint address learning on subnets that are outside the VRF and only learn the addresses when the source IP address is from one of the configured subnets for that VRF. For subnets outside the VRF, enabling this feature will prevent both MAC and IP addresses from being learned for local endpoints, and IP addresses for remote endpoints. This feature provides a better check than the Limit IP Learning to Subnet covered in the next section, which does the subnet check for IP addresses but not for MAC addresses. Also, it does the check only for learning local endpoints and not for remote endpoints. However the Limit IP Learning to Subnet feature is more granular in scope as it does the subnet-check on a per bridge domain basis while the Enforce Subnet Check does a check against all subnets at the VRF level and is enabled/disabled at the fabric level so it applies to all VRFs in the fabric.
This feature is disabled by default. This feature can be enabled in HyperFlex deployments as shown in the figure below.
Figure 28 Cisco ACI Fabric Settings: Enforce Subnet Check
Note the following regarding this feature:
· This feature is available only on second-generation leaf switches. In a mixed environment with first and second-generation leaf switches, the first-generation switches will ignore this feature.
· Enabling this feature will enable it fabric-wide, across all VRFs though the subnet-check is for the subnets in the VRF.
· Available in APIC Releases 2.2(2q) and higher 2.2 releases and in 3.0(2h) and higher. It is not available in 2.3 or 3.0(1x) releases.
· The feature can be enabled/disabled under Fabric > Access Policies > Global Policies > Fabric Wide Setting Policy in earlier releases and under System > System Settings > Fabric Wide Setting in newer releases.
Limit IP Learning to Subnet (Optional)
This feature changes the default endpoint “IP” address learning behavior of the ACI fabric. Enabling this feature will disable IP address learning on subnets that are not part of the bridge domain subnets and only learn if the source IP address belongs to one of the configured subnets for that bridge domain. A bridge domain can have multiple IP subnets and enabling this feature will limit the IP address learning to the bridge-domain subnets but will not learn addresses for subnets outside the bridge-domain.
This feature is available as of APIC release 1.1(1j) and enabled by default as of APIC releases 2.3(1e) and 3.0(1k). This feature can be enabled for HyperFlex deployments as shown in the figure below.
Figure 29 Cisco ACI Fabric Settings: Limit IP Learning to Subnet
Note the following regarding this feature:
· Available on first and second-generations of ACI leaf switches
· If Enforce Subnet Checking is also enabled, it supersedes this feature.
· This feature should be used when subnet-check is for a specific bridge domain (as opposed to all VRF subnets) or when you have an environment with first-generation leaf switches.
· Prior to APIC release 3.0(1k), toggling this feature with Unicast Routing enabled could result in an impact of 120s. In prior releases, ACI flushed all endpoints addresses and suspended learning on the bridge domain for 120s. The behavior in 3.0(1k) and later releases is to only flush endpoint IP addresses that are not part of the bridge domain subnets and there is no suspension of address learning.
Jumbo Frames and MTU
Traditional switching fabrics typically us a 1500B MTU and must be configured to support Jumbo frames. However, the ACI fabric, by default uses an MTU of 9150B on core facing ports of leaf and spine switches and 9000B on access ports of leaf switches. Therefore, no configuration is necessary to support Jumbo frames on an ACI fabric.
ACI Fabric Design – Enabling Access to Cisco HyperFlex Installer and Existing Infrastructure
The HyperFlex installer virtual machine (VM) is used in this design to install HX and must be deployed first. The virtualized environment hosting the HX Installer VM must be different from the HyperFlex system being deployed. In this design, the Installer VM is hosted in customer’s existing environment, outside the ACI fabric. The HX Installer outside the ACI fabric is reachable through a Layer 3 routed connection as outlined in this section.
Other infrastructure services and networks are also required for the initial install for a HyperFlex system such as VMware vCenter, DNS, NTP, out-of-band management access to Cisco UCS Fabric Interconnects and HX servers. In this design, all networks and services for the initial HyperFlex install are outside the ACI Fabric and accessible through the same connection that provides access to the HyperFlex Installer.
ACI Fabric Connectivity to Outside Networks
ACI provides different options to connect to networks outside the ACI fabric using Layer 2 and Layer 3 technologies. A Layer 3 connection provides routed connectivity while a Layer 2 connection provides bridged connectivity to outside networks.
Figure 30 ACI Fabric Connectivity to Outside Networks
In this design, Layer 3 connections are used to provide the connectivity to outside network. Specifically, the Layer 3 Outside (L3Out) connection provides the following connectivity to tenants and endpoints in the ACI fabric:
· Connectivity to HX Installer outside the ACI fabric in order to deploy a Cisco HyperFlex system in a Cisco UCS domain attached to the AVI Fabric.
· Connectivity to existing Infrastructure and Application Services outside the ACI Fabric to access NTP, AD/DNS, vCenter, Cisco Umbrella Virtual Appliance etc. necessary to install and manage a HyperFlex system.
· Connectivity to networks in other parts of the organization – for example, Campus, WAN, Internet etc. This includes connectivity to out-of-band management connectivity to Cisco UCS FI and HX servers, and external connectivity to Cisco Intersight in the cloud to manage the HyperFlex deployment.
ACI Fabric Design – Layer 3 Routed Connectivity to Outside Networks
To connect the ACI fabric to outside networks using Layer 3, leaf nodes in the ACI fabric are connected to routers or other Layer 3 devices in the non-ACI infrastructure. Static or dynamic routing protocols are used across the Layer 3 Outside (also referred to as Shared L3Out) connection to exchange routes between the ACI and non-ACI domains. The L3Out connection can be a shared service where it is shared by multiple tenants or it can be dedicated on a per-tenant basis. Cisco recommends a shared L3Out service. This functionality can be deployed as a shared service in any tenant.
In this design, the L3Out connection is a shared or common service that all tenants use to access the existing infrastructure or networks outside ACI. For this reason, the shared L3Out in this design is part of the common tenant. The common tenant is pre-defined system tenant where any objects defined in this tenant are visible to all other tenants, making it easier for other tenants to enable and access this service.
To validate this design, the ACI fabric is connected to the external network as shown in Figure 31. A pair of Nexus 9372PX leaf switches are connected to Nexus 7000 routers in the external network using 10GbE links. Each leaf switch is connected to both external Nexus 7k routers for redundancy.
The leaf switches used for L3Out are separate from the ones used to attach the HyperFlex domain to the ACI fabric. For larger deployments, Cisco recommends using a dedicated pair of border leaf switches for scale.
Figure 31 Shared L3Out – Physical Connectivity
The links between ACI leaf nodes and external routers are treated as 4 individual access links (no bundling) with a dedicated VLAN and IP subnet for each link. In this design, OSPF is used as the routing protocol to exchange routes across the Shared L3Out connection. On each node, OSPF will establish two neighbor-adjacencies on the two connections from that node to the neighboring nodes. The OSPF and fabric access policies configuration for the shared L3Out service is shown in Figure 32.
Figure 32 Shared L3Out – OSPF Configuration and Fabric Access Policies
In this design, OSPF is used to share ACI tenant “public” routes with the outside infrastructure and routes from the outside network can be shared with the ACI tenants. In this setup, the ACI fabric uses a default route for the outside networks and each tenant advertises one or more "public" routable subnets to the outside infrastructure. Note that this requires ACI tenant routes to be shared outside the VRF. OSPF metrics on Cisco Nexus 7000 switches can be optionally used to influence path preferences. Tenant advertised prefixes for a shared L3Out must to be unique; overlapping tenant subnets are not supported.
The ACI constructs and design for enabling and accessing a shared L3Out service is shown in Figure 33. These include:
· A single External Routed Network under tenant common to connect ACI fabric to Cisco Nexus 7000s using OSPF.
· A unique private VRF (vrf-common-outside) network and a dedicated external facing bridge domain (bd-common-outside) is defined under the common tenant. This private network is setup with OSPF to provide connectivity to external infrastructure.
· The shared L3 Out created in the common Tenant “provides” an external connectivity contract (Allowed-Shared-L3Out) that can be “consumed” from any tenant. Contracts created in common Tenant are visible to all tenants. Similarly, the contract to access Shared L3Out service is accessible by all other tenants.
· When other tenants consume the contract, the public subnets shared by the tenants will get advertised to the outside infrastructure. These tenant will also learn the routes to outside networks, to access the external infrastructure networks and endpoints. The outside routes in this design is a single default route.
Figure 33 ACI Tenant Design – Accessing Shared L3Out Connection
By defining a shared L3Out in common tenant, the contract is provisioned as part of the L3Out configuration and it would automatically be available in all other tenants to consume, without doing any additional configuration since the objects (contracts in this case) from the common tenant are available in all other tenants. If the shared L3Out was deployed in any other tenant, the contract would have to be explicitly exported from that tenant to each tenant where this contract needs to be consumed.
ACI Fabric Design – Enabling Access to Cisco UCS Domain and HyperFlex Nodes
This section covers the access layer design for connecting the ACI fabric to a UCS domain where the Cisco HyperFlex servers reside. The access layer configuration to the UCS domain must be done before the ACI fabric can enable network reachability to endpoints in the UCS domain. In this case, the access layer setup is necessary before the ACI fabric can provide connectivity between the HX installer in the non-ACI fabric (see previous section) and HX nodes in the UCS domain for the initial installation and configuration of the HyperFlex system.
The access layer design or fabric access policies to connect the UCS domain and HyperFlex servers to the ACI fabric includes the following:
· Specifying/Selection Leaf interfaces that connect to the Cisco UCS Fabric Interconnects
· Enabling vPCs to Cisco UCS Fabric Interconnects from the leaf switch pair
· Access Policies for the above vPC interfaces (for example, 40Gbps, CDP, LLDP, Spanning Tree)
· VLANs that should be enabled on the above vPC links (VLAN Pool)
· Domain type and name for the UCS domain
· Attachable Access Entity Profile (AAEP or AEP)
Defining the above fabric access policies for the UCS domain results in the following AEP and relationships. The AEP ties VLAN pools and domains to the vPC, physical interface and interface policies going to the Cisco UCS domain.
Figure 34 Fabric Access Policies – Cisco UCS Domain and HyperFlex Cluster
ACI Fabric Design – Enabling Foundational Infrastructure for Cisco HyperFlex System
This section covers the design of a foundational infrastructure tenant (HXV-Foundation) for enabling infrastructure connectivity to a HyperFlex cluster in a Cisco UCS domain. The same HXV-Foundation tenant can be used across multiple Hyperflex clusters and UCS domains. The HXV-Foundation tenant provides connectivity to all infrastructure VLANs used in a HyperFlex deployment. This tenant is not used for application virtual machines hosted on the HyperFlex cluster.
The infrastructure connectivity enabled by the HXV-Foundation tenant in this design are as follows:
· In-Band Management connectivity to ESXi hosts and HyperFlex Storage Controller VMs (SCVM)
· Storage data connectivity between nodes in the HyperFlex cluster (every cluster should use a unique VLAN and subnet)
· Storage data connectivity between ESXi hosts and HyperFlex storage (datastores)
· Access to vMotion network
· Access to customer’s existing network (outside ACI) and external networks (Internet/Cloud) by ESXi Hosts and by any Infrastructure/Services virtual machines deployed on the HyperFlex cluster.
In an ACI fabric, all applications, services and connectivity are defined using ACI constructs such as tenants, application profiles, bridge domains and EPGs. This design maps the different HyperFlex infrastructure and management networks (Management, vMotion, Storage Data) to different EPGs in the HXV-Foundation tenant. Separate application profiles and bridge domains are used for each EPG. However, the different bridge domains belong a single VRF instance. The design assumes that overlapping IP addressing is not required and therefore a single VRF is used.
The different ACI constructs used in this design for providing infrastructure connectivity to HyperFlex servers in a Cisco UCS domain is shown in Table 22 .
Table 22 Cisco UCS Domain and HyperFlex System – Infrastructure EPGs and EPG VLANs
Table 23 Cisco UCS Domain and HyperFlex System – Infrastructure EPGs, Application Profiles, Contracts
Table 24 Cisco UCS Domain and HyperFlex System – Infrastructure EPGs and Networking
The various ACI constructs used in this design to enable infrastructure connectivity to the HyperFlex system and the relationship between them is shown in Figure 35.
Figure 35 Inband Management Network Connectivity to HyperFlex System (ESXi, SCVM)
Figure 36 vMotion Network Connectivity to HyperFlex System (ESXi, SCVM)
Figure 37 Storage Data Network Connectivity to HyperFlex System (ESXi, SCVM)
The connectivity from existing infrastructure to the HyperFlex system is shown in the figure below. Note that HX Installer VM and other services required for installing the HX system is enabled through this connectivity.
Figure 38 Existing Infrastructure Networks and Services to HyperFlex System Connectivity
Though not shown in the above topology, the Shared L3Out also provides connectivity to external networks for access to cloud-based services such as Cisco Intersight and Cisco Umbrella.
ACI Fabric Design - Integration with Virtual Machine Manager
The Cisco APIC that manages the ACI Fabric automates the networking for all application workloads (virtual and physical) including access policies and L4-L7 services. By integrating with a Virtual Machine Manager (VMM), the Cisco APIC can also control the virtual distributed switching and extend the fabric access policies to the virtual switches running on that VMM. The integration also automates the deployment tasks associated with connecting a virtual machine to the ACI fabric.
The Virtual Machine Manager integration enables Cisco APIC to control and manage the creation and operation of distributed virtual switches running on ESXi hosts. In this design, the APIC integrates with VMware vCenter to manage the creation and operation of virtual distributed switches running on ESXi hosts. To deploy new application virtual machines, the APIC can create EPGs in the ACI fabric, dynamically allocate VLANs to the EPGs, create corresponding port-groups in the VMM domain, and apply network policies. A VMM domain can contain multiple EPGs and hence multiple port groups.
The APIC can manage a VMware vSphere vSwitch, VMware vSphere Distributed Switch (vDS) or a Cisco ACI Virtual Edge (AVE). This design uses an APIC-controlled Cisco ACI Virtual Edge for guest (application) virtual machines hosted on HyperFlex cluster.
ACI Constructs for VMM Integration
VMM Integration requires the following ACI constructs to be defined:
· VMM Domain Profiles
A VMM Domain profile defines a VMM domain and specifies the connectivity policies for connecting VMware vCenter to the ACI fabric. VMMs with common policies can be mapped to a single domain. The access credentials included in the domain enable APIC to VMM communication. This communication channel is used for querying vCenter for info such as VM names and vNIC details, to create port-groups, to push VM policies and to listen for VM events such as VM creation, deletion, etc. A VMM domain provides VM mobility within the domain and support for multi-tenancy within the fabric.
· VLAN Pools
VLAN Pool is a shared resource that can be shared by multiple domains including a VMM domain. However a VMM domain can only be associated with one VLAN pool. VLAN pool specifies the allowed range of VLANs in the VMM domain. Typically, VLANs in a VMM domain pool are dynamically assigned to an EPG by the APIC. APIC also provisions the VMM domain VLAN on the leaf ports.
· Endpoint Groups
A VMM Domain can be associated with multiple EPGs and an EPG can span multiple VMM domains. APIC creates a port-group for every EPG associated with a VMM domain in the ACI fabric. A VMM domain with multiple EPGs will have multiple port groups defined on the virtual distributed switch. To position an application, the application administrator deploys the VMs on VMware vCenter and places the virtual machine’s NIC into the appropriate port group for the application tier.
· Attachable Entity Profile
AEPs associated the domain (and VLAN pool) to the physical infrastructure and can enable VMM policies on leaf switch ports across the fabric.
VMM Integration with Cisco HyperFlex Hosts
For VMM integration to work correctly, it requires Cisco HX nodes and domains to be setup correctly as outlined below.
· The virtual NICs on the HX hosts must be configured to use either CDP or LLDP; only one can be enabled at a time.
· The allowed range of VLANs or the VLAN pool for the VMM domain must be allowed on the HX host’s virtual NIC and on the Fabric Interconnect uplinks connecting to the leaf switches.
The HyperFlex installation will take care of configuring the above if the same VLAN pool was provided during the install process. For additional VLANs, this must be done manually through the Cisco UCS Manager.
ACI Fabric Design – Onboarding Multi-Tier Applications
This section describes the design for deploying applications on a Cisco HyperFlex cluster that connects to a Cisco ACI fabric. The design assumes a multi-tier application where the ACI fabric must explicitly allow access between the different tiers of the application. The design also assumes that the ACI fabric is operational, with reachability to networks and services that the application (tiers) need. This includes integration with the Virtual Machine Manager (VMware vCenter) managing the application virtual machines. This could be external access to access or host cloud services, access to networks within the organizations, access to storage resources and infrastructure services such as NTP, DNS, AD, etc.
The ACI constructs for a multi-tier application deployment include defining a new tenant, VRF(s), bridge domain(s), application profile(s), end point group(s), and the contract(s) to allow communication between various tiers of the application.
To deploy a sample two-tier application, following elements are configured:
· A new Tenant called HXV-App-A is defined to host the application.
· A VRF called HXV-App-A_VRF is defined under the tenant to provide the tenant IP address space.
· A bridge domain HXV-App-A-Int_BD is created in the tenant.
· An optional bridge domain HXV-App-A-Ext_BD is created in the tenant for additional L2 segregation of incoming user traffic.
· An application profile, HXV-App-A_AP is utilized to deploy the application.
· Two EPGs, HXV-A-Web_EPG and HXV-A-App_EPG are associated with the VMM domain to host Web and App/DB tiers of the application.
· A contract to allow communication between the two application tiers is defined. This contract (Allow-Web-to-App) is ”provided” by the App_EPG and ”consumed” by the Web_EPG.
· The Web_EPG also has access to outside networks by consuming the contract (Allowed-Shared-L3Out) provided by the Shared L3Out
· When the EPGs are associated with an APIC-controlled virtual distributed switch (Cisco AVE or vDS) in the VMM domain, Cisco assigns a VLAN to each EPG from a pre-defined pool and uses the established VMM integration to create a corresponding port-group for that VLAN in the virtual distributed switch. These port-groups are then used to deploy the virtual machines (Web, App/dB in the different application tiers.
Figure 39 provides an overview of the ACI constructs required for deploying a sample two-tier application.
Figure 39 Cisco ACI Fabric Design – Multi-Tier Application
Though not shown in the above topology, the Shared L3Out also provides connectivity to external networks for access to cloud-based services such as Cisco Intersight and Cisco Umbrella.
Validated Software and Hardware Components
Table 25 lists the software versions for hardware and virtual components used in this solution.
Table 25 Software and Hardware Components
The versions used for this validation were verified for interoperability using the following matrixes:
· Cisco UCS and HyperFlex Hardware and Software Interoperability Tool
· Cisco ACI Recommended Release
· Cisco ACI Virtualization Compatibility Matrix
· Cisco APIC and ACI Virtual Edge Support Matrix
To use versions that differs from the validated versions above, please confirm interoperability using the above links and review the release notes for the components and releases.
This section provides detailed procedures for configuring an existing Data Center Fabric based on Cisco ACI to support a Cisco HyperFlex deployment. In this section, the following aspects of the deployment are covered.
· Deploying a pair of leaf switches to connect the HyperFlex system and UCS domain to the ACI fabric.
· Enabling Access to HyperFlex installer outside the ACI fabric.
· Enabling Access to a new Cisco UCS Domain where a new HyperFlex system will be installed. This includes access to different HyperFlex networks for Management, Storage and vMotion that the HyperFlex Installer will setup.
· Enabling access to applications hosted on the Cisco HyperFlex system.
ACI Fabric Core
The design assumes that an ACI fabric of Spine switches and APICs already exists in the customer’s environment so this document verifies the existing setup but does not cover the configuration required to bring the initial ACI fabric online.
Review Existing ACI Fabric Topology
Before adding leaf switches to connect to a new Cisco HyperFlex environment, review the topology by completing the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select System > Controllers.
3. From the left navigation pane, select Controllers to review the APICs topology and connectivity to the fabric. In a production environment, ACI recommends a minimum of 3 APICs. In the lab setup used for validating this CVD design, a single APIC is used as shown below.
4. To see the complete ACI fabric topology, from the top menu, select Fabric > Inventory.
5. In the left navigation pane, select the Pod to view the Pod topology and in the right pane, select Topology.
Verify Time Zone and NTP Setup
To review/enable the time zone and NTP configuration on the ACI fabric, complete the following steps:
1. Use a browser to navigate and login to APIC’s Web GUI.
2. From the top menu, select Fabric > Fabric Policies.
3. In the left navigation pane, expand Policies and select Policies > Pod > Date and Time > Policy default.
4. Verify that at least one NTP Server is listed.
5. To add an NTP server, click the [+] sign in the NTP Servers section of Policy default to specify a new NTP server’s IP address – the new server should use the default (Out-of-Band) Management EPG if the NTP server is reachable through the OOB management subnet.
6. (Optional) Select enabled for Server State to enable the ACI fabric switches as NTP servers.
7. Click Submit complete the NTP configuration. Repeat this process to add more NTP servers.
Verify Domain Name Servers
To review/enable Domain Name Server (DNS) configuration on the ACI fabric, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. From the top menu, select Fabric > Fabric Policies.
2. In the left navigation pane, expand Policies and select Policies > Global > DNS Profiles > default.
3. Verify DNS Providers and DNS Domains sections in the default DNS Profile.
4. For the Management EPG, select the default (Out-of-Band) from the drop-down list if the DNS servers are reachable through the out of band management subnet.
5. Use the [+] signs to the right of DNS Providers and DNS Domains to add DNS servers and domains as needed.
Verify BGP Route Reflector Setup
To distribute routes within the ACI fabric, multiprotocol BGP (MP-BGP) is implemented between the leaf and spine switches. The spine switches are set up as BGP route-reflectors (RR) for propagating routes through out fabric and provides a scalable L3 design for supporting a large number of leaf switches. To review/enable the BGP Route Reflector configuration, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. From the top menu, select System > System Settings.
2. In the left navigation pane, select BGP Route Reflector.
3. Verify the BGP Route-Reflector configuration. Specify a unique Autonomous System Number for the Internal BGP domain in the ACI fabric. If necessary, use the [+] sign on the right to add spines to the list of Route Reflector Nodes.
4. Click Submit to complete the BGP Route Reflector configuration.
5. To verify the BGP Route Reflector has been enabled, from top menu, select Fabric > Fabric Policies.
6. From the left pane, select Pods > Policy Groups and select the policy group. The BGP Route Reflector Policy field should be default as shown below.
7. If a Policy Group is not present, from the left navigation pane, expand and select Pods > Policy Groups and right-click to select Create Pod Policy Group. In the Create Pod Policy Group window, specify a Policy Group name and select ‘default’ from the drop-down list for the BGP Route Reflector Policy. Click Submit.
8. To apply the above policy-group, in the left navigation pane, select Pods > Profiles > Pod Profile default > default. If the Fabric Policy Group is not specified, select it from the drop-down list. Click Submit.
Enable Enforce Subnet Check for IP Learning (Fabric Wide Setting)
Enforce Subnet Check in ACI limits both local and remote IP endpoint learning in a VRF to source IP addresses that belong to one of the bridge domain subnets in the VRF. This a system wide policy that impacts data plane learning at the VRF level. It also requires second generation hardware on the leaf switches. Note that for local learning, the source IP address must be in its bridge domain subnet but for remote learning, the source IP just needs to match one of the bridge domain subnets for the VRF.
To enable Enforce Subnet Check for IP Learning feature, complete the following steps:
1. Use a browser to navigate and login to APIC’s Web GUI.
2. From the top menu, select System > System Settings.
3. In the left navigation pane, select Fabric Wide Setting.
4. Enable check box for Enforce Subnet Check. Click Submit.
Enable IP Aging (System Wide Setting)
IP Aging tracks and ages endpoint IP addresses that the fabric has learned, to age out stale entries. This is a system wide feature. To enable IP aging feature, complete the following steps:
1. Use a browser to navigate and login to APIC’s Web GUI.
2. From the top menu, select System > System Settings.
3. In the left navigation pane, select Endpoint Controls.
4. For Administrative State, click Enabled. Click Submit.
COS Preservation (Fabric Wide Setting)
Class Of Service (COS) Preservation feature in ACI preserves the COS setting in the traffic received from the endpoints. When connecting a HyperFlex system to an ACI fabric, this feature should be enabled so that QOS can be ensured within the UCS domain when receiving HyperFlex traffic forwarded by the ACI fabric. This a global access policy that has a fabric wide impact.
To enable COS Preservation, complete the following steps:
1. Use a browser to navigate and login to APIC’s Web GUI.
2. From the top menu, select Fabric > Access Policies.
3. In the left navigation pane, select and expand Policies > Global > QOS Class.
4. Confirm that Preserve QOS is selected.
ACI Fabric Access – Pre-configure Fabric Access Policies
Fabric Access Policies are configurations that can be applied to access layer connections on leaf switches and to VMM domains. The individual configurations are maintained as policies to enable re-use across the fabric. The access layer connections include vPC connections to HyperFlex environment, physical connectivity to outside Management networks and routed connections to external L3 networks. Cisco recommends configuring the policies explicitly rather than using defaults since they may change in future releases. Table 26 provides a summary of the access policies used in this design. This section covers the procedures for creating the policies – they will be used in later configurations.
Table 26 Fabric Access Policies
All policies are configured from the following location in the GUI:
1. Use a browser to navigate and login to the APIC’s Web GUI.
2. From the top menu, select Fabric > External Access Policies.
3. In the left navigation pane, select and expand Policies > Interface.
Create Link Level Policies
To create the link level policies to specify link speeds of 1/10/40-Gbps, complete the following steps:
1. In the left navigation pane, select Link Level. Right-click and select Create Link Level Policy.
2. In the Create Link Level Policy pop-up window, specify the policy Name as 1Gbps-Link. For the Speed, select 1Gbps from the drop-down list.
3. Click Submit to complete creating the policy.
4. Repeat the above steps to create a link policies for 10Gbps and 40Gbps speeds.
Create CDP Interface Policies
To create the CDP interface policies, complete the following steps:
1. In the left navigation pane, select CDP Interface. Right-click and select Create CDP Interface Policy.
2. In the Create CDP Interface Policy pop-up window, specify the policy Name as CDP-Enabled. For the Admin State, click Enabled.
3. Click Submit to complete creating the policy.
4. Repeat the above steps to create a policy that disables CDP called CDP-Disabled.
Create LLDP Interface Policies
To create the LLDP interface policies, complete the following steps:
1. In the left navigation pane, select LLDP Interface. Right-click and select Create LLDP Interface Policy.
2. In the Create LLDP Interface Policy pop-up window, specify the policy Name as LLDP-Enabled. For the Receive and Transmit State, click Enabled.
3. Click Submit to complete creating the policy.
4. Repeat the above steps to create a policy to disable LLDP called LLDP-Disabled. The Receive and Transmit states for this policy should be Disabled.
Create Port Channel Policy
To create the Port Channel policy, complete the following steps:
1. In the left navigation pane, select Port Channel. Right-click and select Create Port Channel Policy.
2. In the Create Port Channel Policy pop-up window, specify the policy Name as LACP-Active. For the mode, select LACP-Active from the drop-down list. Leave everything else as is.
3. Click Submit to complete creating the policy.
4. Repeat the steps above to create a MAC-Pinning policy as shown below.
5. Click Submit to complete creating the policy.
6. Repeat the steps above to create a MAC-Pinning-Phy-NIC-Load policy as shown below.
Create L2 Interface (VLAN Scope) Policies
To create the L2 interface policies, complete the following steps:
1. In the left navigation pane, select L2 Interface. Right-click and select Create L2 Interface Policy.
2. In the Create L2 Interface Policy pop-up window, specify the policy Name as VLAN-Scope-Local. For the VLAN Scope, select Port Local scope.
3. Click Submit to complete creating the policy.
4. Repeat the steps above to create a VLAN-Scope-Global policy as shown below.
Create Spanning Tree Interface Policies
To create the Spanning Tree interface policies, complete the following steps:
1. In the left navigation pane, select Spanning Tree Interface. Right-click and select Create Spanning Tree Interface Policy.
2. In the Create Spanning Tree Interface Policy pop-up window, specify the policy Name as BPDU-FG-Enabled. For Interface Controls, select check-box for BPDU Filter enabled and BPDU Guard enabled.
3. Click Submit to complete creating the policy.
4. Repeat the above steps to create a BPDU-FG-Disabled policy where BPDU Filter and Guard are both disabled.
Create Firewall Policy
Cisco ACI best practices recommend explicitly configuring policies and since a firewall is not used in this design, a policy to disable firewall is created by following these steps:
1. In the left navigation pane, select Firewall. Right-click and select Create Firewall Policy.
2. In the Create Firewall Policy pop-up window, specify the policy Name as Firewall-Disabled.
3. For Mode, select Disabled. Leave all other values as is.
4. Click Submit to complete creating the policy.
Deploy New Leaf Switches for HyperFlex Environment
In ACI, all configuration is centralized and managed through the APIC – there is no individual configuration of the Spine and Leaf switches. APIC discovers the ACI capable Nexus 9000 series switches in the infrastructure using Link Layer Discovery Protocol (LLDP). The newly discovered switches are added, provisioned and managed from the APIC web GUI.
ACI Fabric Discovery of Leaf Switches
When new switches running ACI software are connected to the spines, they are automatically discovered in the fabric through LLDP. To verify that the fabric sees the new leaf switches, complete the following steps:
1. Use a browser to navigate and login to APIC’s Web GUI.
2. From the top menu, select Fabric > Inventory.
3. In the left navigation pane, select Fabric Membership.
4. The newly discovered Leaf Switches will be listed with a Node ID of ‘0’.
5. Note the serial numbers of the newly discovered leaf switches.
Add Nexus 9000 Series Leaf Switches to the ACI Fabric
The newly discovered Nexus 93180YC-EX leaf switches from the previous step can now be added to the ACI fabric. To add the new leaf, complete the following steps:
1. Identify the -1 and -2 switches in the new leaf switch pair.
2. Determine the serial numbers corresponding to the -1 and -2 switches to map it to the ones collected in the previous step. To find the serial number for a given leaf switch, access it’s serial console, login using admin account (no password) and run the command: show inventory.
3. Use a browser to navigate and login to APIC’s Web GUI.
4. From the top menu, select Fabric > Inventory.
5. In the left navigation pane, select Fabric Membership.
6. In the right pane, from the list of switches with Node ID of ‘0’, select and double-click the serial number corresponding to the -1 leaf. Enter a Pod ID, Node ID and a Node Name for the Leaf switch and click Update.
7. Repeat the previous step for -2 switch in the leaf pair.
8. The Fabric Membership information now reflects the updated configuration and the columns for SSL certificate and Status is same as other active switches in the fabric.
9. Navigate to Fabric > Fabric Policies and select the Pod that the N9k switches were added to.
10. From the right-window pane, select Topology tab and the topology should now show the newly added switches.
Setup Out-of-Band Management for New Leaf Switches
To enable out-of-band (OOB) management access to the switches in the ACI fabric, ACI provides a pre-defined mgmt Tenant. To enable OOB connectivity to the new leaf switches, complete the following steps:
1. Use a browser to navigate and login to APIC’s Web GUI.
2. From the top menu, select Tenants > mgmt.
3. In the left navigation pane, select and expand Tenant mgmt > Node Management Addresses > Static Node Management Addresses.
4. Right-click and select Create Static Node Management Addresses.
5. In the Create Static Node Management Addresses pop-up window, enter the node ID range for the new leaf switches (in this case, 1105-1106) and select the check-box for Out-Of-Band Addresses. For the OOB Management EPG, select ‘default’ from the drop-down list. Also provide the IP addresses and Gateway IPs for OOB.
Consecutive IP addresses will be assigned for the range of nodes so only a starting IP address needs to be specified.
6. Click Submit and then Yes to proceed with assigning the IP addresses.
The leaf switches can now be accessed via OOB Management using SSH.
Enabling Access to Cisco HyperFlex Installer and Existing Infrastructure
The HX Installer VM deployed on existing virtualized infrastructure outside the ACI fabric is reachable through a Shared L3Out connection. The VMware vCenter that will manage the HX cluster being deployed will also be reachable through this connection. Other infrastructure services and network connectivity necessary for HyperFlex installation such as DNS, NTP and out-of-band management connectivity to Cisco UCS Fabric Interconnects and CIMC access to HX servers are also reachable through this connection. External connectivity or connectivity outside the customer’s network for accessing cloud services, (for example, Cisco Intersight) is also through this connection.
To enable connectivity to the above networks, services and HX installer VM, a Layer 3 routed connection is established between a pair of leaf switches in the ACI fabric and a pair of Nexus 7000 gateway routes in the customer’s existing, non-ACI network. The leaf switches are a pair of Nexus 9372 routers; these are different leaf switches from the ones that HyperFlex and UCS domain connect to.
Complete the steps outlined in this section to establish Layer 3 connectivity (also referred to as Shared L3Out) between ACI fabric and non-ACI fabric to enable access to HyperFlex installer, VMware vCenter, NTP, DNS, and other internal and external networks.
For details about the design, see the ACI Fabric Design - Layer 3 Routed Connectivity to Outside Networks section of this guide.
ACI Fabric Deployment – Layer 3 Routed Connectivity to Outside Networks
The Shared L3Out connection is established in the common Tenant so that the access can be easily shared by other tenants within the ACI fabric. The configuration utilizes four interfaces between the pair of the ACI leaf switches and the pair of Nexus 7000 switches. The routing protocol being utilized is OSPF. Some highlights of this connectivity are:
· Two Nexus 7000 routers are connected to two Nexus ACI leaf switches – using a total of 4 links.
· Sub-interfaces are configured and used for enabling this connectivity. VLANs are used for connectivity across the 4 links – using a total of 4 VLANs.
· Configure a Fabric Access Policy – specifically an AEP profile that specifies the domain (External Routed) and VLAN pool (vlans: 305-308) to use.
· A dedicated bridge domain bd-common-outside and associated dedicated VRF vrf-common-outside is configured in tenant common for external connectivity.
· The shared Layer 3 Out created in common Tenant “provides” an external connectivity contract that can be “consumed” from any tenant.
· The Nexus 7000s are configured to originate and send a default route to the Nexus 9000 leaf switches using OSPF.
· ACI leaf switches advertise tenant subnets back to Nexus 7000 switches.
Configure External Routed Domain
To configure the external routed domain, complete the following steps:
1. At the top, select Fabric > Access Policies.
2. In the left pane, expand Physical and External Domains > External Routed Domains.
3. Right-click External Routed Domains and select Create Layer 3 Domain.
4. Specify a name for the domain. (For example: AA07N7K-SharedL3Out)
5. From the Associated Attachable Entity Profile drop-down list, select Create Attachable Entity Profile.
6. Specify a name for the AEP profile, for example: AA07N7k-SharedL3Out-AttEntityP. Click Next.
7. Click Finish to continue without specifying interfaces.
8. In the Create Layer 3 Domain window, use the VLAN Pool drop-down list to select Create VLAN Pool.
9. Specify a name for the VLAN Pool, for example: RowA-SharedL3Out-vlan.
10. Select Static Allocation. Click [+] to add an Encap Block.
11. In the Create Ranges window, enter the VLAN range for connectivity to Nexus 7000 switches (305-308). Select Static Allocation.
12. Click OK to complete adding the VLAN range.
13. Click Submit to complete creating the VLAN Pool.
14. Click Submit to complete creating the Layer 3 Domain.
Configure Leaf Switch Interfaces
To configure the leaf switch interfaces, complete the following steps:
1. In the APIC Advanced GUI, select Fabric > Access Policies > Quick Start.
2. In the right pane, select Configure and interface, PC and VPC.
3. Click the [+] icon underneath the listing of Configured Switch Interfaces on the left hand side.
4. From the right-side of the window, click the drop-down list for Switches and select the two leaf switches that have been cabled to connect to the non-ACI gateways (Nexus 7Ks in this design).
5. Click 
in the right side to add switch interfaces.
6. Configure the various fields as shown in the screenshot below. In this design, ports 1/47 and 1/48 are connected to the Nexus 7K switches using 10Gbps links.
7. Click Submit.
8. Repeat for the second leaf switch in the pair (in this design: Node 1102)
Configure External Routed Networks under Tenant Common
To configure the external routed networks under tenant common, complete the following steps:
1. From the top navigation menu, select Tenants > common.
2. In the left pane, select and expand Tenant common > Networking > External Routed Networks.
3. Right-click External Routed Networks and select Create Routed Outside.
4. Name the Routed Outside, for example: Nexus-7K-Shared.
5. Check the check box next to OSPF.
6. Enter 0.0.0.10 (matches Nexus 7000s configuration) as the OSPF Area ID.
7. From the VRF drop-down list, select vrf-common-outside.
8. From the External Routed Domain drop-down list, select AA07N7K-SharedL3Out.
9. Under Nodes and Interfaces Protocol Profiles, click [+] to add a Node Profile.
10. Name the Node Profile Node-1101-1102.
11. Click + to add a Node.
12. In the select Node and Configure Static Routes window, select Leaf switch 1101 from the drop-down list.
13. Provide a Router ID IP address – this address will be configured as the Loopback Address. The address used in this deployment is 10.251.255.1.
14. Click OK to complete selecting the Node.
15. Click [+] to add another Node.
16. In the select Node window, select Leaf switch 1102.
17. Provide a Router ID IP address – this address will be configured as the Loopback Address. The address used in this deployment is 10.251.255.2.
18. Click OK to complete selecting the Node.
19. Click [+] to create an OSPF Interface Profile.
20. Specify a name for the Node Interface profile, for example: Node-1101-1102-IntfPol.
21. Click Next.
22. Using the OSPF Policy drop-down list, select Create OSPF Interface Policy.
23. Name the policy ospf-Nexus-7K.
24. Select the Point-to-Point Network Type.
25. Select the MTU ignore Interface Controls.
26. Click Submit to complete creating the policy.
27. Click Next.
28. Click [+] to add a routed sub-interface.
29. Select Routed Sub-Interface under Interfaces.
For adding Routed Sub-interfaces, refer to Figure 32 for Interface, IP and VLAN details
30. In the Select Routed Sub-Interface window, for Path, select the interface on Nexus 9372-1 (Node 1101) that is connected to Nexus 7004-1.
31. Enter vlan-<interface vlan> (305) for Encap.
32. Enter the IPv4 Address (10.251.231.1/30)
33. Leave the MTU set to inherit.
34. Click OK to complete creating the routed sub-interface.
35. Repeat these steps to all four sub-interfaces.
36. Click OK to complete creating the Interface Profile.
37. Click OK to complete creating the Node Profile.
38. Click Next on the Create Routed Outside Screen.
39. Click [+] to create an External EPG Network.
40. Name the External Network Default-Route.
41. Click [+] to add a Subnet.
42. Enter 0.0.0.0/0 as the IP Address. Select the checkboxes for External Subnets for the External EPG, Shared Route Control Subnet, and Shared Security Import Subnet.
43. Click OK to complete creating the subnet.
44. Click OK to complete creating the external network.
45. Click Finish to complete creating the Routed Outside.
46. In the left pane, right-click Contracts and select Create Contract.
47. Name the contract Allow-Shared-L3-Out.
48. Select the Global Scope to allow the contract to be consumed from all tenants.
49. Click [+] to add a contract subject.
50. Name the subject Allow-Shared-L3-Out.
51. Click [+] to add a filter.
52. From the drop-down list, select common/default to create a default filter for Tenant common.
53. Click Update.
54. Click OK to complete creating the contract subject.
55. Click Submit to complete creating the contract.
56. In the left pane expand Tenant common > Networking > External Routed Networks > Nexus-7K-Shared > Networks. Select Default-Route.
57. In the right pane under Policy, select Contracts.
58. Click [+] to add a Provided Contract.
59. Select the common/Allow-Shared-L3-Traffic contract.
60. Click Update.
Tenant EPGs can now consume the Allow-Shared-L3-Traffic contract and route traffic outside fabric. This deployment example shows default filter to allow all traffic. More restrictive contracts can be created for more restrictive access outside the Fabric.
Nexus 7000 Router Sample Configuration – Layer 3 Gateway to Outside Networks
See the Appendix for the routing configuration on the two Nexus 7000 routers used in this design.
Enable Access to Cisco UCS Domain and HyperFlex Nodes
To use the compute and storage resources provided by a Cisco HyperFlex cluster, the HyperFlex system must first be deployed on Cisco HX-series nodes connected to Cisco UCS Fabric Interconnects. Cisco HyperFlex system can be deployed either:
· From the Cloud using Cisco Intersight or
· Using a HyperFlex Installer deployed in an existing virtualization environment
In this design, the HyperFlex system is installed using an installer VM deployed in an existing virtualization environment, outside the ACI fabric. However, before the HyperFlex system can be deployed, the ACI fabric must provide connectivity from the HX installer to the HX servers connected to Cisco UCS Fabric Interconnects in the Cisco UCS domain. Enabling this connectivity requires the following:
· The ACI fabric must have connectivity to the HX installer and other infrastructure services and networks required to complete the installation. This is required for the HX Installation and was covered in the previous section.
· The Cisco UCS domain (with HX servers) must be physically connected to the ACI fabric.
· The access configuration (or Fabric Access Policies) for enabling connectivity to the Cisco UCS domain from the ACI fabric must be complete.
In this section, the access configuration enable connectivity to Cisco UCS domain from the ACI fabric is covered and assumes that the physical connectivity is in place (HX servers to UCS Fabric Interconnects, UCS Fabric Interconnects to ACI Leaf).
Deployment Workflow
The workflow for configuring the policies on leaf switches that connect to a Cisco UCS domain (where HyperFlex attaches to) is shown in Figure 40.
Figure 40 Fabric Access Policies – To Cisco UCS Domain and HyperFlex Cluster
Create Virtual Port Channels to Cisco UCS Domain for Cisco HyperFlex Deployment
In this section, access layer connectivity is established between the ACI fabric and the Cisco UCS Domain where the Cisco HyperFlex cluster will be deployed. The Cisco UCS Domain consists of a pair of Cisco UCS Fabric Interconnects (FI-A, FI-B) – multiple Cisco HyperFlex clusters and UCS (rack, blade) servers can connect into a pair of Cisco UCS Fabric Interconnects.
To enable this connectivity, two virtual Port Channels (vPCs) are created on a pair of newly deployed Leaf switches (see earlier section) to each Cisco UCS Fabric Interconnect (FI-A, FI-B) where Cisco HyperFlex system will be deployed.
Complete the following steps to create vPCs from the newly deployed ACI leaf switches to the first UCS Fabric Interconnect (HXV1-6300FI-A) in the UCS Domain. Repeat the steps to create a second vPC to the second UCS Fabric Interconnect.
Create VLAN Pool for Cisco UCS Domain
The VLAN Pool is for the HX cluster deployed in the UCS Domain. A single UCS domain can support multiple HX and UCS clusters but in this design, it is a single HX cluster and therefore only HyperFlex VLANs are in the pool. Table 27 shows the VLANs used in this design.
Table 27 VLAN Pool – EPG VLANs to Cisco UCS Domain and HyperFlex Cluster
To configure a VLAN pool to the Cisco UCS domain where the Cisco HX Cluster is deployed, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Fabric > External Access Policies.
3. From the left navigation pane, expand and select Pools > VLAN.
4. Right-click and select Create VLAN Pool.
5. In the Create VLAN Pool pop-up window, specify a Name and for Allocation Mode, select Static Allocation.
6. For Encap Blocks, use the [+] button on the right to add VLANs to the VLAN Pool. In the Create Ranges pop-up window, configure the VLANs that need to be trunked from the Cisco UCS FIs to the ACI Fabric. Leave the remaining parameters as is.
7. Repeat for all VLANs in the VLAN Pool for Cisco UCS Domain (see table at the beginning of this section) – these VLANs will need to be enabled on the Cisco UCS FI uplinks to the ACI Fabric as well. Note that the HyperFlex storage VLANs are local and unique (recommended) to each HyperFlex cluster but are still trunked to the ACI Fabric to handle failure situations where different hosts are forwarding on different UCS fabrics (FI-A, FI-B).
8. Click Submit to complete.
Configure Domain Type for Cisco UCS Domain
To specify the domain type for the access layer connection to the Cisco UCS domain, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Fabric > External Access Policies.
3. From the left navigation pane, expand and select Physical and External Domains > External Bridged Domains.
4. Right-click and select Create Layer 2 Domain.
5. In the Create Layer 2 Domain pop-up window, specify a Name and select the previously created VLAN Pool from the drop-down list.
6. Click Submit to complete.
Create Attachable Access Entity Profile for Cisco UCS Domain
To specify the domain type for the access layer connection to the Cisco UCS domain, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Fabric > External Access Policies.
3. From the left navigation pane, expand and select Policies > Global > Attachable Access Entity Profile.
4. Right-click and select Create Attachable Access Entity Profile.
5. In the Create Attachable Access Entity Profile pop-up window, specify a Name.
6. For the Domains, click the [+] on the right-side of the window to add the previously created domain.
7. Click Update. You can view the selected domain and the associated VLAN Pool as shown below.
8. Click Next and Finish. This profile is not associated with any interfaces at this time as they have not yet been configured.
Create Interface Policies for the vPC Interfaces to Cisco UCS Domain
The pre-configured Fabric Access Policies from an earlier section can be used for configuring the policies for a specific access layer connection. The pre-configured policies selected for the specific access layer connection is configured in the next section using the Interface Policy Group.
Create Interface Policy Group for the vPC Interfaces to Cisco UCS Domain
To create the interface policy group for the vPC interfaces to Cisco UCS domain, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Fabric > External Access Policies.
3. From the left navigation pane, expand and select Interfaces > Leaf Interfaces > VPC Interface.
4. Right-click and select Create VPC Interface Policy Group.
5. In the Create VPC Interface Policy Group pop-up window, select the previously created interface policies from the drop-down list for each field. The same Policy Group can be reused on other Cisco UCS Domains that share the policies selected below. For the Attached Entity Profile, select the previously created AAEP to Cisco UCS Domain.
6. Click Submit to complete.
7. Repeat the above procedure to create vPC Interface Policy Group for the second Fabric Interconnect (FI-B).
Create Leaf Interface Profile for the vPC Interfaces to Cisco UCS Domain
To create the leaf interface profile for the vPC interfaces to Cisco UCS domain, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Fabric > External Access Policies.
3. From the left navigation pane, expand and select Interfaces > Leaf Interfaces > Profiles.
4. Right-click and select Create Leaf Interface Profile.
5. In the Create Leaf Interface Profile pop-up window, specify a profile Name and for Interface Selectors, click the [+] to select access ports to apply interface policies to. In this case, the interfaces are access ports connecting Leaf switches to 1st FI (FI-A) in the Cisco UCS domain. In the Create Access Port Selector pop-up window, specify a selector Name, for the Interface IDs, provide the access ports going to FI-A and select the previously created Policy Group for the Interface Policy Group.
6. Click OK and Submit to complete.
7. Repeat above steps to select the interfaces in the second vPC to FI-B in the Cisco UCS domain by clicking the [+] to add more Access Ports Selectors for the same Interface Profile.
8. Verify that all vPC interfaces to UCS have been added and are listed in the Interface Selectors section.
Create Switch Policies for the vPC Interfaces to Cisco UCS Domain
To create the switch policies for the vPC interfaces to Cisco UCS domain, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Fabric > External Access Policies.
3. From the left navigation pane, expand and select Policies > Switch > Virtual Port Channel default.
4. Right-click and select Create VPC Explicit Protection Group.
5. In the Create VPC Explicit Protection Group pop-up window, specify a Name and for the ID, provide the vPC Domain ID for the Leaf pair. For Switch 1 and Switch 2, select the Node IDs of the leaf pair from the list.
6. Click Submit to complete.
Create Leaf Switch Profile
To create the leaf switch profile, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Fabric > External Access Policies.
3. From the left navigation pane, expand and select Switches > Leaf Switches > Profiles.
4. Right-click and select Create Leaf Profile.
5. In the Create Leaf Profile pop-up window, specify a profile Name and for Leaf Selectors, click the [+] to select the leaf switches to apply the switch policies to. Specify a Leaf Pair Name and select the Node IDs for the Leaf Switches going to Cisco UCS Domain. Click Update. Click Next.
6. In the Associations window, select the previously created Interface Selector Profiles from the list.
7. Click Finish to complete.
Enabling Foundational Infrastructure for Cisco HyperFlex System
In this section, you will create a tenant that provides a foundational infrastructure, management, and services. For HyperFlex, this tenant provides connectivity to the HyperFlex cluster for:
· In-Band Management Access to all ESXi hosts in the HX cluster. This is required not only to manage the HX nodes from VMware vCenter but also for the initial HyperFlex install. The same network will be used to access HyperFlex controller VMs deployed on the ESXi hosts.
· Storage Data Connectivity for all storage data traffic in the HyperFlex cluster. This includes ESXi hosts accessing datastores on HX storage cluster but also for the storage data network that provides the HX storage cluster where the user datastores reside.
· vMotion Network to enable vMotion across the ACI fabric.
Create Foundation Tenant and VRF
To create the foundation tenant and VRF, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Tenants > Add Tenant.
3. In the Create Tenant pop-up window, specify a Name (for example, HXV-Foundation).
4. For the VRF Name, enter a name for the only VRF in this Tenant (for example, HXV-Foundation_VRF)
5. Check the box for “Take me to this tenant when I click finish.”
6. Click Submit to complete.
Configure ACI Fabric for HyperFlex In-Band Management
This section describes configuring the Cisco ACI fabric to support in-band management traffic across the ACI fabric.
Create Bridge Domain for In-Band Management
To create a Bridge Domain for In-Band Management, complete the following steps:
1. Navigate to Tenants > HXV-Foundation.
2. In the left navigation pane, expand Tenant HXV-Foundation > Networking > Bridge Domains.
3. Right-click and select Create Bridge Domain.
4. Specify a Name.
5. For the VRF, select the previously created VRF from the drop-down list.
6. For Forwarding, select Custom and for L2 Unknown Unicast, select Flood. Click Next.
7. In L3 Configurations section, check the box for EP Move Detection Mode – GARP based detection. Click Next.
8. Skip the Advanced/Troubleshooting section. Click Finish to complete.
Create Application Profile for In-Band Management
To create an application profile for In-Band Management, complete the following steps:
1. In the left navigation pane, select Tenant HXV-Foundation. Right-click and select Create Application Profile.
2. In the Create Application Profile pop-up window, specify a Name the Application Profile and click Submit to complete.
Create EPG for In-Band Management and Associate with Bridge Domain
To create an EPG for In-Band Management, complete the following steps:
1. In the left navigation pane, navigate to Tenant HXV-Foundation > Application Profiles > HXV-IB-MGMT_AP.
2. Right-click and select Create Application EPG.
3. In the Create Application EPG pop-up window, specify a Name for the EPG.
4. For Bridge Domain, select the previously created Bridge Domain.
5. Click Finish.
Associate EPG with UCS Domain
To associate the In-Band Management EPG with UCS Domain, complete the following steps:
1. In the left navigation pane, expand the newly created EPG.
2. Select Domains. Right-click and select Add L2 External Domain Association.
3. In the Add L2 External Domain Association pop-up window, select the UCS Domain (for example, HXV-UCS_Domain) from the list.
4. Click Submit.
Create Static EPG and VLAN Binding on vPC Interfaces to UCS Domain
To statically bind the In-Band Management EPG and VLANs to vPC interfaces going to the UCS Domain, complete the following steps:
1. In the left navigation pane, navigate to HXV-IB-MGMT_EPG > Static Ports.
2. Right-click and select Deploy Static EPG on PC, VPC or Interface.
3. In the Deploy Static EPG on PC, VPC or Interface pop-up window, for Path Type, select Virtual Port Channel.
4. For the Path, select the vPC to the first UCS Fabric Interconnect from the drop-down list.
5. For the Port Encap, leave VLAN selected and specify the VLAN ID for the In-Band Management EPG.
6. For the Deployment Immediacy, select Immediate.
7. Click Submit.
8. Repeat these steps to bind the EPG to the vPC going to the second UCS Fabric Interconnect.
Configure Gateway for IB-MGMT EPG
To configure a default gateway for the IB-MGMT EPG, complete the following steps:
1. In the left navigation pane, navigate to HXV-IB-MGMT_EPG > Subnets.
2. Right-click and select Create EPG Subnet.
3. In the Create EPG Subnet pop-up window, specify the Default Gateway IP and for Scope, unselect Private to VRF and select Advertised Externally and Shared between VRFs. Leave everything else as is.
4. Click Submit.
Create Contract to Access Outside Networks via Shared L3Out
To create a contract to access Shared L3Out in the common Tenant, complete the following steps:
1. In the left navigation pane, navigate to HXV-IB-MGMT_EPG > Contracts.
2. Right-click and select Add Consumed Contract.
3. In the Add Consumed Contract pop-up window, select the Allow-Shared-L3Out contract from the drop-down list.
4. Click Submit.
Configure ACI Fabric for HyperFlex Storage Data Traffic
This section covers the configuration of the Cisco ACI fabric to enable forwarding of HyperFlex storage data traffic within the HyperFlex cluster. This is minimally required to support failure scenarios where traffic from one UCS Fabric needs to be forwarded to another when different hosts are forwarding on different fabrics (FI-A, FI-B).
Create Bridge Domain for HyperFlex Storage Data Traffic
To create a Bridge Domain for Storage data traffic, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. Navigate to Tenants > HXV-Foundation.
3. In the left navigation pane, expand Tenant HXV-Foundation > Networking > Bridge Domains.
4. Right-click and select Create Bridge Domain.
5. Specify a Name.
6. For the VRF, select HXV-Foundation_VRF from the drop-down list.
7. For Forwarding, select Custom and for L2 Unknown Unicast, select Flood. Click Next.
8. In the L3 Configurations section, check the box for EP Move Detection Mode – GARP based detection. Click Next.
9. Skip the Advanced/Troubleshooting section.
10. Click Finish to complete.
Create Application Profile for HyperFlex Storage Data Traffic
To create an application profile for HyperFlex Storage data traffic, complete the following steps:
1. In the left navigation pane, select Tenant HXV-Foundation. Right-click and select Create Application Profile.
2. In the Create Application Profile pop-up window, specify a Name the Application Profile.
3. Click Submit to complete.
Create EPG for HyperFlex Storage and Associate with Bridge Domain
To create an EPG for HyperFlex storage data traffic, complete the following steps:
1. In the left navigation pane, navigate to Tenant HXV-Foundation > Application Profiles > HXV-Storage_AP.
2. Right-click and select Create Application EPG.
3. In the Create Application EPG pop-up window, specify a Name for the EPG.
4. For Bridge Domain, select the previously created Bridge Domain.
5. Click Finish.
Associate EPG with UCS Domain
To associate the HyperFlex Storage EPG with UCS Domain, complete the following steps:
1. In the left navigation pane, expand the newly created EPG.
2. Select Domains. Right-click and select Add L2 External Domain Association.
3. In the Add L2 External Domain Association pop-up window, select the UCS Domain (for example, HXV-UCS_Domain) from the list.
4. Click Submit.
Create Static EPG and VLAN Binding on vPC Interfaces to UCS Domain
To statically bind the HyperFlex Storage EPG and VLANs to vPC interfaces going to the UCS Domain, complete the following steps:
1. In the left navigation pane, navigate to HXV-Storage-Data_EPG > Static Ports.
2. Right-click and select Deploy Static EPG on PC, VPC or Interface.
3. In the Deploy Static EPG on PC, VPC or Interface pop-up window, for Path Type, select Virtual Port Channel.
4. For the Path, select the vPC to the first UCS Fabric Interconnect from the drop-down list.
5. For the Port Encap, leave VLAN selected and specify the VLAN ID for the Storage Data EPG.
6. For the Deployment Immediacy, select Immediate.
7. Click Submit.
8. Repeat these steps to bind the EPG to the vPC going to the second UCS Fabric Interconnect.
Configure ACI Fabric for HyperFlex vMotion Traffic
This section details the configuration of the Cisco ACI fabric to enable access to the vMotion network. This is minimally required to support failure scenarios where traffic from one UCS Fabric needs to be forwarded to another when different hosts are forwarding on different fabrics (FI-A, FI-B).
The vMotion network can also be optionally configured with a gateway in the ACI network to enable L3 connectivity to the existing infrastructure. For more information, see the VMware guidelines for L3 vMotion.
Create Bridge Domain for HyperFlex vMotion Traffic
To create a Bridge Domain for HyperFlex vMotion traffic, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. Navigate to Tenants > HXV-Foundation.
3. In the left navigation pane, expand Tenant HXV-Foundation > Networking > Bridge Domains.
4. Right-click and select Create Bridge Domain.
5. Specify a Name (for example, HXV-vMotion_BD).
6. For the VRF, select HXV-Foundation_VRF from the drop-down list.
7. For Forwarding, select Custom and for L2 Unknown Unicast, select Flood. Click Next.
8. In the L3 Configurations section, check the box for EP Move Detection Mode – GARP based detection. Click Next.
9. Skip the Advanced/Troubleshooting section.
10. Click Finish to complete.
Create Application Profile for HyperFlex vMotion Traffic
To create an application profile for HyperFlex vMotion traffic, complete the following steps:
1. In the left navigation pane, select Tenant HXV-Foundation. Right-click and select Create Application Profile.
2. In the Create Application Profile pop-up window, specify a Name (for example, HXV-vMotion_AP) the Application Profile.
3. Click Submit to complete.
Create EPG for HyperFlex vMotion and Associate with Bridge Domain
To create an EPG for HyperFlex vMotion traffic, complete the following steps:
1. In the left navigation pane, navigate to Tenant HXV-Foundation > Application Profiles > HXV-vMotion_AP.
2. Right-click and select Create Application EPG.
3. In the Create Application EPG pop-up window, specify a Name (for example, HXV-vMotion_EPG) for the EPG.
4. For Bridge Domain, select the previously created Bridge Domain.
5. Click Finish.
Associate EPG with UCS Domain
To associate the HyperFlex vMotion EPG with UCS Domain, complete the following steps:
1. In the left navigation pane, expand the newly created EPG.
2. Select Domains. Right-click and select Add L2 External Domain Association.
3. In the Add L2 External Domain Association pop-up window, select the UCS Domain (for example, HXV-UCS_Domain) from the drop-down list.
4. Click Submit.
Create Static EPG and VLAN Binding on vPC Interfaces to UCS Domain
To statically bind the HyperFlex vMotion EPG and VLANs to vPC interfaces going to the UCS Domain, complete the following steps:
1. In the left navigation pane, navigate to HXV-vMotion_EPG > Static Ports.
2. Right-click and select Deploy Static EPG on PC, VPC or Interface.
3. In the Deploy Static EPG on PC, VPC or Interface pop-up window, for Path Type, select Virtual Port Channel.
4. For the Path, select the vPC to the first UCS Fabric Interconnect from the drop-down list.
5. For the Port Encap, leave VLAN selected and specify the VLAN ID for the vMotion EPG.
6. For the Deployment Immediacy, select Immediate.
7. Click Submit.
8. Repeat these steps to bind the EPG to the vPC going to the second UCS Fabric Interconnect.
Configure Gateway for vMotion EPG (Optional)
To configure a default gateway for the vMotion EPG, complete the following steps:
1. In the left navigation pane, navigate to HXV-vMotion_EPG > Subnets.
2. Right-click and select Create EPG Subnet.
3. In the Create EPG Subnet pop-up window, specify the Default Gateway IP and for Scope, unselect Private to VRF and select Advertised Externally and Shared between VRFs. Leave everything else as is.
4. Click Submit.
Create Contract to Access Outside Networks via Shared L3Out (Optional)
To create a contract to access Shared L3Out in the common Tenant, complete the following steps:
1. In the left navigation pane, navigate to HXV-vMotion_EPG > Contracts.
2. Right-click and select Add Consumed Contract.
3. In the Add Consumed Contract pop-up window, select the Allow-Shared-L3Out contract from the drop-down list.
4. Click Submit.
Prerequisites
· HX Installer Deployed in an existing Virtualization environment – separate from the HX cluster being deployed
· Reachability to vCenter Server which will manage the HyperFlex cluster(s) to be installed – must be hosted on an existing virtualization environment, separate from the HX cluster being deployed.
· Reachability to the management interfaces on Fabric Interconnects that the HyperFlex cluster being installed, connects to.
· Reachability to out-of-management (CIMC) interfaces on the servers, reachable via the Fabric Interconnects’ management interfaces. This network (ext-mgmt) should be in the same subnet as the FI management network.
· Reachability to the ESXi management interface of the hosts in the HyperFlex cluster being installed.
· Reachability to the DNS server(s) for use by the HyperFlex cluster being installed.
· Reachability to the NTP server(s) for use by the HyperFlex cluster being installed.
· ACI Fabric Network setup to enable connectivity to HyperFlex and ESXi management, storage data, vMotion and application VM networks.
· For complete details of all ports required for the installation of Cisco HyperFlex, refer to Appendix A of the HyperFlex 3.0 Hardening Guide: https://www.cisco.com/c/dam/en/us/support/docs/hyperconvergedinfrastructure/hyperflex-hx-data-platform/HX-Hardening_Guide_v3-0.pdf
Setup a New Cisco UCS Domain
In this section, you will setup a new Cisco UCS domain in order to connect and bring up a new HyperFlex cluster in this domain. HyperFlex clusters can be deployed in an existing UCS domain but for this CVD, a new UCS domain is deployed as outlined below.
Bring up UCS Domain with Fabric Interconnects
This section provides detailed steps to configure the Cisco Unified Computing System (Cisco UCS) for use in a HyperFlex environment. The steps outlined does an initial setup of a new pair of Cisco UCS Fabric Interconnects which must be completed before connecting and deploying a HyperFlex system to the Cisco UCS Fabric Interconnects.
Cisco UCS Fabric Interconnect A (FI-A)
To start the configuration of the FI-A, connect to the console of the fabric interconnect and step through the Basic System Configuration Dialogue:
---- Basic System Configuration Dialog ----
This setup utility will guide you through the basic configuration of
the system. Only minimal configuration including IP connectivity to
the Fabric interconnect and its clustering mode is performed through these steps.
Type Ctrl-C at any time to abort configuration and reboot system.
To back track or make modifications to already entered values,
complete input till end of section and answer no when prompted
to apply configuration.
Enter the configuration method. (console/gui) ? console
Enter the setup mode; setup newly or restore from backup. (setup/restore) ? setup
You have chosen to setup a new Fabric interconnect. Continue? (y/n): y
Enforce strong password? (y/n) [y]:
Enter the password for "admin":
Confirm the password for "admin":
Is this Fabric interconnect part of a cluster(select 'no' for standalone)? (yes/no) [n]: yes
Enter the switch fabric (A/B) []: A
Enter the system name: HXV1-6300-FI
Physical Switch Mgmt0 IP address : 192.168.167.205
Physical Switch Mgmt0 IPv4 netmask : 255.255.255.0
IPv4 address of the default gateway : 192.168.167.254
Cluster IPv4 address : 192.168.167.204
Configure the DNS Server IP address? (yes/no) [n]: yes
DNS IP address : 10.99.167.244
Configure the default domain name? (yes/no) [n]: yes
Default domain name : hxv.com
Join centralized management environment (UCS Central)? (yes/no) [n]:
Following configurations will be applied:
Switch Fabric=A
System Name=HXV1-6300-FI
Enforced Strong Password=yes
Physical Switch Mgmt0 IP Address=192.168.167.205
Physical Switch Mgmt0 IP Netmask=255.255.255.0
Default Gateway=192.168.167.254
Ipv6 value=0
DNS Server=10.99.167.244
Domain Name=hxv.com
Cluster Enabled=yes
Cluster IP Address=192.168.167.204
NOTE: Cluster IP will be configured only after both Fabric Interconnects are initialized.UCSM will be functional only after peer FI is configured in clustering mode.
Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): yes
Applying configuration. Please wait.
Configuration file - Ok
Cisco UCS 6300 Series Fabric Interconnect
HXV1-6300-FI-A login:
Cisco UCS Fabric Interconnect B (FI-B)
Continue the configuration of Fabric Interconnect B (FI-B) from the console.
Enter the configuration method. (console/gui) ? console
Installer has detected the presence of a peer Fabric interconnect. This Fabric interconnect will be added to the cluster. Continue (y/n) ? y
Enter the admin password of the peer Fabric interconnect:
Connecting to peer Fabric interconnect... done
Retrieving config from peer Fabric interconnect... done
Peer Fabric interconnect Mgmt0 IPv4 Address: 192.168.167.205
Peer Fabric interconnect Mgmt0 IPv4 Netmask: 255.255.255.0
Cluster IPv4 address : 192.168.167.204
Peer FI is IPv4 Cluster enabled. Please Provide Local Fabric Interconnect Mgmt0 IPv4 Address
Physical Switch Mgmt0 IP address : 192.168.167.206
Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): yes
Applying configuration. Please wait.
Wed Jul 11 02:23:14 UTC 2018
Configuration file - Ok
Cisco UCS 6300 Series Fabric Interconnect
HXV1-6300-FI-B login:
Setup Cisco UCS Domain
Log into Cisco UCS Manager
To log in o the Cisco Unified Computing System (UCS) environment, complete the following steps:
1. Open a web browser and navigate to the Cisco UCS fabric interconnect cluster address.
2. Click the Launch UCS Manager link under HTML to launch Cisco UCS Manager.
3. If prompted to accept security certificates, accept as necessary.
4. When prompted, enter admin as the user name and enter the administrative password.
5. Click Login to log in to Cisco UCS Manager.
Upgrade Cisco UCS Manager Software to Version 3.2(3e)
This document is based on Cisco UCS 3.2(3e) release of software for Cisco UCS infrastructure, B-series, and C-Series servers. To upgrade the Cisco UCS Manager software, the Cisco UCS Fabric Interconnect firmware and server bundles to version 3.2(3e), refer to Cisco UCS Manager Firmware Management Guide: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Firmware-Mgmt/3-2/b_UCSM_GUI_Firmware_Management_Guide_3_2.html.
Configure Cisco UCS Call Home and Anonymous Reporting
It is highly recommended by Cisco to configure Call Home in Cisco UCS Manager. Configuring Call Home will accelerate resolution of support cases. To configure Call Home, complete the following steps:
1. In Cisco UCS Manager, click the Admin icon on the left.
2. Select All > Communication Management > Call Home.
3. In the General Tab, change the State to On.
4. Use the other tabs to set Call Home Policies and other preferences, including Anonymous Reporting which selects whether to send anonymous data to Cisco for improving future products.
Configure NTP
To synchronize the Cisco UCS environment to the NTP servers in the Nexus switches, complete the following steps:
1. In Cisco UCS Manager, click the Admin icon on the left.
2. Expand All > Time Zone Management.
3. Select Timezone.
4. In the Properties section, for the Time Zone filed, select the appropriate time zone from the drop-down list.
5. Click Save Changes, and then click OK.
6. In the NTP Servers section, Click [+] Add to add NTP servers.
7. In the Add NTP Server pop-up window, specify the NTP server to use.
8. Click OK and Save Changes to accept.
Configuration of Uplink Ports – To Nexus Leaf Switches in ACI Fabric
The Ethernet ports on Cisco UCS Fabric Interconnects can be configured as network Uplink ports, Server ports, Appliance ports or other port types. By default, all ports are unconfigured.
To configure specific ports as network uplink ports to the upstream network (in this case, ACI Fabric), complete the following steps:
1. In Cisco UCS Manager, click the Equipment icon in the left navigation pane.
2. Navigate to Fabric Interconnects > Fabric Interconnect A > Fixed Module or Expansion Module as appropriate > Ethernet Ports.
3. In the right window pane, select the uplink port and right-click to select Configure as Uplink Port.
4. Click Yes and OK to confirm.
5. Navigate to Fabric Interconnects > Fabric Interconnect B > Fixed Module or Expansion Module as appropriate > Ethernet Ports.
6. In the right window pane, select the uplink port and right-click to select Configure as Uplink Port.
7. Click Yes and OK to confirm.
8. Repeat for remaining uplink ports and verify that they now show up as Network ports.
Bundle Uplink Ports into a Port-Channel on each FI
The uplink ports on each FI are bundled into a port-channel. The ports are connected to different Nexus Leaf switches, configured to be part of a vPC domain, with a vPC to each FI using the two ports going to that FI. On each FI, complete the following steps to configure the uplink networks ports into a port-channel:
1. In Cisco UCS Manager, click the LAN icon in the left navigation pane.
2. Navigate to LAN > LAN Cloud > Fabric A.
3. Right-click and select Create Port Channel from the list.
4. In the Create Port Channel wizard, in the Set Port Channel Name section, specify a unique Port-Channel ID for this UCS Domain. Click Next.
5. In the Add Ports section, select uplink ports from the first table and use the >> to add them to the port-channel (2nd table). Click Finish and OK to complete.
6. Navigate to Fabric Interconnects > Fabric Interconnect B > Fixed Module or Expansion Module as appropriate > Ethernet Ports.
7. Verify the port channel is up and running with active ports.
8. Repeat the above steps under Fabric B to create a port-channel from Fabric B uplink ports to Nexus Leaf switches.
9. Verify the port channel is up and running with active ports.
Modify Chassis Discovery Policy
If the Cisco HyperFlex system uses blades as compute-only nodes in an extended cluster design, chassis discovery policy must be configured for connecting a Cisco UCS 5108 blade chassis. The Chassis Discovery policy defines the number of links between the Fabric Interconnect and the Cisco UCS Fabric Extenders which must be connected and active, before the chassis will be discovered. This also effectively defines how many of those connected links will be used for communication. The Link Grouping Preference setting specifies if the links will operate independently, or if Cisco UCS Manager will automatically combine them into port-channels. Cisco best practices recommends using link grouping, and the number of links per side is dependent on the hardware used in Cisco UCS 5108 chassis, and the model of Fabric Interconnects. For 10 GbE connections Cisco recommends 4 links per side, and for 40 GbE connections Cisco recommends 2 links per side.
To modify the chassis discovery policy, complete the following steps:
1. In Cisco UCS Manager, click the Equipment icon on the left and select Equipment from the left-navigation pane.
2. In the right pane, click-on the Policies tab.
3. Under Global Policies, set the Chassis/FEX Discovery Policy (for Action) to match the minimum number of uplink ports that are cabled between the chassis or fabric extenders (FEXes) and the fabric interconnects.
4. Set the Link Grouping Preference to Port Channel.
5. Click Save Changes and OK to complete.
Configuration of Server Ports – To HyperFlex Hosts
The Ethernet ports of a Cisco UCS Fabric Interconnect connected to the rack-mount servers, or to the blade chassis must be defined as server ports. When a server port is activated, the connected server or chassis will begin the discovery process shortly afterwards. Rack-mount servers and blade chassis are automatically numbered in Cisco UCS Manager in the order which they are first discovered. For this reason, it is important to configure the server ports sequentially in the order you wish the physical servers and/or chassis to appear within Cisco UCS Manager.
Auto-Discovery of Server Ports (Optional)
To enable server ports to be discovered automatically when rack and blade servers are connected to the Cisco UCS Fabric Interconnects, complete the following steps:
1. In Cisco UCS Manager, click the Equipment icon on left-navigation pane.
2. In the right window pane, click the tab for Policies > Port Auto-Discovery Policy.
3. Under Properties, set the Auto Configure Server Port to Enabled.
4. Click Save Changes and OK to complete.
Manual Configuration of Server Ports
To manually define the server ports and have control over the numbering of the servers, complete the following steps:
1. In Cisco UCS Manager, click the Equipment icon on left-navigation pane.
2. Navigate to Equipment > Fabric Interconnects > Fabric Interconnect A > Fixed Module > Ethernet Ports.
3. In the right-window pane, select the first port. Right-click and select Configure as Server Port.
4. Click Yes and OK to confirm.
5. Navigate to Equipment > Fabric Interconnects > Fabric Interconnect B > Fixed Module > Ethernet Ports.
6. In the right-window pane, select the matching port from Fabric Interconnect A. Right-click and select Configure as Server Port.
7. Click Yes and OK to confirm.
8. Repeat the steps for each port that is to be a server port.
9. In the right-window pane, verify that all ports connected to chassis, Cisco FEX and rack servers are configured as Server Ports.
Verify Server Status before HyperFlex Install
When server ports are configured and active, the servers connected to these ports will begin a discovery process. During discovery, hardware inventories are collected, along with their current firmware revisions. Before starting the HyperFlex installation process that will create the service profiles and associate them with the servers, verify that that the servers have finished their discovery process and are listed as unassociated in the Overall Status column. The servers should also be powered off and have no errors.
To verify, complete the following steps:
1. In Cisco UCS Manager, click the Equipment icon in the left navigation pane.
2. In the properties section of the right-window pane, click the Servers tab.
The servers should be in a state similar to the figure above with no faults or errors.
The servers are now ready for installing the HyperFlex Data Platform Software.
Install Cisco HyperFlex
This section describes the installation of a new 4-node HyperFlex system using the Cisco HyperFlex Installer Virtual Machine. The HyperFlex cluster can also be deployed from the cloud using Cisco Intersight.
HX Installer Overview
The Cisco HyperFlex Installer is available as an Open Virtual Appliance (OVA) format file that can be deployed as a virtual machine in an existing VMware vSphere environment. It can also be deployed from other virtualization environments that support OVA format – VMware Workstation, VMware Fusion, etc. The HyperFlex OVA file is available for download from Software Downloads page on cisco.com.
In this CVD, the HyperFlex Installer was deployed on an existing ESXi cluster, managed by VMware vCenter.
Deploy HX Installer OVA in an Existing Virtualization Environment
To deploy the HyperFlex installer in an existing VMware vCenter environment, complete the following steps:
1. Open a web browser to navigate to the IP address of the VMware vCenter server.
2. Click the vSphere web client of your choice and log in as Administrator.
3. From the vSphere Web Client, navigate to Home > Hosts and Clusters.
4. From the left navigation pane, select the Datacenter > Cluster and right-click to select Deploy OVF Template….
5. In the Deploy OVF Template wizard, for Select Template, select Local file and click the Browse button to locate and open the Cisco-HX-Data-Platform-Installer OVA file. Click Next.
6. For Select name and location, specify a name for the virtual machine and select a folder location. Click Next.
7. For Select a resource, select a host or cluster or resource pool to locate the virtual machine. Click Next.
8. Review the details. Click Next.
9. For Select storage, select a datastore and Thin provision virtual disk format for the VM. Click Next.
10. For Select networks, use the drop-down list in the Destination Networks column to specify the network the installer VM will communicate on. Click Next.
11. For Customize template, provide the IP Address, Mask, Gateway, DNS and NTP server info. Click Next.
12. Review the settings. Click Finish.
13. Power on the VM.
14. The Installer VM is now ready to be used for installing a HyperFlex system.
Access HyperFlex Installer
To access the HyperFlex installer webpage, complete the following steps:
1. Open a web browser to navigate to the IP address of the installer VM. If the HyperFlex installer was deployed using DHCP, open the local console of the installer VM to see the leased IP address as shown below.
2. Click accept or continue to bypass any SSL certificate errors.
3. At the login screen, enter the username: root
4. At the login screen, enter the default password: Cisco123
5. Verify the version of the installer in the lower right-hand corner of the Welcome page is the correct version.
6. Check the box for accepting terms and conditions and click Login.
7. You should now be forwarded to the HyperFlex Installer Workflow page where you can install a new Standard Cluster, Stretch Cluster, Edge Cluster or expand an existing cluster. This design is for a Standard Cluster.
Install HyperFlex System
In this section, the HyperFlex installer will be used to set up a HyperFlex cluster. It will configure Cisco UCS policies, templates, service profiles, and settings, as well as assigning IP addresses to the HX servers that come from the factory with ESXi hypervisor software preinstalled. The installer will deploy the HyperFlex controller VMs and software on the nodes, add the nodes to VMware vCenter managing the HX Cluster, and finally create the HyperFlex cluster and distributed filesystem. The above setup is done via a single workflow from the HyperFlex Installer webpage by providing the necessary information through the Installation wizard.
To install a HyperFlex system in a Cisco UCS Domain, complete the following steps:
1. From the main HyperFlex Installer Workflow page, for the Select a Workflow, click Create Cluster and from the drop-down list, select Standard Cluster.
2. In the Credentials section screen, provide the Hostname and Credentials for UCS Manager, vCenter, and ESXi.
3. Click Continue.
4. In the Server Selection screen, select all the Servers that should be part of the HyperFlex cluster.
5. Click Continue.
6. In the UCSM Configuration screen, specify the UCSM related configuration as shown below.
7. Click Continue.
8. In the Hypervisor Configuration screen, specify the ESXi Management IP Addresses and Gateway information for the ESXi hosts in the HyperFlex cluster.
9. Click Continue.
10. In the IP Addresses screen, specify the HyperFlex addressing for the nodes in the HyperFlex cluster.
11. Click Continue.
12. In the Cluster Configuration screen, specify the cluster related configuration as shown below.
13. In the same screen as above, expand Advanced Configuration and select Enable Jumbo Frames.
14. Click Continue and Start.
15. Validation of the configuration will now start. If there are warnings, you can review them and choose to ‘Skip Validation’ if it is acceptable to proceed with the warnings. If there are no warnings, the installer will automatically move on to the configuration stage.
16. When Validation completes, you may see something similar to the screenshot below.
The initial validation will always fail when using a new Cisco UCS 6332 or 6332-16UP model Fabric Interconnects. This is due to the fact that changes to the QoS system classes require these models to reboot. If the validation is skipped, the HyperFlex installer will continue the installation and automatically reboot both Fabric Interconnects sequentially. If this is an initial setup of these Fabric Interconnects, and no other systems are running on them yet, then it is safe to proceed. However, if these Fabric Interconnects are already in use for other workloads, then caution must be taken to ensure that the sequential reboots of both Fabric Interconnects will not interrupt those workloads, and that the QoS changes will not cause traffic drops. Contact Cisco TAC for assistance if this situation applies.
17. When the pre-validation stage completes, the Installer will then proceed to complete the HyperFlex install and configuration by stepping through the stages listed at the top of the page, along with the status.
18. You can also monitor the process from the components being configured such as the Cisco UCS Manager and VMware vCenter.
19. After the installation completes, if everything goes well, you should see the Cluster Creation Successful message.
20. Click View Summary to view a summary of the setup. Note that the HX Cluster is ONLINE and HEALTHY.
21. Export the cluster configuration by clicking the downward arrow icon in the top right of the screen. Click OK to save the configuration to a JSON file. This file can be imported to save time if you need to rebuild the same cluster in the future, and be kept as a record of the configuration options and settings used during the installation.
22. Continue with the Post Installation tasks in the next section. It is particularly important to run the post_install script in order to create the vMotion interfaces, the guest VM port groups, and to enable HA and DRS in the cluster.
Post Installation
After the HyperFlex Installer has completed the installation of the HyperFlex cluster, additional steps are needed to finalize the configuration of the system before placing virtual machine workloads on the cluster. The following scripts and procedures can be used to complete the installation.
Post Installation Script
To automate many of the post installation procedures on the hosts and to verify that the HyperFlex Installer has properly configured Cisco UCS Manager, a script has been provided on the HyperFlex Installer OVA. These steps can also be performed manually in vCenter if preferred.
To run this script, complete the following steps:
1. SSH to the installer OVA IP as root with a password Cisco123.
2. From the CLI of the installer VM, run the script named post_install. Enter the passwords for HX Controller VM, vCenter when prompted.
3. The installer will already have the information from the just completed HX installation and it will be used by the script. Enter the HX Storage Controller VM root password for the HX cluster (use the one entered during the HX Cluster installation), as well as the vCenter user name and password. You can also enter the vSphere license or complete this task later.
4. The health check will show for all hosts in the cluster though the figure above only shows for the first 1st host and will with a summary of the results as shown below.
Syslog
To store diagnostic logs in a central location in case they are needed after a host failure, it is recommended to enable a syslog destination for permanent storage of the ESXi host logs for all Cisco HyperFlex hosts. It is possible to use the vCenter server as the log destination in this case, or another syslog receiver of your choice.
To configure syslog, complete the following steps:
1. Log into the ESXi host via SSH as the root user.
2. Enter the following commands, replacing the IP address in the first command with the IP address of the vCenter server that will receive the syslog logs:
esxcli system syslog config set loghost='udp://10.1.167.120'
esxcli system syslog reload
esxcli network firewall ruleset set -r syslog -e true
esxcli network firewall refresh 3.
3. Repeat for each ESXi host.
Auto-Support and Notifications
Auto-Support should be enabled for all clusters during the initial HyperFlex installation. Auto-Support enables Call Home to automatically send support information to Cisco TAC, and notifications of tickets to the email address specified. If the settings need to be modified, they can be changed in the HyperFlex Connect HTML management webpage.
To change Auto-Support settings, complete the following steps:
1. Use a browser to navigate to HyperFlex Connect.
2. Click the gear shaped icon in the upper right-hand corner, and click Auto-Support Settings.
3. Enable or Disable Auto-Support as needed. Enter the email address to receive alerts for Auto-Support events.
4. Enable or Disable Remote Support as needed. Remote support allows Cisco TAC to connect to the HX cluster and accelerate troubleshooting efforts.
5. Enter in the information for a web proxy if needed. Click OK.
6. To enable Email Notifications, click the gear shaped icon in top right corner, and click Notifications Settings. Enter the outgoing Mail Server Address information, the From Address and the Recipient List. Click OK.
Smart Licensing
HyperFlex 2.5 and later utilizes Cisco Smart Licensing, which communicates with a Cisco Smart Account to validate and check out HyperFlex licenses to the nodes, from the pool of available licenses in the account. At the beginning, Smart Licensing is enabled but the HX storage cluster is unregistered and in a 90-day evaluation period or EVAL MODE. For the HX storage cluster to start reporting license consumption, it must be registered with the Cisco Smart Software Manager (SSM) through a valid Cisco Smart Account. Before beginning, verify that you have a Cisco Smart account, and valid HyperFlex licenses are available to be checked out by your HX cluster.
To create a Smart Account, see Cisco Software Central > Request a Smart Account https://webapps.cisco.com/software/company/smartaccounts/home?route=module/accountcreation.
To activate and configure smart licensing, complete the following steps:
1. Log into a HX Controller VM. Confirm that your HX storage cluster is in Smart Licensing mode.
# stcli license show status
2. Feedback will show Smart Licensing is ENABLED, Status: UNREGISTERED, and the amount of time left during the 90-day evaluation period (in days, hours, minutes, and seconds).
3. Navigate to Cisco Software Central (https://software.cisco.com/) and log in to your Smart Account.
4. From Cisco Smart Software Manager, generate a registration token.
5. In the License pane, click Smart Software Licensing to open Cisco Smart Software Manager.
6. Click Inventory.
7. From the virtual account where you want to register your HX storage cluster, click General, and then click New Token.
8. In the Create Registration Token dialog box, add a short Description for the token, enter the number of days you want the token to be active and available to use on other products, and check Allow export controlled functionality on the products registered with this token.
9. Click Create Token.
10. From the New ID Token row, click the Actions drop-down list, and click Copy.
11. Log into the controller VM.
12. Register your HX storage cluster, where idtoken-string is the New ID Token from Cisco Smart Software Manager.
# stcli license register --idtoken idtoken-string 12.
13. Confirm that your HX storage cluster is registered.
# stcli license show summary
14. The cluster is now ready. You may run any other preproduction tests that you wish to run at this point.
Deploy VMware vCenter Plugin to Manage HyperFlex (Optional)
The Cisco HyperFlex vCenter Web Client Plugin can be deployed as a secondary tool to monitor and configure the HyperFlex cluster.
This plugin is not supported in the HTML5 based VMware vSphere Client for vCenter.
To manage the HyperFlex cluster using the vCenter Web Client Plugin for vCenter 6.5, complete the following steps:
1. Open the vCenter Web Client, and login with admin rights.
2. Navigate to the Home screen and click Global Inventory Lists.
3. In the Navigator pane, click Cisco HX Data Platform.
4. In the Navigator pane, choose the HyperFlex cluster you want to manage and click the name.
Summary
From the Web Client Plugin Summary screen, several elements are accessible:
· Overall cluster usable capacity, used capacity, free capacity, datastore capacity provisioned, and the amount of datastore capacity provisioned beyond the actual cluster capacity.
· Deduplication and compression savings percentages calculated against the data stored in the cluster.
· The cluster operational status, the health state, and the number of node failures that can occur before the cluster goes into read-only or offline mode.
· A snapshot of performance over the previous hour, showing IOPS, throughput, and latencies.
Create Datastores for Virtual Machines
This task can be completed by using the vSphere Web Client HX plugin, or by using the HyperFlex Connect HTML management webpage. To configure a new datastore via the HyperFlex Connect webpage, complete the following steps:
1. Use a web browser to open the HX cluster IP Management URL to open HyperFlex Connect.
2. Enter Login credentials, either a local credential, or a vCenter RBAC credential with administrative rights.
3. Click Login.
4. In the left navigation pane, go to Manage > Datastores. Click-on Create Datastore.
5. In the popup, enter the Datastore Name and size. For most applications, leave the Block Size at the default of 8K. Only dedicated Virtual Desktop Infrastructure (VDI) environments should choose the 4K Block Size option.
6. Click Create Datastore.
Managing HyperFlex
HyperFlex Connect
HyperFlex Connect is an easy to use, powerful primary management tool for managing HyperFlex clusters. HyperFlex Connect is a HTML5 web-based GUI tool that is accessible via the cluster management IP address. It runs on all HX nodes in the cluster for high availability. HyperFlex Connect can be accessed using either pre-defined Local accounts or Role-Based access (RBAC) by integrating authentication with VMware vCenter managing the HyperFlex cluster. With RBAC, you can use VMware credentials either local (for example, administrator@vsphere.local) or Single Sign-On (SSO) credential such as an Active Directory(AD) users defined on vCenter through AD integration.
To manage HyperFlex cluster using HyperFlex Connect, complete the following steps:
1. Open a web browser and navigate to the IP address of the HX cluster (for example, https://10.1.167.210)
2. For initial access, use the admin account for the HX cluster, defined during the HX install process.
3. The Dashboard provides general information about the cluster’s operational status, health, Node failure tolerance, Storage Performance and Capacity Details and Cluster Size and individual Node health.
4. The left navigation menu provides different options to Manage, Monitor, Analyze and Protect the HyperFlex system.
Cisco Intersight Cloud-Based Management
Cisco Intersight provides a dashboard and a single view of all Cisco UCS Domains, and servers within those domains, including Cisco HyperFlex Clusters running in the domains. The dashboard elements can be drilled down to get an overview of their health statuses, storage utilization, port counts, and more. For HyperFlex, Cisco Intersight can be used to do the initial install of a cluster as well. New features and capabilities are continually being added over time. Please visit the Cisco Intersight web page or release notes for the latest information.
Access Cisco Intersight
Cisco Intersight management is enabled via embedded code running on the Cisco UCS Fabric Interconnects, and in the Cisco HyperFlex software, known as Device Connectors. To enable Cisco Intersight management, the device connectors are first registered to Cisco Intersight located in the Cloud and all subsequent management can be done from the Cisco Intersight website.
The Cisco UCS Fabric Interconnects, and the Cisco HyperFlex nodes must do a DNS lookup to access Cisco Intersight on the internet. Internet access can be direct or via an HTTPS proxy server.
To access Cisco Intersight, complete the following steps:
1. Use a web browser to access Cisco Intersight at https://intersight.com/.
2. Login with a valid cisco.com account.
Enable Cisco Intersight Management on Cisco UCS Manager
To enable Cisco Intersight to manage the Cisco UCS Domain(s) in your environments, complete the following steps:
1. In the left navigation menu, select Devices.
2. Click the Claim a New Device button in the top right-hand corner.
3. Open a web browser and navigate to IP address of Cisco UCS Manager and login.
4. In the left navigation menu, select Admin > Device Connector. Notice that the Status is currently Not Claimed.
5. (Optional) For Access Mode, you can set it to Read-only or Allow Control for Cisco Intersight to remotely manage the UCS domain. Alternatively, you can completely Enable/Disable all access.
6. (Optional) If you’re using a HTTP Proxy in your environment to access the Internet and Cisco Intersight, you can provide the necessary information for your environment by clicking the HTTPS Proxy Settings button. In the HTTPS Proxy Settings pop-up window, click the Manual button and enter the information and click Save.
7. In the Connection section, copy the Device ID and a Claim Code for your Cisco UCS Domain and enter it in the Cisco Intersight – Claim a New Device window. Click Claim.
8. You should now see the UCS Manager in the list of Claimed Devices.
Enable Cisco Intersight Management on Cisco HyperFlex Connect
To enable Cisco Intersight to manage the Cisco HyperFlex cluster, complete the following steps:
1. Open a web browser and navigate to HyperFlex Connect and login.
2. From a second browser window or tab, log on to the Cisco Intersight webpage at https://intersight.com if you are not already and navigate to Devices > Claim A New Device.
3. In the HyperFlex Connect Dashboard page, click Edit Settings in the top right-hand corner, then click Device Connector.
4. Notice that the Status is currently Not Claimed in the Device Connector pop-up window.
5. (Optional) For Access Mode, you can set it to Read-only or Allow Control for Cisco Intersight to remotely manage the UCS domain. Alternatively, you can completely Enable/Disable all access.
6. (Optional) If you’re using a HTTP Proxy in your environment to access the Internet and Cisco Intersight, you can provide the necessary information for your environment by clicking on the HTTPS Proxy Settings button. In the HTTPS Proxy Settings pop-up window, click the Manual button and enter the information and click Save.
7. From the Connection section, copy the Device ID and a Claim Code for your Cisco UCS Domain and enter it in the Cisco Intersight – Claim a New Device window. Click Claim.
8. You should now see the UCS Manager and Cisco HyperFlex Cluster in the list of Claimed Devices.
Cisco Intersight Licensing
Cisco Intersight is offered in two editions:
· Base license which is free to use, and offers a large variety of monitoring, inventory and reporting features.
· Essentials license, at an added cost but provides advanced monitoring, server policy and profile configuration, firmware management, virtual KVM features, and more. A 90-day trial of the Essentials license is available for use as an evaluation period.
New features and capabilities will be added to the different licensing tiers over time.
vCenter Web Client Plugin
The Cisco HyperFlex vCenter Web Client Plugin is installed by the HyperFlex installer to the specified vCenter server or vCenter appliance. The plugin is accessed as part of the vCenter Web Client (Flash) interface, and is a secondary tool used to monitor and configure the HyperFlex cluster. This plugin is not integrated into the new vCenter 6.5 HTML5 vSphere Client. In order to manage a HyperFlex cluster via an HTML5 interface, (without the Adobe Flash requirement), use the new HyperFlex Connect management tool.
To manage the HyperFlex cluster using the vCenter Web Client Plugin for vCenter 6.5, complete the following steps:
1. Open the vCenter Web Client, and login with administrative rights.
2. In the home pane, from the Home screen click Global Inventory Lists.
3. In the left navigation pane, click Cisco HX Data Platform.
4. In the left navigation pane, select the HyperFlex cluster to manage (for example, HXV1-StorCluster1).
5. You can switch to the Summary, Monitor or Manage tabs in the right-window pane to monitor and manage the cluster status, storage performance and capacity status, create datastores, upgrade cluster and more.
The virtual networking used in this design uses a dedicated VMware vSphere vSwitch for In-Band Management, vMotion and Storage Data. This is setup by the HyperFlex Installer. The HyperFlex Installer also sets up a dedicated vSwitch for Application VM traffic however, the application traffic will be migrated to either a VMware vSphere vDS or a Cisco AVE in this design. Examples will be provided to first show the application traffic using VMware vSphere vDS, and then the Cisco AVE option will be shown. The VMware vDS and Cisco AVE will be both APIC Controlled, enabled by the VMM Integration between Cisco APIC and VMware vCenter.
This section describes the steps involved in deploying a APIC-controlled VMware vSphere vDS and a Cisco AVE for Application traffic.
Deploying APIC-controlled VMware vSphere vDS for HyperFlex Application Traffic
To deploy a VMware vSphere vDS for Application VMs deployed on the HyperFlex cluster, complete the following steps: The virtual switch configuration will be done from the Cisco APIC GUI.
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Virtual Networking.
3. From the left navigation pane, select Quick Start. From the right-window pane, click Create a vCenter Domain Profile.
4. In the Create vCenter Domain window, specify a Virtual Switch Name (for example, HXV-vDS-vm-network). This is the name of the VMware vSphere vDS switch as it should appear in VMware vCenter.
5. Leave VMware vSphere Distributed Switch selected for the Virtual Switch.
6. For the Associated Attachable Entity Profile, select the AAEP for UCS (for example, HXV-UCS_AAEP).
7. For VLAN Pool, select Create VLAN Pool from the drop-down list.
8. In the Create VLAN Pool pop-up window, specify a name for the pool to be associated with vDS.
9. Leave the Allocation Mode set to Dynamic Allocation.
10. Click the [+] icon at the right side of the Encap Blocks section.
11. In the Create Ranges pop-up window, specify a VLAN range for the pool.
12. Set the Allocation Mode set to Dynamic Allocation.
13. Leave the Role as External or On the wire encapsulations.
14. Click OK to close the Create Ranges pop-up window.
15. Click Submit to close the Create VLAN Pool pop-up window.
16. In the Create vCenter Domain window, click the [+] icon to the right of the vCenter Credentials section.
17. Specify a Name for the credentials, along with the appropriate account Username and Password.
The Administrator account is used in this example, but an APIC account can be created within the vCenter to enable the minimum set of privileges. For more information, see the ACI Virtualization Guide on cisco.com.
18. Click OK.
19. Click the [+] icon at the right side of the vCenter section.
20. In the Add vCenter Controller pop-up window, enter a Name for the vCenter.
21. Enter the vCenter IP Address or Host Name.
22. For DVS Version, leave it as vCenter Default.
23. Set Stats Collection to Enabled.
24. For Datacenter, enter the exact vCenter Datacenter name.
25. Do not select a Management EPG.
26. For vCenter Credential Name, select the vCenter credentials created in the last step (Administrator).
27. Click OK.
28. In the Create vCenter Domain Window, select the MAC Pinning-Physical-NIC-load as the Port Channel Mode.
29. Select CDP for vSwitch Policy.
30. Leave the NetFlow Exporter Policy unselected.
31. Click Submit to create the vDS within the vCenter.
32. Log into the vCenter vSphere Web Client and navigate to Networking.
33. Confirm that the distributed switch was created.
Add HyperFlex ESXi Hosts to VMware vSphere vDS
To add the HyperFlex ESXi Hosts to the newly created vDS, complete the following steps:
1. Use a browser to log into the VMware vSphere Web Client.
2. From the Home screen, select Networking in the Inventories section.
3. In the left navigation pane, expand the Datacenter with the newly deployed vDS. Open the vDS folder and select the vDS that was created by the APIC.
4. Right-click the APIC-controlled vDS switch and select Add and manage hosts.
5. In the Add and Manage Hosts pop-up window, select the Add hosts option. Click Next.
6. In the Select Hosts window, click [+ New host…] to add new host.
7. In the Select new hosts pop-up window, select all of the relevant ESXi hosts. Click OK to complete.
8. Click Next.
9. Leave Manage physical adapters selected and de-select the other options. Click Next.
10. For the first host, select vmnic4 from the Host/Physical Network Adapters column.
11. Click the Assign uplink from the menu above.
12. In the Select an Uplink for vmnic4 pop-up window, leave uplink 1 selected and click OK.
13. Repeat this process for vmnic5 – assign it to uplink2. Click Next.
14. Repeat these assignment for all ESXi hosts being added to vDS. If a server shows no physical adapter available to add to the vDS, complete this wizard without adding, then select the host from left pane and navigate to Configure > virtual Switches (under Networking) and select the vswitch for vm-network (vswitch-hx-vm-network) and remove the physical adapters. Once released from the vswitch, the physical adapters for that host can be added to the vDS.
15. Click Next.
16. In the Analyze impact window, click Next.
17. Review the settings and click Finish to apply.
Deploying Cisco ACI Plug-in for Managing ACI from VMware vCenter
Cisco ACI vCenter Plug-in provides a user interface to manage the ACI Fabric from VMware vCenter web management interface. The plug-in exposes a subset of APIC capabilities to VMware vCenter that are relevant to Virtualization Administrators. The plug-in can also be used to install Cisco’s distributed virtual switches (Cisco AVS and Cisco AVE).
Prerequisites for Cisco ACI Plug-in
The prerequisites for installing the Cisco ACI Plugin on VMware vCenter are as follows:
· At least one VMM domain should already exists between the APIC and the vCenter where the plug-in is being installed.
· HTTPS traffic must be allowed between VMware vCenter server and Cisco APIC – vCenter will directly download the plug-in using HTTPS.
· VMware PowerCLI installed on a Windows machine – the PowerShell scripts for installing the plug-in will be executed from the PowerCLI console.
· VMware vCenter 6.0U3 or higher is recommended.
Installing Cisco ACI Plug-in
To install the Cisco AC Plug-in for VMware vCenter, complete the following steps:
1. Using a web browser, navigate to the APIC at https://<APIC-IP>/vcplugin.
2. Download the ACIPlugin-Install.ps1 script.
3. Copy the script to the system where it will be executed if it was not the download host. The script will be executed from the PowerCLI console.
4. Go to the folder with the installation script.
5. Select the script. Right-click and select Properties.
6. In the pop-up window, click Unblock button. Click Apply. Click OK to close the pop-up.
7. Open the PowerCLI console (Run as Administrator) and execute the ACIPlugin-Install.ps1 script.
8. Enter the address to the vCenter, the http source for the plugin bundle and the HTTPS SHA1 Thumbprint: https://<APIC-IP-Address-or-Hostname>/vcplugin/vcenter-plugin-3.2.2000.12.zip. To determine the HTTP SHA1 Thumbprint, review the Installation instructions in the same location as the .zip file; it varies depending on the browser you are using.
9. In the pop-window, provide the vCenter Administrator credentials.
10. If the installation was successful, you should see something similar to the following:
Verify the ACI Plug-in Installation from VMware vCenter
To verify from vCenter that the installation was successful, complete the following steps:
1. Login to VMware vSphere web client (VMware vCenter) and verify you see the icon as follows. If you are already logged in, you may need to disconnect and re-login back to see the icon.
2. If you do not see the icon, navigate to the following link to check the status of the plug-in registration. If it shows a successful registration but you still do not see the plug-in, a reboot of the vCenter may be necessary.
3. For additional information on VMware vCenter plug-in, see the ACI Virtualization Guide for 3.2(2) ACI release used in this design.
Connecting ACI Plug-in to ACI Fabric
To connect the ACI plug-in to the ACI Fabric, complete the following steps:
1. Use a web browser to navigate to VMware vSphere web client (VMware vCenter) and login.
2. From the Home menu, click Cisco ACI Fabric icon.
3. From the Getting Started tab, select Connect vSphere to your ACI Fabric.
4. In the Register a new ACI Fabric pop-up window, click Yes to register a new ACI fabric.
5. In the Register a new APIC Node pop-up window, specify the IP or hostname of an APIC node.
6. Deselect Use Certificate.
7. For the APIC Credentials, specify the Username and Password.
8. Click OK to complete the configuration and close the window.
9. If the registration was successful, you should see something similar to the following.
10. Click OK to close the pop-up window.
11. In the ACI Fabric tab, you should now see the APICs managing the ACI fabric.
Deploying APIC-controlled Cisco AVE for HyperFlex Application Traffic
To deploy Cisco AVE for Application workloads hosted on the HyperFlex cluster, the following three methods can be used: Cisco ACI vCenter Plug-in, VMware PowerCLI, Python Script. In this section, the detailed procedures for deploying Cisco AVE using the Cisco ACI vCenter Plug-in are explained.
High-level Deployment Steps
· The high-level steps for deploying Cisco AVE in a HyperFlex environment are:
· Verify IP connectivity between Cisco APIC and VMware vCenter.
· Enable Infrastructure VLAN (4093) for VxLAN tunneling between ACI Leaf switch and Cisco AVE.
· Create and configure Infrastructure VLAN on Cisco UCS Manager and on links to HyperFlex servers. The MTU should be 1600 or higher for VxLAN.
· Allocate VLAN pool for Application EPGs and port-groups to use – the pool should accommodate the number of EPGs published to the VMware vCenter domain.
· Create a new vCenter VMM domain for Cisco AVE – allocate multicast address pool to accommodate the number of EPGs that will be published to the VMware vCenter domain.
· Deploy Cisco ACI vCenter plug-in - VMware vCenter 6.0U3 or higher is recommended.
· Download Cisco AVE OVF file to VMware vCenter
· Allocate a Management IP Address for each Cisco AVE Virtual Machine – one VM on each ESXi host. The addresses should be in a contiguous block if the DHCP server is VMware vCenter.
· Configure Virtual Networking (if needed) to add Cisco AVE VMs to Management network (port-group).
· Setup DHCP to allocate IP address to Cisco AVE VM – either an existing DHCP network or use VMware vCenter as DHCP server. VMware vCenter is used in this setup.
· Deploy Cisco AVE Virtual Machine on ESXi hosts.
Enable ACI Infrastructure VLAN for Cisco AVE in the ACI Fabric
To enable the ACI infrastructure VLAN for Cisco AVE in the ACI fabric, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Fabric > Access Policies.
3. From the left navigation pane, expand Policies > Global > Attachable Access Entity Profile.
4. Select AEP for UCS (HXV-UCS_AAEP) and select Enable Infrastructure VLAN.
5. Click Submit.
Enable ACI Infrastructure VLAN in the Cisco UCS Domain
An infrastructure VLAN is required for Cisco AVS deployment in VXLAN mode. To create the infrastructure VLAN and add the VLAN to virtual machine networking virtual network interface card (vNIC) templates, complete the following steps:
1. Use a browser to navigate to Cisco UCS Manager Web GUI and login within an administrator account.
2. In Cisco UCS Manager, click the LAN tab from the left navigation menu.
3. Expand LAN > LAN Cloud.
4. Right-click VLANs and choose Create VLANs.
5. In the Create VLAN pop-up window, specify a VLAN Name (for example, Infra-VLAN).
6. Keep the Common/Global option for VLAN scope.
7. Specify the VLAN ID (4093) for Infrastructure VLAN for VXLAN.
8. Keep the Sharing Type set to None.
9. Click OK twice.
Enable Infrastructure VLAN on HX Server vNIC Templates
To enable the Infrastructure VLAN on the HX Server vNIC templates, complete the following steps:
1. In Cisco UCS Manager, click the LAN icon from the left navigation menu.
2. Expand LAN > Policies > root > Sub-Organizations.
3. Expand the HyperFlex sub-organization (for example, HXV1-Org1) > vNIC Templates.
4. Select the vNIC template for HX virtual machine network for UCS Fabric A side (for example, vm-network-a) from the left navigation pane.
5. On the right-window, select General > Modify VLANs.
6. In the Modify VLANs pop-up window, select the new VLAN (for example, Infra-VLAN).
7. Click OK.
8. In the General tab, scroll down to the MTU field and change the MTU. Increase the MTU to something higher than 1600B for VXLAN traffic – in this design, the default of 1500B is increased to an MTU of 9000 to stay consistent with MTU on other vNICs.
9. Click Save Changes.
10. In the Save Changes pop-up window, select Yes to apply changes and acknowledge the reboot of the servers.
Reboot will NOT be from UCS. Instead, it will be done from vCenter, using HX plug-in and will be done one host at a time so as to ensure the cluster is healthy before proceeding to the next host.
11. Repeat for the second vNIC template to UCS Fabric B.
12. Use a web browser to navigate to vSphere web client and log in.
13. Navigate to Home > Hosts and Clusters and select the first host in the HX cluster to put in HX maintenance mode and reboot to apply the service profile changes done in UCS.
14. Right-click the first host and select Cisco HX Maintenance Mode > Enter HX Maintenance Mode.
15. Click OK to accept changes.
16. When the host is in maintenance mode, right-click the host again and select Power > Reboot option.
17. When the host reboots and comes back up, right-click the host and select Cisco HX Maintenance Mode > Exit HX Maintenance Mode. Reboot takes a few minutes and if you can ping the server but it doesn’t show up in vCenter, you may need to reconnect by selecting Connect > Reconnect.
18. When the cluster comes up and is healthy, repeat above steps for each host in the HX cluster.
Create a new VMM Domain for Cisco AVE
To create a new VMM domain for Cisco AVE, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Virtual Networking.
3. From the left navigation pane, select Quick Start. From the right-window pane, click Create a vCenter Domain Profile.
4. In the Create vCenter Domain window, specify a Virtual Switch Name (for example, HXV-AVE-vm-network). This is the name of the Cisco AVE switch as it should appear in VMware vCenter.
5. For the Virtual Switch, select Cisco AVE.
6. For the Switching Preference, select Local Switching.
7. For the Default Encap Mode, select VXLAN Mode.
8. For the Attachable Entity Profile, select the previously created UCS AAEP (for example, HXV-UCS_AAEP).
9. For the VLAN Pool, select Create VLAN Pool from the pull-down menu options. In VXLAN mode, the internal VLAN range is used for private VLAN allocation son the distributed virtual switch used Cisco AVE. These VLANs will not be seen outside the ESXi host or on the wire.
10. In the Create VLAN Pool pop-up window, specify a name for the pool to be associated with Cisco AVE.
11. Leave the Allocation Mode set to Dynamic Allocation.
12. For the Encap Blocks, click the [+] icon on the right side of the Encap Block section.
13. In the Create Ranges pop-up window, specify a VLAN range for the pool.
14. For the Allocation Mode, select Dynamic Allocation.
15. For the Role, select Internal.
16. Click OK to complete the VLAN range configuration and close the Create Ranges pop-up window.
17. Click Submit to complete the VLAN pool configuration and close the Create VLAN Pool pop-up window.
18. In the Create vCenter Domain window, for the AVE Fabric-Wide Multicast Address, specify an address outside the multicast address pool defined in the next step.
19. For the pool of Multicast Addresses (one per-EPG), select Create Multicast Address Pool from the pull-down menu options.
20. In the Create Multicast Address Pool pop-up window, specify a name for the Address Pool (for example, HXV1-AVE_mPOOL).
21. For the Address Blocks, click the [+] on the right side of the Address Blocks section.
22. In the Create Multicast Address Block pop-up window, specify a range.
23. Click OK to create the multicast address block and close the pop-up window.
24. Click Submit to complete and close the Create Multicast Address Pool pop-up window.
25. For vCenter Credentials, click the [+] icon on the right side of the vCenter Credentials section.
26. In the Create vCenter Credential pop-up window, specify a Name for the credentials, along with the appropriate account Username and Password.
The Administrator account is used in this example, but an APIC account can be created within the vCenter to enable the minimum set of privileges. For more information, see the ACI Virtualization Guide on cisco.com.
27. Click OK to close the Create vCenter Credential pop-up window.
28. In the Create vCenter Domain window, click the [+] icon on the right side of the vCenter section.
29. In the Create vCenter Controller pop-up window, enter a Name for the vCenter.
30. For the Host Name (or IP Address), enter the vCenter IP or Hostname.
31. For the DVS Version, leave it as vCenter Default.
32. For the Datacenter, enter the Datacenter name provisioned on the vCenter. Name is case-sensitive.
33. For the Management EPG, leave it unselected.
34. For the Associated Credential, select the vCenter credentials created in the last step (Administrator).
35. Click OK to close the Create vCenter Controller pop-up window.
36. Leave everything else as is in the Create vCenter Domain window and click Submit.
37. Log into the vCenter vSphere Web Client and navigate to Networking.
38. Confirm that the Cisco AVE distributed virtual switch was created.
Apply Virtual Switch Policies using Pre-configured Policies
All policies applied to the VMM domain in this section were pre-configured earlier in this document for fabric-wide reuse of the policies across many different access layer connections; here it will be applied to the VMM domain.
To apply the virtual switch policies using the pre-configured policies, complete the following steps:
1. Navigate to Virtual Networking > Inventory > VMM Domains > VMware.
2. Select the newly created Cisco AVE VMM Domain from the left navigation pane.
3. Select Policy > vSwitch Policy tab from the right window pane.
4. For the Port Channel Policy, select the pre-configured MAC-Pinning policy from the drop-down list options.
If the AVE is connected through a Cisco UCS FI, MAC Pinning-Physical-NIC-Load is not supported.
5. For LLDP Policy, select the pre-configured LLDP-Disabled policy.
6. For CDP Policy, select the pre-configured CDP-Enabled policy.
7. For STP Policy, select the pre-configured BPDU-FG-Enabled policy.
8. For Firewall Policy, select the pre-configured Firewall-Disabled policy.
9. Leave the NetFlow Exporter Policy unselected.
10. Click Submit and Submit Changes to apply the policy changes and close window.
Enable Statistics Collection
To enable statistics collection, complete the following steps:
1. Navigate to Virtual Networking > Inventory > VMM Domains > VMware.
2. Select the newly created Cisco AVE VMM Domain from the left navigation pane.
3. Select Policy > General tab from the right window pane.
4. In the vCenter section, select and double-click the vCenter configured in the previous step.
5. In the VMM Controller window, for Stats Collection, select Enabled.
6. Click Submit and Submit Changes to accept the change and close the window.
Configure Deployment and Resolution Immediacy for VLAN Allocation
To configure deployment and resolution immediacy, complete the following steps:
1. Navigate to Virtual Networking > Inventory > VMM Domains > VMware.
2. Select the newly created Cisco AVE VMM Domain from the left navigation pane.
3. Select Associated EPGs tab from the right window pane.
4. Select the ave-ctrl EPG and double-click to change Deployment and Resolution Immediacy from On-Demand to Immediate.
5. Click Update and Continue to apply the changes.
Add HyperFlex ESXi Hosts to Cisco AVE
To add the HyperFlex ESXi Hosts to the newly created Cisco AVE distributed virtual switch, complete the following steps:
1. Use a browser to log into the VMware vSphere Web Client.
2. From the Home screen, select Networking in the Inventories section.
3. In the left navigation pane, expand the Datacenter with the newly deployed Cisco AVE. Open the folder and select the AVE distributed virtual switch that was created by the APIC.
4. Right-click the APIC-controlled AVE distributed virtual switch and select Add and Manage hosts….
5. In the Add and Manage Hosts pop-up window, select the Add hosts option. Click Next.
6. In the Select Hosts window, click [+ New host…] to add new host.
7. In the Select new hosts pop-up window, select all of the relevant ESXi hosts. Click OK to complete.
8. Select Configure identical network settings on multiple hosts (template mode) and click Next.
9. Select a template host for configuring. Click Next.
10. Leave Manage physical adapters selected and de-select the other options. Click Next.
11. For the template host, select vmnic4 from the Host/Physical Network Adapters column.
12. Click the Assign uplink from the menu above.
13. In the Select an Uplink for vmnic4 pop-up window, leave uplink 1 selected and click OK.
14. Repeat the above steps to A=assign a second uplink to the same template host by selecting vmnic5 as uplink2. Click Next.
15. Proceed to Step 2 where the same physical adapter assignments are applied to the remaining hosts. Click the Apply to all icon in Step 2.
16. Scroll down and verify that vmnic4 and vmnic5 has been assigned as uplink1 and uplink2 on the remaining hosts. Click Next.
17. In the Analyze impact window, click Next.
18. Review the settings and click Finish to apply.
19. Review the topology to verify the changes.
Download Cisco AVE OVF file to VMware vCenter
To upload the Cisco AVE OVF to the VMware vCenter, complete the following steps:
1. Use a browser to log into the VMware vSphere Web Client.
2. From the Home screen, select Content Libraries in the Inventories section.
3. Create a new content library or go directly to Import Item if one already exists. Click Next.
4. Select Local content library. Click Next.
5. Select a datastore. Click Next.
6. Review and click Finish to complete.
7. Select Import Item.
8. In the Import Item to Content Library, select the source and click OK.
9. Select all files needed. Click OK.
10. Click OK again to import the OVF file.
11. When the OVF file is uploaded to the content library, it appears in the work pane under the Templates tab.
Setup Management Networking for Cisco AVE Virtual Machines
In this section, the following prerequisites for deploying Cisco AVE Virtual machines are addressed:
· Allocate a Management IP Address for each Cisco AVE Virtual Machine – one VM on each ESXi host. The addresses will need to be allocated via DHCP – either using an existing DHCP network or by using VMware vCenter as a DHCP server.
· Configure/Verify Virtual Networking before adding Cisco AVE VMs to Management network (port-group).
· Setup DHCP to allocate IP address to Cisco AVE VM – VMware vCenter is used in this setup. Verify that there is no IP address overlap with other devices in the same network.
In this design, the management IP addresses are allocated by VMware vCenter using DHCP. The addresses are part of the in-band management network used for (1) ESXi management and (2) HX Storage Controller Management. The Cisco AVE VMs will be added to the Storage Controller Management Network port-group that was configured by the HyperFlex Installer. Though the name of the port-group may not be intuitive for Cisco AVE management, it is the in-band management network. Other options for Cisco AVE management networking are:
· Create a port-group with a more intuitive name but still keep it in the same VLAN and subnet as in-band management.
· Create a separate AVE management network dedicated to AVE hosts with appropriate routing enabled in the ACI fabric.
Both of the above options are not used in this design.
If VMware vCenter is used as DHCP server for Cisco AVE Virtual Machines, a block of IP addresses from the inband management network must be first allocated. In this design, the block allocated is: 10.1.167.161 – 10.1.167.164/24.
To setup VMware vCenter for DHCP, complete the following steps:
1. Use a browser to log into the VMware vSphere Web Client.
2. From the Home screen, select Networking in the Inventories section.
3. From the left navigation pane, expand and select the Cisco AVE virtual distributed switch.
4. In the right window, navigate to Configure > Network Protocol Policies.
5. Click the [+] to add a new network control profile.
6. In the Add Network Protocol Profile wizard, specify a Name for the profile.
7. On the same screen, in the Network Associations section, click [+] Select Networks.
8. In the Network Associations pop-up window, select the management port-group for Cisco AVE. Click OK.
9. Click Next.
10. In the Configure IPv4 screen, specify the IP Pool Subnet, Gateway and range of IP addresses. Select Enable IP Pool check box. Click View Range to view the exact IP addresses in the pool.
11. Click Next.
12. In the Configure IPv6 screen, click Next.
13. In the Set other network configurations screen, enter any DNS information that is needed. Click Next.
14. In the Ready to complete screen, review the configuration and click Finish to complete.
15. You should now see a network protocol profile showing the IP Pool the related network and IPv4 configuration.
Deploy Cisco AVE VM on the ESXi Hosts Using the Cisco ACI Plug-In
To ESXi hosts in VMware vCenter, complete the following steps to deploy the Cisco AVE VM:
1. Use a browser to log into the VMware vSphere Web Client.
2. From the Home screen, click the Cisco ACI Fabric icon in the Inventories section.
3. From the left navigation pane, select ACI Virtual Edge.
4. From the right window pane, select a domain from the Select an ACI Virtual Edge Domain drop-down list at the top. Login with VMware vCenter password to access the tools if needed.
5. Expand the datacenter and cluster to select the hosts where Cisco AVE should be deployed. Use the boxes to the right of each host.
6. On the same screen, scroll down to the bottom and for ACI Virtual Edge version drop-down list, choose the version to be deployed.
7. For the Management PortGroup drop-down list, choose the management port group.
8. From the Datastore drop-down list, choose Custom, click Edit.
9. In the Custom AVE Datastore selection pop-up window, select the Use local datastore only check box, and specify local data store for each Cisco ACI Virtual Edge. Click OK. You may not see all types of local storage in vCenter. However, if you uncheck the Use local datastore only check box, vCenter shows all local data stores.
For details, see the document "When installing ESX/ESXi 4.x or 5.x to a physical server, the local SAS drive appears as a remote storage (1027819)" on the VMware website for details.
Cisco ACI Virtual Edge installation is supported only on local data stores in the current release.
10. For the VM Admin Password fields, enter a password for the Cisco ACI Virtual Edge VMs.
11. Click the Install/Upgrade ACI Virtual Edge button.
12. In the Install dialog box, click Yes.
13. When the install is complete, the display should change to something similar as shown below.
Now you are ready to deploy virtual machines on the HyperFlex cluster using Cisco AVE virtual leaf switches.
The high-level steps for deploying multi-tier applications on a Cisco HyperFlex cluster connected to a Cisco ACI fabric are as follows:
· Define ACI Constructs for the new Application – this includes an Application Tenant, VRF, Bridge Domain, Application Profile
· Define End Point Groups – for example, a three-tier application could be deployed with 3 EPGs for each Tier, for example: Web EPG, App EPG, Database EPG
· Enable contracts to allow communication between different tiers of the Application and access to the Application
· Deploy the applications tiers in the HyperFlex cluster
This section details the steps for creating a sample two-tier application called HXV-App-A. This tenant will comprise of a Web and an App Tier which will be mapped to relevant EPGs on the ACI fabric. Integration with Virtual Machine Manager or VMware vCenter for virtual networking should be in place before onboarding applications as outlined in this section.
Create Tenant and VRF for Application
To create a tenant and VRF for application, complete the following steps:
1. Use a browser to navigate to APIC’s Web GUI. Login with the admin user id and password.
2. From the top menu, select Tenants > Add Tenant.
3. In the Create Tenant pop-up window, specify a Name (for example, HXV-App-A).
4. For the VRF Name, enter a name for the only VRF in this Tenant (for example, HXV-App-A_VRF)
5. Leave the checkbox for Take me to this tenant when I click finish checked.
6. Click Submit to complete the configuration.
Configure Bridge Domains
At least one bridge domain will need to be created. In the following steps, an internal versus an external bridge domain is created to allow an optional insertion of a firewall between EPGs connecting from the differing bridge domains. Insertion and configuration of this firewall is not covered in this document.
1. In the left navigation pane, navigate to Tenant HXV-App-A > Networking > Bridge Domains.
2. Right-click Bridge Domains and select Create Bridge Domain.
3. Name the Bridge Domain HXV-App-A-Ext_BD and select HXV-App-A_VRF for the VRF. Select Forwarding as Custom, and change L2 Unknown Unicast to Flood.
4. Click Next.
5. Within the L3 Configurations section, select the checkbox for GARP based detection of the EP Move Detection Mode.
6. Click Next and click Finish to complete adding the Bridge Domain.
7. Repeat the steps above to add another Bridge Domain HXV-App-A-Int_BD under the same VRF.
Configure Application Profile
To configure the application profile, complete the following steps:
1. In the left navigation pane, navigate to Tenant HXV-App-A > Application Profiles.
2. Right-click Application Profiles and select Create Application Profile.
3. Name the Application Profile HXV-App-A_AP and click Submit.
Configure End Point Groups
EPG for Web
1. In the left navigation pane, navigate to Tenant HXV-App-A > Application Profiles > HXV-App-A_AP.
2. Right-click and select Create Application EPG.
3. Name the EPG HXV-A-Web_EPG. Leave Intra EPG Isolation Unenforced.
4. For the Bridge Domain, select HXV-App-A-Ext_BD from the drop-down list.
5. Check the Associate to VM Domain Profiles checkbox.
6. Click Next.
7. Click [+] to Associate VM Domain Profiles.
8. For the Domain Profile, select VMware/HXV-AVE-vm-network from the drop-down list.
9. Change the Deployment Immediacy and Resolution Immediacy to Immediate.
10. Click Update.
11. Click Finish to complete the configuration.
12. In the left navigation pane, navigate to newly created EPG (HXV-A-Web_EPG), right-click and select Create EPG Subnet.
13. For the Default Gateway IP, enter a gateway IP address and mask (for example, 172.19.101.254/24).
14. Since the Web VM Subnet is advertised to networks outside ACI and to App EPG, select checkboxes for Advertised Externally and Shared between the VRFs.
15. Click Submit.
EPG for App
1. In the left navigation pane, navigate to Tenant HXV-App-A > Application Profiles > HXV-App-A_AP.
2. Right-click and select Create Application EPG.
3. Name the EPG HXV-A-App_EPG.
4. Leave Intra EPG Isolation Unenforced.
5. For the Bridge Domain, select HXV-App-A-Int_BD from the drop-down list.
6. Check the box next to Associate to VM Domain Profiles.
7. Click Next.
8. Click [+] to Associate VM Domain Profiles.
9. For the Domain Profile, select VMware/HXV-AVE-vm-network from the drop-down list.
10. Change the Deployment Immediacy and Resolution Immediacy to Immediate.
11. Click Update.
12. Click Finish to complete the configuration.
13. In the left navigation pane, select the newly created EPG (HXV-A-App_EPG), right-click and select Create EPG Subnet.
14. For the Default Gateway IP, enter a gateway IP address and mask (for example, 172.19.102.254/24).
15. Since the App VMs only need to communicate with Web VMs EPG, select checkbox for Private to VRF.
16. Click Submit.
Verify Virtual Networking for the Application EPGs
When the two Application EPGs are provisioned in the ACI fabric and associated with a VMM domain (in this case, HXV-AVE-vm-network), you should now see two port-groups corresponding to the EPGs in the Cisco AVE switch. To verify that the port-groups have been created in the VMM domain (VMware vCenter), complete the following steps:
1. Use a browser to log into the VMware vSphere Web Client.
2. From the Home screen, select Networking in the Inventories section.
3. In the left navigation pane, expand the Datacenter, folder and distributed virtual switch for the ACI VMM domain associated with the EPGs. The distributed virtual switch would’ve been created by the Cisco APIC when the VMM domain was first created.
4. In the right window pane, navigate to Configure > Topology. The port-groups associated with the two EPGs should’ve been automatically created as shown below. When the VMM domain is associated with a Cisco AVE, two VLANs from the VLAN pool are used for each EPG to create a private VLAN.
5. The application virtual machines can now be deployed and added to these port-groups. To enable connectivity outside the EPG, the necessary contracts need to be provided and consumed between the different EPGs as outlined in the next section.
Configure Contracts
App-Tier to Web-Tier Contract
To enable communication between Web and App tiers of the application, complete the following steps:
Customers can use more restrictive contracts to replace the Allow-All contract defined in this example.
Provided Contract in EPG App-A-App
1. In the left navigation pane, navigate to Tenant HXV-App-A > Application Profiles > HXV-App-A_AP.
2. Navigate to Application EPGs > HXV-A-App_EPG.
3. Right-click Contract and select Add Provided Contract.
4. In the Add Provided Contract pop-up window, select Create Contract from end of the drop-down list.
5. Name the Contract Allow-Web-to-App.
6. For Scope, select Tenant from the drop-down list.
7. For Subjects, click [+] to add a Contract Subject.
8. In the Create Contract Subject pop-up window, specify a Name for the subject: Allow-Web-to-App.
9. For Filters, click [+] to add a Contract filter.
10. Click [+] to add a new filter.
11. In the Create Filter pop-up window, specify a Name for the filter: Allow-Web-A-All.
12. Click [+] to add an Entry.
13. Enter a name for the Entry, for example: Allow-All.
14. For the EtherType, select IP from the drop-down list.
15. Click Update.
16. Click Submit.
17. Click Update in the Create Contract Subject pop-up window.
18. Click OK to finish creating the Contract Subject.
19. Click Submit to complete creating the Contract.
20. Click Submit to complete adding the Provided Contract.
Consume Contract in EPG App-A-Web
1. In the left navigation pane, navigate to Tenant HXV-App-A > Application Profiles > HXV-App-A_AP.
2. Navigate to Application EPGs > HXV-A-Web_EPG.
3. Right-click and select Add Consumed Contract.
4. In the Add Consumed Contract pop-up window, select the newly created contract (Allow-Web-to-App) from the drop-down list.
5. Click Submit to complete adding the Consumed Contract.
Web-Tier to Shared L3 Out Contract
To enable App-A’s Web VMs to communicate outside the Fabric, Shared L3 Out contract defined in the Common Tenant will be consumed by the Web EPG. To enable Web VMs to outside the fabric, complete the following steps:
1. In the left navigation pane, navigate to Tenant HXV-App-A > Application Profiles > HXV-App-A_AP.
2. Navigate to Application EPGs > HXV-A-Web_EPG.
3. Right-click and select Add Consumed Contract.
4. In the Add Consumed Contract pop-up window, select the shared L3Out contract (common/Allow-Shared-L3Out).
5. Click Submit to complete adding the Consumed Contract.
6. Log into the Nexus 7000 routers (172.19.101.0/24) which is being advertised. Nexus 7000 routers serve as gateways to outside networks, networks outside ACI, and includes both internal networks and Internet in this design.
With the association of contracts to the Web and App EPGs, the application environment now has access to outside (L3Out) networks and the App tier is limited to accessing only the Web tier.
This solution delivers Virtual Server Infrastructure for Enterprise data centers using Cisco HyperFlex hyperconverged infrastructure and software-defined networking with Cisco’s ACI fabric. The components in the solution include Cisco’s HyperFlex Servers for compute and storage, Cisco UCS 6300 series Fabric Interconnects and Cisco Nexus 9300-EX series cloud-scale switches as ACI leaf switches. The Cisco HyperFlex with ACI solution is designed and validated using compute, network and storage best practices for high performance, high availability, and simplicity in implementation and management. This CVD document covers the design of the end-to-end solution as well as the detailed procedures for deploying it.
The following tools are available for planning and profiling a HyperFlex Deployment.
HyperFlex Sizer
HyperFlex sizer is a cloud based end-to-end tool that can help the customers and partners find out how many Cisco HyperFlex nodes are needed and how the nodes can be configured to meet their needs for the compute resources, storage capacity and performance requirements in the datacenter. The sizing guidance of the HX system is calculated according to the information of workloads collected from the users. This cloud application can be accessed from: https://hyperflexsizer.cloudapps.cisco.com (CCO login required)
The HyperFlex Sizer tool is designed to provide general guidance in evaluating the optimum solution for using selected Cisco products. The tool is not intended as a substitute for your own judgment or for that of your professional advisors.
HyperFlex Workload Profiler
The HyperFlex Workload Profiler tool is used to capture storage usage and performance statistics from an existing VMware ESX cluster, enabling you to use that data to assist with sizing a HyperFlex cluster which would assume that workload. The workload profiler is distributed as an OVA file, which can be deployed using static or DHCP assigned addressing, on an existing VMware ESXi host. Once deployed, the profiler tool connects to an existing VMware vCenter server to gather storage statistics for the selected ESXi hosts. This tool can be access from: https://hyperflexsizer.cloudapps.cisco.com.
Nexus 7000 – Sample Configuration
The following configuration is a sample from the virtual device contexts (VDCs) of two Nexus 7004s.
The Nexus 7000 configuration provided below is not complete and is meant to be used only as a reference.
Nexus 7004-1
feature ospf
!
interface Ethernet4/4
description To 9372-1 E1/47
no shutdown
!
interface Ethernet4/4.305
description To 9372-1 E1/47
encapsulation dot1q 305
ip address 10.251.231.2/30
ip ospf network point-to-point
ip ospf mtu-ignore
ip router ospf 10 area 0.0.0.10
no shutdown
!
interface Ethernet4/8
description To 9372-2 E1/47
no shutdown
!
interface Ethernet4/8.307
description To 9372-2 E1/47
encapsulation dot1q 307
ip address 10.251.231.6/30
ip ospf network point-to-point
ip ospf mtu-ignore
ip router ospf 10 area 0.0.0.10
no shutdown
!
interface loopback0
ip address 10.251.255.21/32
ip router ospf 10 area 0.0.0.0
!
router ospf 10
router-id 10.251.255.21
area 0.0.0.10 nssa no-summary no-redistribution default-information-originate
!
Nexus 7004-2
feature ospf
!
interface Ethernet4/4
description To 93180-1 E1/48
no shutdown
!
interface Ethernet4/4.306
description To 93180-1 E1/48
encapsulation dot1q 306
ip address 10.251.231.10/30
ip ospf network point-to-point
ip ospf mtu-ignore
ip router ospf 10 area 0.0.0.10
no shutdown
!
interface Ethernet4/8
description To 93180-2 E1/48
no shutdown
!
interface Ethernet4/8.308
description To 93180-2 E1/48
encapsulation dot1q 308
ip address 10.251.231.14/30
ip ospf network point-to-point
ip ospf mtu-ignore
ip router ospf 10 area 0.0.0.10
no shutdown
!
interface loopback0
ip address 10.251.255.22/32
ip router ospf 10 area 0.0.0.0
!
router ospf 10
router-id 10.251.255.2
area 0.0.0.10 nssa no-summary no-redistribution default-information-originate
!
Upgrade HyperFlex Components
Upgrade ESXi
The upgrade procedure detailed in this section is for upgrading ESXi on HX nodes in an active HyperFlex cluster. For an install, re-install, or upgrade of ESXI before HyperFlex is installed uses a different procedure; use the HyperFlex Install and Upgrade guides available on cisco.com for more information.
A published Cisco custom ESXi offline bundle file is required to upgrade ESXi on a running cluster.
The ESXi hypervisor version can be upgraded with no disruption to the HyperFlex cluster workload. This is achieved by performing an online rolling upgrade of each node in the HX cluster.
Upgrade Overview
A high-level overview of the upgrade process is provided below along with some additional information and recommendations; review before doing an ESXi upgrade. To upgrade, complete the following steps:
1. ESXi upgrade requires a manual online upgrade.
2. When upgrading VMware ESXi from 5.5 U3b through any version up to 6.0 U2, please contact Cisco TAC.
3. Use the ESXi command line interface esxcli for upgrading or updating ESXi; VMware Update Manager (VUM) is not recommended.
4. Upgrade vCenter to a compatible version before beginning ESXi upgrades on the hosts; use VMware Product Compatibility Matrix to confirm: https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop
5. Complete pre-upgrade validation checks. See Pre-Upgrade Validation Checks in Chapter 3 in the Cisco HyperFlex Systems Upgrade Guide, Release 3.0: https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/HyperFlex_upgrade_guide/3-0/b_HyperFlexSystems_Upgrade_Guide_3_0.html
6. Download ESXi Software from HyperFlex Download website: https://software.cisco.com/download/release.html?mdfid=286305544&flowid=79522&softwareid=286305994
7. Select one of the hosts and put it in HX maintenance mode using the vSphere Web Client, see Entering Cisco HyperFlex Maintenance Mode: https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/HyperFlex_upgrade_guide/3-0/b_HyperFlexSystems_Upgrade_Guide_3_0/b_HyperFlexSystems_Upgrade_Guide_3_0_chapter_011.html#id_45660
8. To copy ESXi offline upgrade bundle file to the ESXi host using SCP, enable SSH service on ESXi hosts.
9. Copy the ESXi offline upgrade bundle file using SCP to a mounted HX datastore or local disk (On HX240 models) on the ESXi host as follows:
scp <local_filename> <user@server:/path/where/file/should/go>
10. Login to ESXi host and execute the following command to query the list of available image profiles and for profile name verification.
esxcli software sources profile list -d <location_of_the_esxi_zip_bundle_on _datastore>
11. Run the following command to perform the upgrade.
esxcli software profile update -d <path_to_profile_ZIP_file> -p <profile name>
12. When upgrade is complete, reboot the ESXi host by issuing the command: reboot
13. Verify that the host booted up with the correct version by issuing the command: vmware -v.
14. Wait for host to reconnect to vCenter and exit maintenance mode– see Ex iting Cisco HyperFlex Maintenance Mode: https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HyperFlex_HX_DataPlatformSoftware/HyperFlex_upgrade_guide/3-0/b_HyperFlexSystems_Upgrade_Guide_3_0/b_HyperFlexSystems_Upgrade_Guide_3_0_chapter_011.html#id_45660
15. Verify that the HX cluster is healthy after each ESXi node upgrade by issuing the command: stcli cluster storage-summary –detail
16. Repeat upgrade steps above for each host in the HyperFlex cluster.
Pre-Upgrade Validation Checks
1. From vSphere Web Client, verify HyperFlex Cluster Health. Navigate to Home > Global Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform. Select the HX Cluster, followed by the Summary tab > Status widget to verify that Operational and Resiliency Status is Online and Healthy.
2. From vSphere Web Client, verify that DRS is enabled. Navigate to Home > Global Inventory Lists > Clusters. Select the HX Cluster, followed by Configure tab > vSphere DRS.
3. From vSphere Web Client, verify that vSphere services are running and ESX Agent Manager (EAM) health is normal.
4. From Cisco UCS Manager, verify that both Fabric Interconnects are up and running and that the data path is up and running. Verify that HyperFlex server and vNICs on servers have no faults.
5. From vSphere Web Client, verify the HX server (UCS C-Bundle) firmware version. Navigate to Server > Policies > Root > Sub-Organizations. Select and expand the HX sub-organization and navigate to Host Firmware Packages > HyperFlex-m5 in the left navigation pane. Note the Rack Package version below.
6. From vSphere Web Client, verify vMotion interface is configured and enabled for Jumbo Frames (MTU of 9000). Navigate to Home > Hosts and Clusters. Navigate to HX Datacenter and HX Cluster, and select the host. Navigate to Configure tab, followed by Networking > Virtual Switches. Select the virtual switch for vMotion and navigate to vMotion vmk in the virtual switch figure shown in the window below. Click the (i) button next to the vMotion vmk and verify that the MTU is 9000. The installer configured the IP address and MTU as shown in the figure below.
7. Verify the HX cluster access policy is set to lenient mode. SSH as root into any of HX Storage Controller VMs and use the following commands to verify and enable as needed.
stcli cluster get-cluster-access-policy
8. The above command can also be executed directly from the HX Connect. Login in as admin and navigate to the >_Web CLI in the left navigation pane and enter the above command as shown below.
9. If set to strict, change it as follows:
stcli cluster set-cluster-access-policy --name lenient
10. Verify the HX Node has sufficient storage. If the available storage is <50%, call Cisco TAC. SSH as root into the controller VM on each HX node and run the command as shown below: df -h /var/stv
11. Repeat for all nodes in the cluster.
Download and Update ESXi Software
To download and update ESXI software, complete the following steps:
1. Download ESXI software. To update ESXI, use the offline bundle (.zip)
2. Put the server in Maintenance mode from HX Connect. Navigate to the menu shown below: System Information > Nodes. Select the node and click Enter HX Maintenance mode from the menu above. Click Enter HX Maintenance Mode button to confirm.
3. SCP the software onto the ESX host. In this case, it was uploaded to an existing datastore on the ESXi node: /vmfs/volumes/HXV1-DS1 as shown below.
4. Before you can update using the offline image from ESXi command line, query the list of available image profiles and determine the profile as shown below; the profile name is necessary for ESXi update.
5. Update ESXI image using offline bundle (.zip) and profile name from above (Vmware-ESXi-6.5.0-HX-7388607-Custom-Cisco-6.5.0.1). Verify the update completed successfully. A partial screenshot that shows the top status of the update is shown below.
6. Restart the ESXi host using the command: reboot.
7. Reconnect to the host and verify it is running the correct version using the command: vmware -v.
8. Wait for host to reconnect to vCenter. If necessary, from vSphere web client and navigate to Hosts and Clusters > Datacenter > Cluster > Host. Right-click and select Connection > Connect. Exiting from maintenance mode in the next step may not work if the host is not connected to vCenter.
9. Exit maintenance mode from HX Connect. This is the same menu that was used to Enter Cisco HyperFlex Maintenance Mode above.
10. Verify that the HX cluster is healthy after each ESXi node upgrade by issuing the command: stcli cluster storage-summary –detail
11. Repeat the above steps for each host in the HyperFlex cluster. Note that the images needs to be copied to the datastore only once since the same datastore is mounted on all hosts in the cluster.
12. Notice that the steps above show the upgrade to ESXi 6.5U1 Patch 02 from the default ESXi version installed by the HX Installer. This system used in this design was updated a second time in validation to ESXi 6.5U2 GA as shown below.
Cisco HyperFlex
Cisco HyperFlex 3.0 for Virtual Server Infrastructure with VMware ESXi:
https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/hyperflex_30_vsi_esxi.pdf
Comprehensive Documentation for Cisco HyperFlex:
https://http://hyperflex.io
Comprehensive Documentation Roadmap for Cisco HyperFlex:
HX220c M5 Server: https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HX_series/HX220c_M5/HX220c_M5.html
HX240c M5 Server: https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HX_series/HX240c_M5/HX240c_M5.html
HX240c M5L Server: https://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HX_series/HX240c_M5/HX240c_M5.html
HX220c M4 Server: http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HX_series/HX220c_M4/HX220c/overvie w.html
HX240c M4 Server: http://www.cisco.com/c/en/us/td/docs/hyperconverged_systems/HX_series/HX240c_M4/HX240c/overvie w.html
Cisco UCS
Cisco Unified Computing System:
http://www.cisco.com/en/US/products/ps10265/index.html
Cisco UCS 6300 Series Fabric Interconnects:
Cisco UCS 5100 Series Blade Server Chassis:
http://www.cisco.com/en/US/products/ps10279/index.html
Cisco UCS 2300 Series Fabric Extenders:
Cisco UCS 2200 Series Fabric Extenders:
Cisco UCS B-Series Blade Servers:
http://www.cisco.com/en/US/partner/products/ps10280/index.html
Cisco UCS C-Series Rack Mount Servers:
http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-c-series-rack-servers/index.html
Cisco UCS VIC Adapters:
http://www.cisco.com/en/US/products/ps10277/prod_module_series_home.html
Cisco UCS Manager:
http://www.cisco.com/en/US/products/ps10281/index.html
Cisco UCS Manager Plug-in for VMware vSphere Web Client:
Datacenter Network Fabric – Cisco ACI
Cisco Nexus 9000 Series Switches:
http://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/index.html
Cisco Application Centric Infrastructure – Datacenter and Virtualization:
https://www.cisco.com/c/en_au/solutions/data-center-virtualization/aci.html
Cisco Application Centric Infrastructure – Cisco Datacenter:
Cisco ACI Fundamentals:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/aci-fundamentals/b_ACI-Fundamentals.html
Cisco ACI Infrastructure Release 2.3 Design Guide:
Cisco ACI Infrastructure Best Practices Guide:
Virtualization Layer
VMware vCenter Server:
http://www.vmware.com/products/vcenter-server/overview.html
VMware vSphere:
https://www.vmware.com/products/vsphere
Security
Integrating Cisco Umbrella to Cisco HyperFlex and Cisco UCS Solutions:
Interoperability Matrixes
Cisco UCS and HyperFlex Hardware Compatibility Matrix:
https://ucshcltool.cloudapps.cisco.com/public/
VMware and Cisco Unified Computing System:
http://www.vmware.com/resources/compatibility
Cisco ACI Virtualization Compatibility Matrix:
Cisco APIC and ACI Virtual Edge Support Matrix:
https://www.cisco.com/c/dam/en/us/td/docs/Website/datacenter/aveavsmatrix/index.html
Archana Sharma, Technical Leader, Cisco UCS Data Center Solutions, Cisco Systems Inc.
Archana Sharma is Technical Marketing Engineer with over 20 years of experience at Cisco on a range of technologies that span Data Center, Desktop Virtualization, Collaboration, and other Layer2 and Layer3 technologies. Archana is focused on systems and solutions for Enterprise and Provider deployments, including delivery of Cisco Validated designs for over 10 years. Archana is currently working on designing and integrating Cisco UCS-based Converged Infrastructure solutions. Archana holds a CCIE (#3080) in Routing and Switching and a Bachelor’s degree in Electrical Engineering from North Carolina State University.
Acknowledgements
· Ramesh Isaac, Technical Marketing Engineer, Cisco Systems, Inc.
· Brian Everitt, Technical Marketing Engineer, Cisco Systems, Inc.
· Haseeb Niazi, Technical Marketing Engineer, Cisco Systems, Inc.
Feedback