Cisco UCS Infrastructure with Red Hat OpenShift Container Platform on VMware vSphere

Design and Deployment Guide for Cisco UCS Infrastructure for Red Hat OpenShift Container Platform 3.9 and VMware vSphere 6.7 on Cisco UCS Manager 3.2, Cisco UCS M5 B-Series, and Cisco UCS M5 C-Series Servers

Cisco UCS Infrastructure with Red Hat OpenShift Container Platform on VMware vSphere PDF

NOTE: Works with document’s Advanced Properties “First Published” property. Click File | Properties | Advanced Properties | Custom.

Last Updated: December 13, 2018

NOTE: Works with document’s Advanced Properties “Last Updated” property. Click File | Properties | Advanced Properties | Custom.

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, see:

http://www.cisco.com/go/designzone.

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS.  CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.  IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE.  USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS.  THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS.  USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS.  RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2018 Cisco Systems, Inc. All rights reserved.

 

Table of Contents

Executive Summary. 6

Solution Overview.. 7

Introduction. 7

Solution Benefits. 7

Audience. 8

Purpose of this Document. 8

What’s New in this Release?. 8

Technology Overview.. 9

Cisco Unified Computing System... 9

Cisco UCS Manager. 10

Cisco UCS Fabric Interconnects. 10

Cisco UCS 5108 Blade Server Chassis. 11

Cisco UCS B200 M5 Blade Server. 11

Cisco UCS C220M5 Rack-Mount Server. 12

Cisco UCS C240M5 Rack-Mount Server. 12

Cisco VIC Interface Cards. 13

Cisco UCS Fabric Extenders. 14

Cisco Nexus 9000 Switches. 14

Intel Scalable Processor Family. 15

Intel^® SSD DC S4500 Series. 15

Red Hat OpenShift Container Platform... 16

Kubernetes Infrastructure. 16

Red Hat OpenShift Integrated Container Registry. 16

Container-native Storage Solution from Red Hat. 16

Docker. 17

Kubernetes. 17

Etcd. 17

Open vSwitch. 17

HAProxy. 17

Red Hat Ansible Automation. 17

Solution Design. 19

Hardware and Software Revisions. 19

Solution Components. 19

Architectural Overview.. 20

Bastion Node. 21

OpenShift Master Nodes. 21

OpenShift Infrastructure Nodes. 21

OpenShift Application Nodes. 22

OpenShift Storage Nodes. 22

Physical Topology. 23

Logical Topology. 23

Virtual Machine Instance Details. 24

Red Hat OpenShift Container Platform Node Placement. 25

HA Proxy Load Balancer. 26

Deployment Hardware and Software. 27

Solution Prerequisites. 27

Required Channels. 27

Deployment Workflow.. 27

DNS (Domain Name Server) Configuration. 28

Application DNS.. 29

VMware vCenter Prerequisites. 30

Networking. 30

vCenter Shared Storage. 31

vSphere Parameter. 31

Resource Pool, Cluster Name, and Folder Location. 32

Prepare RHEL VM Template. 32

Setting Up Bastion Instance. 33

Configure Ansible. 34

Prepare Inventory File. 35

Red Hat OpenShift Container Platform Instance Creation. 39

Setup DRS Anti-Affinity Rules. 42

Configure VM Latency Sensitivity. 44

Red Hat OpenShift Platform Storage Node Setup. 44

Creating Storage Profile. 44

Boot Policy for Storage Node. 47

Service Profile Template for Storage Nodes. 48

Installation of Red Hat Enterprise Linux Operating System in Storage Nodes. 49

Configure Storage Node Interfaces for RHOCP.. 54

Creating an SSH Keypair for Ansible. 55

Configure and Install Prerequisites for Storage Nodes. 57

Instance Verification. 58

Red Hat OpenShift Container Platform Prerequisites Playbook. 59

Deploying Red Hat OpenShift Container Platform... 60

Functional Validation. 60

Sample Application Test Scenario. 68

Web Console – UI Operations. 70

Scale the Environment. 81

Add Metrics to the Installation. 87

Add Logging to the Installation. 87

Resources. 88

Conclusion. 89

About the Authors. 90

Acknowledgements. 90

 

 

Cisco Validated Designs are the foundation of systems design and the centerpiece of facilitating complex customer deployments. The validated designs incorporate products and technologies into a broad portfolio of Enterprise, Service Provider, and Commercial systems that are designed, tested, and fully documented to help ensure faster, reliable, consistent, and more predictable customer deployments.

Cisco’s converge infrastructures integrate systems that can make IT operations more cost-effective, efficient, and agile by integrating systems with Cisco Unified Computing System. These validated converged architectures accelerate customer’s success by reducing risk with strategic guidance and expert advice.

Cisco leads in integrated systems and offers diversified portfolio of converged infrastructure of solutions. They integrate wide range of technologies and products into cohesive validated and supported solutions to address business needs of Cisco’s customers. Furthermore, these converged infrastructures have been previously designed, tested, and validated with VMware vSphere environment.

The recommended solution architecture offers Red Hat OpenShift platform 3.9 on VMware vSphere 6.7 based on Cisco validated converged infrastructure. To learn more about datacenter virtualization on Cisco’s validated converged infrastructure, see: https://www.cisco.com/c/en/us/solutions/data-center-virtualization/converged-infrastructure/index.html.

Introduction

Deployment-centric application platform and DevOps initiatives are driving benefits for organizations in their digital transformation journey. Though still early in maturity, Docker format container packaging and Kubernetes container orchestration are emerging to cater to the rapid digital transformation. Docker format container packaging and Kubernetes container orchestration are emerging as industry standards for state-of-the-art PaaS solutions.

Containers have brought a lot of excitement and value to IT by establishing predictability between building and running applications. Developers can trust and know that their application will perform the same when it’s run on a production environment as it did when it was built, while operations and admins have the tools to operate and maintain applications seamlessly.

Red Hat^® OpenShift^® Container Platform provides a set of container-based open source tools enabling digital transformation, which accelerates application development while making optimal use of infrastructure. Professional developers utilize fine-grained control of all aspects of the application stack, with application configurations enabling rapid response to unforeseen events. Availability of highly secure operating systems assists in standing up an environment capable of withstanding continuously changing security threats, helping deployment with highly secure applications.

Red Hat OpenShift Container Platform helps organizations use the cloud delivery model and simplify continuous delivery of applications and services on Red Hat OpenShift Container Platform, the cloud-native way. Built on proven open source technologies, Red Hat OpenShift Container Platform also provides development teams multiple modernization options to enable a smooth transition to microservices architecture and the cloud for existing traditional applications.

Cisco Unified Computing System™ (Cisco UCS^®) servers adapt to meet rapidly changing business needs, including just-in-time deployment of new computing resources to meet requirements and improve business outcomes. With Cisco UCS, you can tune your environment to support the unique needs of each application while powering all your server workloads on a centrally managed, highly scalable system. Cisco UCS brings the flexibility of non-virtualized and virtualized systems in a way that no other server architecture can, lowering costs and improving your return on investment (ROI).

Cisco UCS M5 servers built on Intel’s powerful Intel^® Xeon Scalable processors are unified yet modular, scalable, high-performing, built on infrastructure-as-code for powerful integrations and continuous delivery of distributed applications.

Cisco, Intel and Red Hat have joined hands to develop a best-in-class solution for delivering PaaS solution to the enterprise with ease. And also, to provide the ability to develop, deploy, and manage containers in an on-premises, Private/ Public cloud environments by bringing automation to the table with a robust platform such as Red Hat OpenShift Container Platform.

Solution Benefits

Some of the key benefits of this solution include:

·         Red Hat OpenShift

-        Strong, role-based access controls, with integrations to enterprise authentication systems.

-        Powerful, web-scale container orchestration and management with Kubernetes.

-        Integrated Red Hat Enterprise Linux^® Atomic Host, optimized for running containers at scale with Security-Enhanced Linux (SELinux) enabled for strong isolation.

-        Integration with public and private registries.

-        Integrated CI/CD tools for secure DevOps practices.

-        A new model for container networking.

-        Modernize application architectures toward microservices.

-        Adopt a consistent application platform for hybrid cloud deployments.

-        Support for remote storage volumes.

-        Persistent storage for stateful cloud-native containerized applications.

·         Cisco UCS

-        Reduced datacenter complexities through Cisco UCS infrastructure with a single management control plane for hardware lifecycle management.

-        Easy to deploy and scale the solution.

-        Superior scalability and high-availability.

-        Compute form factor agnostic.

-        Better response with optimal ROI.

-        Optimized hardware footprint for production and dev/test deployments.

Audience

The audience for this document includes, but is not limited to, sales engineers, field consultants, professional services, IT managers, partner engineers, IT architects, and customers who want to take advantage of an infrastructure that is built to deliver IT efficiency and enable IT innovation. The reader of this document is expected to have the necessary training and background to install and configure Red Hat Enterprise Linux, Cisco Unified Computing System, and Cisco Nexus Switches, Enterprise storage sub-systems, VMware vSphere. Furthermore, knowledge of container platform preferably Red Hat OpenShift Container Platform is required. External references are provided where applicable and familiarity with these documents is highly recommended.

Purpose of this Document

This document highlights the benefits of using Cisco UCS M5 servers for Red Hat OpenShift Container Platform 3.9 on VMware vSphere 6.7 to efficiently deploy, scale, and manage a production-ready application container environment for enterprise customers. This document focuses design choices and best practices of deploying Red Hat OpenShift container platform on converged infrastructure comprised of Cisco UCS, Nexus, and VMware.

What’s New in this Release?

In this solution Red Hat OpenShift Platform 3.9 is validated on Cisco UCS with VMware vSphere. Red Hat OpenShift Container platform nodes such as master, infrastructure, and application nodes are running in VMware vSphere virtualized environment while leveraging VMware HA cluster. Furthermore, GlusterFS is running on bare-metal environment on Cisco UCS C240 M5 which can also be utilized for provisioning containers on bare-metal.

This section provides a brief introduction of the various hardware/ software components used in this solution.

Cisco Unified Computing System

The Cisco Unified Computing System is a next-generation solution for blade and rack server computing. The system integrates a low-latency; lossless 10 Gigabit Ethernet unified network fabric with enterprise-class, x86-architecture servers. The system is an integrated, scalable, multi-chassis platform in which all resources participate in a unified management domain. The Cisco Unified Computing System accelerates the delivery of new services simply, reliably, and securely through end-to-end provisioning and migration support for both virtualized and non-virtualized systems. Cisco Unified Computing System provides:

·         Comprehensive Management

·         Radical Simplification

·         High Performance

The Cisco Unified Computing System consists of the following components:

·         Compute-The system is based on an entirely new class of computing system that incorporates rack mount and blade servers based on Intel® Xeon® scalable processors product family.

·         Network-The system is integrated onto a low-latency, lossless, 40-Gbps unified network fabric. This network foundation consolidates Local Area Networks (LAN’s), Storage Area Networks (SANs), and high-performance computing networks which are separate networks today. The unified fabric lowers costs by reducing the number of network adapters, switches, and cables, and by decreasing the power and cooling requirements.

·         Virtualization-The system unleashes the full potential of virtualization by enhancing the scalability, performance, and operational control of virtual environments. Cisco security, policy enforcement, and diagnostic features are now extended into virtualized environments to better support changing business and IT requirements.

·         Storage access-The system provides consolidated access to both SAN storage and Network Attached Storage (NAS) over the unified fabric. It is also an ideal system for Software Defined Storage (SDS). Combining the benefits of single framework to manage both the compute and Storage servers in a single pane, Quality of Service (QOS) can be implemented if needed to inject IO throttling in the system. In addition, the server administrators can pre-assign storage-access policies to storage resources, for simplified storage connectivity and management leading to increased productivity. In addition to external storage, both rack and blade servers have internal storage which can be accessed through built-in hardware RAID controllers. With storage profile and disk configuration policy configured in Cisco UCS Manager, storage needs for the host OS and application data gets fulfilled by user defined RAID groups for high availability and better performance.

·         Management-the system uniquely integrates all system components to enable the entire solution to be managed as a single entity by the Cisco UCS Manager. The Cisco UCS Manager has an intuitive graphical user interface (GUI), a command-line interface (CLI), and a powerful scripting library module for Microsoft PowerShell built on a robust application programming interface (API) to manage all system configuration and operations.

Cisco Unified Computing System (Cisco UCS) fuses access layer networking and servers. This high-performance, next-generation server system provides a data center with a high degree of workload agility and scalability.

Cisco UCS Manager

Cisco Unified Computing System Manager (Cisco UCS Manager) provides unified, embedded management for all software and hardware components in Cisco UCS. Using Single Connect technology, it manages, controls, and administers multiple chassis for thousands of virtual machines. Administrators use the software to manage the entire Cisco Unified Computing System as a single logical entity through an intuitive GUI, a command-line interface (CLI), or an XML API. The Cisco UCS Manager resides on a pair of Cisco UCS 6300 Series Fabric Interconnects using a clustered, active-standby configuration for high-availability.

Cisco UCS Manager offers unified embedded management interface that integrates server, network, and storage. Cisco UCS Manager performs auto-discovery to detect inventory, manage, and provision system components that are added or changed. It offers comprehensive set of XML API for third part integration, exposes 9000 points of integration and facilitates custom development for automation, orchestration, and to achieve new levels of system visibility and control.

Service profiles benefit both virtualized and non-virtualized environments and increase the mobility of non-virtualized servers, such as when moving workloads from server to server or taking a server offline for service or upgrade. Profiles can also be used in conjunction with virtualization clusters to bring new resources online easily, complementing existing virtual machine mobility.

For more information about Cisco UCS Manager Information, see: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-manager/index.html.

Cisco UCS Fabric Interconnects

The Fabric interconnects provide a single point for connectivity and management for the entire system. Typically deployed as an active-active pair, the system’s fabric interconnects integrate all components into a single, highly-available management domain controlled by Cisco UCS Manager. The fabric interconnects manage all I/O efficiently and securely at a single point, resulting in deterministic I/O latency regardless of a server or virtual machine’s topological location in the system.

Cisco UCS 6300 Series Fabric Interconnects support the bandwidth up to 2.43-Tbps unified fabric with low-latency, lossless, cut-through switching that supports IP, storage, and management traffic using a single set of cables. The fabric interconnects feature virtual interfaces that terminate both physical and virtual connections equivalently, establishing a virtualization-aware environment in which blade, rack servers, and virtual machines are interconnected using the same mechanisms. The Cisco UCS 6332-16UP is a 1-RU Fabric Interconnect that features up to 40 universal ports that can support 24 40-Gigabit Ethernet, Fiber Channel over Ethernet, or native Fiber Channel connectivity. In addition to this it supports up to 16 1- and 10-Gbps FCoE or 4-, 8- and 16-Gbps Fibre Channel unified ports.

Figure 1       Cisco UCS Fabric Interconnect 6332-16UP

For more information, see: https://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-6332-16up-fabric-interconnect/index.html.

Cisco UCS 5108 Blade Server Chassis

The Cisco UCS 5100 Series Blade Server Chassis is a crucial building block of the Cisco Unified Computing System, delivering a scalable and flexible blade server chassis. The Cisco UCS 5108 Blade Server Chassis is six rack units (6RU) high and can mount in an industry-standard 19-inch rack. A single chassis can house up to eight half-width Cisco UCS B-Series Blade Servers and can accommodate both half-width and full-width blade form factors. Four single-phase, hot-swappable power supplies are accessible from the front of the chassis. These power supplies are 92 percent efficient and can be configured to support non-redundant, N+ 1 redundant and grid-redundant configurations. The rear of the chassis contains eight hot-swappable fans, four power connectors (one per power supply), and two I/O bays for Cisco UCS 2304 Fabric Extenders. A passive mid-plane provides multiple 40 Gigabit Ethernet connections between blade serves and fabric interconnects. The Cisco UCS 2304 Fabric Extender has four 40 Gigabit Ethernet, FCoE-capable, Quad Small Form-Factor Pluggable (QSFP+) ports that connect the blade chassis to the fabric interconnect. Each Cisco UCS 2304 can provide one 40 Gigabit Ethernet ports connected through the midplane to each half-width slot in the chassis, giving it a total eight 40G interfaces to the compute. Typically configured in pairs for redundancy, two fabric extenders provide up to 320 Gbps of I/O to the chassis.

For more information, see: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-5100-series-blade-server-chassis/index.html.

Cisco UCS B200 M5 Blade Server

The Cisco UCS B200 M5 Blade Server has the following:

·         Up to two Intel Xeon Scalable CPUs with up to 28 cores per CPU

·         24 DIMM slots for industry-standard DDR4 memory at speeds up to 2666 MHz, with up to 3 TB of total memory when using 128-GB DIMMs

·         Modular LAN On Motherboard (mLOM) card with Cisco UCS Virtual Interface Card (VIC) 1340, a 2-port, 40 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE)–capable mLOM mezzanine adapter

·         Optional rear mezzanine VIC with two 40-Gbps unified I/O ports or two sets of 4 x 10-Gbps unified I/O ports, delivering 80 Gbps to the server; adapts to either 10- or 40-Gbps fabric connections

·         Two optional, hot-pluggable, Hard-Disk Drives (HDDs), Solid-State Disks (SSDs), or NVMe 2.5-inch drives with a choice of enterprise-class RAID or pass-through controllers

Figure 2       Cisco UCS B200 M5 Blade Server

For more information, see: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-b-series-blade-servers/datasheet-c78-739296.html.

Cisco UCS C220M5 Rack-Mount Server

The Cisco UCS C220 M5 Rack Server is among the most versatile general-purpose enterprise infrastructure and application servers in the industry. It is a high-density 2-socket rack server that delivers industry-leading performance and efficiency for a wide range of workloads, including virtualization, collaboration, and bare metal applications. The Cisco UCS C-Series Rack Servers can be deployed as standalone servers or as part of the Cisco Unified Computing System™ (Cisco UCS) to take advantage of Cisco’s standards-based unified computing innovations that help reduce customers’ Total Cost of Ownership (TCO) and increase their business agility. The Cisco UCS C220 M5 server extends the capabilities of the Cisco UCS portfolio in a 1-Rack-Unit (1RU) form factor. It incorporates the Intel^® Xeon^® Scalable processors, supporting up to 20 percent more cores per socket, twice the memory capacity, 20 percent greater storage density, and five times more PCIe NVMe Solid-State Disks (SSDs) compared to the previous generation of servers. These improvements deliver significant performance and efficiency gains that will improve your application performance. The C220 M5 delivers outstanding levels of expandability and performance in a compact package, with:

·         Latest Intel Xeon Scalable CPUs with up to 28 cores per socket

·         Up to 24 DDR4 DIMMs for improved performance

·         Up to 10 Small-Form-Factor (SFF) 2.5-inch drives or 4 Large-Form-Factor (LFF) 3.5-inch drives (77 TB storage capacity with all NVMe PCIe SSDs)

·         Support for 12-Gbps SAS modular RAID controller in a dedicated slot, leaving the remaining PCIe Generation 3.0 slots available for other expansion cards

·         Modular LAN-On-Motherboard (mLOM) slot that can be used to install a Cisco UCS Virtual Interface Card (VIC) without consuming a PCIe slot

·         Dual embedded Intel x550 10GBASE-T LAN-On-Motherboard (LOM) ports

Figure 3       Cisco UCS C220 M5SX

For more information, see: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/datasheet-c78-739281.html.

Cisco UCS C240M5 Rack-Mount Server

The Cisco UCS C240 M5 Rack Server is a 2-socket, 2-Rack-Unit (2RU) rack server offering industry-leading performance and expandability. It supports a wide range of storage and I/O-intensive infrastructure workloads, from big data and analytics to collaboration. Cisco UCS C-Series Rack Servers can be deployed as standalone servers or as part of a Cisco Unified Computing System™ (Cisco UCS) managed environment to take advantage of Cisco’s standards-based unified computing innovations that help reduce customers’ Total Cost of Ownership (TCO) and increase their business agility.

In response to ever-increasing computing and data-intensive real-time workloads, the enterprise-class Cisco UCS C240 M5 server extends the capabilities of the Cisco UCS portfolio in a 2RU form factor. It incorporates the Intel® Xeon® Scalable processors, supporting up to 20 percent more cores per socket, twice the memory capacity, and five times more

Non-Volatile Memory Express (NVMe) PCI Express (PCIe) Solid-State Disks (SSDs) compared to the previous generation of servers. These improvements deliver significant performance and efficiency gains that will improve your application performance. The C240 M5 delivers outstanding levels of storage expandability with exceptional performance, with:

·         Latest Intel Xeon Scalable CPUs with up to 28 cores per socket

·         Up to 24 DDR4 DIMMs for improved performance

·         Up to 26 hot-swappable Small-Form-Factor (SFF) 2.5-inch drives, including 2 rear hot-swappable SFF drives (up to 10 support NVMe PCIe SSDs on the NVMe-optimized chassis version), or 12 Large-Form-Factor (LFF) 3.5-inch drives plus 2 rear hot-swappable SFF drives

·         Support for 12-Gbps SAS modular RAID controller in a dedicated slot, leaving the remaining PCIe Generation 3.0 slots available for other expansion cards

·         Modular LAN-On-Motherboard (mLOM) slot that can be used to install a Cisco UCS Virtual Interface Card (VIC) without consuming a PCIe slot, supporting dual 10- or 40-Gbps network connectivity

·         Dual embedded Intel x550 10GBASE-T LAN-On-Motherboard (LOM) ports

·         Modular M.2 or Secure Digital (SD) cards that can be used for boot

Figure 4       Cisco UCS C240 M5SX

For more information, see: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/datasheet-c78-739279.html.

Cisco VIC Interface Cards

The Cisco UCS Virtual Interface Card (VIC) 1340 is a 2-port 40-Gbps Ethernet or dual 4 x 10-Gbps Ethernet, Fiber Channel over Ethernet (FCoE) capable modular LAN on motherboard (mLOM) designed exclusively for the M4 generation of Cisco UCS B-Series Blade Servers. All the blade servers for both Controllers and Computes will have MLOM VIC 1340 card. Each blade will have a capacity of 40Gb of network traffic. The underlying network interfaces like will share this MLOM card.

The Cisco UCS VIC 1340 enables a policy-based, stateless, agile server infrastructure that can present over 256 PCIe standards-compliant interfaces to the host that can be dynamically configured as either network interface cards (NICs) or host bus adapters (HBAs).

For more information, see: http://www.cisco.com/c/en/us/products/interfaces-modules/ucs-virtual-interface-card-1340/index.html.

The Cisco UCS Virtual Interface Card 1385 improves flexibility, performance, and bandwidth for Cisco UCS C-Series Rack Servers. It offers dual-port Enhanced Quad Small Form-Factor Pluggable (QSFP+) 40 Gigabit Ethernet and Fibre Channel over Ethernet (FCoE) in a half-height PCI Express (PCIe) adapter. The 1385 card works with Cisco Nexus 40 Gigabit Ethernet (GE) and 10 GE switches for high-performance applications. The Cisco VIC 1385 implements the Cisco Data Center Virtual Machine Fabric Extender (VM-FEX), which unifies virtual and physical networking into a single infrastructure. The extender provides virtual-machine visibility from the physical network and a consistent network operations model for physical and virtual servers.

For more information, see: https://www.cisco.com/c/en/us/products/interfaces-modules/ucs-virtual-interface-card-1385/index.html.

Cisco UCS Fabric Extenders

Cisco UCS 2304 Fabric Extender brings the unified fabric into the blade server enclosure, providing multiple 40 Gigabit Ethernet connections between blade servers and the fabric interconnect, simplifying diagnostics, cabling, and management. It is a third-generation I/O Module (IOM) that shares the same form factor as the second-generation Cisco UCS 2200 Series Fabric Extenders and is backward compatible with the shipping Cisco UCS 5108 Blade Server Chassis. The Cisco UCS 2304 connects the I/O fabric between the Cisco UCS 6300 Series Fabric Interconnects and the Cisco UCS 5100 Series Blade Server Chassis, enabling a lossless and deterministic Fibre Channel over Ethernet (FCoE) fabric to connect all blades and chassis together. Because the fabric extender is similar to a distributed line card, it does not perform any switching and is managed as an extension of the fabric interconnects. This approach reduces the overall infrastructure complexity and enabling Cisco UCS to scale to many chassis without multiplying the number of switches needed, reducing TCO and allowing all chassis to be managed as a single, highly available management domain.

The Cisco UCS 2304 Fabric Extender has four 40Gigabit Ethernet, FCoE-capable, Quad Small Form-Factor Pluggable (QSFP+) ports that connect the blade chassis to the fabric interconnect. Each Cisco UCS 2304 can provide one 40 Gigabit Ethernet ports connected through the midplane to each half-width slot in the chassis, giving it a total eight 40G interfaces to the compute. Typically configured in pairs for redundancy, two fabric extenders provide up to 320 Gbps of I/O to the chassis.

Figure 5       Cisco UCS 2304 Fabric Extender

For more information, see: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-6300-series-fabric-interconnects/datasheet-c78-675243.html.

Cisco Nexus 9000 Switches

The Cisco Nexus 9000 Series delivers proven high performance and density, low latency, and exceptional power efficiency in a broad range of compact form factors. Operating in Cisco NX-OS Software mode or in Application Centric Infrastructure (ACI) mode, these switches are ideal for traditional or fully automated data center deployments.

The Cisco Nexus 9000 Series Switches offer both modular and fixed 10/40/100 Gigabit Ethernet switch configurations with scalability up to 30 Tbps of non-blocking performance with less than five-microsecond latency, 1152 x 10 Gbps or 288 x 40 Gbps non-blocking Layer 2 and Layer 3 Ethernet ports and wire speed VXLAN gateway, bridging, and routing.

Figure 6       Cisco UCS Nexus 9396PX

For more information, see: https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/datasheet-c78-736967.html.

Intel Scalable Processor Family

Intel® Xeon® Scalable processors provide a new foundation for secure, agile, multi-cloud data centers. This platform provides businesses with breakthrough performance to handle system demands ranging from entry-level cloud servers to compute-hungry tasks including real-time analytics, virtualized infrastructure, and high performance computing. This processor family includes technologies for accelerating and securing specific workloads.

·         Intel^® Xeon^® Scalable processors are now available in four feature configurations:

·         Intel^® Xeon^® Bronze Processors with affordable performance for small business and basic storage.

·         Intel^® Xeon^® Silver Processors with essential performance and power efficiency

·         Intel^® Xeon^® Gold Processors with workload-optimized performance, advanced reliability

·         Intel^® Xeon^® Platinum Processors for demanding, mission-critical AI, analytics, hybrid-cloud workloads

Figure 7       Intel^® Xeon^® Scalable Processor Family

Intel^® SSD DC S4500 Series

Intel^® SSD DC S4500 Series is a storage inspired SATA SSD optimized for read-intensive workloads. Based on TLC Intel^® 3D NAND Technology, these larger capacity SSDs enable data centers to increase data stored per rack unit. Intel^® SSD DC S4500 Series is built for compatibility in legacy infrastructures so it enables easy storage upgrades that minimize the costs associated with modernizing data center. This 2.5” 7mm form factor offers wide range of capacity from 240 GB up to 3.8 TB.

Figure 8       Intel® SSD DC S4500

Red Hat OpenShift Container Platform

OpenShift Container Platform is Red Hat’s container application platform that brings together Docker and Kubernetes and provides an API to manage these services. OpenShift Container Platform allows you to create and manage containers. Containers are standalone processes that run within their own environment, independent of operating system and the underlying infrastructure. OpenShift helps developing, deploying, and managing container-based applications. It provides a self-service platform to create, modify, and deploy applications on demand, thus enabling faster development and release life cycles. OpenShift Container Platform has a microservices-based architecture of smaller, decoupled units that work together. It runs on top of a Kubernetes cluster, with data about the objects stored in etcd, a reliable clustered key-value store.

Kubernetes Infrastructure

Within OpenShift Container Platform, Kubernetes manages containerized applications across a set of Docker runtime hosts and provides mechanisms for deployment, maintenance, and application-scaling. The Docker service packages, instantiates, and runs containerized applications.

A Kubernetes cluster consists of one or more masters and a set of nodes. This solution design includes HA functionality at the hardware as well as the software stack. Kubernetes cluster is designed to run in HA mode with 3 master nodes and 2 Infra nodes to help ensure that the cluster has no single point of failure.

Red Hat OpenShift Integrated Container Registry

OpenShift Container Platform provides an integrated container registry called OpenShift Container Registry (OCR) that adds the ability to automatically provision new image repositories on demand. This provides users with a built-in location for their application builds to push the resulting images. Whenever a new image is pushed to OCR, the registry notifies OpenShift Container Platform about the new image, passing along all the information about it, such as the namespace, name, and image metadata. Different pieces of OpenShift Container Platform react to new images, creating new builds and deployments.

Container-native Storage Solution from Red Hat

Container-native storage solution from Red Hat makes OpenShift Container Platform a fully hyperconverged infrastructure where storage containers co-reside with the compute containers. Storage plane is based on containerized Red Hat Gluster^® Storage services, which controls storage devices on every storage server. Heketi is a part of the container-native storage architecture and controls all of the nodes that are members of storage cluster. Heketi also provides an API through which storage space for containers can be easily requested. While Heketi provides an endpoint for storage cluster, the object that makes calls to its API from OpenShift clients is called a Storage Class. It is a Kubernetes and OpenShift object that describes the type of storage available for the cluster and can dynamically send storage requests when a persistent volume claim is generated.

Container-native storage for OpenShift Container Platform is built around three key technologies:

·         OpenShift provides the Platform-as-a-Service (PaaS) infrastructure based on Kubernetes container management. Basic OpenShift architecture is built around multiple master systems where each system contains a set of nodes.

·         Red Hat Gluster Storage provides the containerized distributed storage based on Red Hat Gluster Storage container. Each Red Hat Gluster Storage volume is composed of a collection of bricks, where each brick is the combination of a node and an export directory.

·         Heketi provides the Red Hat Gluster Storage volume life cycle management. It creates the Red Hat Gluster Storage volumes dynamically and supports multiple Red Hat Gluster Storage clusters.

Docker

Red Hat OpenShift Container Platform uses Docker runtime engine for containers.

Kubernetes

Red Hat OpenShift Container Platform is a complete container application platform that natively integrates technologies like Docker and Kubernetes; a powerful container cluster management and orchestration system.

Etcd

Etcd is a key-value store used in OpenShift Container Platform cluster. Etcd data store provides complete cluster and endpoint states to the OpenShift API servers. Etcd data store furnishes information to API servers about node status, network configurations, secrets, etc.

Open vSwitch

Open vSwitch is an open-source implementation of a distributed virtual multilayer switch. It is designed to enable effective network automation through programmatic extensions, while supporting standard management interfaces and protocols such as 802.1ag, SPAN, LACP, and NetFlow. Open vSwitch provides software-defined networking (SDN)-specific functions in the OpenShift Container Platform environment.

HAProxy

HAProxy is open source software that provides a high availability load balancer and proxy server for TCP and HTTP-based applications that spreads requests across multiple servers. In this solution, HAProxy is deployed in virtual machine in VMware HA cluster which provides routing and load-balancing functions for Red Hat OpenShift applications. Other instance of HAProxy acts as an ingress router for all applications deployed in Red Hat OpenShift cluster.

Red Hat Ansible Automation

Red Hat Ansible Automation is a powerful IT automation tool. It is capable of provisioning numerous types of resources and deploying applications. It can configure and manage devices and operating system components. Due to the simplicity, extensibility, and portability, this OpenShift solution is based largely on Ansible Playbooks.

Ansible is mainly used for installation and management of the Red Hat OpenShift Container Platform deployment.

For VMware environment, the installation of Red Hat OpenShift Container Platform is done via the Ansible playbooks installed by the openshift-ansible-playbooks rpm package. In order to deploy 3 masters, 3 infrastructure, and 3 app nodes, Ansible playbooks are utilized. These playbooks can be altered for VM sizing and to meet other specific requirements if needed. Details of these playbooks can be obtained from the following https://github.com/openshift/openshift-ansible-contrib/tree/master/reference-architecture/vmware-ansible/playbooks.

This section provides an overview of the hardware and software components used in this solution, as well as the design factors to be considered in order to make the system work as a single, highly available solution.

Hardware and Software Revisions

Table 1 lists the firmware versions validated in this RHOCP solution.

Table 1      Hardware Revisions

Table 2 lists the software versions validated in this RHOCP solution.

Table 2      Software Versions

Solution Components

This solution is validated comprised of following components (see Table 3).

Table 3      Solution Components

Architectural Overview

Red Hat OpenShift Container Platform is managed by the Kubernetes container orchestrator, which manages containerized applications across a cluster of systems running the Docker container runtime. The physical configuration of Red Hat OpenShift Container Platform is based on the Kubernetes cluster architecture. OpenShift is a layered system designed to expose underlying Docker-formatted container image and Kubernetes concepts as accurately as possible, with a focus on easy composition of applications by a developer. For example, install Ruby, push code, and add MySQL. The concept of an application as a separate object is removed in favor of more flexible composition of "services", allowing two web containers to reuse a database or expose a database directly to the edge of the network.

Figure 9       Architectural Overview

 

This Red Hat OpenShift Container Platform reference architecture contains five types of nodes: bastion, master, infrastructure, storage, and application.

Bastion Node

This is a dedicated node that serves as the main deployment and management server for the Red Hat OpenShift cluster. It is used as the logon node for the cluster administrators to perform the system deployment and management operations, such as running the Ansible OpenShift deployment Playbooks and performing scale-out operations. Also, Bastion node runs DNS services for the OpenShift Cluster nodes. The bastion node runs Red Hat Enterprise Linux 7.5.

OpenShift Master Nodes

The OpenShift Container Platform master is a server that performs control functions for the whole cluster environment. It is responsible for the creation, scheduling, and management of all objects specific to Red Hat OpenShift. It includes API, controller manager, and scheduler capabilities in one OpenShift binary. It is also a common practice to install an etcd key-value store on OpenShift masters to achieve a low-latency link between etcd and OpenShift masters. It is recommended that you run both Red Hat OpenShift masters and etcd in highly available environments. This can be achieved by running multiple OpenShift masters in conjunction with an external active-passive load balancer and the clustering functions of etcd. The OpenShift master node runs Red Hat Enterprise Linux Atomic Host 7.5.

OpenShift Infrastructure Nodes

The OpenShift infrastructure node runs infrastructure specific services: Docker Registry*, HAProxy router, and Heketi. Docker Registry stores application images in the form of containers. The HAProxy router provides routing functions for Red Hat OpenShift applications. It currently supports HTTP(S) traffic and TLS-enabled traffic via Server Name Indication (SNI). Heketi provides management API for configuring GlusterFS persistent storage. Additional applications and services can be deployed on OpenShift infrastructure nodes. The OpenShift infrastructure node runs Red Hat Enterprise Linux Atomic Host 7.5.

OpenShift Application Nodes

The OpenShift application nodes run containerized applications created and deployed by developers. An OpenShift application node contains the OpenShift node components combined into a single binary, which can be used by OpenShift masters to schedule and control containers. A Red Hat OpenShift application node runs Red Hat Enterprise Linux Atomic Host 7.5.

OpenShift Storage Nodes

The OpenShift storage nodes run containerized GlusterFS services which configure persistent volumes for application containers that require data persistence. Persistent volumes may be created manually by a cluster administrator or automatically by storage class objects. An OpenShift storage node is also capable of running containerized applications. A Red Hat OpenShift storage node runs Red Hat Enterprise Linux Atomic Host 7.5.

Table 4 lists the functions and roles for each class of node in this solution for the OpenShift Container Platform.

Table 4      Type of Nodes in OpenShift Container Platform Cluster and their Roles

Physical Topology

Figure 10 shows the physical architecture used in this reference design.

Figure 10     Red Hat OpenShift Platform Architectural Diagram

 

3 x C240s are acting as compute nodes as well as providing persistent storage to containers.

Enterprise storage is providing Boot LUNs to all the servers via the boot policy configuration in Cisco UCS Manager. However, Cisco UCS C240s can also be configured to boot from local disk by configuring storage profiles in Cisco UCS Manager with RAID 1 Mirrored.

Enterprise storage also provides data stores to VMware vSphere environment for VM virtual disks.

Logical Topology

Figure 11 illustrates the logical topology of Red Hat OpenShift Container Platform.

Figure 11     Logical Diagram

·         Private Network – This network is common to all nodes. This is internal network and mostly used by bastion node to perform SSH access for Ansible playbook. Hence, bastion node is acting as a jump host for SSH. Outbound NAT has been configured for nodes to access Red Hat content delivery network for enabling the required repos.

·         Bastion node is a part of public subnet and it can also be configured as IP Router that routes traffic across all the nodes via the control network. In this case default gateway of all the nodes will be bastion node private IP. This scenario would limit outside access for all the nodes if bastion node is powered down

·         OpenShift SDN – OpenShift Container Platform uses a software-defined networking (SDN) approach to provide a unified cluster network that enables communication between pods across the OpenShift Container Platform cluster. This pod network is established and maintained by the OpenShift SDN, which configures an overlay network using Open vSwitch (OVS).

·         Public Network – This network provides internet access.

Virtual Machine Instance Details

Table 5 lists the minimum VM requirements for each node.

Table 5      VM Configurations

Master nodes should contain three extra disks used for Docker storage, etcd, and OpenShift volumes for OCP pod storage. All volumes are thin provisioned. Infra and App nodes does not require etcd. In this reference architecture, VMs are deployed in a single cluster in a single datacenter.

Red Hat OpenShift Container Platform Node Placement

The following diagram shows how the master, infrastructure, and application nodes should be placed in vSphere ESXi hosts.

·         It is recommended to configure anti-affinity rule for master VMs after they are installed from Ansible play-book. Setting up the anti-affinity rule is described in a subsequent section of this document.

·         Within vSphere environment, HA is maintained by VMware vSphere High-availability

HA Proxy Load Balancer

In the reference architecture, HA Proxy load balancer is used. However, on premise existing load balancer can also be utilized. HA Proxy is the entry point for many Red Hat OpenShift Container Platform components. OpenShift Container Platform console is accessible via the master nodes, which is spread across multiple instances to provide load balancing as well as high availability.

Application traffic passes through the Red Hat OpenShift Container Platform Router on its way to the container processes. The Red Hat OpenShift Container Platform Router is a reverse proxy service container that multiplexes the traffic to multiple containers making up a scaled application running inside Red Hat OpenShift Container Platform. The load balancer used by infra nodes acts as the public view for the Red Hat OpenShift Container Platform applications.

The destination for the master and application traffic must be set in the load balancer configuration after each instance is created, the floating IP address is assigned and before the installation. A single haproxy load balancer can forward both sets of traffic to different destinations.

Solution Prerequisites

The following prerequisites must be met before starting the deployment of Red Hat OpenShift Container platform:

·         An active Red Hat account with access to the OpenShift Container Platform subscriptions through purchased entitlements.

·         Fully functional DNS server is an absolute MUST for this solution. Existing DNS server can be utilized as long as it is accessible to and from OpenShift nodes and can provide name resolutions to hosts and containers running on the platform.

·         OpenShift Enterprise requires NTP to synchronize the system and hardware clocks. It prevents master and nodes in the cluster from going out of sync.

·         Network Manager is required on the nodes for populating dnsmasq with the DNS IP addresses.

·         Have the Red Hat Enterprise Linux Server 7.5 ISO image (rhel-server-7.5-x86-64-dvd.iso) on hand and readily available.

·         Pre-existing VMware vSphere and vCenter environment with the capability of vCenter High Availability.

Required Channels

Subscription to the following channels are required. Before you start the deployment, make sure your subscriptions have access to these channels.

Table 6      Red Hat OpenShift Container Platform Required Channels

Deployment Workflow

Figure 12 shows the workflow of deployment process.

Figure 12     Deployment Workflow

DNS (Domain Name Server) Configuration

DNS is a mandatory requirement for successful Red Hat OpenShift Container Platform deployment. It should provide name resolution to all the hosts and containers running on the platform.

Below is the sample DNS configuration for reference:

Figure 13 shows forward lookup zone ocp3.cisco.com and the DNS records used to point to sub-domain to an IP address.

Figure 13     DNS Configuration

Application DNS

Applications served by OpenShift are accessible by the router on ports 80/TCP and 443/TCP. The router uses a wildcard record to map all host names under a specific sub domain to the same IP address without requiring a separate record for each name. This allows Red Hat OpenShift Container Platform to add applications with arbitrary names as long as they are under that sub domain.

For example, a wildcard record for *.apps.ocp3.cisco.com causes DNS name lookups for wordpress.apps.ocp3.cisco.com and nginx-01.apps.ocp3.cisco.com to both return the same IP address: 192.168.91.240. All traffic is forwarded to the OpenShift Routers. The Routers examine the HTTP headers of the queries and forward them to the correct destination.

With a load-balancer host address of 192.168.91.240, the wildcard DNS record can be added as follows:

Table 7      Load Balancer DNS records

Figure 14 shows wildcard DNS record that resolves to the IP address of the OpenShift router.

Figure 14     DNS Wildcard Record

VMware vCenter Prerequisites

This reference architecture assumes a pre-existing VMware vCenter environment and is configured based on the best practices for the infrastructure.

VMware HA and storage IO control should already be configured. Once the environment is setup, anti-affinity rules are recommended to be setup for maximum uptime and optimal performance.

Networking

An existing port group and virtual LAN (VLAN) are required for deployment. The initial configuration of the Red Hat OpenShift nodes in this reference architecture assumes you will be deploying VMs to a port group called "VM Network".

The environment can utilize a VMware vSphere Distributed Switch (VDS) or Standard vSwitch. The specifics of that are unimportant and beyond the scope of this document. However, a vDS is required if you wish to utilize network IO control and some of the quality of service (QoS) technologies.

Furthermore, specifics of setting up the VMware environment on Cisco UCS is not covered in this document such as physical NICs, storage configuration whether IP based or Fibre channel, UCS vNIC failover and redundancy, vNIC templates, service profiles, other policies, and so on. These design choices and scenarios have been previously validated in various Cisco Validated Designs. For best practices and supported design, see:

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_esxi65u1_n9kiscsi.html?referring_site=RE&pos=3&page=https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_esxi65u1design.html

The figure below shows the standard vSwitch configuration used in this reference architecture. After the VM deployment is completed from Ansible playbook (described in a subsequent section), vSphere vDS can be configured, details of which are outlined in the link above.

vCenter Shared Storage

All vSphere hosts should have shared storage to provision VMware virtual machine disk files (VMDKs) for templates. It is also recommended to enable storage I/O control (SIOC) to address any latency issues caused by performance. For in-depth overview of storage IO control, see: https://kb.vmware.com/s/article/1022091.

vSphere Parameter

Table 8 lists the vSphere parameter required for inventory file discussed in later section of this document. These parameter must be handy and pre-configured in vCenter environment before preparing the inventory file.

Table 8      vSphere Parameter

Resource Pool, Cluster Name, and Folder Location

This reference architecture assumes some default names as per the playbook used to deploy the cluster VMs. However, it is recommend to follow the information mentioned in Table 8.

Create the following per Table 8:

1.    Create a resource pool named: "ocp3"

2.    Create a folder for the Red Hat OpenShift VMs for logical organization named: "ocp"

3.    Make sure this folder exists under the datacenter and cluster you will use for deployment

If you would like customize the names, remember to specify them later while creating the inventory file.

Prepare RHEL VM Template

This reference architecture is based on RHEL 7.5. VM template and needs to be prepared for use with the RHOCP instance deployment.

To prepare the RHEL VM template, complete the following steps:

1.    Create a Virtual machine using RHEL 7.5 iso. You can create the VM with 2 vCPU, 4GB RAM, and 50GB disk for RHEL 7.5 OS. Specify net.ifnames=0 biosdevname=0 after pressing tab key as boot parameter to get consistent naming for network interfaces.

2.    Power-On VM, configure /etc/sysconfig/network-script/ifcfg-eth0 with IP address, Netmask, Gateway, and DNS. Restart networking by running the following command.

3.    Register VM with Red Hat OpenShift subscription and enable required repositories:

4.    Install following packages:

5.    Unregister the VM:

6.    Follow the steps mentioned in https://access.redhat.com/solutions/198693 to create clean VM for use as a template or cloning.

7.    Change the /etc/sysconfig/network-script/ifcfg-eth0 with the following:

8.    Run the following:

9.    Power-off the VM and convert it into template by Right click VM > Template > Convert to Template.

Setting Up Bastion Instance

The Bastion host serves as the installer of the Ansible playbooks that deploy Red Hat OpenShift Container Platform as well as an entry point for management tasks.

Bastion instance is a non-OpenShift instance accessible from outside of the Red Hat OpenShift Container Platform environment, configured to allow remote access via secure shell (ssh). To remotely access an instance, the systems administrator first accesses the bastion instance, then "jumps" via another ssh connection to the intended OpenShift instance. The bastion instance may be referred to as a "jump host".

To set up the bastion instance, complete the following steps:

1.    Deploy a VM from Rhel75 template. Customize the virtual machine’s hardware as shown below based on the requirements mentioned in Table 1.

Figure 15     Deploy Bastion Instance from Template

2.    After the VM is deployed, configure eth0 interface as shown below:

3.    Set the host name:

4.    Register Bastion instance with Red Hat OpenShift subscription and enable required repositories:

5.    Install atomic-openshift-util package via following command:

Configure Ansible

Ansible is installed on the deployment instance to perform the registration, installation of packages, and the deployment of the Red Hat OpenShift Container Platform environment on the master and node instances.

Before running playbooks, it is important to create an ansible.cfg to reflect the deployed environment:

Prepare Inventory File

Ansible Playbook execution requires an inventory file to perform tasks on multiple systems at the same time. It does this by selecting portions of systems listed in Ansible’s inventory, which defaults to being saved in the location /etc/ansible/hosts on the Bastion node for this solution. This inventory file consists of Hosts and Groups, Host Variables, Group Variables and behavioral Inventory Parameters.

This inventory file is customized for the Playbooks we used in this solution for preparing nodes and installing OpenShift Container Platform. The inventory file also describes the configuration of our OCP cluster and include/exclude additional OCP components and their configurations.

Table 9 table lists the sections used in the inventory file.

Table 9       Inventory File Sections and their Descriptions

Table 10 lists the inventory key variables used in this solution and their descriptions.

Table 10   Inventory Key Variables Used in this Solution and Descriptions

Below is the sample inventory file used in this reference architecture:

For more information about various OpenShift environment variables, see: https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/installation_and_configuration/#configuring-ansible

Red Hat OpenShift Container Platform Instance Creation

In this section, the following instance will be created:

·         1 x bastion, 3 x masters, 3 x infra, 3 x app, and 1 x haproxy

etcd requires that an odd number of cluster members exist. For collocating etcd with master nodes, three masters were chosen to support high availability and etcd clustering. Three infrastructure instances allow for minimal to zero downtime for applications running in the OpenShift environment. Applications instance can be one to many instances depending on the requirements of the organization.

Run the following command:

For haproxy instance, run the following command:

Below is the summary of the Red Hat OpenShift Platform vSphere instance creation playbook.

1.    Register with Red Hat subscription manager.

2.    Enable the required repositories and disable the repositories that should not be enabled.

3.    Setup vmware guest instance by installing prerequisites packages, such as open-vm-tools, PyYAML, perl, net-tools, chrony, python-six, iptables, iptables-services, and docker.

4.    Create a docker storage setup file as follows and start docker service:

5.    A VMDK volume should be created for the directory of /var/lib/origin/openshift.local.volumes that is used with the perFSGroup setting at installation and with the mount option of gquota. These settings and volumes set a quota to make sure that the containers cannot grow to an unreasonable size. Create openshift-volume-quota by creating filesystem for /var/lib/origin/openshift.local.volumes

6.    Create fstab entry as shown below and mount the fstab entry:

To create the master prerequisites, complete the following steps. (These are only performed on master nodes.)

1.    Install git.

2.    Setup etcd storage. A VMDK volume should be created on the master instances for the storage of /var/lib/etcd. Storing etcd allows the similar benefit of protecting /var but more importantly provides the ability to perform snapshots of the volume when performing etcd maintenance. Create lvm volume and create local partition on lvm lv.

3.    Create the following fstab entry and mount the fstab:

Figure 16 shows the VM instances created as a result of ansible playbook.

Figure 16     RHOCP Instances

Setup DRS Anti-Affinity Rules

Create a DRS anti-affinity rules to ensure maximum availability for the cluster before you begin the deployment. To setup DRS anti-affinity rules, complete the following steps:

1.    Open the VMware vCenter web client, select the cluster. Click Configure tab and select VM/Host Rules as shown.

Figure 17     Configure Cluster for VM/Host Rules

2.    Click + Add to Create VM/Host Rule

3.    Type Rule name such as master-away in this case. Make sure rule is enabled. Click + Add and select master-0, master-1, master-2 from Add Virtual Machine pop-up. Click OK to create the rule.

Figure 18     Create VM/Host Rule

Configure VM Latency Sensitivity

Configure all of the VMs created to High VM Latency ass recommended by VMware for latency sensitive workloads.

To configure VM Latency Sensitivity, complete the following steps:

1.    Open the VMware vCenter web client. Right-click VM and select Edit Settings.

2.    Select VM Options tab in Edit Settings window.

3.    Under VM Options tab, expand Advanced.

4.    Select the 'Latency Sensitivity' dropdown and select 'High' as shown in below Figure. Click OK.

Figure 19     VMware High Latency

Red Hat OpenShift Platform Storage Node Setup

In this section, three storage nodes will be setup in Cisco UCS C240M5 Servers.

Creating Storage Profile

Storage Profiles provide a systematic way to automate the steps for provisioning Disk Groups, RAID Levels, LUNs, boot drives, hot spares, and other related resources. They are used in combination with Service Profile Templates to map the associations between logically defined storage resources and servers.

Having a Storage Profile created will reduce the task of configuring two virtual disks in the RAID Controller Option ROM or create a custom file system layout at the time of OS installation.

In this reference architecture, we used C240M5 servers for storage nodes. Cisco UCS C240 M5 servers support 26 drive bays. We populated 20 Intel S4500 Series SSDs to present them as JBOD disks in the solution.

We created separate storage profiles C-Series nodes.

A Storage Profile is created with two local LUNs one each for boot and data. To create a storage profile, complete the following steps:

1.    Click Storage in the left Navigation pane.

2.    Create a disk group policy by right clicking Storage Policies > root > Disk Group Policy.

3.    Enter the Disk Group name (for example, OCP-Glstr-Boot). Keep the RAID Level as RAID 1 Mirrored.

4.    Select Disk Group Configuration (Manual) radio button. Click + to add two slots; slot 25 and 26.

5.    Click OK.

Figure 20     Create Disk Group Policy

6.    Select Storage Profiles.

7.    Right-click Storage Profiles and select Create Storage Profile.

8.    Enter the name for the Storage Profile (for example, OCP-Glstr for Rack servers C240M5) as shown.

Figure 21     Create Storage Profile – OCP-Glstr

9.    Click Add to in the Local LUN tab. Select Create Local LUN radio button. Specify the LUN name for example Boot-LUN, size 50GB, and select OCP-Glstr-Boot from Select Disk Group Configuration drop down as shown.

10.  Click OK to create Boot-LUN.

Figure 22     Create Local LUN

11.  Repeat step 9 to create Docker LUN and OCP-LUN. Final output will look like as shown below.

12.  Click OK to create storage profile with three LUNs; Boot, Docker, and OpenShift local.

Figure 23     Create Local LUN

Boot Policy for Storage Node

To create a boot policy for storage node, complete the following steps:

1.    Click Servers in the left navigation pane. Expand Policies > root > Boot Policies. Right-click Boot Policies.

2.    Provide a boot policy name (for example. Local-lun-boot).

3.    Add Local LUN and Local CD/DVD in the boot order as shown below.

4.    Click OK to create the boot policy.

Figure 24     Create Boot Policy – Storage Nodes

Service Profile Template for Storage Nodes

If Storage profile is used for booting the storage nodes, storage nodes service profile template, then Storage provisioning should be configured with storage profile as shown in Figure 25.

Figure 25     Service Profile Template – Storage Provisioning

Installation of Red Hat Enterprise Linux Operating System in Storage Nodes

To install the Operating System on the storage nodes, complete the following steps:

1.    Download Red Hat Enterprise Linux 7.5 from http://access.redhat.com.

2.    Complete the following to install Red Hat Enterprise Linux 7.5 OS.

3.    From Cisco UCS Manager, click Servers > Service Profiles > Storage-01

4.    Launch the KVM console from General tab. Click >> to launch Java JVM Console. Click OK.

5.    In the KVM Console, select Activate Virtual Devices by clicking the Virtual Media drop-down list.

6.    When the virtual devices get activated, select Map CD/DVD from the Virtual Media drop-down list.

7.    Click Browse in the Virtual Media – Map CD/DVD pop-up window.

8.    Locate the Red Hat Enterprise Linux Server 7.5 installer ISO image file and click Map Device as shown.

9.    The image gets mapped to CD/DVD.

10.  Click the Reset on the menu bar. Click Power Cycle. Click OK.

11.  In KVM window, monitor the reboot. On reboot, the machine detects the presence of the Red Hat Enterprise Linux Server 7.5 install media. At the prompt press F6 to select the Boot Device. Select Cisco vKVM-Mapped vDVD and press Enter key.

12.  Server will detect the installation media of RHEL 7.5

13.  Press tab key for full configuration

14.  Type the following and press the Enter key to proceed to installation.

15.  Select Language and click Continue.

16.  Click DATE & TIME.

17.  Select Region and City in DATE & TIME screen and click Done.

18.  Click INSTALLATION DESTINATION. Select boot LUN. Click Done.

19.  Click Done on device selection for installation destination.

20.  Click Done and click Begin Installation.

21.  While the installation in is progress, click ROOT PASSWORD to assert a password and then click User Creation to set user credentials.

22.  When the installation is complete, click Reboot.

Configure Storage Node Interfaces for RHOCP

To configure the storage node interfaces for RHOCP, complete the following steps:

1.    Add vlan tag interface in all three storage nodes as below. Assign the respective IP addresses to each storage node:

2.    Interface configuration will look like as below:

3.    Configure host name for storage nodes by running the following command:

Creating an SSH Keypair for Ansible

After all the VMs and Cisco UCS C240 Servers are deployed successfully. The VMware infrastructure requires an SSH key on the VMs for Ansible’s use.

In the bastion instance, complete the following steps:

1.    Generate a new SSH key to be used for authentication.

2.    Add the ssh keys to all the deployed virtual machines and storage bare-metal nodes via ssh-copy-id or to the template prior to deployment as shown below:

3.    Verify ssh by the running the following command for Ansible. Ensure all virtual nodes (master, infra, and app) and physical storage nodes are accessible via Ansible through fully qualified domain names.

4.    Make sure there is connectivity to all instances via bastion node as follows:

Configure and Install Prerequisites for Storage Nodes

To verify the following prerequisites in storage nodes before proceeding for RHOCP deployment, complete the following steps:

1.    Verify storage nodes are accessible via Ansible.

2.    Register storage nodes with Red Hat subscription and enable required repositories

3.    Install prerequisites packages such as PyYAML, perl, net-tools, chrony, python-six, iptables, iptables-services, docker:

4.    Create a docker-storage-setup file in bastion node with the following contents and copy it in /etc/sysconfig/ in all storage nodes:

5.    Setup OpenShift local volume:

6.    Create an fstab entry for OpenShift local volume:

7.    Mount the file system mentioned in fstab:

8.    Start and verify the docker service:

Instance Verification

It can be useful to check for potential issues or misconfigurations in the instances before continuing the installation process. Connect to every instance using the deployment host and verify the disks are properly created and mounted:

Where the instance is master-0.ocp3.cisco.com

For reference, the following is an example output of lsblk for the master nodes:

For reference, the following is an example of output of lsblk for the infra and app nodes:

Red Hat OpenShift Container Platform Prerequisites Playbook

The Red Hat OpenShift Container Platform Ansible installation provides a playbook to ensure all prerequisites are met prior to the installation of Red Hat OpenShift Container Platform. This includes steps such as registering all the nodes with Red Hat Subscription Manager and setting up the docker on the docker volumes. The playbook also skips if anything is found already configured and installed. Playbook will also report any error if occurs.

Run the prerequisites ansible-playbook on the bastion node to help ensure all the prerequisites are met using prerequisites.yml playbook as shown in Figure 26:

Figure 26     Prerequisites Playbook

Deploying Red Hat OpenShift Container Platform

With the prerequisites met, the focus shifts to the installation of Red Hat OpenShift Container Platform. The installation and configuration is done via a series of Ansible playbooks and roles provided by the OpenShift RPM packages.

Deploy Red Hat OpenShift Container Platform by running the installer playbook as shown below:

The playbook will run through several tasks while installing Red Hat OpenShift Container Platform. It will report any errors if occur. Figure 27 shows the successful output of the playbook. For detailed information about the tasks, use the -vvv option as shown below:

Figure 27     Deploy Red Hat OpenShift Container Platform Playbook Output

Functional Validation

In this section, you will go through a series of deployment verification tasks; these include both UI and CLI tasks. After the installation has completed successfully, complete the following steps:

1.    Verify if the master is started and nodes are registered and reporting `Ready` status. On one of the master node, run the following as root:

2.    Verify services:

3.    Verify hxproxy-0 instance is up and running and /etc/haproxy/haproxy.cfg file matches with the following and haproxy service is running before validating web console. It is also important to note that openshift.ocp3.cisco.com is resolving to haproxy IP address in DNS records.

4.    Verify if the web console is installed correctly. Use `openshift_master_cluster_public_hostname` value, as defined in the inventory file, with the URL for web-console access: https://openshift.ocp3.cisco.com:8443.

Figure 28     Red Hat OpenShift Container Platform Web Console

5.    Login by providing Username and Password.

Figure 29     OpenShift Container Platform – Browser Catalog

6.     Verify for OpenShift Integrated Container Registry console access, by clicking the registry console URL as displayed on dashboard under default project:

Figure 30     OpenShift Integrated Container Registry Console

7.    Verify the etcd cluster health and membership status. Login to one of the master node and run following commands:

Figure 31 shows the healthy status of the etcd cluster.

Figure 31     ETCD Cluster Health

8.    Verify the member list by running the following command:

Figure 32 shows the etcd member list and mentions which node is participating as a leader.

Figure 32     ETCD Member List

9.    Follow these instructions to setup OpenShift client “oc” on the bastion instance and use this client to further validate the deployment:

a.     Download and install OpenShift v3.9 Linux Client from this URL: https://access.redhat.com/downloads/content/290

b.    Once downloaded extract the .tar file and also copy it in /usr/bin and /usr/sbin folder as shown below.

c.     Access OpenShift Web Console and click `?` on top right corner to get the authentication token as shown below.

Figure 33     OpenShift Web Console - Command Line Tools

d.    Copy login command with the token from the screenshot shown below:

Figure 34     Command Line Tools – Login Command Copy to Clipboard

e.    Use copied command on the Bastion node to login to OpenShift Container cluster:

f.      Login through cli client `oc` can be done without having to supply authentication token by using `insecure` connection as below. Here we need to provide username and password to login.

g.    Verify container native storage PODs and other resources are running:

 

h.     Verify Gluster cluster status and storage resource health by logging in to one of the glusterfs-storage pods:

i.      Verify Router and Registry Health, value in the DESIRED and CURRENT field should match:

j.      Verify PODs are distributed on desired Infra nodes:

Sample Application Test Scenario.

In this section, you will deploy a sample application to validate the complete environmental health of the deployment.

With the OpenShift “oc” client, complete the following steps:

1.    Create a new project named “sample-test” as shown below:

2.    Deploy a sample application `cakephp-mysql-example`:

3.    After the successful build, we should see two POD running i.e. application POD and database POD as shown below.

4.    Verify to access the application via route 'cakephp-mysql-example-sample-test.apps.ocp3.cisco.com' by copy and paste in the browser as shown below.

Figure 35     Access Application via Route cakephp-mysql-example-sample-test.apps.ocp3.cisco.com

Web Console – UI Operations

To create a project and deploy a sample application from the service catalog, complete the following steps:

1.    Select previously created project or create a new one. For example demo project under project-list

2.    Create storage by clicking Storage in the left pane and Create Storage button on the right pane. Select Storage Class, name of the PV claim, and size as shown below:

Figure 36     Create Persistent Storage

3.    Browse Catalog by clicking Catalog in the left pane for MySQL. Click MySQL.

Figure 37     Browse Catalog for MySQL

4.    MySQL Information screen will show up as shown below. Click Next

Figure 38     MySQL - Information

5.    Specify database service name for example mysql in this case. Click Create

Figure 39     MySQL - Configuration

6.    Click Close.

Figure 40     MySQL-Results

7.    Verify persistent volume claim in mysql pod.

Figure 41     Pods > mysql – Persistent Volume Claim

8.    Browse Catalog for PHP application. Click PHP.

Figure 42     Catalog - PHP

9.    Click Next on PHP information screen as shown below

Figure 43     PHP - Information

10.  Specify the PHP application name, for example wordpress, in this case and git repository as https://github.com/wordpress/wordpress. Click Create.

Figure 44     PHP Application Configuration

11.  Wordpress PHP application is created as show below. Click Close.

Figure 45     WordPress PHP Application Results

12.  Verify the Build process is completed as status shows “Running” as shown below.

Figure 46     WordPress Application

As shown in Figure 46, the PHP pod is provisioned in storage node, such as storage-03.ocp3.cisco.com. As mentioned previously in this document, Cisco C240 M5 nodes are not only providing GlusterFS, it is also acting as compute nodes to provision pods. In this reference architecture, pods can be provisioned in virtualized nodes as well as in bare-metal nodes.

13.  Launch wordpress by clicking the Routes that is exposing this service as shown below:

Figure 47     Wordpress - Routes

14.  Go to the Terminal of mysql to capture the mysql user and password that would be needed to configure and install wordpress. This information is also available in the Results screen while create mysql pod.

Figure 48     MySQL – Capture MySQL user and password

15.  Configure wordpress for installation as shown below:

Figure 49     WordPress Configuration

16.  Install wordpress by providing all required fields such as Site Title, Username, Password, and email address.

Figure 50     WordPress Information for Installation

17.  You will receive a successful installation notice and you are ready to login. Click Login and provide the credentials used in setting up the installation in step 15. Figure 51 shows the WordPress Dashboard.

Figure 51     WordPress Dashboard

18.  On CLI using OpenShift Client “OC”, get the application pod and pvc status verified as shown below:

 

 

Scale the Environment

To add more master or app/infra nodes in the cluster, complete the following steps:

1.    Add DNS record for the newly added node.

Figure 52     DNS Record for New Host

2.    Create an inventory file for adding new nodes to be used by prod.yaml. This step provision VM instances for new hosts. You can scale to more than one node at the same time by specifying the nodes. This file can be created by copying the /etc/ansible/hosts file and update accordingly as shown in the below example. Important entries have been highlighted in bold.

3.    Run the prod.yaml file as shown below:

4.    Verify VM in vCenter.

Figure 53     App-3 Instance Verification

5.    Edit your /etc/ansible/hosts file and add new_<host_type> to the [OSEv3:children] section:

6.    To add new master hosts, add new_masters.

7.    Create a [new_<host_type>] section much like an existing section, specifying host information for any new hosts you want to add. For example, when adding a new node:

8.    When adding new masters, hosts added to the [new_masters] section must also be added to the [new_nodes] section. This helps ensure the new master host is part of the OpenShift SDN.

9.    Run the scaleup.yml playbook. If your inventory file is located somewhere other than the default of /etc/ansible/hosts.

10.  For additional nodes:

11.  For additional masters:

Figure 41 shows the successful output of scaleup.yml

Figure 54     Output of scaleup.yml

12.  Verify the installation. ssh to one of the master nodes and run the following command.

13.  Finally, move any hosts you had defined in the [new_<host_type>] section into their appropriate section (but leave the [new_<host_type>] section definition itself in place) so that subsequent runs using this inventory file are aware of the nodes but do not handle them as new nodes. For example, when adding new nodes

Add Metrics to the Installation

1.    After the installation has been completed, metrics can be added via the following process:

2.    Run the following playbook:

Add Logging to the Installation

To add logging, complete the following steps:

1.    Configure /etc/ansible/hosts file with the following:

2.    Run the following playbook to add logging:

Cisco UCS Infrastructure for Red Hat OpenShift Container Platform Deployment Guide: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/ucs_openshift.html

Deploying and Managing OpenShift 3.9 on VMware vSphere: https://access.redhat.com/documentation/en-us/reference_architectures/2018/html-single/deploying_and_managing_openshift_3.9_on_vmware_vsphere/index

FlexPod Datacenter with VMware vSphere 6.5 Design Guide: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_esxi65design.html

OpenShift:  https://www.openshift.com/

Red Hat OpenShift Container Platform 3.9 Architecture: https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/architecture/

Day Two Operations:  https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/day_two_operations_guide/

CLI Reference: https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html-single/cli_reference/

Red Hat OpenShift: https://www.redhat.com/en/technologies/cloud-computing/openshift

Solution involving Cisco UCS Infrastructure and Red Hat OpenShift Container Platform with container-native storage has been created to deliver a production-ready foundation that simplifies the deployment process, shares the latest best practices, and provides a stable, highly available environment to run your containerized production applications. This reference architecture covers the process of provisioning and deploying a highly available Red Hat OpenShift Container Platform cluster on a VMware vSphere private cloud environment with both the registry and the application pods backed by container-native storage solution from Red Hat.

For enterprise IT, this solution provides a quick and easy journey to DevOps and CI/CD model for application development to address immediate business needs and reducing time to market. Enterprises can accelerate on the path to an enterprise-grade Kubernetes solution with Red Hat OpenShift Container Platform running on Cisco UCS infrastructure.

Muhammad Afzal, Engineering Architect, Cisco UCS Solutions Engineering, Cisco Systems, Inc.

Muhammad Afzal is an Engineering Architect and Technical Marketing Engineer at Cisco Systems in Cisco UCS Product Management and Datacenter Solutions Engineering. He is currently responsible for designing, developing, and producing validated converged architectures while working collaboratively with product partners. Previously, Afzal had been a lead architect for various cloud and data center solutions in Solution Development Unit at Cisco. Prior to this, Afzal has been a Solutions Architect in Cisco’s Advanced Services group, where he worked closely with Cisco's large enterprise and service provider customers delivering data center and cloud solutions. Afzal holds an MBA in Finance and a BS in Computer Engineering.

Acknowledgements

·         Vishwanath Jakka, Cisco Systems, Inc.

·         Babu Mahadevan, Cisco Systems, Inc.

·         Antonios Dakopoulos, Red Hat

·         Chris Morgan, Red HatNOTE: Available paragraph styles are listed in the Quick Styles Gallery in the Styles group on the Home tab. Alternatively, they can be accessed via the Styles window (press Alt + Ctrl + Shift + S).