Cisco UCS Integrated Infrastructure for Big Data and Analytics with Hortonworks Data Platform 3.0

Design and Deployment Guide of Cisco Integrated Infrastructure for Big Data with Hortonworks Data Platform 3.0 and Cisco UCS C480 M5L Platform

Last Updated: October 21, 2019

NOTE: Works with document’s Advanced Properties “Last Updated” property. Click File | Properties | Advanced Properties | Custom.

 

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to:

http://www.cisco.com/go/designzone.

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS.  CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.  IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE.  USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS.  THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS.  USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS.  RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2019 Cisco Systems, Inc. All rights reserved.

 

Table of Contents

Executive Summary. 7

Solution Overview.. 8

Introduction. 8

Caveats and Limitations. 8

Audience. 9

Purpose of this Document. 9

Solution Summary. 9

Scaling the Solution. 12

Technology Overview.. 13

Cisco UCS Integrated Infrastructure for Big Data and Analytics. 13

Cisco Unified Computing System.. 13

Cisco UCS 6300 Series Fabric Interconnects. 13

Cisco UCS C-Series Rack-Mount Servers. 13

Cisco UCS Virtual Interface Cards. 18

Cisco UCS Manager. 18

NVIDIA GPU.. 19

NVIDIA CUDA.. 19

Hortonworks Data Platform.. 19

Apache Ambari 19

HDP for Data Access. 20

Docker Containerization. 21

YARN Support For Docker. 21

NVIDIA Docker. 22

GPU Pooling and Isolation. 23

Red Hat Ansible Automation. 23

Solution Design. 25

Requirements. 25

Rack and PDU Configuration. 25

Cabling for Cisco UCS C240 M5. 25

Software Distributions and Versions. 26

Hortonworks Data Platform (HDP 3.0.1) 26

Red Hat Enterprise Linux (RHEL) 26

Software Versions. 26

Fabric Configuration. 27

Perform Initial Setup of Cisco UCS 6332 Fabric Interconnects. 28

Configure Fabric Interconnect A.. 28

Configure Fabric Interconnect B.. 29

Log Into Cisco UCS Manager. 29

Upgrade Cisco UCS Manager Software to Version 4.0(2a) 29

Add a Block of IP Addresses for KVM Access. 30

Enable Uplink Ports. 31

Configure VLANs. 32

Enable Server Ports. 34

Create Pools for Service Profile Templates. 35

Create an Organization. 35

Create MAC Address Pools. 36

Create a Server Pool 38

Create Policies for Service Profile Templates. 40

Create Host Firmware Package Policy. 40

Create QoS Policies. 41

Create the Local Disk Configuration Policy. 43

Create the Server BIOS Policy. 43

Create the Boot Policy. 46

Create Power Control Policy. 48

Create Server BIOS Policy. 50

Create Service Profile Template. 52

Configure the Storage Provisioning for the Template. 54

Configure Network Settings for the Template. 54

Configure the vMedia Policy for the Template. 59

Configure the Server Boot Order for the Template. 60

Configure the Server Assignment for the Template. 61

Configure the Operational Policies for the Template. 62

Install Red Hat Enterprise Linux 7.5. 64

Post OS Install Configuration. 88

Configure /etc/hosts. 88

Set Up the Passwordless Login. 89

Create the Red Hat Enterprise Linux (RHEL) 7.5 Local Repository. 90

Create the Red Hat Repository Database. 91

Set Up Ansible. 92

Install httpd. 94

Set Up All Nodes to Use the RHEL Repository. 94

Upgrade the Cisco Network Driver for VIC1387. 95

Install xfsprogs. 95

Set Up JAVA.. 96

Configure NTP. 97

Enable Syslog. 99

Set ulimit. 99

Disable SELinux. 100

Set TCP Retries. 100

Disable the Linux Firewall 101

Disable Swapping. 101

Disable Transparent Huge Pages. 101

Disable IPv6 Defaults. 102

Configure Data Drives on Name Node and Other Management Nodes. 102

Configure Data Drives on Data Nodes. 105

Configure the Filesystem for NameNodes and Datanodes. 106

Cluster Verification. 107

Install HDP 3.0.1. 110

Prerequisites for HDP Installation. 111

Hortonworks Repository. 111

Downgrade Snappy on All Nodes. 113

HDP Installation. 113

Install and Setup Ambari Server on rhel1. 113

Install PostgreSQL in rhel2. 115

Setup Ambari Server On Admin Node(Rhel1) 119

Launch the Ambari Server. 120

Create the Cluster. 121

Select Version. 122

Select Hosts. 123

Hostname Pattern Expressions. 125

Confirm Hosts. 126

Choose Services. 127

Assign Masters. 128

Assign Slaves and Clients. 128

Customize Services. 129

HDFS. 133

MapReduce2. 135

YARN.. 135

HBase. 136

Zookeeper. 136

Storm.. 137

Ambari Metrics. 137

Accumulo. 138

Atlas. 138

Kafka. 138

Knox. 138

SmartSense. 139

Spark. 139

Review.. 140

Deploy. 140

Summary of the Installation Process. 141

High Availability for HDFS NameNode and YARN ResourceManager. 142

Configure the HDFS NameNode High Availability. 142

Configure the YARN ResourceManger HA.. 150

HDP Post OS Deployment – Enable GPU Isolation and Scheduling. 154

Install the Prerequisites for CUDA.. 155

Install CUDA.. 160

Download and Setup NVIDIA CUDA Deep Neural Network library (cuDNN) 161

Install and Configure Docker. 163

Prerequisites. 163

Install Docker. 164

Install nvidia-docker in GPU Nodes. 164

Configure Docker. 166

Configure YARN to Running Docker Containers. 167

Enable Cgroups. 168

Run Docker on YARN Using the YARN Service API 173

Run Docker Container with GPU on YARN.. 176

Prerequisites. 176

Enable GPU through Ambari 176

Verify YARN Configurations. 184

Setting Up Docker Registry. 187

Apache Hadoop YARN Distributed Shell 188

Run TensorFlow Container Using YARN Distributed Shell 190

Bill of Materials. 195

About the Authors. 201

Acknowledgements. 201

 

 

Years ago, most enterprises relied on batch pro­cessing to unlock the value of big data and achieved intelligence. As data size and data sources grew, it also drove a demand for real-time analysis which was catered by Spark and Spark streaming for business-critical decisions. In today’s digital world where sensors, Internet of Things (IoT) devices, social media, and online transactions are generating enormous amount of data and companies are struggling to find ways how to better process this data to generate insights and innovation. This drove the next generation of Hadoop which enables Artificial Intelligence and Machine Learning (AI/ML) with deep learning allowing deep learning frameworks and faster compute with CPUs and GPUs to solve problems driving the next generation of platforms.

The emerging trend of Artificial Intelligence requires large amounts of data for its training and Hadoop is a natural fit for storing and retrieving these large amounts of data.  Many machine learning tasks especially, deep learning requires the use of GPUs, a specialized, very high-performance processor that is massively parallel in nature. This solution focuses on Hadoop accelerating AI natively where GPUs are part of a Hadoop cluster and are natively scheduled by Hadoop schedulers to process massive amounts of data stored in the same cluster.

Building next-generation big data architecture requires sim­plified and centralized management, high performance, and a linearly-scaling infrastructure and software platform. Cisco UCS Integrated Infrastructure for Big Data and Analytics with Hortonworks Data Platform (HDP) is an optimal choice where world class performance and reliability are base requirements. It is the strong foundation upon which solutions are built. The Cisco UCS reference architecture has been designed to scale from a small starting solution to thousands of servers and hundreds of petabytes of storage with ease, and all managed from a single pane of glass. Cisco UCS reference architecture also provides the customer with the flexibility an AI workload demands proving choice of server with 2,4,6 or 8 GPUs in a single server catering to different AI compute requirements.

The Hortonworks Data Platform (HDP) is the industry’s enterprise-ready open-source Apache Hadoop framework for distributed storage and processing of large datasets. HDP 3.0 bring the capability of managing and scheduling GPUs, Docker containers into Hadoop enabling AI workloads natively in Hadoop. HDP also enables agile application deployment in a containerized micro-services architecture. Containerization makes it possible to run multiple versions of applications for AI/ML/DL workloads. Furthermore, HDP also supports third-party applications in Docker containers and native YARN containers.

Introduction

Both big data and machine learning technology have progressed to the point where they are being implemented in production systems running 24x7. There exists a very clear need for a proven, dependable, high-performance platform for the ingestion, processing, storage and analysis of the data, as well as the seamless dissemination of the output, results and insights of the analysis.

This solution implements the Cisco UCS Integrated Infrastructure for Big Data and Analytics, a world-class platform specifically designed for demanding workloads that is both easy to scale and easy to manage, even as the requirements grow to thousands of servers and petabytes of storage; and Hortonworks Data Platform, an integrated set of tools designed to enable flexible, fast access to the entire data store, while enabling AI workloads on servers with GPUs.

This CVD implements the following:

·         Hortonworks Data Platform 3.0 on Cisco UCS Integrated Infrastructure for Big Data and Analytics

·         Install and Enable Docker to be used by YARN 2.0

·         Enable CUDA for the GPUs

·         Enable GPU as a resource to the Docker Containers through NVIDIA-docker v1

·         Enable GPU isolation and scheduling (with Docker Containers) through YARN 2.0

·         Downloading a TensorFlow image from NVIDIA Cloud (NGC)

·         Adding trusted registries for Docker for YARN 2.0

·         Execute a sample TensorFlow job accessing data from Hadoop and running on a Docker container with GPU as a resource scheduled by YARN 2.0

·         Installation and setup of the above through Apache Ambari

Caveats and Limitations

The following is beyond the scope of this CVD and therefore is not addressed.

Docker Networking

YARN does not manage Docker networks or Docker multi-host networking. In the implementation of this CVD, container(s) in one host cannot communicate with container(s) in another host. 

·         Docker Swarm and other docker container orchestration such as Kubernetes are not supported by HDP as they might be competing with YARN

·         This document considers TensorFlow or other AI applications in only standalone container spawned and scheduled through YARN 2.0. Distributed TensorFlow and other framework support is expected in later release and will be added as part of addendum

NVIDIA-Docker

·         As of this release of Hortonworks and this CVD, only nvidia-docker v1 is supported with HDP 3.0

Many companies recognizing the immense potential of big data and AI/ML technology, are gearing-up to leverage these new capabilities, building-out departments and increasing hiring. However, these efforts face a new set of challenges:

·         Making the data available to the diverse set of people who need it

·         Enabling access to high-performance computing resources, GPUs, that also scale with data growth

·         Allowing people to work with the data using the environments in which they are familiar

·         Publishing their results so the organization can make use of it

·         Enabling the automated production of those results

·         Managing the data for compliance and governance

·         Scaling the system as the data grows

·         Managing and administering the system in an efficient, cost-effective way

This solution is based on the Cisco UCS Integrated Infrastructure for Big Data and Analytics and includes computing, storage, connectivity, and unified management capabilities to help companies manage the immense amount of data being collected. It is built on the Cisco Unified Computing System (Cisco UCS) infrastructure using Cisco UCS 6332 Series Fabric Interconnects and Cisco UCS C-Series Rack Servers. This architecture is specifically designed for performance and linear scalability for big data and machine learning workloads.

Audience

The intended audience of this document includes, but not limited to, sales engineers, field consultants, professional services, IT managers, partner engineering and customers who want to deploy the Hortonworks Data Platform (HDP 3.0) on Cisco UCS Integrated Infrastructure for Big Data and Analytics. You are assumed to have intermediate level of knowledge for Apache Hadoop and Cisco UCS based scale-out infrastructure.

Purpose of this Document

This document describes the architecture and step by step guidelines of deployment procedures for Hortonworks Data Platform (HDP) 3.0.1 on a 28-node Cisco UCS C240 M5 cluster based on Cisco UCS Integrated Infrastructure for Big Data and Analytics.

Solution Summary

This CVD describes in detail the process for installing Hortonworks 3.0.1 with Apache Spark and Docker containers including the configuration details of the cluster. The current version of Cisco UCS Integrated Infrastructure for Big Data and Analytics offers the following configurations depending on the compute and storage requirements as shown in Table 1.

Table 1           Cisco UCS Integrated Infrastructure for Big Data and Analytics Configuration Options

Table 2          High Density GPU Nodes for Data Nodes  

Figure 1 illustrates a 16-node starter cluster. The first rack (left) has 16 Cisco UCS C240 M5 servers. Each link in the figure represents a 40 Gigabit Ethernet link from each of the 16 servers directly connected to a Fabric Interconnect. The second rack (right) has 8 x Cisco UCS C240 M5 servers and 4 x Cisco UCS C480 ML M5 Servers. Every server is connected to both Fabric Interconnects.

Figure 1  Topology

 

 

Scaling the Solution

Figure 2 illustrates how to scale the solution. Each pair of Cisco UCS 6332 Fabric Interconnects has 28 Cisco UCS C240 M5 servers connected to it. This allows for four uplinks from each Fabric Interconnect to the Cisco Nexus 9332 switch. Six pairs of 6332 FI’s can connect to a single switch with four uplink ports each. With 28 servers per FI, a total of 168 servers can be supported. Additionally, the can scale to thousands of nodes with the Nexus 9500 series family of switches.

Figure 2  Scaling the Solution

Cisco UCS Integrated Infrastructure for Big Data and Analytics

The Cisco UCS Integrated Infrastructure for Big Data and Analytics solution for Hortonworks Data Platform on Cisco UCS Integrated Infrastructure for Big Data and Analytics, is a highly scalable architecture designed to meet a variety of scale-out application demands with seamless data integration and management integration capabilities built using the components described in this section.

Cisco Unified Computing System

Cisco Unified Computing System is a next-generation solution for blade and rack server computing. Cisco UCS integrates a low-latency; lossless 10 and 40 Gigabit Ethernet unified network fabric with enterprise-class, x86-architecture servers. Cisco UCS is an integrated, scalable, multi-chassis platform in which all resources participate in a unified management domain. Cisco UCS accelerates the delivery of new services simply, reliably, and securely through end-to-end provisioning and migration support for both virtualized and non-virtualized systems. Cisco UCS fuses access layer networking and servers. This high-performance, next-generation server system provides a data center with a high degree of workload agility and scalability.

Cisco UCS 6300 Series Fabric Interconnects

Cisco UCS 6300 Series Fabric Interconnects provide high-bandwidth, low-latency connectivity for servers, with integrated, unified management provided for all connected devices by Cisco UCS Manager (UCSM). Deployed in redundant pairs, Cisco fabric interconnects offer the full active-active redundancy, performance, and exceptional scalability needed to support the large number of nodes that are typical in clusters serving big data applications. Cisco UCS Manager enables rapid and consistent server configuration using service profiles, automating ongoing system maintenance activities such as firmware updates across the entire cluster as a single operation. Cisco UCS Manager also offers advanced monitoring with options to raise alarms and send notifications about the health of the entire cluster.

The Cisco UCS 6300 series Fabric interconnects are a core part of Cisco UCS, providing low-latency, lossless 10 and 40 Gigabit Ethernet, Fiber Channel over Ethernet (FCoE), and Fiber Channel functions with management capabilities for the entire system. All servers attached to Fabric interconnects become part of a single, highly available management domain.

Figure 3  Cisco UCS 6332 UP 32 -Port Fabric Interconnect

Cisco UCS C-Series Rack-Mount Servers

Cisco UCS C-Series Rack-Mount Servers keep pace with Intel Xeon processor innovation by offering the latest processors with increased processor frequency and improved security and availability features. With the increased performance provided by the Intel Xeon Scalable Family Processors, Cisco UCS C-Series servers offer an improved price-to-performance ratio. They also extend Cisco UCS innovations to an industry-standard rack-mount form factor, including a standards-based unified network fabric, Cisco VN-Link virtualization support, and Cisco Extended Memory Technology.

It is designed to operate both in standalone environments and as part of Cisco UCS managed configuration, these servers enable organizations to deploy systems incrementally—using as many or as few servers as needed—on a schedule that best meets the organization’s timing and budget. Cisco UCS C-Series servers offer investment protection through the capability to deploy them either as standalone servers or as part of Cisco UCS. One compelling reason that many organizations prefer rack-mount servers is the wide range of I/O options available in the form of PCIe adapters. C-Series servers support a broad range of I/O options, including interfaces supported by Cisco and adapters from third parties.

Cisco UCS C240 M5 Rack-Mount Server

The Cisco UCS C240 M5 Rack-Mount Server (Figure 4) is a 2-socket, 2-Rack-Unit (2RU) rack server offering industry-leading performance and expandability. It supports a wide range of storage and I/O-intensive infrastructure workloads, from big data and analytics to collaboration. Cisco UCS C-Series Rack Servers can be deployed as standalone servers or as part of a Cisco Unified Computing System managed environment to take advantage of Cisco’s standards-based unified computing innovations that help reduce customers’ Total Cost of Ownership (TCO) and increase their business agility.

In response to ever-increasing computing and data-intensive real-time workloads, the enterprise-class Cisco UCS C240 M5 server extends the capabilities of the Cisco UCS portfolio in a 2RU form factor. It incorporates the Intel Xeon Scalable processors, supporting up to 20 percent more cores per socket, twice the memory capacity, and five times more

Non-Volatile Memory Express (NVMe) PCI Express (PCIe) Solid-State Disks (SSDs) compared to the previous generation of servers. These improvements deliver significant performance and efficiency gains that will improve your application performance. The Cisco UCS C240 M5 delivers outstanding levels of storage expandability with exceptional performance, along with the following:

·         Latest Intel Xeon Scalable CPUs with up to 28 cores per socket

·         Up to 24 DDR4 DIMMs for improved performance

·         Up to 26 hot-swappable Small-Form-Factor (SFF) 2.5-inch drives, including 2 rear hot-swappable SFF drives (up to 10 support NVMe PCIe SSDs on the NVMe-optimized chassis version), or 12 Large-Form- Factor (LFF) 3.5-inch drives plus 2 rear hot-swappable SFF drives

·         Support for 12-Gbps SAS modular RAID controller in a dedicated slot, leaving the remaining PCIe Generation 3.0 slots available for other expansion cards

·         Modular LAN-On-Motherboard (mLOM) slot that can be used to install a Cisco UCS Virtual Interface Card (VIC) without consuming a PCIe slot, supporting dual 10- or 40-Gbps network connectivity

·         Dual embedded Intel x550 10GBASE-T LAN-On-Motherboard (LOM) ports

·         Modular M.2 or Secure Digital (SD) cards that can be used for boot

Figure 4  Cisco UCS C240 M5 Rack-Mount Server – Front View

Figure 5  Cisco UCS C240 M5 Rack-Mount Server – Rear View

Cisco UCS C480 M5 Rack-Mount Server

The Cisco UCS C480 M5 Rack-Mount Server is a storage and I/O-optimized enterprise-class rack-mount server that delivers industry-leading performance for in-memory databases, big data analytics, virtualization, Virtual Desktop Infrastructure (VDI), and bare-metal applications. The Cisco UCS C480 M5 (Figure 6) delivers outstanding levels of expandability and performance for standalone or Cisco Unified Computing System managed environments in a 4RU form-factor. Because of its modular design, you pay for only what you need. It offers these capabilities:

·         Latest Intel Xeon Scalable processors with up to 28 cores per socket and support for two-or four-processor configurations

·         2666-MHz DDR4 memory and 48 DIMM slots for up to 6 Terabytes (TB) of total memory

·         12 PCI Express (PCIe) 3.0 slots

-        Six x 8 full-height, full length slots

-        Six x16 full-height, full length slots

·         Flexible storage options with support up to 32 Small-Form-Factor (SFF) 2.5-inch, SAS, SATA, and PCIe NVMe disk drives

·         Cisco 12-Gbps SAS Modular RAID Controller in a dedicated slot

·         Internal Secure Digital (SD) and M.2 boot options

·         Dual embedded 10 Gigabit Ethernet LAN-On-Motherboard (LOM) ports

Figure 6  Cisco UCS C480 M5 Rack-Mount Server – Front View

Figure 7  Cisco UCS C480 M5 Rack-Mount Server – Rear View

For more information about Cisco UCS C480 M5 Rack Server, go to: https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/datasheet-c78-739291.html

Cisco UCS C480 ML M5 Rack Server

The Cisco UCS C480 ML M5 Rack Server is a purpose-built server for Deep Learning. It is storage and I/O optimized to deliver an industry-leading performance for training Models. The Cisco UCS C480 ML M5 delivers outstanding levels of storage expandability and performance options for standalone or Cisco Unified Computing System managed environments in a 4RU form factor. Because of its modular design, you pay for only what you need. It offers these capabilities:

·         8 NVIDIA SXM2 V100 32G modules with NVLink interconnect

·         Latest Intel Xeon Scalable processors with up to 28 cores per socket and support for two processor configurations

·         2666-MHz DDR4 memory and 24 DIMM slots for up to 3 terabytes (TB) of total memory

·         4 PCI Express (PCIe) 3.0 slots for 100G UCS VIC 1495

·         Flexible storage options with support for up to 24 Small-Form-Factor (SFF) 2.5-inch, SAS/SATA Solid-State Disks (SSDs) and Hard-Disk Drives (HDDs)

·         Up to 6 PCIe NVMe disk drives

·         Cisco 12-Gbps SAS Modular RAID Controller in a dedicated slot

·         M.2 boot options

·         Dual embedded 10 Gigabit Ethernet LAN-On-Motherboard (LOM) ports

Figure 8  Cisco UCS C480 ML M5 Purpose Built Deep Learning Server – Front View

Figure 9  Cisco UCS C480 ML M5 Purpose Built Deep Learning Server – Rear View

For more information about Cisco UCS C480 ML M5 Server, go to: https://www.cisco.com/c/dam/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/c480m5-specsheet-ml-m5-server.pdf

Table 3 lists the features and benefits of Cisco UCS C480 ML M5 Server.

Table 3           Feature and Benefits for Cisco UCS C480 ML M5 Server

Cisco UCS Virtual Interface Cards

Cisco UCS Virtual Interface Cards (VICs) are unique to Cisco. Cisco UCS Virtual Interface Cards incorporate next-generation converged network adapter (CNA) technology from Cisco and offer dual 10- and 40-Gbps ports designed for use with Cisco UCS servers. Optimized for virtualized networking, these cards deliver high performance and bandwidth utilization, and support up to 256 virtual devices.

The Cisco UCS Virtual Interface Card 1387 offers dual-port Enhanced Quad Small Form-Factor Pluggable (QSFP+) 40 Gigabit Ethernet and Fiber Channel over Ethernet (FCoE) in a modular-LAN-on-motherboard (mLOM) form factor. The mLOM slot can be used to install a Cisco VIC without consuming a PCIe slot providing greater I/O expandability.

Figure 10           Cisco UCS VIC 1387

For more information about Cisco UCS Adapters, go to: https://www.cisco.com/c/en/us/products/interfaces-modules/unified-computing-system-adapters/index.html

Cisco UCS Manager

Cisco UCS Manager (UCSM) resides within the Cisco UCS 6300 Series Fabric Interconnect. It makes the system self-aware and self-integrating, managing all of the system components as a single logical entity. Cisco UCS Manager can be accessed through an intuitive GUI, a CLI, or an XML API. Cisco UCS Manager uses service profiles to define the personality, configuration, and connectivity of all resources within Cisco UCS, radically simplifying provisioning of resources so that the process takes minutes instead of days. This simplification allows IT departments to shift their focus from constant maintenance to strategic business initiatives.

For more information about Cisco UCS Manger, go to: https://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-manager/index.html

NVIDIA GPU

Graphics Processing Units or GPUs are specialized processors designed to render images, animation and video for computer displays. They perform this task by running many operations simultaneously. While the number and kinds of operations they can do are limited, they make up for it by being able run many thousands in parallel. As the graphics capabilities of GPUs increased, it soon became apparent that the massive parallelism of GPUs could be put to other uses beside rendering graphics.

NVIDIA GPU used in this document, NVIDIA Tesla V100, is advanced data center GPU built to accelerate AI, HPC, and graphics. It is powered by NVIDIA Volta architecture, comes in 16 and 32 GB configurations.

NVIDIA GPUs bring two key advantages to the table. First, they make possible solutions that were simply not computationally possible before. Second, by providing the same processing power as scores of traditional CPUs they reduce the requirements for rack space, power, networking and cooling in the data center.

NVIDIA CUDA

GPUs are very good at running the same operation on different data simultaneously. This is often referred to as single instruction, multiple data, or SIMD. This is exactly what’s needed to render graphics but many other computing problems can benefit from this approach. As a result, NVIDIA created CUDA. CUDA is a parallel computing platform and programming model that makes it possible to use a GPU for many general-purpose computing tasks via commonly used programming languages like C and C++.

In addition to the general-purpose computing capabilities that CUDA enables there is also a special CUDA library for deep learning called the CUDA Deep Neural Network library, or cuDNN. cuDNN makes it easier to implement deep machine learning architectures that take full advantage of the GPU’s capabilities.

Hortonworks Data Platform

The Hortonworks Data Platform (HDP 3.0.1) delivers essential capabilities in a completely open, integrated and tested platform that is ready for enterprise usage. With Hadoop YARN at its core, HDP provides flexible enterprise data processing across a range of data processing engines, paired with comprehensive enterprise capabilities for governance, security and operations.

All the integration of the entire solution is thoroughly tested and fully documented. By taking the guesswork out of building out a Hadoop deployment, HDP gives a streamlined path to success in solving real business problems.

Hortonworks Data Platform (HDP) 3.0 delivers significant new features, including the ability to launch apps in a matter of minutes and address new use cases for high-performance deep learning and machine learning apps. In addition, this new version of HDP enables enterprises to gain value from their data faster, smarter, in a hybrid environment.

Apache Ambari

Apache Ambari is a completely open source management platform. It performs provisioning, managing, securing, and monitoring Apache Hadoop clusters. Apache Ambari is a part of Hortonworks Data Platform and it allows enterprises to plan and deploy HDP cluster. It also provides ongoing cluster maintenance and management.

Ambari provides an intuitive Web UI as well as an extensive REST API framework which is very useful for automating cluster operations.

Below are the core benefits that Hadoop operators get with Ambari:

·         Simplified Installation, Configuration and Management. Easily and efficiently create, manage and monitor clusters at scale. Takes the guesswork out of configuration with Smart Configs and Cluster Recommendations.  Enables repeatable, automated cluster creation with Ambari Blueprints.

·         Centralized Security Setup. Reduce the complexity to administer and configure cluster security across the entire platform. Helps automate the setup and configuration of advanced cluster security capabilities such as Kerberos and Apache Ranger.

·         Full Visibility into Cluster Health. Ensure your cluster is healthy and available with a holistic approach to monitoring. Configures predefined alerts — based on operational best practices — for cluster monitoring. Captures and visualizes critical operational metrics — using Grafana — for analysis and troubleshooting. Integrated with Hortonworks SmartSense for proactive issue prevention and resolution.

·         Highly Extensible and Customizable. Fit Hadoop seamlessly into your enterprise environment. Highly extensible with Ambari Stacks for bringing custom services under management, and with Ambari Views for customizing the Ambari Web UI.

HDP for Data Access

With YARN at its foundation, HDP provides a range of processing engines that allow users to interact with data in multiple and parallel ways, without the need to stand up individual clusters for each data set/application. Some applications require batch while others require interactive SQL or low-latency access with NoSQL. Other applications require search, streaming or in-memory analytics. Apache Solr, Storm and Spark fulfill those needs respectively.

To function as a true data platform, the YARN-based architecture of HDP enables the widest possible range of access methods to coexist within the same cluster avoiding unnecessary and costly data silos.

As shown in Figure 11, HDP Enterprise natively provides for the following data access types:

·         Batch – Apache MapReduce has served as the default Hadoop processing engine for years. It is tested and relied upon by many existing applications.

·         Interactive SQL Query - Apache Hive is the de facto standard for SQL interactions at petabyte scale within Hadoop. Hive delivers interactive and batch SQL querying across the broadest set of SQL semantics.

·         Search - HDP integrates Apache Solr to provide high-speed indexing and sub-second search times across all your HDFS data.

·         Scripting - Apache Pig is a scripting language for Hadoop that can run on MapReduce or Apache Tez, allowing you to aggregate, join and sort data.

·         Low-latency access via NoSQL - Apache HBase provides extremely fast access to data as a columnar format, NoSQL database. Apache Accumulo also provides high-performance storage and retrieval, but with fine-grained access control to the data.

·         Streaming - Apache Storm processes streams of data in real time and can analyze and take action on data as it flows into HDFS.

Figure 11           YARN

Docker Containerization

Hortonworks Data Platform (HDP 3.0) makes use of container technology. Containers are conceptually similar to virtual machines, but instead of virtualizing the hardware, a container virtualizes the operating system. With a VM there is an entire operating system sitting on top of the hypervisor. Containers dispense with this time-consuming and resource hungry requirement by sharing the host system’s kernel. As a result, a container is far smaller, and its lightweight nature means they can be instantiated quickly. In fact, they can be instantiated so quickly that new application architectures are possible.

Docker is an open-source project that performs operating-system-level virtualization, also known as "containerization." It uses Linux kernel features like namespaces and control groups to create containers. These features are not new, but Docker has taken these concepts and improved them in the following ways:

·         Ease of use: Docker makes easier for anyone — developers, systems admins, architects and others — to take advantage of containers in order to quickly build and test portable applications. It allows anyone to package an application on their development system, which can then run unmodified on any cloud or bare metal server. The basic idea is to create a “build once, run anywhere” system.

·         Speed: Docker containers are very fast with a small footprint. Ultimately, containers are just sandboxed environments running on the kernel, so they take up few resources. You can create and run a Docker container in seconds. Compare this to a VM which takes much longer because it has to boot up a full virtual operating system every time.

·         Modularity: Docker makes it easy to take an application and breaks its functionality into separate individual containers. These containers can then be spun up and run as needed. This is particularly useful for cases where an application needs to hold and lock a particular resource, like a GPU, and then release it once it’s done using it. Modularity also enables each component, i.e., container to be updated independently.

·         Scalability: modularity enables scalability. With different parts of the system running in different containers it becomes possible, and with Docker, it becomes easy to connect these containers together to create an application, which can then be scaled out as needed.

YARN Support For Docker

Containerization provides YARN support for Docker containers, which makes it easier to bundle libraries and dependencies along with their application, allowing third-party applications to run on Apache Hadoop (for example, containerized applications), enabling:

·         Faster time to deployment by enabling third-party apps.

·         The ability to run multiple versions of an application, enabling users to rapidly create features by developing and testing new versions of services without disrupting old ones.

·         Improved resource utilization and increased task throughput for containers, yielding faster time to market for services.

·         Orchestration of stateless distributed applications.

·         Packaging libraries for Spark application, eliminating the need for operations to deploy those libraries cluster wide.

Figure 12           Containerized Application on Apache Hadoop YARN 3.1

As shown in Figure 12, YARN Services Framework in addition with Docker containerization, it is now possible to run both existing Hadoop frameworks, such as Hive, Spark, etc., and new containerized workloads on the same underlying infrastructure. Apache Hadoop 3.1 further improved these capabilities to enable advanced use cases such as TensorFlow and HBase.

NVIDIA Docker

Docker containers are platform-agnostic, but also hardware-agnostic. This presents a problem when using specialized hardware such as NVIDIA GPUs which require kernel modules and user-level libraries to operate.  As a result, Docker does not natively support NVIDIA GPUs within containers.

One of the early workarounds to this problem was to fully install the NVIDIA drivers inside the container and map in the character devices corresponding to the NVIDIA GPUs (for example, /dev/nvidia0) on launch. This solution is brittle because the version of the host driver must exactly match the version of the driver installed in the container. This requirement drastically reduced the portability of these early containers, undermining one of Docker’s more important features.

To enable portability in Docker images that leverage NVIDIA GPUs, NVIDIA developed nvidia-docker, an open-source project hosted on GitHub that provides the two critical components needed for portable GPU-based containers:

driver-agnostic CUDA images; and a Docker command line wrapper that mounts the user mode components of the driver and the GPUs (character devices) into the container at launch.

nvidia-docker is essentially a wrapper around the docker command that transparently provisions a container with the necessary components to execute code on the GPU.

GPU Pooling and Isolation

GPU pooling and isolation allows GPU to be a first-class resource type in Hadoop, making it easier for customers to run machine learning and deep learning workloads.

·         Compute-intensive analytics require not only a large compute pool, but also a fast and expensive processing pool with GPUs in tandem

·         Customers can share cluster-wide GPU resources without having to dedicate a GPU node to a single tenant or workload

·         GPU isolation dedicates a GPU to an application so that no other application has access to that GPU

When it comes to resource scheduling, it is important to recognize GPU as a resource. YARN extends the resource model to more flexible mode which makes it easier to add new countable resource-types. When GPU is added as resource type, YARN can schedule applications on GPU machines. Furthermore, by specifying the number of requested GPU to containers, YARN can find machines with available GPUs to satisfy container requests.

Figure 13           YARN Scheduling for GPU/Non-GPU Applications

Red Hat Ansible Automation

Red Hat Ansible Automation is a powerful IT automation tool. It is capable of provisioning numerous types of resources and deploying applications. It can configure and manage devices and operating system components. Due to its simplicity, extensibility, and portability, this solution extensively utilizes Ansible for performing repetitive deployment steps across the nodes.

 

Requirements

This CVD describes the architecture and deployment procedures for Hortonworks Data Platform (HDP) 3.0.1 on a 28 Cisco UCS C240 M5 node cluster based on Cisco UCS Integrated Infrastructure for Big Data and Analytics. The solution goes into detail configuring HDP 3.0.1 on the Cisco UCS Integrated infrastructure for Big Data. In addition, it also details the configuration for Hortonworks Dataflow for various use cases.

The cluster configuration consists of the following:

·         2 Cisco UCS 6332UP Fabric Interconnects

·         24 Cisco UCS C240 M5 Rack-Mount servers

·         4 Cisco UCS C480 ML M5 Rack-Mount server

·         8 NVIDIA GPU in each Cisco UCS C480 ML M5

·         2 Cisco R42610 standard racks

·         4 Vertical Power distribution units (PDUs) (Country Specific) per rack

Rack and PDU Configuration

Each rack consists of two vertical PDUs. The first rack consists of two Cisco UCS 6332UP Fabric Interconnects, 16 Cisco UCS C240 M5 Rack Servers connected to each of the vertical PDUs for redundancy; thereby ensuring availability during power source failure. The second rack consists of 8 Cisco UCS C240 M5 Servers and 4 Cisco UCS C480 ML M5 connected to each of the vertical PDUs for redundancy; thereby ensuring availability during power source failure, similar to the first rack.

Table 4          Port Configuration on Fabric Interconnects

Cabling for Cisco UCS C240 M5

The Cisco UCS C240 M5 rack server is equipped with 2 x Intel Xeon Processor Scalable Family 6132 (2 x 14 cores, 2.6 GHz), 192 GB of memory, Cisco UCS Virtual Interface Card 1387 Cisco 12-Gbps SAS Modular Raid Controller with 4-GB FBWC, 26 x 1.8 TB 10K rpm SFF SAS HDDs or 12 x 1.6 TB Enterprise Value SATA SSDs, M.2 with 2 x 240-GB SSDs for Boot.

Figure 14 illustrates the port connectivity between the Fabric Interconnect, and Cisco UCS C240 M5 server. Sixteen Cisco UCS C240 M5 servers are used in Master rack configurations.

 

Figure 14           Cisco UCS C240 M5 and 6300 Series Fabric Interconnect Port Connectivity

For information about physical connectivity and single-wire management go to:

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c-series_integration/ucsm3-2/b_C-Series-Integration_UCSM3-2/b_C-Series-Integration_UCSM3-2_chapter_010.html?bookSearch=true

For more information about physical connectivity illustrations and cluster setup, go to:

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c-series_integration/ucsm3-2/b_C-Series-Integration_UCSM3-2/b_C-Series-Integration_UCSM3-2_chapter_010.html?bookSearch=true

Software Distributions and Versions

The software distributions required versions are listed below.

Hortonworks Data Platform (HDP 3.0.1)

The Hortonworks Data Platform supported is HDP 3.0.1. For more information, go to: http://www.hortonworks.com.

Red Hat Enterprise Linux (RHEL)

The operating system supported is Red Hat Enterprise Linux 7.5. For more information, go to: http://www.redhat.com.

Software Versions

The software versions tested and validated in this document are shown in Table 5.

Table 5           Software Versions

 

 

Fabric Configuration

This section provides the details to configure a fully redundant, highly available Cisco UCS 6332 fabric configuration. The following is the high-level workflow to setup Cisco UCS:

·         Initial setup of the Fabric Interconnect A and B

·         Connect to Cisco UCS Manager using virtual IP address of using the web browser

·         Launch Cisco UCS Manager

·         Enable server and uplink ports

·         Start discovery process

·         Create pools and polices for service profile template

·         Create Service Profile template

·         Create service profile for each server from service profile template

·         Associate Service Profiles to servers5

Perform Initial Setup of Cisco UCS 6332 Fabric Interconnects

This section describes the initial setup of the Cisco UCS 6332 Fabric Interconnects A and B.

Configure Fabric Interconnect A

To configure Fabric Interconnect A, follow these steps:

1.        Connect to the console port on the first Cisco UCS 6332 Fabric Interconnect.

2.       At the prompt to enter the configuration method, enter console to continue.

3.        If asked to either perform a new setup or restore from backup, enter setup to continue.

4.       Enter y to continue to set up a new Fabric Interconnect.

5.        Enter y to enforce strong passwords.

6.       Enter the password for the admin user.

7.        Enter the same password again to confirm the password for the admin user.

8.       When asked if this fabric interconnect is part of a cluster, answer y to continue.

9.       Enter A for the switch fabric.

10.    Enter the cluster name for the system name.

11.     Enter the Mgmt0 IPv4 address.

12.    Enter the Mgmt0 IPv4 netmask.

13.     Enter the IPv4 address of the default gateway.

14.    Enter the cluster IPv4 address.

15.     To configure DNS, answer y.

16.    Enter the DNS IPv4 address.

17.     Answer y to set up the default domain name.

18.    Enter the default domain name.

19.    Review the settings that were printed to the console, and if they are correct, answer yes to save the configuration.

20.    Wait for the login prompt to make sure the configuration has been saved.

Configure Fabric Interconnect B

To configure Fabric Interconnect B, follow these steps:

1.        Connect to the console port on the second Cisco UCS 6332 Fabric Interconnect.

2.       When prompted to enter the configuration method, enter console to continue.

3.        The installer detects the presence of the partner Fabric Interconnect and adds this fabric interconnect to the cluster. Enter y to continue the installation.

4.       Enter the admin password that was configured for the first Fabric Interconnect.

5.        Enter the Mgmt0 IPv4 address.

6.       Answer yes to save the configuration.

7.        Wait for the login prompt to confirm that the configuration has been saved.

For more information about configuring Cisco UCS 6332 Series Fabric Interconnect, go to:

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Getting-Started/3-2/b_UCSM_Getting_Started_Guide_3_2/b_UCSM_Getting_Started_Guide_3_2_chapter_0100.html

Log Into Cisco UCS Manager

To log into Cisco UCS Manager, follow these steps:

1.        Open a Web browser and navigate to the Cisco UCS 6332 Fabric Interconnect cluster address.

2.       Click the Launch link to download the Cisco UCS Manager software.

3.        If prompted to accept security certificates, accept as necessary.

4.       When prompted, enter admin for the username and enter the administrative password.

5.        Click Login to log in to the Cisco UCS Manager.

Upgrade Cisco UCS Manager Software to Version 4.0(2a)

This document assumes the use of UCS 4.0(2a). Refer to the Cisco UCS 4.0 Release  (upgrade Cisco UCS Manager software and UCS 6332 Fabric Interconnect software to version 4.0(2a). Also, make sure the Cisco UCS C-Series version 4.0(2a) software bundles are installed on the Fabric Interconnects.

Add a Block of IP Addresses for KVM Access

To create a block of KVM IP addresses for server access in the Cisco UCS environment, follow these steps:

1.        Select the LAN tab at the top of the left window.

2.       Select Pools > root > IpPools > Ip Pool ext-mgmt.

3.        Right-click IP Pool ext-mgmt.

4.       Select Create Block of IPv4 Addresses.

Figure 15           Adding a Block of IPv4 Addresses for KVM Access Part 1

5.        Enter the starting IP address of the block and number of IPs needed, as well as the subnet and gateway information.

Figure 16           Adding Block of IPv4 Addresses for KVM Access Part 2

6.       Click OK to create the IP block.

7.        Click OK in the message box.

Enable Uplink Ports

To enable uplinks ports, follow these steps:

1.        Select the Equipment tab on the top left of the window.

2.       Select Equipment > Fabric Interconnects > Fabric Interconnect A (primary) > Fixed Module.

3.        Expand the Unconfigured Ethernet Ports section.

4.       Select port 29-32 that is connected to the uplink switch, right-click, then select Reconfigure > Configure as Uplink Port.

5.        Select Show Interface and select 40GB for Uplink Connection.

6.       A pop-up window appears to confirm your selection. Click Yes then OK to continue.

7.        Select Equipment > Fabric Interconnects > Fabric Interconnect B (subordinate) > Fixed Module.

8.       Expand the Unconfigured Ethernet Ports section.

9.       Select port number 29-32, which is connected to the uplink switch, right-click, then select Reconfigure > Configure as Uplink Port.

10.    Select Show Interface and select 40GB for Uplink Connection.

11.     A pop-up window appears to confirm your selection. Click Yes then OK to continue.

Figure 17           Enabling Uplink Ports Part1

Figure 17           Enabling Uplink Ports Part2

Figure 18           Enabling Uplink Ports Part 3

Configure VLANs

VLANs are configured as in shown in Table 6.

Table 6          VLAN Configurations

The NIC will carry the data traffic from VLAN13. A single vNIC is used in this configuration and the Fabric Failover feature in Fabric Interconnects will take care of any physical port down issues. It will be a seamless transition from an application perspective.

To configure VLANs in the Cisco UCS Manager GUI, follow these steps:

1.        Select the LAN tab in the left pane in the UCSM GUI.

2.       Select LAN > LAN Cloud > VLANs.

3.        Right-click the VLANs under the root organization.

4.       Select Create VLANs to create the VLAN.

Figure 19           Creating a VLAN

5.        Enter vlan13 for the VLAN Name.

6.       Keep multicast policy as <not set>.

7.        Select Common/Global for vlan16.

8.       Enter 13 in the VLAN IDs field for the Create VLAN IDs.

9.       Click OK and then, click Finish.

10.    Click OK in the success message box.

Figure 20          Creating VLAN for Data

11.     Click OK and then click Finish.

Enable Server Ports

To enable server ports, follow these steps:

1.        Select the Equipment tab on the top left of the window.

2.       Select Equipment > Fabric Interconnects > Fabric Interconnect A (primary) > Fixed Module.

3.        Expand the Unconfigured Ethernet Ports section.

4.       Select all the ports that are connected to the Servers right-click them and select Reconfigure > Configure as a Server Port.

5.        A pop-up window appears to confirm your selection. Click Yes then OK to continue.

6.       Select Equipment > Fabric Interconnects > Fabric Interconnect B (subordinate) > Fixed Module.

7.        Expand the Unconfigured Ethernet Ports section.

8.       Select all the ports that are connected to the Servers right-click them, and select Reconfigure > Configure as a Server Port.

9.       A pop-up window appears to confirm your selection. Click Yes, then OK to continue.

Figure 21           Enabling Server Ports

After the Server Discovery, Port 29-32 will be a Network Port and 1-28 will be Server Ports.

Figure 22          Ports Status after the Server Discover

Create Pools for Service Profile Templates

Create an Organization            

Organizations are used as a means to arrange and restrict access to various groups within the IT organization, thereby enabling multi-tenancy of the compute resources. This document does not assume the use of Organizations; however, the necessary steps are provided for future reference.

To configure an organization within the Cisco UCS Manager GUI, follow these steps:

1.        Click Quick Action icon on the top right corner in the right pane in the Cisco UCS Manager GUI.

2.       Select Create Organization from the options

3.        Enter a name for the organization.

4.       (Optional) Enter a description for the organization.

5.        Click OK.

6.       Click OK in the success message box.

Create MAC Address Pools

To create MAC address pools, follow these steps:

1.        Select the LAN tab on the left of the window.

2.       Select Pools > root > MAC Pools

3.        Right-click MAC Pools under the root organization.

4.       Select Create MAC Pool to create the MAC address pool. Enter ucs for the name of the MAC pool.

5.        (Optional) Enter a description of the MAC pool.

6.       Select Assignment Order Sequential.

7.        Click Next.

8.       Click Add.

9.       Specify a starting MAC address.

10.    Specify a size of the MAC address pool, which is sufficient to support the available server resources.

11.     Click OK.

Figure 23           Specifying first MAC Address and Size

12.    Click Finish.

13.     When the message box displays, click OK.

Create a Server Pool

A server pool contains a set of servers. These servers typically share the same characteristics. Those characteristics can be their location in the chassis, or an attribute such as server type, amount of memory, local storage, type of CPU, or local drive configuration. You can manually assign a server to a server pool or use server pool policies and server pool policy qualifications to automate the assignment.

To configure the server pool within the Cisco UCS Manager GUI, follow these steps:

1.        Select the Servers tab in the left pane in the Cisco UCS Manager GUI.

2.       Select Pools > root.

3.        Right-click the Server Pools.

4.       Select Create Server Pool.

5.        Enter your required name (ucs) for the Server Pool in the name text box.

6.       (Optional) enter a description for the organization.

7.        Click Next > to add the servers.

8.       Select all the Cisco UCS C240M5 servers to be added to the server pool that was previously created (ucs), then Click >> to add them to the pool.

9.       Click Finish.

10.    Click OK and then click Finish.

Create Policies for Service Profile Templates

Create Host Firmware Package Policy

Firmware management policies allow the administrator to select the corresponding packages for a given server configuration. These include adapters, BIOS, board controllers, FC adapters, HBA options, and storage controller properties as applicable.

To create a firmware management policy for a given server configuration using the Cisco UCS Manager GUI, follow these steps:

1.        Select the Servers tab in the left pane in the UCS Manager GUI.

2.       Select Policies > root.

3.        Right-click Host Firmware Packages.

4.       Select Create Host Firmware Package.

5.        Enter the required Host Firmware package name (ucs).

6.       Select Simple radio button to configure the Host Firmware package.

7.        Select the appropriate Rack package that has been installed.

8.       Click OK to complete creating the management firmware package

9.       Click OK.

Create QoS Policies

To create the QoS policy for a given server configuration using the Cisco UCS Manager GUI, follow these steps:

Platinum Policy

1.        Select the LAN tab in the left pane in the Cisco UCS Manager GUI.

2.       Select Policies > root.

3.        Right-click QoS Policies.

4.       Select Create QoS Policy.

 

5.        Enter Platinum as the name of the policy.

6.       Select Platinum from the drop-down list.

7.        Keep the Burst(Bytes) field set to default (10240).

8.       Keep the Rate(Kbps) field set to default (line-rate).

9.       Keep Host Control radio button set to default (none).

10.    When the pop-up window appears, click OK to complete the creation of the Policy.

Set Jumbo Frames

To set Jumbo frames and enable QoS, follow these steps:

1.        Select the LAN tab in the left pane in the Cisco UCS Manager GUI.

2.       Select LAN Cloud > QoS System Class.

3.        In the right pane, select the General tab

4.       In the Platinum row, enter 9216 for MTU.

5.        Check the Enabled Check box next to Platinum.

6.       In the Best Effort row, select none for weight.

7.        In the Fiber Channel row, select none for weight.

8.       Click Save Changes.

9.       Click OK.

Create the Local Disk Configuration Policy

To create local disk configuration in the Cisco UCS Manager GUI, follow these steps:

1.        Select the Servers tab on the left pane in the Cisco UCS Manager GUI.

2.       Go to Policies > root.

3.        Right-click Local Disk Config Policies.

4.       Select Create Local Disk Configuration Policy.

5.        Enter ucs as the local disk configuration policy name.

6.       Change the Mode to Any Configuration. Check the Protect Configuration box.

7.        Keep the FlexFlash State field as default (Disable).

8.       Keep the FlexFlash RAID Reporting State field as default (Disable).

9.       Click OK to complete the creation of the Local Disk Configuration Policy.

10.    Click OK.

 

Create the Server BIOS Policy

The BIOS policy feature in Cisco UCS automates the BIOS configuration process. The traditional method of setting the BIOS is manually and is often error-prone. By creating a BIOS policy and assigning the policy to a server or group of servers, can enable transparency within the BIOS settings configuration.

To create a server BIOS policy using the Cisco UCS Manager GUI, follow these steps:

1.        Select the Servers tab in the left pane in the UCS Manager GUI.

2.       Select Policies > root.

3.        Right-click BIOS Policies.

4.       Select Create BIOS Policy.

5.        Enter your preferred BIOS policy name (ucs).

6.       Change the BIOS settings as shown in the following figures.

7.        Only changes that need to be made are in the Processor and RAS Memory settings.

Create the Boot Policy

To create boot policies within the Cisco UCS Manager GUI, follow these steps:

1.        Select the Servers tab in the left pane in the UCS Manager GUI.

2.       Select Policies > root.

3.        Right-click the Boot Policies.

4.       Select Create Boot Policy.

5.        Enter ucs as the boot policy name.

6.       (Optional) enter a description for the boot policy.

7.        Keep the Reboot on Boot Order Change check box unchecked.

8.       Keep Enforce vNIC/vHBA/iSCSI Name check box checked.

9.       Keep Boot Mode Default (Legacy).

10.    Expand Local Devices > Add CD/DVD and select Add Local CD/DVD.

11.     Expand Local Devices and select Add Local Disk.

12.    Expand vNICs and select Add LAN Boot and enter eth0.

13.     Click OK to add the Boot Policy.

14.    Click OK.

Create Power Control Policy

To create Power Control policies within the Cisco UCS Manager GUI, follow these steps:

1.        Select the Servers tab in the left pane in the Cisco UCS Manager GUI.

2.       Select Policies > root.

3.        Right-click the Power Control Policies.

4.       Select Create Power Control Policy.

 

5.        Enter ucs as the Power Control policy name.

6.       (Optional) enter a description for the boot policy.

7.        Select Performance for Fan Speed Policy.

8.       Select No cap for Power Capping selection.

9.       Click OK to create the Power Control Policy.

10.    Click OK.

Create Server BIOS Policy

To create a server BIOS policy for the Cisco UCS environment, follow these steps:

1.        In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.       Select Policies > root > Sub-Organization > UCS-HDP > BIOS Policies.

3.        Right-click BIOS Policies.

4.       Select Create BIOS Policy.

5.        Enter C240M5-BIOS as the BIOS policy name.

Figure 24          BIOS Configuration

Create Service Profile Template

To create the Service Profile Template, follow these steps:

1.        Select the Servers tab in the left pane in the Cisco UCS Manager GUI.

2.       Right-click Service Profile Templates.

3.        Select Create Service Profile Template.

The Create Service Profile Template window appears.

To identify the service profile template, follow these steps:

1.        Name the service profile template as ucs. Select the Updating Template radio button.

2.       In the UUID section, select Hardware Default as the UUID pool.

3.        Click Next to continue to the next section.

 

Configure the Storage Provisioning for the Template

To configure storage policies, follow these steps:

1.        Go to the Local Disk Configuration Policy tab and select ucs for the Local Storage.

2.       Click Next to continue to the next section.

 

3.        Click Next once the Networking window appears to go to the next section.

Configure Network Settings for the Template

To configure the network settings for the templates, follow these steps:

1.        Keep the Dynamic vNIC Connection Policy field at the default.

2.       Select Expert radio button for the option how would you like to configure LAN connectivity?

3.        Click Add to add a vNIC to the template.

4.       The Create vNIC window displays. Name the vNIC as eth0.

5.        Select ucs in the Mac Address Assignment pool.

6.       Select the Fabric A radio button and check the Enable failover check box for the Fabric ID.

7.        Check the VLAN13 check box for VLANs and select the Native VLAN radio button.

8.       Select MTU size as 9000.

9.       Select adapter policy as Linux.

10.    Select QoS Policy as Platinum.

11.     Keep the Network Control Policy as Default.

12.    Click OK.

       

       

13.     Click Next to continue with SAN Connectivity.

14.    Select no vHBAs for How would you like to configure SAN Connectivity?

15.     Click Next to continue with Zoning.

16.    Click Next to continue with vNIC/vHBA placement.

17.     Click Next to configure vMedia Policy.

Configure the vMedia Policy for the Template

To configure the vMedia policy for the template, follow these steps:

1.        Click Next once the vMedia Policy window appears to go to the next section.

Configure the Server Boot Order for the Template

To set the boot order for the servers, follow these steps:

1.        Select ucs in the Boot Policy name field.

2.       Review to make sure that all of the boot devices were created and identified.

3.        Verify that the boot devices are in the correct boot sequence.

4.       Click OK.

5.        Click Next to continue to the next section.

6.       In the Maintenance Policy window, apply the maintenance policy.

7.        Keep the Maintenance policy at no policy used by default. Click Next to continue to the next section.

Configure the Server Assignment for the Template

To assign the servers to the pool, In the Server Assignment window, follow these steps:

1.        Select ucs for the Pool Assignment field.

2.       Select the power state to be Up.

3.        Keep the Server Pool Qualification field set to <not set>.

4.       Check the Restrict Migration check box.

5.        Select ucs in Host Firmware Package.

 

Configure the Operational Policies for the Template

To configure the operational policies for the template, in the Operational Policies Window, follow these steps:

1.        Select ucs in the BIOS Policy field.

2.       Select ucs in the Power Control Policy field.

3.        Click Finish to create the Service Profile template.

4.       Click OK in the pop-up window to proceed.

5.        Select the Servers tab in the left pane of the Cisco UCS Manager GUI.

6.       Go to Service Profile Templates > root.

7.        Right-click Service Profile Templates ucs.

8.       Select Create Service Profiles From Template.

 

The Create Service Profiles from Template window appears.

Association of the Service Profiles will take place automatically.

Install Red Hat Enterprise Linux 7.5

This section provides detailed procedures to install Red Hat Enterprise Linux 7.5 using Software RAID (OS based Mirroring) on Cisco UCS C240 M5 servers. There are multiple ways to install the Red Hat Linux operating system. The installation procedure described in this deployment guide uses the KVM console and virtual media from Cisco UCS Manager.

To install the Red Hat Linux 7.5 operating system, follow these steps:

1.        Log into the Cisco UCS 6332 Fabric Interconnect and launch the Cisco UCS Manager application.

2.       Select the Equipment tab.

3.        In the navigation pane expand Rack-Mounts and then Servers.

4.       In the right pane, click the KVM Console >>.

5.        Click OK on the KVM Console – Select IP address pop-up window.

6.       Click the link to launch the KVM console.

7.        Point the cursor over the top right corner, select the Virtual Media tab.

8.       Click the Activate Virtual Devices found in Virtual Media tab.

9.       Click the Virtual Media tab again to select CD/DVD.

10.    Select Choose File in the Virtual Disk Management windows.

11.     Browse to the Red Hat Enterprise Linux Server 7.5 installer ISO image File.

12.    Click Open to add the image to the list of virtual media.

13.     Click Map Drive after selecting the .iso file.

14.    In the KVM window, select the KVM tab to monitor during boot.

15.     In the KVM window, select the Macros > Static Macros > Ctrl-Alt-Del button in the upper left corner.

16.    Click OK.

17.     Click OK to reboot the system.

18.    Press F6 key on the keyboard to select install media.

19.    On reboot, the machine detects the presence of the Red Hat Enterprise Linux Server 7.5 install media.

20.    Select the Install Red Hat Enterprise Linux 7.5.

21.    Skip the Media test and start the installation. Select language of installation and click Continue.

22.    Select Date and time, which pops up another window as shown below:

23.    Select the location on the map, set the time, and click Done.

24.    Click INSTALLATION DESTINATION.

25.    This opens a new window with the boot disks. Make the selection and choose I will configure partitioning. Click Done.

26.    This opens the new window for creating the partitions. Click the + sign to add a new partition as shown below, boot partition of size 2048 MB.

27.    Click Add MountPoint to add the partition.

28.    Change the Device type to RAID and make sure the RAID Level is RAID1 (Redundancy) and click Update Settings to save the changes.

29.    Click the + sign to create the swap partition of size 2048 MB as shown below.

30.    Change the Device type to RAID and RAID level to RAID1 (Redundancy) and click Update Settings.

31.     Click + to add the / partition. The size can be left empty, so it uses the remaining capacity and click Add Mountpoint.

32.    Change the Device type to RAID and RAID level to RAID1 (Redundancy). Click Update Settings.

33.     Click Done to return to the main screen and continue the Installation.

34.    Click SOFTWARE SELECTION.

35.     Select Infrastructure Server and select the Add-Ons as noted below. Click Done.

36.    Click NETWORK & HOSTNAME and configure Hostname and Networking for the Host.

37.     Type in the hostname as shown below.

38.    Click Configure to open the Network Connectivity window. Click IPV4Settings.

39.    Change the Method to Manual and click Add to enter the IP Address, Netmask, and Gateway details.

40.    Click Save and update the hostname and turn Ethernet ON. Click Done to return to the main menu.

41.    Click Begin Installation on the Main menu.

42.    Select Root Password in the User Settings.

43.    Enter the Root Password and click Done.

44.    When the installation is complete reboot the system.

45.    Repeat the steps to install Red Hat Enterprise Linux 7.5 on the remaining servers.

The hostnames and their corresponding IP addresses are shown in Table 7.

Table 7           Hostnames and IP Addresses

 

 

Post OS Install Configuration

Choose one of the nodes of the cluster or a separate node as the Admin Node for management such as HDP installation, Ansible, creating a local Red Hat repo and so on. In this document, rhel1 has been used for this purpose.

Configure /etc/hosts

Setup /etc/hosts on the Admin node; this is a pre-configuration to setup DNS as shown in the next section.

Below are the sample A records for DNS configuration within Linux environment.

To create the host file on the admin node, follow these steps:

1.        Log into the Admin Node (rhel1).

2.       Populate the host file with IP addresses and corresponding hostnames on the Admin node (rhel1) and other nodes as follows:

3.        On Admin Node (rhel1):

Set Up the Passwordless Login

To manage all of the clusters nodes from the admin node password-less login needs to be setup. It assists in automating common tasks with Ansible, and shell-scripts without having to use passwords.

To enable a passwordless login across all the nodes when Red Hat Linux is installed across all the nodes in the cluster, follow these steps:

1.        Log into the Admin Node (rhel1).

2.       Run the ssh-keygen command to create both public and private keys on the admin node.

Figure 25           ssh-keygen

3.        Run the following command from the admin node to copy the public key id_rsa.pub to all the nodes of the cluster.  ssh-copy-id appends the keys to the remote-host’s  .ssh/authorized_keys.

4.       Enter yes for Are you sure you want to continue connecting (yes/no)?

5.        Enter the password of the remote host.

Create the Red Hat Enterprise Linux (RHEL) 7.5 Local Repository

To create the repository using RHEL DVD or ISO on the admin node (in this deployment rhel1 is used for this purpose), create a directory with all the required RPMs, run the createrepo command and then publish the resulting repository.

1.        Log into rhel1. Create a directory that would contain the repository.

2.       Copy the contents of the Red Hat DVD to /var/www/html/rhelrepo

3.        Alternatively, if you have access to a Red Hat ISO Image, Copy the ISO file to rhel1.

4.       Log back into rhel1 and create the mount directory.

5.        Copy the contents of the ISO to the /var/www/html/rhelrepo directory.

6.       On rhel1 create a .repo file to enable the use of the yum command.

7.        Copy rheliso.repo file from /var/www/html/rhelrepo  to /etc/yum.repos.d on rhel1.

8.       To make use of repository files on rhel1 without httpd, edit the baseurl of repo file /etc/yum.repos.d/rheliso.repo to point repository location in the file system.

Create the Red Hat Repository Database

To create the Red Hat repository database, follow these steps:

1.        Install the createrepo package on admin node (rhel1). Use it to regenerate the repository database(s) for the local copy of the RHEL DVD contents.

2.       Run createrepo on the RHEL repository to create the repo database on admin node

Figure 26          createrepo

 

Set Up Ansible

To set up Ansible, follow these steps:

1.        Download Ansible rpm from the following link:

https://releases.ansible.com/ansible/rpm/release/epel-7-x86_64/ansible-2.4.6.0-1.el7.ans.noarch.rpm

2.       Run the following command to install ansible.

3.        Verify Ansible installation by running the following commands.

4.       Prepare the host inventory file for Ansible as shown below. Various host groups have been created based on any specific installation requirements of certain hosts.

5.        Verify host group by running the following commands. Figure 26 shows the outcome of the ping command.

Figure 27           Ansible – Ping Hosts

Install httpd

Setting up the RHEL repository on the admin node requires httpd. To set up RHEL repository on the admin node, follow these steps:

1.        Install httpd on the admin node to host repositories:

2.       Add ServerName and make the necessary changes to the server configuration file:

3.        Start httpd:

Set Up All Nodes to Use the RHEL Repository

To set up all noes to use the RHEL repository, follow these steps:

1.        Copy the rheliso.repo to all the nodes of the cluster:

2.       Copy the /etc/hosts file to all nodes:

3.        Purge the yum caches:

Upgrade the Cisco Network Driver for VIC1387

The latest Cisco Network driver is required for performance and updates. The latest drivers can be downloaded from the link below:

https://software.cisco.com/download/home/283862063/type/283853158/release/4.0.2

In the ISO image, the required driver kmod-enic-…..rpm can be located at  Network\Cisco\VIC\RHEL\RHEL7.5.

1.        From a node connected to the Internet, download, extract and transfer kmod-enic-.rpm to rhel1 (admin node).

2.       Copy the rpm on all nodes of the cluster using the following Ansible commands. For this example, the rpm is assumed to be in present working directory of rhel1:

3.        Use the yum module to install the enic driver rpm file on all the nodes through Ansible:

4.       Make sure that the above installed version of kmod-enic driver is being used on all nodes by running the command "modinfo enic" on all nodes:

5.        It is recommended to download the kmod-megaraid driver for higher performance. The RPM can be found in the same package at: \Linux\Storage\LSI\Cisco_Storage_12G_SAS_RAID_controller\RHEL\RHEL7.5

Install xfsprogs

From the admin node rhel1 run the command shown below to Install  xfsprogs on all the nodes for xfs filesystem:

Set Up JAVA

To setup JAVA, follow these steps:

1.        Download jdk-7u75-linux-x64.rpm and scp the rpm to admin node (rhel1) from the link  (http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html )

2.       Copy JDK rpm to all nodes:

3.        Extract and Install JDK on all nodes:

 

4.       Create the following files java-set-alternatives.sh and java-home.sh on admin node (rhel1):

5.        Make the two java scripts created above executable:

6.       Copying java-set-alternatives.sh to all nodes.

7.        Setup Java Alternatives

8.       Make sure correct java is setup on all nodes (should point to newly installed java path).

9.       Setup JAVA_HOME on all nodes.

10.    Display JAVA_HOME on all nodes.

11.     Display current java –version.

 

Configure NTP

The Network Time Protocol (NTP) is used to synchronize the time of all the nodes within the cluster. The Network Time Protocol daemon (ntpd) sets and maintains the system time of day in synchronism with the timeserver located in the admin node (rhel1). Configuring NTP is critical for any Hadoop Cluster. If server clocks in the cluster drift out of sync, serious problems will occur with HBase and other services.

1.        Configure /etc/ntp.conf on the admin node only with the following contents:

2.       Create /root/ntp.conf on the admin node and copy it to all nodes:

3.        Copy ntp.conf file from the admin node to /etc of all the nodes by executing the following commands in the admin node (rhel1):

4.       Run the following to syncronize the time and restart NTP daemon on all nodes:

5.        Make sure to restart of NTP daemon across reboots:

6.       Verify NTP is up and running in all nodes by running the following commands:

 

7.        Install the Chrony service:

8.       Activate the Chrony service at boot:

9.       Start the Chrony service:

The Chrony configuration is in the /etc/chrony.conf file, configured similar to /etc/ntp.conf.

Enable Syslog

Syslog must be enabled on each node to preserve logs regarding killed processes or failed jobs. Modern versions such as syslog-ng and rsyslog are possible, making it more difficult to be sure that a syslog daemon is present.

One of the following commands should suffice to confirm that the service is properly configured:

Set ulimit

On each node, ulimit -n specifies the number of inodes that can be opened simultaneously. With the default value of 1024, the system appears to be out of disk space and shows no inodes available. This value should be set to 64000 on every node.

Higher values are unlikely to result in an appreciable performance gain.

To set ulimit, follow these steps:

1.        For setting the ulimit on Redhat, edit  /etc/security/limits.conf on admin node rhel1 and add the following lines:

root soft nofile 64000

root hard nofile 64000

2.       Copy the /etc/security/limits.conf file from admin node (rhel1) to all the nodes using the following command:

3.        Make sure that the /etc/pam.d/su file contains the following settings:

Disable SELinux

SELinux must be disabled during the install procedure and cluster setup. SELinux can be enabled after installation and while the cluster is running.

SELinux can be disabled by editing /etc/selinux/config and changing the SELINUX line to SELINUX=disabled.

To disable SELinux, follow these steps:

1.        The following command will disable SELINUX on all nodes:

 

2.       Reboot the machine, if needed for SELinux to be disabled in case it does not take effect. It can be checked using the following command:

 

Set TCP Retries

Adjusting the tcp_retries parameter for the system network enables faster detection of failed nodes. Given the advanced networking features of UCS, this is a safe and recommended change (failures observed at the operating system layer are most likely serious rather than transitory).

To set TCP retries, follow these steps:

1.        Edit the file /etc/sysctl.conf and on admin node rhel1 and add the following lines:

2.       Copy the /etc/sysctl.conf file from admin node (rhel1) to all the nodes using the following command:

3.        Load the settings from default sysctl file /etc/sysctl.conf by running the following command:

Disable the Linux Firewall

The default Linux firewall settings are far too restrictive for any Hadoop deployment. Since the UCS Big Data deployment will be in its own isolated network there is no need for that additional firewall.

Disable Swapping

To disable swapping, follow these steps:

1.        In order to reduce Swapping, run the following on all nodes. Variable vm.swappiness defines how often swap should be used, 60 is default:

2.       Load the settings from default sysctl file /etc/sysctl.conf and verify the content of sysctl.conf:

 

Disable Transparent Huge Pages

Disabling Transparent Huge Pages (THP) reduces elevated CPU usage caused by THP.

To disable Transparent Huge Pages, follow these steps:

1.        The following commands must be run for every reboot, so copy this command to /etc/rc.local so they are executed automatically for every reboot:

2.       On the Admin node, run the following commands:

3.        Copy file to each node:

4.       Append the content of file thp_disable to /etc/rc.local:

Disable IPv6 Defaults

To disable IPv6 defaults, follow these steps:

1.        Disable IPv6 as the addresses used are IPv4:

2.       Load the settings from default sysctl file /etc/sysctl.conf:

Configure Data Drives on Name Node and Other Management Nodes

This section describes the steps to configure non-OS disk drives as RAID1 using the StorCli command. All drives are part of a single RAID1 volume. This volume can be used for staging any client data to be loaded to HDFS. This volume will not be used for HDFS data.

To configure data drives on Name node and other nodes, follow these steps:

1.        From the website download storcli https://www.broadcom.com/support/download-search/?pg=&pf=&pn=&po=&pa=&dk=storcli.

2.       Extract the zip file and copy storcli-1.14.12-1.noarch.rpm from the linux directory.

3.        Download storcli and its dependencies and transfer to Admin node:

4.       Copy storcli rpm to all the nodes using the following commands:

5.        Run this command to install storcli on all the nodes:

6.       Run this command to copy storcli64 to root directory:

7.        Run this command to check the state of the disks:

8.       If the drive state shows up as JBOD, it can be converted into Unconfigured Good using Cisco UCSM or storcli64 command. Following steps should be performed if the state is JBOD.

9.       Get the enclosure id as follows:

10.    Convert to unconfigured good:

11.      Verify status by running the following command:

12.    Run this script as root user on rhel1 to rhel3 to create the virtual drives for the management nodes:

13.     Run the following command to get enclosure id:

14.    Run MegaCli script:

15.     Run the following command. State should change to Online:

16.    State can also be verified in UCSM as show below in Equipment>Rack-Mounts>Servers>Server # under Inventory/Storage/Disk tab:

Configure Data Drives on Data Nodes

To configure non-OS disk drives as individual RAID0 volumes using StorCli command, follow these steps. These volumes will be used for HDFS Data.

1.        Issue the following command from the admin node to create the virtual drives with individual RAID 0 configurations on all the data nodes:

Configure the Filesystem for NameNodes and Datanodes

The following script formats and mounts the available volumes on each node whether it is NameNode or Data node. OS boot partition will be skipped. All drives are mounted based on their UUID as /data/disk1, /data/disk2, etc. To configure the filesystem for NameNodes and DataNodes, follow these steps:

1.        From the Admin node, create partition tables and file systems on the local disks supplied to each of the nodes, run the following script as the root user on each node:

2.       Run the following command to copy driveconf.sh to all the nodes:

3.        Run the following command from the admin node to run the script across all data nodes:

4.       Run the following from the admin node to list the partitions and mount points:

Delete Partitions

To delete a partition, follow these steps:

1.        Run the mount command (‘mount’) to identify which drive is mounted to which device /dev/sd<?>

2.       umount the drive for which partition is to be deleted and run fdisk to delete as shown below.

Cluster Verification

This section describes the steps to create the script cluster_verification.sh that helps to verify the CPU, memory, NIC, and storage adapter settings across the cluster on all nodes. This script also checks additional prerequisites such as NTP status, SELinux status, ulimit settings, JAVA_HOME settings and JDK version, IP address and hostname resolution, Linux version and firewall settings.

To verify a cluster, follow these steps:

1.        Create the script cluster_verification.sh as shown, on the Admin node (rhel1).

2.       Change permissions to executable:

3.        Run the Cluster Verification tool from the admin node. This can be run before starting Hadoop to identify any discrepancies in Post OS Configuration between the servers or during troubleshooting of any cluster / Hadoop issues:

Install HDP 3.0.1

HDP is an enterprise grade, hardened Hadoop distribution. HDP combines Apache Hadoop and its related projects into a single tested and certified package. HPD 3.0.1 components are depicted in below. This section details how to install HDP 3.0.1 on the cluster.

Prerequisites for HDP Installation

This section details the prerequisites for the HDP installation, such as setting up HDP repositories.

Hortonworks Repository

1.        From a host connected to the Internet, create a Hortonworks folder and download the Hortonworks repositories as shown below, then transfer it to the admin node.

2.       Download Hortonworks HDP repo:

3.        Download Hortonworks HDP-Utils repo:

4.       Download HDP-GPL repo:

5.        Download the Ambari repo:

6.       Copy the repository directory to the admin node (rhel1):

7.        Extract the tar ball:

8.       Create HDP repo with the following contents:

9.       Create the Ambari repo:

10.    From the admin node copy the repo files to /etc/yum.repos.d/ of all the nodes of the cluster:

Downgrade Snappy on All Nodes

Downgrade snappy on all data nodes by running this command from admin node:

HDP Installation

To install HDP, complete the following the steps:

Install and Setup Ambari Server on rhel1

1.        Run the following command in rhel1 to install ambary-server:

Install PostgreSQL in rhel2

The PostgreSQL database is used by Ambari, Hive, and Oozie services.

The rhel2 hosts the Hive and Oozie services and Ambari Server is installed on rhel1. To install, follow these steps:

1.        Log into rhel2.

2.       Install Red Hat Package Manager (RPM) according to the requirements of your operating system:

3.        Install PostgreSQL version 9.5 or later:

4.       Initialize the database as shown in the below figure by running the following command:

5.        Start PostgreSQL:

6.       Open /var/lib/pgsql/9.6/data/postgresql.conf and update to the following:

7.        Update these files on rhel2 in the location chosen to install the databases for Hive, Oozie and Ambari, using the host ip addresses:

8.       Restart PostgreSQL:

9.       Run the following:

Create Database for Ambari

To create the database for Ambari, follow these steps:

1.        Run the following commands mentioned below in bold to create and prepare database for Ambari:

2.       Restart PostgreSQL:

3.        Verify the ambari user by logging into psql:

4.       Load the Ambari Server database schema:

5.        Find the Ambari-DDL-Postgres-CREATE.sql file in the /var/lib/ambari-server/resources/ directory of the Ambari Server host after you have installed Ambari Server.

6.       Copy /var/lib/ambari-server/resources/ from rhel1 to rhel2:/tmp/.

7.        Run the following command to launch the Ambari-DDL-Postgres-CREATE.sql script:

8.       Check the table is created by running \dt command:

9.       Restart PostgreSQL:

Create Database for Hive

Run the following command as shown in bold to create and prepare database for Hive:

Create Database for Oozie

Run the following command to create and prepare database for Oozie:

Setup Ambari Server On Admin Node(Rhel1)

To setup the Ambari server, follow these steps:

1.        Install the PostgreSQL JDBC driver:

2.       Configure Ambari server to use the JDBC driver for connectivity to Ambari database in PostgreSQL:

3.        Setup Ambari Server by running the following command:

4.       Start the Ambari Server:

 

5.        To check status of Ambari Server, run the following command:

 

Launch the Ambari Server

When the Ambari service starts, access the Ambari Install Wizard through the browser. To launch the Ambari server, follow these steps:

1.        Point the browser to http://<ip address for rhel1>:8080 or http://rhel1.hdp3.cisco.com:8080

The Ambari Login screen opens.

2.       Log into the Ambari Server using the default username/password: admin/admin. This can be changed at a later period of time.

3.        When logged in the “Welcome to Apache Ambari” window opens.

Create the Cluster

To create the cluster, follow these steps:

1.        To create a cluster click “LAUNCH INSTALL WIZARD.”

2.       From the Get started page type “Cisco_HDP” for the name for the cluster.

3.        Click Next.

Select Version

To select the version, follow these steps:

1.        In the Select Version section, choose the HDP 3.0.1.0 version.

2.       Under Repositories, select “Use Local Repository.”

3.        Update the Redhat 7 HDP-3.0 URL to http://rhel1.hdp3.cisco.com/Hortonworks/HDP/centos7/3.0.1.0-187/

4.       Update the Redhat 7 HDP-3.0-GPL URL to http://rhel1.hdp3.cisco.com/Hortonworks/HDP-GPL/centos7/3.0.1.0-187/

5.        Update the Redhat 7 HDP-UTILS-1.1.0.22 to http://rhel1.hdp3.cisco.com/Hortonworks/HDP-UTILS/centos7/1.1.0.22/

Select Hosts

To build up the cluster, you need to provide the general information about how you want to set up the cluster. This requires providing the Fully Qualified Domain Name (FQDN) of each of the hosts. You also need to provide access to the private key file that was created in Set Up Password-less SSH; this is used to locate all the hosts in the system and to access and interact with them securely.

1.        Use the Target Hosts text box to enter the list of host names, one per line. Ranges inside brackets can also be used to indicate larger sets of hosts.

2.       Select the option Provide your SSH Private Key in the Ambari cluster install wizard.

3.        Copy the contents of the file /root/.ssh/id_rsa on rhel1 and paste it in the text area provided by the Ambari cluster install wizard.

4.       Click the Register and Confirm to continue.

Figure 28          Install Options

Hostname Pattern Expressions

1.        Click OK in the Host Name Pattern Expressions popup.

Confirm Hosts

Confirm Hosts helps ensure that Ambari has located the correct hosts for the cluster and checks those hosts to make sure they have the correct directories, packages, and processes to continue the install.

To confirm host, follow these steps:

1.        If any host was selected in error, remove it by selecting the appropriate checkboxes and clicking the grey Remove Selected button.

2.       To remove a single host, click the small white Remove button in the Action column.

3.        When the list of hosts is confirmed, click Next.

Choose Services

HDP is made up of a number of components. Go to Hortonworks Understand the Basics for more information.

To choose services, follow these steps:

1.        Select all to preselect all items.

2.       When you have made your selections, click Next.

Assign Masters

The Ambari installation wizard attempts to assign the master nodes for various services that have been selected to appropriate hosts in the cluster, as listed in Table 8. The right column shows the current service assignments by host, with the hostname and its number of CPU cores and amount of RAM indicated.

1.        Reconfigure the service assignments to match Table 8 shown below.

Table 8          Reconfigure the service assignments

2.       Click Next.

Assign Slaves and Clients

The Ambari install wizard attempts to assign the slave components (DataNodes, NFSGateway, NodeManager, RegionServers, Supervisor, and Client) to appropriate hosts in the cluster.

To assign slaves and clients, follow these steps:

1.        Reconfigure the service assignment to match the values shown in Table 9.

2.       Assign DataNode, NodeManager, RegionServer, and Supervisor on nodes rhel3- rhel28.

3.        Assign Client to all nodes.

4.       Click Next.

Table 9          Services and Hostnames

 

Customize Services

This section shows the tabs that manage configuration settings for Hadoop components. The wizard attempts to set reasonable defaults for each of the options here, but this can be modified to meet specific requirements. The following sections provide configuration guidance that should be refined to meet specific use case requirements.

The following changes need to be made:

·         Memory and service level settings for each component and service level tuning.

·         Customize the log locations of all the components to make sure growing logs do not cause the SSDs to run out of space.

Credentials

Specify the credentials as per your organizational policies for services in CREDENTIALS tab as shown below.

Databases

In the DATABASES tab, to configure the database for DRUID, HIVE, OOZIE, and RANGER, follow these steps:

1.        Configure DRUID as shown below.

2.       Change the default log location by finding the Log Dir property and modifying the /var prefix to /data/disk1.

3.        Configure HIVE:

a.        Select Existing PostgreSQL database in the Hive Database drop-down list.

b.       Enter Database Name as hive

c.        Enter Database Username as hive

d.       Enter Database URL as jdbc:postgresql://rhel2.hdp3.cisco.com/hive

e.       Enter Database password (use the password created during hive database setup in earlier steps; for example, bigdata)

f.         Click TEST CONNECTION to verify the connectivity

g.       In Advanced tab, Change the default log location by filtering the Log Dir property and modifying the /var prefix to /data/disk1.

h.       Change the WebHCat log directory by filtering the Log Dir property and modifying the /var prefix to /data/disk1.

4.       Configure OOZIE:

a.        Select Existing PostgreSQL database in the Hive Database drop-down list.

b.       Enter Database Name as oozie.

c.        Enter Database Username as oozie.

d.       Enter Database URL as jdbc:postgresql://rhel2.hdp3.cisco.com/oozie.

e.       Enter Database password (use the password created during hive database setup in earlier steps; for example, bigdata).

f.         Click TEST CONNECTION to verify the connectivity.

g.       In Advanced tab, Change the default log location by filtering the Log Dir property and modifying the /var prefix to /data/disk1.

5.        Configure RANGER:

a.        Select POSTGRES from DB FLAVOR drop-down list.

b.       Provide a Ranger DB name. For example, ranger.

c.        Provide Ranger DB Username such as rangeradmin.

d.       Enter JDBC connect string for a Ranger Database as jdbc:postgresql//rhel2.dhp3.cisco.com:54323/ranger.

e.       Ranger DB Host as rhel2.hdp3.cisco.com.

f.         Enter Ranger DB password. (Ranger database is not previously created. provide password string that would be configured in the DB. For example, bigdata).

g.       Select “Yes” for Setup Database and Database User.

h.       Enter “postgres” in Database Administrator (DBA) username.

i.         Enter Database Administrator (DBA) password.

j.         Enter jdbc:postgresql://rhel2.hdp3.cisco.com:5432/postgres in JDBC connect string for root user.

k.        Update Ranger Admin Log Dir from /var to /data/disk1.

HDFS

1.        In Ambari, select the HDFS Service tab and use the “Search” box to filter for the properties mentioned in Table 10 and update their values.

2.       Update the following HDFS configurations in Ambari.

Table 10       HDFS Configurations in Ambari

3.        Change the default log location by filtering the Log Dir property and modifying the /var prefix to /data/disk1.

MapReduce2

1.        In Ambari, choose the MapReduce Service tab and update the values as shown below.

2.       Under the MapReduce2 tab, change the default log location by finding the Log Dir property and modifying the /var prefix to /data/disk1.

YARN

1.        In Ambari, select the YARN Service, and update the following as shown in Table 11.

Table 11        YARN Configuration

2.       Under YARN tab, change the default log location by filtering the Log Dir property and modifying the /var prefix to /data/disk1.

 

HBase

Under the HBase tab, change the default log location by finding the Log Dir property and modifying the /var prefix to /data/disk1.

Zookeeper

Under the Zookeeper tab, change the default log location by filtering the Log Dir property and modifying the /var prefix to /data/disk1.

Storm

Under the Storm tab, change the default log location by finding the Log Dir property and modifying the /var prefix to /data/disk1.

Ambari Metrics

1.        Choose the Ambari Metrics Service and expand the General tab and make the changes shown below.

2.       Enter the Grafana Admin password as per organizational policy.

3.        Change the default log location for Metrics Collector, Metrics Monitor and Metrics Grafana by finding the Log Dir property and modifying the /var prefix to /data/disk1.

4.       Change the default data dir location for Metrics Grafana by finding the data Dir property and modifying the /var prefix to /data/disk1.

Accumulo

Select Accumulo Service and change the default log location by finding the Log Dir property and modifying the /var prefix to /data/disk1.

Atlas

Under the Atlas tab, change the default log location by finding the Log Dir property and modifying the /var prefix to /data/disk1.

Kafka

Under the Kafka tab, change the default log location by finding the Log Dir property and modifying the /var prefix to /data/disk1.

Knox

1.        Select the Knox Service tab and expand the Knox gateway tab and make the changes shown below.

2.       Enter the Knox Master Secret password as per organizational policy.

3.        For Knox, change the gateway port to 8444 to ensure no conflicts with local HTTP server.

SmartSense

The SmartSense account requires the Hortonworks support subscription. Subscribers can populate the properties as shown below.

Spark

Select the Spark tab, change the default log location by finding the Log Dir property and modifying the /var prefix to /data/disk1.

Review

The assignments that have been made are displayed. Check to make sure all is correct before clicking the Deploy button. If any changes are necessary, use the left navigation bar to return to the appropriate screen.

Deploy

When the review is complete, click the DEPLOY button.

Follow the installation process. Watch for warnings and failure by clicking on the link as shown below.

Summary of the Installation Process

On the Summary page click “COMPLETE.”

High Availability for HDFS NameNode and YARN ResourceManager

High availability for NameNode and YARN Resource Manager can be configured using Ambari or also on non-Ambari clusters. This deployment guide covers the configuration of high availability using Ambari – Use the Ambari wizard interface to configure HA of the components.

The Ambari web UI provides a wizard-driven user experience that allows to configure high availability of the components in many Hortonworks Data Platform (HDP) stack services. The high availability of the components are achieved by setting up primary and secondary components. In the event that the primary component fails or becomes unresponsive, services failover to secondary component. After configuring the high availability for a service, Ambari enables you to manage and disable (roll back) the high availability of components within that service.

Configure the HDFS NameNode High Availability

The HDFS NameNode high availability feature enables you to run redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby. This eliminates the NameNode as a potential single point of failure (SPOF) in an HDFS cluster. With the release of Hadoop 3.0, you can configure more than one backup NameNode.

Prior to the release of Hadoop 2.0, the NameNode represented a single point of failure (SPOF) in an HDFS cluster. Each cluster had a single NameNode, and if that machine or process became unavailable, the cluster as a whole would be unavailable until the NameNode was either restarted or brought up on a separate machine. This situation impacted the total availability of the HDFS cluster in two major ways:

·         In the case of an unplanned event such as a machine crash, the cluster would be unavailable until an operator restarted the NameNode.

·         Planned maintenance events such as software or hardware upgrades on the NameNode machine would result in periods of cluster downtime.

HDFS NameNode HA avoids this by facilitating either a fast failover to one or more backup NameNodes during machine crash, or a graceful administrator-initiated failover during planned maintenance.

Active NameNode honors all the client requests and the Standby NameNode acts as a backup. The Standby NameNode keeps its state synchronized with Active NameNode through a group of JournalNodes(JNs). When the Active node performs any namespace modification, the Active node durably logs a modification record to a majority of these JNs. The Standby node reads the edits from the JNs and continuously watches the JNs for changes to the edit log.

Prerequisites for NameNode High Availability

The following are the prerequisites for NameNode high availability:

·         NameNode Machine: Hardware for Active and Standby node should be exactly identical.

·         JournalNodes Machine: JournalNode daemon is relatively lightweight, therefore it can be co-located on machines with other Hadoop daemons; it is typically located on the management nodes.

·         There MUST be at least three JournalNodes, because the edit log modifications must be written to a majority of JNs. This allows the system tolerate failure of a single machine. You may also run more than three JournalNodes, but in order to increase the number of failures that the system can tolerate, you must run an odd number of JNs (3, 5, 7, and so on).

·         ZooKeeper Machines: For automatic failover capability, an existing Zookeeper cluster must exist. The Zookeeper service can also co-exist with other Hadoop daemons.

·         In HA Cluster, the Standby NameNode also performs the checkpoints of the namespace state, therefore do not deploy a Secondary NameNode, CheckpointNode, or BackupNode in an high availability cluster.

Deploy the NameNode High Availability Cluster

To deploy the NameNode high availability on a Ambari managed cluster, follow these steps:

1.        Log into Ambari. Click HDFS > CONFIGS. Click the ACTIONS drop-down list and click Enable NameNode HA to launch the wizard.

Figure 29          Enable NameNode HA

2.       Step 1 launches the Enable NameNode HA wizard. On the Get Started page, specify the Nameservice ID as shown below. Click Next.

Figure 30           Enable NameNode HA Wizard – Get Started

3.        On the Select Hosts page, select the Additional NameNode and JournalNode. Click Next.

Figure 31           Enable NameNode HA Wizard – Select Hosts

4.       On the Review page, confirm the selection. To change any values, click Back, or to continue click Next.

Figure 32           Enable NameNode HA Wizard – Review

5.        Create a checkpoint on the NameNode on the linux server (rhel1.hdp3.cisco.com) as shown below.

Figure 33           Enable NameNode HA Wizard – Create Checkpoint

6.       SSH to current NameNode, rhel1.hdp3.cisco.com and run the following commands:

Figure 34           Current NameNode – Safe Mode and Create Checkpoint Command

7.        Return to the Ambari web UI, verify that the Checkpoint was created. Click Next.

8.       See the progress bar on the Configure Components page. When the configuration steps are completed, click Next.

Figure 35           Enable NameNode HA Wizard – Configure Components

9.       Initialize the JournalNodes as shown below.

Figure 36           Enable NameNode HA Wizard – Initialize JournalNodes

10.    SSH to the current NameNode, for example rhel1.hdp3.cisco.com.

11.     Run the following command:

Figure 37           Initialize JournalNodes

12.    Return to the Ambari UI, when Ambari detects success, click Next.

13.     On the Start Components page, when completed, click Next.

Figure 38           Enable NameNode HA Wizard – Start Components

14.    On the Initialize Metadata page, add the information as shown below.

Figure 39           Enable NameNode HA Wizard – Initialize Metadata

15.     SSH to rhel1.hdp3.cisco.com and run the following command:

Figure 40          Initialize the Metadata for NameNode

16.    SSH to an additional NameNode, for example, rhel3.hdp3.cisco.com and run the following command:

Figure 41           Initialize the Metadata for Additional NameNode

17.     Return to the Ambari web UI, click NEXT. Click OK on the confirmation pop-up window. Make sure the initialization of metadata was performed in NameNode and an additional NameNode as mentioned in step 15 and 16.

Figure 42          Enable NameNode HA Wizard – Confirmation for Initialize the Metadata

18.    On the Finalize HA Setup page, you can see the progress bar as the high availability completes.

Figure 43           Enable NameNode HA Wizard – Finalize High Availability Setup

19.    Click COMPLETE when done.

20.    Click HDFS > SUMMARY tab, verify the Active and Standby NameNode. The Quick Links pane also shows that rhel1.hdp3.cisco.com is running the Active NameNode and rhel3.hdp3.cisco.com is running in Standby NameNode.

Figure 44          Ambari – HDFS – Summary Information

Configure the YARN ResourceManger HA

This section provides instructions on setting up the ResourceManager (RM) HA feature in a HDFS cluster. The Active and Standby ResourceManagers embed the ZooKeeper based ActiveStandbyElector to determine which RM should be active.

Prerequisites for ResourceManager HA

·         The servers where Active and Standby RMs are run should have identical hardware.

·         For automated failover configurations, there must be an existing Zookeeper cluster available. The ZooKeeper service nodes can be co-located with the other Hadoop daemons.

Deploy the ResourceManager HA

ResourceManager HA can be configured manually or through Ambari. These instructions are based on configuring ResourceManager HA using Ambari.

To setup ResourceManager HA, follow these steps:

1.        From the Ambari web UI, click Services > YARN. Click the ACTIONS drop-down list and select Enable ResourceManager HA.

Figure 45           Services/YARN - Enable ResourceManger HA

 

2.       This launches the ResourceManager HA wizard as shown below. Click NEXT.

Figure 46          Enable ResourceManager HA Wizard – Get Started

3.        From the Select Host page, select the host for Additional Resource Manager. Click NEXT.

Figure 47           Enable ResourceManger HA Wizard – Select Host

4.       Proceed to the Review page.

Figure 48          Enable ResourceManager HA Wizard - Review

5.        The Configure Components page shows the progress bar as the Additional ResourceManager is being deployed.

Figure 49          Enable ResourceManager HA Wizard – Configure Components

6.       Click COMPLETE when done.

7.        Verify the ResourceManger HA setup by clicking Services > YARN > SUMMARY tab. The Quick Links pane identifies Active and Standby ResourceManager.

Figure 50           Services/YARN/Summary – Verify ResourceManger HA

HDP Post OS Deployment – Enable GPU Isolation and Scheduling

One of the prerequisites for HDP to enable GPU isolation and scheduling, is CUDA. CUDA itself also has prerequisites. The order of installation is as follows:

·         Install CUDA prerequisites

·         CUDA

·         Cisco UCS C480 ML M5 resource inventory as seen through Cisco UCS Manager

Install the Prerequisites for CUDA

To install the prerequisites for CUDA, follow these steps:

 

1.        List GPUs and CPUs installed:

 

 

Install GCC

To install GCC, follow these steps:

1.        Make sure gcc is installed in the system:

Install Kernel Headers and Installation Packages

The CUDA Driver requires that the kernel headers and development packages for the running version of the kernel are installed at the time of the driver installation, as well as whenever the driver is rebuilt. For example, if your system is running kernel version 3.17.4-301, the 3.17.4-301 kernel headers and development packages must also be installed.

 

 

Install DKMS

The NVIDIA driver RPM packages depend on other external packages, such as DKMS. Those packages are only available on third-party repositories, such as EPEL.

To install DKMS, follow these steps:

http://rpmfind.net/linux/rpm2html/search.php?query=dkms  for RHEL 7.x

RHEL 7.x http://rpmfind.net/linux/epel/7/x86_64/Packages/d/dkms-2.6.1-1.el7.noarch.rpm

1.        Copy dkms rpm to all the GPU servers:

2.       Install dkms with yum install:

Install NVIDIA GPU Drivers

To install the NVIDIA GPU drivers, follow these steps:

1.        Download this NVIDIA GPU driver from http://www.nvidia.com/Download/index.asp?lang=en-us

2.       For the NVIDIA driver download, select the product type, Series, Product, OS, and CUDA toolkit.

3.        Click SEARCH. The selected driver is shown below.

4.       Click DOWNLOAD. The download link can be captured by right-clicking AGREE & DOWNLOAD as shown below.

5.        Copy the .rpm file in all the GPU nodes as shown below.

6.       Install the driver by running the following command:

Install CUDA

To install CUDA, follow these steps:

1.        Download CUDA 9.2.

The latest version of CUDA is available here:

https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=RHEL&target_version=7&target_type=rpmlocal

2.       Copy .rpm file to all GPU nodes as follows:

3.        Install CUDA in all GPU nodes by running the following set of commands:

Download and Setup NVIDIA CUDA Deep Neural Network library (cuDNN)

Download cuDNN 9.2

1.        Download cuDNN from https://developer.nvidia.com/cudnn  for the same CUDA version.

2.       Copy cuNN into all the GPU servers.

https://developer.nvidia.com/rdp/cudnn-archive

3.        Run the following Ansible commands in all GPU nodes to setup cuDNN:

Post Installation Steps

1.        Add CUDA in PATH and LD_LIBRARY_PATH variable in all the GPU nodes.

2.       The PATH  and LD_LIBRARY_PATH variable need to include /usr/local/<cuda>/bin:

3.        Reboot the GPU nodes:

 

Verify Drivers

To verify the drivers have been installed, run the following commands in all GPU nodes as shown below:

 

 

Install and Configure Docker

Prerequisites

Docker Version

Docker only supports systems with modern kernel as it is highly dependent on Linux OS kernel version and its capabilities.

YARN requires Docker should be setup and installed in all NodeManger hosts where Docker containers will run. It is important to consider these recommendations before installing Docker to be used by YARN.

Docker 1.12.5 is required at a minimum. However, it is recommended to use the latest version.

Storage Driver

It was observed through testing that device mapper using LVM is generally stable. However, high writes to the container root file system, device mapper exhibit panics. Overlay and Overlay2 perform significantly better than device mapper and recommended. In this reference architecture Overlay2 has been used.

Networking

YARN has support for running Docker containers on a user specified network, however, it does not manage the Docker networks. Administrators are expected to create the networks prior to running the containers. Node labels can be used to isolate particular networks. It is vital to read and understand the Docker networking documentation. Swarm based options are not recommended, however, overlay networks can be used if setup using an external store, such as etcd.

YARN will ask Docker for the networking details, such as IP address and hostname. As a result, all networking types are supported. Set the environment variable  YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK  to specify the network to use.

Host networking is only recommended for testing. If the network where the NodeManagers are running has a sufficient number of IP addresses. The bridge networking with --fixed-cidr option works well. Each NodeManager is allocated a small portion of the larger IP space, and then allocates those IP addresses to containers.

To use an administrator defined network, add the network to docker.allowed.networks  in container-executor.cfg  and yarn.nodemanger.runtime.linux.docker.allowed-container-networks in yarn-site.xml .

Image Management

Images can be preloaded on all NodeManager hosts or they can be implicitly pulled at runtime if they are available in a public Docker registry, such as Docker hub. If the image does not exist on the NodeManager and cannot be pulled, the container will fail.

Install Docker

To install Docker, follow these steps:

1.        Uninstall any previous version of Docker completely by running the following command:

2.       Remove any existing Docker repository:

3.        Docker has been released as part of Extra channel in Red Hat Enterprise Linux. When the extra channel has been enabled, docker packages can be installed. To register and subscribe a system to Red Hat customer portal, attached the required pool, and enable the extras channel, run the following commands:

4.       Install Docker by running the following command:

5.        Start the Docker service:

6.       Verify Docker by running the following command:

Install nvidia-docker in GPU Nodes

To install nvidia-docker in GPU nodes, follow these steps:

1.        In order for the Docker container to see GPU as a resource, you need to install nvidia-docker v1 plugin.

2.       Add the package repositories:

3.        Install nvidia-docker v1:

4.       Start the nvidia-docker service:

5.        Test nvidia-smi with the latest CUDA image. SSH to the node with GPU and run the following command:

 

6.       Verify that the /var/lib/nvidia-docker/volumes/nvidia_driver/$NVIDIA_DRIVER_VERSION/ directory was created:

 

7.        Run the following command to verify the nvidia-docker plugin:

 

8.       Use the following Docker command to verify that HDP can access the GPU:

 

Configure Docker

After Docker and nvidia-docker are installed, the following process is the minimum recommended configuration. To configure Docker, follow these steps:

1.        Edit /etc/docker/daemon.json and add the following options:

2.       Restart Docker service:

Configure YARN to Running Docker Containers

YARN provides isolation through the use of cgroups. Docker also has cgroup management built in. If isolation through cgroups if desired, the only recommended solution is to use YARN’s cgroup management at this time. YARN will create the cgroup hierarchy and set the --cgroup-parent flag when launching the container.

The cgroupdriver must be set to cgroupfs in all datanodes where Docker is running. Ensure that Docker is running using the --exec-opt native.cgroupdriver=cgroupfs docker daemon option.

To configure YARN to run Docker containers, follow these steps:

1.        Find and fix the cgroupdriver. If it is not there, it should be added.

 

2.       The version of Docker may include oci-hooks that expect to use the systemd cgroup driver. Search for oci on your system and remove these files. For example:

3.        If it exists, run the following:

 

4.       If it exists, remove the file:

Enable Cgroups

To set up the CGroup hierarchy, run the following command:

1.        Create the following script in admin node:

2.       Copy the script in all datanodes and run the script as shown below:

3.        To enable cgroups on an Ambari cluster, select YARN > CONFIGS on the Ambari dashboard, then click CPU Isolation under CPU. Click Save.

4.       Enable Docker Runtime by selecting YARN > CONFIGS > SETTINGS on the Ambari dashboard as shown below.

5.        Set the following properties for Advanced yarn-site in Ambari.

Table 12       Advanced yarn-site Properties

6.       Configure the following isolation properties in YARN. In Ambari, click YARN > CONFIGS > ADVANCED. Properties can also be configured in the /etc/hadoop/conf/yarn-site.xml on the ResourceManager and NodeManager nodes. Configure and verify properties listed in Table 13 in Ambari so that changes get applied to all the respective nodes.

Table 13        YARN – Isolation Properties

7.        Configure the following properties for Container Executor in YARN. Some properties are pre-set as per YARN recommendation.

Table 14       YARN – Container Executor Properties

8.       The administrator must define the volume whitelist in container-executor.cfg by setting docker.allowed.ro-mounts and docker.allowed.rw-mounts to the list of parent directories that are allowed to be mounted.

The application submitter requests the required volumes at application submission time using the YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS environment variable.

CPU jobs are constrained with CPU scheduling and cgroups enabled, but by default these are flexible limits. If spare CPU cycles are available, containers are allowed to exceed the CPU limits set for them. With flexible limits, the amount of CPU resources available for containers to use can vary based on cluster usage; the amount of CPU available in the cluster at any given time.

You can use cgroups to set strict limits on CPU usage. When strict limits are enabled, each process receives only the amount of CPU resources it requests. With strict limits, a CPU process will receive the same amount of cluster resources every time it runs.

Strict limits are not enabled (set to false) by default for property  yarn.nodemanager.linux-container-executor.cgroups.strict-resourceusage as shown above.

9.       Set the Percentage of CPU used by YARN. Set the percentage of CPU that can be allocated for YARN containers. In most cases, the default value of 100 percent should be used. If you have another process that needs to run on a node that also requires CPU resources, you can lower the percentage of CPU allocated to YARN to free up resources for the other process.

1

10.    Click SAVE. Restart All Affected Services as shown below.

Run Docker on YARN Using the YARN Service API

To deploy the web server using YARN service API, follow these steps:

1.        Create the following Yarn file and save it to /tmp/httpdserver.json.

2.       Submit the HTTP application using the YARN CLI:

 

3.        Get the IP address httpd container and the node where it is scheduled by running the following command:

4.       Verify the application through YARN ResourceManager web UI:

5.        SSH to rhel7 and using curl, perform a GET request to the HTTP server on port 8080 using the IP address obtained in previous step:

 

6.       Run the following command to verify if Docker container is provisioned:

 

Run Docker Container with GPU on YARN

A GPU is a specialized processor that can be used to accelerate highly parallelized computationally-intensive workloads. Because of their computational power, GPUs have been found to be particularly well-suited to deep learning workloads. Ideally, CPUs and GPUs should be used in tandem for data engineering and data science workloads. A typical machine learning workflow involves data preparation, model training, model scoring, and model fitting. You can use existing general-purpose CPUs for each stage of the workflow, and optionally accelerate the math-intensive steps with the selective application of special-purpose GPUs. For example, GPUs allow you to accelerate model fitting using frameworks such as TensorFlow, Caffe, PyTorch, Keras, MXNet, and Microsoft Cognitive Toolkit (CNTK).

By enabling GPU support, data scientists can share GPU resources available on HDP nodes. Users can request a specific number of GPU instances, up to the total number available on a node, which are then allocated to the running session or job for the duration of the run.

Prerequisites

·         Currently, only NVIDIA GPUs are supported by YARN

·         NVIDIA drivers must be pre-installed in YARN node managers

When Docker is used as container runtime context, nvidia-docker 1.0 needs to be installed (the current supported version is YARN for nvidia-docker).

Enable GPU through Ambari

This section provides instructions about how to enable GPU support in HDP through Ambari.

If you have HDP cluster where datanodes with GPU installed and datanodes without GPU installed, you must create host groups. Enabling GPU in Ambari for non-GPU nodes will complain on restarting NodeManager service. Therefore, it is important to separate out the GPU nodes configuration by creating host group for GPU nodes.

Ambari initially assigns all hosts in your cluster to one default configuration group for each service you install. For example, after deploying a three-node cluster with default configuration settings, each host belongs to one configuration group that has default configuration settings for the HDFS service.

Create a New Host Configuration Group

To create host group, follow these steps:

1.        Click a service name, for example, here we need to create host group for YARN. Click YARN > CONFIGS.

2.       In Configs, click Manage Config Groups.

3.        Click + icon on the left pane to create new configuration group. Create New Configuration Group window will pop-up as shown below. Provide a name to the configuration group. For example, GPU-Nodes

4.       Click the newly created configuration group in the left pane and click + icon to add hosts to this group as shown below. Select hosts that has GPU and leave the non-GPU nodes in default configuration group.

5.        Click the check box on the hosts that have GPU installed as shown below. Click OK when done.

6.       Click SAVE to save the configuration group as shown below.

Enable GPU

To enable GPU, follow these steps:

1.        Under default config group. In the GPU section, provide absolute path for nvidia-smi as shown below.

2.       Select the GPU Nodes configuration group from Config group drop-down list created in the earlier steps as shown below. Select YARN > CONFIGS.

3.        Click Enabled for GPU.

4.       Verify Advanced yarn-site with the following:

5.        Click ADVANCED tab and verify the config for yarn.resource-types in Resource Types section as shown below.

6.       Verify and/or apply the following settings for Container Executor in YARN config for GPU-nodes configuration group.

Table 15        YARN – Container Executor for GPU Properties

 

7.        Click SAVE and RESTSRT for all affected services in Ambari.

Verify YARN Configurations

All of the configurations make changes in /etc/hadoop/conf/yarn-site.xml and /etc/hadoop/conf/container-executor.cfg file. To verify the YARN configurations, follow these steps:

1.        For Non-GPU nodes, contents of /etc/hadoop/conf/container-executor.cfg file are in this reference design.

2.       For GPU nodes, contents of /etc/hadoop/conf/container-executor.cfg file are in this reference design.

3.        Launch the YARN Web UI and verify cluster overview as shown below.

4.       Click Nodes and select a node as shown below.

Setting Up Docker Registry

Private trusted registry is required to provision YARN container. This topic provides basic information about deploying and configuring a registry.

Designate a Server for Docker and Start the Registry

To designate a server for Docker and start the registry, follow these steps:

1.        Designate a server in the cluster for the Docker registry. Minimal resources are required, but sufficient disk space is needed to store the images and metadata. Docker must be installed and running.

2.       Optional: By default, data will only be persisted within the container. If you would like to persist the data on the host, you can customize the bind mounts using the -v option.

3.        Create /var/lib/registry folder:

4.       Configure Docker to allow pulling from this insecure registry. Modify /etc/docker/daemon.json on all nodes in the cluster to include the following configuration options:

5.        Restart Docker on all nodes.

6.       Provision the registry container by running the following command:

7.        Verify the registry container is provisioned by running docker ps command:

Test the Docker Registry

To test the Docker registry, follow these steps:

1.        Pull the Docker image as shown in below:

2.       Tag the image:

3.        Push the image into private registry:

Apache Hadoop YARN Distributed Shell

To run the YARN distributed shell with docker container having GPU as a resource, follow these steps:

1.        Use the following command to run the distributed shell and GPU without a Docker container:

2.       Use the following command to run the distributed shell and GPU with a Docker container:

3.        On the YARN web UI, find the application and the node where it is provisioned as shown below and click prelaunch.out for _000002 container

4.       As shown in the figure above, the container is running in rhel14 node. ssh to rhel14 and run the following command:

Run TensorFlow Container Using YARN Distributed Shell

In this example, Nvidia GPU Cloud (NGC) is used as the repository for AI frameworks Docker image. The TensorFlow Docker container image is pulled from NGC which is aligned to the CUDA installed on the server pulling the image.

To deploy TensorFlow container with YARN distributed shell, follow these steps:

1.        Pull the tensorflow docker image as shown below:

2.       Tag the image:

3.        Push the image to private registry:

4.       Run YARN distributed shell to provision tensorflow docker container in GPU nodes:

 

Below is the output of this command. Output has been truncated for simplicity:

5.        Log into YARN Web UI and click the Applications tab as shown below.

6.       Click the application_1547666438624_0011 for details about the application, such as where it is provisioned by YARN. As shown below, this particular container is provisioned in rhel20.hdp3.cisco.com by YARN.

7.        Click rhel20.hdp3.cisco.com. Detailed node information is displayed as shown below. Under Node Information tab, panel Resource yarn.io/gpu shows 6 GPUs is used and 2 GPUs are available. In the previous YARN distributed shell command mentioned in step 4, 6 GPUs were requested for the container.

8.       Click List of Containers on this Node. This displays the containers provisioned in this node, as shown below.

9.       Click prelaunch.out to see the output of the container, as shown below

This section provides the BOM for the 24 Nodes Hadoop Base Rack and 8 Nodes Hadoop Expansion Rack. See Table 16 for the BOM for the Hadoop Base rack, Table 17 for BOM for Hadoop Expansion Rack, Table 18 for BOM for Hadoop GPU Rack, Table 19 for Red Hat Enterprise Linux License, and Table 20 lists Cloudera SKUs available from Cisco.

Table 16       Bill of Materials for Cisco UCS C240 M5 SX Hadoop Nodes Base Rack

Table 17        Bill of Materials for Hadoop Nodes Expansion Rack

Table 18       Bill of Materials for Cisco UCS C480 ML Nodes Rack

Table 19       Red Hat Enterprise Linux License

Table 20       Cisco SKUs with Cisco PID and Product Names

Muhammad Afzal, Engineering Architect, Computing Systems Product Group, Cisco Systems, Inc.

Muhammad Afzal is an Engineering Architect and Technical Marketing Engineer in Cisco UCS Product Management and Datacenter Solutions Engineering. He is currently responsible for designing, developing, and producing validated architectures for Big Data and analytics while working collaboratively with product partners. Previously, Afzal had been a lead architect for various cloud and data center solutions in Solution Development Unit at Cisco. Prior to this, Afzal has been a Solutions Architect in Cisco’s Advanced Services group, where he worked closely with Cisco's large enterprise and service provider customers delivering data center and cloud solutions. Afzal holds an MBA in Finance and a BS in Computer Engineering.

Acknowledgements

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, we would like to acknowledge the following for their significant contribution and expertise that resulted in developing this document:

·         Karthik Kulkarni, Architect, Computing Systems Product Group, Cisco Systems, Inc.

·         Silesh Bijjahalli, Product Management, Computing Systems Product Group, Cisco Systems, Inc.

·         Ali Bajwa, Hortonworks

·         Dhananjay Mehta, Hortonworks

·         Harsh Shah, Hortonworks

·         Wangda Tan, Hortonworks

·         Sicong Ji, NVIDIA

·         Chetan Tekur, NVIDIA