Guest

Cisco UCS Common Platform Architecture Version 2 (CPA v2) for Big Data with Comprehensive Data Protection using Intel Distribution for Apache Hadoop

  • Viewing Options

  • PDF (14.0 MB)
  • Feedback
Cisco UCS Common Platform Architecture Version 2 (CPA v2) for Big Data with Comprehensive Data Protection using Intel Distribution for Apache Hadoop

Table Of Contents

About the Authors

Acknowledgment

About Cisco Validated Design (CVD) Program

Cisco UCS Common Platform Architecture Version 2 (CPA v2) for Big Data with Comprehensive Data Protection using Intel Distribution for Apache Hadoop

Audience

Introduction

Cisco UCS Common Platform Architecture for Big Data

Intel Distribution for Apache Hadoop (ID)

ID - Key Features and Benefits

Access controls and Data protection options

Intel Advanced Encryption Standard New Instructions (AES-NI)

Kerberos Authentication (secure mode)

Role based Access Control

Hive and Pig

Solution Overview

Rack and PDU Configuration

Server Configuration and Cabling

Software Distributions and Versions

Intel Distribution for Apache Hadoop (ID)

RHEL

Software Versions

Fabric Configuration

Performing Initial Setup of Cisco UCS 6296 Fabric Interconnects

Configure Fabric Interconnect A

Configure Fabric Interconnect B

Logging Into Cisco UCS Manager

Upgrading UCSM Software to Version 2.1(1e)

Adding Block of IP Addresses for KVM Access

Editing The Chassis Discovery Policy

Enabling The Server Ports and Uplink Ports

Creating Pools for Service Profile Templates

Creating an Organization

Creating MAC Address Pools

Configuring VLANs

Creating a Server Pool

Creating Policies for Service Profile Templates

Creating a Host Firmware Package Policy

Creating QoS Policies

Creating the Best Effort Policy

Creating a Platinum Policy

Setting Jumbo Frames

Creating a Local Disk Configuration Policy

Creating a Server BIOS Policy

Creating a Boot Policy

Creating a Service Profile Template

Configuring the Network Settings for the Template

Configuring a Storage Policy for the Template

Configuring a vNIC/vHBA Placement for the Template

Configuring a Server Boot Order for the Template

Configuring Server Assignment for the Template

Configuring Operational Policies for the Template

Configuring Disk Drives for Operating System on NameNode

Configuring Disk Drives for Operating System on DataNodes

Installing Red Hat Linux 6.4 with KVM

Post OS Install Configuration

Setting Up Password-less Login

Installing and Configuring Parallel SSH

Installing Parallel-SSH

Installing Cluster Shell

Configuring /etc/hosts and DNS

Creating RedHat Local Repository

Upgrading LSI driver

Installing httpd

Enabling Syslog

Setting Ulimit

Disabling SELinux

Setting TCP Retries

Disabling the Linux Firewall

Configuring Data Drives on NameNode

Configuring the Filesystem for NameNodes

Configuring Data Drives on DataNodes

Configuring the Filesystem for DataNodes

Installing Intel Distribution for Apache Hadoop

Prerequisites for ID Installation

Copy Intel Distribution to Admin Node

Create Intel Repo

Kerberos Setup

Install OpenLDAP and Integrate with Kerberos

Install Kerberos Server

Import the Kerberos Schema and Setup LDAP Identities for Kerberos

Edit Kerberos Configuration files

Create KDC Entries in LDAP

Set up KDC Logging and Starting the KDC Daemons

Generating KeyTab Files

ID Installation

Role Assignment

Installing Intel Manager

Installing Intel Distribution through Intel Manager

Troubleshooting Installation and Deployment

Kerberos (Secure Mode) Install

Memory Configuration

HDFS

MapReduce

HBase

Starting all the Services

Post ID Installation

Integration with Existing Identity Store for Access Control

Pig and Hive with Encryption

Conclusion

Bill of Material


Cisco UCS Common Platform Architecture Version 2 (CPA v2) for Big Data with Comprehensive Data Protection using Intel Distribution for Apache Hadoop

Building Architectures to Solve Business Problems

About the Authors

Raghunath Nambiar, Cisco Systems

Raghunath Nambiar is a Distinguished Engineer at Cisco's Data Center Business Group. His current responsibilities include emerging technologies and big data strategy.

Suyash Ramineni, Intel Corporation

Suyash Ramineni is a Software Engineer in the Software and Services Group at Intel. He's part of the Pre-sales Engineering team focusing on Integration with Partner products.

Karthik Kulkarni, Cisco Systems

Karthik Kulkarni is a Technical Marketing Engineer at Cisco Data Center Business Group focusing on Big Data and Hadoop technologies.

Acknowledgment

The authors acknowledge contributions of Ashwin Manjunatha, and Sindhu Sudhir in developing the Cisco UCS Common Platform Architecture Version 2 (CPA v2) for Big Data with Comprehensive Data Protection using Intel Distribution for Apache Hadoop Cisco Validated Design.

About Cisco Validated Design (CVD) Program


The CVD program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information visit:

http://www.cisco.com/go/designzone

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2013 Cisco Systems, Inc. All rights reserved.

Cisco UCS Common Platform Architecture Version 2 (CPA v2) for Big Data with Comprehensive Data Protection using Intel Distribution for Apache Hadoop


Audience

This document describes the architecture and deployment procedures of Intel Distribution for Apache Hadoop on a 64 node cluster based Cisco UCS Common Platform Architecture Version 2 (CPA v2) for Big Data. The intended audience of this document includes, but is not limited to, sales engineers, field consultants, professional services, IT managers, partner engineering and customers who want to deploy Intel Distribution for Apache Hadoop on the Cisco UCS CPA v2 for Big Data.

Introduction

Hadoop has become a strategic data platform embraced by mainstream enterprises as it offers the fastest path for businesses to unlock value in big data while maximizing existing investments. The Intel Distribution for Apache Hadoop is a 100% open source distribution of Apache Hadoop that is truly enterprise grade having been built, tested and hardened with enterprise rigor. The combination of ID and Cisco UCS provides industry-leading platform for Hadoop based applications.

Cisco UCS Common Platform Architecture for Big Data

Cisco UCS Common Platform Archtiecture (CPA) is a popular big data solution. It has been widely adopted for finance, healthcare, service provider, entertainment, insurance, and public-sector environments. The new Cisco UCS CPA Version 2 (v2) for Big Data improves both performance and capacity. With complete, easy-to-order packages that include computing, storage, connectivity, and unified management features, Cisco UCS CPA v2 for Big Data helps enable rapid deployment, delivers predictable performance, and reduces total cost of ownership (TCO). Cisco UCS CPA v2 for Big Data offers:

Cisco UCS servers with the versatile Intel® Xeon® E5-2600 v2 product family

Transparent cache acceleration option with Cisco UCS Nytro MegaRAID technology

Unified management and unified fabric across enterprise applications.

The Cisco UCS solution for Intel Distribution for Apache Hadoop is based on Cisco Common Platform Architecture (CPA) for Big Data, a highly scalable architecture designed to meet a variety of scale-out application demands with seamless data integration and management integration capabilities built using the following components:

Cisco UCS 6200 Series Fabric Interconnects—provide high-bandwidth, low-latency connectivity for servers, with integrated, unified management provided for all connected devices by Cisco UCS Manager. Deployed in redundant pairs, Cisco fabric interconnects offer the full active-active redundancy, performance, and exceptional scalability needed to support the large number of nodes that are typical in clusters serving Big Data applications. Cisco UCS Manager enables rapid and consistent server configuration using service profiles and automation of the ongoing system maintenance activities such as firmware updates across the entire cluster as a single operation. Cisco UCS Manager also offers advanced monitoring with options to raise alarms and send notifications about the health of the entire cluster.

Cisco UCS 2200 Series Fabric Extenders—extends the network into each rack, acting as remote line cards for fabric interconnects and providing highly scalable and extremely cost-effective connectivity for a large number of nodes.

Cisco UCS C-Series Rack-Mount Servers—Cisco UCS C240M3 Rack-Mount Servers are 2-socket servers based on Intel Xeon E-2600 v2 series processors and supporting up to 768 GB of main memory. 24 Small Form Factor (SFF) disk drives are supported in performance optimized option and 12 Large Form Factor (LFF) disk drives are supported in capacity option, along with 4 Gigabit Ethernet LAN-on-motherboard (LOM) ports.

Cisco UCS Virtual Interface Cards (VICs)—the unique Cisco UCS Virtual Interface Cards incorporate next-generation converged network adapter (CNA) technology from Cisco, and offer dual 10Gbps ports designed for use with Cisco UCS C-Series Rack-Mount Servers. Optimized for virtualized networking, these cards deliver high performance and bandwidth utilization and support up to 256 virtual devices.

Cisco UCS Manager—resides within the Cisco UCS 6200 Series Fabric Interconnects. It makes the system self-aware and self-integrating, managing the system components as a single logical entity. Cisco UCS Manager can be accessed through an intuitive graphical user interface (GUI), a command-line interface (CLI), or an XML application-programming interface (API). Cisco UCS Manager uses service profiles to define the personality, configuration, and connectivity of all resources within Cisco UCS, radically simplifying provisioning of resources so that the process takes minutes instead of days. This simplification allows IT departments to shift their focus from constant maintenance to strategic business initiatives.

Intel Distribution for Apache Hadoop (ID)

Intel Distribution for Apache Hadoop software (Intel Distribution) is a software platform that provides distributed data processing and data management for enterprise applications that analyze massive amounts of diverse data. The Intel Distribution includes Apache Hadoop and other software components with enhancements from Intel. Proven in production at some of the most demanding enterprise deployments in the world, the Intel Distribution of Hadoop is supported by experts at Intel with deep optimization experience in the Apache Hadoop software stack as well knowledge of the underlying processor, storage, and networking components.

Figure 1 Intel Distribution for Apache Hadoop 2.x Stack

ID - Key Features and Benefits

Intel has developed a solution for Big Data that includes a feature enhanced controlled distribution of Apache Hadoop, with optimizations for better hardware performance, and services to streamline deployment and improve the end user experience.

The Intel distribution of Hadoop includes:

The Intel Manager to install, configure, monitor and administer the Hadoop cluster.

Enhancements to HBase and Hive for improved real time query performance and end user experience.

Resource monitoring capability using Nagios and Ganglia in the Intel Manager.

Superior security and performance through better integrated encryption and compression.

Packaged Hadoop ecosystem that includes HBase, Hive and Pig among other tools.

This solution provides a foundational platform for Intel to offer additional solutions as the Hadoop ecosystem evolves and expands.

Aside from the Hadoop core technology (HDFS, MapReduce, etc.) Intel has designed additional capabilities to address specific customer needs for Big Data applications such as:

Optimally Installing and configuring the Hadoop cluster.

Monitoring, reporting, and alerting of the hardware and software components.

Providing Job level metrics for analyzing specific workloads deployed in the cluster.

Infrastructure configuration automation.

Extensions to HBase and Hive to improve real time transactional performance and features.

Enhancements to security and access control with better encryption and decryption capabilities.

Intel's Hadoop solution focuses on efficient integration of Apache Open source based Hadoop software distribution with commodity servers to deliver optimal solutions for a variety of use cases while minimizing total cost of ownership.

Access controls and Data protection options

Intel Distribution provides security and access control features, each of which can be applied in conjunction with others or independently if needed. They are,

Hardware assisted AES encryption of data

Kerberos authentication for services and users

Role base Access Control

These features are explained as below:

Intel Advanced Encryption Standard New Instructions (AES-NI)

ID is optimized for Intel Advanced Encryption Standard New Instructions (AES-NI), a technology that is built into Intel Xeon processors. Encryption and decryption are compute-intensive processes that traditionally add considerable latency and consume substantial processing resources. The Intel Distribution for Apache Hadoop software running on Intel Xeon processors helps to eliminate much of the latency and greatly reduce the load on the processors. Encryption and decryption are performed using OpenSSL 1.0.1c. This version of OpenSSL has been optimized by Intel for AES-NI. Intel AES-NI provides seven instructions that help to accelerate the most complex and compute-intensive steps of the AES algorithms.


Note No changes are required from the configuration point of view for AES-NI encryption optimizations, as mapreduce code which has support for encrypting and decrypting data (that is, the map-reduce code will decrypt data when read from HDFS and encrypt back when writing to HDFS) running on an encrypted workload will automatically trigger the AES-NI.


Kerberos Authentication (secure mode)

Kerberos is a network authentication protocol that uses symmetric key cryptography to provide strong authentication for client-server applications. The protocol requires mutual authentication, which means the client and the server must verify one another's identity before the client is permitted to use resources on the server. The purpose of Kerberos is to enable applications that communicate over a non-secure network to prove their identity to one another in a secure manner.

In ID cluster, Kerberos authentication is used in the following ways:

Trusted Services —Apache Hadoop daemons must pass Kerberos authentication before access is granted to Kerberized services. Kerberized services include MapReduce, HDFS, and HBase.

For example, if there is a 16 node cluster where the primary NameNode service runs on node 1 and a datanode service runs on node 5, then the datanode service must pass Kerberos authentication before it is permitted to communicate with the primary NameNode service.

Trusted Clients — users, which include human or system, must pass Kerberos authentication before consuming Kerberized services on the cluster. Unlike in simple authentication, the user's UID is not used to authenticate the user. Instead, the primary component of the user's Kerberos principal is the username that is authenticated.

For example, if the Unix user jdoe needs to run a MapReduce application, then that user must pass Kerberos authentication before the application is permitted to run. In addition, the MapReduce application runs as the authenticated user.


Note Configuration for Kerberos authentication (secure mode) should be done prior to ID setup and necessary keytab files need to be provided during installation process in order to support Kerberos authenticated communication between services and users instead of simple ssh (simple mode). These steps are detailed in ID Installation as prerequisite steps if we are choosing secure mode.


Role based Access Control

Authentication is the process of verifying a client's identity. Authorization is the process of determining whether an authenticated identity is entitled to use a particular resource.

ID includes built-in support for enterprise-class access controls. Apache Hadoop authenticates a user based on the username in the Unix Shell, if in simple mode, or the user's Kerberos principal, if in secure mode. The cluster uses Access Control Lists (ACL) to perform service level authorization. An ACL is a list of users and groups permitted to take a particular action for an Apache Hadoop service.

After a user is authenticated, the Apache Hadoop daemon checks if the user is in the ACL or is a part of a group that is in the ACL. If either condition is true, then the user is authorized to perform the action or actions controlled by the ACL. Intel Distribution for Apache Hadoop software has implemented a role based authorization tool that uses the LDAP based group mapping functionality.

Roles

A role is a set of permissions for creating, reading, and modifying data as well as service administration for the following Apache Hadoop services.

HDFS

MapReduce

HBase

Hive

Oozie

A role can consist of permissions to one service or many services. When a role is assigned to a LDAP group, any user in that group gains the set of permissions that the role consists of. If a user is placed in multiple LDAP groups and each LDAP group is assigned a role, then the user gains the permissions of each role assigned to each group. Consequently, you can easily increase or decrease a user's access to Apache Hadoop services by adding or remove the user from LDAP groups.

Role based authorization is enabled on a per service basis. Once enabled, a user must be in a role with the appropriate permissions to perform any read/write action on a service. A role can have permissions from multiple services.

Role based authorization provides the following features and functionality:

Unified security management tool — from Intel Manager, a system administrator has a central administrative point of control to define roles, assign users and groups to those roles, then propagate those changes across the cluster with a click of a button.

Integration with an existing identity store — instead of having to manually reconstruct an identity store from scratch, the role based authorization feature retrieves users and groups from an existing LDAP server and then binds those identities to roles in Intel Manager.

Security, consistency, and reliability — since Intel Manager retrieves and assigns users and groups from a single LDAP server, a system administrator avoids potential ACL and permission leakages that occur when multiple identity stores are used and ACLs are manually provisioned across the cluster.


Note Configuration for Integration with Identity Store for Access control is done after ID setup. These steps are detailed in ID Installation as Post Installation steps.


Hive and Pig

Hive is the query engine framework for Intel Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in HDFS and HBase. With SQL like semantics, Hive makes it easy for RDBMS users to transition into querying unstructured data in Hadoop. The Intel Manager provides the option to select Hive optionally, during installation and prompts the user to choose the nodes that will run the meta-store and Hive engine components. The Intel distribution of Hadoop includes enhancements to run Hive queries on data in Hbase and as a result, can run queries faster by a few orders of magnitude when compared to queries that are run on data in HDFS.

Additionally Intel Distribution supports encryption/decryption inherently in the hive queries, spawning map reduce jobs, which automatically decrypt encrypted data and further encrypt the result once the credentials are provided

Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets. The Intel Manager optionally, allows the administrator during installation to select and install Pig.

Additionally Intel Distribution supports encryption/decryption inherently in the pig queries, spawning map reduce jobs, which automatically decrypt encrypted data and further encrypt the result once the credentials are provided

Solution Overview

The current version of the Cisco UCS CPA Version 2 for Big Data offers the following configuration depending on the compute and storage requirements:

Table 1 Cisco UCS CPA v2 Configuration Details

Performance and Capacity Balanced
Capacity Optimized
Capacity Optimized with Flash Memory

16 Cisco UCS C240 M3 Rack Servers, each with:

2 Intel Xeon processors E5-2660 v2

256 GB of memory

LSI MegaRaid 9271CV 8i card

24 1-TB 7.2K SFF SAS drives (384 TB total)

16 Cisco UCS C240 M3 Rack Servers, each with:

2 Intel Xeon processors E5-2640 v2

128 GB of memory

LSI MegaRaid 9271CV 8i card

12 4-TB 7.2 LFF SAS drives (768 TB total)

16 Cisco UCS C240 M3 Rack Servers, each with:

2 Intel Xeon processors E5-2660 v2

128 GB of memory

Cisco UCS Nytro MegaRAID 200-GB Controller

12 4-TB 7.2K LFF SAS drives (768 TB total)



Note This CVD describes the installation process for a 64-node Performance and Capacity Balanced Cluster configuration.


The Performance and Capacity Balanced Cluster configuration consists of the following:

Two Cisco UCS 6296UP Fabric Interconnects

Eight Cisco Nexus 2232PP Fabric Extenders (two per rack)

64 Cisco UCS C240M3 Rack-Mount Servers (16 per rack)

Four Cisco R42610 standard racks

Eight vertical power distribution units (PDU) (country specific)

Rack and PDU Configuration

Each rack consists of two vertical PDU. The master rack consists of two Cisco UCS 6296UP Fabric Interconnects, two Cisco Nexus 2232PP Fabric Extenders and sixteen Cisco UCS C240M3 Servers, connected to each of the vertical PDUs for redundancy; thereby, ensuring availability during power source failure. The expansion racks also consists of two Cisco Nexus 2232PP Fabric Extenders and sixteen Cisco UCS C240M3 Servers are connected to each of the vertical PDUs for redundancy; thereby, ensuring availability during power source failure, similar to master rack.


Note Contact your Cisco representative for country specific information.


Table 2 and Table 3 describe the rack configurations of rack 1 (master rack) and racks 2-4 (expansion racks).

Table 2 Rack Configuration For The Master Rack (Rack-1)

Cisco 42U Rack
Master Rack

42

Cisco UCS FI 6296UP

41

40

Cisco UCS FI 6296UP

39

38

Cisco Nexus FEX 2232PP

37

Cisco Nexus FEX 2232PP

36

Unused

35

Unused

34

Unused

33

Unused

32

Cisco UCS C240M3

31

30

Cisco UCS C240M3

29

28

Cisco UCS C240M3

27

26

Cisco UCS C240M3

25

24

Cisco UCS C240M3

23

22

Cisco UCS C240M3

21

20

Cisco UCS C240M3

19

18

Cisco UCS C240M3

17

16

Cisco UCS C240M3

15

14

Cisco UCS C240M3

13

12

Cisco UCS C240M3

11

10

Cisco UCS C240M3

9

8

Cisco UCS C240M3

7

6

Cisco UCS C240M3

5

4

Cisco UCS C240M3

3

2

Cisco UCS C240M3

1


Table 3 Rack Configuration for the Expansion Rack (Racks 2-4)

Cisco 42U Rack
Master Rack

42

Unused

41

Unused

40

Unused

39

Unused

38

Cisco Nexus FEX 2232PP

37

Cisco Nexus FEX 2232PP

36

Unused

35

Unused

34

Unused

33

Unused

32

Cisco UCS C240M3

31

30

Cisco UCS C240M3

29

28

Cisco UCS C240M3

27

26

Cisco UCS C240M3

25

24

Cisco UCS C240M3

23

22

Cisco UCS C240M3

21

20

Cisco UCS C240M3

19

18

Cisco UCS C240M3

17

16

Cisco UCS C240M3

15

14

Cisco UCS C240M3

13

12

Cisco UCS C240M3

11

10

Cisco UCS C240M3

9

8

Cisco UCS C240M3

7

6

Cisco UCS C240M3

5

4

Cisco UCS C240M3

3

2

Cisco UCS C240M3

1


Server Configuration and Cabling

The Cisco UCS C240M3 Rack Server is equipped with Intel Xeon E5-2660 v2 processors, 256 GB of memory, Cisco UCS Virtual Interface Card (VIC)1225, LSI MegaRAID SAS 9271 CV-8i storage controller and 24 x 1TB 7.2K Serial Attached SCSI (SAS) disk drives.

Figure 2 illustrates the ports on the Cisco Nexus 2232PP fabric extender connecting to the Cisco UCS C240M3 servers. Sixteen Cisco UCS C240M3 servers are used in the master rack configurations.

Figure 2 Fabric Topology

Figure 3 illustrates the port connectivity between the Cisco Nexus 2232PP FEX and the Cisco UCS C240M3 server.

Figure 3 Connectivity Diagram of Cisco Nexus 2232PP FEX and Cisco UCS C240M3 Servers

For more information on physical connectivity and single-wire management, see:

http://www.cisco.com/en/US/docs/unified_computing/ucs/c-series_integration/ucsm2.1/b_UCSM2-1_C-Integration_chapter_010.html

For more information on physical connectivity illustrations and cluster setup, see:

http://www.cisco.com/en/US/docs/unified_computing/ucs/c-series_integration/ucsm2.1/b_UCSM2-1_C-Integration_chapter_010.html#reference_FE5B914256CB4C47B30287D2F9CE3597

Figure 4 depicts a 64-node cluster, and each link represents 8 x 10 Gigabit link.

Figure 4 64 -Node Cluster Configuration

Software Distributions and Versions

Intel Distribution for Apache Hadoop (ID)

The ID version supported is 2.5. For more information, see: http://hadoop.intel.com/

RHEL

The Operating System supported is Red Hat Enterprise Linux Server 6.4. For more information on the Linux support, see:

www.redhat.com.

Software Versions

Table 4 describes the software versions tested and validated in this document.

Table 4 Software Versions Summary

Layer
Component
Version or Release

Compute

Cisco UCS C240M3

1.4.7cc

Network

Cisco UCS 6296UP

UCS 2.1(1e)

Cisco UCS VIC1225 Firmware

2.1(1a)

Cisco UCS VIC1225 Driver

2.1.1.41

Cisco Nexus 2232PP

5.1(3)N2(2.11a)

Storage

LSI 9271 CV-8i Firmware

23.7.0-0039

LSI 9271 CV-8i Driver

06.601.06.00

Software

Red Hat Enterprise Linux Server

6.4 (x86_64)

Cisco UCS Manager

2.1(1e)

Intel Distribution for Apache Hadoop

2.5



Note To download the latest drivers, see: http://software.cisco.com/download/release.html?mdfid=284296254&flowid=31743&softwareid=283853158&release=1.5.1&relind=AVAILABLE&rellifecycle=&reltype=latest


Fabric Configuration

This section provides details for configuring a fully redundant, highly available Cisco UCS 6296 Fabric Interconnect.

1. Initial setup of the Fabric Interconnect A and B.

2. Connect to IP address of Fabric Interconnect A using web browser.

3. Launch the Cisco UCS Manager.

4. Edit the chassis discovery policy.

5. Enable server and uplink ports.

6. Create pools and polices for service profile template.

7. Create Cisco Service Profile template and 64 service profiles.

8. Start discover process.

9. Associate to server.

Performing Initial Setup of Cisco UCS 6296 Fabric Interconnects

This section describes the steps to perform the initial setup of the Cisco UCS 6296 Fabric Interconnects A and B.

Configure Fabric Interconnect A

Follow these steps to configure the Fabric Interconnect A:

1. Connect to the console port on the first Cisco UCS 6296 Fabric Interconnect.

2. At the prompt to enter the configuration method, enter console to continue.

3. If asked to either perform a new setup or restore from backup, enter setup to continue.

4. Enter y to continue to set up a new Fabric Interconnect.

5. Enter y to enforce strong passwords.

6. Enter the password for the admin user.

7. Enter the same password again to confirm the password for the admin user.

8. When asked if this fabric interconnect is part of a cluster, enter y to continue.

9. Enter A for the switch fabric.

10. Enter the cluster name for the system name.

11. Enter the Mgmt0 IPv4 address.

12. Enter the Mgmt0 IPv4 netmask.

13. Enter the IPv4 address of the default gateway.

14. Enter the cluster IPv4 address.

15. To configure DNS, enter y.

16. Enter the DNS IPv4 address.

17. Enter y to set up the default domain name.

18. Enter the default domain name.

19. Review the settings that were printed to the console, and enter yes to save the configuration.

20. Wait for the login prompt to make sure the configuration has been saved.

Configure Fabric Interconnect B

Follow these steps to configure the Fabric Interconnect B:

1. Connect to the console port on the second Cisco UCS 6296 Fabric Interconnect.

2. When prompted to enter the configuration method, enter console to continue.

3. The installer detects the presence of the partner fabric interconnect and adds this fabric interconnect to the cluster. Enter y to continue the installation.

4. Enter the admin password that was configured for the first Fabric Interconnect.

5. Enter the Mgmt0 IPv4 address.

6. Enter yes to save the configuration.

7. Wait for the login prompt to confirm that the configuration has been saved.

For more information on configuring Cisco UCS 6200 Series Fabric Interconnect, see:

http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_0100.html

Logging Into Cisco UCS Manager

Follow these steps to login to Cisco UCS Manager.

1. Open a Web browser and navigate to the Cisco UCS 6296 Fabric Interconnect cluster address.

2. Click the Launch link to download the Cisco UCS Manager software.

3. If prompted, accept the security certificates.

4. When prompted, enter the username as admin and the administrative password.

5. Click Login.

Upgrading UCSM Software to Version 2.1(1e)

This document assumes the uses of Cisco UCS 2.1(1e). Make sure that the Cisco UCS C-Series version 2.1(1e) software bundle is installed on the Cisco UCS Fabric Interconnects.

To upgrade the Cisco UCS Manager software and Cisco UCS 6296 Fabric Interconnect software to version 2.1(1e), see: Upgrading Cisco UCS from Release 2.0 to Releases 2.1

Adding Block of IP Addresses for KVM Access

Follow these steps to create a block of KVM IP addresses for the server access in Cisco UCS environment.

1. Click the LAN tab.

2. Select Pools > IPPools > IP Pool ext-mgmt.

3. Right-click Management IP Pool.

4. Select Create Block of IP Addresses as sown in Figure 5

Figure 5 Adding Block of IP Addresses for KVM Access Part 1

5. Enter the starting IP address of the block and number of IPs needed, the subnet and the gateway information as shown in Figure 6.

Figure 6 Adding Block of IP Addresses for KVM Access Part 2

6. Click OK to create the IP Address block as shown in Figure 7.

Figure 7 Adding Block of IP Addresses for KVM Access Part 3

7. Click OK.

Editing The Chassis Discovery Policy

This section provides details for modifying the chassis discovery policy. Setting the discovery policy ensures easy addition of the Cisco UCS B-Series chassis or fabric extenders for the Cisco UCS C-Series servers in future.

1. Click the Equipment tab.

2. In the right pane, click the Policies tab.

3. Click the Global Policies tab.

4. In the Chassis/FEX Discovery Policy area, select 8-link from the drop-down list for Action field as shown in Figure 8.

Figure 8 Changing The Chassis Discovery Policy

5. Click Save Changes.

6. Click OK.

Enabling The Server Ports and Uplink Ports

Follow these steps to enable the server and configure the uplink ports:

1. Click the Equipment tab.

2. Select Equipment > Fabric Interconnects > Fabric Interconnect A (primary) > Fixed Module.

3. Expand the Unconfigured Ethernet Ports.

4. Select all the ports that are connected to the Cisco 2232PP FEX (eight per FEX), right-click and select Reconfigure > Configure as a Server Port.

5. Select port 1 that is connected to the uplink switch, right-click, then select Reconfigure > Configure as Uplink Port.

6. Select Show Interface and select 10GB for Uplink Connection.

7. Click Yes and then OK to continue.

8. Select Equipment > Fabric Interconnects > Fabric Interconnect B (subordinate) > Fixed Module.

9. Expand the Unconfigured Ethernet Ports section.

10. Select all the ports that are connected to the Cisco 2232 Fabric Extenders (eight per Fex), right-click and select Reconfigure > Configure as Server Port.

11. Click Yes and then OK to continue.

12. Select port number 1, which is connected to the uplink switch, right-click and select Reconfigure > Configure as Uplink Port.

Figure 9 Enabling Server Ports

13. Select Show Interface and select 10GB for Uplink Connection.

14. Click Yes and then OK to continue.

Figure 10 shows all the configured uplink and Server ports.

Figure 10 Server and Uplink Ports Summary

Creating Pools for Service Profile Templates

Creating an Organization

Organizations are used as a means to arrange and restrict access to various groups within the IT organization, and enable multi-tenancy of the compute resources. This document does not use organizations; however, the steps to create an organizations are given for future reference.

Follow these steps to configure an organization in the Cisco UCS Manager:

1. Click New in the left corner of the UCS Manager GUI.

2. Select Create Organization from the options.

3. Enter a name for the organization.

4. (Optional) Enter a description for the organization.

5. Click OK.

Creating MAC Address Pools

Follow these steps to create MAC address pools:

1. Click the LAN tab.

2. Select Pools > root.

3. Right-click MAC Pools under the root organization.

4. Select Create MAC Pool to create the MAC address pool. Enter ucs as the name of the MAC pool.

5. (Optional) Enter a description of the MAC pool.

6. Click Next.

7. Click Add.

8. Specify a starting MAC address.

9. Specify a size of the MAC address pool, which is sufficient to support the available server resources as shown in Figure 11.

10. Click OK.

Figure 11 Specifying the First MAC Address and Size

11. Click Finish as shown in Figure 12.

Figure 12 Adding MAC Addresses

12. Click OK to confirm the addition of the MAC addresses.

Configuring VLANs

Table 5 describes the VLANs that are configured in this design solution.

Table 5 VLAN Configurations

VLAN
Fabric
NIC Port
Function
Failover

vlan160_mgmt

A

eth0

Management, user connectivity

Fabric Failover B

vlan12_HDFC

B

eth1

Hadoop

Fabric Failover A

vlan11_DATA

A

eth2

Hadoop and/or SAN/NAS access, ETL

Fabric Failover B


All of the VLANs created should be trunked to the upstream distribution switch connecting the fabric interconnects. In this deployment, vlan160_mgmt is configured for management access and user connectivity, vlan12_HDFS is configured for Hadoop interconnect traffic, and vlan11_DATA is configured for optional secondary interconnect and/or SAN/NAS access, heavy ETL, and so on.

Follow these steps to configure VLANs in Cisco UCS Manager:

1. Click the LAN tab.

2. Select LAN > VLANs.

3. Right-click the VLANs under the root organization.

4. Select Create VLANs to create the VLAN as shown in Figure 13.

Figure 13 Creating VLAN

5. Enter vlan160_mgmt in the VLAN Name/Prefix text box as shown in Figure 14.

6. Click the Common/Global radio button.

7. Enter 160 in the VLAN IDs text box.

8. Click OK and then click Finish.

9. Click OK.

Figure 14 Creating Management VLAN

10. Click the LAN tab.

11. Select LAN > VLANs.

12. Right-click the VLANs under the root organization.

13. Select Create VLANs to create the VLAN as shown in Figure 15.

14. Enter vlan11_DATA in the VLAN Name/Prefix text box.

15. Click the Common/Global radio button.

16. Enter 11 in the VLAN IDs text box.

17. Click OK and then click Finish.

18. Click OK.

Figure 15 Creating VLAN for Data

19. Click the LAN tab.

20. Select LAN > VLANs.

21. Right-click the VLANs under the root organization.

22. Select Create VLANs to create the VLAN.

23. Enter vlan12_HDFS in the VLAN Name/Prefix text box as shown in Figure 16.

24. Click the Common/Global radio button.

25. Enter 12 in the VLAN IDs text box.

26. Click OK and then click Finish.

Figure 16 Creating VLAN for Hadoop Data

Creating a Server Pool

A server pool contains a set of servers. These servers typically share the same characteristics such as their location in the chassis, server type, amount of memory, local storage, type of CPU, or local drive configuration. You can manually assign a server to a server pool, or use the server pool policies and server pool policy qualifications to automate the server assignment.

Follow these steps to configure the server pool within the Cisco UCS Manager:

1. Click the Servers tab.

2. Select Pools > root.

3. Right-click the Server Pools.

4. Select Create Server Pool.

5. Enter the required name (ucs) for the server pool in the name text box as shown in Figure 17.

6. (Optional) Enter a description for the organization.

Figure 17 Setting Name and Description of the Server Pool

7. Click Next to add the servers.

8. Select all the Cisco UCS C240M3 servers to be added to the server pool that were previously created (ucs), then Click >> to add them to the pool as shown in Figure 18.

9. Click OK, and then click Finish.

Figure 18 Adding Servers to the Server Pool

Creating Policies for Service Profile Templates

This section provides you the procedure to create the following policies for the service profile template:

Creating a Host Firmware Package Policy

Creating QoS Policies

Creating a Local Disk Configuration Policy

Creating a Server BIOS Policy

Creating a Boot Policy

Creating a Host Firmware Package Policy

Firmware management policies allow the administrator to select the corresponding firmware packages for a given server configuration. The components that can be configured include adapters, BIOS, board controllers, FC adapters, HBA options, ROM and storage controller.

Follow these steps to create a host firmware management policy for a given server configuration using the Cisco UCS Manager:

1. Click the Servers tab in the UCS Manager.

2. Select Policies > root.

3. Right-click Host Firmware Packages.

4. Select Create Host Firmware Package.

5. Enter the required host firmware package name (ucs) as shown in Figure 19.

6. Click the Simple radio button to configure the host firmware package.

7. Select the appropriate Rack Package value.

8. Click OK to complete creating the management firmware package.

9. Click OK.

Figure 19 Creating Host Firmware Package

Creating QoS Policies

This section describes the procedure to create the Best Effort QoS Policy and Platinum QoS policy.

Creating the Best Effort Policy

Follow these steps to create the Best Effort Policy:

1. Click the LAN tab.

2. Select Policies > root.

3. Right-click QoS Policies.

4. Select Create QoS Policy as shown in Figure 20.

Figure 20 Creating QoS Policy

5. Enter BestEffort as the name of the policy as shown in Figure 21.

6. Select BestEffort from the drop down menu.

7. Keep the Burst (Bytes) field as default (10240).

8. Keep the Rate (Kbps) field as default (line-rate).

9. Keep the Host Control radio button as default (none).

10. Click OK to complete creating the Policy.

11. Click OK.

Figure 21 Creating BestEffort QoS Policy

Creating a Platinum Policy

Follow these steps to create the Platinum QoS policy:

1. Click the LAN tab.

2. Select Policies > root.

3. Right-click QoS Policies.

4. Select Create QoS Policy.

5. Enter Platinum as the name of the policy as shown in Figure 22.

6. Select Platinum from the drop down menu.

7. Keep the Burst (Bytes) field as default (10240).

8. Keep the Rate (Kbps) field as default (line-rate).

9. Keep the Host Control radio button as default (none).

10. Click OK to complete creating the Policy.

11. Click OK.

Figure 22 Creating Platinum QoS Policy

Setting Jumbo Frames

Follow these steps to set up Jumbo frames and enable the QoS:

1. Click the LAN tab in the Cisco UCS Manager.

2. Select LAN Cloud > QoS System Class.

3. In the right pane, click the General tab.

4. In the Platinum row, enter 9000 for MTU as shown in Figure 23.

5. Check the Enabled check box.

6. Click Save Changes.

7. Click OK.

Figure 23 Setting Jumbo Frames

Creating a Local Disk Configuration Policy

Follow these steps to create a local disk configuration in the Cisco UCS Manager:

1. Click the Servers tab.

2. Select Policies > root.

3. Right-click Local Disk Config Policies.

4. Select Create Local Disk Configuration Policy.

5. Enter ucs as the local disk configuration policy name as shown in Figure 24.

6. Select Any Configuration from the drop-down list to set the Mode.

7. Uncheck the Protect Configuration check box.

8. Click OK to complete creating the Local Disk Configuration Policy.

9. Click OK.

Figure 24 Configuring Local Disk Policy

Creating a Server BIOS Policy

The BIOS policy feature in Cisco UCS automates the BIOS configuration process. The traditional mode of setting the BIOS is manual and is often error-prone. By creating a BIOS policy and assigning the policy to a server or group of servers, you can enable transparency within the BIOS settings configuration.


Note BIOS settings can have a significant performance impact, depending on the workload and the applications. The BIOS settings listed in this section is for configurations optimized for best performance which can be adjusted based on the application, performance and energy efficiency requirements.


Follow these steps to create a server BIOS policy using the Cisco UCS Manager:

1. Select the Servers tab.

2. Select Policies > root.

3. Right-click BIOS Policies.

4. Select Create BIOS Policy.

5. Enter the preferred BIOS policy name.

6. Change the BIOS settings as shown in Figure 25.

Figure 25 Creating Server BIOS Policy

Figure 26 and Figure 27 show the Processor and Intel Directed IO properties settings in the BIOS Policy.

Figure 26 Creating Server BIOS Policy for Processor

Figure 27 Creating Server BIOS Policy for Intel Directed IO

7. Set the RAS Memory settings and click Next as shown in Figure 28.

Figure 28 Creating Server BIOS Policy for Memory

8. Click Finish to complete creating the BIOS Policy.

9. Click OK.

Creating a Boot Policy

Follow these steps to create a boot policy within Cisco UCS Manager:

1. Select the Servers tab.

2. Select Policies > root.

3. Right-click the Boot Policies.

4. Select Create Boot Policy as shown in Figure 29.

Figure 29 Creating Boot Policy Part 1

5. Enter ucs as the boot policy name as shown in Figure 30.

6. (Optional) Enter a description for the boot policy.

7. Keep the Reboot on Boot Order Change check box as default (unchecked).

8. Expand Local Devices and select Add CD-ROM.

9. Expand Local Devices and select Add Local Disk.

10. Expand vNICs and select Add LAN Boot and enter eth0.

11. Click OK to add the Boot Policy.

12. Click OK.

Figure 30 Creating Boot Policy Part 2

Creating a Service Profile Template

Follow these steps to create a service profile template in Cisco UCS Manager:

1. Click the Servers tab.

2. Select Policies > root.

3. Right-click root.

4. Select Create Service Profile Template as shown in Figure 31.

Figure 31 Creating Service Profile Template

5. The Create Service Profile Template window appears. Do the following (see Figure 32):

a. In the Identify Service Profile Template window, enter the name of the service profile template as ucs.

b. Click the Updating Template radio button.

c. In the UUID section, select Hardware Default as the UUID pool.

6. Click Next to continue.

Figure 32 Identify Service Profile Template

Configuring the Network Settings for the Template

In the Networking window, follow these steps to configure the network settings in the Cisco UCS Manager:

1. Keep the Dynamic vNIC Connection Policy field at the default as shown in Figure 33.

2. Click the Expert radio button to define How would you like to configure LAN connectivity?

3. Click Add to add a vNIC to the template. The Modify vNIC window appears.

Figure 33 Configuring Network Settings for the Template

4. In the Modify vNIC window, enter name for the vNIC as eth0 as shown in Figure 34.

5. Select ucs in the MAC Address Assignment pool.

6. Click the Fabric A radio button and check the Enable failover check box for the Fabric ID.

7. Check the vlan160_mgmt check box for VLANs.

8. Click the Native VLAN radio button.

9. Select MTU size as 1500.

10. Select adapter policy as Linux.

11. Keep the Dynamic vNIC connection policy as <no set>.

12. Select QoS Policy as BestEffort.

13. Keep the Network Control Policy as default.

14. Click OK.

Figure 34 Configuring vNIC eth0

15. The Modify vNIC window appears. Enter the name of the vNIC as eth1 as shown in Figure 35.

16. Select ucs in the MAC Address Assignment pool.

17. Click the Fabric B radio button and check the Enable failover check box for the Fabric ID.

18. Check the vlan12_HDFS check box for VLANs and select the Native VLAN radio button.

19. Select MTU size as 9000.

20. Select adapter policy as Linux.

21. Keep the Dynamic vNIC Connection Policy as <no set>.

22. Select QoS Policy as Platinum.

23. Keep the Network Control Policy as default.

24. Click OK.

Figure 35 Configuring vNIC eth1

25. The Create vNIC window appears. Enter the name of the vNIC as eth2 as shown in Figure 36.

26. Select ucs in the MAC Address Assignment pool.

27. Click the Fabric A radio button and check the Enable failover check box for the Fabric ID.

28. Check the vlan11_DATA check box for VLANs and select the Native VLAN radio button.

29. Select MTU size as 9000.

30. Select adapter policy as Linux.

31. Keep the Dynamic vNIC Connection Policy as <no set>.

32. Select QoS Policy as Platinum.

33. Keep the Network Control Policy as default.

34. Click OK.

35. Click Next in the Networking window to continue.

Figure 36 Configuring vNIC eth2

Configuring a Storage Policy for the Template

In the Storage window, follow these steps to configure a storage policy in Cisco UCS Manager:

1. Select ucs for the local disk configuration policy as shown in Figure 37.

2. Click the No vHBAs radio button to define How would you like to configure SAN connectivity?

3. Click Next to continue.

Figure 37 Configuring Storage settings

4. Click Next in the Zoning window to continue.

Configuring a vNIC/vHBA Placement for the Template

In the vNIC/vHBA window, follow these steps to configure a vNIC/vHBA placement policy in Cisco UCS Manager:

1. Select the Default Placement Policy option for the Select Placement field as shown in Figure 38.

2. Select eth0, eth1 and eth2 assign the vNICs in the following order:

a. eth0

b. eth1

c. eth2

Review to make sure that all vNICs are assigned in the appropriate order.

3. Click Next to continue.

Figure 38 vNIC/vHBA Placement

Configuring a Server Boot Order for the Template

In the Server Boot Order window, follow these steps to set the boot order for servers in Cisco UCS Manager:

1. Select ucs in the Boot Policy name field as shown in Figure 39.

2. Check the Enforce vNIC/vHBA/iSCSI Name check box.

Review to make sure that all the boot devices are created and identified.

3. Verify that the boot devices are in the correct boot sequence.

4. Click OK.

Figure 39 Creating Boot Policy

5. Click Next to continue.

In the Maintenance Policy window, keep the default no policy as we have not created a policy. Click Next to continue to the next window.

Configuring Server Assignment for the Template

In the Server Assignment window, follow these steps to assign the servers to the pool in Cisco UCS Manager:

1. Select ucs for the Pool Assignment field as shown in Figure 40.

2. Keep the Server Pool Qualification field as default.

3. Select ucs in Host Firmware Package.

Figure 40 Server Assignment

Configuring Operational Policies for the Template

In the Operational Policies window, follow these steps:

1. Select ucs in the BIOS Policy field as shown in Figure 41.

2. Click Finish to create the Service Profile template.

3. Click OK.

Figure 41 Selecting BIOS Policy

4. Click the Servers tab.

a. Select Service Profile Templates > root.

b. Right-click root and select Create Service Profile Template as shown in Figure 42.

Figure 42 Creating Service Profiles from Template

c. The Create Service Profile from Template window appears. Enter the name and number of nodes in the Name and Number fields as shown in Figure 43.

Figure 43 Selecting Name and Total Number of Service Profiles

The Cisco UCS Manager discovers the servers and automatically associate these servers with service profiles. Figure 44 illustrates the service profiles associated with all the 64-nodes.

Figure 44 Cisco UCS Manager showing 64 Nodes

Configuring Disk Drives for Operating System on NameNode

Namenode and Secondary Namenode have a different RAID configuration compared to Datanodes. This section details the configuration of disk drives for OS on these nodes (rhel1 and rhel2). The disk drives are configured as RAID1, read ahead cache and write cache are enabled when the battery is available. The first two disk drives are used for the Operating System and the remaining 22 disk drives are used for the HDFS as described in the following sections.

There are several ways to configure RAID such as using the LSI WebBIOS Configuration Utility embedded in the MegaRAID BIOS, booting DOS and running MegaCLI commands, using Linux-based MegaCLI commands, or using third party tools that have MegaCLI integrated. For this deployment, the first disk drive is configured using the LSI WebBIOS Configuration Utility and the remaining drives are configured using Linux-based MegaCLI commands after the completion of the Operating System installation.

Figure 45 RAID 1 Configured Using LSI WebBIOS Utility and MegaCLI

Follow these steps to create RAID1 on the first disk drive to install the Operating System:

1. Boot the server, and do the following:

a. Press <Ctrl><H> to launch the WebBIOS.

b. Press Ctrl+H immediately. The Adapter Selection window appears.

2. Click Start to continue as shown in Figure 46.

3. Click Configuration Wizard.

Figure 46 Adapter Selection for RAID Configuration

4. In the Configuration Wizard window, click the Clear Configuration radio button as shown in Figure 47.

5. Click Next to clear the existing configuration.

Figure 47 Clearing Current configuration on the controller

6. Click Yes.

7. In the Physical View, ensure that all the drives are Unconfigured Good.

8. In the Configuration Wizard window, click the New Configuration radio button as shown in Figure 48.

9. Click Next.

Figure 48 Choosing to create a New Configuration

10. Click the Manual Configuration radio button. This enables complete control over all attributes of the new storage configuration, such as, configuration of the drive groups, virtual drives and setting the parameters as shown in Figure 49.

Figure 49 Choosing Manual Configuration Method

11. Click Next. The Drive Group Definition window appears.

12. In the Drive Group Definition window, choose the first two drives to create drive groups as shown in Figure 50.

13. Click Add to Array to move the drives to a proposed drive group configuration in the Drive Groups pane.

14. Click Accept DG and click Next.

Figure 50 Selecting first drive and Adding to Drive Group

15. In the Span Definitions window, click Add to SPAN and click Next as shown in Figure 51.

Figure 51 Span Definition Window

16. In the Virtual Drive definitions window, do the following (see Figure 52):

a. Click on Update Size.

b. Change Strip Size to 1MB. A larger strip size ensures higher read performance.

c. From the Read Policy drop-down list, choose Always Read Ahead.

d. From the Write Policy drop-down list, choose Write Back with BBU.

e. Make sure RAID Level is set to RAID1.

f. Click Accept to accept the changes to the virtual drive definitions.

g. Click Next.


Note Clicking on Update Size can change some of the settings in the window. Make sure all settings are correct before submitting the changes.


Figure 52 Defining Virtual Drive

17. Click Yes to save the configuration.

18. In the Managing SSD Caching window, click Cancel as shown in Figure 53.

Figure 53 SSD Caching Window

19. Click Yes in the confirmation page.

20. Set VD0 as the Boot Drive and click Go as shown in Figure 54.

Figure 54 Setting Virtual Drive as Boot Drive

21. Click Home.

22. Review the configuration and click Exit.

Configuration of disks 2 to 24 are done using Linux based MegaCLI commands described in "Configuring Data Drives on NameNode" section.

Configuring Disk Drives for Operating System on DataNodes

Nodes 3 through 64 are configured as DataNodes. This section details the configuration of disk drives for OS on the data nodes. The focus of this CVD is on the High Performance Configuration, featuring 24 1TB SFF disk drives. The disk drives are configured as individual RAID0 volumes with 1MB strip size. Read ahead cache and write cache are enabled when the battery is available. The first disk drive is used for the Operating System and the remaining 23 disk drives are used for the HDFS as described in the following sections.


Note In case of the High Capacity Configuration featuring 12 4TB LFF disk drives, the disk drives are configured as individual RAID0 volumes with 1MB strip size. Read ahead cache and write cache are enabled when the battery is available. Two partitions of 1TB and 3TB are created on the first disk drive, the 1TB partition is used for the Operating System and the 3TB partition is used for the HDFS along with disk drives 2 through 12.


There are several ways to configure RAID. RAID can be configured using LSI WebBIOS Configuration Utility embedded in the MegaRAID BIOS, booting DOS and running MegaCLI commands, Linux based MegaCLI commands, or by third party tools having MegaCLI. For this deployment, the first disk drive is configured using LSI WebBIOS Configuration Utility and the rest of them are configured using a Linux based MegaCLI commands after the completion of OS installation.

Follow these steps to create RAID0 on the first disk drive to install the Operating System:

1. Boot the server, and do the following:

a. Press <Ctrl><H> to launch the WebBIOS.

b. Press Ctrl+H immediately. The Adapter Selection window appears.

2. Click Start to continue as shown in Figure 46.

3. Click Configuration Wizard.

Figure 55 Adapter Selection for RAID Configuration

4. In the Configuration Wizard window, click the Clear Configuration radio button as shown in Figure 47.

5. Click Next to clear the existing configuration.

Figure 56 Clearing Current configuration on the controller

6. Click Yes.

7. In the Physical View, ensure that all the drives are Unconfigured Good.

8. In the Configuration Wizard window, click the New Configuration radio button as shown in Figure 48.

9. Click Next.

Figure 57 Choosing to create a New Configuration

10. Click the Manual Configuration radio button. This enables complete control over all attributes of the new storage configuration, such as, configuration of the drive groups, virtual drives and setting the parameters as shown in Figure 49.

Figure 58 Choosing Manual Configuration Method

11. Click Next. The Drive Group Definition window appears.

12. In the Drive Group Definition window, choose the first drive to create drive groups as shown in Figure 50.

13. Click Add to Array to move the drives to a proposed drive group configuration in the Drive Groups pane.

14. Click Accept DG and click Next.

Figure 59 Selecting first drive and Adding to Drive Group

15. In the Span Definitions window, click Add to SPAN and click Next as shown in Figure 51.

Figure 60 Span Definition Window

16. In the Virtual Drive definitions window, do the following (see Figure 52):

a. Click on Update Size.

b. Change Strip Size to 1MB. A larger strip size ensures higher read performance.

c. From the Read Policy drop-down list, choose Always Read Ahead.

d. From the Write Policy drop-down list, choose Write Back with BBU.

e. Make sure RAID Level is set to RAID0.

f. Click Accept to accept the changes to the virtual drive definitions.

g. Click Next.


Note Clicking on Update Size can change some of the settings in the window. Make sure all settings are correct before submitting the changes.


Figure 61 Defining Virtual Drive

17. Click Yes to save the configuration.

18. In the Managing SSD Caching window, click Cancel as shown in Figure 53.

Figure 62 SSD Caching Window

19. Click Yes in the confirmation page.

20. Set VD0 as the Boot Drive and click Go as shown in Figure 54.

Figure 63 Setting Virtual Drive as Boot Drive

21. Click Home.

22. Review the configuration and click Exit.

Configuration of disks 3 to 24 are done using Linux based MegaCLI commands described in "Configuring Data Drives on DataNodes" section.

Installing Red Hat Linux 6.4 with KVM

The following section provides detailed procedures for installing Red Hat Linux 6.4.

There are multiple methods to install Red Hat Linux Operating System. The installation procedure described in this design guide uses KVM console and virtual media from Cisco UCS Manager.

1. Log in to the Cisco UCS 6296 Fabric Interconnect and launch the Cisco UCS Manager application.

2. Click the Equipment tab.

3. In the navigation pane expand Rack-Mounts and Servers.

4. Right-click on the Server and select KVM Console as shown in Figure 64.

Figure 64 Selecting KVM Console Option

5. In the KVM window, select the Virtual Media tab as shown in Figure 65.

6. Click Add Image button in the Client View selection window.

7. Browse to the Red Hat Enterprise Linux Server 6.4 installer ISO image file.


Note The Red Hat Enterprise Linux 6.4 DVD is assumed to be available on the client machine.


Figure 65 Adding an ISO Image

8. Click Open to add the image to the list of virtual media.

9. Check the Mapped check box for the image you just added as shown in Figure 66.

Figure 66 Mapping ISO Image

10. In the KVM console, select the KVM tab to monitor the bootup.

11. In the KVM console, click Boot Server.

12. Click OK.

13. Click OK to reboot the system.

On reboot, the server detects the presence of the Red Hat Enterprise Linux Server 6.4 install media.

14. Select Install or Upgrade an Existing System option as shown in Figure 67.

Figure 67 Select Install Option

15. Skip the Media test as the ISO Image is used for the installation.

16. Click Next. The Red Hat Linux Welcome Screen appears.

17. Select the Language for the installation.

18. Click the Basic Storage Devices radio button.

19. Click the Fresh Installation radio button.

20. Enter the host name of the server and click Next.

21. Click Configure Network. The Network Connections window appear.

22. In the Network Connections window, select the Wired tab.

23. Select the interface System eth0 and click Edit.

24. Editing System eth0 appears as shown in Figure 68.

25. Check the Connect automatically check box.

26. Select Manual in the Method drop-down list.

27. Click Add and enter IP Address, the netmask and the gateway.

For this demonstration, the following values have been used:

IP Address: 10.29.160.53

Netmask: 255.255.255.0

Gateway: 10.29.160.1

28. Add DNS servers (optional).

29. Click Apply.

Figure 68 Configuring Network for eth0

30. Repeat the steps 26 to steps 32 to configure the network for the System eth1. The following values have been used (see Figure 69):

IP Address: 192.168.12.11

Netmask: 255.255.255.0

Figure 69 Configuring Network for eth1

31. Repeat the steps 26 to steps 32 to configure the network for System eth2. The following values have been used:

IP Address: 192.168.11.11

Netmask: 255.255.255.0

32. Select the appropriate time zone.

33. Enter the root password and click Next.

34. Select Use All Space and click Next as shown in Figure 70.

35. Choose an appropriate boot drive.

Figure 70 Selecting Install Option

36. Click Write changes to the disk and click Next.

37. Select Basic Server and click Next as shown in Figure 71.

Figure 71 Selecting Type of Installation

38. After the installer has finished loading, it will continue with the installation.

39. Reboot the system after the installation is complete.

Repeat the above steps (1 to 39) to install the Red Hat Linux on servers 2 through 64.


Note You can automate the OS installation and configuration of the nodes through the Preboot Execution Environment (PXE) boot or through third party tools.


Table 6 describes the hostnames and their corresponding IP addresses.

Table 6 Hostnames and IP Addresses

Hostname
eth0
eth1
eth2

rhel1

10.29.160.53

192.168.12.11

192.168.11.11

rhel2

10.29.160.54

192.168.12.12

192.168.11.12

rhel3

10.29.160.55

192.168.12.13

192.168.11.13

rhel4

10.29.160.56

192.168.12.14

192.168.11.14

rhel5

10.29.160.57

192.168.12.15

192.168.11.15

rhel6

10.29.160.58

192.168.12.16

192.168.11.16

rhel7

10.29.160.59

192.168.12.17

192.168.11.17

rhel8

10.29.160.60

192.168.12.18

192.168.11.18

rhel9

10.29.160.61

192.168.12.19

192.168.11.19

rhel10

10.29.160.62

192.168.12.20

192.168.11.20

rhel11

10.29.160.63

192.168.12.21

192.168.11.21

rhel12

10.29.160.64

192.168.12.22

192.168.11.22

rhel13

10.29.160.65

192.168.12.23

192.168.11.23

rhel14

10.29.160.66

192.168.12.24

192.168.11.24

rhel15

10.29.160.67

192.168.12.25

192.168.11.25

rhel16

10.29.160.68

192.168.12.26

192.168.11.26

...

...

...

...

rhel64

10.29.160.116

192.168.12.74

192.168.11.74


Post OS Install Configuration

Choose one of the nodes of the cluster or a separate node as an Admin Node for management such as ID installation, parallel shell, creating a local Red Hat repo and others. In this document, we have used rhel1 for management. The manager node is where the Intel ® Manager for Apache Hadoop software is installed and the web server that hosts the Intel Manager runs.

Setting Up Password-less Login

To manage all of the cluster nodes from the admin node we need to setup password-less login. It assists in automating common tasks with Parallel-SSH (pssh) and shell-scripts without having passwords.

Once Red Hat Linux is installed across all the nodes in the cluster, follow these steps in order to enable password less login across all the nodes.

1. Login to the admin node (rhel1).

ssh 10.29.160.53
 
   

2. Run the ssh-keygen command to create both public and private keys on the admin node.

3. Run the following commands from the admin node to copy the public key id_rsa.pub to all the nodes of the cluster. The .ssh-copy-id command appends the keys to the remote-host. .ssh/authorized_key.

for IP in {53..116}; do echo -n "$IP -> "; ssh-copy-id -i ~/.ssh/id_rsa.pub 
10.29.160.$IP; done
 
   

4. Enter yes at the command prompt to continue connecting.

5. Enter the password of the remote host to login.

Installing and Configuring Parallel SSH

Installing Parallel-SSH

Parallel-ssh is used to run commands on several hosts at the same time. It takes a file of hostnames and a few common ssh parameters as parameters, and executes the given command in parallel on the specified nodes.

1. Download the pssh.

wget https://parallel-ssh.googlecode.com/files/pssh-2.3.1.tar.gz

scp pssh-2.3.1.tar.gz rhel1:/root
 
   

2. Run the following command to copy pssh-2.3.1.tar.gz to the admin node:

ssh rhel1 
tar xzf pssh-2.3.1.tar.gz
cd pssh-2.3.1
python setup.py install

3. Extract and install pssh on the admin node.

4. Create a host file containing the IP addresses of all the nodes and all the DataNodes in the cluster. This file is passed as a parameter to pssh to identify the nodes and run the commands on them.

vi /root/allnodes 
# This file contains ip address of all nodes of the cluster 
#used by parallel-shell (pssh). For Details man pssh
10.29.160.53
10.29.160.54
10.29.160.55
10.29.160.56
10.29.160.57
10.29.160.58
10.29.160.59
10.29.160.60
10.29.160.61
10.29.160.62
10.29.160.63
10.29.160.64
10.29.160.65
10.29.160.66
10.29.160.67
10.29.160.68
...
10.29.160.116
 
   
vi /root/datanodes 
10.29.160.55
10.29.160.56
10.29.160.57
10.29.160.58
10.29.160.59
10.29.160.60
10.29.160.61
10.29.160.62
10.29.160.63
10.29.160.64
10.29.160.65
10.29.160.66
10.29.160.67
10.29.160.68
...
10.29.160.116
 
   

Installing Cluster Shell

1. Download cluster shell (clush) and install it on rhel1.

Cluster shell is available from the Extra Packages for Enterprise Linux (EPEL) repository.

wget 
http://dl.fedoraproject.org/pub/epel//6/x86_64/clustershell-1.6-1.el6.noarch.rpm
scp clustershell-1.6-1.el6.noarch.rpm rhel1:/root/
 
   

2. Login to rhel1 and install cluster shell.

yum install clustershell-1.6-1.el6.noarch.rpm
 
   

3. Edit /etc/clustershell/groups file to include hostnames for all the nodes of the cluster.

For 64 node cluster all: rhel[1-64]

Note Configuring EPEL repository is discussed in detail in another section.


Configuring /etc/hosts and DNS

Follow these steps to create the host file across all the nodes in the cluster:

1. Run the following command to populate the host file with IP addresses and corresponding hostnames on the admin node (rhel1):

On Admin Node (rhel1)

vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.29.160.53    rhel1.mgmt
10.29.160.54    rhel2.mgmt
10.29.160.55    rhel3.mgmt
10.29.160.56    rhel4.mgmt
10.29.160.57    rhel5.mgmt
10.29.160.58    rhel6.mgmt
10.29.160.59    rhel7.mgmt
10.29.160.60    rhel8.mgmt
10.29.160.61    rhel9.mgmt
10.29.160.62    rhel10.mgmt
10.29.160.63    rhel11.mgmt
10.29.160.64    rhel12.mgmt
10.29.160.65    rhel13.mgmt
10.29.160.66    rhel14.mgmt
10.29.160.67    rhel15.mgmt
10.29.160.68    rhel16.mgmt
... 
 
   
192.168.12.11 rhel1
192.168.12.12 rhel2
192.168.12.13 rhel3
192.168.12.14 rhel4
192.168.12.15 rhel5
192.168.12.16 rhel6
192.168.12.17 rhel7
192.168.12.18 rhel8
192.168.12.19 rhel9
192.168.12.20 rhel10
192.168.12.21 rhel11
192.168.12.22 rhel12
192.168.12.23 rhel13
192.168.12.24 rhel14
192.168.12.25 rhel15
192.168.12.26 rhel16
...

On Other nodes (rhel2-rhel64)

 
   
vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
 
   

2. Update /etc/resolv.conf file to point to Admin Node

vi /etc/resolv.conf
nameserver 10.29.160.53
 
   

Note This step is required if you are setting up dnsmasq on Admin node, else this file should be updated with the correct nameserver.


3. Deploy /etc/resolv.conf from the admin node (rhel1) to all the nodes via the following pscp command:

pscp -h /root/allnodes /etc/resolv.conf /etc/resolv.conf
 
   

4. Start dnsmasq on Admin node.

service dnsmasq start
 
   

5. Ensure DNS is working fine by running the following command on Admin node and any data-node.

[root@rhel2 ~]# nslookup rhel1.mgmt
Server: 10.29.160.53
Address: 10.29.160.53#53
 
   
Name: rhel1.mgmt
Address: 10.29.160.53
 
   
[root@rhel2 ~]# nslookup rhel1
Server: 10.29.160.53
Address: 10.29.160.53#53
 
   
Name: rhel1
Address: 192.168.12.11
 
   

Creating RedHat Local Repository

To create a repository using RHEL DVD or ISO on the admin node (in this deployment rhel1 is used for this purpose), create a directory with all the required rpms, run the createrepo command and then publish the resulting repository.

1. Login to rhel1 node, and run the following command to create a directory that would contain the repository:

mkdir -p /var/www/html/rhelrepo64
 
   

2. Copy the contents of the Red Hat DVD to /var/www/html/rhelrepo64.

3. Alternatively, if you have access to a Red Hat ISO Image, copy the ISO file to rhel1.

scp rhel-server-6.4-x86_64-dvd.iso rhel1:/root
 
   

Assuming the Red Hat ISO file is located in your working directory.

mkdir -p /mnt/rheliso
mount -t iso9660  -o loop /root/rhel-server-6.4-x86_64-dvd.iso /mnt/rheliso/
 
   

4. Copy the contents of the ISO to the /var/www/html/rhelrepo directory.

cp -r /mnt/rheliso/* /var/www/html/rhelrepo64
 
   

5. Run the following command on the rhel1 to create a .repo file that enables the use of the yum command:

vi /var/www/html/rhelrepo/rheliso.repo
[rhel6.4]
name=Red Hat Enterprise Linux 6.4
baseurl=http://10.29.160.53/rhelrepo64
gpgcheck=0
enabled=1
 
   

Note The yum command based on the repo file requires httpd to be running on rhel1 so that the other nodes can access the repository. Steps to install and configure httpd are given in the following section.


6. Copy the rheliso.repo to all the nodes of the cluster.

pscp -h /root/allnodes /var/www/html/rhelrepo/rheliso.repo /etc/yum.repos.d/

7. To use the repository files on rhel1 without httpd, edit the baseurl of the repo file. etc/yum.repos.d/rheliso.repo to point repository location in the file system.

vi /etc/yum.repos.d/rheliso.repo
[rhel6.4]
name=Red Hat Enterprise Linux 6.4
baseurl=file:///var/www/html/rhelrepo64
gpgcheck=0
enabled=1
 
   

8. Run pssh -h /root/allnodes "yum clean all" command:

Creating the Red Hat Repository Database

1. Install the createrepo package.

2. Use the createrepo package to regenerate the repository database(s) for the local copy of the RHEL DVD contents.

3. Purge the yum caches:

yum -y install createrepo
cd /var/www/html/rhelrepo64
createrepo .
yum clean all

Upgrading LSI driver

The latest LSI driver is essential for performance and bug fixes.

To download the latest LSI drivers, see:

http://software.cisco.com/download/release.html?mdfid=284296254&flowid=31743&softwareid=283853158&release=1.5.1&relind=AVAILABLE&rellifecycle=&reltype=latest

1. In the ISO image, the required driver kmod-megaraid_sas-v06.504.01.00.rpm can be located at: ucs-cxxx-drivers.1.5.1\Linux\Storage\LSI\92xx\RHEL\RHEL6.4

2. Download and transfer kmod-megaraid_sas-v06.504.01.00.rpm driver to the admin node (rhel1).

3. Run the following commands to install the rpm on all nodes of the cluster:

pscp -h /root/allnodes kmod-megaraid_sas-v06.504.01.00_rhel6.4-2.x86_64.rpm /root/
 
   

pssh -h /root/allnodes  "rpm -ivh 
kmod-megaraid_sas-v06.504.01.00_rhel6.4-2.x86_64.rpm"

4. Run the following command to verify the version of kmod-megaraid_sas driver is used on all the nodes (confirm all versions are same):

pssh -h /root/allnodes  "modinfo megaraid_sas | head -5"

Installing httpd

1. Install httpd on the admin node to host repositories.

The Red Hat repository is hosted using http on the admin node, and this machine is accessible by all the hosts in the cluster.

yum -y install httpd
 
   

2. Add ServerName, and make the necessary changes to the server configuration file.

/etc/httpd/conf/httpd.conf
SeverName 10.29.160.53:80

3. Run the following command to make sure that the httpd is able to read the repofiles:

chcon -R -t httpd_sys_content_t /var/www/html/rhelrepo64
 
   

4. Run the following command to start httpd:

service httpd start
chkconfig httpd on

Enabling Syslog

Syslog must be enabled on each node to preserve logs regarding killed processes or failed jobs. Modern versions such as syslog-ng and rsyslog are possible, making it more difficult to ascertain if a syslog daemon is present.

Run any of the commands to confirm if the service is properly configured:

clush -B -a rsyslogd -v
clush -B -a service rsyslog status

Setting Ulimit

On each node, ulimit -n specifies the number of inodes that can be opened simultaneously. With the default value of 1024, the system appears to be out of disk space and shows no inodes available. This value should be set to 64000 on every node.

Higher values are unlikely to result in an appreciable performance gain.

1. For setting ulimit on Redhat, run the command Edit /etc/security/limits.conf and add the following lines:

root soft nofile 64000
root hard nofile 64000
 
   

Note The ulimit values are applied on a new shell. Running the command on a node on an earlier instance of a shell shows old values.


2. To verify the ulimit setting, run the following command:

clush -B -a ulimit -n
 
   

The command should report 64000 as the ulimit.

Disabling SELinux

SELinux must be disabled during the ID installation procedure and cluster setup. SELinux can be enabled after installation and while the cluster is running.

SELinux can be disabled by editing /etc/selinux/config and changing the SELINUX line to SELINUX=disabled.

1. Run the following command to disable SELINUX on all nodes:

pssh -h /root/allnodes "sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' 
/etc/selinux/config"

 
   
pssh -h /root/allnodes "setenforce 0"

Note This command fails if SELinux is already disabled.


Setting TCP Retries

Adjusting the tcp_retries parameter for the system network enables faster detection of failed nodes. Given the advanced networking features of UCS, this is a safe and recommended change (failures observed at the Operating System layer are mostly serious rather than transitory). On each node, set the number of TCP retries to 5 can help detect unreachable nodes with less latency.

1. Edit the file /etc/sysctl.conf and add the following line:

net.ipv4.tcp_retries2=5
 
   

2. Save the file and run the following command.

clush -B -a sysctl -p

Disabling the Linux Firewall

The default Linux firewall settings are far too restrictive for any Hadoop deployment. Since the Cisco UCS Big Data deployment is performed in the isolated network, there is no need to leave the iptables service running.

1. Run the following commands to disable the iptables:

pssh -h /root/allnodes  "service iptables stop"

2. Run the following command to check if the iptables are disabled:

pssh -h /root/allnodes "chkconfig iptables off"

Configuring Data Drives on NameNode

This section provides the steps to configure data drives on the NameNode.

The first two disk drives are configured for the Operating System on the nodes, rhel1 and rhel2, as shown in "Configuring Disk Drives for Operating System on NameNode" section. The remaining disk drives can be configured similarly or by using MegaCli.

1. From the LSI website http://www.lsi.com/support/Pages/Download-Results.aspx?keyword=9266-8i, download MegaCli and its dependencies and transfer the to the admin node.

scp  /root/MegaCli64  rhel1:/root/
scp  /root/Lib_Utils-1.00-08.noarch.rpm rhel1:/root/
scp  /root/Lib_Utils2-1.00-01.noarch.rpm rhel1:/root/
 
   

2. To copy all the three files to all the nodes, run the following commands:

pscp -h  /root/allnodes  /root/MegaCli64  /root/

 
   
pscp -h  /root/allnodes  /root/Lib_Utils*  /root/

3. Run the following command to install the rpms on all the nodes:

pssh -h /root/allnodes "rpm -ivh Lib_Utils*"

4. Run the following script as root user on NameNode and Secondary NameNode to create the virtual drives.

vi /root/raid1.sh
./MegaCli64 -cfgldadd 
r1[$1:3,$1:4,$1:5,$1:6,$1:7,$1:8,$1:9,$1:10,$1:11,$1:12,$1:13,$1:14,$1:15,$1:16,$1
:17,$1:18,$1:19,$1:20,$1:21,$1:22,$1:23,$1:24] wb ra nocachedbadbbu strpsz1024 -a0
 
   
The above script requires enclosure ID as a parameter. Run the following command 
to get enclousure id.
 
   
./MegaCli64 pdlist -a0 | grep Enc | grep -v 252 | awk '{print $4}' | sort | uniq 
-c | awk '{print $2}'
 
   
chmod 755 raid1.sh
 
   
Run MegaCli script as follows
 
   
./raid1.sh <EnclosureID obtained by running the command above>
 
   
WB: Write back
RA: Read Ahead
NoCachedBadBBU: Do not write cache when the BBU is bad. 
Strpsz1024: Strip Size of 1024K
 
   

Note The command above will not override any existing configuration. To clear and reconfigure the existing configurations, see Embedded MegaRAID Software Users Guide available at: www.lsi.com.


Configuring the Filesystem for NameNodes

To Configure the filesystem for NameNodes, run the following script:

vi /root/driveconf.sh
#!/bin/bash
disks_count=`lsblk -id | grep sd  | wc -l`
if [ $disks_count -eq 2 ];  then
    echo "Found 2 disks"
else
    echo "Found $disks_count disks. Expecting 2. Exiting.."
    exit 1
fi
[[ "-x" == "${1}" ]] && set -x && set -v && shift 1
for X in /sys/class/scsi_host/host?/scan
do
echo '- - -' > ${X}
done
for X in /dev/sd?
do
echo $X
if [[ -b ${X} && `/sbin/parted -s ${X} print quit|/bin/grep -c boot` -ne 0 ]]
then
echo "$X bootable - skipping."
continue
else
Y=${X##*/}1
/sbin/parted  -s  ${X} mklabel gpt quit
/sbin/parted  -s  ${X} mkpart 1 6144s 100% quit
/sbin/mkfs.xfs -f -q -l size=65536b,lazy-count=1,su=256k -d sunit=1024,swidth=6144 -r 
extsize=256k -L ${Y} ${X}1
(( $? )) && continue
/bin/mkdir  -p  /IDH/${Y}
(( $? )) && continue
/bin/mount  -t xfs  -o allocsize=128m,noatime,nobarrier,nodiratime  ${X}1  /IDH/${Y}
(( $? )) && continue
echo "LABEL=${Y} /IDH/${Y} xfs allocsize=128m,noatime,nobarrier,nodiratime 0 0" >> 
/etc/fstab
fi
done
 
   

Configuring Data Drives on DataNodes

This section provides the steps to configure data drives on DataNodes.

The first disk drive is configured for the Operating System on all the DataNodes, rhel3 to rhel64 as shown in "Configuring Disk Drives for Operating System on DataNodes" section. The remaining disk drives can be configured similarly or by using MegaCli.

Run the following command from the admin node to create the virtual drives with RAID 0 configurations on all the DataNodes.

pssh -h /root/datanodes "./MegaCli64 -cfgeachdskraid0 WB RA direct NoCachedBadBBU 
strpsz1024 -a0"
 
   

WB: Write back

RA: Read Ahead

NoCachedBadBBU: Do not write cache when the BBU is bad

Strpsz1024: Strip Size of 1024K


Note The above command will not override existing configurations. To clear and reconfigure the existing configurations, see Embedded MegaRAID Software Users Guide available at: www.lsi.com.


Configuring the Filesystem for DataNodes

This section describes the procedure to configure the filesystem for DataNodes.

1. On the Admin node, create a file containing the following script.

To create partition tables and file systems on the local disks of each nodes, run the following script as the root user on all the nodes.

vi /root/driveconf.sh
#!/bin/bash
disks_count=`lsblk -id | grep sd  | wc -l`
if [ $disks_count -eq 24 ];  then
    echo "Found 24 disks"
else
    echo "Found $disks_count disks. Expecting 24. Exiting.."
    exit 1
fi
[[ "-x" == "${1}" ]] && set -x && set -v && shift 1
for X in /sys/class/scsi_host/host?/scan
do
echo '- - -' > ${X}
done
count=1
for X in /dev/sd?
do
echo $X
if [[ -b ${X} && `/sbin/parted -s ${X} print quit|/bin/grep -c boot` -ne 0 ]]
then
echo "$X bootable - skipping."
continue
else
Y=${X##*/}1
/sbin/parted  -s  ${X} mklabel gpt quit
/sbin/parted  -s  ${X} mkpart 1 6144s 100% quit
/sbin/mkfs.xfs -f -q -l size=65536b,lazy-count=1,su=256k -d sunit=1024,swidth=6144 
-r extsize=256k -L ${Y} ${X}1
(( $? )) && continue
/bin/mkdir  -p  /mnt/disk$count
(( $? )) && continue
/bin/mount  -t xfs  -o allocsize=128m,noatime,nobarrier,nodiratime  ${X}1  
/mnt/disk$count
(( $? )) && continue
echo "LABEL=${Y} /mnt/disk$count xfs allocsize=128m,noatime,nobarrier,nodiratime 0 
0" >> /etc/fstab
((count++))
fi
done
 
   

Note This script would mount the non-bootable 23 drives, that is, /dev/sd<b-x> on /mnt/disk<1-23>. ID choose /mnt/disk<1-n> as the default location for formatting and storing HDFS data on a data-node. This could be overridden during installation.


2. Run the following command to copy driveconf.sh to all the DataNodes.

pscp -h /root/datanodes /root/driveconf.sh /root/
 
   

3. Run the following command from the admin node to run the script across all DataNodes.

pssh -h /root/datanodes "./driveconf.sh" 

Installing Intel Distribution for Apache Hadoop

Prerequisites for ID Installation

This section details the pre-requisites for ID Installation including setting up of local Intel Repo, Kerberos Setup if planning for Kerberos authenticated communication between nodes instead of just simple ssh. More information on ID installation is available at:

http://hadoop.intel.com/resources

Copy Intel Distribution to Admin Node

Download Intel Distribution tarball from http://hadoop.intel.com and extract Intel Distribution on the Admin node:

tar -xvf intelhadoop-2.5-en-<commercial/evaluation>.el6.x86_64.tar.gz
 
   

This creates the directory "intelhadoop" which has the following contents.

Figure 72 Created Intelhadoop Directory

Create Intel Repo

Create /var/www/html/intel directory on the Admin Node and copy directories "idh" and "manager" from intelhadoop to /var/www/html/intel

cp -r idh /var/www/html/intel/
cp -r manager /var/www/html/intel/
 
   

Run createrepo in order to create the repo from /var/www/html/intel/

cd /var/www/html/intel
createrepo

Kerberos Setup

This is needed if ID cluster deployment is going to use Kerberos authentication. This document will describe Kerberos setup with openLDAP.


NoteSimple ssh authentication doesn't require this step.

This is a simple installation to demonstrate the integration of Kerberos with ldap and Intel Manager. You may have to customize it based on your needs.

Install OpenLDAP and Integrate with Kerberos

Basic OpenLDAP Setup


Note The following steps are not required if OpenLDAP is installed already.


1. On the admin node install the following RPMs:

yum -y install openldap-servers openldap-clients krb5-server-ldap
 
   

2. Open file vi /etc/openldap/slapd.d/cn\=config/olcDatabase\=\{2\}bdb.ldif in the text editor and change the following two lines as follows

olcSuffix: dc=my-domain,dc=com
olcRootDN: cn=Manager,dc=my-domain,dc=com

to

olcSuffix: dc=IDH, dc=COM
olcRootDN: cn=Manager,dc=IDH,dc=COM
 
   

3. Insert the following line after the olcRootDN and save changes

olcRootPW: {SSHA}pW8s+vP2UbJxSa4Obts5h2iQ2qy/tmGr
 
   

This is a SHA1 hash of the root DN's password. This password in cleartext is "passwd".


Note This value is obtained by running slappasswd command as shown below.


#slappasswd
    New password: <entered passwd>
    Re-enter new password: <passwd again>
    {SSHA}pW8s+vP2UbJxSa4Obts5h2iQ2qy/tmGr
 
   

4. cp /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG

5. chown -R ldap:ldap /var/lib/ldap

6. chmod -R 700 /var/lib/ldap

7. Replace the contents of /etc/sysconfig/ldap with the following:

# Options of slapd (see man slapd)
# Use this one to debug ACL...
# SLAPD_OPTIONS="-4 -d 128"
# Use this one for day-to-day production usage.
SLAPD_OPTIONS="-4"
 
   
# Run slapd with -h "... ldap:/// ..."
#   yes/no, default: yes
SLAPD_LDAP=yes
 
   
# Run slapd with -h "... ldapi:/// ..."
#   yes/no, default: yes
SLAPD_LDAPI=yes
 
   
# Run slapd with -h "... ldaps:/// ..."
#   yes/no, default: no
SLAPD_LDAPS=no
 
   
# Maximum allowed time to wait for slapd shutdown on 'service ldap 
# stop' (in seconds)
SLAPD_SHUTDOWN_TIMEOUT=15
 
   

8. Edit the file /etc/openldap/ldap.conf to reflect the following changes:

BASE  dc=IDH,dc=COM
URI  ldap://<FQDN of LDAP server>  
#  for the cluster used in the document it is ldap://rhel1
TIMELIMIT 15
TIMEOUT  20
 
   

9. Insert the following lines /etc/rsyslog.conf.

# Send slapd(8c) logs to /var/log/slapd.log
if $programname == 'slapd' then /var/log/slapd.log
& ~
 
   

10. touch /var/log/slapd.log

11. service rsyslog restart

12. Start up the LDAP server as follows. By default, the server listens on port 389.

service slapd start
 
   

13. Verify that root DN can bind by running the command

ldapwhoami -D cn=Manager,dc=IDH,dc=COM -w passwd
 
   

This should give the output

dn:cn=Manager,dc=IDH,dc=COM
 
   

14. Create hadoop.ldif file in the home directory with the following data

version: 1
 
   
dn: dc=IDH,dc=COM
objectClass: top
objectClass: dcObject
objectClass: organization
dc: IDH
o: iDevelopment.info LDAP Server
 
   
dn: ou=users,dc=IDH,dc=COM
objectClass: top
objectClass: organizationalUnit
ou: users
description: All users in iDevelopment.info
 
   
dn: ou=groups,dc=IDH,dc=COM
objectClass: top
objectClass: organizationalUnit
ou: groups
description: All groupss in iDevelopment.info
 
   
dn: cn=Manager,dc=IDH,dc=COM
objectClass: organizationalRole
cn: Manager
description: Rootdn
 
   
dn: cn=John Doe,ou=users,dc=IDH,dc=COM
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
cn: John Doe
sn: Doe
description: All users in iDevelopment.info
ou: users
uid: jdoe
 
   
dn: cn=administrators,ou=groups,dc=IDH,dc=COM
objectClass: groupOfNames
objectClass: top
cn: administrators
member: cn=John Doe,ou=users,dc=IDH,dc=COM
 
   
dn: ou=kerberos,dc=IDH,dc=COM
ou: kerberos
objectClass: top
objectClass: organizationalUnit
description: Kerberos OU to store Kerberos principals. 
 
   

15. Before running a MapReduce job, the user under which the MapReduce job runs must have a unix account on each tasktracker and kinit must have been run thereby caching kerberos token. Please check "Running a Mapreduce Job after Kerberos Setup" section for more details on how to do this.

16. Load the LDIF into the LDAP with the following command:

ldapadd -H ldap://localhost:389 -x -a -D "cn=Manager,dc=IDH,dc=COM" -f hadoop.ldif 
-w passwd
 
   

This gives the output as follows:

adding new entry "dc=IDH,dc=COM"
 
   
adding new entry "ou=users,dc=IDH,dc=COM"
 
   
adding new entry "ou=groups,dc=IDH,dc=COM"
 
   
adding new entry "cn=Manager,dc=IDH,dc=COM"
 
   
adding new entry "cn=John Doe,ou=users,dc=IDH,dc=COM"
 
   
adding new entry "cn=administrators,ou=groups,dc=IDH,dc=COM"
 
   
adding new entry "ou=kerberos,dc=IDH,dc=COM"
 
   

17. To view the entries in the LDAP, execute the following command:

ldapsearch -h localhost -D "cn=Manager,dc=IDH,dc=com" -w passwd -b "dc=IDH,dc=com"
 
   

18. To set password for the jdoe user:

ldappasswd -xZWD cn=Manager,dc=IDH,dc=COM -S "cn=John Doe,ou=users,dc=IDH,dc=COM"

Install Kerberos Server

krb5-workstation rpm should be installed on all nodes in the cluster.

Run the following command from the admin node:.

clush -B -a yum -y install krb5-workstation 
 
   

Install the following rpms as follows on the Admin Node.

yum -y install krb5-pkinit-openssl krb5-libs krb5-server-ldap krb5-server 
 
   

Note Make sure that the krb5-workstation is installed on the Admin node before running the above command.


Import the Kerberos Schema and Setup LDAP Identities for Kerberos

1. Create directory /root/ldap and cd to /root/ldap.

2. cp /usr/share/doc/krb5-server-ldap-1.10.3/kerberos.schema /etc/openldap/schema/.

3. Create a configuration file named schema_convert.conf within /root/ldap, or a similar descriptive name, containing the following lines:

include /etc/openldap/schema/core.schema
include /etc/openldap/schema/collective.schema
include /etc/openldap/schema/corba.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/duaconf.schema
include /etc/openldap/schema/dyngroup.schema
include /etc/openldap/schema/inetorgperson.schema
include /etc/openldap/schema/java.schema
include /etc/openldap/schema/misc.schema
include /etc/openldap/schema/nis.schema
include /etc/openldap/schema/openldap.schema
include /etc/openldap/schema/ppolicy.schema
include /etc/openldap/schema/kerberos.schema
 
   

4. mkdir /root/ldap/ldif_output.

5. Use slapcat to convert the schema files:

slapcat -f schema_convert.conf -F /root/ldap/ldif_output -n0 -s 
"cn={12}kerberos,cn=schema,cn=config" > cn=kerberos.ldif
 
   

6. In /root/ldap/cn\=kerberos.ldif file, remove the sequence number from in front of the kerberos CN. Remove the number and brackets.

That is in file cn=kerberos.ldif replace
dn: cn={12}kerberos,cn=schema,cn=config
cn: {12}kerberos
as 
dn: cn=kerberos,cn=schema,cn=config
cn: kerberos
 
   

7. Remove the following lines from the end of the cn=kerberos.ldif:

structuralObjectClass: olcSchemaConfig
entryUUID: 18ccd010-746b-102d-9fbe-3760cca765dc
creatorsName: cn=config
createTimestamp: 20090111203515Z
entryCSN: 20090111203515.326445Z#000000#000#000000
modifiersName: cn=config
modifyTimestamp: 20090111203515Z
 
   

8. Add the kerberos ldif.

ldapadd -c -Y EXTERNAL -H ldapi:/// -f cn\=kerberos.ldif
 
   

This gives the output

SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
adding new entry "cn=kerberos,cn=schema,cn=config"
 
   

9. Add an index for the krb5principalname attribute by inserting the following lines into /etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif.

vi /etc/openldap/slapd.d/cn\=config/olcDatabase\=\{2\}bdb.ldif
olcDbIndex: krbPrincipalName eq,pres,sub
Ensure the line is added as a last entry along with other olcDbindex as shown 
below
olcDbIndex: nisMapEntry pres,eq,sub
olcDbIndex: krbPrincipalName eq,pres,sub
olcDbLinearIndex: FALSE

Edit Kerberos Configuration files

/var/kerberos/krb5kdc/kdc.conf

In /var/kerberos/krb5kdc/kdc.conf, remove any existing content and then insert the following:

[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88
 v4_mode = nopreauth
 
   
[realms]
IDH.COM = {
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  max_life = 10h 0m 0s
  max_renewable_life = 7d 0h 0m 0s
  master_key_type = des3-hmac-sha1
supported_enctypes = arcfour-hmac:normal des3-hmac-sha1:normal des-cbc-crc:normal 
des:normal des:v4 des:norealm des:onlyrealm des:afs3
  default_principal_flags = +preauth
 }
 
   

Note If this file doesn't exist, create this file with the above content. The realm needs to be changed accordingly.


Change Kerberos ACL

In /var/kerberos/krb5kdc/kadm5.acl, change the value as follows
*/admin@IDH.COM     *
 
   

Change /etc/krb5.conf

Change /etc/krb5.conf with the following content. 
 
   
[libdefaults]
default_realm = IDH.COM
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
allow_weak_crypto = true
default_tkt_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1 
default_tgs_enctypes = des-cbc-md5 des-cbc-crc des3-cbc-sha1 
[realms]
IDH.COM = {
kdc = rhel1  # FQDN for the admin node
admin_server = rhel1  # FQDN for the admin node 
database_module = openldap_ldapconf
}
 
   
 
   
[domain_realm] 
.rhel1 = IDH.COM
rhel1 = IDH.COM
 
   
 
   
[appdefaults] 
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
}
 
   
[dbmodules] 
openldap_ldapconf = {
db_library = kldap
ldap_kerberos_container_dn = ou=kerberos,dc=IDH,dc=COM
ldap_kdc_dn = cn=Manager,dc=IDH,dc=COM
# this object needs to have read rights on
# the realm container, principal container and realm sub-trees
ldap_kadmind_dn = cn=Manager,dc=IDH,dc=COM
# this object needs to have read and write rights on
# the realm container, principal container and realm sub-trees
ldap_service_password_file = /etc/krb5.d/stash.keyfile
ldap_servers = ldapi:///
ldap_conns_per_server = 5
}
 
   
Copy krb5.conf to all nodes
 
   
clush -b -w rhel[2-64] -c /etc/krb5.conf --dest /etc/

Create KDC Entries in LDAP

Execute the following command and provide the password "passwd" when prompted.

kdb5_ldap_util -D "cn=Manager,dc=IDH,dc=COM" create -subtrees 
"ou=kerberos,dc=IDH,dc=COM" -r IDH.COM -s
 
   

This prompts the following:

Password for "cn=Manager,dc=IDH,dc=COM":  ? Provide "passwd"
Initializing database for realm 'IDH.COM'
You will be prompted for the database Master Password.  ? This is setting Kerberos 
Password
It is important that you NOT FORGET this password.
Enter KDC database master key: ? Provided "passwd" for simplicity
Re-enter KDC database master key to verify: 
 
   

Verify LDAP tree by running the following command

ldapsearch -LLLY EXTERNAL -H ldapi:/// -b ou=kerberos,dc=IDH,dc=COM dn
 
   

The result should be:

SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
dn: ou=kerberos,dc=IDH,dc=COM
 
   
dn: cn=IDH.COM,ou=kerberos,dc=IDH,dc=COM
 
   
dn: krbPrincipalName=K/M@IDH.COM,cn=IDH.COM,ou=kerberos,dc=IDH,dc=COM
 
   
dn: krbPrincipalName=krbtgt/IDH.COM@IDH.COM,cn=IDH.COM,ou=kerberos,dc=IDH,dc=C
 OM
 
   
dn: krbPrincipalName=kadmin/admin@IDH.COM,cn=IDH.COM,ou=kerberos,dc=IDH,dc=COM
 
   
dn: krbPrincipalName=kadmin/changepw@IDH.COM,cn=IDH.COM,ou=kerberos,dc=IDH,dc=
 COM
 
   
dn: krbPrincipalName=kadmin/history@IDH.COM,cn=IDH.COM,ou=kerberos,dc=IDH,dc=C
 OM
 
   
dn: krbPrincipalName=kadmin/rhel1@IDH.COM,cn=IDH.COM,ou=kerberos,dc=IDH,dc=COM
 
   

Create a keyfile as follows for passwordless communication between LDAP and Kerberos

mkdir /etc/krb5.d
 
   

Execute the following command

#kdb5l_dap_util -D "cn=Manager,dc=IDH,dc=COM" stashsrvpw -f /etc/krb5.d/stash.keyfile 
"cn=Manager,dc=IDH,dc=COM"
 
   
Password for "cn=Manager,dc=IDH,dc=COM": 
Password for "cn=Manager,dc=IDH,dc=COM":
Re-enter password for "cn=Manager,dc=IDH,dc=COM":
 
   
#cat /etc/krb5.d/stash.keyfile
cn=Manager,dc=IDH,dc=COM#{HEX}706173737764
 
   

Set up KDC Logging and Starting the KDC Daemons

Run the following commands to setup KDC logging and start Kerberos daemons

1. touch /var/log/krb5kdc.log /var/log/kadmind.log

2. Insert the following lines /etc/rsyslog.conf***ter the following lines to integrate KDC logging

After
# Send slapd(8c) logs to /var/log/slapd.log
if $programname == 'slapd' then /var/log/slapd.log
Insert
# Send kadmind(8) logs to /var/log/kadmind.log
if $programname == 'kadmind' then /var/log/kadmind.log
& ~
# Send krb5kdc(8) logs to /var/log/krb5kdc.log
if $programname == 'krb5kdc' then /var/log/krb5kdc.log
& ~
 
   

3. Run the command to restart rsyslog

service rsyslog restart
 
   

4. chkconfig krb5kdc on

5. chkconfig kadmin on

6. Start KDC Daemons as follows:

# /etc/init.d/krb5kdc start
Starting Kerberos 5 KDC: [ OK 
# /etc/init.d/kadmin start
Starting Kerberos 5 Admin Server: [ OK ] 

Generating KeyTab Files

The keytab files are needed when configuring Kerberos though Intel Manager. Follow these steps to generate the keytab files:

1. Create the script keytab.sh in the home directory with following content

#! /bin/bash 
set -u 
 
   
if [ $# -lt 1 ]
then
    echo 
    echo "ERROR: neeed to specify hostnames that keytabs will be created for"
    exit 1
fi
 
   
realm=IDH.COM
 
   
keytabdir=/tmp/keytabs/clusterkeytabs$$
mkdir -p $keytabdir
 
   
while [ $# -gt 0 ]
do
    hostname=${1}
    shift 1
    
    nodedir=$keytabdir/$hostname
 
   
    mkdir -p ${nodedir}
    if [ $? -ne 0 ]
    then
	echo "Failed to created ${nodedir}"
	exit 1
    fi
 
   
    cd ${nodedir}
 
   
    echo
    echo "generating keytabs for ${hostname} in ${nodedir}"
    
    echo
    echo "Generating HDFS keytabs for ${hostname}"
    #create keytabs for hdfs
    kadmin.local -w secure -q "addprinc -randkey hdfs/${hostname}@${realm}"
    kadmin.local -w secure -q "xst -k hdfs.keytab hdfs/${hostname}@${realm}"
    kadmin.local -w secure -q "addprinc -randkey host/${hostname}@${realm}"
    kadmin.local -w secure -q "xst -k hdfs.keytab host/${hostname}@${realm}"
    kadmin.local -w secure -q "addprinc -randkey HTTP/${hostname}@${realm}"
    kadmin.local -w secure -q "xst -k HTTP.keytab HTTP/${hostname}@${realm}"
 
   
    echo
    echo "Generating MapReduce keytabs for ${hostname}"
    #create keytabs for mapreduce
 
   
    kadmin.local -w secure -q "addprinc -randkey mapred/${hostname}@${realm}"
    kadmin.local -w secure -q "xst -k mapred.keytab mapred/${hostname}@${realm}"
 
   
    #create keytabs for hive
    echo
    echo "Generating Hive keytabs for ${hostname}"
    kadmin.local -w secure -q "addprinc -randkey hive/${hostname}@${realm}"
    kadmin.local -w secure -q "xst -k hive.keytab hive/${hostname}@${realm}"
 
   
    
    #create keytab for hbase
    echo
    echo "Generating hbase keytabs for ${hostname}"
    kadmin.local -w secure -q "addprinc -randkey hbase/${hostname}@${realm}"
    kadmin.local -w secure -q "xst -k hbase.keytab hbase/${hostname}@${realm}"
 
   
    #create keytab for zookeeper
    echo
    echo "Generating zookeeper keytab for ${hostname}"
    kadmin.local -w secure -q "addprinc -randkey zookeeper/${hostname}@${realm}"
    kadmin.local -w secure -q "xst -k zookeeper.keytab 
zookeeper/${hostname}@${realm}"
 
   
 
   
    #create keytab for oozie
    echo
    echo "Generating oozie keytab for ${hostname}"
    kadmin.local -w secure -q "addprinc -randkey oozie/${hostname}@${realm}"
    kadmin.local -w secure -q "xst -k oozie.keytab oozie/${hostname}@${realm}"
 
   
 
   
    #regenerating keytab so that it avoids the gss init error
    kinit -k -t hdfs.keytab hdfs/${hostname}@${realm}
    kinit -k -t HTTP.keytab HTTP/${hostname}@${realm}
    kinit -k -t hive.keytab hive/${hostname}@${realm}
    kinit -k -t mapred.keytab hdfs/${hostname}@${realm}
    kinit -k -t hbase.keytab hbase/${hostname}@${realm}
    kinit -k -t zookeeper.keytab zookeeper/${hostname}@${realm}
    kinit -k -t oozie.keytab oozie/${hostname}@${realm}
 
   
done
 
   
kinit -R
 
   
cd /tmp/keytabs
 
   
basename="$(basename ${keytabdir})"
 
   
tar cvf ${basename}.tar ${basename}
 
   
echo "Check /tmp/keytabs for the tarball that contains all the keytabs"
 
   

2. Run the keytab script as follows:

./keytab.sh rhel1 rhel2 rhel3 rhel4 rhel5 rhel6 rhel7 rhel8 rhel9 rhel10 rhel11 
rhel12 rhel13 rhel14 rhel15 rhel16 ... rhel64
 
   
The sample output of the script for one of the nodes (rhel16) is as follows
 
   
rhel16 in /tmp/keytabs/clusterkeytabs3803/rhel16
 
   
Generating HDFS keytabs for rhel16
Authenticating as principal oozie/admin@IDH.COM with password.
WARNING: no policy specified for hdfs/rhel16@IDH.COM; defaulting to no policy
Principal "hdfs/rhel16@IDH.COM" created.
Authenticating as principal oozie/admin@IDH.COM with password.
Entry for principal hdfs/rhel16@IDH.COM with kvno 2, encryption type arcfour-hmac 
added to keytab WRFILE:hdfs.keytab.
Entry for principal hdfs/rhel16@IDH.COM with kvno 2, encryption type des3-cbc-sha1 
added to keytab WRFILE:hdfs.keytab.
Entry for principal hdfs/rhel16@IDH.COM with kvno 2, encryption type des-cbc-crc 
added to keytab WRFILE:hdfs.keytab.
Authenticating as principal oozie/admin@IDH.COM with password.
WARNING: no policy specified for host/rhel16@IDH.COM; defaulting to no policy
Principal "host/rhel16@IDH.COM" created.
Authenticating as principal oozie/admin@IDH.COM with password.
Entry for principal host/rhel16@IDH.COM with kvno 2, encryption type arcfour-hmac 
added to keytab WRFILE:hdfs.keytab.
Entry for principal host/rhel16@IDH.COM with kvno 2, encryption type des3-cbc-sha1 
added to keytab WRFILE:hdfs.keytab.
Entry for principal host/rhel16@IDH.COM with kvno 2, encryption type des-cbc-crc 
added to keytab WRFILE:hdfs.keytab.
Authenticating as principal oozie/admin@IDH.COM with password.
WARNING: no policy specified for HTTP/rhel16@IDH.COM; defaulting to no policy
Principal "HTTP/rhel16@IDH.COM" created.
Authenticating as principal oozie/admin@IDH.COM with password.
Entry for principal HTTP/rhel16@IDH.COM with kvno 2, encryption type arcfour-hmac 
added to keytab WRFILE:HTTP.keytab.
Entry for principal HTTP/rhel16@IDH.COM with kvno 2, encryption type des3-cbc-sha1 
added to keytab WRFILE:HTTP.keytab.
Entry for principal HTTP/rhel16@IDH.COM with kvno 2, encryption type des-cbc-crc 
added to keytab WRFILE:HTTP.keytab.
 
   
Generating MapReduce keytabs for rhel16
Authenticating as principal oozie/admin@IDH.COM with password.
WARNING: no policy specified for mapred/rhel16@IDH.COM; defaulting to no policy
Principal "mapred/rhel16@IDH.COM" created.
Authenticating as principal oozie/admin@IDH.COM with password.
Entry for principal mapred/rhel16@IDH.COM with kvno 2, encryption type 
arcfour-hmac added to keytab WRFILE:mapred.keytab.
Entry for principal mapred/rhel16@IDH.COM with kvno 2, encryption type 
des3-cbc-sha1 added to keytab WRFILE:mapred.keytab.
Entry for principal mapred/rhel16@IDH.COM with kvno 2, encryption type des-cbc-crc 
added to keytab WRFILE:mapred.keytab.
 
   
Generating Hive keytabs for rhel16
Authenticating as principal oozie/admin@IDH.COM with password.
WARNING: no policy specified for hive/rhel16@IDH.COM; defaulting to no policy
Principal "hive/rhel16@IDH.COM" created.
Authenticating as principal oozie/admin@IDH.COM with password.
Entry for principal hive/rhel16@IDH.COM with kvno 2, encryption type arcfour-hmac 
added to keytab WRFILE:hive.keytab.
Entry for principal hive/rhel16@IDH.COM with kvno 2, encryption type des3-cbc-sha1 
added to keytab WRFILE:hive.keytab.
Entry for principal hive/rhel16@IDH.COM with kvno 2, encryption type des-cbc-crc 
added to keytab WRFILE:hive.keytab.
 
   
Generating hbase keytabs for rhel16
Authenticating as principal oozie/admin@IDH.COM with password.
WARNING: no policy specified for hbase/rhel16@IDH.COM; defaulting to no policy
Principal "hbase/rhel16@IDH.COM" created.
Authenticating as principal oozie/admin@IDH.COM with password.
Entry for principal hbase/rhel16@IDH.COM with kvno 2, encryption type arcfour-hmac 
added to keytab WRFILE:hbase.keytab.
Entry for principal hbase/rhel16@IDH.COM with kvno 2, encryption type 
des3-cbc-sha1 added to keytab WRFILE:hbase.keytab.
Entry for principal hbase/rhel16@IDH.COM with kvno 2, encryption type des-cbc-crc 
added to keytab WRFILE:hbase.keytab.
 
   
Generating zookeeper keytab for rhel16
Authenticating as principal oozie/admin@IDH.COM with password.
WARNING: no policy specified for zookeeper/rhel16@IDH.COM; defaulting to no policy
Principal "zookeeper/rhel16@IDH.COM" created.
Authenticating as principal oozie/admin@IDH.COM with password.
Entry for principal zookeeper/rhel16@IDH.COM with kvno 2, encryption type 
arcfour-hmac added to keytab WRFILE:zookeeper.keytab.
Entry for principal zookeeper/rhel16@IDH.COM with kvno 2, encryption type 
des3-cbc-sha1 added to keytab WRFILE:zookeeper.keytab.
Entry for principal zookeeper/rhel16@IDH.COM with kvno 2, encryption type 
des-cbc-crc added to keytab WRFILE:zookeeper.keytab.
 
   
Generating oozie keytab for rhel16
Authenticating as principal oozie/admin@IDH.COM with password.
WARNING: no policy specified for oozie/rhel16@IDH.COM; defaulting to no policy
Principal "oozie/rhel16@IDH.COM" created.
Authenticating as principal oozie/admin@IDH.COM with password.
Entry for principal oozie/rhel16@IDH.COM with kvno 2, encryption type arcfour-hmac 
added to keytab WRFILE:oozie.keytab.
Entry for principal oozie/rhel16@IDH.COM with kvno 2, encryption type 
des3-cbc-sha1 added to keytab WRFILE:oozie.keytab.
Entry for principal oozie/rhel16@IDH.COM with kvno 2, encryption type des-cbc-crc 
added to keytab WRFILE:oozie.keytab.
kinit: Keytab contains no suitable keys for hdfs/rhel16@IDH.COM while getting 
initial credentials
 
   
clusterkeytabs3803/rhel6/
clusterkeytabs3803/rhel6/hbase.keytab
clusterkeytabs3803/rhel6/zookeeper.keytab
clusterkeytabs3803/rhel6/mapred.keytab
clusterkeytabs3803/rhel6/hive.keytab
clusterkeytabs3803/rhel6/oozie.keytab
clusterkeytabs3803/rhel6/HTTP.keytab
clusterkeytabs3803/rhel6/hdfs.keytab
 
   

3. Ensure the script has generating keytabs for all nodes in /tmp/keytabs as follows. The tar file will be uploaded to Kerberos as Keytab files for authentication.

# ls /tmp/keytabs/ -l
total 136
drwxr-xr-x 18 root root   4096 Sep 30 17:55 clusterkeytabs3803
-rw-r--r--  1 root root 133120 Sep 30 17:55 clusterkeytabs3803.tar
 
   
 
   
 
   

Note Proceed to the Kerberos Secure Install for installing with Kerberos Authentication.


ID Installation

This section details installation of Intel Manager, Role Assignment to nodes in the cluster, Installation of ID through Intel Manager.

Role Assignment

The install wizard attempts to assign the master nodes for various services that have been selected to appropriate hosts in the cluster. Reconfigure the service assignment to match the Table 7.

Table 7 Role Assignment of Intel Distribution on CPA v2

Service Name
Host

NameNode

rhel1

SNameNode

rhel2

JobTracker

rhel2

Region Server

rhel[3-64]

ZooKeeper

rhel1, rhel2, rhel3

Hive Server

rhel2

HBase Master

rhel1, rhel2, rhel3

Oozie Server

rhel1

TasktrackerNodes

rhel[3-64]

DataNodes

rhel[3-64]



Note On a small cluster (<16 nodes), consolidate all master services to run on a single node.


Installing Intel Manager

Intel Manager can be installed in the following ways:

Quick install—In this installation mode, a user passes a few command line arguments to the installer script and then the script automates the installation process of Intel Manager and requires minimal input from the user executing the installation.

Interactive install—In the interactive mode, a wizard guides the user through the step-by-step process of installing Intel Manager. This mode requires input from the user at each step.

The document explains how to perform interactive install of Intel Manager. For more information on Quick Install, see ID Installation guide at: http://hadoop.intel.com/resources

Follow these steps to instal Intel Manager in the interactive mode:

1. Change directory to ui-installer under intelhadoop and run the executable ./install.

Figure 73 Running the Intel Manager Executable

2. Make sure the timestamp and time are correct as this is used to set the cluster time. Click Yes.

3. Click Accept to agree the on terms of Java licensing.

4. Verify the hostnames and FQDN (Fully Qualified Domain Name) settings are correct. Click Yes to continue.

5. In the Firewall Setting window, choose Disable Firewall and click OK.

6. To create a repo file for OS repo. Choose Create a repo file for an existing OS repository and click Yes.

7. Enter the location of the local Redhat 6.4 repo created. Make sure that the link opens in a regular browser and is working fine. From section "Creating Redhat local repo", the location is http://10.29.160.53/rhelrepo64. Click Yes.

Figure 74 Specifying Base URL for Creating Repo File

8. This will install intelcloudui.Click Continue.

9. Bind Intel Manager to eth0 and click Yes.

Figure 75 Choose eth0 for Admin (Intel Manager) Traffic

10. Click Yes and click Continue in the Install Apache Hadoop RPMs and Configure Management Software window.

11. Click Continue in the Server Startup Check window.

Figure 76 Intel Manager Server Startup

12. After the completion of Intel Manager installation, note down the URL. This URL is required to install Intel Distribution for Apache Hadoop through the Intel Manager. This URL will be the IP that is bound to the Intel Manager as shown in Figure 76. Click Ok and this completes the installation of Intel Manager.

Figure 77 Completion of Intel Manager Installation

Installing Intel Distribution through Intel Manager

The following steps detail the installation of Intel Distribution for Apache Hadoop using the Intel Manager that was just installed.

1. Open Intel Manager through the browser opening the Intel Manager url https://10.29.160.53:9443 and enter "admin" for both username and password.

Figure 78 Intel Manager login

2. Click Accept in the License Agreement page.

3. Enter relevant Cluster name and choose all the Cluster Components.

Figure 79 Cluster Name and Cluster Components

4. Choose Access Server via DNS for Network Environment and click Add Nodes.

Figure 80 Accessing Cluster Through DNS

5. In Add Nodes, choose Batch for the Add method and provide start and end ip from rhel1 to rhel64. The IP that is provided here is the eth1 IP. Eth0 is used for management traffic and Intel Manager is bound to eth0 IP. See Table 6.

Figure 81 Adding Nodes Through Batch Mode

6. Choose all IP Addresses and click Add.

Figure 82 Discovering Nodes

7. Click OK in the Confirmation window.

8. Ensure all nodes are Configured and then click OK.

Figure 83 Nodes Configured for Password-less SSH Access

9. This takes us back to Step 2 window showing all the configured nodes. Click Next.

10. In Step 3 for configuring NTP, provide location of external NTP server and click Next. If left blank the Intel Manager will setup NTP server on the Intel Manager Node.

Figure 84 Configuring External NTP

11. In the Edit Rack window, rename /Default rack to /Rack1.

Figure 85 Editing the Rack Name

12. In Step 4, select all the Available Nodes and move them to Selected Nodes and click Next.

Figure 86 Choosing Nodes for the Rack

13. In Step 5, for Security Policy, we can either choose Simple User Based Authentication, or Kerberos Authentication. The following steps continues with choosing the option Simple User Based Authentication. For Kerberos Authentication, see the next section Kerberos Install.

Choose Simple User Based Authentication and click Next.

Figure 87 Security Policy used in the Cluster

14. The next step, Intel Manager configures NTP and JDK and other steps on all nodes and gives out a message when it is complete. Click OK in the completion message window.

Figure 88 Setting up JDK, Yum and Nagios

15. In Step 6, verify the status against each node as Successful on all nodes and click Next.

16. In Step 7, configure Primary Namenode and Secondary NameNode as rhel1 and rhel2 respectively. The following steps would configure Master nodes as described in the Role Assignment section.

Figure 89 Configuring Primary NameNode and Secondary NameNode

17. Choose all nodes except NameNodes (rhel1 and rhel2) as DataNodes and click Next.

Figure 90 Configuring DataNodes

18. Choose rhel2 as JobTracker and click Next.

Figure 91 Configuring JobTracker

19. Select all the nodes except NameNodes (rhel1 and rhel2) as Tasktracker Nodes and click Next.

Figure 92 Selecting Tasktracker Nodes

20. In Step 11, leave dfs and mapred directories as default and click Next.


Note The script to format and mount the disks, "Configuring the Filesystem for DataNodes" takes care of creating the correct mount points which Intel Manager uses as default.


21. In Step 12, select rhel1, rhel2, rhel3 as ZooKeeper Servers and click Next.

Figure 93 Selecting Zookeeper Servers

22. In Step 13, select rhel1, rhel2, rhel3 as HBase Masters and click Next.

Figure 94 Select HBase Masters

23. In Step 14, select all the nodes except NameNodes (rhel1 and rhel2) as nodes which run RegionServer daemons and click Next.

Figure 95 Selecting Region Servers

24. In Step 15, select rhel2 as Hive Server and click Next.

Figure 96 Selecting Hive Servers

In Step 16, select rhel1 as Oozie Server and click Next.

Figure 97 Selecting Oozie Servers

25. In Step 17, click Edit Roles to verify the role assignments.

Figure 98 Verifying the Roles

26. Choose each Hadoop project from the drop-down and verify the role assignment. Choose HDFS and verify the role assignments.

Figure 99 HDFS Role Assignment

27. Choose MapReduce from drop-down and verify the role assignments.

Figure 100 Role Assignment for MapReduce

28. Choose ZooKeeper and verify role assignments.

Figure 101 Role Assignment for zookeeper

29. Choose HBase and verify the role assignments.

Figure 102 Role Assignment for HBase

30. Choose Hive and verify the role assignments.

Figure 103 Role Assignment for Hive

31. Choose Oozie and verify the role assignments. Click OK.

Figure 104 Role Assignment for Oozie Server

32. Click Finish in Step 17 and click OK in Confirmation window.

33. The next window shows Intel Manager formatting HDFS on all nodes and deploying Intel Distribution for Apache Hadoop on all nodes as completed.

Figure 105 Progress of Provisioning Service

34. Click OK and ensure all nodes have passed.

35. This opens the Intel Manager Dashboard with all the services status showing as Stopped.

Figure 106 Intel Manager Dashboard

Troubleshooting Installation and Deployment

If any nodes failed in the last step, click Configuration and then Nodes. Do the following in the same order:

1. Reformat the Cluster

2. Provisioning Service Properties

This will re-configure the nodes.

Figure 107 Intel Manager Node Configuration

Kerberos (Secure Mode) Install

There are different options to setup Kerberos (Secure Mode) Authenticated Intel Distribution:

Installing Intel Distribution through Intel Manager

Post Simple Secure Authentication Installation, setting up of Kerberos

In the first case, during Intel Distribution installation, choose "Kerberos Authentication" for Secure Policy. Continue with the installation steps mentioned above after choosing Kerberos Authentication.

This section details Post Simple Secure Authentication Kerberos set up:

1. Choose Configuration > Nodes.

2. Click Create Hadoop Instance in the top right corner of Intel Manager.

Figure 108 Intel Manager Create Hadoop Instance

3. Follow the steps as mentioned in "Installing Intel Distribution through Intel Manager" section and in step 5 choose Kerberos Authentication for Security Policy.

Figure 109 Specifying Kerberos Authentication for Security Policy

4. At the end of the wizard, a warning message window appears asking you to upload the keytab files. Click OK.

Figure 110 Message Window to Upload keytab Files

5. This automatically provisions all the nodes in the cluster for Kerberos authentication.

Figure 111 Provisioning all nodes for Kerberos authentication

6. After provisioning is complete, under Configuration in Intel Manager, you will see a new tab Kerberos.

Figure 112 New Kerberos Tab under Configuration

7. Click Kerberos and a message window pops up with a warning to Provision the service properties after the Keytab files are added. Click OK.

8. Under Configuration > Kerberos, click Change Realm.

Figure 113 Kerberos Page in Intel Manager

9. Update realm to IDH.COM.


Note The realm name is as per the current example and this is used for Kerberos setup. If you have specified a different realm name during Kerberos setup, you need to enter it here.


Figure 114 Modify Kerberos Realm

10. Click OK in the pop-up message window.

11. The above steps would update Principal from NODOMAIN or <DEFAULT DOMAIN>to IDH.COM. If not updated, click Refresh. Then click Batch Update Keytab.

Figure 115 Kerberos Page with Updated Principal

12. Upload the keytab tar file generated by Generating Keytab files.

Figure 116 Uploading keytab File

13. Click OK in Confirmation message window.

14. This would provision the clusters immediately. Ensure provisioning on all nodes finished successfully, else re-provision the cluster by going to Configuration >Nodes > Provision Cluster.

Figure 117 Provisioning Cluster Nodes

This completes Kerberos Integration with Intel Manager.

The keytab files are copied under /etc/ as follows:

[root@rhel1 ~]# ls /etc/*.keytab
/etc/hbase.keytab  /etc/hive.keytab  /etc/mapred.keytab  /etc/zookeeper.keytab
/etc/hdfs.keytab   /etc/http.keytab  /etc/oozie.keytab
 
   
To test if the keytab is valid and authenticate user run the following command 
 
   
kinit hdfs/rhel1@IDH.COM -kt /etc/hdfs.keytab
 
   

Note Make sure that /etc/krb5.conf is copied to all the nodes.


Following steps sets up the user as provided in the Kerberos configuration file to run jobs:

Running a MapReduce Job after Kerberos Setup


Note User jdoe has been already added to configuration files for Kerberos in "Install Kerberos Server" section.


1. Create user jdoe on all the nodes.

# clush -b -a useradd jdoe
 
   

2. On Admin node, run the following kadmin command to add jdoe@IDH.COM and jdoe/admin@IDH.COM.

[root@rhel1 ~]# kadmin.local
Authenticating as principal jdoe/admin@IDH.COM with password.
kadmin.local:  addprinc jdoe@IDH.COM
WARNING: no policy specified for jdoe@IDH.COM; defaulting to no policy
Enter password for principal "jdoe@IDH.COM":
Re-enter password for principal "jdoe@IDH.COM":
Principal "jdoe@IDH.COM" created.
 
   
kadmin.local:  addprinc jdoe/admin@IDH.COM
WARNING: no policy specified for jdoe/admin@IDH.COM; defaulting to no policy
Enter password for principal "jdoe/admin@IDH.COM":
Re-enter password for principal "jdoe/admin@IDH.COM":
Principal "jdoe/admin@IDH.COM" created.
 
   

Change user to jdoe from root

[root@rhel1 ~]# su jdoe
[jdoe@rhel1 root]$
 
   
Get the token/ticket for jdoe running kinit
 
   
[jdoe@rhel1 root]$ kinit jdoe
Password for jdoe@IDH.COM:
[jdoe@rhel1 root]$
 
   

Now run hadoop commands/jobs as shown with example

[jdoe@rhel1 root]$ hadoop dfs -ls /
Found 21 items
drwxrwxrwx   - root        supergroup          0 2013-09-12 18:03 /data300
drwxrwxrwx   - root        supergroup          0 2013-09-12 18:44 /data300aes
drwxrwxrwx   - root        supergroup          0 2013-09-18 15:45 /data3aes
drwxr-xr-x   - hbase       hbase               0 2013-09-30 17:38 /hbase
drwxr-xr-x   - hbase       hbase               0 2013-09-12 17:44 /hbck
drwxr-xr-x   - mapred      hadoop              0 2013-09-12 17:44 /mapred
drwxrwxrwx   - root        supergroup          0 2013-09-26 14:46 /result300
drwxrwxrwx   - root        supergroup          0 2013-09-26 14:23 /result300aes
drwxrwxrwx   - root        supergroup          0 2013-09-18 15:40 /result3aes
drwxrwxrwx   - hdfs        hadoop              0 2013-09-26 14:46 /tmp
drwxrwxrwx   - hdfs        hadoop              0 2013-10-01 17:56 /user
[jdoe@rhel1 root]$
 
   

Run sample hadoop job to verify as jdoe user

[jdoe@rhel1 root]$ hadoop jar /usr/lib/hadoop/hadoop-examples-1.0.3-Intel.jar pi 3 
3
Number of Maps  = 3
Samples per Map = 3
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Starting Job
13/10/02 16:04:57 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 10 for 
jdoe on 192.168.12.11:8020
13/10/02 16:04:57 INFO security.TokenCache: Got dt for 
hdfs://rhel1:8020/tmp/hadoop-mapred/mapred/staging/jdoe/.staging/job_201310011722_
0003;uri=192.168.12.11:8020;t.service=192.168.12.11:8020
13/10/02 16:04:57 INFO mapred.FileInputFormat: Total input paths to process : 3
13/10/02 16:04:57 INFO mapred.JobClient: Running job: job_201310011722_0003
13/10/02 16:04:58 INFO mapred.JobClient:  map 0% reduce 0%
13/10/02 16:05:13 INFO mapred.JobClient:  map 33% reduce 0%
13/10/02 16:05:17 INFO mapred.JobClient:  map 100% reduce 0%
...
Job Finished in 36.484 seconds
13/10/02 16:05:34 INFO util.NativeCodeLoader: Loaded the native-hadoop library
Estimated value of Pi is 3.55555555555555555556

Note Since root is not added to Kerberos, running hadoop as root throws an error as shown below.


[root@rhel1 ~]# hadoop dfs -ls /
13/10/02 16:06:32 ERROR security.UserGroupInformation: PriviledgedActionException 
as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate 
failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]

Memory Configuration

This section describes procedure to change settings in order to optimize the following services for performance and better utilization of resources for Performance and Capacity Balanced Configuration CPA v2:

HDFS

MapReduce

HBase

Stop all the services before making these changes.

HDFS

Table 8 HDFS Configurations

Property Name
Value
Property Name

NameNode Java Heap Size

4GB

hadoop.namenode.memory

Reserved space for HDFS

4GB

hadoop.datanode.memory

DataNode Volumes Failure Toleration

5

dfs.datanode.failed.volumes.tolerated


1. In Intel Manager, click Configuration > HDFS > Advanced Configuration.

2. Search for memory in the keyword text area.

Figure 118 HDFS Memory Configuration

3. For hadoop.namenode.memory, manually set the Default Value to 4096 and click Confirm.

Figure 119 HDFS NameNode Memory Configuration

4. Datanode heapsize "hadoop.datanode.memory" is automatically set to 4096, confirm that by clicking on the value.

5. Enter tolerated in the keyword text box to set the Datanode Volumes Failure Toleration.

6. Update the value of dfs.datanode.failed.volumes.tolerated to 5.

Figure 120 HDFS Volumes Tolerated Configuration

7. Click Save. This pops up a window to add the notes for these changes as audit notes. This can be viewed later on in the Administration tab.


Note These changes will not change throughout immediately. This requires another step, which is to provision service properties, which will be done once all the changes are made.


MapReduce

Update the MapReduce configuration as shown in Table 9:

Table 9 MapReduce Configurations

Property Name
Value
Property Name

Job Tracker Maximum Java Heap Size

4GB

hadoop.jobtracker.memory

Number of Map Slots per Node

24

mapred.tasktracker.map.tasks.maximum

Number of Reduce Slots per Node

12

mapred.tasktracker.reduce.tasks.maximum

Java Options for MapReduce Tasks

4GB

hadoop.tasktracker.memory

Map-side sort buffer memory

1GB

io.sort.mb

Java Child Opts

1536M

Mapred.child.java.opts


To configure MapReduce, follow these steps:

1. In the Intel Manager, click Configuration > MapReduce > Advanced Configuration.

2. Enter map.task in the Keyword text box and double click on the mapred.tasktracker.map.tasks.maximum property.

Figure 121 MapReduce Configuration

3. Update the Value manually to 24 on all the nodes individually.

Figure 122 Tasktracker Map Tasks Configuration

4. Repeat the same process for property mapred.tasktacker.reduce.tasks.maximum and set the value manually to 12.

Figure 123 Tasktracker Map Tasks Configuration

5. Enter child in the Keyword text box and double click on mapred.child.java.opts property and update -Xmx value to -Xmx1536M.

Figure 124 MapReduce Child Java Options

6. Enter sort in the Keyword text box and double click on io.sort.mb property and update value to 1024.

Figure 125 MapReduce Sort Buffer Memory

HBase

Update the HBase configurations as shown in Table 10:

Table 10 HBase Configurations

Property Name
Value
Property Name

HBase Master Maximum Java Heap Size

4GB

hbase.master.memory

HBase RegionServers Maximum Java Heap Size

32GB

hbase.regionserver.memory


1. In the Intel Manager, click Configuration > HBase > Advanced Configuration.

2. Enter memory in the Keyword text box and double click on hbase.regisonserver.memory property.

Figure 126 Hbase Memory Configuration

3. Update the value manually to 32768 and click Refresh all nodes and click Confirm.

Figure 127 Hbase Region Server Memory Configuration

4. Similarly ensure hbase.master.memory is 4GB which is automatically set by Intel Manager.

5. Click Save on the top left hand corner of the Advanced Configuration window. This pops up a window to add notes for these changes as audit notes. This can be viewed later on in the Administration tab.


Note These changes will not be reflected throughout immediately. This requires another step, which is to provision service properties.


Provisioning Service Properties

In order to make sure all the properties set above are deployed, navigate to Configuration > Nodes and click Provisioning Service Properties.

Figure 128 Provisioning Service Properties

This starts the tasks on all the nodes and configures them.

Figure 129 Provisioning Service Properties in Progress

Viewing Audits

Click Administration > Audits to view all the changes that have been made to the cluster and the notes we have entered.

Figure 130 Administration Audits

Starting all the Services

Now that all the configuration changes are made, follow these steps to start all the services:

1. Click Monitoring > Dashboard.

2. All services will have the status as stopped right after the deployment. In order to start the services, click Start All.

Figure 131 Dashboard before Starting Services

3. Start All, starts all services on all nodes in an order as:

a. ZooKeeper

b. Namenode and the Datanodes

c. Job tracker and task tracker

d. Other services

Figure 132 Starting Services

4. Once all the services start, a window pops up with a message that all the component services have started, click OK.

5. The Dashboard in Figure 133 shows all services as running. Individual services can be stopped or restarted by clicking on the Action button under operation column.

Figure 133 Dashboard Showing all Services Up and Running

Post ID Installation

This section describes how to integrate access control to Intel Manager in-order to enable role based access control and permissions to access HDFS, run MapReduce jobs, Hbase, Hive and Oozie. The section also describes how to configure Pig and Hive for encryption

Integration with Existing Identity Store for Access Control

ID supports OpenLDAP and Microsoft Active Directory (AD) for integration with Identity Store. This section describes Integration with OpenLDAP which has users jdoe, clee, jsmith.

Follow these steps to Integrate ID with OpenLDAP Identity Store:

1. In the Intel Manager GUI, click Configuration > Security > Authorization.

Figure 134 OpenLDAP Authorization Step

2. Click Configure Identity Store and provide the details of the LDAP as follows:

a. Check the Enable LDAP check box.

b. Provide the Server and Port details.

Figure 135 Adding LDAP to Integrate with Intel Manager

3. Click Verify before Saving to verify server connection. Click Save.

4. Choose the services in order to add access control role. In this case, all are selected. That is, HDFS, MapReduce, Hbase, Hive and Oozie.

Figure 136 Setting Up Access Control

5. Click Click to Add under Component in the right pane in the Authorization window and provide the role details. Each tab has specific access control settings for each service. This step creates an access control role. When done, click OK.

Figure 137 Adding Roles for Individual Services

6. In this step, we add users to the above created access control profile. Click Click to add under User/Group to newly create Access control role.

Figure 138 Access Control with New Role

7. This pulls all users from the Identity Store. Check the check box to choose the users/groups that need to be assigned to the above created access control role and Click OK.

Figure 139 Adding Users and Groups for the Role Defined

8. Once done, the Users and group for the Role are displayed. Click Save on the top left corner in the right pane of the Authorization window.

Figure 140 Access Control Role and User Group Defined

9. In order to enforce the Role based access control, do the following:

Monitoring >Dashboard > Stop all

Configuration > Nodes > Provisioning Service Properties

Monitoring > Dashboard > Start all

10. Once done, verify the settings by running ls command.

Pig and Hive with Encryption

ID allows encryption of Hive and Pig queries:

Hive

To enable encryption in Hive, set hive.encrypt properties as follows:

The following example demonstrates adding a table and encrypting it. Query to the table throws an error if no key credentials are provided. This example demonstrates that by passing a key, we will be able to access and decrypt the data.

Creating data.keystore

Use java keytool to generate a keystore. HIVE encryption feature requires a java keystore.

#keytool -genkey -alias KEY00 -keyalg RSA -keystore data.keystore -storepass 12345678 
-keypass 12345678 -dname "CN=John Doe OU=Development, O=Intel, L=Chicago, S=IL, C=US" 
-storetype JCEKS -keysize 1024
 
   

Storing Data in a Table

Data from customers.tbl will be loaded into a HIVE table which is going to encrypt and store the data in said HIVE table. Contents of the customers.tbl file are listed as:

cat ../files/customers.tbl
1|Customer#000000001|IVhzIApeRb ot,c,E|15|25-989-741-2988|711.56|BUILDING|to the even, 
regular platelets. regular, ironic epitaphs nag e|
2|Customer#000000002|XSTf4,NCwDVaWNe6tEgvwfmRchLXak|13|23-768-687-3665|121.65|AUTOMOBI
LE|l accounts. blithely ironic theodolites integrate boldly: caref|
3|Customer#000000003|MG9kdTD2WBHm|1|11-719-748-3364|7498.12|AUTOMOBILE| deposits eat 
slyly ironic, even instructions. express foxes detect slyly. blithely even accounts 
above|
 
   

Create directories in HDFS

hadoop fs -mkdir /user/securitylab/data/customers
hadoop fs -chmod 755 /user/securitylab/data/customers
hadoop fs -chown securitylab:hadoop /user/securitylab/data/customers
 
   
hadoop fs -copyFromLocal ./files/customers.tbl /user/securitylab/data/customers
 
   

Hive Script

Here we set hive.encrypt properties providing Key and Keystore file and also set hive.encrypt.enable in Hive query in order to enable encryption. This creates the table which stores the data encrypted.

# cat exercise-1.hive
set hive.encrypt.master.keyName=KEY00;
set 
hive.encrypt.master.keyProviderParameters=keyStoreUrl=file:////root/intel/lab/files/da
ta.keystore&keyStoreType=JCEKS&password=12345678;
 
   
-- prepare
drop table customer;
create external table customer (C_CUSTKEY INT, C_NAME STRING, C_ADDRESS STRING, 
C_NATIONKEY INT, C_PHONE STRING, C_ACCTBAL DOUBLE, C_MKTSEGMENT STRING, C_COMMENT 
STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE LOCATION 
'/user/securitylab/data/customers';
 
   
-- create the encrypted table
drop table customer_encrypted;
create table customer_encrypted (C_CUSTKEY INT, C_NAME STRING, C_ADDRESS STRING, 
C_NATIONKEY INT, C_PHONE STRING, C_ACCTBAL DOUBLE, C_MKTSEGMENT STRING, C_COMMENT 
STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE 
TBLPROPERTIES("hive.encrypt.enable"="true");
 
   
-- insert data from plain text table to the encrypted table
insert overwrite table customer_encrypted select * from customer;
 
   
-- show data from the encrypted customer table
select * from customer_encrypted;
 
   

Run the Hive Script as follows:

hive -f exercise-1.hive
 
   

Query the data

Directly querying data throws an error unless the right credentials are provided.

Query the data with credentials

By providing the correct credentials, we can now access the data:

Pig

To enable encryption in Pig, we pass the key and keystore file and set pig.encrypt properties as shown below through an example

Following example demonstrates adding the same table created above and encrypting it. Query to the table gives result encrypted if no key credentials are provided. The example demonstrates that by passing key, we are able to access and decrypt the data.

Creating public.keystore

Use java keytool to generate a keystore. Pig encryption feature requires a java keystore.

#keytool -genkey -alias CLUSTERPUBLICKEY -keyalg RSA -keystore public.keystore 
-storepass public123  -dname "CN=John Doe OU=Development, O=Intel, L=Chicago, S=IL, 
C=US" -storetype JKS -keysize 1024
 
   

A key mapping.xml file is required with the following contents:

<mapping>
	<default>
		<matches>
			<match>
				<expr>^default\.customer_encrypted_resolvedkey$</expr>
				<key>
					<keyName>KEY01</keyName>
				</key>
			</match>
		</matches>
	</default>
</mapping>
 
   

Pig Job Properties

This provides the Key and Keystore files which will be used by the pig script

cat job.parameters
# Key provider parameters
KEY_PROVIDER_PARAMETERS="keyStoreUrl=file:////root/intel/lab/files/data.keystore&keySt
oreType=JCEKS&password=12345678"
KEY_MAPPING_FILE="file:////root/intel/lab/files/keymapping.xml"
 
   
AGENT_SECRETS_PROTECTOR="com.intel.hadoop.mapreduce.cryptocontext.provider.AgentSecret
sProtector"
AGENT_PUBLIC_KEY_PROVIDER="org.apache.hadoop.io.crypto.KeyStoreKeyProvider"
AGENT_PUBLIC_KEY_PROVIDER_PARAMETERS="keyStoreUrl=file:////root/intel/lab/files/public
.keystore&keyStoreType=JKS&password=public123"
AGENT_PUBLIC_KEY_NAME="CLUSTERPUBLICKEY"
 
   

Pig Script

This creates 'pig-customer-encrypted' where the customer data is stored encrypted:

# cat exercise-1.pig
/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
 
   
-- Key store information
set pig.encrypt.keyProviderParameters '$KEY_PROVIDER_PARAMETERS';
 
   
-- Use the PigStorage function to load the customer file
customer = load '/user/securitylab/data/customers' USING PigStorage('|') as 
(c_custkey:long,c_name:chararray, c_address:chararray, c_nationkey:int, 
c_phone:chararray, c_acctbal:double, c_mktsegment:chararray, c_comment:chararray);
 
   
-- Use the PigStorge to store the customer file encrypted
STORE customer INTO 'pig-customer-encrypted' USING PigStorage('|', '-keyName KEY01');
 
   

Running the Pig Job

Run the command as follows

pig -param_file job.parameters exercise-1.pig
 
   

Query the Result

In order to query the result, run pig and run the following commands:

This spawn's a map reduce job to provide the result in encrypted format.

Query the Result with Key

Now run the same command passing the key as follows in order to decrypt the result:

This spawn's a map reduce job to provide the result in decrypted format.

Conclusion

Hadoop has become a popular data management across all verticals. The Cisco CPA v2 for Big Data with Intel Distribution for Apache Hadoop offers a dependable deployment model for enterprise Hadoop that offer a fast and predictable path for businesses to unlock value in big data.

The configuration detailed in the document can be extended to clusters of various sizes depending on what application demands. Up to 160 servers (10 racks) can be supported with no additional switching in a single UCS domain. Each additional rack requires two Cisco Nexus 2232PP 10GigE Fabric Extenders and 16 Cisco UCS C240 M3 Rack-Mount Servers. Scaling beyond 10 racks (160 servers) can be implemented by interconnecting multiple UCS domains using Nexus 6000/7000 Series switches, scalable to thousands of servers and to hundreds of petabytes storage, and managed from a single pane using Cisco UCS Central.

Bill of Material

This section provides the hardware and software components used in the design setup for deploying the 64-node Performance and Capacity Balanced Cluster.

Table 11 describes the BOM for the master rack; Table 12 describes the BOM for expansion racks (rack 2 to 4); and Table 13 and Table 14 describe the BOM for the software components

Table 11 Bill of Material for Base Rack

Part Number
Description
Quantity

UCS-SL-CPA2-PC

Performance and Capacity Balanced Cluster

1

UCSC-C240-M3S

UCS C240 M3 SFF w/o CPU mem HD PCIe w/ rail kit expdr

16

UCS-RAID9271CV-8I

MegaRAID 9271CV with 8 internal SAS/SATA ports with Supercap

16

UCSC-PCIE-CSC-02

Cisco VIC 1225 Dual Port 10Gb SFP+ CNA

16

CAB-9K12A-NA

Power Cord 125VAC 13A NEMA 5-15 Plug North Americ

32

UCSC-PSU2-1200

1200W 2u Power Supply For UCS

32

UCSC-RAIL-2U

2U Rail Kit for UCS C-Series servers

16

UCSC-HS-C240M3

Heat Sink for UCS C240 M3 Rack Server

32

UCSC-PCIF-01F

Full height PCIe filler for C-Series

48

UCS-CPU-E52660B

2.20 GHz E5-2660 v2/95W 10C/25MB Cache/DDR3 1866MHz

128

UCS-MR-1X162RZ-A

16GB DDR3-1866-MHz RDIMM/PC3-14900/dual rank/x4/1.5v

256

UCS-HD1T7KS2-E

1TB SAS 7.2K RPM 2.5 inch HDD/hot plug/drive sled mounted

384

UCS-SL-BD-FI96

Cisco UCS 6296 FI w/ 18p LIC, Cables Bundle

2

N2K-UCS2232PF

Cisco Nexus 2232PP with 16 FET (2 AC PS, 1 FAN (Std Airflow)

2

SFP-H10GB-CU3M=

10GBASE-CU SFP+ Cable 3 Meter

28

RACK-UCS2

Cisco R42610 standard rack w/side panels

1

RP208-30-1P-U-2=

Cisco RP208-30-U-2 Single Phase PDU 20x C13 4x C19 (Country Specific)

2

CON-UCW3-RPDUX

UC PLUS 24X7X4 Cisco RP208-30-U-X Single Phase PDU 2x (Country Specific)

6


Table 12 Bill of Material for Expansion Racks

Part Number
Description
Quantity

UCSC-C240-M3S

UCS C240 M3 SFF w/o CPU mem HD PCIe w/ rail kit expdr

48

UCS-RAID9271CV-8I

MegaRAID 9271CV with 8 internal SAS/SATA ports with Supercap

48

UCSC-PCIE-CSC-02

Cisco VIC 1225 Dual Port 10Gb SFP+ CNA

48

CAB-9K12A-NA

Power Cord 125VAC 13A NEMA 5-15 Plug North America

96

UCSC-PSU2-1200

1200W 2u Power Supply For UCS

96

UCSC-RAIL-2U

2U Rail Kit for UCS C-Series servers

48

UCSC-HS-C240M3

Heat Sink for UCS C240 M3 Rack Server

96

UCSC-PCIF-01F

Full height PCIe filler for C-Series

144

UCS-CPU-E52660B

2.20 GHz E5-2660 v2/95W 10C/25MB Cache/DDR3 1866MHz

96

UCS-MR-1X162RZ-A

16GB DDR3-1866-MHz RDIMM/PC3-14900/dual rank/x4/1.5v

768

UCS-HD1T7KS2-E

1TB SAS 7.2K RPM 2.5 inch HDD/hot plug/drive sled mounted

1152

N2K-UCS2232PF

Cisco Nexus 2232PP with 16 FET (2 AC PS, 1 FAN (Std Airflow)

6

CON-SNTP-UCS2232

SMARTNET 24X7X4 Cisco Nexus 2232PP

6

SFP-H10GB-CU3M=

10GBASE-CU SFP+ Cable 3 Meter

84

RACK-UCS2

Cisco R42610 standard rack w/side panels

3

RP208-30-1P-U-2=

Cisco RP208-30-U-2 Single Phase PDU 20x C13 4x C19 (Country Specific)

6

CON-UCW3-RPDUX

UC PLUS 24X7X4 Cisco RP208-30-U-X Single Phase PDU 2x (Country Specific)

18


Table 13 RedHat Enterprise Linux License

Red Hat Enterprise Linux

RHEL-2S-1G-3A

Red Hat Enterprise Linux

64

CON-ISV1-RH2S1G3A

3 year Support for Red Hat Enterprise Linux

64


Table 14 Intel Distribution for Apache Hadoop Software

Intel

Intel Distribution Software

Intel Distribution for Apache Hadoop

64