Guest

Design Zone for Data Centers

Cisco UCS Common Platform Architecture (CPA) for Big Data with MapR

  • Viewing Options

  • PDF (7.1 MB)
  • Feedback
Cisco UCS Common Platform Architecture (CPA) for Big Data with MapR

Table Of Contents

About the Authors

Acknowledgment

About Cisco Validated Design (CVD) Program

Cisco UCS Common Platform Architecture (CPA) for Big Data with MapR

Audience

Introduction

Cisco UCS Common Platform Architecture for Big Data

MapR: A Complete Hadoop Platform

Solution Overview

Rack and PDU Configuration

Server Configuration and Cabling

Software Distributions and Versions

MapR

Red Hat Enterprise Linux

Software Versions

Fabric Configuration

Performing Initial Setup of Cisco UCS 6296 Fabric Interconnects

Configure Fabric Interconnect A

Configure Fabric Interconnect B

Logging Into Cisco UCS Manager

Upgrading UCSM Software to Version 2.1(1a)

Adding Block of IP Addresses for KVM Access

Editing The Chassis Discovery Policy

Enabling The Server Ports and Uplink Ports

Creating Pools for Service Profile Templates

Creating an Organization

Creating MAC Address Pools

Configuring VLANs

Creating a Server Pool

Creating Policies for Service Profile Templates

Creating a Host Firmware Package Policy

Creating QoS Policies

Creating the Best Effort Policy

Creating a Platinum Policy

Setting Jumbo Frames

Creating a Local Disk Configuration Policy

Creating a Server BIOS Policy

Creating a Boot Policy

Creating a Service Profile Template

Configuring the Network Settings for the Template

Configuring a Storage Policy for the Template

Configuring a vNIC/vHBA Placement for the Template

Configuring a Server Boot Order for the Template

Configuring Server Assignment for the Template

Configuring Operational Policies for the Template

Configuring a Disk Drive for Operating System

Installing Red Hat Linux 6.2 with KVM

Post OS Install Configuration

Setting Up Password-less Login

Installing and Configuring Parallel SSH

Installing Parallel-SSH

Installing Cluster Shell

Configuring /etc/hosts file

Creating Red Hat Local Repository

Upgrading LSI driver

Installing HTTPD

Configuring Data Disk Drives

MapR Overview

MapR Editions

Unique Features of MapR Hadoop

Installation Process

Planning the Cluster

MapR Services

Hostnames and Roles

Prerequisites for MapR Software

Package Dependencies for Red Hat

Creating User Accounts

Configuring Password-less ssh for MapR User (Optional)

Installing JDK

MySQL Installation

NTP Configuration

Enabling Syslog

Setting ulimit

Setting ulimit on RHEL

Disabling SELinux

Setting TCP Retries Parameter

Disabling the Linux Firewall

Cluster Verification and Prerequisite Checks

Running STREAM Benchmark

Running RPCtest

Running IOzone Benchmark

Prerequisite Checks

Installing MapR Software

Preparing Packages and Repositories

Using a Local Repository

MapR Software Installation

Installing MapR Packages

Verifying the MapR Installation

Configuring the Node Using configure.sh Sript

Configuring All Nodes with CLDB and Zookeeper Locations

Formatting Disks Using the disksetup Script

Identify and Format the Data Disks for MapR

Updating Environment Variables in /opt/mapr/conf/env.sh

Bringing Up The Cluster

Initialization Sequence

Installing Cluster License

Using web-based MapR Control System to Install the License

Installing a License From the Command Line

Restarting M5-enabled Services After License Installation

Verifying Cluster Status

Verify Cluster Status Using the Command Line Interface

Specifying the Physical Topology of the Cluster

Adding Volumes

Installing Hadoop Components

Post Installation

Additional Information

Services and Ports Quick Reference

Avoiding Port Conflicts

MapR Repositories and Package Archives

RPM Repositories for MapR Core Software

RPM Repositories for Hadoop Ecosystem Tools

Package Archive for All Releases of Hadoop Ecosystem Tools

GitHub Repositories for Source Code

Maven Repositories for Application Developers

Other Scripts and Tools

Updating Physical Topology

Conclusion

Bill of Material


Cisco UCS Common Platform Architecture (CPA) for Big Data with MapR
Building a 64-Node Hadoop Cluster
Last Updated: October 25, 2013

Building Architectures to Solve Business Problems

About the Authors

Raghunath Nambiar, Cisco Systems

Raghunath Nambiar is a Distinguished Engineer at Cisco's Data Center Business Group. His current responsibilities include emerging technologies and big data strategy.

Jabari Norton, MapR Technologies

Jabari Norton is responsible for Worldwide Alliances at MapR Technologies. He and his team are responsible for the strategy and execution of MapR's alliances strategy. Jabari brings over 20 years' experience in enterprise software sales with a proven track record of building successful companies in high growth technology areas. Prior to MapR, Jabari led North American Sales for ScanSafe Technologies which was acquired by Cisco Systems in December 2009. Previous to ScanSafe/ Cisco Jabari held senior management roles at Qualys, Postini, and Lucent.

Karthik Kulkarni, Cisco Systems

Karthik Kulkarni is a Technical Marketing Engineer at Cisco Data Center Business Group focusing on Big Data and Hadoop technologies.

David Tucker, MapR Technologies

David Tucker is a Solutions Architect with MapR Technologies, specializing in performance tuning and system integration. He has over 15 years of experience working with enterprise hardware and software companies on wide range of data management solutions.

Acknowledgment

The authors acknowledge contributions of John Benninghoff, Manankumar Trivedi, Ashwin Manjunatha, and Nicolas Panayiotou in developing the Cisco UCS Common Platform Architecture (CPA) for Big Data with MapR Cisco Validated Design.

About Cisco Validated Design (CVD) Program


The CVD program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information visit:

http://www.cisco.com/go/designzone

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2013 Cisco Systems, Inc. All rights reserved.

Cisco UCS Common Platform Architecture (CPA) for Big Data with MapR


Audience

This document is intended to assist solution architects, sales engineers, field consultants, professional services, IT managers, partner engineering and customers in deploying the MapR on the Cisco Common Platform Architecture (CPA) for Big Data. This document assumes that the reader has an architectural understanding of Cisco Unified Computing (Cisco UCS), Cisco CPA and MapR technologies.

Introduction

Big Data technology and Apache Hadoop in particular, are finding use in an enormous number of applications and are being evaluated and adopted by enterprises of all kinds. As this important technology helps transform large volumes of data into actionable information, many organizations are struggling to deploy effective and reliable Hadoop infrastructure that performs and scales and is appropriate for mission-critical applications in the enterprise. Deployed as part of a comprehensive data center architecture, the Cisco UCS with MapR delivers a powerful and flexible infrastructure that increases business and IT agility, reduces total cost of ownership (TCO), and delivers exceptional return on investment (ROI) at scale, while fundamentally transforming the way that organizations do business with Hadoop technology.

Cisco UCS Common Platform Architecture for Big Data

The Cisco UCS solution for MapR is based on Cisco CPA for Big Data. The Cisco CPA offers a highly scalable architecture designed to meet a variety of scale-out application demands with seamless data integration and management integration capabilities built using the following components:

Cisco UCS 6200 Series Fabric Interconnects—provide high-bandwidth, low-latency connectivity for servers, with integrated, unified management provided for all connected devices by Cisco UCS Manager. Deployed in redundant pairs, Cisco fabric interconnects offer the full active-active redundancy, performance, and exceptional scalability needed to support the large number of nodes that are typical in clusters serving Big Data applications. Cisco UCS Manager enables rapid and consistent server configuration using service profiles and automation of the ongoing system maintenance activities such as firmware updates across the entire cluster as a single operation. Cisco UCS Manager also offers advanced monitoring with options to raise alarms and send notifications about the health of the entire cluster.

Cisco UCS 2200 Series Fabric Extenders—extends the network into each rack, acting as remote line cards for fabric interconnects and providing highly scalable and extremely cost-effective connectivity for a large number of nodes.

Cisco UCS C-Series Rack-Mount Servers—Cisco UCS C240M3 Rack-Mount Servers are 2-socket servers based on Intel Xeon E-2600 series processors and supporting up to 768 GB of main memory. 24 Small Form Factor (SFF) disk drives are supported in performance optimized option and 12 Large Form Factor (LFF) disk drives are supported in capacity option, along with 4 Gigabit Ethernet LAN-on-motherboard (LOM) ports.

Cisco UCS Virtual Interface Cards (VICs)—the unique Cisco UCS Virtual Interface Cards incorporate next-generation converged network adapter (CNA) technology from Cisco, and offer dual 10Gbps ports designed for use with Cisco UCS C-Series Rack-Mount Servers. Optimized for virtualized networking, these cards deliver high performance and bandwidth utilization and support up to 256 virtual devices.

Cisco UCS Manager—resides within the Cisco UCS 6200 Series Fabric Interconnects. It makes the system self-aware and self-integrating, managing the system components as a single logical entity. Cisco UCS Manager can be accessed through an intuitive graphical user interface (GUI), a command-line interface (CLI), or an XML application-programming interface (API). Cisco UCS Manager uses service profiles to define the personality, configuration, and connectivity of all resources within Cisco UCS, radically simplifying provisioning of resources so that the process takes minutes instead of days. This simplification allows IT departments to shift their focus from constant maintenance to strategic business initiatives.

MapR: A Complete Hadoop Platform

As the technology leader in Hadoop, the MapR distribution provides enterprise-class Hadoop solutions that are fast to develop and easy to administer. With significant investment in critical technologies, MapR offers the industry's most comprehensive Hadoop platform, which is fully optimized for performance scalability. MapR's distribution delivers more than a dozen tested and validated Hadoop software modules over a fortified data platform, offering exceptional ease of use, reliability, and performance for Hadoop solutions. MapR offers the following capabilities:

Ease of use—MapR is POSIX-compliant file system allows users to access the Hadoop cluster through industry-standard APIs such as Network File Service (NFS), Open Database Connectivity (ODBC), Linux Pluggable Authentication Modules (PAM), and Representational State Transfer (REST). MapR also provides multi-tenancy, data-placement control, and hardware-level monitoring of the cluster.

Reliability—MapR distribution provides lights-out data center capabilities for Hadoop. Features include self-healing of the critical services that maintain the cluster nodes and jobs, snapshots that allow point-in-time recovery of data, mirroring that allows wide-area intercluster replication, and rolling upgrades that prevent service disruption.

Performance—MapR distribution is twice as fast as any other Hadoop distribution. To provide superior and exceptional performance over other Hadoop distributions, MapR uses an optimized shuffle algorithm, direct access to the disk, built-in compression, and code written in advanced C++ rather than in Java. As a result, the MapR distribution provides the best hardware utilization when compared to any other distribution.

Solution Overview

The current version of the Cisco UCS CPA for Big Data offers two options depending on the compute and storage requirements:

High Performance Cluster Configuration—offers a balance of compute power with IO bandwidth optimized for price and performance. It is built using Cisco UCS C240M3 Rack-Mount Servers powered by two Intel Xeon E5-2665 processors (16 cores) with 256 GB of memory and 24 1TB SFF disk drives.

High Capacity Cluster Configuration—optimized for low cost per terabyte, is built using Cisco UCS C240M3 Rack-Mount Servers powered by two Intel Xeon E5-2640 processors (12 cores) with 128GB memory and 12 3TB LFF disk drives.


Note This CVD describes the installation process for a 64-node High Performance Cluster configuration.


The High Performance Cluster configuration consists of the following:

Two Cisco UCS 6296UP Fabric Interconnects

Eight Cisco Nexus 2232PP Fabric Extenders (two per rack)

64 Cisco UCS C240M3 Rack-Mount Servers (16 per rack)

Four Cisco R42610 standard racks

Eight vertical power distribution units (PDU) (country specific)

Rack and PDU Configuration

Each rack consists of two vertical PDU. The master rack consists of two Cisco UCS 6296UP Fabric Interconnects, two Cisco Nexus 2232PP Fabric Extenders and sixteen Cisco UCS C240M3 Servers, connected to each of the vertical PDUs for redundancy. The expansion racks also consists of two Cisco Nexus 2232PP Fabric Extenders and sixteen Cisco UCS C240M3 Servers are connected to each of the vertical PDUs for redundancy. This rack and PDU configuration ensures availability during power source failure.

Table 1 and Table 2 describe the rack configurations of rack 1 (master rack) and racks 2-4 (expansion racks).

Table 1 Rack Configuration For The Master Rack (Rack-1)

Cisco 42U Rack
Master Rack

42

Cisco UCS FI 6296UP

41

40

Cisco UCS FI 6296UP

39

38

Cisco Nexus FEX 2232PP

37

Cisco Nexus FEX 2232PP

36

Unused

35

Unused

34

Unused

33

Unused

32

Cisco UCS C240M3

31

30

Cisco UCS C240M3

29

28

Cisco UCS C240M3

27

26

Cisco UCS C240M3

25

24

Cisco UCS C240M3

23

22

Cisco UCS C240M3

21

20

Cisco UCS C240M3

19

18

Cisco UCS C240M3

17

16

Cisco UCS C240M3

15

14

Cisco UCS C240M3

13

12

Cisco UCS C240M3

11

10

Cisco UCS C240M3

9

8

Cisco UCS C240M3

7

6

Cisco UCS C240M3

5

4

Cisco UCS C240M3

3

2

Cisco UCS C240M3

1


Table 2 Rack Configuration for the Expansion Rack (Racks 2-4)

Cisco 42U Rack
Master Rack

42

Unused

41

Unused

40

Unused

39

Unused

38

Cisco Nexus FEX 2232PP

37

Cisco Nexus FEX 2232PP

36

Unused

35

Unused

34

Unused

33

Unused

32

Cisco UCS C240M3

31

30

Cisco UCS C240M3

29

28

Cisco UCS C240M3

27

26

Cisco UCS C240M3

25

24

Cisco UCS C240M3

23

22

Cisco UCS C240M3

21

20

Cisco UCS C240M3

19

18

Cisco UCS C240M3

17

16

Cisco UCS C240M3

15

14

Cisco UCS C240M3

13

12

Cisco UCS C240M3

11

10

Cisco UCS C240M3

9

8

Cisco UCS C240M3

7

6

Cisco UCS C240M3

5

4

Cisco UCS C240M3

3

2

Cisco UCS C240M3

1



Note Please contact the Cisco representative for country specific information.


Server Configuration and Cabling

The Cisco UCS C240M3 Rack Server is equipped with Intel Xeon E5-2665 processors, 256 GB of memory, Cisco UCS Virtual Interface Card (VIC)1225, LSI MegaRAID SAS 9266-8i storage controller and 24 x 1TB 7.2K Serial Advance Technology Attachment (SATA) disk drives. Figure 3 illustrates the physical connectivity of Cisco UCS C240M3 servers to Cisco Nexus 2232PP fabric extenders and Cisco UCS 6296UP fabric interconnects.

Figure 1 illustrates the ports on the Cisco Nexus 2232PP fabric extender connecting to the Cisco UCS C240M3 servers. Sixteen Cisco UCS C240M3 servers are used in the master rack configurations.

Figure 1 Fabric Topology

Figure 2 illustrates the port connectivity between the Cisco Nexus 2232PP FEX and the Cisco UCS C240M3 server.

Figure 2 Connectivity Diagram of Cisco Nexus 2232PP FEX and Cisco UCS C240M3 Servers

For more information on physical connectivity and single-wire management, see:

http://www.cisco.com/en/US/docs/unified_computing/ucs/c-series_integration/ucsm2.1/b_UCSM2-1_C-Integration_chapter_010.html

For more information on physical connectivity illustrations and cluster setup, see:

http://www.cisco.com/en/US/docs/unified_computing/ucs/c-series_integration/ucsm2.1/b_UCSM2-1_C-Integration_chapter_010.html#reference_FE5B914256CB4C47B30287D2F9CE3597

Figure 3 depicts a 64-node cluster, and each link represents 8 x 10 Gigabit link.

Figure 3 64 -Node Cluster Configuration

Software Distributions and Versions

MapR

MapR Hadoop is API-compatible and includes and works with the family of Hadoop ecosystem components such as HBase, Hive, Pig, Flume, and others. For more information on MapR software, see: www.maprtech.com

Red Hat Enterprise Linux

The operating system supported is Red Hat Enterprise Linux (RHEL) 6.2. For more information on RHEL, see: http://www.redhat.com.

Software Versions

Table 3 describes the software versions tested and validated in this document.

Table 3 Software Versions Summary

Layer
Component
Version or Release

Compute

Cisco UCS C240M3

1.4.7cc

Network

Cisco UCS 6296UP

UCS 2.1(1a)

Cisco UCS VIC1225 Firmware

2.1(1a)

Cisco UCS VIC1225 Driver

2.1.1.41

Cisco Nexus 2232PP

5.1(3)N2(2.11a)

Storage

LSI 9266-8i Firmware

23.7.0-0039

LSI 9266-8i Driver

06.504.01.00

Software

Red Hat Enterprise Linux Server

6.2 (x86_64)

Cisco UCS Manager

2.1(a)

MapR

M5



Note To download the latest drivers, see: http://software.cisco.com/download/release.html?mdfid=284296254&flowid=31743&softwareid=283853158&release=1.5.1&relind=AVAILABLE&rellifecycle=&reltype=latest


Fabric Configuration

This section provides details for configuring a fully redundant, highly available Cisco UCS 6296 Fabric Interconnect.

1. Initial setup of the Fabric Interconnect A and B.

2. Connect to IP address of Fabric Interconnect A using web browser.

3. Launch the Cisco UCS Manager.

4. Edit the chassis discovery policy.

5. Enable server and uplink ports.

6. Create pools and polices for service profile template.

7. Create Cisco Service Profile template and 64 service profiles.

8. Start discover process.

9. Associate to server.

Performing Initial Setup of Cisco UCS 6296 Fabric Interconnects

This section describes the steps to perform the initial setup of the Cisco UCS 6296 Fabric Interconnects A and B.

Configure Fabric Interconnect A

Follow these steps to configure the Fabric Interconnect A:

1. Connect to the console port on the first Cisco UCS 6296 Fabric Interconnect.

2. At the prompt to enter the configuration method, enter console to continue.

3. If asked to either perform a new setup or restore from backup, enter setup to continue.

4. Enter y to continue to set up a new Fabric Interconnect.

5. Enter y to enforce strong passwords.

6. Enter the password for the admin user.

7. Enter the same password again to confirm the password for the admin user.

8. When asked if this fabric interconnect is part of a cluster, enter y to continue.

9. Enter A for the switch fabric.

10. Enter the cluster name for the system name.

11. Enter the Mgmt0 IPv4 address.

12. Enter the Mgmt0 IPv4 netmask.

13. Enter the IPv4 address of the default gateway.

14. Enter the cluster IPv4 address.

15. To configure DNS, enter y.

16. Enter the DNS IPv4 address.

17. Enter y to set up the default domain name.

18. Enter the default domain name.

19. Review the settings that were printed to the console, and enter yes to save the configuration.

20. Wait for the login prompt to make sure the configuration has been saved.

Configure Fabric Interconnect B

Follow these steps to configure the Fabric Interconnect B:

1. Connect to the console port on the second Cisco UCS 6296 Fabric Interconnect.

2. When prompted to enter the configuration method, enter console to continue.

3. The installer detects the presence of the partner fabric interconnect and adds this fabric interconnect to the cluster. Enter y to continue the installation.

4. Enter the admin password that was configured for the first Fabric Interconnect.

5. Enter the Mgmt0 IPv4 address.

6. Enter yes to save the configuration.

7. Wait for the login prompt to confirm that the configuration has been saved.

For more information on configuring Cisco UCS 6200 Series Fabric Interconnect, see:

http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/2.0/b_UCSM_GUI_Configuration_Guide_2_0_chapter_0100.html

Logging Into Cisco UCS Manager

Follow these steps to login to Cisco UCS Manager.

1. Open a Web browser and navigate to the Cisco UCS 6296 Fabric Interconnect cluster address.

2. Select the launch link to download the Cisco UCS Manager software.

3. If prompted, accept the security certificates.

4. When prompted, enter admin for the username and enter the administrative password.

5. Click Login.

Upgrading UCSM Software to Version 2.1(1a)

This design solution uses the Cisco UCS 2.1(1a). Make sure that the Cisco UCS C-Series version 2.1(1a) software bundle is installed on the Cisco UCS Fabric Interconnects.

To upgrade the Cisco UCS Manager software and Cisco UCS 6296 Fabric Interconnect software to version 2.1(1a), see: Upgrading Cisco UCS from Release 2.0 to Releases 2.1

Adding Block of IP Addresses for KVM Access

Follow these steps to create a block of KVM IP addresses for the server access in Cisco UCS environment.

1. Click the LAN tab.

2. Select Pools > IPPools > IP Pool ext-mgmt.

3. Right-click Management IP Pool.

4. Select Create Block of IP Addresses. See Figure 4

Figure 4 Adding Block of IP Addresses for KVM Access Part 1

5. Enter the starting IP address of the block and number of IPs needed, the subnet and the gateway information. See Figure 5.

Figure 5 Adding Block of IP Addresses for KVM Access Part 2

6. Click OK to create the IP block. See Figure 6.

Figure 6 Adding Block of IP Addresses for KVM Access Part 3

7. Click OK.

Editing The Chassis Discovery Policy

This section provides details for modifying the chassis discovery policy. Setting the discovery policy ensures easy addition of the Cisco UCS B-Series chassis or fabric extenders for the Cisco UCS C-Series servers in future.

1. Click the Equipment tab.

2. In the right pane, click the Policies tab.

3. Click the Global Policies tab.

4. In the Chassis/FEX Discovery Policy area, change the chassis discovery policy to 8-link. See Figure 7.

Figure 7 Changing The Chassis Discovery Policy

5. Click Save Changes.

6. Click OK.

Enabling The Server Ports and Uplink Ports

Follow these steps to enable the server and configure the uplink ports:

1. Click the Equipment tab.

2. Select Equipment > Fabric Interconnects > Fabric Interconnect A (primary) > Fixed Module.

3. Expand the Unconfigured Ethernet Ports section. See Figure 8.

4. Select all the ports that are connected to the Cisco 2232PP FEX (eight per FEX), right-click and select Reconfigure > Configure as a Server Port.

5. Select port 1 that is connected to the uplink switch, right-click, then select Reconfigure > Configure as Uplink Port.

6. Select Show Interface and select 10GB for Uplink Connection.

7. Click Yes and then OK to continue.

8. Select Equipment > Fabric Interconnects > Fabric Interconnect B (subordinate) > Fixed Module.

9. Expand the Unconfigured Ethernet Ports section.

10. Select all the ports that are connected to the Cisco 2232 Fabric Extenders (eight per Fex), right-click and select Reconfigure > Configure as Server Port.

11. Click Yes and then OK to continue.

12. Select port number 1, which is connected to the uplink switch, right-click and select Reconfigure > Configure as Uplink Port.

Figure 8 Enabling Server Ports

13. Select Show Interface and select 10GB for Uplink Connection.

14. Click Yes and then OK to continue.

Figure 9 illustrates all the uplink ports configured on the servers.

Figure 9 Servers and Uplink Ports Summary

Creating Pools for Service Profile Templates

Creating an Organization

Organizations are used as a means to arrange and restrict access to various groups within the IT organization, and enable multi-tenancy of the compute resources. (This feature is not used in this design solution.)

Follow these steps to configure an organization in the Cisco UCS Manager:

1. Click New.

2. Select Create Organization from the options.

3. Enter a name for the organization.

4. (Optional) Enter a description for the organization.

5. Click OK.

Creating MAC Address Pools

Follow these steps to create MAC address pools:

1. Click the LAN tab.

2. Select Pools > root.

3. Right-click MAC Pools under the root organization.

4. Select Create MAC Pool to create the MAC address pool. Enter ucs as the name of the MAC pool.

5. (Optional) Enter a description of the MAC pool.

6. Click Next.

7. Click Add.

8. Specify a starting MAC address.

9. Specify a size of the MAC address pool, which is sufficient to support the available server resources. See Figure 10.

Figure 10 Specifying the First MAC Address and Size

10. Click OK.

11. Click Finish. See Figure 11.

Figure 11 Adding MAC Addresses

12. Click OK to confirm the addition of the MAC addresses.

Configuring VLANs

Table 4 describes the VLANs that are configured in this design solution.

Table 4 VLAN Configurations

VLAN
Fabric
NIC Port
Function
Failover

vlan160_mgmt

A

eth0

Management, user connectivity

Fabric Failover B

vlan12_HDFC

B

eth1

Hadoop

Fabric Failover A

vlan11_DATA

A

eth2

Hadoop and/or SAN/NAS access, ETL

Fabric Failover B


All of the VLANs created should be trunked to the upstream distribution switch connecting the fabric interconnects. In this deployment, vlan160_mgmt is configured for management access and user connectivity, vlan12_HDFS is configured for Hadoop interconnect traffic, and vlan11_DATA is configured for optional secondary interconnect and/or SAN/NAS access, heavy ETL.

Follow these steps to configure VLANs in Cisco UCS Manager:

1. Click the LAN tab.

2. Select LAN > VLANs.

3. Right-click the VLANs under the root organization.

4. Select Create VLANs to create the VLAN. See Figure 12.

Figure 12 Creating VLAN

5. Enter vlan160_mgmt in the VLAN Name/Prefix text box. See Figure 13.

6. Click the Common/Global radio button.

7. Enter 160 in the VLAN IDs text box.

8. Click OK and then click Finish.

9. Click OK.

Figure 13 Creating Management VLAN

10. Click the LAN tab.

11. Select LAN > VLANs.

12. Right-click the VLANs under the root organization.

13. Select Create VLANs to create the VLAN. See Figure 14.

14. Enter vlan11_DATA in the VLAN Name/Prefix text box.

15. Click the Common/Global radio button.

16. Enter 11 in the VLAN IDs text box.

17. Click OK and then click Finish.

18. Click OK.

Figure 14 Creating VLAN for Data

19. Click the LAN tab.

20. Select LAN > VLANs.

21. Right-click the VLANs under the root organization.

22. Select Create VLANs to create the VLAN.

23. Enter vlan12_HDFS in the VLAN Name/Prefix text box. See Figure 15.

24. Click the Common/Global radio button.

25. Enter 12 in the VLAN IDs text box.

26. Click OK and then click Finish.

Figure 15 Creating VLAN for Hadoop Data

Creating a Server Pool

A server pool contains a set of servers. These servers typically share the same characteristics such as their location in the chassis, server type, amount of memory, local storage, type of CPU, or local drive configuration. You can manually assign a server to a server pool, or use the server pool policies and server pool policy qualifications to automate the server assignment.

Follow these steps to configure the server pool within the Cisco UCS Manager:

1. Click the Servers tab.

2. Select Pools > root.

3. Right-click the Server Pools.

4. Select Create Server Pool.

5. Enter the required name (ucs) for the server pool in the name text box. See Figure 16.

6. (Optional) Enter a description for the organization.

Figure 16 Setting Name and Description of the Server Pool

7. Click Next > to add the servers.

8. Select all the Cisco UCS C240M3 servers to be added to the server pool that were previously created (ucs), then Click >> to add them to the pool. See Figure 17.

9. Click OK, and then click Finish.

Figure 17 Adding Servers to the Server Pool

Creating Policies for Service Profile Templates

This section describes the steps to create the following policies for the service profile template:

Host Firmware Package policy

Quality-of-Service (QoS) policy

Local Disk Configuration

Server BIOS Policy

Boot Policy

Creating a Host Firmware Package Policy

You can use firmware packages in service profiles to upgrade the server and adapter firmware, including the BIOS on the server, by defining a host firmware policy and including it in the service profile associated with a server.

Firmware management policies allow the administrator to select the corresponding firmware packages for a given server configuration. The components that can be configured include adapters, BIOS, board controllers, FC adapters, HBA options, ROM and storage controller.

Follow these steps to create a host firmware management policy for a given server configuration using the Cisco UCS Manager:

1. Click the Servers tab.

2. Select Policies > root.

3. Right-click Host Firmware Packages.

4. Select Create Host Firmware Package.

5. Enter the required host firmware package name (ucs). See Figure 18.

6. Click the Simple radio button to configure the host firmware package.

7. Select the appropriate Rack Package value.

8. Click OK to complete creating the management firmware package.

9. Click OK.

Figure 18 Creating Host Firmware Package

Creating QoS Policies

This section describes the steps to create the Best Effort QoS Policy and Platinum QoS policy.

Creating the Best Effort Policy

Follow these steps to create the Best Effort Policy:

1. Click the LAN tab.

2. Select Policies > root.

3. Right-click QoS Policies.

4. Select Create QoS Policy. See Figure 19.

Figure 19 Creating QoS Policy

5. Enter BestEffort as the name of the policy. See Figure 20.

6. Select BestEffort from the drop down menu.

7. Retain the Burst (Bytes) field as default (10240).

8. Retain the Rate (Kbps) field as default (line-rate).

9. Retain the Host Control radio button as default (none).

10. Click OK to complete creating the Policy.

11. Click OK.

Figure 20 Creating BestEffort QoS Policy

Creating a Platinum Policy

Follow these steps to create the Platinum QoS policy:

1. Click the LAN tab.

2. Select Policies > root.

3. Right-click QoS Policies.

4. Select Create QoS Policy.

5. Enter Platinum as the name of the policy. See Figure 21.

6. Select Platinum from the drop down menu.

7. Retain the Burst (Bytes) field as default (10240).

8. Retain the Rate (Kbps) field as default (line-rate).

9. Retain the Host Control radio button as default (none).

10. Click OK to complete creating the Policy.

11. Click OK.

Figure 21 Creating Platinum QoS Policy

Setting Jumbo Frames

Follow these steps to set up Jumbo frames and enable the QoS:

1. Click the LAN tab in the Cisco UCS Manager.

2. Select LAN Cloud > QoS System Class.

3. In the right pane, click the General tab.

4. In the Platinum row, enter 9000 for MTU. See Figure 22.

5. Check the Enabled check box.

6. Click Save Changes.

7. Click OK.

Figure 22 Setting Jumbo Frames

Creating a Local Disk Configuration Policy

Follow these steps to create a local disk configuration in the Cisco UCS Manager:

1. Click the Servers tab.

2. Select Policies > root.

3. Right-click Local Disk Config Policies.

4. Select Create Local Disk Configuration Policy.

5. Enter ucs as the local disk configuration policy name. See Figure 23.

6. Select Any Configuration from the drop-down list to set the Mode.

7. Uncheck the Protect Configuration check box.

8. Click OK to complete creating the Local Disk Configuration Policy.

9. Click OK.

Figure 23 Configuring Local Disk Policy

Creating a Server BIOS Policy

The BIOS policy feature in Cisco UCS automates the BIOS configuration process. The traditional mode of setting the BIOS is manual and is often error-prone. By creating a BIOS policy and assigning the policy to a server or group of servers, you can enable transparency within the BIOS settings configuration.


Note BIOS settings can have a significant performance impact, depending on the workload and the applications. The BIOS settings listed in this section is for configurations optimized for best performance which can be adjusted based on the application, performance and energy efficiency requirements.


Follow these steps to create a server BIOS policy using the Cisco UCS Manager:

1. Select the Servers tab.

2. Select Policies > root.

3. Right-click BIOS Policies.

4. Select Create BIOS Policy.

5. Enter the preferred BIOS policy name.

6. Change the BIOS settings. See Figure 24.

Figure 24 Creating Server BIOS Policy

Figure 25 and Figure 26 show the Processor and Intel Directed IO properties settings in the BIOS Policy.

Figure 25 Creating Server BIOS Policy for Processor

Figure 26 Creating Server BIOS Policy for Intel Directed IO

7. Set the RAS Memory settings and click Next. See Figure 27.

Figure 27 Creating Server BIOS Policy for Memory

8. Click Finish to complete creating the BIOS Policy.

9. Click OK.

Creating a Boot Policy

Follow these steps to create a boot policy within Cisco UCS Manager:

1. Select the Servers tab.

2. Select Policies > root.

3. Right-click the Boot Policies.

4. Select Create Boot Policy. See Figure 28.

Figure 28 Creating Boot Policy Part 1

5. Enter ucs as the boot policy name. See Figure 29.

6. (Optional) Enter a description for the boot policy.

7. Retain the Reboot on Boot Order Change check box as default (unchecked).

8. Expand Local Devices and select Add CD-ROM.

9. Expand Local Devices and select Add Local Disk.

10. Expand vNICs and select Add LAN Boot and enter eth0.

11. Click OK to add the Boot Policy.

12. Click OK.

Figure 29 Creating Boot Policy Part 2

Creating a Service Profile Template

Follow these steps to create a service profile template in Cisco UCS Manager:

1. Click the Servers tab.

2. Select Policies > root.

3. Right-click root.

4. Select Create Service Profile Template. See Figure 30.

Figure 30 Creating Service Profile Template

5. The Create Service Profile Template window appears. Do the following (See Figure 31):

a. In the Identify Service Profile Template window, enter the name of the service profile template as ucs.

b. Click the Updating Template radio button.

c. In the UUID section, select Hardware Default as the UUID pool.

6. Click Next to continue to the next section.

Figure 31 Identify Service Profile Template

Configuring the Network Settings for the Template

In the Network window, follow these steps to configure the network settings in the Cisco UCS Manager:

1. Retain the Dynamic vNIC Connection Policy field at the default. See Figure 32.

2. Click the Expert radio button to define How would you like to configure LAN connectivity?

3. Click Add to add a vNIC to the template. The Modify vNIC window appears.

Figure 32 Configuring Network Settings for the Template

4. In the Modify vNIC window, enter name for the vNIC as eth0. See Figure 33.

5. Select ucs in the MAC Address Assignment pool.

6. Click the Fabric A radio button and check the Enable failover check box for the Fabric ID.

7. Check the vlan160_mgmt check box for VLANs.

8. Click the Native VLAN radio button.

9. Select MTU size as 1500.

10. Select adapter policy as Linux.

11. Retain the Dynamic vNIC connection policy as <no set>.

12. Select QoS Policy as BestEffort.

13. Retain the Network Control Policy as default.

14. Click OK.

Figure 33 Configuring vNIC eth0

15. The Modify vNIC window appears. Enter the name of the vNIC as eth1. See Figure 34.

16. Select ucs in the MAC Address Assignment pool.

17. Click the Fabric B radio button and check the Enable failover check box for the Fabric ID.

18. Check the vlan12_HDFS check box for VLANs and select the Native VLAN radio button.

19. Select MTU size as 9000.

20. Select adapter policy as Linux.

21. Retain the Dynamic vNIC Connection Policy as <no set>.

22. Select QoS Policy as Platinum.

23. Retain the Network Control Policy as default.

24. Click OK.

Figure 34 Configuring vNIC eth1

25. The Create vNIC window appears. Enter the name of the vNIC as eth2. See Figure 35.

26. Select ucs in the MAC Address Assignment pool.

27. Click the Fabric A radio button and check the Enable failover check box for the Fabric ID.

28. Check the vlan11_DATA check box for VLANs and select the Native VLAN radio button.

29. Select MTU size as 9000.

30. Select adapter policy as Linux.

31. Retain the Dynamic vNIC Connection Policy as <no set>.

32. Select QoS Policy as Platinum.

33. Retain the Network Control Policy as default.

34. Click OK.

35. Click Next in the Networking window to move to the next section.

Figure 35 Configuring vNIC eth2

Configuring a Storage Policy for the Template

In the Storage window, follow these steps to configure a storage policy in Cisco UCS Manager:

1. Select ucs for the local disk configuration policy. See Figure 36.

2. Click the No vHBAs radio button to define How would you like to configure SAN connectivity?

3. Click Next to continue to the next section.

Figure 36 Configuring Storage settings

4. Click Next in the Zoning window to move to the next section.

Configuring a vNIC/vHBA Placement for the Template

In the vNIC/vHBA window, follow these steps to configure a vNIC/vHBA placement policy in Cisco UCS Manager:

1. Select the Default Placement Policy option for the Select Placement field. See Figure 37.

2. Select eth0, eth1 and eth2 assign the vNICs in the following order:

a. eth0

b. eth1

c. eth2

Review to make sure that all vNICs are assigned in the appropriate order.

3. Click Next to continue to the next section.

Figure 37 vNIC/vHBA Placement

Configuring a Server Boot Order for the Template

In the Server Boot Order window, follow these steps to set the boot order for servers in Cisco UCS Manager:

1. Select ucs in the Boot Policy name field. See Figure 38.

2. Check the Enforce vNIC/vHBA/iSCSI Name check box.

Review to make sure that all the boot devices are created and identified.

3. Verify that the boot devices are in the correct boot sequence.

4. Click OK.

Figure 38 Creating Boot Policy

5. Click Next to continue to the next section.

6. In the Maintenance Policy window retain the default <no policy>.

7. Click Next to continue to the next section.

Configuring Server Assignment for the Template

In the Server Assignment window, follow these steps to assign the servers to the pool in Cisco UCS Manager:

1. Select ucs for the Pool Assignment field. See Figure 39.

2. Retain the Server Pool Qualification field as default.

3. Select ucs in Host Firmware Package.

Figure 39 Server Assignment

Configuring Operational Policies for the Template

In the Operational Policies window, follow these steps:

1. Select ucs in the BIOS Policy field. See Figure 40.

2. Click Finish to create the Service Profile template.

3. Click OK.

Figure 40 Selecting BIOS Policy

4. Click the Servers tab.

a. Select Service Profile Templates > root.

b. Right-click root and select Create Service Profile Template. See Figure 41.

Figure 41 Creating Service Profiles from Template

c. The Create Service Profile from Template window appears. Enter the name and number in the fields. See Figure 42.

Figure 42 Selecting Name and Total Number of Service Profiles

The Cisco UCS Manager discovers the servers and automatically associate these servers with service profiles. Figure 43 illustrates the service profiles associated with all the 64-nodes.

Figure 43 Cisco UCS Manager showing 64 Nodes

Configuring a Disk Drive for Operating System

This CVD describes the High Performance Configuration featuring 24 1TB SFF disk drives. The disk drives are configured as individual RAID0 volumes with 1MB stripe size. The Read ahead cache and write cache is enabled while the battery is present. The first disk drive is used by the operating system and remaining 23 disk drives are used for the Hadoop Distributed File System (HDFS).


Note Two partitions of 1TB and 2TB are created on the first disk drive, the 1TB partition is used for operating system and the 2TB partition is used for HDFS along with disk drives 2 to 12.


There are several ways to configure RAID such as using the LSI WebBIOS Configuration Utility embedded in the MegaRAID BIOS, booting DOS and running MegaCLI commands, using Linux-based MegaCLI commands, or using third party tools that have MegaCLI integrated. For this deployment, the first disk drive is configured using the LSI WebBIOS Configuration Utility and the remaining drives are configured using Linux-based MegaCLI commands after the operating system (OS) is installed.

Follow these steps to create RAID0 on the first disk drive to install the operating system:

1. Bootup the server, and do the following:

a. Press <Ctrl><H> to launch the WebBIOS.

b. Press Ctrl+H immediately. The Adapter Selection window appears.

2. Click Start to continue. See Figure 44.

3. Click Configuration Wizard.

Figure 44 Adapter Selection for RAID Configuration

4. In the Configuration Wizard window, click the Clear Configuration radio button. See Figure 45.

5. Click Next to clear the existing configuration.

Figure 45 Clearing Current configuration on the controller

6. Click Yes.

7. In the Physical View, ensure that all the drives are Unconfigured Good.

8. In the Configuration Wizard window, click the New Configuration radio button. See Figure 46.

9. Click Next.

Figure 46 Choosing to create a New Configuration

10. Click the Manual Configuration radio button. This enables complete control over all attributes of the new storage configuration, such as, configuration of the drive groups, virtual drives and setting the parameters. See Figure 47.

Figure 47 Choosing Manual Configuration Method

11. Click Next. The Drive Group Definition window appears.

12. In the Drive Group Definition window, choose the first drive to create drive groups. See Figure 48.

13. Click Add to Array to move the drives to a proposed drive group configuration in the Drive Groups pane.

14. Click Accept DG and click Next.

Figure 48 Selecting first drive and Adding to Drive Group

15. In the Span Definitions window, click Add to SPAN and click Next. See Figure 49.

Figure 49 Span Definition Window

16. In the Virtual Drive definitions window, do the following (See Figure 50):

a. Click on Update Size.

b. Change Strip Size to 1MB. A larger strip size ensures higher read performance.

c. From the Read Policy drop-down list, choose Always Read Ahead.

d. From the Write Policy drop-down list, choose Write Back with BBU.

e. Make sure RAID Level is set to RAID0.

f. Click Accept to accept the changes to the virtual drive definitions.

g. Click Next.


Note Clicking on Update Size can change some of the settings in the window. Make sure all settings are correct before submitting the changes.


Figure 50 Virtual Drive Definition Window

17. Click Yes to save the configuration.

18. In the Managing SSD Caching window, click Cancel. See Figure 51.

Figure 51 SSD Caching Window

19. Click Yes in the confirmation page.

20. Set VD0 as the Boot Drive and click Go. See Figure 52.

Figure 52 Setting Virtual Drive as Boot Drive

21. Click Home.

22. Review the configuration and click Exit.

Repeat the procedure to configure disks 2 to 24 or use the Linux based MegaCLI commands described in section Configuring Data Disk Drives.

Installing Red Hat Linux 6.2 with KVM

The following section provides detailed procedures for installing Red Hat Linux 6.2.

There are multiple methods to install Red Hat Linux operating system. The installation procedure described in this design guide uses KVM console and virtual media from Cisco UCS Manager.

1. Log in to the Cisco UCS 6296 Fabric Interconnect and launch the Cisco UCS Manager application.

2. Click the Equipment tab.

3. In the navigation pane expand Rack-Mounts and Servers.

4. Right-click on the Server and select KVM Console. See Figure 53.

Figure 53 Selecting KVM Console Option

5. In the KVM window, select the Virtual Media tab. See Figure 54.

6. Click Add Image button in the Client View selection window.

7. Browse to the Red Hat Enterprise Linux Server 6.2 installer ISO image file.


Note The Red Hat Enterprise Linux 6.2 DVD should be available on the client machine.


Figure 54 Adding an ISO Image

8. Click Open to add the image to the list of virtual media.

9. Check the Mapped check box for the image you just added. See Figure 55.

Figure 55 Mapping ISO Image

10. In the KVM console, select the KVM tab to monitor the bootup.

11. In the KVM console, click Boot Server.

12. Click OK.

13. Click OK to reboot the system.

On reboot, the server detects the presence of the Red Hat Enterprise Linux Server 6.2 install media.

14. Select Install or Upgrade an Existing System. See Figure 56.

Figure 56 Select Install Option

15. Skip the Media test as the ISO Image is used for the installation.

16. Click Next. The Red Hat Linux Welcome Screen appears.

17. Select the Language for the installation.

18. Click the Basic Storage Devices radio button.

19. Click the Fresh Installation radio button.

20. Enter the host name of the server and click Next.

21. Click Configure Network. The Network Connections window appear.

22. In the Network Connections window, select the Wired tab.

23. Select the interface System eth0 and click Edit.

24. Editing System eth0 appears. See Figure 57.

25. Check the Connect automatically check box.

26. Select Manual in the Method drop-down list.

27. Click Add and enter IP Address, the netmask and the gateway.

For this demonstration, the following values have been used:

IP Address: 10.29.160.53

Netmask: 255.255.255.0

Gateway: 10.29.160.1

28. Add DNS servers (optional).

29. Click Apply.

Figure 57 Configuring Network for eth0

30. Repeat the steps 26 to steps 32 to configure the network for the System eth1. The following values have been used (See Figure 58):

IP Address: 192.168.12.11

Netmask: 255.255.255.0

Figure 58 Configuring Network for eth1

31. Repeat the steps 26 to steps 32 to configure the network for System eth2. The following values have been used:

IP Address: 192.168.11.11

Netmask: 255.255.255.0

32. Select the appropriate time zone.

33. Enter the root password and click Next.

34. Select Use All Space and click Next. See Figure 59.

35. Choose an appropriate boot drive.

Figure 59 Selecting Install Option

36. Click Write changes to the disk and click Next.

37. Select Basic Server and click Next. See Figure 60.

Figure 60 Selecting Type of Installation

38. After the installer has finished loading, it will continue with the installation.

39. Reboot the system after the installation is complete.

Repeat the above steps (1 to 39) to install the Red Hat Linux on servers 2 to 64.


Note You can automate the OS installation and configuration of the nodes through the Preboot Execution Environment (PXE) boot or through third party tools.


Table 5 describes the hostnames and their corresponding IP addresses.

Table 5 Hostnames and IP Addresses

Hostname
eth0
eth1
eth2

rhel1

10.29.160.53

192.168.12.11

192.168.11.11

rhel2

10.29.160.54

192.168.12.12

192.168.11.12

rhel3

10.29.160.55

192.168.12.13

192.168.11.13

rhel4

10.29.160.56

192.168.12.14

192.168.11.14

rhel5

10.29.160.57

192.168.12.15

192.168.11.15

rhel6

10.29.160.58

192.168.12.16

192.168.11.16

rhel7

10.29.160.59

192.168.12.17

192.168.11.17

rhel8

10.29.160.60

192.168.12.18

192.168.11.18

rhel9

10.29.160.61

192.168.12.19

192.168.11.19

rhel10

10.29.160.62

192.168.12.20

192.168.11.20

rhel11

10.29.160.63

192.168.12.21

192.168.11.21

rhel12

10.29.160.64

192.168.12.22

192.168.11.22

rhel13

10.29.160.65

192.168.12.23

192.168.11.23

rhel14

10.29.160.66

192.168.12.24

192.168.11.24

rhel15

10.29.160.67

192.168.12.25

192.168.11.25

rhel16

10.29.160.68

192.168.12.26

192.168.11.26

...

...

...

...

rhel64

10.29.160.116

192.168.12.74

192.168.11.74


Post OS Install Configuration

For management activities such as MapR installation, parallel shell, creating a local Red Hat repo and others, an admin node is defined. In this document, rhel1 is used for the management purpose.

Setting Up Password-less Login

To manage all the cluster nodes from the admin node it is necessary to setup the password-less login. It assists in automating common tasks with parallel-ssh (pssh) and shell-scripts without having to use passwords.

Follow these steps to enable password-less login across all the nodes following the RHEL installation across all the nodes in the cluster:

1. Login to the admin node (rhel1).

ssh 10.29.160.53
 
   

2. Run the ssh-keygen command to create both public and private keys on the admin node.

3. Run the following commands from the admin node to copy the public key id_rsa.pub to all the nodes of the cluster. The .ssh-copy-id command appends the keys to the remote-host.

.ssh/authorized_key. 
for IP in {53..116}; do echo -n "$IP -> "; ssh-copy-id -i ~/.ssh/id_rsa.pub 
10.29.160.$IP; done
 
   

4. Enter yes at the command prompt to continue connecting.

5. Enter the password of the remote host to login.

Installing and Configuring Parallel SSH

Installing Parallel-SSH

Parallel-ssh is used to run commands on several hosts at the same time. It takes a file of hostnames and a few common ssh parameters as parameters, and executes the given command in parallel on the specified nodes.

1. Download the pssh.

wget https://parallel-ssh.googlecode.com/files/pssh-2.3.1.tar.gz
scp pssh-2.3.1.tar.gz rhel1:/root
 
   

2. Run the following command to copy pssh-2.3.1.tar.gz to the admin node:

ssh rhel1 
tar xzf pssh-2.3.1.tar.gz
cd pssh-2.3.1
python setup.py install

3. Extract and install pssh on the admin node.

4. Create a host file containing the IP addresses of all the nodes in the cluster. This file is passed as a parameter to pssh to identify the nodes and run the commands on them.


Note You can create multiple host files based on roles such as datanodes, zookeepernodes, allnodes.


vi /root/allnodes 
# This file contains ip address of all nodes of the cluster 
#used by parallel-shell (pssh). For Details man pssh
10.29.160.53
10.29.160.54
10.29.160.55
10.29.160.56
10.29.160.57
10.29.160.58
10.29.160.59
10.29.160.60	
10.29.160.61
10.29.160.62
10.29.160.63
10.29.160.64
10.29.160.65
10.29.160.66
10.29.160.67
10.29.160.68
...
10.29.160.116

Installing Cluster Shell

1. Download cluster shell (clush) and install it on rhel1.

Cluster shell is available from the Extra Packages for Enterprise Linux (EPEL) repository.

wget 
http://dl.fedoraproject.org/pub/epel//6/x86_64/clustershell-1.6-1.el6.noarch.rpm
scp clustershell-1.6-1.el6.noarch.rpm rhel1:/root/
 
   

2. Login to rhel1 and install cluster shell.

yum install clustershell-1.6-1.el6.noarch.rpm
 
   

3. Edit /etc/clustershell/groups file to include hostnames for all the nodes of the cluster.

For 64 node cluster all: rhel[1-64]

Configuring /etc/hosts file

Follow these steps to create the host file across all the nodes in the cluster:

1. Run the following commands to populate the host file with IP addresses and corresponding hostnames on the admin node (rhel1):

vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.12.11 rhel1
192.168.12.12 rhel2
192.168.12.13 rhel3
192.168.12.14 rhel4
192.168.12.15 rhel5
192.168.12.16 rhel6
192.168.12.17 rhel7
192.168.12.18 rhel8
192.168.12.19 rhel9
192.168.12.20 rhel10
192.168.12.21 rhel11
192.168.12.22 rhel12
192.168.12.23 rhel13
192.168.12.24 rhel14
192.168.12.25 rhel15
192.168.12.26 rhel16
...
192.168.12.74 rhel64
 
   

2. Run the following commands to deploy /etc/hosts from the admin node (rhel1) to all the nodes:

pscp -h /root/allnodes /etc/hosts /etc/hosts

Creating Red Hat Local Repository

To create a repository using RHEL DVD or ISO on the admin node (in this deployment rhel1 is used for this purpose), create a directory with all the required rpms, run the createrepo command and then publish the resulting repository.

1. Login to rhel1 node, and run the following command to create a directory that would contain the repository:

mkdir -p /var/www/html/rhelrepo
 
   

2. Copy the contents of the Red Hat DVD to /var/www/html/rhelrepo.

3. Alternatively, if you have access to a Red Hat ISO Image, copy the ISO file to rhel1.

scp rhel-server-6.2-x86_64-dvd.iso rhel1:/root
 
   

The Red Hat ISO file is located in the present working directory.

mkdir -p /mnt/rheliso
mount -t iso9660  -o loop /root/rhel-server-6.2-x86_64-dvd.iso /mnt/rheliso/
 
   

4. Copy the contents of the ISO to the /var/www/html/rhelrepo directory.

cp -r /mnt/rheliso/* /var/www/html/rhelrepo
 
   

5. Run the following command on the rhel1 to create a .repo file that enables the use of the yum command:

vi /var/www/html/rhelrepo/rheliso.repo
[rhel6.2]
name=Red Hat Enterprise Linux 6.2
baseurl=http://10.29.160.53/rhelrepo
gpgcheck=0
enabled=1
 
   

Note The yum command based on the repo file requires httpd to be running on rhel1 so that the other nodes can access the repository.


6. Copy the rheliso.repo to all the nodes of the cluster.

pscp -h /root/allnodes /var/www/html/rhelrepo/rheliso.repo /etc/yum.repos.d/

7. To use the repository files on rhel1 without httpd, edit the baseurl of the repo file.

/etc/yum.repos.d/rheliso.repo to point repository location in the file system.
vi /etc/yum.repos.d/rheliso.repo
[rhel6.2]
name=Red Hat Enterprise Linux 6.2
baseurl=file:///var/www/html/rhelrepo
gpgcheck=0
enabled=1
 
   

8. Run pssh -h /root/allnodes "yum clean all" command:

Creating the Red Hat Repository Database

1. Install the createrepo package.

2. Regenerate the repository database(s) for the local copy of the RHEL DVD contents.

3. Run the following commands to purge the yum caches:

yum -y install createrepo
cd /var/www/html/rhelrepo
createrepo .
yum clean all

Upgrading LSI driver

The latest LSI driver is essential for performance and bug fixes.

To download the latest LSI drivers, see:

http://software.cisco.com/download/release.html?mdfid=284296254&flowid=31743&softwareid=283853158&release=1.5.1&relind=AVAILABLE&rellifecycle=&reltype=latest

1. In the ISO image, the required driver kmod-megaraid_sas-v06.504.01.00.rpm can be located at: ucs-cxxx-drivers.1.5.1\Linux\Storage\LSI\92xx\RHEL\RHEL6.2

2. Download and transfer kmod-megaraid_sas-v06.504.01.00.rpm driver to the admin node (rhel1).

3. Run the following commands to install the rpm on all nodes of the cluster:

pscp -h /root/allnodes kmod-megaraid_sas-v06.504.01.00_rhel6.2-2.x86_64.rpm /root/
 
   

In this design solution the rpm is located in the working directory of rhel1.

pssh -h /root/allnodes  "rpm -ivh 
kmod-megaraid_sas-v06.504.01.00_rhel6.2-2.x86_64.rpm"

Run the following command to verify the version of kmod-megaraid_sas driver that is being used on all the nodes (confirm all versions are same):

pssh -h /root/allnodes  "modinfo megaraid_sas | head -5"

Installing HTTPD

1. Install httpd on the admin node to host repositories.

The Red Hat repository is hosted using HTTP on the admin node, and this machine is accessible by all the hosts in the cluster.

yum -y install httpd
 
   

2. Add the ServerName as 10.29.160.53:80, and make the necessary changes to the server configuration file.

/etc/httpd/conf/httpd.conf 

3. Run the following command to make sure that the httpd is able to read the repofiles:

chcon -R -t httpd_sys_content_t /var/www/html/rhelrepo
 
   

4. Run the following command to start httpd:

service httpd start
chkconfig httpd on

Configuring Data Disk Drives

This section describes how to configure the disk drives using the LSI MegaCLI. The section Configuring a Disk Drive for Operating System describes the steps to configure the first disk drive for the operating system.

1. Download the MegaCLI (from LSI website) and its dependencies and transfer it to the admin node. http://www.lsi.com/support/Pages/Download-Results.aspx?keyword=9266-8i

a. scp /root/MegaCli64 rhel1:/root/

b. scp /root/Lib_Utils-1.00-08.noarch.rpm rhel1:/root/

c. scp /root/Lib_Utils2-1.00-01.noarch.rpm rhel1:/root/

2. Run the following commands to copy the three files to all the nodes:

a.	 pscp -h  /root/allnodes  /root/MegaCli64  /root/

b.	 pscp -h  /root/allnodes  /root/Lib_Utils*  /root/

3. Run the following command to install the rpms on all the nodes:

pssh -h /root/allnodes "rpm -ivh Lib_Utils*"

4. Run the following command from the admin node to create the virtual drives with RAID 0 configurations on all the nodes:

pssh -h /root/allnodes "./MegaCli64 -cfgeachdskraid0 WB RA direct NoCachedBadBBU 
strpsz1024 -a0"
 
   

WB: Write Back

RA: Read Ahead

NoCachedBadBBU: Do not write cache when the BBU is bad.

Strpsz1024: Strip Size of 1024K


Note The commands do not override any existing configuration. To clear and reconfigure existing configurations see: Embedded MegaRAID Software Users Guide


MapR Overview

MapR is a complete, industry-standard, Hadoop distribution with key improvements. With MapR, there are no Java dependencies or reliance on the Linux file system. MapR provides a dynamic read-write data layer that brings unprecedented dependability, ease-of-use, and world-record speed to Hadoop, NoSQL, database and streaming applications in one unified Big Data platform.

Figure 61 Services Surrounding The Hadoop Map Reduce

Figure 61 illustrates the services surrounding the basic Hadoop idea of Map and Reduce operations performed across a distributed storage system. Some services provide management and others run at the application level.

For more information on MapR software, see:

http://www.mapr.com/doc/display/MapR/Installation+Guide

MapR Editions

Table 6 describes the MapR Hadoop features that are available with M5 and M7 editions. These features are licensed separately based on customer requirements.

Table 6 MapR Editions

Edition
Description

M5

Offers high availability and data protection, including multinode NFS

M7

Offers M5 features and support for NoSQL HBase tables that are independent of the Apache HBase RegionServer architecture


A trial license for MapR M5 is available online, and can be easily deployed after the software is installed. Permanent M5 or M7 licenses are available from MapR technologies.

Unique Features of MapR Hadoop

MapR is a complete Hadoop distribution, but not all services are required on every node. Every Hadoop cluster requires JobTracker and TaskTracker services to manage Map/Reduce tasks. In addition, MapR requires the ZooKeeper service to coordinate the cluster, and at least one node must run the CLDB service. The WebServer service is required to support browser-based cluster management through the MapR Control System. MapR data nodes typically run TaskTracker and FileServer.

MapR offers real-time read/write storage layer, which is highly appreciated by Hadoop administrators. The API-compatible with HDFS, MapR Hadoop does not require Namenodes. Furthermore, MapR utilizes raw disks and partitions without Logical Volume Manager. Many Hadoop installation documents spend pages discussing HDFS and Namenodes, and MapR Hadoop's solution is simpler to install and offers higher performance.

The MapR Data Platform (MDP) stores data in volumes, conceptually in a set of containers distributed across a cluster. Each container includes its own metadata, eliminating the central Namenode single point of failure. A required directory of container locations, the Container Location Database (CLDB), can improve network performance and provide high availability. Data stored by the MDP can be files or NoSQL tables that support HBase semantics.

The warden process runs on all nodes to manage, monitor, and report on the other services on each node. The MapR cluster uses Apache ZooKeeper to coordinate services. ZooKeeper prevents service conflicts by enforcing a set of rules and conditions that determine which instance of each service is the master. The warden does not start any services unless ZooKeeper is reachable and more than half of the configured ZooKeeper nodes (a quorum) are live.

MapR manages the data in a cluster in a set of volumes. Volumes can be mounted in the Linux filesystem in a hierarchical directory structure, but volumes do not contain other volumes. Each volume has its own policies and other settings, so it is important to define a number of volumes in order to segregate and classify the data.

It is important to define volumes for each user, for each project, and so on. For streaming data, you can create a new volume to store new data every day or week or month. Higher volume granularity, makes it easy to specify backup or other policies for subsets of the data.

Installation Process

This section provides the steps required to install the MapR Hadoop.

Planning the cluster

Determine which services run on each node.

Installing MapR

Each node in the cluster, even purely data/compute nodes, runs several services. Obtain and install MapR packages, using a package manager, a local repository, or a downloaded tarball.

After installing services on a node, configure it to participate in the cluster, and then initialize the raw disk resources.

Bringing up the cluster

Start the nodes and check the cluster. Verify node communication and that services are up and running.

Create one or more volumes to organize data.

Installing Hadoop ecosystem components

Install additional Hadoop components alongside MapR services.

Planning the Cluster

The first step towards deploying the MapR is planning which nodes contribute to the cluster, and selecting the services that will run on each node.

MapR Services

Table 7 describes the MapR services and the corresponding descriptions.

Table 7 MapR Services and Description

Service
Description

Warden

The Warden service runs on every node, coordinating the node's contribution to the cluster.

TaskTracker

The TaskTracker service starts and tracks MapReduce tasks on a node. The TaskTracker service receives task assignments from the JobTracker service and manages task execution.

FileServer

FileServer is the MapR service which manages disk storage for the MapR Data Platform on each node.

CLDB

Maintains the container location database (CLDB) service. The CLDB service maintains the MapR Data Platform storage (MapR-FS) and is aware of all NFS and data nodes in the cluster. The CLDB service coordinates data storage services among MapR File System (MapR-FS) nodes, MapR NFS gateways, and MapR clients.

NFS

Provides read-write MapR Direct Access NFS™ access to the cluster, with full support for concurrent read and write access.

JobTracker

Hadoop JobTracker service. The JobTracker service coordinates the execution of MapReduce jobs by assigning tasks to TaskTracker nodes and monitoring task execution.

ZooKeeper

Provides coordination. Enables high availability (HA) and fault tolerance for MapR clusters by providing coordination.

HBase Master

The HBase master service manages the region servers that make up HBase table storage.

Web Server

Runs the MapR Control System and provides the MapR Heatmap™.

Metrics

Provides real-time analytics data on cluster and job performance through the Job Metrics interface.

HBase Region Server

HBase region server is used with the HBase Master service and provides storage for an individual HBase region.

Pig

Pig is a high-level data-flow language and execution framework.

Hive

Hive is a data warehouse that supports SQL-like adhoc querying and data summarization.

Mahout

Mahout is a set of scalable machine-learning libraries that analyze user behavior.

Sqoop

Sqoop is a tool for transferring bulk data between Hadoop and relational databases.


Hostnames and Roles

Table 8 describes the cluster plan of a 64-node cluster with hostnames and roles assignments for TaskTracker (TT), JobTracker (JT), FileServer (FS), Container Location Database (CLDB), Zookeeper, Metrics, and Webserver, used in this design solution.

Table 8 Host names and Role Assignment

Hostname
Rack-1
Hostname
Rack-2
Hostname
Rack-3
Hostname
Rack-4

rhel1

CLDB,TT, FS

rhel17

CLDB,TT, FS

rhel33

CLDB,TT, FS

rhel49

TT, FS

rhel2

ZooKeeper

TT, FS

rhel18

ZooKeeper

TT, FS

rhel34

ZooKeeper

TT, FS

rhel50

TT, FS

rhel3

Webserver,

TT, FS

rhel19

Webserver,

TT, FS

rhel35

Webserver,

TT, FS

rhel51

Webserver,

TT, FS

rhel4

Metrics, TT, FS

rhel20

Metrics, TT, FS

rhel36

Metrics, TT, FS

rhel52

TT, FS

rhel5

JT, TT, FS

rhel21

JT, TT, FS

rhel37

JT, TT, FS

rhel53

TT, FS

rhel6

TT, FS

rhel22

TT, FS

rhel38

TT, FS

rhel54

TT, FS

rhel7

TT, FS

rhel23

TT, FS

rhel39

TT, FS

rhel55

TT, FS

rhel8

TT, FS

rhel24

TT, FS

rhel40

TT, FS

rhel56

TT, FS

rhel9

TT, FS

rhel25

TT, FS

rhel41

TT, FS

rhel57

TT, FS

rhel10

TT, FS

rhel26

TT, FS

rhel42

TT, FS

rhel58

TT, FS

rhel11

TT, FS

rhel27

TT, FS

rhel43

TT, FS

rhel59

TT, FS

rhel12

TT, FS

rhel28

TT, FS

rhel44

TT, FS

rhel60

TT, FS

rhel13

TT, FS

rhel29

TT, FS

rhel45

TT, FS

rhel61

TT, FS

rhel14

TT, FS

rhel30

TT, FS

rhel46

TT, FS

rhel62

TT, FS

rhel15

TT, FS

rhel31

TT, FS

rhel47

TT, FS

rhel63

TT, FS

rhel16

TT, FS

rhel32

TT, FS

rhel48

TT, FS

rhel64

TT, FS


Prerequisites for MapR Software

This section details the prerequisites for the MapR software. There are a few customizations to the standard linux installation that are required for the MapR deployment to function properly. These include the NTP configuration, enabling syslog, setting ulimit, disabling the SELinux, setting TCP retires, and disabling linux firewall and few package dependencies.

Package Dependencies for Red Hat

Make sure the following packages are installed on each node:

bash

rpcbind

Ajaxterm

dmidecode

glibc

hdparm

iputils

irqbalance

libgcc

libstdc++

redhat-lsb

rpm-libs

sdparm

shadow-utils

syslinux

unzip

zip

On ZooKeeper nodes, nc is required for netcat.

On nodes running the Metrics service, soci-mysql is required.

Creating User Accounts

Users of the cluster must have the same credentials and uid on every node in the cluster. Each user (or department) that runs the MapR jobs needs an account and must belong to a common group (gid).

In addition, a MapR user with full privileges to administer the cluster is created. If a directory service, such as LDAP, is not used, this user is created on each node. If a user named 'mapr' does not exist, the installing MapR creates it. It is recommended that you create the user named 'mapr' in advance in order to test the connectivity issues prior to the installation step.

Every user (including the 'mapr' user) must have the same uid and primary gid on every node.

1. Run the following command to create the mapr account on all the cluster nodes:

clush -B -a 'useradd -u 3000 -m -s /bin/bash mapr'

Note To create a user, run the command as root. If you get an error "cannot lock /etc/passwd" implies that the command was not run as root.


2. Run the su mapr command to test the mapr user has been created.

Verify that a home directory has been created (usually /home/mapr) and that the mapr user has read-write access to it.


Note The mapr user must have write access to the /tmp directory, or the warden fails to start services.


Configuring Password-less ssh for MapR User (Optional)

It is important that the common user be able to ssh from each node to the other nodes without using a password. If the password-less ssh is not enabled for nodes running the MapR webserver service, certain MapR features, such as centralized disk management and rolling upgrades, will not be available.

To set up the password-less ssh, generate a key pair on each node, and append the key to an authorization file. Then copy this authorization file to each node, so that each node is available from every other node. You can either allow all the nodes to access the other nodes, or enable access only for the MCS nodes (rhel3, rhel19, and rhel35).

1. Run the following commands to create a key as the mapr user:

mapr>   ssh-keygen -t rsa -P "" -f ~mapr/.ssh/id_rsa
 
   

2. Run the following command to generate a key across all nodes:

clush -a 'sudo -u mapr ssh-keygen -t rsa -P "" -f ~mapr/.ssh/id_rsa'
 
   

3. Run the following commands to append the key and then redistribute it across all the nodes:

clush -w rhel3 'cat /home/mapr/.ssh/id_rsa.pub'  >> /tmp/ak.mapr
clush -w rhel19 'cat /home/mapr/.ssh/id_rsa.pub'  >> /tmp/ak.mapr
clush -w rhel35 'cat /home/mapr/.ssh/id_rsa.pub'  >> /tmp/ak.map
clush -a -copy /tmp/ak.map --dest=/home/mapr/.ssh/authorized_keys
clush -a chmod 600 /home/mapr/.ssh/authorized_keys
clush -a chown mapr:mapr /home/mapr/.ssh/authorized_keys

Note The file containing the permissions and ownership of the authorized-keys file should be handled carefully to ensure the proper functioning of the ssh.


Installing JDK

MapR services require the Java runtime environment. For this design solution Java SE Development Kit 6 update 45 has been used, however, Java 7 can also be used.

1. Download Java SE Development Kit 6 Update 45 from the Oracle website:

http://www.oracle.com/technetwork/java/javase/downloads/jdk6downloads-1902814.html

2. Transfer the file to rhel1.

scp jdk-6u45-linux-x64-rpm.bin  rhel1:/root/
 
   

3. Run the following commands to create a post java install script on rhel1, and transfer it to all the nodes:

vi /root/java-post-install.sh
#!/bin/bash
for item in java javac javaws jar jps javah; do
alternatives --install /usr/bin/$item $item /usr/java/jdk1.6.0_45/bin/$item 9
alternatives --set $item /usr/java/jdk1.6.0_45/bin/$item
done
clush -b -a -c /root/java-post-install.sh --dest=/root/
 
   

4. Run the following command to create a script on rhel1, and then install java on all the nodes:

#!/bin/bash
echo Copying JDK to Nodes
clush -B -a -c ./jdk-6u45-linux-x64-rpm.bin --dest=/root/
echo Copying post-install to Nodes
clush -B -a -c ./java-post-install.sh --dest=/root/
echo Extract JDK on Nodes
clush -B -a /root/jdk-6u45-linux-x64-rpm.bin
echo Install JDK on Nodes
clush -B -a "rpm -i /root/sun-javadb*.rpm"
clush -B -a "rpm -i /root/jdk-6u45-linux-amd64.rpm"
clush -B -a "rpm -qa | grep jdk"
echo Post Java Install Configure
clush -a -B /root/java-post-install.sh
 
   

Note Verify that the Java SE Development Kit 6 update 45 is properly installed, and JAVA_HOME and PATH are correctly set on all the nodes before proceeding.


MySQL Installation

The MapR Metrics service requires access to a MySQL server running version 5.1 or later. MySQL does not have to be installed on a node in the cluster, but it must be on the same network as the cluster. For more information on metrics configuration in MapR, see:

http://www.mapr.com/doc/display/MapR/Setting+up+the+MapR+Metrics+Database

http://www.oracle.com/technetwork/java/javase/downloads/jdk6downloads-1902814.html

NTP Configuration

The Network Time Protocol (NTP) is used to synchronize the time of all the nodes within the cluster. The Network Time Protocol daemon (ntpd) sets and maintains the system time of day in sync with the timeserver located in the admin node (rhel1). Configuring NTP is critical for any Hadoop Cluster. If server clocks in the cluster drift out of sync, serious problems can occur in the HBase and other MapR services. MapR raises a Time Skew alarm on any out of sync nodes.

Installing an internal NTP server keeps the cluster synchronized even when an outside NTP server is inaccessible.

1. Configure /etc/ntp.conf on the admin node with the following contents:

vi /etc/ntp.conf
driftfile /var/lib/ntp/drift
restrict 127.0.0.1
restrict -6 ::1
server 127.127.1.0 
fudge 127.127.1.0 stratum 10
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
 
   

2. Run the following commands to create /root/ntp.conf on the admin node and copy it to all the nodes:

vi /root/ntp.conf
server 10.29.160.53
driftfile /var/lib/ntp/drift
restrict 127.0.0.1
restrict -6 ::1
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
 
   

3. Run the following command in the admin node (rhel1) to copy ntp.conf file from the admin node to /etc of all the nodes:

for SERVER in {54..116}; do scp /root/ntp.conf 10.29.160.$SERVER:/etc/ntp.conf; 
done

Note Do not use pssh /root/allnodes command without editing the host file allnodes as it overwrites /etc/ntp.conf from the admin node.


4. Run the following commands to synchronize the time and restart NTP daemon on all the nodes:

pssh -h /root/allnodes "yum install -y ntpdate"
pssh -h /root/allnodes "service ntpd stop"
pssh -h /root/allnodes "ntpdate rhel1"
pssh -h /root/allnodes "service ntpd start"

5. Run the following command to restart the NTP daemon on all the reboots:

pssh -h /root/allnodes "chkconfig ntpd on"

Enabling Syslog

Syslog must be enabled on each node to preserve logs regarding killed processes or failed jobs. Modern versions such as syslog-ng and rsyslog are possible, making it more difficult to ascertain if a syslog daemon is present.

Run any of the commands to confirm if the service is properly configured:

clush -B -a rsyslogd -v
clush -B -a service rsyslog status

Setting ulimit

The mapr-warden script uses the ulimit command to set the maximum amount of file descriptors and processes to 64000. Higher values are unlikely to result in an appreciable performance gain.

On each node, ulimit -n specifies the number of inodes that can be opened simultaneously. With the default value of 1024, the system appears to be out of disk space and shows no inodes available. This value should be set to 64000 on every node.

Setting ulimit on RHEL

1. Edit /etc/security/limits.conf and add the following lines:

root soft nofile 64000
root hard nofile 64000
 
   

2. Check that /etc/pam.d/su file contains the following settings:

   #%PAM-1.0
   auth            sufficient      pam_rootok.so
   # Uncomment the following line to implicitly trust users in the "wheel" group.
   #auth           sufficient      pam_wheel.so trust use_uid
   # Uncomment the following line to require a user to be in the "wheel" group.
   #auth           required        pam_wheel.so use_uid
   auth            include         system-auth
   account         sufficient      pam_succeed_if.so uid = 0 use_uid quiet
   account         include         system-auth
   password        include         system-auth
   session         include         system-auth
   session optional        pam_xauth.so
 
   

3. Verify the ulimit setting. Run the following command at the command line:

clush -B -a ulimit -n
 
   
The command should report 64000 as the ulimit.

Note The ulimit values are applied on a new shell. Running the command on a node on an earlier instance of a shell shows old values.


Disabling SELinux

SELinux must be disabled during the MapR install procedure and cluster setup. SELinux can be enabled after installation and while the cluster is running.

SELinux can be disabled by editing /etc/selinux/config and changing the SELINUX line to SELINUX=disabled.

1. Run the following command to disable SELINUX on all nodes:

pssh -h /root/allnodes "sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' 
/etc/selinux/config "

 
   
pssh -h /root/allnodes "setenforce 0"

Note This command fails if SELinux is already disabled.


Setting TCP Retries Parameter

Adjusting the tcp_retries parameter for the system network enables faster detection of failed nodes within the MapR cluster. The advanced networking features of Cisco UCS Manager, ensure that setting TCP retries is safe and is recommended. It is crucial since failures observed at the operating system layer are most likely serious rather than transitory. On each node, set the number of TCP retries to five so that MapR can detect unreachable nodes with less latency.

1. Edit the file /etc/sysctl.conf and add the following line:

net.ipv4.tcp_retries2=5
 
   

2. Save the file and run the following command.

clush -B -a sysctl -p

Disabling the Linux Firewall

The default Linux firewall settings are far too restrictive for any Hadoop deployment. Since the Cisco UCS Big Data deployment is performed in the isolated network, there is no need to leave the iptables service running.

Disabling iptables is not mandatory. Enabling iptables on a node may close ports that are used by MapR. If you enable iptables, make sure that required ports defined remain open. To verify the ports that should remain open, see:

http://www.mapr.com/doc/display/MapR/Ports+Used+by+MapR

1. Run the following commands to disable the iptables:

pssh -h /root/allnodes  "service iptables stop"

2. Run the following command to check if the iptables are disabled:

pssh -h /root/allnodes "chkconfig iptables off"

Cluster Verification and Prerequisite Checks

This section provides a set of micro-benchmarks and prerequisites scripts to verify that all the systems are configured correctly:

STREAM benchmark to test memory bandwidth

RPCtest to test network bandwidth

IOzone to test I/O

Prerequisite script to verify configuration across the cluster

Running these tests are optional. Test results can vary based on topology and configuration.

Running STREAM Benchmark

The STREAM benchmark measures sustainable memory bandwidth (in MB/s) and the corresponding computation rate for simple vector kernels. To download the STREAM benchmark, see:

http://www.cs.virginia.edu/stream/

Follow these steps to run the STREAM benchmark:

1. Log on to the admin node. Copy and extract STREAM file to each node (/root/).

clush -B -a "tar -xvf stream.tgz"
 
   

2. Run the following command to run the STREAM benchmark on all nodes:

clush -B -a "/root/stream/runme.sh > /root/stream.log"
 
   

3. Run the following command to verify the results:

Extract the five lines of the result as shown and verify it on all the nodes.

$clush -B -a "grep -A5 \"Function     \" stream.log"
---------------
rhel1
---------------
Function      Rate (MB/s)   Avg time     Min time     Max time
Copy:        53289.0222       0.0241       0.0240       0.0243
Scale:        73664.0430       0.0175       0.0174       0.0177
Add:          75339.0246       0.0257       0.0255       0.0259
Triad:        76845.8770       0.0252       0.0250       0.0254

Note Results can vary based on the configuration.


Running RPCtest

MapR RPCtest is network bandwidth measurement test. In this solution the methodology adopted to verify the network bandwidth across the cluster requires configuring half the nodes as senders and remaining half as receivers. This test is included in MapR software available at /opt/mapr/servers/tools/rpctest as part of the installation.

Follow the steps below to run RPCtest:

1. Log on to the admin node and run the following commands to create the script:

#!/bin/bash
# Define sender nodes
# 8 servers in each rack act as servers and the other half as clients 
senders=( 192.168.12.11 192.168.12.12 192.168.12.13 192.168.12.14 
192.168.12.19 192.168.12.20 192.168.12.21 192.168.12.22 
192.168.12.27 192.168.12.28 192.168.12.29 192.168.12.30 
192.168.12.35 192.168.12.36 192.168.12.37 192.168.12.38 
192.168.12.43 192.168.12.44 192.168.12.45 192.168.12.46 
192.168.12.51 192.168.12.52 192.168.12.53 192.168.12.54
192.168.12.59 192.168.12.60 192.168.12.61 192.168.12.62 
192.168.12.67 192.168.12.68 192.168.12.69 192.168.12.70 )
for node in "${half1[@]}"; do
  ssh -n $node /opt/mapr/servers/tools/rpctest -server &
done
sleep 9 # let the servers set up
# Define receiver nodes
receivers=( 192.168.12.15 192.168.12.16 192.168.12.17 192.168.12.18 
192.168.12.23 192.168.12.24 192.168.12.25 192.168.12.26 
192.168.12.31 192.168.12.32 192.168.12.33 192.168.12.34
192.168.12.39 192.168.12.40 192.168.12.41 192.168.12.42  
192.168.12.47 192.168.12.48 192.168.12.49 192.168.12.50
192.168.12.55 192.168.12.56 192.168.12.57 192.168.12.58 
192.168.12.63 192.168.12.64 192.168.12.65 192.168.12.66
192.168.12.71 192.168.12.72 192.168.12.73 192.168.12.74 )
i=0
for node in "${receivers[@]}"; do
  ssh -n $node "/opt/mapr/servers/tools/rpctest -client 5000 \ ${senders[$i]} > 
rpctest.log" &
  ((i++))
done
#wait $!   # Comment/uncomment this to make it sequential/concurrent
sleep 5
tmp=${half1[@]}
clush -w ${tmp// /,} pkill rpctest
 
   

2. Run runRPCtest.sh command from the admin node.

3. Results are generated on receiver nodes. Verify results for all the nodes.

$clush -B -w 192.168.12. [19-26, 35-42, 51-58,67-74] cat rpctest.log
---------------
Rhel19
---------------
23:49:42  rpcs 17620,  mb 1150.6
23:49:43  rpcs 17772,  mb 1164.7
23:49:44  rpcs 17771,  mb 1164.6
23:49:45  rpcs 17024,  mb 1115.7
Rate: 1108.93 mb/s, time: 4.73158 sec, #rpcs 80063, rpcs/sec 16921
---------------

Note Results can vary based on the topology and configuration.


Running IOzone Benchmark

IOzone is a filesystem benchmark that measures the performance of various I/O operations, such as read, write, re-read, re-write, fread, fwrite, random read and random write.


Warning IOzone is data destructive. Do not run the test on disks with data.

Follow these steps to run IOzone benchmark test:

1. Download IOzone from http://www.iozone.org/ and copy to all nodes at /root/.

2. Create the following script, runIOzone.sh on the admin node.

#!/bin/bash
# Parallel IOzone tests to stress/measure disk controller
# These tests are destructive therefore
# Test must be run BEFORE MapR filesystem is formatted with disksetup 
# Run iozone command once on a single device to verify iozone command
 
   
D=$(dirname "$0")
abspath=$(cd "$D" 2>/dev/null && pwd || echo "$D")
 
   
# run iozone with -h option for usage, adjust path below for iozone location
# Set list of device names for the 'for' loop
disks=$(lsblk -id | grep -o ^sd. | grep -v ^sda |sort); #echo $disks; exit
 
   
set -x
for disk in $disks; do
   $abspath/iozone -I -r 1M -s 80G -i 0 -i 1 -i 2 -f /dev/$disk > 
$disk-iozone.log&
   sleep 3 #Some controllers seem to lockup without a sleep
done

Note Script assumes sda is an OS drive and removes it from the list. If OS is installed on the other drive, change the script accordingly.


3. Copy runIOzone.sh to all the nodes at location /root/.

4. Run the following command to start the test:

clush -B -a  runIOzone.sh
 
   

5. Verify that the tests are running and wait for its completion.

clush -B -a "ps -aef | grep iozone | wc -l"
---------------
rhel[1-64] (64)
---------------
 
   

6. Run the following command to verify the test results.

Result is generated for each disk as sd<x>-iozone.log, where <x> is the device id. These logs have sequential and random write and read latencies from each disks.

$ grep " 83886080    " sd*.log
sdb-iozone.log: 83886080    1024 97978 97951 100673 99254 49002 66552
sdc-iozone.log: 83886080   1024 101290 100745 97803 97006 48863 66671
sdd-iozone.log:  83886080    1024 94286 94937 96752 95872 48871 65605

Note Results can vary based on configuration.


Prerequisite Checks

The section describes the steps to create the prerequisite script that helps to verify CPU, memory, NIC, storage adapter settings. This script also checks additional prerequisites such as NTP status, SELinux status, ulimit settings, JAVA_HOME settings and JDK version, IP address and hostname resolution, Linux version, firewall settings and any missing rpms.

1. Run the following commands to create a script file cpa-pre-requisite.sh on the admin node (rhel1):

#!/bin/bash
# Parallel shell for pre-requisite verification 
shopt -s expand_aliases
 
   
sep='==============================================================='
D=$(dirname "$0")
abspath=$(cd "$D" 2>/dev/null && pwd || echo "$D")
eval enpath=$(echo /sys/kernel/mm/*transparent_hugepage/enabled)
 
   
#Setup Parallel Shell to use
alias psh=clush
alias dshbak=clubak
parg="-a"
 
   
date; 
echo ================ System =======================
echo $sep
 
   
# System info
psh $parg "dmidecode |grep -A2 '^System Information'" | dshbak -c
echo $sep
 
   
# BIOS 
psh $parg "dmidecode | grep -A3 '^BIOS I'" | dshbak -c
echo $sep
 
   
# Memory 
psh $parg "cat /proc/meminfo | grep -i ^memt | uniq" | dshbak -c
echo $sep
 
   
# Number of DIMMs
psh $parg "echo -n 'DIMM slots: '; dmidecode |grep -c '^[[:space:]]*Locator:'" | 
dshbak -c
echo $sep
psh $parg "echo -n 'DIMM count is: '; dmidecode | grep -c 	'Size: [0-9]* MB'" | 
dshbak -c
echo $sep
psh $parg "dmidecode | awk '/Memory Device$/,/^$/ {print}' | grep -e '^Mem' -e 
Size: -e Speed: -e Part | sort -u | grep -v -e 'NO DIMM' -e 'No Module Installed' 
-e Unknown" | dshbak -c
echo $sep
 
   
# CPU 
psh $parg "grep '^model name' /proc/cpuinfo | sort -u" | dshbak -c
#echo $sep
psh $parg "lscpu | grep -v -e op-mode -e ^Vendor -e family -e Model: -e Stepping: 
-e BogoMIPS -e Virtual -e ^Byte -e '^NUMA node(s)'" | dshbak -c
echo $sep
 
   
# NIC 
psh $parg "lspci | grep -i ether" | dshbak -c
psh $parg "ip link show | sed '/ lo: /,+1d' | awk '/UP/{sub(\":\",\"\",\$2);print 
\$2}' | xargs -l ethtool | grep -e ^Settings -e Speed" | dshbak -c
echo $sep
 
   
# Storage 
psh $parg "echo 'Storage Controller: '; lspci | grep -i -e raid -e storage -e lsi" 
| dshbak -c
psh $parg "dmesg | grep -i raid | grep -i scsi" | dshbak -c
echo $sep
psh $parg "lsblk -id | awk '{print \$1,\$4}'|sort | nl" | dshbak -c
echo $sep
 
   
echo $sep
echo ================ Software  ===========================
echo $sep
 
   
# Linux Release 
psh $parg "cat /etc/*release | uniq" | dshbak -c
echo $sep
 
   
# Linux Version
psh $parg "uname -srvm | fmt" | dshbak -c
echo $sep
 
   
# CPU Settings
#psh $parg "grep AUTOCONF /etc/sysconfig/network" | dshbak -c; echo $sep
psh $parg "echo -n 'Transparent Huge Pages: '; cat $enpath" | dshbak -c
echo $sep
psh $parg "echo -n 'CPUspeed Service: '; service cpuspeed status" |dshbak -c
psh $parg "echo -n 'CPUspeed Service: '; chkconfig --list cpuspeed" |dshbak -c
echo $sep
 
   
# Firewall 
psh $parg "chkconfig --list iptables" | dshbak -c
echo $sep
psh $parg "service iptables status | head -10" | dshbak -c
echo $sep
 
   
# NTP status
psh $parg date | dshbak -c
echo $sep
psh $parg "ntpstat | head -1" | dshbak -c
echo $sep
psh $parg "echo -n 'SElinux status: '; grep ^SELINUX=\ /etc/selinux/config" | 
dshbak -c
echo $sep
 
   
# JAVA_HOME and Version
psh $parg "echo Check Permissions; ls -ld / /tmp | awk '{print \$1,\$3,\$4,\$9}'" 
| dshbak -c; echo $sep
psh $parg 'java -version; echo JAVA_HOME is ${JAVA_HOME:-Not Defined!}' |& dshbak 
-c; echo $sep
 
   
# Hostname 
echo Hostname lookup
psh $parg 'hostname -I'; echo $sep
 
   
# Open File Limit
psh $parg 'echo -n "Open file limit(should be >32K): "; ulimit -n' | dshbak -c; 
echo $sep
 
   
# MapR related RPMs 
psh $parg 'rpm -qa | grep -i nfs |sort' | dshbak -c; echo $sep
psh $parg 'echo Missing RPMs: ; for each in make patch redhat-lsb irqbalance 
syslinux hdparm sdparm dmidecode nc; do rpm -q $each | grep "is not installed"; 
done' | dshbak -c; echo $sep
psh $parg "ls -d /opt/mapr/* | head" | dshbak -c; echo $sep
 
   
# mapr login for hadoop
psh $parg 'echo "mapr login for Hadoop "; getent passwd mapr && { echo ~mapr/.ssh; 
ls ~mapr/.ssh; }' | dshbak -c
echo $sep
psh $parg 'echo "Root login "; getent passwd root && { echo ~root/.ssh; ls 
~root/.ssh; }' | dshbak -c; echo $sep
 
   
exit
 
   

2. Run the cpa-pre-requisite.sh script from the admin node to verify configuration and settings on all nodes in the cluster prior to installing the MapR software.

Installing MapR Software

Installing MapR software across the cluster involves performing several steps on each node. To make the installation process simpler, we will postpone the installation of ancillary Hadoop components, such as HBase or Hive, until after the base MapR platform is deployed. However, experienced administrators can install these components at the same time as MapR software. This section refers to Table 8 for the role assignments for installation of services on the 64-node cluster.

The following sections describe the steps and options for installing MapR software:

Preparing packages and repositories

MapR installation

Installing MapR packages

Verify successful installation

Configure the node with the configure.sh script

Formatting disks with the disksetup script

Preparing Packages and Repositories

To install the MapR software successfully, each node must have access to the package files. There are several ways to specify the location of the source packages:

Using internet repository of MapR

Using a local repository

Using a local path containing rpm package files

It is important to consider all the packages that the MapR software depends on. You can install dependencies on each node before beginning the MapR installation process, or you can specify repositories and allow the package manager on each node to resolve dependencies. In this design solution for the targeted Cisco UCS deployment the local repository mechanism is used. For more information on how to prepare packages and repositories, see: http://www.mapr.com/doc/display/MapR/Installing+MapR+Software#InstallingMapRSoftware-PreparingPackagesandRepositories

Using a Local Repository

You can set up a local repository on the admin node to provide access to installation packages. In this case the package manager on each node installs from packages in the local repository.

This section provide instructions to set up a local repository for Red Hat Linux distribution. These instructions create a single repository that includes both MapR components and the Hadoop ecosystem components.

Follow these steps to create the local repositories:

1. Login as root on the admin node (rhel1).

2. Create the following directory if it does not exist: /var/www/html/mapr.local.

3. Download the following files by substituting the appropriate <version> and <datestamp>:

http://package.mapr.com/releases/v<version>/redhat/mapr-v<version>GA.rpm.tgz

http://package.mapr.com/releases/ecosystem/redhat/mapr-ecosystem-<datestamp>.rpm.tgz

4. Copy the files to /var/www/html/mapr.local on the admin node, and extract the files.

tar -xvzf mapr-v<version>GA.rpm.tgz
tar -xvzf mapr-ecosystem-<datestamp>.rpm.tgz
 
   

5. Create the base repository headers:

createrepo /var/www/html/mapr.local
 
   

Verify the contents of the new /var/www/html/mapr.local/repodata directory as filelists.xml.gz, other.xml.gz, primary.xml.gz, repomd.xml.

Adding Repository on Node

Follow these steps to add the repository on each node:

1. Create repo file /etc/yum.repos.d/maprtech.repo on the admin node (rhel1):

vi /etc/yum.repos.d/maprtech.repo
[maprtech]
name=MapR Technologies, Inc.
baseurl=http://10.29.160.53/mapr.local
enabled=1
gpgcheck=0
 
   

2. Copy the maprtech.repo specification to all the nodes of the cluster. Update the yum metadata cache so that the repository files are properly accessed.

pscp -h /root/allnodes /etc/yum.repos.d/maprtech.repo /etc/yum.repos.d/
pscp -h /root/allnodes yum makecache

MapR Software Installation

Follow these steps to install the MapR software on each node:

1. Install the planned MapR services as shown in Table 8.

2. Run the configure.sh script to configure the node.

3. Format raw drives and partitions allocated to MapR using the disksetup script.

Table 9 describes the MapR services and corresponding packages to be installed.

Table 9 MapR Services and Packages

Service
Package

CLDB

mapr-cldb

JobTracker

mapr-jobtracker

MapR Control Server

mapr-webserver

MapR Data Platform

mapr-fileserver

Metrics

mapr-metrics

NFS

mapr-nfs

TaskTracker

mapr-tasktracker

ZooKeeper

mapr-zookeeper

Hadoop Ecosystem Components

Use MapR-tested versions, compatible and in some cases improved components

Cascading

mapr-cascading

Flume

mapr-flume

HBase

mapr-hbase-master

mapr-hbase-regionserver

HCatalog

mapr-hcatalog

mapr-hcatalog-server

Hive

mapr-hive

Mahout

mapr-mahout

Oozie

mapr-oozie

Pig

mapr-pig

Sqoop

mapr-sqoop

Whirr

mapr-whirr


Installing MapR Packages

This section describes the commands to install the appropriate packages for each node. The packages are installed based on the cluster plan ( see Table 8). Configuring the local yum repository ensures that the package dependencies are managed correctly.

Installing CLDB and Mapr-NFS
clush -B -w rhel[1,17,33] 'yum -y install mapr-cldb'
 
   
Installing Mapr Webserver
clush -B -w rhel[4,17,33,19] 'yum -y install mapr-webserver'
 
   
Installing Mapr-Zookeeper
clush -B -w rhel[2,18,34] 'yum -y install mapr-zookeeper'
 
   
Installing JobTracker, Mapr-Metrics

Note The install mapr-metrics cannot be installed until mysql and soci-mysql is installed from the epel(which is not available within isolated cluster). The detailed steps to configure the MySQL for metrics is described in the section Post Installation.


clush -B -w rhel[4,19,35] 'yum -y install mapr-jobtracker mapr-metrics'
Installing NFS, Fileserver and Tasktracker on all cluster nodes
clush -B -a 'yum -y install mapr-fileserver mapr-tasktracker'

Verifying the MapR Installation

To verify that the MapR software has been installed successfully, check the /opt/mapr/roles directory on each node. The software is installed in directory /opt/mapr and a file is created in /opt/mapr/roles for every service that installs successfully. Check this directory to verify installation for the node.

For example:

# ls -l /opt/mapr/roles
total 0
-rwxr-xr-x 1 root root 0 May 24 17:59 fileserver
-rwxr-xr-x 1 root root 0 May 24 17:58 tasktracker
-rwxr-xr-x 1 root root 0 May 24 17:58 webserver
-rwxr-xr-x 1 root root 0 May 24 17:58 zookeeper

Configuring the Node Using configure.sh Sript


Note Firstly configure the node,and then prepare raw disks and partitions using the disksetup command.


The script configure.sh configures a node to be a part of a MapR cluster, or modifies services running on an existing node in the cluster. This script creates (or updates) configuration files related to the cluster and the services running on the node.

Configuring All Nodes with CLDB and Zookeeper Locations

Prior to configuring all nodes with CLDB and ZooKeeper, make sure that you have a list of the hostnames for CLDB and ZooKeeper nodes. You can specify the ports for the CLDB and ZooKeeper nodes. If you do not specify them, the following default ports selected:

CLDB - 7222

ZooKeeper - 5181

1. The script configure.sh takes an optional cluster name and log file, and comma-separated lists of CLDB and ZooKeeper host names or IP addresses (and optionally ports) by using the following syntax:

/opt/mapr/server/configure.sh -C <host>[:<port>][,<host>[:<port>]...] -Z 
<host>[:<port>][,<host>[:<port>]...] [-L <logfile>][-N <cluster name>]
 
   

2. Run the following commands to configure all the Nodes with CLDB and Zookeeper Locations:

clush -B -a '/opt/mapr/server/configure.sh -C rhel1,rhel17,rhe33 -Z 
rhel2,rhel18,rhel34 -N <cluster-name> -u mapr -g mapr'

Formatting Disks Using the disksetup Script

If mapr-fileserver is installed on this node, use the following procedure to format disks and partitions for use by MapR cluster.

1. Run the configure.sh script.

2. Run the disksetup script.

3. Create a text file /tmp/disks.txt listing the disks and partitions for use by MapR on the node.

Each line lists either a single disk or all applicable partitions on a single disk.

4. When listing multiple partitions on a line, separate them by spaces.

Identify and Format the Data Disks for MapR

The commands explained in this section considers the OS to be installed in sda and formats the rest of the drives. If OS is installed on any other drive, change the script accordingly to prevent it from accidentally formatting the drive.

Follow these steps to identify and format the data disks:

Run the following command to create the list of disks to be formatted:

clush -B -a "lsblk -id | grep -o ^sd. | grep -v ^sda | sort | sed 's,^,/dev/,' | 
tee /tmp/MapR.disks"
 
   

Confirm that the disks are not in use.

The cfdisk, mount, and pvdisplay utilities can be used to confirm that the system is not using the disks listed in /tmp/MapR.disks. This confirmation is not necessary during the initial setup, however, is important when the nodes are removed or re-added to the cluster.

Run the following command to format the disks to MapR-FS:

clush -B -a '/opt/mapr/server/disksetup -F -W 5 /tmp/MapR.disks'

Note The script disksetup removes all data from the specified disks. Make sure you specify the disks correctly, and also backup essential data.


If you are reusing a node that was used previously in another cluster, it is important to format the disks to remove any traces of data from the old cluster.

Updating Environment Variables in /opt/mapr/conf/env.sh

There are a few key environment variables for the MapR software saved in /opt/mapr/conf/env.sh. These values must be properly configured before launching the cluster software.

1. Run the following commands to check the default files:

#!/bin/bash
# Copyright (c) 2009 & onwards. MapR Tech, Inc., All rights reserved
# Please set all environment variable you want to be used during MapR cluster
# runtime here.
# namely MAPR_HOME, JAVA_HOME, MAPR_SUBNETS
 
   
#export JAVA_HOME=
#export MAPR_SUBNETS=
#export MAPR_HOME=
#export MAPR_ULIMIT_U=
#export MAPR_ULIMIT_N=
#export MAPR_SYSCTL_SOMAXCONN=
 
   

2. Run the following command to set the required environment values:

For this deployment, we need to explicitly set values for JAVA_HOME, MAPR_SUBNETS, and MAPR_HOME (make sure to remove the '#' comment character).

export JAVA_HOME=/usr/java/jdk1.6.0_45
export MAPR_SUBNETS=192.168.11.0/24,192.168.12.0/24
export MAPR_HOME=/opt/mapr
 
   

3. Make all the changes in rhel1:/opt/mapr/conf/env.sh

4. Run the following command to distribute environment values to the entire cluster:

$ clush -B -a -c /opt/mapr/conf/env.sh

Bringing Up The Cluster

The installation of software across a cluster of nodes is smooth, if the services have been pre-planned and each node has been validated. Make sure that each node has been prepared and that the MapR packages have been installed on each node in accordance with the plan described in the section Planning the Cluster. The process for launching the cluster can be broken down into several simple steps as follows:

Initialization sequence

Troubleshooting

Installing the cluster license

Verifying cluster status

Adding volumes

The initialization sequence involves starting the ZooKeeper service, starting the CLDB service, setting up the administrative user, and installing a MapR license. Following the initial steps the cluster is functional on a limited set of nodes. All the services are not functional, however, you can use the MapR Control System Dashboard, or the MapR Command Line Interface, to examine nodes and activity on the cluster. You can then proceed to start services on all remaining nodes.

Initialization Sequence

1. Start the ZooKeeper service. It is important that all ZooKeeper instances start up, because the rest of the system cannot start unless a majority of ZooKeeper instances are up and running.

2. Start the warden service on each node, or at least on the nodes that host the CLDB and webserver services. The warden service manages all MapR services on the node (except ZooKeeper) and helps coordinate communications. Starting the warden automatically starts the CLDB.

Starting the Cluster

1. Run the following command to start ZooKeeper on all the nodes:

clush -B -w rhel[2,18,34] service mapr-zookeeper start
 
   

2. Run the following command to verify that the ZooKeeper service is running properly:

clush -B -w rhel[2,18,34] service mapr-zookeeper status
 
   

The servers should display the running pid for the zookeeper process. If any of the zookeepers is not running properly, check the logs on that system and restart if necessary.

3. Run the following command on the nodes running CLDB or webserver to start the warden:

If the CLDB and webserver are on separate nodes, run the command on each node. If they are on the same node, run the command on only that one node.

clush -B -w rhel[1,17,33] service mapr-warden start

Note Wait for 30 to 60 seconds for the warden to start the CLDB service. Calls to MapR (such as maprcli) may fail if executed before the CLDB has started successfully.


4. Log in to any node that is running the CLDB, and run the following command to give full permission to the administrative user:

clush -B -w rhel1 /opt/mapr/bin/maprcli acl edit -type cluster -user mapr:fc
 
   

5. Run the following command to confirm that the cluster is up before starting other nodes:

clush -B -w rhel[17,33] 'hadoop fs -ls /'
 
   

6. Run the following command to start warden on all nodes:

clush -B -a rhel[3-16,18-32,35-64] service mapr-warden start

Note Wait for 60 seconds for the system to initialize the first time.


Troubleshooting

This sections provides some of the troubleshooting tips while bringing up the cluster. For the latest support tips, see: http://answers.mapr.com

Can each node connect with the others? Check the list of ports that must be open.

Is the warden running on each node? On the node, run the following command as root:

$ service mapr-warden status
 
   

WARDEN running as process 18732.

If the warden service is not running, check the warden log file, /opt/mapr/logs/warden.log, for clues.

Run the following command to restart the warden service:

$ service mapr-warden start
 
   

The ZooKeeper service is not running on one or more nodes.

Check the warden log file for errors related to resources, such as low memory.

Check the warden log file for errors related to user permissions.

Check for DNS and other connectivity issues between ZooKeeper nodes.

The MapR CLI program /opt/mapr/bin/maprcli is not running.

Did you configure this node? See Installing MapR Software section.

Permission errors appear in the log.

Check that MapR changes to the following files have not been overwritten by automated configuration management tools. Table 10 describes the MapR files.

Table 10 MapR Files

MapR Files
Description

/etc/sudoers

Allows the mapr user to invoke commands as root

/etc/security/limits.conf

Allows MapR services to increase limits on resources such as memory, file handles, threads and processes, and maximum priority level

/etc/udev/rules.d/99-mapr-disk.rules

Covers permissions and ownership of raw disk devices



Note Before contacting MapR support, gather the cluster logs by running the mapr-support-collect script.


Installing Cluster License

It is essential to register the MapR cluster and obtain a valid license key. Even for the free M3 Community Edition, the license is necessary to enable the enterprise-class features of the MapR packages (NFS, active/passive JobTracker, storage snaphots and mirrors, and other features).

Using web-based MapR Control System to Install the License

Follow these steps to use the web-based MapR Control System (MCS) and install the license:

1. Navigating to the node that is running the MapR Control System (rhel4,rhel17,rhel19,rhel33):

https://<MCS node>:8443

If the system throws an error that the connection is not trustworthy, since the HTTPS certificate is missing, ignore the warning. The first time MapR starts, you must accept the Terms of Use and choose whether to enable the MapR Dial Home service.

2. Log in to the MapR Control System as the administrative user.

3. In MapR Control System, expand the System Settings Views group and click Manage Licenses.

4. The MapR License Management dialog opens. Do the following:

a. Click Add Licenses via Web.

b. Click OK to register the cluster on MapR.com and follow the instructions.

If the cluster is already registered, the license is applied automatically. See Figure 62 and Figure 63.


Note Until a license is applied, the MapR Control System dashboard might show some nodes in the amber "degraded" state.


Figure 62 MapR Control System

Figure 63 MapR License Management dialog

Installing a License From the Command Line

1. Obtain a valid license file from MapR.

2. Copy the license file to a cluster node.

3. Run the following command to add the license:

maprcli license add [ -cluster <name> ] -license <filename> -is_file true

Note This method is useful when it is not possible to connect to the cluster and the internet at the same time.


Restarting M5-enabled Services After License Installation

After successfully installing the license, launch the distributed CLDB services, as well as the JobTracker service and the NFS service.


Note Some of the HA features of the MapR cluster will not start properly until a valid license is installed.


1. Run the following command on any node in the cluster to launch the HA features:

maprcli node services -cldb start -filter [service==cldb]
maprcli node services -jobtracker start -filter [service==jobtracker]
 
   

2. Run the following command to launch the NFS:

maprcli node services -nfs start -filter [service==nfs]

The commands are used to start the respective services on all nodes in the cluster configured with those services. Nodes on which the service is already running is not affected.

Verifying Cluster Status

Follow these steps to verify the cluster status using the web interface:

1. Log in to the MapR Control System.

2. Under the Cluster group click Dashboard.

3. Check the Services pane and make sure each service is running the correct number of instances, according to the cluster plan.

Verify Cluster Status Using the Command Line Interface

1. Log in to a cluster node.

2. Run the following command to list MapR services:

$ maprcli service list
name        state  logpath                        displayname
fileserver  0      /opt/mapr/logs/mfs.log         FileServer
webserver   0      /opt/mapr/logs/adminuiapp.log  WebServer
cldb        0      /opt/mapr/logs/cldb.log        CLDB
hoststats   0      /opt/mapr/logs/hoststats.log   HostStats
$ maprcli license list
$ maprcli disk list -host <name or IP address>

Specifying the Physical Topology of the Cluster

The MapR software can leverage the physical layout of the cluster nodes to maximize cluster availability and minimize the risk of data loss due to hardware failure. You can define the physical topology of the MapR cluster (See Figure 3) using the maprcli commands or graphically using the MCS console.

Topology is specified with the 'maprcli node move' command. Since each group of 16 Cisco UCS nodes is in its own rack, use /data/rack1, /data/rack2, for each rack to define the physical topology. For more information, see the example in Additional Information section.

Adding Volumes

You can use the MapR Control System or the maprcli command to create and mount distinct volumes. This allows more granularity when specifying the policy for subsets of data. See the volume plan created in the section Planning the Cluster.

A common practice is to create separate volumes for users or departments that use the cluster, along with separate volumes for particular projects. See Figure 64.


Note If you do not set up volumes, and store all data in the single volume mounted at /, it can cause problems during data center scale out and administering the data policy.


Figure 64 Adding Volumes

Installing Hadoop Components

The final step in installing the MapR cluster is to install and bring up Hadoop ecosystem components and integrating them with the MapR cluster. The following Hadoop ecosystem components that are installed and integrated in this design solution:

Flume—Installing and using Flume on a MapR cluster

HBase—Installing and using HBase on MapR

Hive—Installing and using Hive on a MapR cluster, and setting up a MySQL metastore

Pig—Installing and using Pig on a MapR cluster

Mahout—Environment variable settings needed to run Mahout on MapR and others

For more information on installation and configuration of Hadoop ecosystem components, see:

http://mapr.com/doc/display/MapR/Installing+Hadoop+Components

Post Installation

After installing the MapR core and the desired Hadoop components, you might need to perform additional steps to ready the cluster for production environment.

The following topics should be reviewed and performed on the cluster as per the requirement:

Setting up the MapR metrics database

Setting up topology

Setting up volumes

Setting up central configuration

Designating NICs for MapR

Setting up MapR NFS

Configuring authentication

Configuring permissions

Setting usage quotas

Configuring alarm notifications

Setting up a client to access the cluster

Working with multiple lusters

For more information on post installation processes of MapR, see:

http://mapr.com/doc/display/MapR/Next+Steps+After+Installation

Additional Information

Services and Ports Quick Reference

Table 11 describes the ports used by the MapR cluster, along with the default port numbers. Table 12 describes the port numbers.

Table 11 Services and Ports

Service
Port

CLDB

7222

CLDB JMX monitor port

7220

CLDB web port

7221

HBase Master

60000

Hive Metastore

9083

JobTracker

9001

JobTracker web

50030

LDAP

389

LDAPS

636

MFS server

5660

NFS

2049

NFS monitor (for HA)

9997

NFS management

9998

NFS VIP service

9997 and 9998

Oozie

11000

Port mapper

111

SMTP

25

SSH

22

TaskTrackerweb

50060

Web UI HTTPS

8443

Web UI HTTP

8080

Zookeeper

5181

Zookeeper follower-to-leader communication

2888

Zookeeper leader election

3888


Table 12 Port Details

Port
Destination
Source
Purpose
Set In File

22

Nodes running any MapR services

Nodes/client running mapr-support-collect.sh or "maprcli disk" API calls

mapr-support-collect.sh leverages SSH over port 22 to connect to a shell environment on cluster nodes in which the mapr-support-dump.sh script will be run

N/A

111

Nodes running MapR NFS Services

Nodes/clients accessing MapRFS via the NFS protocol

RPC Portmap services used to connect to MapRFS via NFSv3

N/A

2049

Nodes running MapR NFS Services

Nodes/clients accessing MapRFS via the NFS protocol

NFSv3 access to MapRFS

N/A

2888

Nodes running ZooKeeper services

Nodes running ZooKeeper services

ZooKeeper Server > Server Communication

/opt/mapr/zookeeper/zookeeper-3.3.2/conf/zoo.cfg

3306

Nodes running the SQL database backend for the Hive metastore

Nodes running Hive metastore services

Used for SQL traffic between the Hive metastore service and its SQL backend server

/opt/mapr/hive/hive-0.7.1/conf/hive-site.xml

3888

Nodes running ZooKeeper services

Nodes running ZooKeeper services

ZooKeeper Server > Server Communication

/opt/mapr/zookeeper/zookeeper-3.3.2/conf/zoo.cfg

5181

Nodes running ZooKeeper services

odes running ZooKeeper services, clients executing ZooKeeper API calls

NZooKeeper API calls

/opt/mapr/zookeeper/zookeeper-3.3.2/conf/zoo.cfg, /opt/mapr/conf/warden.conf, /opt/mapr/conf/cldb.conf, /opt/mapr/hbase/hbase-0.90.4/conf/hbase-site.xml, /opt/mapr/hive/hive-0.7.1/conf/hive-site.xml

5660

Nodes running FileServer services

odes running any MapR services, clients interacting with MapRFS

MapRFS API calls

N/opt/mapr/conf/mfs.conf, /opt/mapr/conf/warden.conf

7221

Nodes running CLDB services

Nodes/clients connecting to the CLDB GUI

CLDB GUI

/opt/mapr/conf/cldb.conf

7222

Nodes running CLDB services

Nodes running any MapR services, clients interacting with MapRFS

MapRFS API calls

/opt/mapr/conf/cldb.conf, /opt/mapr/conf/warden.conf, /opt/mapr/conf/mapr-clusters.conf

8443

Nodes running MapR GUI services

Nodes/clients connecting to the MapR GUI

MapR HTTPS GUI

/opt/mapr/conf/web.conf

9001

Nodes running JobTracker Services

Nodes running TaskTracker services, clients submitting/interacting with Map/Reduce jobs

JobTracker <--> TaskTracker communication, Hadoop API calls that require interaction with JobTracker services

/opt/mapr/conf/warden.conf, /opt/mapr/hadoop/hadoop-0.20.2/conf/mapred-site.xml

9083

Nodes running the Hive metastore services

Nodes/clients performing Hive queries/operations

Used by Hive clients to query/access the Hive metastore

/opt/mapr/hive/hive-0.7.1/conf/hive-site.xml

9997

Nodes running NFS services

Nodes running NFS services

NFS VIP Management

/opt/mapr/conf/nfsserver.conf

9998

Nodes running NFS services

Nodes running NFS services

NFS VIP Management

/opt/mapr/conf/nfsserver.conf

11000

Nodes running Oozie services

Nodes/clients accessing Oozie services

Used by Oozie clients to access the Oozie server

/opt/mapr/oozie/oozie-3.0.0/conf/oozie-env.sh

50030

Nodes running JobTracker Services

Nodes/clients connecting to the JobTracker GUI

JobTracker HTTP GUI

/opt/mapr/conf/warden.conf, /opt/mapr/hadoop/hadoop-0.20.2/conf/mapred-site.xml

50060

Nodes running TaskTracker Services

Nodes/clients connecting to the TaskTracker GUI

TaskTracker HTTP GUI

/opt/mapr/hadoop/hadoop-0.20.2/conf/mapred-site.xml

60000

Nodes running HBase Master services

Nodes running HBase RegionServer services, clients executing HBase API calls

HBase Server > Server communication, HBase API calls

HBase Server > Server communication, HBase API calls

60010

Nodes running HBase Master services

Nodes/clients connecting to the HBase GUI

HBase Master HTTP GUI

/opt/mapr/hbase/hbase-0.90.4/conf/hbase-site.xml

60020

Nodes running HBase RegionServer services

Nodes running HBase RegionServer services, clients executing HBase API calls

HBase Server > Server communication, HBase API calls

N/opt/mapr/hbase/hbase-0.90.4/conf/hbase-site.xml


Avoiding Port Conflicts

To avoid eventual trouble with port conflicts on your MapR clusters, do one of the following:

Remap the ports for the JobTracker, TaskTracker, HBaseMaster, and HBaseRegionServer services to ports below 32768.

Set the ephemeral port range to stop at 50029 by changing the value in the file /proc/sys/net/ipv4/ip_local_port_range.


Note This setting changes the available number of ephemeral ports from the default of 28,233 ports to 17,233.


MapR Repositories and Package Archives

The online repositories and archives for the MapR software are as follows:

RPM repositories for MapR core software

RPM repositories for Hadoop ecosystem tools

Package archive for all releases of Hadoop ecosystem tools

GitHub repositories for source code

Maven repositories for application developers

Other scripts and tools

History of RPM repository URLs

RPM Repositories for MapR Core Software

MapR hosts RPM repositories for installing the MapR core software using Linux package management tools. For every release of the core MapR software, a repository is created for each supported platform.

The platform-specific repositories are hosted at the following location: http://package.mapr.com/releases/<version>/<platform>

RPM Repositories for Hadoop Ecosystem Tools

MapR hosts RPM repositories for installing Hadoop ecosystem tools, such as Cascading, Flume, HBase, HCatalog, Hive, Mahout, Oozie, Pig, Sqoop, and Whirr. The MapR's recommended versions of ecosystem tools that work with the latest version of MapR core software are available in the RPM Repositories.

The platform-specific repositories are hosted at the following location: http://package.mapr.com/releases/ecosystem/<platform>

Package Archive for All Releases of Hadoop Ecosystem Tools

To download any of the MapR's past and present releases of Hadoop ecosystem tools, such as HBase, Hive, and Oozie, see: http://package.mapr.com/releases/ecosystem-all/<platform>


Note This is not a repository, however, the RPM files are archived here, and can be downloaded and installed manually from the mentioned location.


GitHub Repositories for Source Code

MapR releases the source code for Hadoop ecosystem components to GitHub, including all patches MapR has applied to the components. The MapR repositories included in the GitHub are Cascading, Flume, HBase, HCatalog, Hive, Mahout, Oozie, Pig, Sqoop, and Whirr. For more information on the latest source code, see: http://github.com/mapr

Maven Repositories for Application Developers

MapR hosts a Maven repository where application developers can download dependencies on MapR software or Hadoop ecosystem components. Maven artifacts for all releases since March 2013 are available on the MapR website. For more details see the Maven Repository and Artifacts for MapR.

Other Scripts and Tools

For more information on other MapR scripts and tools, see:

http://package.mapr.com/scripts/

http://package.mapr.com/tools/

Updating Physical Topology

1. The physical topology of the MapR cluster node is set using the maprcli command.

maprcli node move -serverids <ids> -topology <topology>

2. To deploy the Cisco UCS, use the following script on each node that allows a simple and easy migration to a new topology.

#!/bin/bash
 
   
if [ -z "${1}" ] ; then
echo "usage: $0 <topology>"
exit 1
fi
 
   
hexid=`cat /opt/mapr/hostid`
myid=` printf "%d" "0x$hexid"
 
   
maprcli node move -serverids $myid -topology $1
 
   

3. Save the above commands into a script file (set_topology) on the administrative node. Distribute it to all the other nodes in the cluster

$ chmod a+x /root/set_topology
$ clush -B -a -c /root/set_topology
 
   

4. After it is distributed, set the topology appropriately for the four racks of nodes as follows:

$ clush -B -w rhel[1-16] '/root/set_topology /data/rack1'
$ clush -B -w rhel[17-32] '/root/set_topology /data/rack2'
$ clush -B -w rhel[33-48] '/root/set_topology /data/rack3'
$ clush -B -w rhel[49-64] '/root/set_topology /data/rack4'

Conclusion

The Cisco Common Platform Architecture for MapR offers a dependable deployment model enterprise Hadoop deployments that can be implemented rapidly and customized to meet the demanding Hadoop deployments.

The configuration detailed in the document can be extended to larger clusters. In a single Cisco UCS domain up to 160 servers (10 racks) can be supported with no additional switching. Each additional rack requires two Cisco Nexus 2232PP 10GigE Fabric Extenders and 16 Cisco UCS C240M3 Rack-Mount Servers. Scaling beyond 10 racks (160 servers) can be implemented by interconnecting multiple Cisco UCS domains using Cisco Nexus 6000/7000 series switches, scalable to thousands of servers and to hundreds of petabytes storage, and managed from a single pane using the Cisco UCS Central.

Bill of Material

This section provides the hardware and software components used in the design setup for deploying the 64-node High Performance Cluster.

Table 13 describes the BOM for the master rack; Table 14 describes the BOM for expansion racks (rack 2 to 4); and Table 15 and Table 16 describe the BOM for the software components

Table 13 Bill of Material for Base Rack

Part Number
Description
Quantity

UCS-EZ-BD-HC

High Capacity Rack

1

UCS-EZ-INFRA-FI96

Cisco UCS 6296 FI w/ 18p LIC, Cables Bundle

2

(included)

N2K-UCS2232PF

Cisco Nexus 2232PP with 16 FET (2 AC PS, 1 FAN (Std Airflow)

2

(included)

UCS-EZ-C240-2640

Cisco UCS C240M3 LFF w/ 2640, 16x16GB, VIC 1225, 2PS (if High Capacity Config)

16 (included)

UCS-EZ-C240-2665

Cisco UCS C240 M3 SFF w/ 2665, 16x16GB, VIC 1225, 2PS (if High Performance Config)

16 (included)

SFP-H10GB-CU3M=

10GBASE-CU SFP+ Cable 3 Meter

28 (included)

RACK-UCS2

Cisco R42610 standard rack w/side panels

1

RP208-30-1P-U-2=

Cisco RP208-30-U-2 Single Phase PDU 20x C13 4x C19 (Country Specific)

2

CON-UCW3-RPDUX

UC PLUS 24X7X4 Cisco RP208-30-U-X Single Phase PDU 2x (Country Specific)

6


Table 14 Bill of Material for Expansion Racks

Part Number
Description
Quantity

N2K-UCS2232PF

Cisco Nexus 2232PP with 16 FET

6

CON-SNTP-UCS2232

SMARTNET 24X7X4 Cisco Nexus 2232PP

6

UCS-EZ-C240-2640

Cisco UCS C240M3 LFF w/ 2640, 16x16GB, VIC 1225, 2PS (if High Capacity Config)

48

UCS-EZ-C240-2665

Cisco UCS C240 M3 SFF w/ 2665, 16x16GB, VIC 1225, 2PS (if High Performance Config)

48

CON-SNTP-C24065EZ

SMARTNET 24X7X4 Cisco UCS C240M3 Server

48

SFP-H10GB-CU1M

10GBASE-CU SFP+ Cable 1 Meter

(server 1-8 to FEX in rack 1-4, 8 x FEX 1 to FI A, 8 x FEX 2 to FI B in rack 1)

80

FP-H10GB-CU2M

10GBASE-CU SFP+ Cable 2 Meter

(server 9-16 to FEX in rack 1-4)

64

SFP-H10GB-CU3M

10GBASE-CU SFP+ Cable 3 Meter

(8 x FEX 1 to FI A, 8 x FEX 2 to FI B in rack 2 and 3

32

SFP-H10GB-CU5M

10GBASE-CU SFP+ Cable 5 Meter

(8 x FEX 1 to FI A, 8 x FEX 3 to FI B in rack 2 and 3

16

RACK-UCS2

Cisco R42610 standard rack w/side panels

4

RP208-30-1P-U-2=

Cisco RP208-30-U-2 Single Phase PDU 20x C13 4x C19 (Country Specific)

8

CON-UCW3-RPDUX

UC PLUS 24X7X4 Cisco RP208-30-U-X Single Phase PDU 2x (Country Specific)

24


Table 15 RedHat Enterprise Linux License

Red Hat Enterprise Linux

RHEL-2S-1G-3A

Red Hat Enterprise Linux

64

CON-ISV1-RH2S1G3A

3 year Support for Red Hat Enterprise Linux

64


Table 16 MapR Software

MapR

MAPR-CSCO-CPA

MapR Enterprise

64