Managing a Hadoop Cluster

This chapter contains the following sections:

Managing a Hadoop Cluster

You can manage an existing cluster.

    Step 1   On the menu bar, choose Solutions > Big Sata > Accounts.
    Step 2   Click on the Big Data Accounts tab.
    Step 3   Select an account and click on View Details.
    Step 4   Click on each tab to view the details.
    Name Description

    Summary

    Displays statistics data report for the selected Big Data Account and the high level report on the cluster and node account.

    Hosts

    See Managing Nodes in a Cluster to know the functionality of this tab.

    Hadoop Clusters

    Displays the list of actions that you can perform on an existing Hadoop cluster.

    • (This tab is only for information) Click Role Topology to view the topology of the nodes.

    • (This tab is only for information) Click View Details to view the inputs for the Hadoop cluster you have created and the virtual network interface configuration.

    • Click Shutdown Cluster that stops all the services and power off all the nodes in the Hadoop cluster.

    • Click Start Cluster that powers on all the nodes in the Hadoop cluster and starts all the services in the Hadoop cluster.

    • Click Restart Cluster to power off and then power on all the nodes in the Hadoop cluster.

    • Click Rebalance to configure the threshold percentage to re-balance Hadoop clusters. For MapR cluster, configure the threshold percentage using CLDB Balancer Disk Paused and CLDB Balancer Disk Max Switches in Nodes.

    • Upgrade Hadoop distributions from the current version, if available.
      Note   

      For a derived account, pssh and clush should be installed and configured for all the nodes in the Hadoop cluster and password less between nodes.

      1. Click Upgrade Cluster.

      2. Choose the Hadoop distribution that you want to upgrade from the current version from the Available Version drop-down list.

      3. Check the Enable HA check box to enable the Hadoop cluster for high availability.

      4. Click Submit.

    • Enable Hadoop cluster high availability.

      1. Click Enable High Availability.

      2. From the Enable High Availability dialog box, choose the Standby Name Node from the Standby Name Node drop-down list.

      3. Check a minimum of three nodes from the Journal Nodes table.

      4. Click Submit.

    • Disable Hadoop cluster high availability.

      1. Click Diaable High Availabuilty.

      2. Form the Disable High Availability dialog box, choose the Name Node from the Select Name Node drop-down list.

      3. Check the Node Name check box from the Select Secondary Name Node table.

        Note   

        Name Node and Secondary Name Node should not be mapped to a single node.

      4. Click Submit.

    • Click Cluster Snapshot to view the snapshot.

    • Click View Reports to view performance and monitoring reports.

    Hadoop Services

    Displays the list of Hadoop services and their status. You can do the following:

    1. Start All Services - Start all Hadoop services depending on their current status.

    2. Stop All Services - Stop all Hadoop services depending on their current status.

    3. Add New Service - Add a new Hadoop service.

    4. Edit Service - Start and stop a particular Hadoop service.

    Hadoop Service Roles

    Displays the list of Hadoop services.

    To add a role to the cluster, do the following in the Add Role dialog box:

    1. From the Hadoop Service Name drop-down list, choose the Hadoop service.

    2. From the Role Type drop-down list, choose the role type.

    3. From the Node Name drop-down list, choose the node name.

    4. From the Role Name drop-down list, choose the role name.

    5. Click Submit.

    To start or stop any role that you have created, do the following in the Start/Stop Role dialog box:

    1. From the Hadoop Service Roles tab, choose the Hadoop service.

    2. Click Start/Stop Role.

    3. Click Submit.

    To delete a role in the cluster, do the following in the Delete Role dialog box:

    1. From the Hadoop Service Roles tab, choose the Hadoop service.

    2. Click Delete.

    3. click Submit.

    More Reports

    Additional reports that you can generate for data usage and CPU utilization.

    View Hadoop Cluster Details

    For each Big Data Account, you can view the details of all the Hadoop clusters that are associated with the Big Data account from the Hadoop Clusters tab. See Managing a Hadoop Cluster

    Name

    Description

    Big Data Account name

    The name of the Big Data account.

    UCS SP Template for Big Data

    The UCS SP Template for Big Data that you used to create service profiles for the servers in the Hadoop cluster.

    Hadoop Cluster Profile Template

    The Hadoop cluster profile template that you used to configure the cluster services.

    Hadoop Cluster Deploy Template

    A unique name that you used for the Hadoop cluster deployment template.

    UCSM Policy Name Prefix

    The UCSM Policy Name prefix.

    Hadoop Cluster Name

    A unique name that you used for the Hadoop cluster.

    Hadoop Node Count

    The number of nodes in the Hadoop cluster.

    Hadoop Node Prefix

    The Host Node prefix for the cluster.

    OS Version

    The operating system that you installed on the servers for the Hadoop cluster.

    Hadoop Distribution

    Hadoop distribution that you used for this cluster.

    Hadoop Distribution Version

    Hadoop distribution version that used for this cluster.

    PXE VLAN ID

    The VLAN ID used for PXE boot of the servers.

    UCS Service Profile Template

    The UCS Service Profile Template that you specified parameters for installing the OS and Hadoop distribution software.

    vNIC: eth0

    Displays IPv4 network information for management interface and the management VLAN ID.

    vNIC: eth1

    Displays IPv4 network information for DATA1 interface and the VLAN ID.

    vNIC: eth2

    Displays IPv4 network information for DATA2 interface and the VLAN ID.

    Viewing a Cluster Snapshot

    A cluster snapshots displays configuration details of a Hadoop cluster, such as hosts, roles, and services. To view the current snapshot of a cluster, do the following:

      Step 1   On the menu bar, choose Solutions > Big Data > Accounts.
      Step 2   Click the Big Data Accounts tab.
      Step 3   Choose the Big Data Account for which you want to view the snapshot and click View Details.
      Step 4   Click the Hadoop Clusters tab.
      Step 5   Choose the Hadoop cluster for which you want to view the snapshot and click Cluster Snapshot.
      Step 6   Click Submit. You can view the snapshot for the selected Hadoop cluster.

      Adding a New Hadoop Service

        Step 1   On the menu bar, choose Solutions > Big Sata > Accounts.
        Step 2   Choose a Cloudera account for which you can add a new Hadoop service.

        Add new service is not supported for MapR and Hortonworks distributions.

        Step 3   Click View Details.
        Step 4   Click the Hadoop Services tab.
        Step 5   Click Add New Service.

        In the Add New Service dialog box, complete the following fields:

        Name

        Description

        Workflow Inputs

        Big Data Account Name

        Choose the Big Data Account (Cloudera).

        Service Type

        Enter the Hadoop service for Cloudera.

        Role Assignment Pairs

        Enter Role Assignment Pairs separated by a comma. For example, RoleType1:hostname1, RoleType2:hostname2

        Dependant Services

        Enter the list of dependant services separated by a comma

        Pre Install Commands

        Enter the list of commands separated by a new line ("\n")

        Post Install Commands

        Enter the list of commands separated by a new line ("\n")

        Step 6   Click Submit.

        Managing Nodes in a Cluster

        You can add, delete, re-commission and decommission nodes in a cluster.

        • Managed Node—Any node that was already a member (managed) in the cluster and deleted, which can be added again in the cluster.

        • Live Node—Any node that has the operating system installed and reachable from the Hadoop cluster.

        • Baremetal Node—Any node that is available and not associated with the Hadoop cluster.

          Step 1   On the menu bar, choose Solutions > Big Sata > Accounts.
          Step 2   Click on the Big Data Accounts tab.
          Step 3   Select an account and click View Details.
          Step 4   Click the Hosts tab to perform the following actions:
          Name Description

          Refresh

          Refreshes the page.

          Favorite

          Adds the page to Favorites.

          Add Managed Node

          Add managed nodes to the Hadoop cluster.

          Add Live Node

          Add live nodes to the Hadoop cluster.

          Add BareMetal Nodes

          Add baremetal nodes to the Hadoop cluster.

          Step 5   Select a host that allows you to perform the following actions:
          Name Description

          View Details

          Displays the summary of the CPU usage, the I/O status of the hosts disks and so on.

          Note   

          If you see a License Status tab, it indicates a licensing issue.

          Delete Node

          Deletes node from the cluster.

          Assign Rack

          Assigns the node to the rack server.

          Recommission Node/ Decommission Node

          Decommissioning or Re-commissioning a node depends on its current status.
          Note   

          When the node is in decommissioned status, it means that all the roles for that node have been withdrawn.

          Delete Node to BareMetal

          The node is removed from the cluster and disassociated from the service profile. The node becomes a BareMetal server.

          Host Mappings

          Lists DNS entries of all the hosts in the Hadoop cluster.

          Delete Node and Delete Node to BareMetal Actions in Cloudera and Hortonworks

          When you perform the Delete Node or Delete Node to BareMetal actions, the UCS CPA Delete Node (a new workflow) is created for Cloudera and Hortonworks. You can also execute this workflow for MapR to perform the delete node operation. If you execute the UCS CPA Delete Node workflow for Cloudera and Hortonworks, this workflow also provides the functionality of Delete Node to BareMetal action based on the Delete Node to BareMetal flag setting to true or false. In addition, the Rollback UCS CPA Node BareMetal workflow is created.

          Adding Managed Nodes to the Hadoop Cluster

          Add managed nodes to the Hadoop cluster.

          This feature allows you to add nodes that are available only from the below URLs, but not the members of the cluster.

          • Cloudera— http://serverip:7180/api/v6/hosts where the serverIP is the IPv4 address of the administration node.

          • Hortonworks—http://serverIP:8080/api/v1/hosts where the serverIP is the IPv4 address of the administration node

            Step 1   On the menu bar, choose Solutions > Big Sata > Accounts.
            Step 2   Click on the Big Data Accounts tab.
            Step 3   Double-click on the Big Data account.
            Step 4   Click the Hosts tab.
            Step 5   Click Add Managed Node.
            Step 6   From the Host Name drop-down list, choose the host name.
            Step 7   Click Submit.

            Adding Live Nodes to the Hadoop Cluster

            Add live nodes to the Hadoop cluster.

              Step 1   On the menu bar, choose Solutions > Big Sata > Accounts.
              Step 2   Click on the Big Data Accounts tab.
              Step 3   Double-click on the Big Data account.
              Step 4   Click the Hosts tab.
              Step 5   Click Add Live Node.
              Step 6   Enter the IPv4 address in the Host Management IPV4 Address field.
              Step 7   Enter the name of the rack server in the Rack Name field.
              Step 8   Enter the password in the (New Node) Password field for that rack server.
              Step 9   Choose the Cluster Manager Version for the Hadoop distribution from the Cluster Management Version drop-down list.
              Step 10   Choose the operating system to be installed on the servers in this cluster from the OS Version drop-down list.
              Step 11   Click Submit.

              Adding BareMetal Nodes to the Hadoop Cluster

              Add baremetal nodes to the Hadoop cluster.

                Step 1   On the menu bar, choose Solutions > Big Sata > Accounts.
                Step 2   Click on the Big Data Accounts tab.
                Step 3   Double-click on the Big Data account.
                Step 4   Click the Hosts tab.
                Step 5   Click Add BareMetal Nodes.
                Step 6   In the Add Baremetal Nodes dialog box, complete the following fields:
                Name Description

                Big Data Account Name field

                The name of the Big Data account.

                UCSM Policy Name Prefix field

                The UCSM Policy Name prefix.

                Hadoop Cluster Name field

                A unique name for the Hadoop cluster.

                Hadoop Node Count field

                The number of nodes in the Hadoop cluster.

                Host Node Prefix field

                The Host Node prefix for the cluster.

                OS Version drop-down list

                Choose the operating system to be installed on the servers in this cluster.

                Hadoop Distribution drop-down list

                Choose the Hadoop distribution to be used for this cluster.

                Hadoop Distribution Version drop-down list

                Choose the Hadoop distribution version.

                Multi UCSM check box

                Check the Multi UCSM check box if you use multiple UCSM accounts.

                The following workflows are created during an Instant/Customized Hadoop Cluster creation:

                • UCS CPA Multi-UCSM Hadoop Cluster WF.

                • Single UCSM Server Configuration WF (This WF will be triggered per UCSM Account. For example, UCSM 120, UCSM121).

                • UCS CPA Node BareMetal (This WF will be triggered per Node).

                UCS Manager Account drop-down list

                Choose the Cisco UCS Manager account for this cluster.

                Organization drop-down list

                Choose the organization in which the servers for this cluster are located.

                Server UUID Pool drop-down list

                Choose the UUID pool to be used for servers in this cluster.

                Note   

                A UUID pool is a collection of SMBIOS UUIDs that are available to be assigned to servers. The first number of digits that constitute the prefix of the UUID are fixed. The remaining digits, the UUID suffix, are variable. A UUID pool ensures that these variable values are unique for each server associated with a service profile which uses that particular pool to avoid conflicts.

                If you use UUID pools in service profiles, you do not have to manually configure the UUID of the server associated with the service profile.

                UUID Prefix drop-down list

                Choose the UUID prefix.

                UUID Pool-From field

                To specify manually, enter the UUID pool to start from.

                UUID Pool-Size field

                To specify manually, enter the UUID pool size.

                PXE VLAN ID field

                Enter the PXE VLAN ID.

                UCSTemplate Name table

                Check the UCS Service Profile Template check box that you want to use and click Submit.

                Server Pool table

                The server pool that you want to use for this cluster.

                The Cisco UCS Manager account and the organization that you choose determine which server pools are displayed in this area.

                Step 7   In the vNIC Template table, review and, if desired, edit the vNIC templates available for the cluster.
                Step 8   If you want to edit a vNIC template, select the row for that template and click Edit.
                Step 9   In the Edit vNIC Template Entry dialog box, complete the following fields and click Submit.
                Name Description

                vNIC Name drop-down list

                This field is for your information only.

                IP Pool drop-down list

                Choose the big data IP pool that you want to use for IP addresses assigned to this vNIC.

                MAC Address Pool drop-down list

                Choose the MAC address pool that you want to use for this cluster.

                First MAC Address field

                To specify manually, enter the MAC address.

                Size field

                To specify manually, enter the size .

                VLAN ID field

                The VLAN ID for this cluster.

                Step 10   Click Submit.

                Service Roles

                For each Hadoop distribution, specific roles are added for all nodes that are newly added by using Add Node BareMetal, Add Managed Node, and Add Live Node options to the Hadoop cluster.

                Service Roles

                Cloudera

                MapR

                Hortonworks

                FileServer

                No

                Yes

                No

                DataNode

                Yes

                No

                Yes

                NodeManager

                Yes

                Yes

                Yes

                Ganglia Monitor

                No

                No

                Yes