Cisco Prime Central 1.4 HA Guide

RHCS Components and Functionality

RHCS is included with the Red Hat Enterprise Linux 6.5 (RHEL 6.5) Advanced Program and has the following components:

• Cluster infrastructure—Basic functions that enable a group of computers (nodes) to work together as a cluster. The RHCS cluster infrastructure performs cluster management, lock management, fencing, and cluster configuration management.
• High availability service management—Failover of services from one cluster node to another when a node becomes inoperative.
• Cluster administration tools—Configuration and management tools for setting up, configuring, and managing a Red Hat cluster, including cluster infrastructure components, high availability service management components, and storage.

RHCS HA Local Redundancy Requirements

We strongly recommend that you use a hardware installation designed to avoid a single point of failure. See Figure 2-2.

Figure 2-2 Local Redundancy Hardware Installation Designed to Avoid a Single Point of Failure

Configuring Hardware for Prime Central RHCS Local Redundancy HA

Figure 2-3 shows the recommended hardware configuration for Prime Central RHCS local redundancy HA.

Figure 2-3 Prime Central Dual-Node Cluster for Local Redundancy HA

Configure the external storage so all disks and logical unit numbers (LUNs) are accessible from both servers in the cluster. The disk and LUN configuration depends on the storage type:

• If you are using JBOD disks, provide enough physical disks to create the volumes listed in Table 2-2 to satisfy the Oracle performance requirements.
• If you are using storage that supports hardware RAID, divide the physical disks into LUNs so that the volumes listed in Table 2-2 can be created and configured to satisfy the Oracle performance requirements and can be protected with RAID5, RAID1, or RAID10. The Oracle volumes can be created on a single LUN.

Configuring the OS for Prime Central Local Redundancy HA Managed by RHCS

1. Install the OS and all recommended patches on both servers in the cluster. Installations on both servers must be identical.

2. Verify that access is available to all external disks.

3. Create the internal disk partitions listed in Table 2-3 . Placing the individual directories in separate partitions is recommended, but not required.

4. Complete the internal disk partitions for both servers.

5. Keep the nodes synchronized:

Installing the Red Hat Cluster Service

Using the procedures in the Red Hat user documentation, install RHEL 5.8/6.5 with the RHCS. When you set up the RHCS disk groups and volumes, keep the following in mind:

• All of the shared storage should have an ext3 file system installed and a label set.
• Shared storage must be accessible from all cluster nodes.

Verifying the Prime Central RHCS HA Installation

Before You Begin

• Ensure that both PC and FM are installed on different VMs.
• Verify that your system meets all the hardware and software requirements in “Installation Requirements” in the Cisco Prime Central 1.4 Quick Start Guide.
• Set up two nodes that have:

– Static IP addresses and hostnames that are registered correctly in the DNS.

– The same root password, which cannot contain a percent sign (%, ^, $, *).

• Set up one virtual IP address and hostname that are registered correctly in the DNS. In this section, the virtual IP address is 192.168.1.130.
• Set up shared storage that is compatible with RHEL device-mapper (DM) multipath and cluster fencing.
• Install RHEL 6.5 on both nodes.
• If the default folder location and names are changed, look for the section “Require Manual Definition” to make corresponding changes in the below mentioned files:

– /root/ha-stuff/pc/PrimeCentral.sh

– /root/ha-stuff/pc/UninstallPrimeCentral.sh

– /usr/local/bin/pc.sh

– /usr/local/bin/createUserGroup.sh

Prime Central HA Setup on RHEL 6.5

Step 1 Create three VMs with the configuration specified in the Prime Central 1.4 quick start guide.

In this setup:

• Two VMs act as cluster nodes.
• A third VM runs luci, the web-based high-availability administration application. By running luci on a third system, you can still manage the cluster from another system if either the primary or standby node goes down.
• Shared network storage is required.

Step 2 Install RHEL 6.5 on all three VMs with the Desktop option selected.

Ensure that:

• All three VMs can ping each other via IP address and hostname.
• Both the active and standby cluster nodes have a 200 GB shared repository, which is located in the /dev/sdb directory.

Step 3 Disable the firewall on all three VMs:

Step 4 On both cluster nodes, switch the network daemons:

Step 5 Disable Security-Enhanced Linux (SELinux) on all three VMs:

Step 6 Reboot the VMs.

Adding Clustering to the Installed Red Hat Server (Prime Central)

To add clustering to the newly installed Red Hat server, complete Steps 1 through 6 of the following procedure on both cluster nodes and the VM running luci. Then complete Steps 7 through 12 on just the cluster nodes.

Step 1 Create local directories named /rhel and /cdrom-6.5.

Step 2 Copy the.iso file that was used for the virtual machine (VM) RHCS installation to the /rhel directory.

Step 3 Mount the /rhel.iso file to /cdrom:

Note To permanently mount the drive, update the /etc/fstab file. See http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/4/html/Introduction_To_System_Administration/s2-storage-mount-fstab.html.

Step 4 Create a file named /etc/yum.repos.d/local.repo. Use UNIX format and be sure there are no spaces before lines.

Step 5 Save the newly created file in local.repo, as follows:

Step 6 Add the information for both cluster nodes and the VM running luci to the /etc/hosts file; for example:

Step 7 Generate a Secure Shell (SSH) key for the root user:

Step 8 (On the first node only) Share the node’s public key with the other node so that dynamically creating a secure shell between the nodes does not prompt for a password:

Step 9 Verify that the.ssh directory has 700 permission and the.ssh/id_rsa file has 600 permission:

Step 10 Verify that your SSH is working without an authentication or password prompt:

Caution The Prime Central service will not start if SSH prompts for authentication or a password. Be sure to complete all of the following substeps.

---

a. On node prime-ha-node1.cisco.com, enter:

b. On node prime-ha-node2.cisco.com, enter:

c. If you are prompted for a password, check the permissions of all folders and files that you modified in the preceding steps.

d. If you are prompted to continue connecting, enter yes. (The prompt should appear only the first time you use SSH to connect to the node.)

Step 11 Verify that the virtual IP address is accessible from outside the cluster’s subnet:

Step 12 On a computer outside the cluster’s subnet, ping the virtual IP address:

If you do not get a valid response, determine which part of the OS or network setup is blocking.

Adding Shared Partitions to Prime Central

To add shared partitions, complete the following steps in parallel on both nodes, except where noted:

Note The examples provided use device mapping names such as mpatha and mpathap1; yours may be different.

Step 1 Install and set up multipath:

Step 2 Check for available disks:

In the output, note mpath2, which is the multipath virtual device or disk that you will use later as shared storage.

Note If you previously set up a partition on the disk, you might see output such as mpath2p. You must delete that partition before proceeding to the next step.

Step 3 (On the first node only) Create a shared partition:

Step 4 Reboot both nodes.

Step 5 Check for new partitions:

Step 6 (On the first node only) Format the new shared partition:

Step 7 Create target locations on both nodes:

Step 8 Verify that both nodes can mount and unmount the shared storage:

a. On the first node, mount the shared storage and save a file that contains only the value 1 to the shared storage. The test.txt file should exist in the list of contents of /opt/pc:

b. On the second node, mount the shared storage and verify that the test.txt file exists and contains the value 1 :

If you cannot mount or unmount the shared storage, or if the test.txt file does not exist when you mount it to the second node, your multipath is not set up correctly.

Accessing and Distributing the Prime Central.tar File

Step 1 Insert the Cisco Prime Central 1.4 USB, navigate to the High Availability/RHCS VMWare ESXi Local HA/Prime Central folder, and locate the primecentral_v1.4_ha_vm.tar.gz file.

Step 2 Use SSH to connect to the first node.

Step 3 Copy the primecentral_v1.4_ha_vm.tar.gz file to the first node.

Step 4 Back up the following directories on both nodes:

• /root/ha-stuff/pc
• /usr/local/bin

Step 5 Distribute the file:

Installing Prime Central in an HA Setup

Step 1 Mount the shared partitions on first node:

Step 2 Navigate to the Base Application folder and copy primecentral*.bin and all Oracle zip files to the /root/ha-stuff/pc folder:

Step 3 Add a virtual IP cluster service address for the Prime Central service:

Make sure that virtual IP address is reachable.

Step 4 Update the install.properties file (located at /root/ha-stuff/pc) and verify that all required properties have values. Refer to the comments at the top of the install.properties file for details.

Step 5 Install Prime Central:

Step 6 In another terminal window, check the installation process:

Step 7 After the installation completes, start Prime Central:

Step 8 Verify that Prime Central is running correctly; then, stop it:

Step 9 Remove the virtual IP addresses:

Step 10 Unmount the shared partitions:

Setting Up the Prime Central Cluster Service

To set up the Prime Central cluster service, complete the following steps in parallel on all the nodes including luci, except where noted:

Step 1 Install the ricci RPMs:

Step 2 Start the ricci daemon and configure it to start on boot:

Step 3 Set the ricci user password:

You will need to enter this password later.

At this point, both cluster nodes should be running the ricci servers and be ready to be managed by the cluster web user interface (luci).

Installing the Cluster Web User Interface (luci)

Complete the following procedure to install and configure luci on the third VM.

Step 1 Install the luci RPMs:

Step 2 Start the luci daemon and configure it to start on boot:

Step 3 Launch the URL listed in the last line of the system output (https://prime-ha-luci.cisco.com:8084, in this example) and log in as the root user when prompted.

Step 4 Verify that the RHEL DVD is mounted on both cluster nodes.

Step 5 Select Manage Clusters > Create and then specify a name for the cluster (for example, PrimePCCluster).

Step 6 Enter the name (fully-qualified domain name or name specified in the /etc/hosts directory) and password (the password for the user ricci) for the first cluster node.

Step 7 Click Add Another Node and enter the same information described in Step 6 for the second cluster node.

Step 8 Do the following, and then click Create Cluster:

• Use the Same Password for All Nodes: Leave this check box unchecked.
• Download Packages: Select this radio button.
• Reboot Nodes Before Joining Cluster: Select this check box.
• Enable Shared Storage Support: Leave this unchecked.

Assuming that the nodes can be contacted, luci will set up each cluster node and add each node to the cluster.

After each node has been set up, the High Availability Management screen opens.

Step 9 Create a failover domain:

a. Select the Failover Domains tab and then click Add.

b. Do the following:

– Name: Enter a name for the failover domain.

– Prioritized: Select this check box.

– Restricted: Select this check box.

– No Failback: Select this check box.

– Member: Select the check box for each node.

– Priority. In the Priority column, enter 1 for node1 and 2 for node2.

c. Click Create.

Step 10 Configure fence devices appropriate for the hardware you have, adding a fence device and instance for each node.

The settings you need to configure depend on your particular hardware and software configuration. Refer to the Cluster Administration Guide for help with configuring fence devices.

Complete the following steps to configure VMware fencing:

a. Retrieve the UUID for prime-pc-node1.cisco.com:

This returns: VM-Name,UUID

Keep in mind that VM-Name is the name given to the VM during VM creation, not its hostname.

b. Repeat the previous step for prime-pc-node2.cisco.com.

c. In luci, select the Fence Devices tab and then click Add.

d. Do the following:

– Fence type: Select VMWare Fencing (SOAP Interface).

– Name: Enter a descriptive name for the fence device.

– IP Address or Hostname: Enter the appropriate VCenter IP address or hostname.

– IP Port (optional): Leave blank.

– Login: Enter the VCenter username.

– Password: Enter the VCenter password.

Leave the rest of the fields as is or blank.

e. Click Submit.

f. Select the Nodes tab and then select your first node by clicking its name.

g. At the bottom of the node’s window, click Add Fence Method.

h. Enter a name and then click Submit. Now select Add Fence Instance that appears inside the method box and fill it out as described below:

i. Select the fencing device you configured in Step 10d and do the following:

– VM Name: Leave blank.

– VM UUID: Enter the UUID you retrieved in Step 10a (for example, 4209838d-1e91-104c-9f5c-1181ad87ddd8).

– Use SSL: Select this check box. Fencing will not work unless you do so.

j. Repeat Steps 10f through 10i for your second node, as well.

Note We recommend that you check the bottom of this window for each node and verify that all of the required daemons are running before you test fencing.

Setting Up Shared Resources

You will next create the cluster service (named PrimePCService in this example). To begin, open the High Availability Management page (luci) in a browser. From there, you will:

• Identify the virtual IP address for this cluster
• Identify the PrimeService service and add the virtual IP address to this service

At several points, you will run commands from the cluster nodes (ricci) to verify that the configuration you set in luci is working.

Identifying the Cluster Service's Virtual IP Address

From luci, do the following to identify the cluster service's virtual IP address:

Step 1 Select the cluster name (for example, PrimePCCluster).

Step 2 Select the Resources tab and then click Add.

Step 3 Select IP Address.

Step 4 Enter the following information:

• IP Address: Specify a valid IP address, which will be used to access the service from a web browser.
• Monitor Link: Select this check box.

Step 5 Click Submit.

Creating the Prime Service

From luci, with the cluster still selected, add a new service and associate it with the IP address.

Step 1 Select the Service Groups tab and then click Add.

Step 2 Enter the following information for the new service group:

• Service name: Name of the service group (for example, PrimePCService)
• Automatically start this service: Select this check box.
• Failover Domain: Select the PCFailOver you created while installing luci.
• Recovery Policy: Select Relocate.

Step 3 Click Add Resource.

Step 4 Select the IP address you added earlier and then click Submit.

Step 5 Wait 5 minutes and then reboot both cluster nodes.

After the VMs come back online, the PrimePCService service should up and running.

Verifying the PrimeService Service Can Be Reached

From each of the cluster nodes, check the status of the PrimePCService services you created. Then, from any machine on the network, verify that you can reach the cluster from the IP address you assigned.

Step 1 From a shell on each of the cluster nodes, verify that the service group is running by entering the following commands (in bold):

Step 2 From a shell anywhere that can reach the cluster over the network, verify that you can ping the IP address associated with the IP address resource you created previously (see Creating the Prime Service):

Note To end the ping, enter Ctrl + C.

Step 3 Relocate the service:

Step 4 Verify that the service has been relocated successfully to the other node.

Adding Mount Points and Setting Up File Systems

In the following procedure, you will create mount points on the two nodes. After the mount points have been set up, you can then configure the resources necessary to mount file systems to the proper locations.

Step 1 Create a file system resource:

a. From luci, select the Resources tab and then click Add.

b. From the Add Resource to Cluster window, select Filesystem from the dropdown list and then do the following:

– Name: Enter the name of the file system resource.

– Filesystem type: Select the ext4 option.

– Mount point: Enter the mount point path—/opt/pc

– Device, FS label, or UUID: Enter the appropriate path—in this example, /dev/mapper/mpathap1

– Leave the Mount options and Filesystem ID (optional) field blank.

– Select the Force Unmount check box.

– Select the Reboot host node if unmount fails check box.

c. Click Submit.

Step 2 Add the file system resource to the PrimePCService service group:

a. From luci, select the Service Groups tab.

b. In the Name column, select the PrimePCService service group name.

c. At the bottom of the page, click Add Resource.

d. Click Select a Resource Type.

e. The filesystem resources you added should appear in the list.

f. From the list, click the filesystem entries you added in the previous step.

g. Click Submit.

Step 3 Log in to the node currently running the PrimePCService service.

Step 4 Verify that the logical volume ext4 filesystems are mounted with read/write privileges on the node running the PCService service:

Step 5 Verify that the logical volume ext4 filesystems are not mounted on the node which is not running the PCService service:

Adding Scripts to the Prime Central Cluster Service

Step 1 Add the pc.sh script resource:

a. From luci, select the Resources tab and then click Add.

b. From the Add Resource to Cluster window, select Script from the dropdown list and then enter the following information:

– Name: Name of the script

– Full Path to Script File: Directory in which the script will reside—/usr/local/bin/pc.sh

Step 2 Add script resources to the PrimePCService service group:

a. From luci, select the Service Groups tab.

b. In the Name column, select the PrimePCService service group name.

c. At the bottom of the page, click Add Resource.

d. Click Select a Resource Type.

The script resource you added should appear in the list.

e. Click the script entries from the list you added in the previous step.

f. Click Submit.

Checking the Cluster Services

On both nodes, check the status of the cluster:

The output is similar to the following:

Next Steps

Complete the following steps on both nodes, except where noted:

Step 1 Verify that the required ports are open. For a list of ports that Prime Central requires, see “Prime Central Protocols and Ports” in the Cisco Prime Central 1.4 Quick Start Guide.

Step 2 Enable the firewall:

Before You Begin

• Ensure that both PC and FM are installed on different VMs.
• Verify that your system meets all the hardware and software requirements in “Installation Requirements” in the Cisco Prime Central 1.4 Quick Start Guide.
• If you changed the default installation folder (/opt/fm/primeusr/faultmgmt), make the equivalent changes in the following files (look for the section titled “Require manual definition” in each file):

– /usr/local/bin/fm.sh

Fault Management HA Setup on RHEL 6.5

Step 1 Create two VMs with the configuration specified in the Prime Central 1.4 quick start guide.

In this setup:

• Two VMs act as cluster nodes.
• You will create two VMs—one with the virtual IP address (virtual hostname) and the other with the node2 IP address (node2 hostname). After the fault managment installation is complete, you will then change the virtual IP address to node 1’s IP address (node1 hostname) and then configure the cluster services.
• Shared network storage is required.

Step 2 Install RHEL 6.5 on the cluster nodes with the Desktop option selected.

Ensure that:

• The cluster nodes can ping each other via IP address and hostname.
• Both the active and standby cluster nodes have a 200 GB shared repository, which is located in the /dev/sdb directory.

Step 3 Disable the firewall on the VMs:

Step 4 On both cluster nodes, switch the network daemons:

Step 5 Disable Security-Enhanced Linux (SELinux) on the VMs:

Step 6 Reboot the VMs.

Adding Clustering to the Installed Red Hat Server (Fault Management)

To install RHEL 6.5, complete the following steps in parallel on both nodes, except where noted:

Step 1 Create local directories named /rhel and /cdrom-6.5.

Step 2 Copy the.iso file that was used for the node installation to the /rhel directory.

Step 3 Mount the /rhel.iso file to /cdrom-6.5:

Note To permanently mount the drive, update the /etc/fstab file. See http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/4/html/Introduction_To_System_Administration/s2-storage-mount-fstab.html.

Step 4 Create a file named /etc/yum.repos.d/local.repo. Use UNIX format and be sure there are no spaces before lines.

Step 5 Save the newly created file in local.repo, as follows:

Step 6 Keep the nodes synchronized:

Step 7 Edit the /etc/hosts file to add the node information; for example:

Step 8 Generate an SSH key for the root user:

Step 9 (On the first node only) Share the node’s public key with the other node so that dynamically creating a secure shell between the nodes does not prompt for a password:

Step 10 Verify that the.ssh directory has 700 permission and the.ssh/id_rsa file has 600 permission:

Step 11 Verify that your SSH is working without an authentication or password prompt:

Caution The Fault Management service will not start if SSH prompts for authentication or a password. Be sure to complete all of the following substeps.

---

a. On node fm-ha-node1.cisco.com, enter:

b. On node fm-ha-node2.cisco.com, enter:

c. If you are prompted for a password, check the permissions of all folders and files that you modified in the preceding steps.

d. If you are prompted to continue connecting, enter yes. (The prompt should appear only the first time you use SSH to connect to the node.)

Configuring Multipath

To configure multipath, complete the following steps in parallel on both nodes, except where noted:

Note The examples provided use device mapping names such as mpatha and mpathap1; yours may be different.

Step 1 Install multipath:

Step 2 Configure and start the services:

Step 3 Check for available disks. The names of the multipath disks must be identical on both nodes:

In the output, note mpatha, which is the multipath virtual device or disk that you will use later as shared storage.

Adding Shared Partitions

To add shared partitions, complete the following steps in parallel on both nodes, except where noted:

Note The examples provided use device mapping names such as mpatha and mpathap1; yours may be different.

Step 1 (On the first node only) Create a 100-GB, shared partition:

Step 2 Reboot both nodes.

Step 3 Check for new partitions:

Step 4 (On the first node only) Format the new shared partition:

Step 5 Create target locations on both nodes:

Step 6 Verify that both nodes can mount and unmount the shared storage:

a. On the first node, mount the shared storage and save a file that contains only the value 1 to the shared storage. The test.txt file should exist in the list of contents of /opt/fm:

b. On the second node, mount the shared storage and verify that the test.txt file exists and contains the value 1 :

If you cannot mount or unmount the shared storage, or if the test.txt file does not exist when you mount it to the second node, your multipath is not set up correctly.

Installing Prime Central Fault Management

Step 1 Verify that both fault management VMs can ping each other via the hostname and IP address. Also verify that the both VMs can ping the PC cluster nodes and luci node via the hostname and IP address.

Step 2 Verify that both VMs can ssh to each other without using the configured password.

Step 3 Insert the Cisco Prime Central 1.4 USB, navigate to the High Availability/RHCS VMWare ESXi Local HA/Fault Management folder, and locate the primefm_v1.4_ha_node.tar.gz file.

Step 4 Use SSH to connect to the virtual IP node.

Step 5 Copy the primefm_v1.4_ha_node.tar.gz file to the virtual IP node.

Step 6 Back up the /root/ha-stuff/fm and /usr/local/bin directories.

Step 7 Mount the partition to the /opt/fm directory on the virtual IP node:

Step 8 Distribute the file:

Step 9 Navigate to the top-level Fault Management folder and copy the FM1.4Build.tar.gz file to the /root/ha-stuff/fm directory:

Step 10 Specify all necessary properties in the fm_install.properties file.

Step 11 Change HOSTNAME in /etc/sysconfig/network to match the value of PRIMEFM_SERVER_HOSTNAME in fm_install.properties file.

Note The hostname must be a Fully Qualified Distinguished Name (FQDN).
For example: prime-fm-service.cisco.com

Step 12 Change active hostname to the host name used in step 11 (the virtual FQDN hostname) using the command:

Step 13 Install Fault Management:

Step 14 After the installation is complete, make sure that FM can be started properly.

Step 15 Freeze the Prime Central cluster and then restart the integration layer so that Prime Central displays Fault Management’s status as UP.

Step 16 Stop Fault Management:

Step 17 Change the IP address from virtual IP (hostname) to node 1’s IP address (hostname).
Revert to use the original hostname, if there are any changes in /etc/sysconfig/network file (in Step 11).

Step 18 Reboot the VM.

Setting Up the Fault Management Cluster Service

On both cluster nodes:

Step 1 Install the ricci RPMs:

Step 2 Start the ricci daemon and configure it to start on boot:

Step 3 Set the ricci user password:

You will need to enter this password later.

At this point, both cluster nodes should be running the ricci servers and be ready to be managed by the cluster web user interface (luci).

Configuring the Cluster Web User Interface (luci)

Complete the following procedure to configure luci on the third VM for use with Prime Central Fault Management.

Step 1 Launch the URL listed in the last line of the system output (https://prime-ha-luci.cisco.com:8084, in this example) and log in as the root user when prompted.

Step 2 Verify that the RHEL DVD is mounted on both cluster nodes.

Step 3 Select Manage Clusters > Create and then specify a name for the cluster (for example, PrimeFMCluster).

Step 4 Enter the name (fully-qualified domain name or name specified in the /etc/hosts directory) and password (the password for the user ricci) for the first cluster node.

Step 5 Click Add Another Node and enter the same information described in Step 4 for the second cluster node.

Step 6 Do the following, and then click Create Cluster:

• Use the Same Password for All Nodes: Leave this check box unchecked.
• Download Packages: Select this radio button.
• Reboot Nodes Before Joining Cluster: Select this check box.
• Enable Shared Storage Support: Leave this unchecked.

Assuming that the nodes can be contacted, luci will set up each cluster node and add each node to the cluster.

After each node has been set up, the High Availability Management screen opens.

Step 7 Create a failover domain:

a. Select the Failover Domains tab and then click Add.

b. Do the following:

– Name: Enter a name for the failover domain.

– Prioritized: Select this check box.

– Restricted: Select this check box.

– No Failback: Select this check box.

– Member: Select the check box for each node.

– Priority. In the Priority column, enter 1 for node1 and 2 for node2.

c. Click Create.

Step 8 Configure fence devices appropriate for the hardware you have, adding a fence device and instance for each node.

The settings you need to configure depend on your particular hardware and software configuration. Refer to the Cluster Administration Guide for help with configuring fence devices.

Complete the following steps to configure VMware fencing:

a. Retrieve the UUID for prime-ha-node1.cisco.com:

This returns: VM-Name,UUID

Keep in mind that VM-Name is the name given to the VM during VM creation, not its hostname.

b. Repeat the previous step for prime-ha-node2.cisco.com.

c. In luci, select the Fence Devices tab and then click Add.

d. Do the following:

– Fence type: Select VMWare Fencing (SOAP Interface).

– Name: Enter a descriptive name for the fence device.

– IP Address or Hostname: Enter the appropriate VCenter IP address or hostname.

– IP Port (optional): Leave blank.

– Login: Enter the VCenter username.

– Password: Enter the VCenter password.

Leave the rest of the fields as is or blank.

e. Click Submit.

f. Select the Nodes tab and then select your first node by clicking its name.

g. At the bottom of the node’s window, click Add Fence Method.

h. Enter a name and then click Submit. Now select Add Fence Instance that appears inside the method box and fill it out as described below:

i. Select the fencing device you configured in Step 8d and do the following:

– VM Name: Leave blank.

– VM UUID: Enter the UUID you retrieved in Step 8a (for example, 4209838d-1e91-104c-9f5c-1181ad87ddd8).

– Use SSL: Select this check box. Fencing will not work unless you do so.

j. Repeat Steps 8f through 8i for your second node.

Note We recommend that you check the bottom of this window for each node and verify that all of the required daemons are running before you test fencing.

Step 9 Set up the network configuration:

a. Select the Configure tab.

b. Select the Network tab.

c. Select the UDP Unicast (UDPU) radio button and then click Apply.

Setting Up Shared Resources

You will next create the cluster service (named PrimeService in this example). To begin, open the High Availability Management page (luci) in a browser. From there, you will:

• Identify the virtual IP address for this cluster
• Identify the PrimeService service and add the virtual IP address to this service

At several points, you will run commands from the cluster nodes (ricci) to verify that the configuration you set in luci is working.

Identifying the Cluster Service's Virtual IP Address

From luci, do the following to identify the cluster service's virtual IP address:

Step 1 Select the cluster name (for example, PrimeCluster).

Step 2 Select the Resources tab and then click Add.

Step 3 Select IP Address.

Step 4 Enter the following information:

• IP Address: Specify a valid IP address, which will be used to access the service from a web browser.
• Monitor Link: Select this check box.

Step 5 Click Submit.

Creating the Prime Fault Management Service

From luci, with the cluster still selected, add a new service and associate it with the IP address.

Step 1 Select the Service Groups tab and then click Add.

Step 2 Enter the following information for the new service group:

• Service name: Name of the service group (for example, PrimeFMService)
• Automatically start this service: Select this check box.
• Failover Domain: Select the PCFailOver you created while installing luci.
• Recovery Policy: Select Relocate.

Step 3 Click Add Resource.

Step 4 Select the IP address you added earlier and then click Submit.

Step 5 Wait 5 minutes and then reboot both cluster nodes.

After the VMs come back online, the PrimeFMService service should up and running.

Verifying the PrimeService Service Can Be Reached

From each of the cluster nodes, check the status of the PrimeService services you created. Then, from any machine on the network, verify that you can reach the cluster from the IP address you assigned.

Step 1 From a shell on each of the cluster nodes, verify that the service group is running by entering the following commands (in bold):

Step 2 From a shell anywhere that can reach the cluster over the network, verify that you can ping the IP address associated with the IP address resource you created previously (see Creating the Prime Service):

Note To end the ping, enter Ctrl + C.

Step 3 Relocate the service:

Step 4 Verify that the service has been relocated successfully to the other node.

Adding Mount Points and Setting Up File Systems

In the following procedure, you will create mount points on the two nodes. After the mount points have been set up, you can then configure the resources necessary to mount file systems to the proper locations.

Step 1 Create a file system resource:

a. From luci, select the Resources tab and then click Add.

b. From the Add Resource to Cluster window, select Filesystem from the dropdown list and then do the following:

– Name: Enter the name of the file system resource.

– Filesystem type: Select the ext4 option.

– Mount point: Enter the mount point path—/opt/fm

– Device, FS label, or UUID: Enter the appropriate path—in this example, /dev/mapper/mpath2p1

– Leave the Mount options and Filesystem ID (optional) field blank.

– Select the Force Unmount check box.

– Select the Reboot host node if unmount fails check box.

c. Click Submit.

Step 2 Add the file system resource to the PrimeService service group:

a. From luci, select the Service Groups tab.

b. In the Name column, select the PrimeFMService service group name.

c. At the bottom of the page, click Add Resource.

d. Click Select a Resource Type.

The filesystem resources you added should appear in the list.

e. From the list, click the filesystem entries you added in the previous step.

f. Click Submit.

Step 3 Log in to the node currently running the PrimeFMService service.

Step 4 Verify that the logical volume ext4 filesystems are mounted with read/write privileges on the node running the PCService service:

Step 5 Verify that the logical volume ext4 filesystems are not mounted on the node which is not running the PCService service:

Adding Scripts to the Prime Central Cluster Service

Step 1 Add the fm.sh script resource:

a. From luci, select the Resources tab and then click Add.

b. From the Add Resource to Cluster window, select Script from the dropdown list and then enter the following information:

– Name: Name of the script

– Full Path to Script File: Directory in which the script will reside—/usr/local/bin/fm.sh

Step 2 Add script resources to the PrimeService service group:

a. From luci, select the Service Groups tab.

b. In the Name column, select the PrimeFMService service group name.

c. At the bottom of the page, click Add Resource.

d. Click Select a Resource Type.

The script resource you added should appear in the list.

e. Click the script entries from the list you added in the previous step.

f. Click Submit.

Checking the Cluster Services

Step 1 Review the cluster log file in /var/log/messages.

Step 2 After the Fault Management service is running in an HA cluster, you cannot restart its components (such as Netcool/Impact, OMNIbus, and Tivoli Common Reporting [TCR]) without first freezing the cluster. After you restart the component, you can unfreeze the cluster.

To restart a Fault Management component:

a. On the active Fault Management node, enter:

b. Use SSH to connect to the Fault Management virtual machine and enter:

c. Use SSH to connect to the active Fault Management node and enter: