Mobility Unified Reporting System Clustering Support for High Availability

This chapter describes how to install and configure the Veritas cluster for high availability in MUR.

This chapter describes the following topics:

System and Hardware Recommendations for HA Deployment

The MUR can be configured in HA mode with asymmetric or active/passive configuration. The following requirements must be met in order to ensure the HA deployment in the MUR system:

  • 2 UCS 460 M1 servers for cluster nodes
  • Up to 4 Internal disks (depending upon the fault tolerance for boot disk/OS)
  • Qlogic QLE2462 4Gb dual port FC HBA / Qlogic QLE2560 8 Gbps FC 1 port
  • FC cables
  • External Storage with multiple disks (EMC/Sun Storage)
  • Veritas (VxFS) file system

The hardware setup for the Veritas Cluster Server (VCS) consists of two cluster nodes connected with an external shared storage. UCS 460 supports multiple PCI slots for attaching the FC HBA cards. Both the cluster nodes must be connected to the external storage. The cluster nodes must be installed with the Cisco MITG RHEL operating system, Veritas Storage Foundation (Veritas Volume Manager and Veritas File System), and VCS (for High Availability).

The Veritas Volume Manager (VxVM) is used to create a single Disk Group (DG) containing multiple disks. A separate group of disks is allocated for postgres data directory to achieve good performance.

The RAID recommendations are the same as that of standalone installation i.e. RAID-0 for MUR application and RAID-5 for database.

Separate disk/LUN from the shared storage is required for I/O fencing. I/O fencing is part of the VCS administration. It is assumed that I/O fencing is already configured on the Veritas Cluster setup before the MUR application is installed for HA.

Configuring the External Storage Disk on UCS for HA Deployment

This section describes the procedure to configure an external storage disk on a UCS server for HA deployment.

It is recommended to create separate disk groups for:
  • MUR Application
  • Postgres data directory
  • MUR Archive (This can be either part of MUR Application disk group OR created separately)

The number of disks in each group depends on the total throughput required and the size of disks in the external storage. Please contact your system administrator for setting up the external storage to make the required number of disks (Logical Disk Units (LUNs)) accessible from both the cluster nodes. When the external storage disks are made accessible, you can see them connected using multipath command:

$ multipath
-l

mpath6 (36006016069902d008892bc0dec14e111)
dm-7 DGC,RAID 5

[size=300G][features=1
queue_if_no_path][hwhandler=1 emc][rw]

\_ round-robin
0 [prio=0][active]

\_ 8:0:0:1
sdi 8:128 [active][undef]

 \_ 8:0:1:1
sdk 8:160 [active][undef]

mpath5 (36006016069902d00563e8a20ec14e111)
dm-6 DGC,RAID 5

[size=300G][features=1
queue_if_no_path][hwhandler=1 emc][rw]

\_ round-robin
0 [prio=0][active]

\_ 8:0:0:0
sdh 8:112 [active][undef]

\_ 8:0:1:0
sdj 8:144 [active][undef]

To configure an external storage disk into separate DGs:

  1. Rebuild the disk lists with the new disks detected by the kernel using the following command: 
    $ vxdctl initdmp

    $ vxdctl enable
    To see the status of the new disk, use the following command:
    $ vxdisk -o
alldgs list

    DEVICE             TYPE        DISK       GROUP         STATUS

    disk_0            auto:none     -           -          online
invalid

    disk_1            auto:none     -           -          online
invalid

    disk_2            auto:none     -           -          online
invalid

    disk_3            auto:none     -           -          online
invalid

    emc_clariion0_28  auto          -           -            error

    emc_clariion0_29  auto          -           -            error
  2. To setup the disks, use the following commands: 
    $ /etc/vx/bin/vxdisksetup
-i emc_clariion0_28

    $ /etc/vx/bin/vxdisksetup
-i emc_clariion0_29
    To see the status of the new disk, use the command:
    $ vxdisk -o
alldgs list

    DEVICE             TYPE        DISK       GROUP         STATUS

    disk_0            auto:none     -           -          online
invalid

    disk_1            auto:none     -           -          online
invalid

    disk_2            auto:none     -           -          online
invalid

    disk_3            auto:none     -           -          online
invalid

    emc_clariion0_28  auto:cdsdisk  -           -            online

    emc_clariion0_29  auto:cdsdisk  -           -            online
  3. With the newly initialized disks, create two separate DGs for apps (MUR) and db (postgres data directory) partitions using the following command: 
    vxdg init apps_dg apps_dg01=emc_clariion0_28

    $ vxdg init
db_dg db_dg01=emc_clariion0_29

    IMPORTANT:

    You can specify more disks (using Step 2) and add the disk to disk groups.

    After adding the disk into respective DGs, you can verify them using the following command:
    $ vxdisk -o
alldgs list

    DEVICE             TYPE          DISK        GROUP         STATUS

    disk_0            auto:none       -            -          online
invalid

    disk_1            auto:none       -            -          online
invalid

    disk_2            auto:none       -            -          online
invalid

    disk_3            auto:none       -            -          online
invalid

    emc_clariion0_28  auto:cdsdisk  apps_dg01   apps_dg01         online

    emc_clariion0_29  auto:cdsdisk  db_dg0       db_dg            online
    VxVM will ensure that the newly created DGs are visible from both the cluster nodes. These disk groups can be used only from one node at a time. You will have to import/deport a disk group from either node to use the disk groups and their volumes.
  4. Create volumes in the disk groups using the following commands: 
    $ vxassist
-g apps_dg make apps_vol 299g

    $ vxassist
-g db_dg make db_vol 299g
  5. Initialize the volumes with the VxFS file system using the following commands: 
    $ mkfs -t
vxfs -o bsize=4096,largefiles /dev/vx/rdsk/db_dg/db_vol

    $ mkfs -t
vxfs -o bsize=4096,largefiles /dev/vx/rdsk/apps_dg/apps_vol
    For better performance, use a 4 KB block size and enable support for large files (more than 1 TB).
  6. Create the mount points and mount the volumes using the following commands: 
    $ mount -t
vxfs -o largefiles /dev/vx/dsk/apps_dg/apps_vol /shared_apps

    $ mount -t
vxfs -o largefiles /dev/vx/dsk/db_dg/db_vol /shared_db

Tuning the VxFS File System

The VxFS file system can be tuned for better performance. The vxtunefs command can be used to set the tuning parameters. The default values of these parameters are set when the volume is mounted.

The performance of the MUR application can improve when the following tuning parameters are changed:

  • read_pref_io: The preferred read request size. The filesystem uses this in conjunction with the read_nstream value to determine how much data to read ahead. The default value is 64000. The MUR performance can improve when this value is set to 128000.
  • read_nstream: This is the desired number of parallel read requests of the size specified in the read_pref_io parameter to have outstanding at one time. The file system uses the value specified in the read_nstream parameter multiplied by the value specified in the read_pref_io parameter to determine its read ahead size. The default value for the read_nstream parameter is 1. If you know the hardware RAID configuration on the external storage, then set the read_nstream parameter value to be the number of columns (disks) in the disk array.
  • write_pref_io: The preferred write request size. The filesystem uses this in conjunction with the value specified in the write_nstream parameter to determine how to flush behind on writes. The default value is 64000. The MUR performance can improve when this value is set to 128000.
  • write_nstream: This is the desired number of parallel write requests of the size specified in the write_pref_io parameter to have outstanding at one time. The file system uses the value specified in the write_nstream parameter multiplied by the value specified in the write_pref_io parameter to determine when to flush behind on writes. The default value for the write_nstream parameter is 1. For disk striping configurations, set the value of the write_pref_io and write_nstream parameters to the same values as the read_pref_io and read_nstream parameters.
  1. Use the following command to tune Veritas file system: $/opt/VRTS/bin/vxtunefs -o read_pref_io=131072,read_unit_io=131072,write_pref_io=131072,write_unit_io=131072 /shared_db
  2. To ensure that these values are not lost after a reboot, create the file /etc/vx/tunefstab and add entries using the command: $ cat /etc/vx/tunefstab /dev/vx/dsk/apps_dg/apps_vol read_pref_io=131072,read_unit_io=131072,write_pref_io=131072,write_unit_io=131072 Please create these files on both the nodes. Now, you can install the MUR application from one of the cluster nodes. Please ensure that the check box to start the MUR processes is not selected during the MUR installation. MUR application will be started after the required resources are configured through VCS. For more information on how to install the MUR application, refer to the Managing Mobility Unified Reporting System Installation chapter of this guide. Prior to installing MUR, MUR Administrator user and Group need to be created and activated. MUR Administrator user is the user who will own the MUR installation. Execute the following steps as root user for creating and activating MUR user and group:
    1. Create MUR group using the following command:groupadd -g <groupID> <groupname>For example: groupadd -g 500 murgroup
    2. Create MUR user using the following command:useradd -u <userID> -c "MUR-administrator" -g <groupname> -m <username>For example: useradd -u 100014 -c "MUR-administrator" -g murgroup -m muradmin
    3. Activate the MUR user using a password:passwd <username>For example: passwd muradminDuring installation, provide this user as the Administrator user input.

      IMPORTANT:

      For HA mode, the user and group creation must be done on both the nodes in the cluster (Active and Passive). Ensure that you provide the same userID, groupID and password on both the nodes. Also, note that HA mode installation is only supported on RHEL platform.

Configuring Resources for High Availability

After installation of MUR application, the following resources need to be configured with the Veritas cluster:

  • NIC — To monitor a Network Interface Card (NIC)
  • IP — To monitor an IP address
  • Disk Group, Volume, and Mount — for shared storage
  • MUR Application — comprising of all the MUR related processes

Assume the following configuration resources as:

  • Two disk groups — apps_dg and db_dg
  • Two volumes — apps_vol and db_vol mounted on /shared_apps and /shared_db respectively
  • MUR installation directory — /shared_apps/starbi
  • MUR postgres data directory — /shared_db/data
  • MUR Administrator — muradmin
  • Shared/Floating IP address — 10.4.83.151 (on NIC eth0). The floating/shared IP used should be a public IP in DNS, pingable from the client machine.

To configure these resources:

IMPORTANT:

The following configurations should be performed only on the node where the MUR application is installed.

  1. Log on as super user (root).
  2. Make the Veritas config file (main.cf) writable using the following command: 
    $ haconf -makerw
  3. Create resource group using the following commands: 
    $ hagrp -add
mur-ha

    $ hagrp –modify
mur-ha SystemList <Node1> 0 <Node2> 1

    $ hagrp –modify
mur-ha NumRetries 1
    Where, Node1 and Node2 are the hostnames of the active and passive nodes.
  4. Create DG resource for the MUR partition using the following commands: 
    $ hares -add
mur-apps-dg DiskGroup mur-ha

    $ hares -modify
mur-apps-dg DiskGroup apps_dg

    $ hares -modify
mur-apps-dg Enabled 1
  5. Create DG resource for postgres data partition using the following commands: 
    $ hares -add
mur-db-dg DiskGroup mur-ha

    $ hares -modify
mur-db-dg DiskGroup db_dg

    $ hares -modify
mur-db-dg Enabled 1
  6. Create Volume resource for the MUR partition using the following commands: 
    $ hares -add
mur-apps-vol Volume mur-ha

    $ hares -modify
mur-apps-vol DiskGroup apps_dg

    $ hares -modify
mur-apps-vol Volume apps_vol

    $ hares -modify
mur-apps-vol Enabled 1
  7. Create Volume resource for the postgres data partition using the following commands: 
    $ hares -add
mur-db-vol Volume mur-ha

    $ hares -modify
mur-db-vol DiskGroup db_dg

    $ hares -modify
mur-db-vol Volume db_vol

    $ hares -modify
mur-apps-vol Enabled 1
  8. Create Mount resource for the MUR partition using the following commands: 
    $ hares -add
mur-apps-mnt Mount mur-ha

    $ hares -modify
mur-apps-mnt MountPoint /shared_apps

    $ hares -modify
mur-apps-mnt BlockDevice /dev/vx/dsk/apps_dg/apps_vol

    $ hares -modify
mur-apps-mnt FSType vxfs

    $ hares -modify
mur-apps-mnt FsckOpt %-y

    $ hares -modify
mur-apps-mnt MountOpt largefiles

    $ hares -modify
mur-apps-mnt Enabled 1
  9. Create Mount resource for the postgres data partition using the following commands: 
    $ hares -add
mur-db-mnt Mount mur-ha

     $ hares -modify
mur-db-mnt MountPoint /shared_db

    $ hares -modify
mur-db-mnt BlockDevice /dev/vx/dsk/db_dg/db_vol

    $ hares -modify
mur-db-mnt FSType vxfs

    $ hares -modify
mur-db-mnt FsckOpt %-y

    $ hares -modify
mur-db-mnt  MountOpt largefiles

    $ hares -modify
mur-db-mnt  Enabled 1
  10. Create Application resource for the MUR processes using the following commands: 
    $ hares -add
mur-app Application mur-ha

    $ hares -modify
mur-app User muradmin

    $ hares -modify
mur-app StartProgram "/shared_apps/starbi/starbi/bin/serv
start"

    $ hares -modify
mur-app StopProgram "/shared_apps/starbi/starbi/bin/serv
forcestop"

    $ hares -modify
mur-app PidFiles "/shared_apps/starbi/starbi/server/sysmon/psmon.pid

    $ hares -modify
mur-app  Enabled 1
  11. Create the NIC resource using the following commands: 
    $ hares -add
mur-nic NIC mur-ha

    $ hares -modify
mur-nic Device eth0

    $ hares -modify
mur-nic Enabled 1
  12. Create the IP resource using the following commands: 
    $ hares -add
mur-ip IP mur-ha

    $ hares -modify
mur-ip Device eth0

    $ hares -modify
mur-ip Address <ip-address>

    IMPORTANT:

    The floating or shared IP address should be a public IP in the DNS to which the client machine can successfully ping.

    $ hares -modify
mur-ip NetMask 255.255.255.0

    $ hares -modify
mur-ip Enabled 1
  13. Set the resource dependencies using the following commands: 
    $ hares -link
mur-app mur-apps-mnt

    $ hares -link
mur-app mur-db-mnt

    $ hares -link
mur-apps-mnt mur-apps-vol

    $ hares -link
mur-db-mnt mur-db-vol

    $ hares -link
mur-apps-vol mur-apps-dg

    $ hares -link
mur-db-vol mur-db-dg

    $ hares -link
mur-app mur-ip

    $ hares -link
mur-ip mur-nic
  14. Dump the configuration to the Veritas config file using the following command: 
    $ haconf -dump
-makero
  15. Bring the MUR HA application online on Node 1 using the following command: 
    $ hagrp -online
mur-ha -sys <Node1>
    Once the above steps are performed, the MUR HA application will start running and you can access GUI using shared IP specified earlier.

Recovering MUR in HA

For recovering MUR in HA mode:

  1. Make the Veritas config file writable using the following command: 
    $ haconf –makerw
  2. Disable MUR HA application on the standby node using the following command: 
    $ hagrp -disable
mur-ha -sys <standby-node>
  3. Make MUR Application (mur-app) resource offline on the active node using the following command: 
    $ hares -offline
mur-app -sys <active-node>
  4. Disable MUR-APP resource using the following command: 
    $ hares -modify
mur-app Enabled 0
  5. Perform usual MUR recovery process on the active-node (Install same MUR version, start postgres and run recovery script).
  6. Once recovery is successful, enable MUR-APP resource using the following command: 
    $ hares -modify
mur-app Enabled 1
  7. Bring the MUR-APP resource online on the active node using the following command: 
    $ hares -online
mur-app -sys <active-node>
  8. Enable MUR HA application on the standby node using the following command: 
    $ hagrp -enable
mur-ha -sys <standby-node>
  9. Dump the configuration to Veritas config file using the following command: 
    $ haconf -dump
-makero