InTracer Clustering Support for High Availability

This chapter describes how to install and configure the Veritas cluster for high availability in InTracer. InTracer application traces and analyzes near real time subscriber calls. InTracer heartbeat daemon monitors all the InTracer processes and starts them when any process gets stopped. But if heart beat daemon itself goes down then there is no any process which monitors the heart beat daemon. In clustering we monitor the heart beat daemon and whenever the heart beat daemon gets stopped/crashed, clustering tries to restart it. Whenever machine gets crashed or if there is any loss of communication between resources which clustering is going to monitor then failover happens to restore the loss of communication.

This chapter contains the following sections:

System and Hardware Recommendations for HA Deployment

The InTracer can be configured in HA mode with asymmetric or active/passive configuration. The following requirements must be met in order to ensure the HA deployment in the InTracer system:
  • 2 UCS 460 M1 servers for cluster nodes
  • Up to 4 Internal disks (depending upon the fault tolerance for boot disk/OS)
  • Qlogic QLE2462 4Gb dual port FC HBA/Qlogic QLE2560 8 Gbps FC 1 port
  • FC cables
  • External Storage with multiple disks (EMC/Sun Storage)
  • Veritas (VxFS) file system

The hardware setup for the Veritas Cluster Server (VCS) consists of two cluster nodes connected with an external shared storage. UCS 460 supports multiple PCI slots for attaching the FC HBA cards. Both the cluster nodes must be connected to the external storage. The cluster nodes must be installed with the Cisco MITG RHEL operating system, Veritas Storage Foundation (Veritas Volume Manager and Veritas File System), and VCS (for High Availability).

The Veritas Volume Manager (VxVM) is used to create a single Disk Group (DG) containing multiple disks. A separate group of disks is allocated for postgres data directory to achieve good performance.

The RAID recommendations are the same as that of standalone installation i.e. RAID-0 for InTracer application and RAID-5 for database.

Separate disk/LUN from the shared storage is required for I/O fencing. I/O fencing is part of the VCS administration. It is assumed that I/O fencing is already configured on the Veritas Cluster setup before the InTracer application is installed for HA.

Configuring the External Storage Disk on UCS for HA Deployment

This section describes the procedure to configure an external storage disk on a UCS server for HA deployment.

It is recommended to create single disk group for the components C-InTracer / R-TCE:
  • Configuration files, PostgreSQL data directory, Trace data files and actual data
It is recommended to create two separate disk groups for R-InTracer [Nodal]:
  • Configuration files
  • SGSN data

The number of disks in each group depends on the total throughput required and the size of disks in the external storage. Please contact your system administrator for setting up the external storage to make the required number of disks (Logical Disk Units (LUNs)) accessible from both the cluster nodes. When the external storage disks are made accessible, you can see them connected using multipath command:

multipath -l
mpath6 (36006016069902d008892bc0dec14e111)
dm-7 DGC,RAID 5
[size=300G][features=1
queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin
0 [prio=0][active]
\_ 8:0:0:1
sdi 8:128 [active][undef]
 \_ 8:0:1:1
sdk 8:160 [active][undef]
mpath5 (36006016069902d00563e8a20ec14e111)
dm-6 DGC,RAID 5
[size=300G][features=1
queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin
0 [prio=0][active]
\_ 8:0:0:0
sdh 8:112 [active][undef]
\_ 8:0:1:0
sdj 8:144 [active][undef]

To configure an external storage disk into separate DGs:

  1. Rebuild the disk lists with the new disks detected by the kernel using the following command:
    vxdctl initdmp
    
    vxdctl enable
    
    To see the status of the new disk, use the following command:
    vxdisk -o alldgs list
    
    DEVICE             TYPE        DISK       GROUP         STATUS
    
    disk_0            auto:none     -           -          online
    invalid
    
    disk_1            auto:none     -           -          online
    invalid
    
    disk_2            auto:none     -           -          online
    invalid
    
    disk_3            auto:none     -           -          online
    invalid
    
    emc_clariion0_28  auto          -           -            error
    
    emc_clariion0_29  auto          -           -            error
    
  2. To setup the disks, use the following commands:
    /etc/vx/bin/vxdisksetup
    -i emc_clariion0_28
    
    /etc/vx/bin/vxdisksetup
    -i emc_clariion0_29
    
    To see the status of the new disk, use the command:
    vxdisk -o alldgs list
    
    DEVICE             TYPE        DISK       GROUP         STATUS
    
    disk_0            auto:none     -           -          online
    invalid
    
    disk_1            auto:none     -           -          online
    invalid
    
    disk_2            auto:none     -           -          online
    invalid
    
    disk_3            auto:none     -           -          online
    invalid
    
    emc_clariion0_28  auto:cdsdisk  -           -            online
    
    emc_clariion0_29  auto:cdsdisk  -           -            online
    
  3. With the newly initialized disks, create two separate DGs for one for storing the configuration files and one for SGSN data storage only in case of R-InTracer [Nodal] installation:
    vxdg init config_dg config_dg01=emc_clariion0_28
    
    vxdg init db_dg db_dg01=emc_clariion0_29
    

    IMPORTANT:

    You can specify more disks (using Step 2) and add the disk to disk groups.

    After adding the disk into respective DGs, you can verify them using the following command:
    vxdisk -o alldgs list
    
    DEVICE             TYPE          DISK        GROUP         STATUS
    
    disk_0            auto:none       -            -          online
    invalid
    
    disk_1            auto:none       -            -          online
    invalid
    
    disk_2            auto:none       -            -          online
    invalid
    
    disk_3            auto:none       -            -          online
    invalid
    
    emc_clariion0_28  auto:cdsdisk  config_dg01   config_dg01         online
    
    emc_clariion0_29  auto:cdsdisk  db_dg0       db_dg            online
    
    VxVM ensures that the newly created DGs are visible from both the cluster nodes. These disk groups can be used only from one node at a time. You must import/deport a disk group from either node to use the disk groups and their volumes.
  4. Create volumes in the disk groups using the following commands:
    vxassist -g config_dg
    make config_vol 299g
    
    For SGSN data storage, create four separate volumes of same size for db_db disk group depending upon the disk space.
    vxassist -g db_dg
    make db_vol0 70gvxassist -g db_dg
    make db_vol1 70gvxassist -g db_dg
    make db_vol2 70gvxassist -g db_dg
    make db_vol3 70g
    
  5. Initialize the volumes with the VxFS file system using the following commands:
    mkfs -t vxfs -o bsize=4096,largefiles /dev/vx/rdsk/db_dg/db_vol0mkfs -t vxfs -o bsize=4096,largefiles /dev/vx/rdsk/db_dg/db_vol1mkfs -t vxfs -o bsize=4096,largefiles /dev/vx/rdsk/db_dg/db_vol2mkfs -t vxfs -o bsize=4096,largefiles /dev/vx/rdsk/db_dg/db_vol3
    
    For better performance, use a 4 KB block size and enable support for large files (more than 1 TB).
  6. Create the mount points and mount the volumes using the following commands:
    mount -t vxfs -o largefiles /dev/vx/dsk/apps_dg/config_vol /shared_intracer_config
    
    mount -t vxfs -o largefiles /dev/vx/dsk/db_dg/db_vol0 /shared_intracer_db0mount -t vxfs -o largefiles /dev/vx/dsk/db_dg/db_vol1 /shared_intracer_db1mount -t vxfs -o largefiles /dev/vx/dsk/db_dg/db_vol2 /shared_intracer_db2mount -t vxfs -o largefiles /dev/vx/dsk/db_dg/db_vol3 /shared_intracer_db3 
    

Setting up VERITAS Cluster Resources

The following are the directories and files needed to be installed on shared disk:

  1. C-InTracer
    1. /etc/ -> ipms_system.cfg and bootstrap.cfg file
    2. /data/-> Component specific configuration
    3. PostgreSQL data directory
  2. R-InTracer - LTE
    1. /etc/ -> ipms_system.cfg and bootstrap.cfg file
    2. /data/-> Component specific configuration,, TCE data and archived TCE files
  3. R-InTracer - Nodal
    1. /etc/ -> ipms_system.cfg and bootstrap.cfg file
    2. /data/-> Component specific configuration
    3. All four volumes for storing SGSN data

The following are simple steps to configure the services and applications in Cluster:

  1. Create a service group comprising of all resources required for the application
  2. Add required resources to the service group and configure them
  3. Assign dependencies between resources
  4. Bring the service group online to make the resources available

After installation of InTracer application, the following resources need to be configured with the Veritas cluster:

  • NIC — To monitor a Network Interface Card (NIC)
  • IP — To monitor an IP address
  • Disk Group, Volume, and Mount — for shared storage
  • InTracer Application — comprising of all the InTracer related processes

The disk group, volume and mount points required on shared disk are as follows:

C-InTracer and R-TCE:

  • Intracer-apps-dg -> One disk group required for storing the configuration files, PostgreSQL data directory in case of C-InTracer and Subscriber Trace data (Berkeley DB) in case R-TCE and system and configuration backup files.
  • Intracer-apps-vol0 -> Only one volume is required for storing the above data

R-InTracer (NODAL): Currently SGSN data is stored on separate volumes. So to achieve this and for the performance, two disk groups are required as below:

  • Intracer-apps-cfg-dg->' required for storing configuration files.
    Intracer-apps-vol0-> only one volume is required for storing above data
  • Intracer-apps-db-db -> Required for storing SGSN data.
    In this disk group, 4 separate volumes, as given below are required, which are mapped to respective eventproc process.
    • intercer-apps-vol0
    • intracer-apps-vol1
    • intracer-apps-vol2
    • intracer-apps-vol3

Basic Flow of Resource Configuration

Overall design of application considering the shared disk and cluster nodes is as follows:


Figure 1. C-InTracer and R-TCE

Figure 2. R-InTracer (Nodal)

Installation Steps for Cluster Setup

This section describes installation steps for cluster setup.

Installing Cluster Failover for InTracer
  1. During installation process, there are two opt:
    • Standalone installation
    • Cluster failover
      Choose “Cluster Failover” option to install Cluster Failover for InTracer.
  2. Selecting “Cluster Failover” option displays the following screen to the user to get the shared directory path. All of the configuration files, PostgreSQL data dir and subscriber trace data is stored on this location. After getting the shared disk path, installer installs all the following directories on shared disk (if not already present) under the directory “intracer” which is the mount point created on the shared directory.
    • /etc/
    • /data/
    • pgdata ' PostgreSQL data directory
    Installer creates the symbolic link to the above directories in application code so that there are no code related changes for changing the location. In case of NODAL installation, all the data volumes required for storing SGSN data are on the shared disk, so in case of cluster setup a note “Path should be on the shared disk” is shown on installer at the time of getting the SGSN data directory path:
  3. Click Next to go to the Component Configuration screen to configure the selected component.
  • The following Pre-Installation summary is displayed when Cluster Failover option is selected. Configured shared disk path is shown along with the other configuration summary of the component.
  • The following Installation summary is displayed when Cluster Failover option is selected.

VCS Commands for Resource Configuration

This section decribes how to configure VCS commands for Veritas Cluster.

Note:

Resource configuration for R-InTracer [Nodal], C-InTracer and R-TCE is same, except the disk group”config_dg” for R-InTracer. Resource configuration for the components C-InTraaer and R-TCE does not create db_dg, db_vol and db_mnt.

Assume the configuration resources as follows:

Two disk-groups are - config_dg & db_dg.

The following is the resource configuration for R-InTracer [Nodal]:

Volumes are config_vol, db_vol0, db_vol1, db_vol2, db_vol3 mounted on /shared_intracer_config, /shared_intracer_db0, /shared_intracer_db1, /shared_intracer_db2, /shared_intracer_db3 respectively.

InTracer installation directory - /<node1_path>/<component_name>

Component: R-InTracer [Nodal]

Path: /export/home/<username>/Nodal

Configuration data directories are as follows:
  • /shared_intracer_config/data
  • /shared_intracer_config/etc
  • /shared_intracer_config/pgdata
SGSN data directories are as follows:
  • /shared_intracer_db0
  • /shared_intracer_db1
  • /shared_intracer_db2
  • /shared_intracer_db3

Shared ip-address - 10.4.83.151 (on NIC eth0)

Query Server main.cs

  1. The following commands are used to create Main Disk Group resource for Configuration data partition:
    haconf -makerw
    
    hagrp -add intracer-ha
    
    hagrp -modify intracer-ha
    SystemList -add pnstextappsucs1 0 pnstextappsucs3 1
    
    hagrp -modify intracer-ha
    AutoStartList pnstextappsucs1
    
    hagrp -list
    
  2. The following commands are used to create Disk Group resource for Configuration data partition:
    hares -add intracer-config-dg
    DiskGroup intracer-ha
    
    hares -modify intracer-config-dg
    DiskGroup intracer_dg
    
    hares -modify intracer-config-dg
    Enabled 1
    
  3. The following commands are used to create Volume resource for Configuration data partition:
    hares -add intracer-config-vol
    Volume intracer-ha
    
    hares -modify intracer-config-vol
    DiskGroup intracer_dg
    
    hares -modify intracer-config-vol
    Volume intracer_data
    
    hares -modify intracer-config-vol
    Enabled 1 
    
  4. The following commands are used to create Mount resource for configuration partition:
    hares -add intracer-config-mnt
    Mount intracer-ha
    
    hares -modify intracer-config-mnt
    MountPoint /intracer_data
    
    hares -modify intracer-config-mnt
    BlockDevice /dev/vx/dsk/intracer_dg/intracer_data
    
    hares -modify intracer-config-mnt
    FSType vxfs
    
    hares -modify intracer-config-mnt
    FsckOpt %-y
    
    hares -modify intracer-config-mnt
    MountOpt largefiles
    
    hares -modify intracer-config-mnt
    Enabled 1
    
  5. The following commands are used to create Application resource for INTRACER processes:
    hares -add intracer-app
    Application intracer-ha
    
    hares -modify intracer-app
    User sdeshmuk
    
    hares -modify intracer-app
    StartProgram "/apps/intracer/182/QS/ipmssys
    start with-monitor"
    
    hares -modify intracer-app
    StopProgram "/apps/intracer/182/QS/ipmssys
    stop with-monitor"
    
    hares -modify intracer-app
    CleanProgram "/apps/intracer/182/QS/ipmssys
    stop with-monitor"
    
    hares -modify intracer-app
    PidFiles "/apps/intracer/182/QS/data/QS/.ipmsqshbd.pid"
    
    hares -modify intracer-app
     Enabled 1
    
  6. The following commands are used to create the NIC resource:
    hares -add intracer-nic
    NIC intracer-ha 
    
    hares -modify intracer-nic
    Device eth0
    
    hares -modify intracer-nic
    Enabled 1
    
  7. The following commands are used to create the IP resource:
    hares -add intracer-ip
    IP intracer-ha
    
    hares -modify intracer-ip
    Device eth0
    
    hares -modify intracer-ip
    Address 10.105.85.128
    
    hares -modify intracer-ip
    NetMask 255.255.255.0
    
    hares -modify intracer-ip
    Enabled 1
    
  8. The following commands are used to set the resource dependencies:
    hares -link intracer-app
    intracer-config-mnt
    
    hares -link intracer-config-mnt
    intracer-config-vol
    
    hares -link intracer-config-vol
    intracer-config-dg
    
    hares -link intracer-app
    intracer-ip
    
    hares -link intracer-ip
    intracer-nic
    

TCE Main.cs

  1. The following commands are used to create Main Disk Group resource for Configuration / TCE data partition:
    haconf -makerw
    
    hagrp -add intracer-ha
    
    hagrp -modify intracer-ha
    SystemList -add pnstextappsucs1 0 pnstextappsucs3 1
    
    hagrp -modify intracer-ha
    AutoStartList pnstextappsucs1
    
    hagrp -list
    
  2. The following commands are used to create Disk Group resource for Configuration / TCE data partition:
    hares -add intracer-config-dg
    DiskGroup intracer-ha
    
    hares -modify intracer-config-dg
    DiskGroup intracer_dg
    
    hares -modify intracer-config-dg
    Enabled 1
    
  3. The following commands are used to create Volume resource for Configuration / TCE data partition:
    hares -add intracer-config-vol
    Volume intracer-ha
    
    hares -modify intracer-config-vol
    DiskGroup intracer_dg
    
    hares -modify intracer-config-vol
    Volume intracer_data
    
    hares -modify intracer-config-vol
    Enabled 1 
    
  4. The following commands are used to create Mount resource for configuration / TCE partition:
    hares -add intracer-config-mnt
    Mount intracer-ha
    
    hares -modify intracer-config-mnt
    MountPoint /intracer_data
    
    hares -modify intracer-config-mnt
    BlockDevice /dev/vx/dsk/intracer_dg/intracer_data
    
    hares -modify intracer-config-mnt
    FSType vxfs
    
    hares -modify intracer-config-mnt
    FsckOpt %-y
    
    hares -modify intracer-config-mnt
    MountOpt largefiles
    
    hares -modify intracer-config-mnt
    Enabled 1
    
  5. The following commands are used to create Application resource for INTRACER processes:
    hares -add intracer-app
    Application intracer-ha
    
    hares -modify intracer-app
    User sdeshmuk
    
    hares -modify intracer-app
    StartProgram "/apps/intracer/182/3GPP/ipmssys
    start"
    
    hares -modify intracer-app
    StopProgram "/apps/intracer/182/3GPP/ipmssys
    stop"
    
    hares -modify intracer-app
    CleanProgram "/apps/intracer/182/3GPP/ipmssys
    stop"
    
    hares -modify intracer-app
    PidFiles "/apps/intracer/182/3GPP/data/TCE/tce1/.ipmstcehbd.pid"
    
    hares -modify intracer-app
     Enabled 1
    
  6. The following commands are used to create the NIC resource:
    hares -add intracer-nic
    NIC intracer-ha 
    
    hares -modify intracer-nic
    Device eth0
    
    hares -modify intracer-nic
    Enabled 1
    
  7. The following commands are used to create the IP resource:
    hares -add intracer-ip
    IP intracer-ha
    
    hares -modify intracer-ip
    Device eth0
    
    hares -modify intracer-ip
    Address 10.105.85.128
    
    hares -modify intracer-ip
    NetMask 255.255.255.0
    
    hares -modify intracer-ip
    Enabled 1
    
  8. The following commands are used to set the resource dependencies:
    hares -link intracer-app
    intracer-config-mnt
    
    hares -link intracer-config-mnt
    intracer-config-vol
    
    hares -link intracer-config-vol
    intracer-config-dg
    
    hares -link intracer-app
    intracer-ip
    
    hares -link intracer-ip
    intracer-nic
    

Nodal main.cs

  1. The following commands are used to create Main Disk Group resource for Configuration / TCE data partition:
    haconf -makerw
    
    hagrp -add intracer-sgsn
    
    hagrp -modify intracer-sgsn
    SystemList -add pnstextappsucs1 0 pnstextappsucs3 1
    
    hagrp -modify intracer-sgsn
    AutoStartList pnstextappsucs1
    
    hagrp -list
    
  2. The following commands are used to create Disk Group resource for Configuration / TCE data partition:
    hares -add intracer-config-dg
    DiskGroup intracer-sgsn
    
    hares -modify intracer-config-dg
    DiskGroup intracer_dg
    
    hares -modify intracer-config-dg
    Enabled 1
    
  3. The following commands are used to create Disk Group resource for SGSN data partition:
    hares -add intracer-sgsn-dg
    DiskGroup intracer-sgsn
    
    hares -modify intracer-sgsn-dg
    DiskGroup intracer_sgsn
    
    hares -modify intracer-sgsn-dg
    Enabled 1
    
  4. The following commands are used to create Volume resource for Configuration / TCE data partition:
    hares -add intracer-config-vol
    Volume intracer-sgsn
    
    hares -modify intracer-config-vol
    DiskGroup intracer_dg
    
    hares -modify intracer-config-vol
    Volume intracer_data
    
    hares -modify intracer-config-vol
    Enabled 1 
    
  5. The following commands are used to create Volume resource for SGSN data partition:
    hares -add intracer-sgsn-vol0
    Volume intracer-sgsn
    
    hares -modify intracer-sgsn-vol0
    DiskGroup intracer_sgsn
    
    hares -modify intracer-sgsn-vol0
    Volume intracer_vol0
    
    hares -modify intracer-sgsn-vol0
    Enabled 1
    
    hares -add intracer-sgsn-vol1
    Volume intracer-sgsn
    
    hares -modify intracer-sgsn-vol1
    DiskGroup intracer_sgsn
    
    hares -modify intracer-sgsn-vol1
    Volume intracer_vol1
    
    hares -modify intracer-sgsn-vol1
    Enabled 1
    
    hares -add intracer-sgsn-vol2
    Volume intracer-sgsn
    
    hares -modify intracer-sgsn-vol2
    DiskGroup intracer_sgsn
    
    hares -modify intracer-sgsn-vol2
    Volume intracer_vol2
    
    hares -modify intracer-sgsn-vol2
    Enabled 1
    
    hares -add intracer-sgsn-vol3
    Volume intracer-sgsn
    
    hares -modify intracer-sgsn-vol3
    DiskGroup intracer_sgsn
    
    hares -modify intracer-sgsn-vol3
    Volume intracer_vol3
    
    hares -modify intracer-sgsn-vol3
    Enabled 1
    
  6. The following commands are used to create Mount resource for configuration partition:
    hares -add intracer-config-mnt
    Mount intracer-sgsn
    
    hares -modify intracer-config-mnt
    MountPoint /intracer_data
    
    hares -modify intracer-config-mnt
    BlockDevice /dev/vx/dsk/intracer_dg/intracer_data
    
    hares -modify intracer-config-mnt
    FSType vxfs
    
    hares -modify intracer-config-mnt
    FsckOpt %-y
    
    hares -modify intracer-config-mnt
    MountOpt largefiles
    
    hares -modify intracer-config-mnt
    Enabled 1
    
  7. The following commands are used to create Mount resource for SGSN partition:
    hares -add intracer-sgsn-mnt0
    Mount intracer-sgsn
    
    hares -modify intracer-sgsn-mnt0
    MountPoint /intracer_vol0
    
    hares -modify intracer-sgsn-mnt0
    BlockDevice /dev/vx/dsk/intracer_sgsn/intracer_vol0
    
    hares -modify intracer-sgsn-mnt0
    FSType vxfs
    
    hares -modify intracer-sgsn-mnt0
    FsckOpt %-y
    
    hares -modify intracer-sgsn-mnt0
    MountOpt largefiles
    
    hares -modify intracer-sgsn-mnt0
    Enabled 1
    
    hares -add intracer-sgsn-mnt1
    Mount intracer-sgsn
    
    hares -modify intracer-sgsn-mnt1
    MountPoint /intracer_vol1
    
    hares -modify intracer-sgsn-mnt1
    BlockDevice /dev/vx/dsk/intracer_sgsn/intracer_vol1
    
    hares -modify intracer-sgsn-mnt1
    FSType vxfs
    
    hares -modify intracer-sgsn-mnt1
    FsckOpt %-y
    
    hares -modify intracer-sgsn-mnt1
    MountOpt largefiles
    
    hares -modify intracer-sgsn-mnt1
    Enabled 1
    
    hares -add intracer-sgsn-mnt2
    Mount intracer-sgsn
    
    hares -modify intracer-sgsn-mnt2
    MountPoint /intracer_vol2
    
    hares -modify intracer-sgsn-mnt2
    BlockDevice /dev/vx/dsk/intracer_sgsn/intracer_vol2
    
    hares -modify intracer-sgsn-mnt2
    FSType vxfs
    
    hares -modify intracer-sgsn-mnt2
    FsckOpt %-y
    
    hares -modify intracer-sgsn-mnt2
    MountOpt largefiles
    
    hares -modify intracer-sgsn-mnt2
    Enabled 1
    
    hares -add intracer-sgsn-mnt3
    Mount intracer-sgsn
    
    hares -modify intracer-sgsn-mnt3
    MountPoint /intracer_vol3
    
    hares -modify intracer-sgsn-mnt3
    BlockDevice /dev/vx/dsk/intracer_sgsn/intracer_vol3
    
    hares -modify intracer-sgsn-mnt3
    FSType vxfs
    
    hares -modify intracer-sgsn-mnt3
    FsckOpt %-y
    
    hares -modify intracer-sgsn-mnt3
    MountOpt largefiles
    
    hares -modify intracer-sgsn-mnt3
    Enabled 1
    
  8. The following commands are used to create Application resource for INTRACER processes:
    hares -add intracer-app
    Application intracer-sgsn
    
    hares -modify intracer-app
    User sdeshmuk
    
    hares -modify intracer-app
    StartProgram "/apps/intracer/182/NODAL/ipmssys
    start"
    
    hares -modify intracer-app
    StopProgram "/apps/intracer/182/NODAL/ipmssys
    stop"
    
    hares -modify intracer-app
    CleanProgram "/apps/intracer/182/NODAL/ipmssys
    stop"
    
    hares -modify intracer-app
    PidFiles "/apps/intracer/182/NODAL/data/TCE/tce1/.ipmstcehbd.pid"
    
    hares -modify intracer-app
     Enabled 1
    
  9. The following commands are used to create the NIC resource:
    hares -add intracer-nic
    NIC intracer-sgsn 
    
    hares -modify intracer-nic
    Device eth0
    
    hares -modify intracer-nic
    Enabled 1
    
  10. The following commands are used to create EVT-SOURCE NIC resource:
    hares -add intracer-sgsn-nic
    NIC intracer-sgsn 
    
    hares -modify intracer-sgsn-nic
    Device eth1
    
    hares -modify intracer-sgsn-nic
    Enabled 1
    
  11. The following commands are used to create the IP resource:
    hares -add intracer-ip
    IP intracer-sgsn
    
    hares -modify intracer-ip
    Device eth0
    
    hares -modify intracer-ip
    Address 10.105.85.128
    
    hares -modify intracer-ip
    NetMask 255.255.255.0
    
    hares -modify intracer-ip
    Enabled 1
    
  12. The following commands are used to create EVT-SOURCE IP resource:
    hares -add intracer-ip
    IP intracer-sgsn
    
    hares -modify intracer-sgsn-ip
    Device eth1
    
    hares -modify intracer-sgsn-ip
    Address 192.168.2.1
    
    hares -modify intracer-sgsn-ip
    NetMask 255.255.255.0
    
    hares -modify intracer-sgsn-ip
    Enabled 1
    
  13. The following commands are used to set the resource dependencies:
    hares -link intracer-app
    intracer-config-mnt
    
    hares -link intracer-config-mnt
    intracer-config-vol
    
    hares -link intracer-config-vol
    intracer-config-dg
    
    hares -link intracer-app
    intracer-sgsn-mnt0
    
    hares -link intracer-sgsn-mnt0
    intracer-sgsn-vol0
    
    hares -link intracer-sgsn-vol0
    intracer-sgsn-dg
    
    hares -link intracer-app
    intracer-sgsn-mnt1
    
    hares -link intracer-sgsn-mnt1
    intracer-sgsn-vol1
    
    hares -link intracer-sgsn-vol1
    intracer-sgsn-dg
    
    hares -link intracer-app
    intracer-sgsn-mnt2
    
    hares -link intracer-sgsn-mnt2
    intracer-sgsn-vol2
    
    hares -link intracer-sgsn-vol2
    intracer-sgsn-dg
    
    hares -link intracer-app
    intracer-sgsn-mnt3
    
    hares -link intracer-sgsn-mnt3
    intracer-sgsn-vol3
    
    hares -link intracer-sgsn-vol3
    intracer-sgsn-dg
    
    hares -link intracer-app
    intracer-ip
    
    hares -link intracer-app
    intracer-sgsn-ip
    
    hares -link intracer-ip
    intracer-nic
    
    hares -link intracer-sgsn-ip
    intracer-nic
    

VCS Resource Configuration File

VCS resource configuration file (main.cf file) is created after executing resource configuration commands.

group intracer-ha (
       SystemList = { pnstextappsucs3 = 0, pnstextappsucs1 = 1 } 
       )
       Application
intracer-app ( 
              User = sdeshmuk
                 
              StartProgram = "/export/home/sdeshmuk/Nodal/ipmssys
start"
              StopProgram = "/export/home/sdeshmuk/Nodal/ipmssys
stop"
              PidFiles = {
                     "/export/home/sdeshmuk/Nodal/data/<component_name/.ipmsservhbd.pid"}
              )
                 
       DiskGroup
intracer-config-dg ( (
              DiskGroup = config_dg
              )
       DiskGroup
intracer-db-dg (
              DiskGroup = db
              )
       IP
intracer-ip (
              Device = eth0
              Address = "10.4.83.151"
              NetMask = "255.255.255.0"
              )
       Mount
intracer-config-mnt (
              MountPoint = "/shared_apps"
              BlockDevice = "/dev/vx/dsk/apps_dg/config_vol"
              FSType = vxfs
              MountOpt = largefiles
              FsckOpt = "-y"
              )
       Mount
intracer-db0-mnt (
              MountPoint = "/shared_db"
              BlockDevice = "/dev/vx/dsk/db_dg/db_vol0"
              FSType = vxfs
              MountOpt = largefiles
              FsckOpt = "-y"
              )
       Mount
intracer-db1-mnt (
              MountPoint = "/shared_db"
              BlockDevice = "/dev/vx/dsk/db_dg/db_vol1"
              FSType = vxfs
              MountOpt = largefiles
              FsckOpt = "-y"
              )
       Mount
intracer-db2-mnt (
              MountPoint = "/shared_db"
              BlockDevice = "/dev/vx/dsk/db_dg/db_vol2"
              FSType = vxfs
              MountOpt = largefiles
              FsckOpt = "-y"
              )
       Mount
intracer-db3-mnt (
              MountPoint = "/shared_db"
              BlockDevice = "/dev/vx/dsk/db_dg/db_vol3"
              FSType = vxfs
              MountOpt = largefiles
              FsckOpt = "-y"
              )
       NIC
intracer-nic (
              Device = eth0
              )
       Volume
intracer-config-vol (
              DiskGroup = config_dg
              Volume = config_vol
              )
       Volume
intracer-config-vol0 (
              DiskGroup = config_dg
              Volume = config_vol0
              )
       Volume
intracer-config-vol1 (
              DiskGroup = config_dg
              Volume = config_vol1
              )
       Volume
intracer-config-vol2 (
              DiskGroup = config_dg
              Volume = config_vol2
              )
       Volume
intracer-config-vol3 (
              DiskGroup = config_dg
              Volume = config_vol3
              )
       intracer-app
requires intracer-config-mnt
       intracer-app
requires intracer-config-mnt
       intracer-app
requires intracer-db-mnt
       intracer-app
requires intracer-ip
       intracer-config-mnt
requires intracer-config-vol
       intracer-config-vol
requires intracer-config-dg
       intracer-db0-mnt
requires intracer-db-vol0
       intracer-db-vol0
requires intracer-db-dg
       intracer-db1-mnt
requires intracer-db-vol1
       intracer-db-vol1
requires intracer-db-dg
       intracer-db2-mnt
requires intracer-db-vol1
       intracer-db-vol2
requires intracer-db-dg
       intracer-db3-mnt
requires intracer-db-vol3
       intracer-db-vol3
requires intracer-db-dg
       intracer-ip
requires intracer-nic

Main.cf File Configuration

Below is the main.cf file for resource-groups and resources. Details are as below:Primary machine: pnqainpilot-x4270-1Secondary machine: pnqainpilot-x4270-2C-InTracer is installed at location: /xfs/InTracer/C-InTracer/R-TCE is installed at location: /xfs/InTracer/R-TCEUsername: abcd

main.cf file contents:

include "OracleASMTypes.cf
include "types.cf"
include "Db2udbTypes.cf"
include "OracleTypes.cf"
include "SybaseTypes.cf"
cluster suncluster (
    UserNames = { admin = dijBidIfjEjjHrjDig }
    Administrators = { admin }
    }
system pnqainpilot-x4270-1 (
    }
system pnqainpilot-x4270-2 (
    )

//C-InTracer configuration

group C-intracer-ha (
    SystemList = { pnqainpilot-x4270-1 = 1, pnqainpilot-x4270-2 = 2 }
    }
Application C-intracer-app (
    User = abcd
    StartProgram = "/xfs/InTracer/C-InTracer/ipmssys
start with-monitor"
    StopProgram = "/xfs/InTracer/C-InTracer/ipmssys
stop with-monitor"
    CleanProgram = "/xfs/InTracer/C-InTracer/ipmssys
stop with-monitor"
    PidFiles = { "/xfs/InTracer/C-InTracer/data/QS/.ipmsqshbd.pid" }
    }
DiskGroup C-intracer-dg (
    DiskGroup = config_dg
    }
IP C-intracer-ip (
     Device = eth0
     Address = "10.105.87.247"
     NetMask = "255.255.255.0"
     }
Mount C-intracer-mnt (
    MountPoint = "/shared_intracer_config"
    BlockDevice = "/dev/vx/dsk/config_dg/config_vol"
    FSType = vxfs
    MountOpt = largefiles
    FsckOpt = "-y"
    )
NIC C-intracer-nic (
    Device = eth0
    )
Volume C-intracer-vol (
    DiskGroup = config_dg
    Volume = config_vol
    }
C-intracer-app requires
C-intracer-ip
C-intracer-app requires
C-intracer-mnt
C-intracer-ip requires
C-intracer-nic
C-intracer-mnt requires
C-intracer-vol
C-intracer-vol requires
C-intracer-dg

//R-TCE Configuration

group R-TCE-ha (
   SystemList = { pnqainpilot-x4270-1 = 1, pnqainpilot-x4270-2 = 2 }
   )
   Application
R-tce-app (
      User = hryadav
      StartProgram = "/xfs/InTracer/R-TCE/ipmssys
start"
      StopProgram = "/xfs/InTracer/R-TCE/ipmssys
stop"
      CleanProgram = "/xfs/InTracer/R-TCE/ipmssys
stop"
      PidFiles = { "/xfs/InTracer/R-TCE/data/TCE/Ind/.ipmstcehbd.pid" }
      )
   DiskGroup
R-tce-dg (
      DiskGroup = db_dg
      )
   IP
R-tce-ip (
       Device = eth3
       Address = "10.105.87.248"
       NetMask = "255.255.255.0"
       )
   Mount
R-tce-mnt (
       MountPoint = "/shared_intracer_db"
       BlockDevice = "/dev/vx/dsk/db_dg/db_vol"
       FSType = vxfs
       MountOpt = largefiles
       FsckOpt = "-y"
       )
   NIC
R-tce-nic (
       Device = eth3
       )
   Volume
R-tce-vol (
       DiskGroup = db_dg
       DiskGroup = db_dg
       )
R-tce-app requires
R-tce-ip
R-tce-app requires
R-tce-mnt
R-tce-ip requires R-tce-nic
R-tce-mnt requires
R-tce-vol
R-tce-vol requires
R-tce-dg