Veritas Cluster Installation and Management

The cluster mode functionality enables ESS to provide high availability and critical redundancy support to retrieve CDRs in failure of any one of the systems. An ESS cluster comprises of two ESS systems, or nodes, that work together as a single, continuously available system to provide applications, system resources, and data to ESS users. Each ESS node on a cluster is a fully functional, standalone system. However, in a clustered environment, the ESS nodes are connected by an interconnected network and work together as a single entity to provide increased data availability.

The ESS application consists of internal entities such as the ESS process and process monitor which run on a machine and communicate with the external entities such as the ASR 5x00 chassis. Whenever the machine or ESS process fails, there are chances of loss of communication between internal and external entities. To avoid downtime and ensure continuous availability of ESS application, High Availability (HA) support using Veritas Clustering has been provided.

The hardware setup for Veritas Cluster Server (VCS) solution consists of two cluster nodes connected with an external shared storage. Both the cluster nodes are connected to the external storage. Cluster nodes must be installed with the Cisco MITG RHEL OS, Veritas Storage Foundation (Veritas Volume Manager and Veritas File System), and Veritas Cluster Server (for High Availability).

The Veritas Volume Manager (VxVM) can be used to create a single disk group (DG) containing multiple disks. Separate disk/LUN from the shared storage is required for I/O fencing. I/O fencing is part of the VCS administration. It is assumed that I/O fencing is already configured on the Veritas Cluster setup before the ESS application is installed for HA.

The cluster setup offers several advantages over traditional single-server systems. These advantages include:
  • Support for failover and scalable services
  • Capacity for modular growth
  • Low entry price compared to traditional hardware fault-tolerant systems
  • Reduce or eliminate system downtime because of software or hardware failure
  • Ensure availability of data and applications to ESS user, regardless of the kind of failure that would normally take down a single-server system.
  • Provide enhanced availability of the system by enabling you to perform maintenance without shutting down the entire cluster.
Following are the cluster components that work with ESS to provide this functionality:
  • ESS Cluster Node
    A ESS cluster node is a ESS server that runs both the ESS Application software and Cluster Agent software. The Cluster Agent enables carrier to network two ESS nodes in a cluster. Every ESS node in the cluster is aware when another ESS node joins or leaves the cluster. Also, every ESS node in the cluster is aware of the resources that are running locally as well as the resources that are running on the other ESS cluster nodes.
    Each ESS cluster node is a standalone server that runs its own processes. These processes communicate with one another to form what looks like (to a network client) a single system that co-operatively provides applications, system resources, and data to ESS users.
  • Common Storage System
    A common storage system is a fiber channel (FC) -based cluster storage with FC drives for the servers in the cluster environment. It is interconnected with ESS cluster nodes with carrier class network connectivity to provide high level redundant storage and backup support for CDRs. It serves as common storage for all connected ESS cluster nodes.
    This system provides high storage scalability and redundancy with RAID support.
This chapter includes the following topics:

Once the Veritas Volume Manager and Veritas Cluster are configured, install the ESS application on the ESS Server. For detailed instructions, refer to the ESS Installation and Configuration chapter of this guide. Then, configure the resources for high availability, and perform the cluster monitoring and rootdisk encapsulation processes.

ESS Cluster Functional Description

ESS clustering application provides the support to two discreet ESS servers for retrieving and storing xDRs from the chassis at a distribution node on a single IP address/network element for the billing system.

Both the ESS nodes (ESS1 and ESS2) are configured identically from the standpoint of the retrieval and storage of the xDRs to support the following:
  • The active ESS (either ESS1 or ESS2) is configured to retrieve xDRs from any and all local chassis in pre-defined intervals and the xDRs are stored on shared disk (between active and standby) by the active so that whenever active goes down and standby takes over, it has access to fetched data as data is on the shared disk.
  • The directory structure of both ESS1 and ESS2 is identical and conform to the carrier standards. A /fetched_data directory under <less_install_dir>/ess is used to store initial retrieval of the xDRs from the chassis.
From a process flow perspective, the interaction of the clustered ESS and the ECS is as follows:
  • The ESS(s) is statically configured with chassis to pull xDRs.
  • The chassis continually generates and groups individual records into xDRs, which are marked as a 'closed' xDR file based on pre-defined criteria.
  • The active ESS uses SFTP to access the chassis and retrieve all closed xDRs for storage in the /fetched_data directory.
  • Active ESS fetches xDR files for eventual retrieval by the billing system.

Installing Hardware

To install the hardware components required for the installation of ESS cluster:

  1. Rack the Sun Netra T5220 servers and storage array and connect power to each of them.
  2. Connect Ethernet port 0 on each server to an Ethernet switch.
  3. Connect Ethernet port 1 on server 1 to Ethernet port 1 on server 2 with a cross-over cable.
  4. Connect Ethernet port 2 on server 1 to Ethernet port 2 on server 2 with a cross-over cable.
  5. Connect a terminal (pc with terminal emulation such as HyperTerm) to the console port. Settings for the console are 9600 8, 1, N. Console cable and DB9 to RJ45 adapter are included with each server.
  6. Connect one SCSI cable from CH 0 on the Storage Array to Single Bus Conf as shown in the following figure. DO NOT make any connections to Sun Servers at this time.
  7. Connect the Ethernet ports on each array controller to an Ethernet switch.
  8. Insert install DVD into DVD-ROM in the first Sun server. Make sure the server is NOT cabled to the storage array.
  9. Power on the server.
  10. Wait for the ok prompt on the console.
  11. To boot the machine from the DVD, enter: ok> boot cdrom – install
  12. The install will run for some time. After the image has been loaded, you will be prompted for the host information shown below:
    # Please enter
    the desired hostname for this machine.
    
    # Please enter
    the desired IP address for bge0.
    
    # Please enter
    the netmask for bge0.
    
    # Please enter
    the default router for bge0.
    
  13. After entering hostname, IP address, netmask, and default router information, you must confirm the inputs.
    Please verify your configuration information:
    
       hostname: 
    
       ip:       
    
       netmask:  
    
       router:   
    
    Are these correct? (y/n)
    
  14. The machine will reboot, and comes up in multi-user mode.
  15. Log on as root with the corresponding password.
  16. Remove the “Boot/Install DVD” from the DVD-ROM.
  17. Set the Ethernet interface to full-duplex mode.
    1. Create the script /etc/rc2.d/S68net_tume as shown below:
      -------cut from here------
      
      #!/sbin/sh
      
      # /etc/rc2.d/S68net-tune
      
      PATH=/usr/bin:/usr/sbin
      
      echo "Implementing Solaris
      ndd Tuning Changes "
      
      # bge-Interfaces
      
      # Force bge0
      to 100fdx autoneg off
      
      ndd -set /dev/bge0
      adv_1000fdx_cap 0
      
      ndd -set /dev/bge0
      adv_1000hdx_cap 0
      
      ndd -set /dev/bge0
      adv_100fdx_cap 1
      
      ndd -set /dev/bge0
      adv_100hdx_cap 0
      
      ndd -set /dev/bge0
      adv_10fdx_cap 0
      
      ndd -set /dev/bge0
      adv_10hdx_cap 0
      
      ndd -set /dev/bge0
      adv_autoneg_cap 0
      
      -------end script-------
      
    2. Make the script executable.
      # chmod 755 /etc/rc2.d/S68net_tune
      
  18. Edit the file, /etc/ssh/sshd_config, and change the line, “#PermitRootLogin yes”, so that it reads, “PermitRootLogin yes”. This will only be a temporary change to allow remote access until user accounts are created.
  19. Restart the SSH daemon to make changes take effect.
    #/etc/init.d/sshd stop
    
    #/etc/init.d/sshd start
    
  20. Transfer the three script files to the /mnt directory on the server using FTP.
  21. Change the attributes of the scripts to allow execution.
    #cd /mnt
    
    #chmod 777 *.sh
    
  22. Execute the script, user_config.sh, and specify passwords for the users prompted.
    # ./user_config.sh
    
    Enter password for user
    ssmon.
    
    New Password:
    
    Re-enter new Password:
    
    passwd: password successfully
    changed for ssmon
    
    Enter password for user
    ssadmin.
    
    New Password:
    
    Re-enter new Password:
    
    passwd: password successfully
    changed for ssadmin
    
    Enter password for user
    ssconfig.
    
    New Password:
    
    Re-enter new Password:
    
    passwd: password successfully
    changed for ssconfig
    
    Enter password for user
    essadmin.
    
    New Password:
    
    Re-enter new Password:
    
    passwd: password successfully
    changed for essadmin
    
    Enter password for user.
    
    New Password:
    
    Re-enter new Password:
    
    passwd: password successfully
    changed for user
    
  23. Connect the storage array to server 1 only.
  24. Type the following command to reboot the server and make the storage array known to the server.
    #reboot --
    -r 
    
  25. Repeat Step 8 through Step 21 on the server 2.
  26. Execute the format command on both the servers, and verify if the drives are correctly labeled and cabled. For more detailed information, refer to the Sun Documentation.

Configuring Storage Array on Solaris

To configure the storage array using the graphical interface:

  1. Log on to a workstation, with an X Window server, with access to the machine to be installed.
  2. Start the X Window server. (Hummingbird Exceed)
  3. Using Putty (http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe), setup a new connection, with X11 forwarding enabled, to the server.
  4. Log on as root user with the corresponding password.
  5. Type the following commands:
    # exec bash
    
    # export DISPLAY=<local_IP_address>:0.0
    
    # /usr/openwin/bin/xhost +
    
  6. Invoke the Sun Storage Configuration GUI by typing the following command:
    #ssconsole
    
  7. Click Hide to terminate server discovery, if necessary.
  8. Click Server List Setup on the File menu of the Sun Storage Configuration Service Console to configure the server to monitor.
  9. Click Remove All to remove any old data from the list.
  10. Click Add to add a new server.
  11. Enter the name of the server being configured, its IP address, and the password that you set for the ssmon user in the fields, then click OK.
  12. If you do not want to set up the mail server for event notification, click No when the warning message appears.
  13. Select the server you just created in the Available Servers list, then click >Add> to add it to the Managed Servers list.
  14. Click Controller Assignment on the Array Administration menu.

    IMPORTANT:

    If array has previously been configured, quit the SSCONSOLE.

  15. Select the ID listed, then, in the pop-up at the bottom, select the name of the server, and click Apply.
  16. When prompted, enter the password for the ssadmin user that you selected earlier, and then click OK.
  17. Click Close.
  18. Double-click the server in the main dialog. Refer to the following figure for details.
  19. Double-click the array in the main dialog. Refer to the following figure for details.
  20. Select the array, and click Standard Configure on the Configuration menu.
  21. Enter the password for the ssconfig user that you selected earlier.
  22. Select RAID 5, then select Use a standby drive, and Write a new label to the new LD check boxes.
  23. Click OK to verify that you want to overwrite all data on the array.
  24. A progress dialog appears showing you the status of the array format.
  25. When complete, the below dialog appears. Click Close.
  26. Click Custom Configure on the Configuration menu.
  27. Select Change Controller Parameters.
  28. Click on Channel 1, and then click Change Settings.
  29. Click on 2 under Available SCSI IDs, then click >> Add SID >>, and click OK.
  30. Click on Channel 3, and then click Change Settings.
  31. Select 3 under Available SCSI IDs, then click >> Add PID >>, and click OK.
  32. Click Custom Configure on the Configuration menu.
  33. Select Change Host LUN Assignments.
  34. From Select Host Channel and SCSI ID, select Phy Ch 1(SCSI) – PID 0. Under Partitions, select LD 0, then click Assign Host LUN, and OK.
  35. Repeat the same for Phy Ch 3(SCSI) – PID 3, and assign LD 0 to it.
  36. Click Custom Configure on the Configuration menu.
  37. Select Change Controller Parameters.
  38. Click on Network tab of the Change Controller Parameters screen, and then click Change Settings.
  39. Enter the IP address for the array, and subnet mask, then click OK.
  40. Click Custom Configure on the Configuration menu.
  41. Select Make or Change Standby Drives.
  42. Click the radio button next to, Local Standby for LD#, and make sure that the popup has 0 shown, then click Apply.
  43. Quit ssconsole.

Configuring Storage using CAM

Installing the Management Software (CAM)

Sun Storage Common Array Manager (CAM) provides an easy way to manage storage environment. It provides a common, simple-to-use, interface for Sun Storage arrays. It can be downloaded from www.oracle.com. Once you copy the storage software on a machine, please make sure that following directories and files have execute permissions.

  • linux/util/*
  • linux/bin/tools/*
  • linux/components/lockhartLunux/*
  • linux/RunMe.bin
[root@intracerR
CAM_linux]# ./RunMe.bin -c




          Initializing
Wizard........
          Launching
InstallShield Wizard........


-------------------------------------------------------------------------------
Sun StorageTek(TM)
Common Array Manager 6.2


The InstallShield Wizard
will install Sun StorageTek(TM)
Common Array Manager
on your computer.


To continue, choose
Next.


     Sun
StorageTek(TM) Common Array Manager 6.2
     Sun
Microsystems, Inc.
     http://www.sun.com




Press 1 for Next, 3
to Cancel or 5 to Redisplay [1] 


-------------------------------------------------------------------------------
Sun StorageTek(TM)
Common Array Manager 6.2


Please read the following
license agreement carefully.


Sun StorageTek(TM)
Common Array Manager


Copyright 2008 Sun
Microsystems, Inc. All rights reserved. Sun Microsystems,
Inc. has intellectual
property rights relating to technology embodied in the
product that is described
in this document. In particular, and without
limitation, these intellectual
property rights may include one or more of the
U.S. patents listed
at http://www.sun.com/patents and one or
more additional
patents or pending
patent applications in the U.S. and in other countries. U.S.
Government Rights -
Commercial software. Government users are subject to the
Sun Microsystems, Inc.
standard license agreement and applicable provisions of
the FAR and its supplements.
Use is subject to license terms. This distribution
may include materials
developed by third parties. Portions may be derived from
Berkeley BSD systems,
licensed from U. of CA. Sun, Sun Microsystems, the Sun
logo, Java, Solaris
and Sun StorageTek Common Array Manager are trademarks or
registered trademarks
of Sun Microsystems, Inc. in the U.S. and other
countries. All SPARC
trademarks are used under license and are trademarks or
registered trademarks
of SPARC International, Inc. in the U.S. and other
countries.


Please choose from
the following options:
[ ] 1
- I accept the terms of the license agreement.
[X] 2
- I do not accept the terms of the license agreement.


To select an item enter
its number, or 0 when you are finished: [0] 1


[X] 1
- I accept the terms of the license agreement.
[ ] 2
- I do not accept the terms of the license agreement.


To select an item enter
its number, or 0 when you are finished: [0] 


Press 1 for Next, 2
for Previous, 3 to Cancel or 5 to Redisplay [1] 
-------------------------------------------------------------------------------
Sun StorageTek(TM)
Common Array Manager 6.2


Choose the installation
type that best suits your needs.


[X] 1
- Typical
        The
program will be installed with the suggested configuration.
        Recommended
for most users.
                                                                                 
[ ] 2
- Custom
        The
program will be installed with the features you choose.
        Recommended
for advanced users.


Select the number corresponding
to the type of install you would like: [0] 
Press 1 for Next, 2
for Previous, 3 to Cancel or 5 to Redisplay [1] 


-------------------------------------------------------------------------------
Checking current system
...


|-----------|-----------|-----------|------------|
0%         25%         50%         75%        100%
||||||||||||||||||||||||||||||||||||||||||||||||||
-------------------------------------------------------------------------------
Sun StorageTek(TM)
Common Array Manager 6.2


Software To Be Installed:
Full Install
* Browser
User Interface (BUI)
* Local and
Remote CLI
* Array Firmware




Press 1 for Next, 2
for Previous, 3 to Cancel or 5 to Redisplay [1] Preparing
for installation ...


Pre Uninstall Old Action
...


Removing old features
...


-------------------------------------------------------------------------------
Sun StorageTek(TM)
Common Array Manager 6.2


Installing Sun StorageTek(TM)
Common Array Manager 6.2. Please wait...


|-----------|-----------|-----------|------------|
0%         25%         50%         75%        100%
||||||||||||||||||||||||||||||||||||||||||||||||||


Installing Java 2 Standard
Edition


-------------------------------------------------------------------------------
Sun StorageTek(TM)
Common Array Manager 6.2


|-----------|-----------|-----------|------------|
0%         25%         50%         75%        100%
||||||||||||||||||||||||||||||||||||||||||||||||||
-------------------------------------------------------------------------------
Sun StorageTek(TM)
Host Software Installation Summary


View results:
Info:
Installation success.
The following have
been installed: Browser User Interface (BUI), Local and
Remote CLI, and Array
Firmware.
To access the Browser
User Interface point a browser at:
https://installation_host:6789
The logs may be found
in /var/opt/cam/




Press 3 to Finish or
5 to Redisplay [3] 


Accessing the Storage Management GUI

Follow these steps to access the storage management GUI.

  1. Access the Management GUI using a browser on PC (If 65.198.111.26 is the public IP of the node on which the management software was installed) https://65.198.111.26:6789
  2. The first time login to the CAM software is always through the admin user of the operating system. For example, Administrator on Windows and root on the Unix/Linux.

Installing the hardware

Connect the hardware as shown in the figure. There are two controllers, A and B, on the storage array. Connect both the controllers to each node. They can either be directly connected or they can be connected via switch as shown.

The following is a figure of the StorageTek 2540 Array Direct-Connect Configuration:

The following is a figure of the StorageTek 2540 Array Switched Configuration:

Also, connect the management console to the network. This console will be used to detect the storage and configure it via CAM.

Configuring the Storage System

Perform the following steps to configure the storage system:

  1. Discover the storage system by either clicking Storage Systems -> Register -> Scan the local network or you can specify the IP address of the storage.
  2. Once the storage is added to CAM, create a pool for LESS volume by selecting < storage_name > -> Pools -> New.
  3. In the form, enter the following details:
    1. Name: LESSPool
    2. Description: Storage Pool for LESS
    3. Storage Profile: RAID5-256KB-ReadAhead
    Click OK. If this profile is not available, add a new profile with these values.
  4. Create a volume and map it by selecting < storage_name > -> Volumes -> New.
  5. From the pop-up window, select LESSPool, that was created in Step 3, and click Next.
  6. Select Storage Selected Automatically by CAM and click Next.
  7. In the form, enter the following volume parameters:
    1. Volume Name: LESSVol
    2. Number to Create: 1
    3. Size: Select either Fill one Virtual Disk or Specify size.
    4. Controller: Any
    Click Next.
  8. To limit the disk visibility to a particular set of hosts, map the volume to a particular Host Group. Otherwise, select Map to an Existing Host/Host Group or the Default Storage Domain. Click Next.
  9. Select the Default Storage Domain. Click Next.
  10. Check the parameters and click Finish.

Configuring Veritas Volume Manager and Veritas Cluster

To configure the Veritas Volume Manager:

  1. Start the installation by entering the following command:
    vxinstall
    
    The following prompt appears:
    Are you sure that
    you want to reinstall [y,n,q,?] (default: n)
    
    Type y if you want to reinstall the Volume Manager.
  2. Type y to review the licenses that are already installed when the following prompt appears:
    Some licenses are
    already installed. Do you wish to review them [y,n,q] (default:
    y)
    
  3. Type y when prompted for entering another license key, if necessary. Then enter the key data for this server.

    IMPORTANT:

    The key on the Install DVD is a demo key and will expire in 60 days.

  4. Press Enter to accept the default of n in the following prompt:
    Do you want to use
    enclosure based name for all disks? [y,n,q,?] (default:
    n)
    
  5. Press Enter to accept the default of y in the following prompt:
    Do you want to setup
    a system wide default disk group? [y,n,q,?] (default:
    y)
    
    Enter a default disk group name of rootdg. The installation of Veritas Volume Manager is now successfully completed.
  6. Repeat Step 1 through Step 5 on the server 2.
  7. Copy the provided default LLT and GAB configuration files to configure LLT and GAB. Type the following:
    # for file
    in llttab gabtab llthosts; do cp /etc/$file.server1 /etc/$file;
    done
    
  8. Edit /etc/llttab, if there are more than one cluster on the network. If this is the case, change the cluster ID, after set-cluster to a unique integer. If not, no changes are necessary.
  9. Edit /etc/llthosts, and replace less1 with the name of the first server in the pair, and less2 with the name of the second server.
  10. Restart LLT and GAB, by executing the following commands:
    #/etc/init.d/gab stop
    
    #/etc/init.d/llt.rc stop
    
    #/etc/init.d/llt.rc start
    
    #/etc/init.d/gab start
    
  11. Repeat Step 7 through Step 10 on second server.
  12. Verify that both servers’ cluster communication modules see each other by typing the following command:
    gabconfig –a
    
    If you see a line with membership “01”, then both nodes are talking. If you see the message as membership “;1” or “0;” then the node that has the “;” (semi-colon) is misconfigured. Verify your configurations in the /etc/llttab, /etc/llthosts, and /etc/gabtab files.
  13. Type hastart and then hastatus. When the last line reads “<system name> RUNNING”, VCS engine has started. If you get “VCS ERROR V-16-1-10600 Cannot connect to VCS engine” repeatedly, the VCS engine has failed to start. Refer to the /var/VRTSvcs/log/engine_A.log for possible problems.
  14. Execute the first VCS configuration script, /mnt/vcs_basic.sh, from the configuration DVD. Enter the data for cluster node names, cluster name, virtual IP address and virtual netmask. A lot of warning messages will be displayed, but there should be no errors.
  15. Stop the cluster by typing the following command:
    root@less4 # hastop –all
    
    Then, copy the new types.cf from the /mnt directory to /etc/VRTSvcs/conf/config.
  16. Regenerate and populate the config on both the nodes by executing the following commands:
    root@less4 # hacf –verify /etc/VRTSvcs/conf/config/
    
    root@less4 # hacf –generate /etc/VRTSvcs/conf/config/
    
    Then execute hastart on both the nodes.
  17. Validate that the cluster has probed all resources by waiting until the hastatus -sum command output looks similar to the following:
    root@less4 # hastatus
    -sum
    
    -- SYSTEM STATE
    
    -- System               State            Frozen
    
    A  less3               RUNNING              0
    
    A  less4               RUNNING              0
    
    -- GROUP STATE
    
    -- Group          System        Prob       AutoDisabled    State
    
    B  LAPP            less3         Y              N          OFFLINE
    
    B  LAPP            less4         Y              N          OFFLINE
    
  18. Create the resource groups for ESS application and define the dependencies between the resource groups as outlined below.
    1. Change the VCS configuration as rewritable.
      haconf -makerw
      
    2. Add the application resource group and change the required attributes.
      hares -add <APP_RES> Application lesssg
      
      hares -modify <APP_RES> Enabled 1
      
      hares -modify <APP_RES> PidFiles <PSMON_PID_PATH> FOR EG /users/ess/psmon.pid
      
      hares -modify <APP_RES> StartProgram FOR EG /users/ess/start_serv.sh
      
      hares -modify <APP_RES> StopProgram FOR EG
      "/users/ess/serv forcestop"
      
      hares -modify <APP_RES> User <USERNAME> For EG essadmin
      
    3. Now add the dependencies.
      hares -link <APP_RES> LESS-VIP
      
      hares -link <APP_RES> lessmount
      
    4. To verify the dependencies, enter the following command:
      hares -disp <APP_RES>
      
      The output for this command will display all the required attributes.
       hares -dep <APP_RES>
      
      This command displays the dependencies of the application.
    5. Shutdown the ESS.
    6. Now bring the APP_RES online by entering the following command:
      hares -online <APP_RES> -sys <NODE_NAME>
      
    7. Once the application is up and running, dump the configuration using the following command:
      haconf -dump makero 
      
      Also, perform a cat of main.cf file. A sample output is shown below.
              Application
      T5220-Application (
      
                      User = root
      
                      StartProgram = "/less/LESS/ess/start_serv.sh"
      
                      StopProgram = "/less/LESS/ess/serv forcestop"
      
                      PidFiles = { "/less/LESS/ess/psmon.pid" }
      
                       ) 
      
               // resource
      dependency tree
      
              //
      
              // 
          group T5220-SG
      
              //
      
            {
      
              // 
          Application T5220-Application
      
              // 
              {
      
              // 
              IP T5220-IP
      
              // 
                  {
      
              // 
                  NIC T5220-NIC
      
              // 
                  }
      
             // 
              Mount T5220-mount
      
              // 
                  {
      
              // 
                  Volume T5220-Vol
      
              // 
                      {
      
              // 
                      DiskGroup T5220-DG
      
              // 
                      }
      
              // 
                  }
      
              // 
              }
      
              // 
          }
      
    8. Perform a switch over and verify that the application comes up on the other node as well.
      hagrp -offline <RG_name> -sys <Node1>
      
      hagrp -online <RG_name> -sys <Node2>
      
  19. Validate that it is online on the first node.
    root@less4 # hastatus
    -sum
    
    -- SYSTEM STATE
    
    -- System            State            Frozen
    
    A  less3            RUNNING             0
    
    A  less4            RUNNING             0
    
    -- GROUP STATE
    
    -- Group         System        Probed       AutoDisabled        State
    
    B  LAPP          less3           Y               N              ONLINE
    
    B  LAPP          less4           Y               N              OFFLINE
    
  20. Enable the SNMP traps/alarms and edit the main.cf file under /etc/VRTSvcs/conf/config to add the following entries:
    NotifierMngr LAPP-Notif-Mgr
    (
    
                 SnmpdTrapPort = 162
    
                 SnmpConsoles = { "<SNMP_IP_Address>" = Information }
    
                 )
    
  21. Verify and regenerate the new config.
    root@less3 # hacf –verify /etc/VRTSvcs/conf/config/
    
    root@less3 # hacf –generate /etc/VRTSvcs/conf/config/
    
    Then, restart both the nodes by executing the following command:
    root@less3 # hastart
    
    root@less4 # hastart
    
    For more details on the installation of Veritas Volume Manager and Veritas Cluster configuration, refer to the Veritas Documentation.

Tuning the VxFS File System for Better Performance

The VxFS file system can be tuned for better performance using the vxtunefs command to set the tuning parameters. The default values of these parameters are set when the volume is mounted.

The performance of the ESS application can improve when the following tuning parameters are changed:

  • read_pref_io: The preferred read request size. The filesystem uses this in conjunction with the read_nstream value to determine how much data to read ahead. The default value is 64000. The ESS performance can improve when this value is set to 128000.
  • read_nstream: This is the desired number of parallel read requests of the size specified in the read_pref_io parameter to have outstanding at one time. The file system uses the value specified in the read_nstream parameter multiplied by the value specified in the read_pref_io parameter to determine its read ahead size. The default value for the read_nstream parameter is 1. If you know the hardware RAID configuration on the external storage, then set the read_nstream parameter value to be the number of columns (disks) in the disk array.
  • write_pref_io: The preferred write request size. The filesystem uses this in conjunction with the value specified in the write_nstream parameter to determine how to flush behind on writes. The default value is 64000. The ESS performance can improve when this value is set to 128K.
  • write_nstream: This is the desired number of parallel write requests of the size specified in the write_pref_io parameter to have outstanding at one time. The file system uses the value specified in the write_nstream parameter multiplied by the value specified in the write_pref_io parameter to determine when to flush behind on writes. The default value for the write_nstream parameter is 1. For disk striping configurations, set the value of the write_pref_io and write_nstream parameters to the same values as the read_pref_io and read_nstream parameters.

Use the following command to tune Veritas file system:

$ /opt/VRTS/bin/vxtunefs
-o read_pref_io=131072,read_unit_io=131072,write_pref_io=131072,write_unit_io=131072 /shared_db

To ensure that these values are not lost after a reboot, create the file /etc/vx/tunefstab using the following command:

$ cat /etc/vx/tunefstab /dev/vx/dsk/apps_dg/apps_vol read_pref_io=131072,read_unit_io=131072,write_pref_io=131072,write_unit_io=131072
command.

Configuring Resources for High Availability

After installation of the storage array, Veritas cluster, and the ESS server, the following resources need to be configured with the Veritas cluster:

  • NIC — To monitor an NIC (Network Interface Card)
  • IP — To monitor an IP address
  • Disk Group, Volume, and Mount — for shared storage
  • ESS Application — comprising of all the ESS-related processes
  • Volume — With apps_vol mounted on the /shared_app directory
  • ESS installation directory — /shared_apps/less
  • ESS Administrator — lessadmin
  • Shared/Floating IP address (on NIC eth0)

Figure 1. Resources for high availability

To configure these resources:

IMPORTANT:

The following configurations should be performed only on the node where the ESS application is installed.

  1. Log on as super user (root).
  2. Make the veritas config file writable using the following command:
    $ haconf -makerw
    
  3. Create resource group using the following commands:
    $hagrp -add less-ha
    
    $hagrp –modify
    less-ha SystemList  <Node1> 0<Node2> 1
    
    $hagrp –modify
    less-ha NumRetries 1
    
    Where, Node1 and Node2 are the hostnames of the active and passive nodes.
  4. Create Disk Group resource for the ESS partition using the following commands:
    $ hares -add less-apps-dg
    DiskGroup less-ha
    
    $ hares -modify less-apps-dg
    DiskGroup apps_dg
    
    $ hares -modify less-apps-dg
    Enabled 1
    
  5. Create Volume resource for the ESS partition using the following commands:
    $ hares -add less-apps-vol
    Volume less-ha
    
    $ hares -modify less-apps-vol
    DiskGroup apps_dg
    
    $ hares -modify less-apps-vol
    Volume apps_vol
    
    $ hares -modify less-apps-vol
    Enabled 1
    
  6. Create Mount resource for the ESS partition using the following commands:
    $ hares -add less-apps-mnt
    Mount less-ha
    
    $ hares -modify less-apps-mnt
    MountPoint /shared_apps
    
    $ hares -modify less-apps-mnt
    BlockDevice /dev/vx/dsk/apps_dg/apps_vol
    
    $ hares -modify less-apps-mnt
    FSType vxfs
    
    $ hares -modify less-apps-mnt
    FsckOpt %-y
    
    $ hares -modify less-apps-mnt
    MountOpt largefiles
    
    $ hares -modify less-apps-mnt
    Enabled 1
    
  7. Create Application resource for the ESS processes using the following commands:
    $ hares -add less-app
    Application less-ha
    
    $ hares -modify less-app
    User lessadmin
    
    $ hares -modify less-app
    StartProgram "/<installed dir>/ess/serv
    start"
    
    $ hares -modify less-app
    StopProgram "/<installed dir>/ess/serv
    forcestop"
    
    $ hares -modify less-app
    PidFiles "<installed dir>/ess/psmon.pid"
    
    $ hares -modify less-app
     Enabled 1
    
  8. Create the NIC resource using the following commands:
    $ hares -add less-nic
    NIC less-ha
    
    $ hares -modify less-nic
    Device eth0
    
    $ hares -modify less-nic
    Enabled 1
    
  9. Create the IP resource using the following commands:
    $ hares -add less-ip
    IP less-ha
    
    $ hares -modify less-ip
    Device eth0
    

    IMPORTANT:

    The floating or shared IP address should be a public IP in the DNS to which the client machine can successfully ping.

    $ hares -modify less-ip
    Enabled 1
    
  10. Set the resource dependencies using the following commands:
    $ hares -link less-app
    less-apps-mnt
    
    $ hares -link less-apps-mnt
    less-apps-vol
    
    $ hares -link less-apps-vol
    less-apps-dg
    
    $ hares -link less-app
    less-ip
    
    $ hares -link less-ip
    less-nic
    
  11. Dump the configuration to the Veritas config file using the following command:
    $ haconf -dump -makero
    
  12. Bring the ESS HA application online on Node 1 using the following command:
    $ hagrp -online less-ha
    -sys <Node1>
    

Once the above steps are performed, the ESS HA application will start running.

Creating Disk Group for ESS

Use the following instructions to create disk groups for ESS:

  1. Rebuild the disk lists with the new disks detected by the kernel using the following command:
    $ vxdctl initdmp
    
    $ vxdctl enable
    
    To see the status of the new disk, use the command:
    $ vxdisk -o alldgs list
    
    DEVICE       
         TYPE         
              DISK             GROUP      
         STATUS
    
    disk_0       
         auto:none       
        -                -     
              online invalid
    
    disk_1       
         auto:none       
         -               -     
              online invalid
    
    disk_2       
         auto:none       
        -                   -       
           online invalid
    
    disk_3       
         auto:none       
        -                -         
         online invalid
    
    emc_clariion0_28    auto      
             -              -  
                error
    
    emc_clariion0_29    auto
                   -     
           -                error
    
  2. To setup the disks, use the following commands:
    $ /etc/vx/bin/vxdisksetup
    -i emc_clariion0_28
    
    $ /etc/vx/bin/vxdisksetup
    -i emc_clariion0_29
    
    To see the status of the new disk, use the command:
    $ vxdisk -o alldgs list
    
    DEVICE       
         TYPE       
              DISK             GROUP      
         STATUS
    
    disk_0       
         auto:none       
         -                -     
              online invalid
    
    disk_1       
         auto:none        
        -                -     
              online invalid
    
    disk_2       
         auto:none       
         -                -          
         online invalid
    
    disk_3       
         auto:none       
         -                -      
             online invalid
    
    emc_clariion0_28    auto:cdsdisk    
      -              -     
              online
    
    emc_clariion0_29     auto:cdsdisk     
      -                -     
             online
    
  3. With the newly initialized disks, create disk groups for ESS using the following command:
    $ vxdg init apps_dg apps_dg01=emc_clariion0_28
    

    IMPORTANT:

    You can specify more disks (using Step 2) and add the disk to disk groups.

    VxVM will ensure that the newly created disk groups are visible from both the cluster nodes. These disk groups can be used only from one node at a time. You will have to import/deport a disk group from either node to use the disk groups and their volumes.
  4. Create volumes in the disk groups using the following command:
    $ vxassist -g apps_dg
    make apps_vol 299g
    
  5. Initialize the volumes with the VxFS file system using the following command: On Solaris:
    $ mkfs -F vxfs -o bsize=4096,largefiles /dev/vx/rdsk/apps_dg/apps_vol
    
    On Linux:
    $ mkfs -t vxfs -o bsize=4096,largefiles /dev/vx/rdsk/apps_dg/apps_vol
    
    For better performance, use a 4Kb block size and enable support for large files (more than 1 TB).
  6. Create the mount points and mount the volumes using the following commands: On Solaris:
    $ mount -F vxfs -o largefiles /dev/vx/dsk/apps_dg/apps_vol /shared_apps
    
    On Linux
    $ mount -t vxfs -o largefiles /dev/vx/dsk/apps_dg/apps_vol /shared_apps
    

Monitoring Veritas Cluster

To monitor the status of the Veritas cluster:

  1. Create the following script under /export/home/scripts to monitor the status of the cluster.
    #!/bin/sh
    
    ## script
    to monitor the status of Veritas... if both nodes are offline, force
    to start 
    
    ## the
    number one (1) node.
    
    ## put
    this in crontab as:
    
    ## 
          0,15,30,45 * * * * /export/home/scripts/keep_vcs_active.sh
    2>>/var/adm/messages
    
    VCS=' [VCS] == '
    
    CHECK_VCS=`hastatus
    -sum|grep -c ONLINE`
    
    DATE=`date
    "+%m/%d/%Y %T"`
    
    echo $DATE $VCS
    "Checking for Veritas started..." >> /var/adm/messages
    
    if [ ${CHECK_VCS} -eq
    0 ]
    
    then
    
      DATE=`date
    "+%m/%d/%Y %T"`
    
      echo $DATE $VCS
    "Both nodes are offline... Making first node active" >> /var/adm/messages
    
      hagrp -clear LAPP
    >> /var/adm/messages
    
      hagrp -online LAPP
    -sys less3
    
      DATE=`date
    "+%m/%d/%Y %T"`
    
      echo $DATE $VCS
    "First node activated. " >> /var/adm/messages
    
    else
    
      DATE=`date
    "+%m/%d/%Y %T"`
    
      echo $DATE $VCS
    "Veritas is running normally." >> /var/adm/messages
    
    fi
    
  2. Change the permission of the script to make it executable.
    # chmod 755 /export/home/scripts/keep_vcs_active.sh
    
  3. Copy the cron jobs to a file by executing the following command:
    # crontab –l
    > /tmp/CRON
    
  4. Edit the file and add the following line:
    0,15,30,45 * * * * /export/home/scripts/keep_vcs_active.sh
    2>>/var/adm/messages
    
  5. Activate the new cron jobs.
    # crontab /tmp/CRON
    
    On executing the script, the status of the Veritas cluster can be verified every 15 minutes.

Setup of rootdisk Encapsulation and Mirroring

To setup the rootdisk encapsulation and mirroring:

  1. Log on as super user (root) on the first server. Execute format, and then select disk “1”.
  2. Type y when prompted for labeling the disk.
  3. Exit format, by pressing CTRL+D.
  4. Execute vxdiskadm, and select option “2”.
  5. Type list to see a list of available disks.
  6. Select the rootdisk, “c1t0d0” for encapsulation.
  7. Verify that the rootdisk is the desired disk to encapsulate.
  8. Specify rootdg, the default, as the disk group to add the rootdisk to. Then, confirm that rootdg should be created.
  9. When prompted, type n to specify that you do not wish to use the default disk name for the rootdisk. Then, confirm that you wish to continue with the encapsulation.
  10. Enter a name for the rootdisk, and press ENTER to accept the default private region length.
  11. The rootdisk has been configured for encapsulation. Reboot the server using the following command:
    #shutdown –g0 –y –i6
    
  12. Repeat Step 1 through Step 11 on the second server.
  13. Initialize the second disk that was formatted in Step 2.
    # vxdisksetup
    -i c1t1d0 format=sliced
    
  14. Add the disk to the rootdg disk group.
    # vxdg -g
    rootdg adddisk rootmirror=c1t1d0s2
    
  15. Initialize the mirror process by executing the following command:
    #  vxrootmir
    -v rootmirror
    
    The output should be similar to the following:
    ! vxassist -g bootdg
    mirror rootvol layout=contig,diskalign rootmirror
    
    ! vxbootsetup -g bootdg
    -v rootmirror
    
    ! vxmksdpart -f -v -g
    rootdg rootmirror-01 0 0x2 0x200
    
    ! vxpartadd /dev/rdsk/c1t1d0s2
    0 0x2 0x200 20352 62918208
    
    ! /usr/sbin/installboot /usr/platform/SUNW,Netra-T5220/lib/fs/ufs/bootblk /dev/rdsk/c1t1d0s0
    
    ! vxeeprom devalias
    vx-rootmirror /dev/dsk/c1t1d0s0
    

    IMPORTANT:

    This process will take a long time. It can be monitored from another terminal by typing vxtaskmonitor.

  16. Mirror the additional volumes with the following command:
    #  vxmirror
    rootdisk rootmirror
    
    The output should be as follows:
    ! vxassist -g defaultdg
    mirror swapvol rootmirror
    
    ! vxassist -g defaultdg
    mirror export rootmirror
    

    IMPORTANT:

    This process will take a long time. It can be monitored from another terminal by typing vxtaskmonitor.

  17. Eject the DVD from the drive, by typing /usr/sbin/umount /mnt and eject /dev/dsk/c0t0d0s0.
  18. When complete, shut down both the nodes, by typing, hastop –all –force. Then, type hacf –verify /etc/VRTSvcs/conf/config/ to verify the cluster configuration. Next, type hastart on node 1, then hastatus. On node 2, type hastart. The cluster should then start, and can be monitored in the hastatus window open on node 1. Refer to /var/VRTSvcs/log/engine_A.log if you get any errors.

Testing Veritas Cluster

There are two ways to check the status of the cluster, either interactive or by its summary. To verify the status, type the following:

root@LESS1 # hastatus
attempting to connect....connected
group           resource
            system               message             
-------- --------------------
-------------------- --------------------
                   
                 JPTRFLGN-LESS1       RUNNING             
                   
                 JPTRFLGN-LESS2       RUNNING             
LAPP               
                 JPTRFLGN-LESS1       OFFLINE             
LAPP               
                 JPTRFLGN-LESS2       ONLINE              
-----------------------------------------------------------------------
LAPP               
                 JPTRFLGN-LESS1       OFFLINE             
LAPP               
                 JPTRFLGN-LESS2       ONLINE              
                LAPP-app-ess
        JPTRFLGN-LESS1       OFFLINE             
                LAPP-app-ess
        JPTRFLGN-LESS2       ONLINE              
                LAPP-vmdg-lessdg
    JPTRFLGN-LESS1       OFFLINE             
-----------------------------------------------------------------------
                LAPP-vmdg-lessdg
    JPTRFLGN-LESS2       ONLINE              
                LAPP-ip-vip_ext
     JPTRFLGN-LESS1       OFFLINE             
                LAPP-ip-vip_ext
     JPTRFLGN-LESS2       ONLINE              
                LAPP-mnt-less
       JPTRFLGN-LESS1       OFFLINE             
                LAPP-mnt-less
       JPTRFLGN-LESS2       ONLINE              
-----------------------------------------------------------------------
                LAPP-mnt-less-bk
    JPTRFLGN-LESS1       OFFLINE             
                LAPP-mnt-less-bk
    JPTRFLGN-LESS2       ONLINE              
                LAPP-nic-bge0
       JPTRFLGN-LESS1       ONLINE              
                LAPP-nic-bge0
       JPTRFLGN-LESS2       ONLINE              
                LAPP-app-ess
        JPTRFLGN-LESS1       OFFLINE             
-----------------------------------------------------------------------
                LAPP-app-ess
        JPTRFLGN-LESS2       ONLINE              
                LAPP-vmdg-lessdg
    JPTRFLGN-LESS1       OFFLINE             
                LAPP-vmdg-lessdg
    JPTRFLGN-LESS2       ONLINE              
                LAPP-ip-vip_ext
     JPTRFLGN-LESS1       OFFLINE             
                LAPP-ip-vip_ext
     JPTRFLGN-LESS2       ONLINE              
-----------------------------------------------------------------------
                LAPP-mnt-less
       JPTRFLGN-LESS1       OFFLINE             
                LAPP-mnt-less
       JPTRFLGN-LESS2       ONLINE              
                LAPP-mnt-less-bk
    JPTRFLGN-LESS1       OFFLINE             
                LAPP-mnt-less-bk
    JPTRFLGN-LESS2       ONLINE              
                LAPP-nic-bge0
       JPTRFLGN-LESS1       ONLINE              
-----------------------------------------------------------------------
                LAPP-nic-bge0
       JPTRFLGN-LESS2       ONLINE              

IMPORTANT:

This will continue to gather the status unless interrupted by pressing Ctrl+C key.

root@LESS1 # hastatus –sum
-- SYSTEM STATE
-- System               State            Frozen
A  JPTRFLGN-LESS1      RUNNING             0
A  JPTRFLGN-LESS2      RUNNING             0
-- GROUP STATE
-- Group         System            Probed      AutoDisabled      State
B  LAPP         JPTRFLGN-LESS1       Y              N           OFFLINE
B  LAPP         JPTRFLGN-LESS2       Y              N           ONLINE

ESS Cluster Failure Handling

The ESS clustering application is configured to monitor the health of both the hardware and software components of the ESS(s). The most typical error conditions that are accounted for, along with the expected resolution, are as follows:

  • Failure of a physical interface on the active ESS
    • In this case, communication will be shifted to the redundant interface on the active ESS.
  • Failure of a software process/application on the active ESS
    • In this case, the software process will be attempted to restart.
    • If this cannot be achieved in a reasonable time frame a switchover to the standby ESS will be initiated.
  • Failure of the redundant private interconnect between the active and standby ESS
    • In this case, the node with maximum quorum votes will become active node and the other will be rebooted in the standalone mode.
  • Failure of the active ESS as a whole (e.g. power failure)
    • In this case, a switchover to the standby ESS will be initiated.

Any and all failure scenarios, be it software or hardware, will be handled in a manner such that from the network/billing system perspective, the ESS is always reachable with a consistent set of directory structures and contents.