WEM High Availability Redundancy Installations

Systems that rely on running Web Element Manager on a single server to manage their networks face the possibility of service disruption should the server fail. By using Symantec Veritas Cluster Software (VCS), it is now possible to create redundant Web Element Manager servers with a primary host server running an active instance of Web Element Manager, and a redundant server in standby mode. This appendix provides information to help you successfully configure redundant instances of Web Element Manager over multiple servers. This appendix works closely with the Installing the WEM Software and the WEM Port and Hardware chapters in this guide, and with the VCS user documentation provided by Symantec.

This section provides instructions specific to a Symantec VCS installation to provide redundancy to multiple WEM servers. This software is documented by Symantec, and you will also need to refer to the Install, the Uninstall, and the Upgrade chapters in this guide. Server hardware requirements are in theWEM Port and Hardware Information chapter.

IMPORTANT:

Veritas Cluster is supported on both Sun servers using the Solaris Operating System and Cisco UCS servers using the RHEL OS. The VCS installation in this section is directed to installments on the RHEL platform.

IMPORTANT:

There are configuration changes required when installing WEM in Failover Mode rather than Standalone Mode. These are described in the Installing the WEM Software chapter. For this release, please use the command line to perform the installation rather than the GUI.

IMPORTANT:

During backup or restore functions, if failover happens, postgres stops functioning. As a result, backup or restore is done partially and these functions need to be recalled.

Installation

Following are the installation steps on Active Node and Standby Node.

Active Node

  1. Configure HA resources as mentioned in the Configuring HA Group section. Make sure they are all online except application resource.
    $ hares -state
    
  2. Disable the WEM resource group on standby node.
    $ hagrp -disable <Resource_group>
    
  3. Install WEM in Veritas Cluster Mode in Active Node as follows ( below is an example):
    cd <EMS_Package>/
    
    $ ./inst
    -console 
    
    InstallShield Wizard
    
     Initializing InstallShield
    Wizard...
    
    Preparing Java(tm) Virtual
    Machine...
    
    ...................................
    
    ...................................
    
    ...................................
    
    ...................................
    
    ...................................
    
    ...................................
    
    ...................................
    
    --------------------------------------
    
    Please read the information
    below.
    
    Welcome to the InstallShield
    Wizard for Element Management System.
    
    The Installshield Wizard
    will install Element Management System on your computer. 
    
    To continue, choose
    Next.
    
    Element Management System
    
    Cisco Systems
    
    http://www.cisco.com 
    
    Press 1 for Next panel,
    3 to Cancel or 4 to Redisplay [1]
    
    Deleting old temporary
    installed ems files...
    
    --------------------------------------
    
    EMS Installation Mode
    
    [ ] 1
    - Standalone EMS
    
    [X] 2
    - Cluster Failover
    
    To select a choice enter
    its number, or 0 when you are finished [0]:
    
    Failover Cluster Administrator
    
    [X] 1
    - VERITAS Cluster Failover
    
    To select a choice enter
    its number, or 0 when you are finished [0]:
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay [1]
    
    -------------------------------------
    
    Destination directory
    for Element Management System 14.0.2255
    
    Please specify a directory
    or press Enter to accept the default directory [/users/ems]  /users/ems 
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay [1]
    
    --------------------------------------
    
    Enter the global disk
    path for shared data directory. 
    
    [] /users/ems-shared/ems-sd
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay [1]
    
    Directory Validation
    ...
    
    Fetching all IP Addresses
    ...
    
    --------------------------------------
    
    Global Configuration 
    
    [X] 1
    - Select the IP Address
    
    [ ] 2
    - Enter the IP Address
    
    To select a choice enter
    its number, or 0 when you are finished [0]:
    
    [X] 1
    - 10.105.80.174
    
    [ ] 2
    - 10.105.80.6 [ ]
    
    3 - 10.105.87.222
    
    To select a choice enter
    its number, or 0 when you are finished [0]: 2 
    
    [ ] 1
    - 10.105.80.174 [X]
    
    2 - 10.105.80.6 [ ]
    
    3 - 10.105.87.222
    
    To select a choice enter
    its number, or 0 when you are finished [0]: 
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay [1]
    
    Global Configuration
    ...
    
    -------------------------------------
    
    SSL Configuration
    
    [X] 1
    - Enforce SSL
    
    To select a choice enter
    its number, or 0 when you are finished [0]: 1
    
    [ ] 1
    - Enforce SSL 
    
    To select a choice enter
    its number, or 0 when you are finished [0]: 
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay [1]
    
    SSL Configuration ...
    
    --------------------------------------
    
    Content Filtering / Blacklisting Configuration
    
    [ ] 1
    - Enable Content Filtering
    
    To select a choice enter
    its number, or 0 when you are finished [0]:
    
    [ ] 1
    - Enable Blacklisting
    
    To select a choice enter
    its number, or 0 when you are finished [0]:
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay 
    
    [1] Content
    Filtering / Blacklisting Configuration ...
    
    -------------------------------------
    
    EMS Client Configuration
    
    [ ] 1
    - Fixed Port
    
    To select a choice enter
    its number, or 0 when you are finished [0]: 
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay [1]
    
    EMS Client Configuration
    ...
    
    --------------------------------------
    
    Apache Server Configuration
    
    Apache Port
    
    [80]
    
    Apache SSL Port
    
    [443]
    
    Administrator Email-ID
    you@your.com
    
    [root@localhost.com] 
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay [1]
    
    Apache Server Configuration
    ...
    
    --------------------------------------
    
    EMS Server Configuration
    
    MG Session Management
    (ISM)
    
     CORBA Communication
    Port
    
    [14132] 14135
    
    Application Server ID 
    
    [CISCO] SQA 
    
    Application Server Password
    :
    
     Network Management
    System (NMS)
    
    FTP User Name
    
    [staradmin]
    
    FTP Password :
    
    User Limit
    
    [10]
    
    Security Admin Limit
    
    [5]
    
    User Session Limit
    
    [5]
    
    Context User Limit
    
    [5]
    
    FTP Session Limit
    
    [5]
    
    Storage Server Dir 
    
    [./flash]
    
    Syslog Config File
    
    [/etc/syslog.conf]
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay [1]
    
    EMS Server Configuration
    ...
    
    --------------------------------------
    
    EMS Server Configuration
    (contd.)
    
    Fault Management (FM)
    
    SNMP Trap Port 
    
    [162] 1060
    
    SMTP Server name
    
    [] outbound.cisco.com
    
    Sender Email-ID
    
    [] root@localhost.com
    
    Bulk Statistics
    
    Bulkstat FTP User Name
    
    [anonymous] guest
    
    Bulkstat FTP Password
    : 
    
    Host IP Address
    
    [127.0.0.1]
    
    Host Base Directory 
    
    [/tmp]
    
    [ ] 1
    - Archive Bulkstat Files
    
    To select a choice enter
    its number, or 0 when you are finished [0]: 1
    
    [X] 1
    - Archive Bulkstat Files 
    
    To select a choice enter
    its number, or 0 when you are finished [0]:
    
    [ ] 1
    - Generate XML Files To select a choice enter its number, or 0 when
    you are finished [0]:
    
    1 [X] 1
    - Generate XML Files To select a choice enter its number, or 0 when
    you are finished [0]:
    
    [ ] 1
    - Bulkstat FTP Perform OperationTo select a choice enter
    its number, or 0 when you are finished [0]:
    
    User Administration
    EMS Administrator E-Mail Address
    
    [] root@localhost.com
    Hardware Inventory Hardware Notifier Email-ID
    
    [] root@localhost.com
    Press 1 for Next panel, 2 for Previous panel, 3 to Cancel or 4 to
    Redisplay [1]
    
    EMS Server Configuration
    (contd.) ...
    
    returning from cdp configuration
    ....
    
    -------------------------------------
    
    EMS Process Monitor
    Configuration
    
    Poll Interval (sec)
    
    [30]
    
    Number Of Retries 
    
    [10]
    
    Timeout Interval (sec) 
    
    [330] SMTP
    Server Name [outbound.cisco.com]
    
    To Email-ID xyz@any.com
    
    [] root@localhost.com
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay [1]
    
    EMS Process Monitor
    Configuration ... 
    
    -------------------------------------
    
    Postgres Configuration
    
    User Name
    
    [postgres]
    
    Password :
    
    NOTE: Postgres database
    installed using the EMS Installer should be used for EMS purposes
    only.
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay [1]
    
    Postgres Configuration
    ...
    
    --------------------------------------
    
    Element Management System
    will be installed in the following location:
    
    /users/ems
    
    with the following features:
    
    Apache Server
    
    PostgreSQL
    
    EMS Server
    
    EMS Client
    
    for a total size: 806MB
    
    Press 1 for Next panel,
    2 for Previous panel, 3 to Cancel or 4 to Redisplay [1]
    
    Cleaning all the directories...
    
    Installing Element Management
    System 14.0.2255...
    
    Please wait.
    
     |-----------|-----------|-----------|------------| 
    
    0% 25% 50% 75% 100%
    
    ||||||||||||||||||||||||||||||||||||||||||||||||||
    
    Updating the RPM database
    creating the softlinks for server libraries...
    
    Updating runtime linking
    environment... 
    
    Setting User Id to root...
    
    Installing Database...
    
     \c 
    
    The files belonging
    to this database system will be owned by user "postgres".
    
    This user must also
    own the server process.
    
    The database cluster
    will be initialized with locale C.
    
     The default text search
    configuration will be set to "english".
    
    fixing permissions on
    existing directory /users/ems-shared/ems-sd/pgdata
    ... ok
    
    creating subdirectories
    ... ok 
    
    selecting default max_connections
    ... 100
    
    selecting default shared_buffers
    ... 32MB
    
    creating configuration
    files ... ok
    
    creating template1 database
    in /users/ems-shared/ems-sd/pgdata/base/1
    ... ok 
    
    initializing pg_authid
    ... ok
    
    initializing dependencies
    ... ok
    
    creating system views
    ... ok 
    
    loading system objects'
    descriptions ... ok
    
    creating conversions
    ... ok 
    
    creating dictionaries
    ... ok
    
    setting privileges on
    built-in objects ... ok
    
    creating information
    schema ... ok vacuuming database template1 ... ok
    
    copying template1 to
    template0 ... ok
    
    copying template1 to
    postgres ... ok
    
    Success. You can now
    start the database server using:
    
    /users/ems/postgres8.4.2/bin/postgres
    -D /users/ems-shared/ems-sd/pgdata
    
    or 
    
    /users/ems/postgres8.4.2/bin/pg_ctl
    -D /users/ems-shared/ems-sd/pgdata
    -l logfile
    
    start
    
     WARNING: enabling "trust"
    authentication for local connections
    
    You can change this
    by editing pg_hba.conf or using the -A option the next
    time you run initdb.
    
    Creating database and
    tables... 
    
    \c
    
    Creating postgresctl
    for PSMON... 
    
    Changing directory ownership
    to postgres... Changing file ownership...
    
     Cleaning obsolete packages...
    
    Updating PostgreSQL
    config file...
    
    Updating /users/ems/postgres/data/postgresql.conf
    
    Stopping postmaster...
    
    --------------------------------------
    
    The InstallShield wizard
    has successfully installed Element Management System.
    
    Choose Finish to exit
    the wizard. Press 3 to Finish or 4 to Redisplay [3]
    
  4. Enable and online application resource in Active node.
    $ hares -modify
    <App_resource> Enabled 1
    
    $ hares -online
    <App_resource> -sys <Node1>
    

Standby Node

  1. Offline all the resources on Active Node and online them on standby node except application resource.
    $ hares -offline
    <App_resource> -sys <Node1>
    
    $ hares -modify
    <App_resource> Enabled 0
    
    $ hares -offline
    <IP_resource> -sys <Node1>
    
    $ hares -offline
    <Mount_resource> -sys <Node1>
    
    $ hares -offline
    <Vol_resource> -sys <Node1>
    
    $ hares -offline
    <Disk_resource> -sys <Node1>
    
    $ hares -online
    <IP_resource> -sys <Node2>
    
    $ hares -online
    <Disk_resource> -sys <Node2>
    
    $ hares -online
    <Vol_resource> -sys <Node2>
    
    $ hares -online
    <Mount_resource> -sys <Node2>
    
  2. Disable the Resource group on Active Node.
    $ hagrp -disable <Resource_group>
    
  3. Install WEM in standby node. (Follow step 3 of Install WEM in Veritas Cluster Mode in Active Node section).
  4. Enable and online application resource in standby node.
    $ hares -modify
    <App_resource> Enabled 1
    
    $ hares -online
    <App_resource> -sys <Node2>
    
  5. Configure application resource as critical.
    $ hares -modify
    <App_resource> Critical 1
    
  6. Switchover to Active Node.
    $ hagrp -switch
    <Resource_group> -to <Node1>
    

Main.cf File Configuration Example

The following is an example of the main.cf file for resource-groups and resources.

group wemFailover ( 
   SystemList = { pnstextappsucs1 = 0,
pnstextappsucs3 = 1 }        
   AutoStartList = { pnstextappsucs3 }
  )
Application wemService
(
   StartProgram = "/users/ems/postgres/bin/emsctl
start"                
   StopProgram = "/users/ems/postgres/bin/emsctl
forcestop"                
   PidFiles = { "/users/ems/server/psmon.pid" }
   RestartLimit = 1
)
DiskGroup wemDG ( 
   DiskGroup = wemdg
)
IP wemIP (
   Device = eth0
   Address = "10.4.83.151" 
   NetMask = "255.255.255.0"
)
 Mount wemMount ( 
   MountPoint = "/apps/wem/" 
   BlockDevice = "/dev/vx/dsk/wemdg/w
   FSType = vxfs
   FsckOpt = "-y"
)
NIC wemNIC (
   Device = eth0
)
 Volume wemVolume (
   DiskGroup = wemdg
   Volume = wemvol
)
where:
  • wemMount requires wemVolume
  • wemVolume requires wemDG
  • wemIP requires wemNIC
  • wemService requires wemMount
  • wemService requires wemIP

as indicated below:

// resource dependency tree // // group wemFailover // { // Application wemService // { // Mount wemMount // { // Volume wemVolume // { // DiskGroup wemDG // } // // IP wemIP // { // NIC wemNIC // } // } // } // }

Upgrading WEM with VCS

  1. Disable WEM resource group on standby node
    $ hagrp -modify
    <Resource_group> Enabled 0 -sys <Node2>
    
  2. Upgrade WEM on standby node.
  3. Untar the sqlfiles provided with the installation on the active node.
  4. Go to sqlfiles folder and run dbClusterUpgrade.sh on the Active node.
  5. Offline application resource in the Active node.
    $ hares -offline
    <App_resource> -sys <Node1>
    
  6. Upgrade WEM on Active node.
  7. Enable the resource group on Standby node.
    $ hagrp -modify
    <Resource_group> Enabled 1 -sys <Node2> 
    
  8. Online application resource on Active node.
    $ hares -online
    <App_resource> -sys <Node1>
    

Uninstalling WEM with VCS

  1. Disable resource group on Standby node.
    $ hagrp -disable
    <Resource_group> -sys <Node2>
    
  2. Uninstall WEM on standby node.
  3. On Active node offline and disable application resource.
    $ hares -offline
    <App_resource> -sys <Node1>
    
    $ hares -modify
    <App_resource> Enabled 0
    
  4. Uninstall WEM on Active node.

Configuring HA Group

Prerequisites

  1. Veritas SFHA package is installed and nodes Node1/Node2 are in a clustered environment.
  2. The diskgroup/volume (For example: emsDG/emsVol) is available and can be mounted on both the nodes at location /users/ems-shared.

Configuration Steps

  1. Create Main Resource group.
    $ hagrp -add <ResourceGroup_name>
    
    $ hagrp -modify
    <Resource_group> SystemList <Node1> <Node2> 2
    
  2. Add a Disk group resource.
    $ hares -add
    <Disk_resource> DiskGroup <Resource_group>
    
    $ hares -modify
    <Disk_resource> Enabled 1
    
    $ hares -modify
    <Disk_resource> DiskGroup <DisKGroup_name>
    
    $ hares -modify
    <Disk_resource> Critical 1
    
    $ hares -online
    <Disk_resource> -sys <Node1>
    
    $ hares -offline
    <Disk_resource> -sys <Node2>
    
  3. Add Volume resource.
    $ hares -add
    <Vol_resource> Volume <Resource_group>
    
    $ hares -modify
    <Vol_resource> Enabled 1
    
    $ hares -modify
    <Vol_resource> Critical 1
    
    $ hares -modify
    <Vol_resource> DiskGroup <DiskGroup_name>
    
    $ hares -modify
    <Vol_resource> Volume <Volume_name>
    
    $ hares -online
    <Vol_resource> -sys <Node1>
    
    $ hares -offline
    <Vol_resource> -sys <Node2>
    
  4. Add Mount resource. (Create directory /users/ems-shared on both the nodes before proceeding.)
    $ hares -add
    <Mount_resource> Mount <Resource_group>
    
    $ hares -modify
    <Mount_resource> Enabled 1
    
    $ hares -modify
    <Mount_resource> Critical 1
    
    $ hares -modify
    <Mount_resource> MountPoint /users/ems-shared
    
    $ hares -modify
    <Mount_resource> BlockDevice <diskgroup/volume>
    
    $ hares -modify
    <Mount_resource> FSType vxfs
    
    $ hares -modify
    <Mount_resource> FsckOpt %-y
    
    $ hares -online
    <Mount_resource> -sys <Node1>
    
    $ hares -offline
    <Mount_resource> -sys <Node2>
    
  5. Add NIC resource.
    $ hares -add
    <NIC_resource> NIC <resource_group>
    
    $ hares -modify
    <NIC_resource> Enabled 1
    
    $ hares -modify
    <NIC_resource> Device eth0
    
  6. Add IP resource.
    $ hares -add
    <IP_resource> IP <Resource_group>
    
    $ hares -modify
    <IP_resource> Enabled 1
    
    $ hares -modify
    <IP_resource> Critical 1
    
    $ hares -modify
    <IP_resource> Device eth0
    
    $ hares -modify
    <IP_resource> Address <IP Add>
    
    $ hares -modify
    <IP_resource> NetMask <subnet mask>
    
    $ hares -online
    <IP_resource> -sys <Node1>
    
    $ hares -offline
    <IP_resource> -sys <Node2>
    
  7. Add Application resource.
    $ hares -add
    <App_resource> Application <Resource_group>
    
    $ hares -modify
    <App_resource> StartProgram "/users/ems/server/bin/emsctl
    start"
    
    $ hares -modify
    <App_resource> StopProgram "/users/ems/server/bin/emsctl
    forcestop"
    
    $ hares -modify
    <App_resource> PidFiles /users/ems/psmon.pid
    
  8. Link all the resources that are created above.
    $ hares -link
    <Mount_resource> <Vol_resource>
    
    $ hares -link
    <Vol_resource> <Disk_resource>
    
    $ hares -link
    <IP_resource> <NIC_resource>
    
    $ hares -link
    <App_resource> <Mount_resource>
    
    $ hares -link
    <App_resource> <IP_resource>