> WEM High Availability Redundancy with Cluster Application
> Systems that rely on running Web Element Manager on a single server to manage their networks face the possibility of service disruption should the server fail. By using Oracle Solaris Cluster software, it is now possible to create redundant Web Element Manager servers with a primary host server running an active instance of Web Element Manager, and a redundant server in standby mode. This appendix provides information to help you successfully configure redundant instances of Web Element Manager over multiple servers. This appendix works closely with the Installing the WEM Software chapter in this guide.
> Important: Cluster configuration is not supported on Cisco Red Hat Enterprise Linux (RHEL) operating systems for this software release.
> This appendix addresses the following topics:

> • | System Requirements.

> • | Installing Web Element Manager for Failover Mode.

> • | Creating and Configuring a Cluster Resource Group.

> • | Upgrading Web Element Manager in a Clustered Environment.

> System Requirements
> Requirements for implementing High Availability are as follows:
> Web Element Manager must be installed on a minimum of two Sun Netra™ T5220 servers equipped with the hardware described in the Server Hardware Requirements section of this guide. We recommend a cluster installation restricted to two servers configured similar to that shown in the diagram below. The sample configurations in this appendix assume such an installation.
> Important: Ensure you have installed the latest version of Oracle Solaris software and all appropriate software patches as described in the Operating System Requirements section.
> IPMP is a feature supported on Oracle Solaris. For more complete configuration information, refer to Configuring IPMP for WEM Server and also to the Oracle product documentation.
> Oracle Solaris Cluster is a feature provided and supported by Oracle. For more complete information on configuring Resource Groups, refer to the Oracle Solaris Cluster product documentation.
> Installing Web Element Manager for Failover Mode
> This section specifies the configuration changes required when installing WEM in Failover Mode rather than Standalone Mode when following the installation instructions in the Installing the WEM Software chapter. For this release, please use the GUI to perform the installation rather than the command line.
> Important: Install and configure Web Element Manager in Failover Mode on both servers before configuring a cluster resource group.
> The following items are either different from, or prerequisites for, the installation steps defined in the Installing the WEM Software chapter:

> • | Create a file directory path <ems_dir> or use the default path: /users/ems.

> • | The logical hostname and a floating IP address shared between the two nodes must be configured in /etc/hosts. Ems-Service is used as the logical hostname in the examples in the rest of this appendix.

> • | Create the global disk path for a shared data directory, for example: /shareddir/ems-share.

> Important: The following options are not set when installing in Failover Mode:

> • | WEM Service started by default and monitored by Process Monitor. (See the WEM Process Monitor chapter for more information on processes.)

> • | Start EMS on machine start-up.

> Creating and Configuring a Cluster Resource Group
> This section explains how to create a Resource Group specifically for WEM servers in this cluster and configure it appropriately.
> Important: This process is configured on only one server in the cluster. It is reflected on both.
> Creating a Resource Group
> The following describe how to create a Resource Group.
> We recommend the cluster binary path is set in the shell environment as this means you can execute the cluster commands from any directory path.
> Before clsetup can create a network resource for any logical hostname, that hostname and a common floating IP address associated with it must be specified in the /etc/hosts directory on both servers. This example uses ems-service as the logical hostname.

> Step 1 | Login as root and run clsetup to open the Main Menu.

> Step 2 | From the Main Menu select Option 2: Resource Groups.

> Step 3 | From the Resource Groups Menu, select Option 1: Create a Resource Group.

> A resource group is a container into which you can place resources of various types, such as network and data service resources and then manage them. Only failover resource groups can contain network resources. A network resource would include logical hostname.

> Step 4 | When prompted to create a failover group, enter yes and select Option 1: Create a Failover Group. For this example, call the group ems-rg.

> Step 5 | When you are prompted to select a preferred server enter yes and enter the name of the Preferred server; for this example use Node1. Enter yes to continue the update.

> The screen will display the following message:
> clresourcegroup create -n <Node-1 Node Name> <Node-2 Node Name> ems-rg
> Command completed successfully.
> With the Resource Group created successfully, you can move on to the next step and add the logical hostname.
> Adding a Logical Hostname to a Failover Resource Group
> Follow steps 1 - 5 to add a logical hostname.

> Step 1 | After the confirmation screen from the last task displays, press Enter to continue. Enter yes when prompted to add network resources.

> Step 2 | From the Network Resources Menu, select Option 1: Add a Logical Hostname.

> If a failover resource group contains logical hostname resources, the most common configuration is to have one logical hostname resource for each subnet. Enter 1 to create a single resource.

> Step 3 | When prompted for a logical hostname, enter the logical hostname configured in /etc/hosts for the floating IP address, For this example use ems-service.

> Step 4 | Press Enter to continue. The screen displays:

> clreslogicalhostname create -g ems-rg -p R_description="LogicalHostname resource for ems-service" ems-service

> Step 5 | Enter no when prompted to add any additional network resources.

> Adding a Data Service Resource
> Follow steps 1 - 4 to add a data service.

> Step 1 | After the logical hostname confirmation screen, enter yes when prompted to begin adding data services.

> Step 2 | From the Data Services Menu select Option 1: EMSSCFO Server for Sun Cluster, and use the name ems-dsr for this example.

> The screen displays the following message:
> This data service uses the "Port_list" property. The default "Port_list" for this data service is as follows: <NULL>

> Step 3 | Enter no when prompted to override the default.

> Step 4 | Enter no when prompted to add more properties, then enter yes to continue.

> The screen displays the following message:
> Commands completed successfully
> Bringing the Resource Group Online
> Follow steps 1 - 2 to bring the Resource Group online.

> Step 1 | After the completion confirmation screen, press Enter to continue. Enter no when prompted to add any additional data service resources. Enter yes when prompted to manage and bring this resource group online.

> The screen displays the following message:
> clresourcegroup online -M ems-rg
> Commands completed successfully

> Step 2 | Press Enter to continue, then select Option q to Quit and return to the Main Menu.

> The process is now complete. At this point you can enter the scstat command to display the current online/offline status if required.
> Upgrading Web Element Manager in a Clustered Environment
> This section describes the process for upgrading Web Element Manager in a two-server cluster.
> Important: Network administrators are advised that they should have any connected clients log out at this time. If clients cannot reconnect after the upgrade, please refer to the Troubleshooting appendix for information on any Java-related errors.
> Prerequisite Steps for the Upgrade Process
> For the example configuration that follows you should confirm the following:

> • | The same version of Web Element Manager software (12.0 or newer) has been installed on both servers and configured in Failover Mode.

> • | Config files and scripts are identical.

> • | Devices can be failed over with no loss in connectivity.

> This can be confirmed either by a software switchover, or by running the scstat command to confirm the current node status.

> • | Resources have been configured in the Oracle Solaris Cluster software using the following example names:

> • | Two Cluster Nodes: N-1 (initially this is the active node) and N-2 (initially this is the redundant node).

> • | A Resource Group managed by Web Element Manager exists. ems-rg is used for this example.

> • | A logical hostname and floating (shared) IP address has been configured on both servers. This example uses the default /etc/hosts folder. Use ems-service as the hostname for this example.

> Upgrade Process Overview
> This section provides a broad overview of the procedures to follow.

> Step 1 | Start with two server nodes: N-1 and N-2. They share both data files and database information. N-1 is currently active and N-2 is currently in standby mode, as shown below:

> Step 2 | With Web Element Manager running on N-1, put N-2 into maintenance mode (this is described in Removing an Inactive Node from the Resource Group below). Then upgrade WEM to version 12.0 or later.

> At this point, the database is not yet updated. In cluster mode, when WEM is upgraded on N-2 the postgres database is not started. so a new DB schema is not created. This is because the schema will be updated through the currently active server N-1 with SQL scripts that are provided as part of the installation package; this is explained in the section Updating the Databases. At this point N-2 is still in maintenance mode but now with updated WEM software, and the database schema has now been updated as shown below:

> Step 3 | To prepare for the switchover, take N-2 out of maintenance mode.

> Step 4 | To upgrade WEM on N-1, bring N-2 back into the cluster and do a switchover from N-1 to N-2 so now N-2 is the active host and WEM starts running processes.

> Step 5 | Now place the formerly active node N-1 into maintenance mode and upgrade WEM software while N-2 is the active node as shown below:

> Step 6 | Once the upgrade is complete, N-1 can be switched back to its role of active node if necessary by using the Cluster commands, or N-2 can continue as the active node.

> Important: Refer to the Installing the WEM Software chapter for more complete information about scripts and files. All configuration files placed in the <EMS_HOME>/server/etc folder and the script files in the <EMS_HOME>/server/scripts folder must be identical. This ensures that after the switch between servers, the behavior of EMS does not change in any way.
> Removing an Inactive Node from the Resource Group
> Complete the following steps to remove N-2 from the Resource Group. Since the cluster resource group configuration will be same for both nodes, the cluster-related commands can be run on either node.

> Step 1 | Run the scstat command. scstat is used to verify the current status of the cluster resource group and ensures that on switchover/failover the servers will switch correctly. The following screen display reflects a properly configured cluster:

> Two cluster nodes: Online
> Two cluster transport paths: Online
> Quorum votes by node: Online
> Quorum votes by device: Online
> Resource Groups and Resources: ems-rg, ems-service, ems-dsr
> Ems-rg group N-1: Online N-2: Offline
> IPMP groups: Online
> Important: N-2 must not be allowed to run any WEM processes. This prevents the secondary node from taking ownership of resources. Removing it from the Resource Group prevents a failover from happening and N-1 continues to behave like a standalone WEM thus ensuring a successful upgrade. To do this:

> Step 2 | Enter clsetup to open the Main Menu and select Option 2: Resource Groups Menu.

> Step 3 | From the Resource Groups menu, select Option 8: Change the Properties of a Resource Group.

> Step 4 | Enter yes when prompted to continue.

> Step 5 | Select the group to be changed by selecting Option 1: ems-rg.

> Step 6 | Select Option 1: Change the Nodelist Resource Group Properties

> Step 7 | Enter yes when prompted to continue. Both N-1 and N-2 should now appear in the nodelist.

> Step 8 | Select Option 2: Remove a Node from the Nodelist, then select Option 1 to remove N-2.

> The nodelist now contains only N-1. When prompted to update the nodelist property, enter yes. If your update was successful you will see the following screen confirmation:
> command completed successfully
> Press Enter to continue. You will receive confirmation that only N-1 remains in the nodelist. Select Option q to Quit and exit back to the Resource Group Menu.

> Step 9 | From the Resource Group Menu select Option s: Show Current Status to confirm the current network resources if required.

> Upgrading WEM on the Inactive Server
> Complete the following steps to upgrade WEM on the inactive server, N-2.

> Step 1 | Web Element Manager begins installing along with Apache Server, PostgreSQL Server, and EMS Server. Installation continues until the following warning message appears:

> Updating PostgreSQL config file...This is an upgrade in Cluster mode; not updating postgres config.
> This message is normal because the database is to be updated from the active server, N-1.

> Step 2 | With the installation complete, select Option 3 to finish.

> Updating the Databases
> Complete the following steps to update the databases from node N-1.

> Step 1 | Copy the sqlfiles.tar file from the N-2 installation to a folder on N-1 and untar the file. This process is described fully in the Installing the WEM Software chapter.

> This will create a folder called sqlfiles.

> Step 2 | Go to the sqlfiles folder and run dbClusterUpgrade.sh.

> Step 3 | At the prompt, enter the EMS directory name and press Enter.

> Step 4 | Enter a complete directory path for saving the new SQL files and press Enter.

> Step 5 | Enter the postgres administrator name assigned during the installation process. This is postgres by default.

> Step 6 | Press Enter.

> Step 7 | Enter the database port number and press Enter.

> Important: Currently, this is port 5432, but this may change in a later release.

> Step 8 | The databases update and the screen displays the following message:

> Database schema upgraded successfully...
> Returning the Inactive Node to the Resource Group
> Complete the following steps to return N-2 to the Resource Group and take over resource ownership in order to upgrade the software on N-1.

> Step 1 | Run clsetup and then log in to access the Main Menu and select Option 2: Resource Groups.

> Step 2 | From the Resource Groups Menu select Option 8: Change the Properties of a Resource Group.

> Step 3 | Enter yes when prompted to continue.

> Step 4 | From the next screen select the Resource Group name.

> Step 5 | When prompted for the property to change, select Option 1: Nodelist.

> Step 6 | Enter yes to continue and open the next screen.

> Step 7 | Select Option 1: Add a Node/Zone to the Top of the Nodelist.

> Step 8 | Select Option 1: N-2.

> Step 9 | Enter yes when prompted to update the nodelist property.

> The screen will display the following message:
> Command completed successfully.

> Step 10 | Press Enter to continue and select Option q to Quit and return to the Resource Groups Menu.

> Switching Active Servers
> Complete the following steps to make N-2 the active node so N-1 can be updated.

> Step 1 | From the Resource Group Menu Select Option 5: Switch over a Resource.

> Step 2 | Scroll through the on-screen description and enter yes to continue.

> Step 3 | Select the name of the resource group. In this example it would be ems-rg.

> Step 4 | Select Option 1: Switch Group Ownership.

> Step 5 | Select the node to take ownership of ems-rg, which would be N-2. Enter yes to confirm. The screen will display the following message:

> Command completed successfully.

> Step 6 | Press Enter to continue and select Option q to Quit and return to the Resource Group Menu.

> Step 7 | From the Resource Group Menu select Option s Show Current Status. This shows that N-2 is now online and N-1 is offline.

> At this point return to Removing an Inactive Node from the Resource Group and begin the update process for N-1.
> Important: Since the database schema were previously updated and both N-1 and N-2 share the same database, it is not necessary to run the SQL scripts again for N-1.

Cisco Systems Inc.
Tel: 408-526-4000
Fax: 408-527-0883