Contents

Configuring High Availability

This chapter describes how to configure high availability on the Cisco Mobility Services Engine. The MSE is a platform for hosting multiple mobility applications. Every active MSE is backed up by another inactive instance. The active MSE is called the Primary MSE and the inactive MSE is called the Secondary MSE.

The main component of high availability system is the health monitor. The health monitor configures, manages, and monitors the high availability setup. Heartbeat is maintained between the primary and secondary MSE. Health monitor is responsible for setting up the database, file replication, and monitoring the application. When the primary MSE fails and the secondary MSE takes over, the virtual address of the primary MSE is switched transparently to the secondary MSE.


Note


The MSEs, Synchronize Services, Synchronization History, High Availability, Context Aware Notifications, and Mobile Concierge pages on the Services tab are available only in the virtual domain in Release 7.3.

Overview of the High Availability Architecture

This section provides an overview of the high availability architecture.

  • Every active primary MSE is backed up by another inactive instance. The purpose of the secondary MSE is to monitor the availability and state of the primary MSE. The secondary MSE becomes active only after the failover procedure is initiated.
  • One secondary MSE can support one primary MSE.

Pairing Matrix

The following table lists the server type pairing matrix information.

Table 1 Pairing Matrix
Secondary Server Type
Primary Server Type 3355 VA-2 VA-3 VA-4 VA-5
3355 Y N N N N
VA-2 N Y Y Y Y
VA-3 N N Y Y Y
VA-4 N N N Y Y
VA-5 N N N N Y

Guidelines and Limitations for High Availability

  • Both the health monitor IP and virtual IP should be accessible from Prime Infrastructure.
  • The health monitor IP and virtual IP should always be different. The health monitor and virtual interface can be on the same network interface or different interfaces.
  • You can use either manual or automatic failover. Failover should be considered temporary. The failed MSE should be restored to normal as soon as possible, and failback should be reinitiated. The longer it takes to restore the failed MSE, the longer you are running with a single MSE without high availability support.
  • You can use either manual or automatic failback.
  • Both the primary and secondary MSE should be running the same software version.
  • High Availability over WAN is not supported.
  • High Availability over LAN is supported only when both the primary and secondary MSEs are in the same subnet.
  • The ports over which the primary and secondary MSEs communicate must be open (not blocked with network firewalls, application fireways, gateways, and so on).

Failover Scenario for High Availability

When a primary MSE failure is detected, the following events occur:

  • The primary MSE is confirmed as non-functioning (hardware fail, network fail, and so on) by the health monitor on the secondary MSE.
  • If automatic failover has been enabled, the secondary MSE is starts immediately. If automatic failover is disabled, an e-mail is sent to the administrator asking if they want to manually start failover.
  • When manual failover is configured, an e-mail is sent only if the e-mail is configured for MSE alarms.
  • The result of the failover operation is indicated as an event in the Health Monitor UI, and a critical alarm is sent to Prime Infrastructure.

Failback Scenario for High Availability

When the primary MSE is restored to its normal state, if the secondary MSE is already in failover state for the primary, then failback can be invoked.

Failback can occur only if the secondary MSE is in one of the following states for the primary instance:

  • The secondary MSE is actually failing over for the primary MSE.
  • Manual failover is configured but the administrator did not invoke it.
  • The primary MSE failed but the secondary MSE cannot take over because it has encountered errors.
  • Failback can occur only if the administrator starts up the failed primary MSE.

HA Licensing

For high availability, an activation license is required on the primary and secondary virtual appliances. No other service license is required on the secondary MSE. It is required only on the primary MSE.

Configuring High Availability on the MSE

During the installation of the MSE software (or using the MSE setup script), configure some critical elements. Pair up the primary and secondary MSE from the Prime Infrastructure UI.


Note


By default, all MSEs are configured as primary. If you do not want high availability support and are upgrading from an earlier release, you can continue to use the IP address for the MSE. If you want to set up high availability, then you must configure the health monitor IP address. The health monitor then becomes a virtual IP address.


To configure high availability on the primary MSE, follow these steps:


    Step 1   Ensure that the network connectivity between the primary and secondary MSEs is functioning and that all the necessary ports are open.
    Step 2   Install the correct version of MSE on the primary MSE.
    Step 3   Make sure that the same MSE version is installed on the secondary MSE.
    Step 4   On the intended primary MSE, enter the following command:
    
    /opt/mse/setup/setup.sh
    --------------------------------------------------------------
    Welcome to the Cisco Mobility Services Engine Appliance Setup.
    
    You may exit the setup at any time by typing <Ctrl+c>.
    -----------------------------------------------------------
    
    Would you like to confgire MSE using:
                1. Menu mode
                2. Wizard mode
                Choose 1 or 2: 1
    --------------------------------------------------
    
    Mobility Services Engine Setup
    
    Please select a configuration option belwo and enter the
    requested information. You may exit setup at any time by typing <Ctrl +C>.
    
    You will be prompted to choose whether you wish to configure a
    parameter, skip it, or reset it to its initial default value.
    Skipping a parameter will leave it unchanged from its current
    value.
    
    Please note that the following parameters are mandatory and must be configured at lease once.
            -> Hostname
            -> Network interface eth0
            -> Timezone settings
            -> Root password
            -> NTP settings
            -> Prime Infrastructure password
    
    You must select option 24 to verify and apply any changes made during this session.
    --------------------------------------------------------------
    
    PRESS <ENTER> TO CONTINUE:
    
    
    --------------------------------------------------------------
    
    Configure MSE:
    
    	1) Hostname *																					    13) Remote syslog settings
     2) Network interface eth0 settings*		 14) Host access control settings
     3) Timezone settings*																	15) Audit Rules
     4) Root password *																				16) Login banner
     5) NTP settings *																					17) System console restrictions
     6) Prime Infrastructure password *				18) SSH root access
     7) Display current configuration						19) Single user password check
     8) Domain																													20) Login and password settings
     9) High availability role													21) GRUB password
    10) Network interface eth1 settings				22) Root access control
    11) DNS settings																							23) Auto start MSE on system boot up
    12) Future restart time																24) ## Verify and apply changes ##
    
    Please enter your choice [1 - 24]:
    
    --------------------------------------------------------------
    
    Step 5   Configure the hostname:
    
    Please enter your choice [1 - 24]: 1
    
    Current Hostname=[mse]
    Configure Hostname?  (Y)es/(S)kip/(U)se default [Skip]: y
    The host name should be a unique name that can identify
    the device on the network. The hostname should start with
    a letter, end with a letter or number, and contain only
    letters, numbers, and dashes.
    
    Enter a Host name [mse]:mse1
    
    
    Step 6   Configure the domain:
    Please enter your choice [1-24]: 8
    Current domain=[ ]
    Configure domain name?  (Y)es/(S)kip/(U)se default [Skip]: S
    Step 7   Configure the Network interface eth0 settings.
    
    Please enter your choice [1 - 24]: 2
    
    Current eth0 interface IP address=[10.0.0.1]
    Current eth0 interface netmask=[255.0.0.0]
    Current IPv4 gateway address=[172.20.104.123]
    Configure eth0 interface parameters? (Y)es/(S)kip/(U)se default [Skip]:y
    
    Enter an IP address for first ethernet interface of this machine.
    
    Enter eth0 IP address  [10.0.0.2]: 
    
    Enter the network mask for IP address 172.21.105.126
    
    Enter network mask [255.255.255.224]: 
    
    Enter the default gateway address for this machine.
    
    Note that the default gateway must be reachable from
    the first ethernet interface.
    
    Enter default gateway address [172.20.104.123]:
    
    
    Step 8   Configure the Root password:
    
    Please enter your choice [1 - 24]: 4
    
    Root password has not been configured
    Configure root password? (Y)es/(S)kip/(U)se default [Skip]: 
    Changing password for user root.
    
    You can now choose the new password.
    
    A valid password should be a mix of upper and lower case letters, digits, and other characters. You can use an 8 character long password with characters from all of these classes. An upper case letter that begins the password and a digit that ends it do not count towards the number of character classes used.
    
    Enter new password:
    
    
    Step 9   Configure the High availability role:
    
    Current role=[Primary]
    Configure High Availability?  (Y)es/(S)kip/(U)se default [Skip]:
    
    High availability role for this MSE (Primary/Secondary)
    
    Select role [1 for Primary, 2 for Secondary] [1]: 1
    
    Health monitor interface holds physical IP address of this MSE server.
    This IP address is used by Secondary, Primary MSE servers and Prime Infrastructure to 
    communicate among themselves
    Select Health Monitor Interface [eth0/eth1] [eth0]: eth0 
    -------------------------------------------------------------------
    Direct connect configuration facilitates use of a direct cable connection between the primary and secondary MSE servers.
    This can help reduce latencies in heartbeat response times, data replication and failure detection times.
    Please choose a network interface that you wish to use for direct connect. You should appropriately configure the respective interfaces.
    "none" implies you do not wish to use direct connect configuration.
    -------------------------------------------------------------------
    
    Select direct connect interface [eth0/eth1/none] [none]:
    Enter a Virtual IP address for the Primary MSE server
    
    Enter Virtual IP address [1.1.1.1]: 10.10.10.11
    
    Enter network mask for IP address 10.10.10.1
    
    Enter network mask [1.1.1.1]: 255.255.255.0
    
    Select to start the sever in recovery mode.
    You should choose yes only if this primary MSE was paired earlier and you have nowlost the 
    configuration from this box.
    And, now you want to restore the configuration from Secondary via Cisco  Prime Infrastructure
    Do you wish to start this MSE in HA receovery mode?: (yes/no} [no]:no
    
    Current IP address = [1.1.1.10]
    Current eth0 netmask=[255.255.255.0]
    Current gateway address=[1.1.1.1]
    Configure eth0 interface parameters? (Y)es/(S)kip/(U)se default [Skip]:
    
    Enter an IP address for first ethernet interface of this machine.
    
    Enter eth0 IP address [1.1.1.10]: 10.10.10.12
    
    Enter the network mask for IP address 10.10.10.12
    
    Enter network mask [255.255.255.0]:
    
    Enter an default gateway address for this machine.
    
    Note that the default gateway must be reachable from the first ethernet interface.
    
    Enter default gateway address [1.1.1.1]:10.10.10.1
    
    The second ethernet interface is currently disabled for this machine.
    Configure eth1 interface parameters? (Y)es/(S)kip/(U)se default [Yes]: S
    
    
    
    Step 10   Configure the Timezone settings:
    
    Please enter your choice [1 - 24]: 3
    
    Current Timezone=[America/New_York]
    Configure Timezone? (Y)es/(S)kip/(U)se default [Skip]: y
    
    Enter the current date and time.
    
    Please identify a location so that time zone rules can be set correctly.
    Please select a continent or ocean.
     1) Africa
     2) Americas
     3) Antarctica
     4) Arctic Ocean
     5) Asia
     6) Atlantic Ocean
     7) Australia
     8) Europe
     9) Indian Ocean
    10) Pacific Ocean
    11) UTC - I want to use Coordinated Universal Time.
    #? 2
    Please select a country.
     1) Anguilla                 27) Honduras
     2) Antigua & Barbuda        28) Jamaica
     3) Argentina                29) Martinique
     4) Aruba                    30) Mexico
     5) Bahamas                  31) Montserrat
     6) Barbados                 32) Netherlands Antilles
     7) Belize                   33) Nicaragua
     8) Bolivia                  34) Panama
     9) Brazil                   35) Paraguay
    10) Canada                   36) Peru
    11) Cayman Islands           37) Puerto Rico
    12) Chile                    38) St Barthelemy
    13) Colombia                 39) St Kitts & Nevis
    14) Costa Rica               40) St Lucia
    15) Cuba                     41) St Martin (French part)
    16) Dominica                 42) St Pierre & Miquelon
    17) Dominican Republic       43) St Vincent
    18) Ecuador                  44) Suriname
    19) El Salvador              45) Trinidad & Tobago
    20) French Guiana            46) Turks & Caicos Is
    21) Greenland                47) United States
    22) Grenada                  48) Uruguay
    23) Guadeloupe               49) Venezuela
    24) Guatemala                50) Virgin Islands (UK)
    25) Guyana                   51) Virgin Islands (US)
    26) Haiti
    #? 47
    Please select one of the following time zone regions.
     1) Eastern Time
     2) Eastern Time - Michigan - most locations
     3) Eastern Time - Kentucky - Louisville area
     4) Eastern Time - Kentucky - Wayne County
     5) Eastern Time - Indiana - most locations
     6) Eastern Time - Indiana - Daviess, Dubois, Knox & Martin Counties
     7) Eastern Time - Indiana - Pulaski County
     8) Eastern Time - Indiana - Crawford County
     9) Eastern Time - Indiana - Pike County
    10) Eastern Time - Indiana - Switzerland County
    11) Central Time
    12) Central Time - Indiana - Perry County
    13) Central Time - Indiana - Starke County
    14) Central Time - Michigan - Dickinson, Gogebic, Iron & Menominee Counties
    15) Central Time - North Dakota - Oliver County
    16) Central Time - North Dakota - Morton County (except Mandan area)
    17) Mountain Time
    18) Mountain Time - south Idaho & east Oregon
    19) Mountain Time - Navajo
    20) Mountain Standard Time - Arizona
    21) Pacific Time
    22) Alaska Time
    23) Alaska Time - Alaska panhandle
    24) Alaska Time - Alaska panhandle neck
    25) Alaska Time - west Alaska
    26) Aleutian Islands
    27) Hawaii
    #? 21
    
    The following information has been given:
    
            United States
            Pacific Time
    
    Therefore TZ='America/Los_Angeles' will be used.
    Local time is now:      Sun Apr  6 18:45:27 PDT 2014.
    Universal Time is now:  Mon Apr  7 01:45:27 UTC 2014.
    Is the above information OK?
    1) Yes
    2) No
    #? 1
    
    
    Step 11   Configure the DNS settings:
    
    Please enter your choice [1 - 24]: 11
    
    Domain Name Service (DNS) Setup
    
    Enable DNS (yes/no) [no]: y
    Default DNS server 1=[8.8.8.8]
    Enter primary DNS server IP address: 
    DNS server address must be in the form #.#.#.#, where # is 0 to 255 or hexadecimal : 
    separated v6 address
    
    Enter primary DNS server IP address [8.8.8.8]: 
    Enter backup DNS server IP address (or none) [none]:
    
    

    Step 12   Configure the NTP settings:
    
    Please enter your choice [1 - 24]: 5
    
    Network Time Protocol (NTP) Setup.
    
    If you choose to enable NTP, the system time will be
    configured from NTP servers that you select.  Otherwise,
    you will be prompted to enter the current date and time.
    
    NTP is currently disabled.
    Configure NTP related parameters? (Y)es/(S)kip/(U)se default [Skip]: y
    
    Enter whether or not you would like to set up the
    Network Time Protocol (NTP) for this machine.
    
    If you choose to enable NTP, the system time will be
    configured from NTP servers that you select.  Otherwise,
    you will be prompted to enter the current date and time.
    
    Enable NTP (yes/no) [no]: y
    Default NTP server 1=[time.nist.gov]
    Enter NTP server name or address:
    NTP server address must be in the form #.#.#.3, where # is 0 to 255 hexadecimal : 
    separated v6 address.
    Enter NTP server name or [time.nist.gov]:
    Enter another NTP server IP address (or none) [none]:
    Configure NTP Authentication ? (Y)es/(S)kip/(U)se default [Skip]: y
    Enter NTP Auth key Number [1]:
    Enter NTP Auth key Value (String) [Secret]:
    Do you want to continue (yes/no) [no]: y
    
    
    Step 13   Configure the Prime Infrastructure password:
    
    Please enter your choice [1 - 24]: 6
    
    Cisco Prime Infrastructure communication password has not been configured.
    Configure Prime Infrastructure password? (Y)es/(S)kip/(U)se default [Yes]: 
    
    Enter a password for the admin user.
    The admin user is used by the Prime Infrastructure and other northbound systems to authenticate 
    their SOAP/XML session with the server. Once this password is updated, it must correspondingly 
    be updated on the NCS page for MSE General Parameters so that the Prime Infrastructure can 
    communicate with the MSE.
    
    Step 14   Verify and apply changes
    Please enter your choice: 24
    
    Please verify the following setup information.
    
    -----------------------------BEGIN-------------------------------------------
    
         Hostname=mse1
         Role= 1, Health Monitor Intercace=eth0, Direct connect interface=none
    					Virtual IP Address=10.10.10.11, Virtual IP Netmask=255.255.255.0
    					Eth0 IP address=10.10.10.12, Eth0 network mask=255.0.0.0
         Default Gateway=10.10.10.1
    	    Time zone=America/Los_Angeles
    	    Enable DNS=yes, DNS servers=8.8.8.8
    	    Enable NTP=yes, NTP servers=time.nist.gov
    	    Time zone=America/Los_Angeles
    	    Root password is changed.
    	    Cisco Prime Infrastructure password is changed.
    
    ------------------------------END-----------------------------
    
    You may enter "yes" to proceed with configuration, "no" to make
    more changes.
    
    
    Configuration Changed
    Is the above information correct (yes or no): yes
    
    --------------------------------------------------------------
    Checking mandatory configuration information...
    
    Root password: Not configured
    
    **WARNING**
    The above parameters are mandatory and need to be configured.
    
    -------------------------------------------------------------
    Ignore and proceed (yes/no): yes
    Setup will now attempt to apply the configuration.
    Restarting network services with new settings.
    Shutting down interface eth0:
    
    The system is minimally configured right now. It is strongly recommended that you run the setup 
    script under /opt/mse/setup/setup.sh command to configure all appliance related parameters 
    immediately after installation is complete. 
    
    PRESS <ENTER> TO EXIT THE INSTALLER:
    Step 15   Reboot the system:
    [root@mse1]# reboot
    Stopping MSE Platform
    
    Flushing firewall rules:																			[OK]
    Setting chains to policy ACCEPT: nat filter 																			[OK]
    Unloading iptables modules:																			[ok]
    
    Broadcast message from root (pts/0) (Tue Apr29 14:15:27:2014):
    
    The system is going down for reboot NOW:
    Step 16   Start the MSE services:
    [root@mse1]# /etc/init.d/msed start
    Starting MSE Platform.
    
    Starting Health Monitor, Waiting to check the status.
    Starting Health Monitor, Waiting to check the status.
    Health Monitor successfully started
    Starting Admin process...
    Started Admin process.
    Starting database .....
    Database started successfully. STarting framework and services.......
    FRamework and services successfully started
    
    
    Step 17   After all services have started, confirm MSE services are working properly by entering the following command:
    [root@mse1]# getserverinfo
    
    
    Step 18   After the new installation, the initial login starts the Setup wizard where in you can configure the secondary MSE.
    Current hostanme=[mse1]
    Configure hostname? (Y)es/(S)kip/(U)se default [Yes]: yes
    
    The host name should be a unique name that can identifu the device on the network. The hostname
    should start with a letter, end with a letter or number, and contain only letters, numbers, and 
    dashes.
    
    Enter a hostname [mse]: mse2
    
    
    
    Step 19   Configure the domain:
    Please enter your choice [1-24]: 8
    Current domain=[ ]
    Configure domain name?  (Y)es/(S)kip/(U)se default [Skip]: S
    Step 20   Configure the High availability role:
    Current role=[Primary]
    Configure High Availability?  (Y)es/(S)kip/(U)se default [Skip]:
    
    High availability role for this MSE (Primary/Secondary)
    
    Select role [1 for Primary, 2 for Secondary] [1]: 2
    
    Health monitor interface holds physical IP address of this MSE server.
    This IP address is used by Secondary, Primary MSE servers and Prime Infrastructure to 
    communicate among themselves
    Select Health Monitor Interface [eth0/eth1] [eth0]: eth0 
    -------------------------------------------------------------------
    Direct connect configuration facilitates use of a direct cable connection between the primary and 
    secondary MSE servers.This can help reduce latencies in heartbeat response times, data 
    replication and failure detection times.Please choose a network interface that you wish to use 
    for direct connect. You should appropriately configure the respective interfaces.
    "none" implies you do not wish to use direct connect configuration.
    -------------------------------------------------------------------
    
    Select direct connect interface [eth0/eth1/none] [none]:
    
    Current IP address=[1.1.1.10]
    Current eth0 netmask=[255.255.255.0]
    Current gatewat address=[1.1.1.1]
    Configure eth0 interface parameters? (Y)es/(S)kip/(U)se default [Yes]:
    
    
    Enter an IP address for first ethernet interface of this machine.
    
    Enter eth0 IP address [1.1.1.10]: 10.10.10.13
    
    Enter the network mask for IP address 10.10.10.13
    
    Enter network mask [255.255.255.0]:
    
    Enter an default gateway address for this machine.
    
    Note that the default gateway must be reachable from the first ethernet interface.
    
    Enter default gateway address [1.1.1.1]:10.10.10.1
    
    The second ethernet interface is currently disabled for this machine.
    Configure eth1 interface parameters? (Y)es/(S)kip/(U)se default [Yes]: S
    
    
    
    Step 21   Configure the Timezone settings:
    Please enter your choice [1 - 24]: 3
    
    Current Timezone=[America/New_York]
    Configure Timezone? (Y)es/(S)kip/(U)se default [Skip]: y
    
    Enter the current date and time.
    
    Please identify a location so that time zone rules can be set correctly.
    Please select a continent or ocean.
     1) Africa
     2) Americas
     3) Antarctica
     4) Arctic Ocean
     5) Asia
     6) Atlantic Ocean
     7) Australia
     8) Europe
     9) Indian Ocean
    10) Pacific Ocean
    11) UTC - I want to use Coordinated Universal Time.
    #? 2
    Please select a country.
     1) Anguilla                 27) Honduras
     2) Antigua & Barbuda        28) Jamaica
     3) Argentina                29) Martinique
     4) Aruba                    30) Mexico
     5) Bahamas                  31) Montserrat
     6) Barbados                 32) Netherlands Antilles
     7) Belize                   33) Nicaragua
     8) Bolivia                  34) Panama
     9) Brazil                   35) Paraguay
    10) Canada                   36) Peru
    11) Cayman Islands           37) Puerto Rico
    12) Chile                    38) St Barthelemy
    13) Colombia                 39) St Kitts & Nevis
    14) Costa Rica               40) St Lucia
    15) Cuba                     41) St Martin (French part)
    16) Dominica                 42) St Pierre & Miquelon
    17) Dominican Republic       43) St Vincent
    18) Ecuador                  44) Suriname
    19) El Salvador              45) Trinidad & Tobago
    20) French Guiana            46) Turks & Caicos Is
    21) Greenland                47) United States
    22) Grenada                  48) Uruguay
    23) Guadeloupe               49) Venezuela
    24) Guatemala                50) Virgin Islands (UK)
    25) Guyana                   51) Virgin Islands (US)
    26) Haiti
    #? 47
    Please select one of the following time zone regions.
     1) Eastern Time
     2) Eastern Time - Michigan - most locations
     3) Eastern Time - Kentucky - Louisville area
     4) Eastern Time - Kentucky - Wayne County
     5) Eastern Time - Indiana - most locations
     6) Eastern Time - Indiana - Daviess, Dubois, Knox & Martin Counties
     7) Eastern Time - Indiana - Pulaski County
     8) Eastern Time - Indiana - Crawford County
     9) Eastern Time - Indiana - Pike County
    10) Eastern Time - Indiana - Switzerland County
    11) Central Time
    12) Central Time - Indiana - Perry County
    13) Central Time - Indiana - Starke County
    14) Central Time - Michigan - Dickinson, Gogebic, Iron & Menominee Counties
    15) Central Time - North Dakota - Oliver County
    16) Central Time - North Dakota - Morton County (except Mandan area)
    17) Mountain Time
    18) Mountain Time - south Idaho & east Oregon
    19) Mountain Time - Navajo
    20) Mountain Standard Time - Arizona
    21) Pacific Time
    22) Alaska Time
    23) Alaska Time - Alaska panhandle
    24) Alaska Time - Alaska panhandle neck
    25) Alaska Time - west Alaska
    26) Aleutian Islands
    27) Hawaii
    #? 21
    
    The following information has been given:
    
            United States
            Pacific Time
    
    Therefore TZ='America/Los_Angeles' will be used.
    Local time is now:      Sun Apr  6 18:45:27 PDT 2014.
    Universal Time is now:  Mon Apr  7 01:45:27 UTC 2014.
    Is the above information OK?
    1) Yes
    2) No
    #? 1
    
    
    Step 22   Configure the NTP settings:
    Please enter your choice [1 - 24]: 5
    
    Network Time Protocol (NTP) Setup.
    
    If you choose to enable NTP, the system time will be
    configured from NTP servers that you select.  Otherwise,
    you will be prompted to enter the current date and time.
    
    NTP is currently disabled.
    Configure NTP related parameters? (Y)es/(S)kip/(U)se default [Skip]: y
    
    Enter whether or not you would like to set up the
    Network Time Protocol (NTP) for this machine.
    
    If you choose to enable NTP, the system time will be
    configured from NTP servers that you select.  Otherwise,
    you will be prompted to enter the current date and time.
    
    Enable NTP (yes/no) [no]: y
    Default NTP server 1=[time.nist.gov]
    Enter NTP server name or address:
    NTP server address must be in the form #.#.#.3, where # is 0 to 255 hexadecimal : 
    separated v6 address.
    Enter NTP server name or [time.nist.gov]:
    Enter another NTP server IP address (or none) [none]:
    Configure NTP Authentication ? (Y)es/(S)kip/(U)se default [Skip]: y
    Enter NTP Auth key Number [1]:
    Enter NTP Auth key Value (String) [Secret]:
    Do you want to continue (yes/no) [no]: y
    
    
    Step 23   Verify and apply changes
    Please enter your choice: 24
    
    Please verify the following setup information.
    
    -----------------------------BEGIN-------------------------------------------
    
         Hostname=mse2
         Role= 2, Health Monitor Intercace=eth0, Direct connect interface=none
    					Eth0 IP address=10.10.10.13, Eth0 network mask=255.255.255.0
         Default Gateway=10.10.10.1
    	    Time zone=America/Los_Angeles
    	    Enable NTP=yes, NTP servers=time.nist.gov
    	    Time zone=America/Los_Angeles
    	    
    ------------------------------END-----------------------------
    
    You may enter "yes" to proceed with configuration, "no" to make
    more changes.
    
    
    Configuration Changed
    Is the above information correct (yes or no): yes
    
    --------------------------------------------------------------
    Checking mandatory configuration information...
    
    Root password: Not configured
    
    **WARNING**
    The above parameters are mandatory and need to be configured.
    
    -------------------------------------------------------------
    Ignore and proceed (yes/no): yes
    Setup will now attempt to apply the configuration.
    Restarting network services with new settings.
    Shutting down interface eth0:
    
    The system is minimally configured right now. It is strongly recommended that you run the setup 
    script under /opt/mse/setup/setup.sh command to configure all appliance related parameters 
    immediately after installation is complete. 
    
    PRESS <ENTER> TO EXIT THE INSTALLER:
    Step 24   Reboot the system:
    [root@mse2 installers]# reboot
    Stopping MSE Platform
    
    Flushing firewall rules:																			[OK]
    Setting chains to policy ACCEPT: nat filter 																			[OK]
    Unloading iptables modules:																			[ok]
    
    Broadcast message from root (pts/0) (Tue Apr29 14:15:27:2014):
    
    The system is going down for reboot NOW:
    Step 25   Start the MSE services:
    [root@mse2]# /etc/init.d/msed start
    Starting MSE Platform.
    
    Starting Health Monitor, Waiting to check the status.
    Starting Health Monitor, Waiting to check the status.
    Health Monitor successfully started
    Starting Admin process...
    Started Admin process.
    Starting database .....
    Database started successfully. STarting framework and services.......
    FRamework and services successfully started
    
    
    Step 26   Once you configure both the primary MSE and secondary MSE, the Prime Infrastructure UI should be used to set up a pairing between the primary and secondary MSE.
    Step 27   Once you add the primary MSE successfully, choose Services > High Availability or click the primary MSE device in the Services > Mobility Services Engine > System > Services High Availability > HA Configuration.

    The HA Configuration page appears.

    Step 28   Enter the secondary device name with which you want to pair the primary MSE.
    Step 29   Enter the secondary IP address which is the health monitor IP address of the secondary MSE.
    Step 30   Enter the secondary password. This is the Prime Infrastructure communication password configured on the MSE.
    Step 31   Specify the failover type. You can choose either Manual or Automatic from the Failover Type drop-down list.
    Step 32   Specify the failback type by choosing either Manual or Automatic from the Failback Type drop-down list.
    Step 33   Specify the Long Failover Wait in seconds.

    After 10 seconds, the system fails over. The maximum failover wait is two minutes.

    Step 34   Click Save.

    The pairing and the synchronization happens automatically.

    Step 35   To check whether the heartbeat is received from the primary MSE or not, choose Services > Mobility Services Engine > System > Services High Availability > HA Status.

    Viewing Configured Parameters for High Availability

    To view the configured parameters for high availability, follow these steps:


      Step 1   Choose Services > High Availability.
      Step 2   Click MSE Name to view its configured fields.

      The HA configuration page appears.

      Step 3   Choose Services High Availability > HA Configuration from the left sidebar menu. The HA Configuration page provides the following information:
      • Primary Health Monitor IP
      • Secondary Device Name
      • Secondary IP Address
      • Secondary Password
      • Failover Type
      • Failback Type
      • Long Failover Wait

      Viewing High Availability Status

      To view the high availability status, follow these steps:


        Step 1   Choose Services > High Availability.
        Step 2   Click MSE Name to view the desired status.

        The HA Configuration page appears.

        Step 3   Choose HA Status from the left sidebar menu. The HA Configuration page provides the following information:
        • Current High Availability Status
          • Status—Shows whether the primary and secondary MSE instances are correctly synchronized or not.
          • Heartbeats—Shows whether the heartbeat is received from the primary MSE or not.
          • Data Replication—Shows whether the data replication between the primary and secondary databases is happening or not.
          • Mean Heartbeat Response Time—Shows the mean heartbeat response time between the primary and secondary MSE instance.
        • Event Log—Shows all the events generated by the MSE. The last 20 events can be viewed.


        Configuring High Availability

        Configuring High Availability

        This chapter describes how to configure high availability on the Cisco Mobility Services Engine. The MSE is a platform for hosting multiple mobility applications. Every active MSE is backed up by another inactive instance. The active MSE is called the Primary MSE and the inactive MSE is called the Secondary MSE.

        The main component of high availability system is the health monitor. The health monitor configures, manages, and monitors the high availability setup. Heartbeat is maintained between the primary and secondary MSE. Health monitor is responsible for setting up the database, file replication, and monitoring the application. When the primary MSE fails and the secondary MSE takes over, the virtual address of the primary MSE is switched transparently to the secondary MSE.


        Note


        The MSEs, Synchronize Services, Synchronization History, High Availability, Context Aware Notifications, and Mobile Concierge pages on the Services tab are available only in the virtual domain in Release 7.3.

        Overview of the High Availability Architecture

        This section provides an overview of the high availability architecture.

        • Every active primary MSE is backed up by another inactive instance. The purpose of the secondary MSE is to monitor the availability and state of the primary MSE. The secondary MSE becomes active only after the failover procedure is initiated.
        • One secondary MSE can support one primary MSE.

        Pairing Matrix

        The following table lists the server type pairing matrix information.

        Table 1 Pairing Matrix
        Secondary Server Type
        Primary Server Type 3355 VA-2 VA-3 VA-4 VA-5
        3355 Y N N N N
        VA-2 N Y Y Y Y
        VA-3 N N Y Y Y
        VA-4 N N N Y Y
        VA-5 N N N N Y

        Guidelines and Limitations for High Availability

        • Both the health monitor IP and virtual IP should be accessible from Prime Infrastructure.
        • The health monitor IP and virtual IP should always be different. The health monitor and virtual interface can be on the same network interface or different interfaces.
        • You can use either manual or automatic failover. Failover should be considered temporary. The failed MSE should be restored to normal as soon as possible, and failback should be reinitiated. The longer it takes to restore the failed MSE, the longer you are running with a single MSE without high availability support.
        • You can use either manual or automatic failback.
        • Both the primary and secondary MSE should be running the same software version.
        • High Availability over WAN is not supported.
        • High Availability over LAN is supported only when both the primary and secondary MSEs are in the same subnet.
        • The ports over which the primary and secondary MSEs communicate must be open (not blocked with network firewalls, application fireways, gateways, and so on).

        Failover Scenario for High Availability

        When a primary MSE failure is detected, the following events occur:

        • The primary MSE is confirmed as non-functioning (hardware fail, network fail, and so on) by the health monitor on the secondary MSE.
        • If automatic failover has been enabled, the secondary MSE is starts immediately. If automatic failover is disabled, an e-mail is sent to the administrator asking if they want to manually start failover.
        • When manual failover is configured, an e-mail is sent only if the e-mail is configured for MSE alarms.
        • The result of the failover operation is indicated as an event in the Health Monitor UI, and a critical alarm is sent to Prime Infrastructure.

        Failback Scenario for High Availability

        When the primary MSE is restored to its normal state, if the secondary MSE is already in failover state for the primary, then failback can be invoked.

        Failback can occur only if the secondary MSE is in one of the following states for the primary instance:

        • The secondary MSE is actually failing over for the primary MSE.
        • Manual failover is configured but the administrator did not invoke it.
        • The primary MSE failed but the secondary MSE cannot take over because it has encountered errors.
        • Failback can occur only if the administrator starts up the failed primary MSE.

        HA Licensing

        For high availability, an activation license is required on the primary and secondary virtual appliances. No other service license is required on the secondary MSE. It is required only on the primary MSE.

        Configuring High Availability on the MSE

        During the installation of the MSE software (or using the MSE setup script), configure some critical elements. Pair up the primary and secondary MSE from the Prime Infrastructure UI.


        Note


        By default, all MSEs are configured as primary. If you do not want high availability support and are upgrading from an earlier release, you can continue to use the IP address for the MSE. If you want to set up high availability, then you must configure the health monitor IP address. The health monitor then becomes a virtual IP address.


        To configure high availability on the primary MSE, follow these steps:


          Step 1   Ensure that the network connectivity between the primary and secondary MSEs is functioning and that all the necessary ports are open.
          Step 2   Install the correct version of MSE on the primary MSE.
          Step 3   Make sure that the same MSE version is installed on the secondary MSE.
          Step 4   On the intended primary MSE, enter the following command:
          
          /opt/mse/setup/setup.sh
          --------------------------------------------------------------
          Welcome to the Cisco Mobility Services Engine Appliance Setup.
          
          You may exit the setup at any time by typing <Ctrl+c>.
          -----------------------------------------------------------
          
          Would you like to confgire MSE using:
                      1. Menu mode
                      2. Wizard mode
                      Choose 1 or 2: 1
          --------------------------------------------------
          
          Mobility Services Engine Setup
          
          Please select a configuration option belwo and enter the
          requested information. You may exit setup at any time by typing <Ctrl +C>.
          
          You will be prompted to choose whether you wish to configure a
          parameter, skip it, or reset it to its initial default value.
          Skipping a parameter will leave it unchanged from its current
          value.
          
          Please note that the following parameters are mandatory and must be configured at lease once.
                  -> Hostname
                  -> Network interface eth0
                  -> Timezone settings
                  -> Root password
                  -> NTP settings
                  -> Prime Infrastructure password
          
          You must select option 24 to verify and apply any changes made during this session.
          --------------------------------------------------------------
          
          PRESS <ENTER> TO CONTINUE:
          
          
          --------------------------------------------------------------
          
          Configure MSE:
          
          	1) Hostname *																					    13) Remote syslog settings
           2) Network interface eth0 settings*		 14) Host access control settings
           3) Timezone settings*																	15) Audit Rules
           4) Root password *																				16) Login banner
           5) NTP settings *																					17) System console restrictions
           6) Prime Infrastructure password *				18) SSH root access
           7) Display current configuration						19) Single user password check
           8) Domain																													20) Login and password settings
           9) High availability role													21) GRUB password
          10) Network interface eth1 settings				22) Root access control
          11) DNS settings																							23) Auto start MSE on system boot up
          12) Future restart time																24) ## Verify and apply changes ##
          
          Please enter your choice [1 - 24]:
          
          --------------------------------------------------------------
          
          Step 5   Configure the hostname:
          
          Please enter your choice [1 - 24]: 1
          
          Current Hostname=[mse]
          Configure Hostname?  (Y)es/(S)kip/(U)se default [Skip]: y
          The host name should be a unique name that can identify
          the device on the network. The hostname should start with
          a letter, end with a letter or number, and contain only
          letters, numbers, and dashes.
          
          Enter a Host name [mse]:mse1
          
          
          Step 6   Configure the domain:
          Please enter your choice [1-24]: 8
          Current domain=[ ]
          Configure domain name?  (Y)es/(S)kip/(U)se default [Skip]: S
          Step 7   Configure the Network interface eth0 settings.
          
          Please enter your choice [1 - 24]: 2
          
          Current eth0 interface IP address=[10.0.0.1]
          Current eth0 interface netmask=[255.0.0.0]
          Current IPv4 gateway address=[172.20.104.123]
          Configure eth0 interface parameters? (Y)es/(S)kip/(U)se default [Skip]:y
          
          Enter an IP address for first ethernet interface of this machine.
          
          Enter eth0 IP address  [10.0.0.2]: 
          
          Enter the network mask for IP address 172.21.105.126
          
          Enter network mask [255.255.255.224]: 
          
          Enter the default gateway address for this machine.
          
          Note that the default gateway must be reachable from
          the first ethernet interface.
          
          Enter default gateway address [172.20.104.123]:
          
          
          Step 8   Configure the Root password:
          
          Please enter your choice [1 - 24]: 4
          
          Root password has not been configured
          Configure root password? (Y)es/(S)kip/(U)se default [Skip]: 
          Changing password for user root.
          
          You can now choose the new password.
          
          A valid password should be a mix of upper and lower case letters, digits, and other characters. You can use an 8 character long password with characters from all of these classes. An upper case letter that begins the password and a digit that ends it do not count towards the number of character classes used.
          
          Enter new password:
          
          
          Step 9   Configure the High availability role:
          
          Current role=[Primary]
          Configure High Availability?  (Y)es/(S)kip/(U)se default [Skip]:
          
          High availability role for this MSE (Primary/Secondary)
          
          Select role [1 for Primary, 2 for Secondary] [1]: 1
          
          Health monitor interface holds physical IP address of this MSE server.
          This IP address is used by Secondary, Primary MSE servers and Prime Infrastructure to 
          communicate among themselves
          Select Health Monitor Interface [eth0/eth1] [eth0]: eth0 
          -------------------------------------------------------------------
          Direct connect configuration facilitates use of a direct cable connection between the primary and secondary MSE servers.
          This can help reduce latencies in heartbeat response times, data replication and failure detection times.
          Please choose a network interface that you wish to use for direct connect. You should appropriately configure the respective interfaces.
          "none" implies you do not wish to use direct connect configuration.
          -------------------------------------------------------------------
          
          Select direct connect interface [eth0/eth1/none] [none]:
          Enter a Virtual IP address for the Primary MSE server
          
          Enter Virtual IP address [1.1.1.1]: 10.10.10.11
          
          Enter network mask for IP address 10.10.10.1
          
          Enter network mask [1.1.1.1]: 255.255.255.0
          
          Select to start the sever in recovery mode.
          You should choose yes only if this primary MSE was paired earlier and you have nowlost the 
          configuration from this box.
          And, now you want to restore the configuration from Secondary via Cisco  Prime Infrastructure
          Do you wish to start this MSE in HA receovery mode?: (yes/no} [no]:no
          
          Current IP address = [1.1.1.10]
          Current eth0 netmask=[255.255.255.0]
          Current gateway address=[1.1.1.1]
          Configure eth0 interface parameters? (Y)es/(S)kip/(U)se default [Skip]:
          
          Enter an IP address for first ethernet interface of this machine.
          
          Enter eth0 IP address [1.1.1.10]: 10.10.10.12
          
          Enter the network mask for IP address 10.10.10.12
          
          Enter network mask [255.255.255.0]:
          
          Enter an default gateway address for this machine.
          
          Note that the default gateway must be reachable from the first ethernet interface.
          
          Enter default gateway address [1.1.1.1]:10.10.10.1
          
          The second ethernet interface is currently disabled for this machine.
          Configure eth1 interface parameters? (Y)es/(S)kip/(U)se default [Yes]: S
          
          
          
          Step 10   Configure the Timezone settings:
          
          Please enter your choice [1 - 24]: 3
          
          Current Timezone=[America/New_York]
          Configure Timezone? (Y)es/(S)kip/(U)se default [Skip]: y
          
          Enter the current date and time.
          
          Please identify a location so that time zone rules can be set correctly.
          Please select a continent or ocean.
           1) Africa
           2) Americas
           3) Antarctica
           4) Arctic Ocean
           5) Asia
           6) Atlantic Ocean
           7) Australia
           8) Europe
           9) Indian Ocean
          10) Pacific Ocean
          11) UTC - I want to use Coordinated Universal Time.
          #? 2
          Please select a country.
           1) Anguilla                 27) Honduras
           2) Antigua & Barbuda        28) Jamaica
           3) Argentina                29) Martinique
           4) Aruba                    30) Mexico
           5) Bahamas                  31) Montserrat
           6) Barbados                 32) Netherlands Antilles
           7) Belize                   33) Nicaragua
           8) Bolivia                  34) Panama
           9) Brazil                   35) Paraguay
          10) Canada                   36) Peru
          11) Cayman Islands           37) Puerto Rico
          12) Chile                    38) St Barthelemy
          13) Colombia                 39) St Kitts & Nevis
          14) Costa Rica               40) St Lucia
          15) Cuba                     41) St Martin (French part)
          16) Dominica                 42) St Pierre & Miquelon
          17) Dominican Republic       43) St Vincent
          18) Ecuador                  44) Suriname
          19) El Salvador              45) Trinidad & Tobago
          20) French Guiana            46) Turks & Caicos Is
          21) Greenland                47) United States
          22) Grenada                  48) Uruguay
          23) Guadeloupe               49) Venezuela
          24) Guatemala                50) Virgin Islands (UK)
          25) Guyana                   51) Virgin Islands (US)
          26) Haiti
          #? 47
          Please select one of the following time zone regions.
           1) Eastern Time
           2) Eastern Time - Michigan - most locations
           3) Eastern Time - Kentucky - Louisville area
           4) Eastern Time - Kentucky - Wayne County
           5) Eastern Time - Indiana - most locations
           6) Eastern Time - Indiana - Daviess, Dubois, Knox & Martin Counties
           7) Eastern Time - Indiana - Pulaski County
           8) Eastern Time - Indiana - Crawford County
           9) Eastern Time - Indiana - Pike County
          10) Eastern Time - Indiana - Switzerland County
          11) Central Time
          12) Central Time - Indiana - Perry County
          13) Central Time - Indiana - Starke County
          14) Central Time - Michigan - Dickinson, Gogebic, Iron & Menominee Counties
          15) Central Time - North Dakota - Oliver County
          16) Central Time - North Dakota - Morton County (except Mandan area)
          17) Mountain Time
          18) Mountain Time - south Idaho & east Oregon
          19) Mountain Time - Navajo
          20) Mountain Standard Time - Arizona
          21) Pacific Time
          22) Alaska Time
          23) Alaska Time - Alaska panhandle
          24) Alaska Time - Alaska panhandle neck
          25) Alaska Time - west Alaska
          26) Aleutian Islands
          27) Hawaii
          #? 21
          
          The following information has been given:
          
                  United States
                  Pacific Time
          
          Therefore TZ='America/Los_Angeles' will be used.
          Local time is now:      Sun Apr  6 18:45:27 PDT 2014.
          Universal Time is now:  Mon Apr  7 01:45:27 UTC 2014.
          Is the above information OK?
          1) Yes
          2) No
          #? 1
          
          
          Step 11   Configure the DNS settings:
          
          Please enter your choice [1 - 24]: 11
          
          Domain Name Service (DNS) Setup
          
          Enable DNS (yes/no) [no]: y
          Default DNS server 1=[8.8.8.8]
          Enter primary DNS server IP address: 
          DNS server address must be in the form #.#.#.#, where # is 0 to 255 or hexadecimal : 
          separated v6 address
          
          Enter primary DNS server IP address [8.8.8.8]: 
          Enter backup DNS server IP address (or none) [none]:
          
          

          Step 12   Configure the NTP settings:
          
          Please enter your choice [1 - 24]: 5
          
          Network Time Protocol (NTP) Setup.
          
          If you choose to enable NTP, the system time will be
          configured from NTP servers that you select.  Otherwise,
          you will be prompted to enter the current date and time.
          
          NTP is currently disabled.
          Configure NTP related parameters? (Y)es/(S)kip/(U)se default [Skip]: y
          
          Enter whether or not you would like to set up the
          Network Time Protocol (NTP) for this machine.
          
          If you choose to enable NTP, the system time will be
          configured from NTP servers that you select.  Otherwise,
          you will be prompted to enter the current date and time.
          
          Enable NTP (yes/no) [no]: y
          Default NTP server 1=[time.nist.gov]
          Enter NTP server name or address:
          NTP server address must be in the form #.#.#.3, where # is 0 to 255 hexadecimal : 
          separated v6 address.
          Enter NTP server name or [time.nist.gov]:
          Enter another NTP server IP address (or none) [none]:
          Configure NTP Authentication ? (Y)es/(S)kip/(U)se default [Skip]: y
          Enter NTP Auth key Number [1]:
          Enter NTP Auth key Value (String) [Secret]:
          Do you want to continue (yes/no) [no]: y
          
          
          Step 13   Configure the Prime Infrastructure password:
          
          Please enter your choice [1 - 24]: 6
          
          Cisco Prime Infrastructure communication password has not been configured.
          Configure Prime Infrastructure password? (Y)es/(S)kip/(U)se default [Yes]: 
          
          Enter a password for the admin user.
          The admin user is used by the Prime Infrastructure and other northbound systems to authenticate 
          their SOAP/XML session with the server. Once this password is updated, it must correspondingly 
          be updated on the NCS page for MSE General Parameters so that the Prime Infrastructure can 
          communicate with the MSE.
          
          Step 14   Verify and apply changes
          Please enter your choice: 24
          
          Please verify the following setup information.
          
          -----------------------------BEGIN-------------------------------------------
          
               Hostname=mse1
               Role= 1, Health Monitor Intercace=eth0, Direct connect interface=none
          					Virtual IP Address=10.10.10.11, Virtual IP Netmask=255.255.255.0
          					Eth0 IP address=10.10.10.12, Eth0 network mask=255.0.0.0
               Default Gateway=10.10.10.1
          	    Time zone=America/Los_Angeles
          	    Enable DNS=yes, DNS servers=8.8.8.8
          	    Enable NTP=yes, NTP servers=time.nist.gov
          	    Time zone=America/Los_Angeles
          	    Root password is changed.
          	    Cisco Prime Infrastructure password is changed.
          
          ------------------------------END-----------------------------
          
          You may enter "yes" to proceed with configuration, "no" to make
          more changes.
          
          
          Configuration Changed
          Is the above information correct (yes or no): yes
          
          --------------------------------------------------------------
          Checking mandatory configuration information...
          
          Root password: Not configured
          
          **WARNING**
          The above parameters are mandatory and need to be configured.
          
          -------------------------------------------------------------
          Ignore and proceed (yes/no): yes
          Setup will now attempt to apply the configuration.
          Restarting network services with new settings.
          Shutting down interface eth0:
          
          The system is minimally configured right now. It is strongly recommended that you run the setup 
          script under /opt/mse/setup/setup.sh command to configure all appliance related parameters 
          immediately after installation is complete. 
          
          PRESS <ENTER> TO EXIT THE INSTALLER:
          Step 15   Reboot the system:
          [root@mse1]# reboot
          Stopping MSE Platform
          
          Flushing firewall rules:																			[OK]
          Setting chains to policy ACCEPT: nat filter 																			[OK]
          Unloading iptables modules:																			[ok]
          
          Broadcast message from root (pts/0) (Tue Apr29 14:15:27:2014):
          
          The system is going down for reboot NOW:
          Step 16   Start the MSE services:
          [root@mse1]# /etc/init.d/msed start
          Starting MSE Platform.
          
          Starting Health Monitor, Waiting to check the status.
          Starting Health Monitor, Waiting to check the status.
          Health Monitor successfully started
          Starting Admin process...
          Started Admin process.
          Starting database .....
          Database started successfully. STarting framework and services.......
          FRamework and services successfully started
          
          
          Step 17   After all services have started, confirm MSE services are working properly by entering the following command:
          [root@mse1]# getserverinfo
          
          
          Step 18   After the new installation, the initial login starts the Setup wizard where in you can configure the secondary MSE.
          Current hostanme=[mse1]
          Configure hostname? (Y)es/(S)kip/(U)se default [Yes]: yes
          
          The host name should be a unique name that can identifu the device on the network. The hostname
          should start with a letter, end with a letter or number, and contain only letters, numbers, and 
          dashes.
          
          Enter a hostname [mse]: mse2
          
          
          
          Step 19   Configure the domain:
          Please enter your choice [1-24]: 8
          Current domain=[ ]
          Configure domain name?  (Y)es/(S)kip/(U)se default [Skip]: S
          Step 20   Configure the High availability role:
          Current role=[Primary]
          Configure High Availability?  (Y)es/(S)kip/(U)se default [Skip]:
          
          High availability role for this MSE (Primary/Secondary)
          
          Select role [1 for Primary, 2 for Secondary] [1]: 2
          
          Health monitor interface holds physical IP address of this MSE server.
          This IP address is used by Secondary, Primary MSE servers and Prime Infrastructure to 
          communicate among themselves
          Select Health Monitor Interface [eth0/eth1] [eth0]: eth0 
          -------------------------------------------------------------------
          Direct connect configuration facilitates use of a direct cable connection between the primary and 
          secondary MSE servers.This can help reduce latencies in heartbeat response times, data 
          replication and failure detection times.Please choose a network interface that you wish to use 
          for direct connect. You should appropriately configure the respective interfaces.
          "none" implies you do not wish to use direct connect configuration.
          -------------------------------------------------------------------
          
          Select direct connect interface [eth0/eth1/none] [none]:
          
          Current IP address=[1.1.1.10]
          Current eth0 netmask=[255.255.255.0]
          Current gatewat address=[1.1.1.1]
          Configure eth0 interface parameters? (Y)es/(S)kip/(U)se default [Yes]:
          
          
          Enter an IP address for first ethernet interface of this machine.
          
          Enter eth0 IP address [1.1.1.10]: 10.10.10.13
          
          Enter the network mask for IP address 10.10.10.13
          
          Enter network mask [255.255.255.0]:
          
          Enter an default gateway address for this machine.
          
          Note that the default gateway must be reachable from the first ethernet interface.
          
          Enter default gateway address [1.1.1.1]:10.10.10.1
          
          The second ethernet interface is currently disabled for this machine.
          Configure eth1 interface parameters? (Y)es/(S)kip/(U)se default [Yes]: S
          
          
          
          Step 21   Configure the Timezone settings:
          Please enter your choice [1 - 24]: 3
          
          Current Timezone=[America/New_York]
          Configure Timezone? (Y)es/(S)kip/(U)se default [Skip]: y
          
          Enter the current date and time.
          
          Please identify a location so that time zone rules can be set correctly.
          Please select a continent or ocean.
           1) Africa
           2) Americas
           3) Antarctica
           4) Arctic Ocean
           5) Asia
           6) Atlantic Ocean
           7) Australia
           8) Europe
           9) Indian Ocean
          10) Pacific Ocean
          11) UTC - I want to use Coordinated Universal Time.
          #? 2
          Please select a country.
           1) Anguilla                 27) Honduras
           2) Antigua & Barbuda        28) Jamaica
           3) Argentina                29) Martinique
           4) Aruba                    30) Mexico
           5) Bahamas                  31) Montserrat
           6) Barbados                 32) Netherlands Antilles
           7) Belize                   33) Nicaragua
           8) Bolivia                  34) Panama
           9) Brazil                   35) Paraguay
          10) Canada                   36) Peru
          11) Cayman Islands           37) Puerto Rico
          12) Chile                    38) St Barthelemy
          13) Colombia                 39) St Kitts & Nevis
          14) Costa Rica               40) St Lucia
          15) Cuba                     41) St Martin (French part)
          16) Dominica                 42) St Pierre & Miquelon
          17) Dominican Republic       43) St Vincent
          18) Ecuador                  44) Suriname
          19) El Salvador              45) Trinidad & Tobago
          20) French Guiana            46) Turks & Caicos Is
          21) Greenland                47) United States
          22) Grenada                  48) Uruguay
          23) Guadeloupe               49) Venezuela
          24) Guatemala                50) Virgin Islands (UK)
          25) Guyana                   51) Virgin Islands (US)
          26) Haiti
          #? 47
          Please select one of the following time zone regions.
           1) Eastern Time
           2) Eastern Time - Michigan - most locations
           3) Eastern Time - Kentucky - Louisville area
           4) Eastern Time - Kentucky - Wayne County
           5) Eastern Time - Indiana - most locations
           6) Eastern Time - Indiana - Daviess, Dubois, Knox & Martin Counties
           7) Eastern Time - Indiana - Pulaski County
           8) Eastern Time - Indiana - Crawford County
           9) Eastern Time - Indiana - Pike County
          10) Eastern Time - Indiana - Switzerland County
          11) Central Time
          12) Central Time - Indiana - Perry County
          13) Central Time - Indiana - Starke County
          14) Central Time - Michigan - Dickinson, Gogebic, Iron & Menominee Counties
          15) Central Time - North Dakota - Oliver County
          16) Central Time - North Dakota - Morton County (except Mandan area)
          17) Mountain Time
          18) Mountain Time - south Idaho & east Oregon
          19) Mountain Time - Navajo
          20) Mountain Standard Time - Arizona
          21) Pacific Time
          22) Alaska Time
          23) Alaska Time - Alaska panhandle
          24) Alaska Time - Alaska panhandle neck
          25) Alaska Time - west Alaska
          26) Aleutian Islands
          27) Hawaii
          #? 21
          
          The following information has been given:
          
                  United States
                  Pacific Time
          
          Therefore TZ='America/Los_Angeles' will be used.
          Local time is now:      Sun Apr  6 18:45:27 PDT 2014.
          Universal Time is now:  Mon Apr  7 01:45:27 UTC 2014.
          Is the above information OK?
          1) Yes
          2) No
          #? 1
          
          
          Step 22   Configure the NTP settings:
          Please enter your choice [1 - 24]: 5
          
          Network Time Protocol (NTP) Setup.
          
          If you choose to enable NTP, the system time will be
          configured from NTP servers that you select.  Otherwise,
          you will be prompted to enter the current date and time.
          
          NTP is currently disabled.
          Configure NTP related parameters? (Y)es/(S)kip/(U)se default [Skip]: y
          
          Enter whether or not you would like to set up the
          Network Time Protocol (NTP) for this machine.
          
          If you choose to enable NTP, the system time will be
          configured from NTP servers that you select.  Otherwise,
          you will be prompted to enter the current date and time.
          
          Enable NTP (yes/no) [no]: y
          Default NTP server 1=[time.nist.gov]
          Enter NTP server name or address:
          NTP server address must be in the form #.#.#.3, where # is 0 to 255 hexadecimal : 
          separated v6 address.
          Enter NTP server name or [time.nist.gov]:
          Enter another NTP server IP address (or none) [none]:
          Configure NTP Authentication ? (Y)es/(S)kip/(U)se default [Skip]: y
          Enter NTP Auth key Number [1]:
          Enter NTP Auth key Value (String) [Secret]:
          Do you want to continue (yes/no) [no]: y
          
          
          Step 23   Verify and apply changes
          Please enter your choice: 24
          
          Please verify the following setup information.
          
          -----------------------------BEGIN-------------------------------------------
          
               Hostname=mse2
               Role= 2, Health Monitor Intercace=eth0, Direct connect interface=none
          					Eth0 IP address=10.10.10.13, Eth0 network mask=255.255.255.0
               Default Gateway=10.10.10.1
          	    Time zone=America/Los_Angeles
          	    Enable NTP=yes, NTP servers=time.nist.gov
          	    Time zone=America/Los_Angeles
          	    
          ------------------------------END-----------------------------
          
          You may enter "yes" to proceed with configuration, "no" to make
          more changes.
          
          
          Configuration Changed
          Is the above information correct (yes or no): yes
          
          --------------------------------------------------------------
          Checking mandatory configuration information...
          
          Root password: Not configured
          
          **WARNING**
          The above parameters are mandatory and need to be configured.
          
          -------------------------------------------------------------
          Ignore and proceed (yes/no): yes
          Setup will now attempt to apply the configuration.
          Restarting network services with new settings.
          Shutting down interface eth0:
          
          The system is minimally configured right now. It is strongly recommended that you run the setup 
          script under /opt/mse/setup/setup.sh command to configure all appliance related parameters 
          immediately after installation is complete. 
          
          PRESS <ENTER> TO EXIT THE INSTALLER:
          Step 24   Reboot the system:
          [root@mse2 installers]# reboot
          Stopping MSE Platform
          
          Flushing firewall rules:																			[OK]
          Setting chains to policy ACCEPT: nat filter 																			[OK]
          Unloading iptables modules:																			[ok]
          
          Broadcast message from root (pts/0) (Tue Apr29 14:15:27:2014):
          
          The system is going down for reboot NOW:
          Step 25   Start the MSE services:
          [root@mse2]# /etc/init.d/msed start
          Starting MSE Platform.
          
          Starting Health Monitor, Waiting to check the status.
          Starting Health Monitor, Waiting to check the status.
          Health Monitor successfully started
          Starting Admin process...
          Started Admin process.
          Starting database .....
          Database started successfully. STarting framework and services.......
          FRamework and services successfully started
          
          
          Step 26   Once you configure both the primary MSE and secondary MSE, the Prime Infrastructure UI should be used to set up a pairing between the primary and secondary MSE.
          Step 27   Once you add the primary MSE successfully, choose Services > High Availability or click the primary MSE device in the Services > Mobility Services Engine > System > Services High Availability > HA Configuration.

          The HA Configuration page appears.

          Step 28   Enter the secondary device name with which you want to pair the primary MSE.
          Step 29   Enter the secondary IP address which is the health monitor IP address of the secondary MSE.
          Step 30   Enter the secondary password. This is the Prime Infrastructure communication password configured on the MSE.
          Step 31   Specify the failover type. You can choose either Manual or Automatic from the Failover Type drop-down list.
          Step 32   Specify the failback type by choosing either Manual or Automatic from the Failback Type drop-down list.
          Step 33   Specify the Long Failover Wait in seconds.

          After 10 seconds, the system fails over. The maximum failover wait is two minutes.

          Step 34   Click Save.

          The pairing and the synchronization happens automatically.

          Step 35   To check whether the heartbeat is received from the primary MSE or not, choose Services > Mobility Services Engine > System > Services High Availability > HA Status.

          Viewing Configured Parameters for High Availability

          To view the configured parameters for high availability, follow these steps:


            Step 1   Choose Services > High Availability.
            Step 2   Click MSE Name to view its configured fields.

            The HA configuration page appears.

            Step 3   Choose Services High Availability > HA Configuration from the left sidebar menu. The HA Configuration page provides the following information:
            • Primary Health Monitor IP
            • Secondary Device Name
            • Secondary IP Address
            • Secondary Password
            • Failover Type
            • Failback Type
            • Long Failover Wait

            Viewing High Availability Status

            To view the high availability status, follow these steps:


              Step 1   Choose Services > High Availability.
              Step 2   Click MSE Name to view the desired status.

              The HA Configuration page appears.

              Step 3   Choose HA Status from the left sidebar menu. The HA Configuration page provides the following information:
              • Current High Availability Status
                • Status—Shows whether the primary and secondary MSE instances are correctly synchronized or not.
                • Heartbeats—Shows whether the heartbeat is received from the primary MSE or not.
                • Data Replication—Shows whether the data replication between the primary and secondary databases is happening or not.
                • Mean Heartbeat Response Time—Shows the mean heartbeat response time between the primary and secondary MSE instance.
              • Event Log—Shows all the events generated by the MSE. The last 20 events can be viewed.