Before deploying the VMs, build the VM images by executing the following command from the Cluster Manager VM:

/var/qps/install/current/scripts/build_all.sh

Building /etc/broadhop...
Copying to /var/qps/images/etc.tar.gz...
...
Copying wispr.war to /var/qps/images/wispr.war
Output images to /var/qps/images/
[root@hostname]#

This section describes the steps to deploy each VM in the CPS deployment individually. To deploy all of the VMs in parallel using a single command refer to Automatic Deployment of All CPS VMs in Parallel. To deploy a selective list of VMs in parallel using a single command refer to Automatic Deployment of Selective CPS VMs in Parallel.

Note	Before proceeding, refer to License Generation and Installation to confirm you have installed the license correctly.

For each host that is defined in the Hosts tab of the CPS Deployment Template spreadsheet execute the following:

Note	The following command uses the short alias name (qns01 qns02 etc.) as defined in the Hosts tab of the CPS Deployment Template. It will not work if you enter the full hostname.

/var/qps/install/current/scripts/deployer/deploy.sh $host

where, $host is the short alias name and not the full host name.

For example,

./deploy.sh qns01 < === passed

./deploy.sh NDC2BSND2QNS01 < === failed

Important

Newly deployed VM/VMs need to be shutdown cleanly and started with your preferred method to reserve memory: To shutdown individual VM: cd /var/qps/install/current/scripts/deployer ./deploy.sh <vm alias> --shutdownvm Start the VM: ./deploy.sh <vm alias> --poweronvm

This section describes the steps to deploy all VMs in parallel in the CPS deployment.

Note	Before proceeding, refer to License Generation and Installation to confirm you have installed the license correctly.

Execute the following command:

python /var/qps/install/current/scripts/deployer/support/deploy_all.py

The order in which VMs are deployed is managed internally.

Note	The amount of time needed to complete the entire deployment process depends on the number of VMs being deployed as well as the hardware on which it is being deployed.

The following is a sample list of VM hosts deployed. The list varies according to the type of CPS deployment as well as the information you entered in the CPS Deployment Template.

• pcrfclient01

• pcrfclient02

• sessionmgr01

• sessionmgr02

• lb01

• lb02

• qns01

• qns02

• qns03

• qns04

Note	To install the VMs using shared or single storage, you must use /var/qps/install/current/scripts/deployer/deploy.sh $host command. For more information, refer to Manual Deployment.

This section describes the steps to deploy a selective list of VMs in parallel in the CPS deployment.

Note	Before proceeding, refer to License Generation and Installation to confirm you have installed the license correctly.

Execute the following command:

python /var/qps/install/current/scripts/deployer/support/deploy_all.py --vms <filename-of-vms>

where, <filename-of-vms> is the name of the file containing the list of VMs such as:

pcrfclient01

lb01

qns01

Note	The amount of time needed to complete the entire deployment process depends on the number of VMs being deployed as well as the hardware on which it is being deployed.

Important

After deployment of load balancer VM, verify monit service status by executing the following command on deployed Load Balancer (lb) VM: /bin/systemctl status monit.service If monit service on load balancer VM is not running, then execute the following command on that VM to start it: /bin/systemctl start monit.service

Important

Newly deployed VM/VMs need to be shutdown cleanly and started with your preferred method to reserve memory: Shut down and start Selective CPS VMs in Parallel. Use your preferred editor and create /tmp/vm-list file and add VMs which you want to shut down and start. To shutdown VMs from the given list. cd /var/qps/install/current/scripts/deployer/support python deploy_all.py --vms /tmp/vm-list --poweroffvm Note Make sure that all the VMs in the list are powered OFF by using the above command. To start all the VMs in list. python deploy_all.py --vms /tmp/vm-list --poweronvm

The standard definition for supported replica-set is defined in the mongoConfig.cfg file.

Use the /etc/broadhop/ha_mongoconfig_template file to create the /etc/broadhop/mongoConfig.cfg and modify it to your requirements.

Note	If you are using VIP for arbiter, it is always recommended to keep VIP and all mongod processes on pcrfclient02 (by default).

Consider the following guidelines for choosing MongoDB ports for replica-sets:

• Port must not be in use by any other application. To check whether the port is in use, login to VM on which replica-set is to be created and execute the following command:

  netstat -lnp | grep <port_no>

  If no process is using same port, then port can be chosen for replica-set for binding.

• Port number used should be greater than 1024 and not in ephemeral port range i.e, not in between following range :

  net.ipv4.ip_local_port_range = 32768 to 61000

• While configuring MongoDB ports in a geographically redundant environment, there should be a difference of 100 ports between two respective sites. For example, consider there are two sites: Site1 and Site2. For Site1, if the port number used is 27717, then you can configure 27817 as the port number for Site2. This is helpful to identify a MongoDB member’s site. By looking at first three digits, you can decide where the MongoDB member belongs to. However, this is just a guideline. You must avoid having MongoDB ports of two different sites to close to each other (for example, 27717 on Site-1 and 27718 on Site2).

  Reason: The build_set.sh script fails when you create shards on the site (for example, Site1). This is because the script calculates the highest port number in the mongoConfig on the site where you are creating shards. This creates a clash between the replica-sets on both sites because the port number which it allocates might overlap with the port number of mongoConfig on other site (for example, Site2). This is the reason why there should be some gap in the port numbers allocated between both the sites.

The replica-set script is used to create replica-sets for the following databases. For more information about the script, see Script Usage.

• session

• spr

• balance

• report

• audit

• admin

• It is recommended to use the specific option for creating a single replica-set rather than --all option as it is easy to recreate it again if it fails to create.

• If recreating a replica-set on a production system, make sure to back up the database (Refer CPS Backup and Restore Guide).

• Auto Intelligent DB Operations (AIDO) server is running on Cluster Manager or third-party site Arbiter.

  • It is not active on third-party site Arbiter node, i.e., using monit summary you can see aido_server is running but in /var/log/aido_server.log you can see the following message:

    AIDO server is not needed on arbiter/site

  • It pushes latest or updated mongoConfig.cfg file to all database members every 60 seconds interval.

  • It checks if any database member is UP and ready to join a replica-set. If Yes, then checks whether replica-set exist or not. If replica-set exists, then join as a member in the existing replica-set. If replica-set does not exist, then create new replica sets

  • Monit process name is aido_server.

  • AIDO server status can be checked by using /etc/init.d/aido_server status and systemctl status aido_server

  • Log rotate file is available at: /etc/logrotate.d/aido_server, size limit is 10 M and 5 rotation

• AIDO client is running on sessionmgr, pcrfclient and third-party site Arbiter.

  • mongoConfig.cfg file is received from AIDO servers (in GR, multiple AIDO servers are available).

    mongoConfig.cfg file is available at: /var/aido

    File name format is: /var/aido/mongoConfig.cfg.<<cluman-host-name>>-<<--cluman-eth0-IP-->>

    AIDO server pushes mongoConfig.cfg file to all database members i.e., AIDO clients.

  • AIDO client status can be checked by using /etc/init.d/aido_client status and systemctl status aido_client

  • Log rotate file is availabe at: /etc/logrotate.d/aido_client, size limit is 10 M and 5 rotation

Note

You have to refer to /etc/broadhop/ha_mongoconfig_template file and use this file to create /etc/broadhop/mongoConfig.cfg file based on your requirements. All the replica set members and required information like Host Name and port number arbiter host name and port number should be defined in /etc/broadhop/mongoConfig.cfg file.

Note	Make sure all the replica set ports defined in the mongoConfig.cfg file are outside the range 32768 to 61000. For more information about the port range, refer to http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html.

The following example shows replica-set set04:

Run the /var/qps/install/current/scripts/build/build_etc.sh script from the Cluster Manager to finalize mongoConfig.cfg file after AIDO automatically takes care of updating it.

build_set.sh script copies /etc/broadhop/mongoConfig.cfg file to /var/www/html/images/mongoConfig.cfg file.

build_set.sh script is used to verify replica-set creation.

Option to view help: /var/qps/bin/support/mongo/build_set.sh --help

build_set.sh --help


              Replica-set Configuration
-------------------------------------------------------------------------------

Usage: build_set.sh <--option1> <--option2> [--setname SETNAME] [--help]
option1: Database name
option2: Build operations (create, add or remove members)
option3: Use --setname SETNAME to build or alter a specific replica-set
         replica-set setnames are defined in the /etc/broadhop/mongoConfig.cfg file

The script applies to Database: session, spr, balance, report, portal, admin, audit and bindings db replica-sets
                 Config Server: session_configs, spr_configs and bindings_configs db replica-sets

--all                     : Alias for all databases in the configuration
--create                  : Create a replica-set if force option is given, else it just validate
--create-asc              : Create a replica-set with set priority in the ascending format if
                            force option is given, else it just validate
--create-des              : Create a replica-set with set priority in the descending format if
                            force option is given, else it just validate
--add-members             : Add members to a replica-set if force option is given, else it just validate
                            This applies to members which have been removed from the replica-set using the
                            --remove-members and --remove-failed-members operations
--remove-members          : Remove specific members from a replica-set
                            For example, a non-active member
--remove-failed-members   : Remove failed/not reachable members from a replica-set
                            On occasion, replica-set members are not reachable due to network isues
--remove-replica-set      : Remove a replica-set
--create-scripts          : Create init.d script for the replica-set members if force option is given
--setname                 : The name of a replica-set as configured in /etc/broadhop/mongoConfig.cfg
--force                   : This option can be used with create & add-members

Examples:
  General operation

    build_set.sh --all --create
    build_set.sh --session --create
    build_set.sh --session --create-asc
    build_set.sh --session --create-des
    build_set.sh --session --add-members
    build_set.sh --session --remove-members
    build_set.sh --session --remove-failed-members
    build_set.sh --session --remove-replica-set
    build_set.sh --session --create-scripts
    build_set.sh --help

  To perform build operations on a sepecific replica-set:

    build_set.sh --spr --create --setname set04
    build_set.sh --spr --create-asc --setname set04
    build_set.sh --spr --create-des --setname set04
    build_set.sh --spr --add-members --setname set04
    build_set.sh --spr --remove-failed-members --setname set04
    build_set.sh --spr --remove-replica-set --setname set04
    build_set.sh --spr --create-scripts --setname set04

If you want to use build_set.sh to create replica-set then use option --force.

Note

When you execute execute build_set.sh <databasename> –-remove_replica_set <setname>, it creates a /var/tmp/stopped-XXXX ( XXXX is the port number of the replica-set member) file on the respective sessionmgr and arbiter VM. If you want to recreate the same replica-set again on the same port then you have to manually remove the /var/tmp/stopped-XXXX file from respective sessionmgr or arbiter VM. AIDO monitors the /var/tmp/stopped-XXXX file on the VM and don’t do any action on the replica-set member if file is present.

You must create the database replica-set members on the same VM and the same port on both sites.

For example: For session manager database, among four replica-set members (except arbiter), if sessionmgr01:27717 and sessionmgr02:27717 are two members of replica-set from SITE1, then choose sessionmgr01:27717 and sessionmgr02:27717 of SITE2 as other two replica-set members as shown in following example:

[SESSION-SET]
         SETNAME=set01
         OPLOG_SIZE=5120
         ARBITER1=SITE-ARB-sessionmgr05:27717
         ARBITER_DATA_PATH=/var/data/sessions.1/set1
         PRIMARY-MEMBERS
         MEMBER1=SITE1-sessionmgr01:27717
         MEMBER2=SITE1-sessionmgr02:27717
         SECONDARY-MEMBERS
         MEMBER1=SITE2-sessionmgr01:27717
         MEMBER2=SITE2-sessionmgr02:27717
         DATA_PATH=/var/data/sessions.1/set1
         [SESSION-SET-END]

If it is required to configure more than seven members (including arbiters), then data members must be defined as non-voting-members in /etc/broadhop/mongoConfig.cfg file.

Non-voting members allow you to add additional data members for read distribution beyond the maximum seven voting members.

To configure a member as non-voting, votes and priority value must be configured to 0.

This configuration is done by build_set.sh and set_priority.sh scripts. So, it is expected to have priority as 0 for non-voting-member.

For more information, see https://docs.mongodb.com/manual/tutorial/configure-a-non-voting-replica-set-member/ (select appropriate mongo version).

Configure Non-Voting Members

If there are total eight data members and one arbiter (i.e. total nine members), six must be defined as MEMBERn and all other remaining data members must be defined as NON-VOTING-MEMBERn in /etc/broadhop/mongoConfig.cfg file.

where, n in MEMBERn and NON-VOTING-MEMBERn represents number 1, 2, 3 and so on.

[SPR-SET1]
SETNAME=set04
OPLOG_SIZE=3072
ARBITER=site3-arbiter:27720
ARBITER_DATA_PATH=/var/data/sessions.4
PRIMARY-MEMBERS
MEMBER1=site1-sessionmgr01:27720
MEMBER2=site1-sessionmgr02:27720
MEMBER3=site1-sessionmgr03:27720
NON-VOTING-MEMBER4=site1-sessionmgr04:27720
SECONDARY-MEMBERS
MEMBER1=site2-sessionmgr01:27720
MEMBER2=site2-sessionmgr02:27720
MEMBER3=site2-sessionmgr03:27720
NON-VOTING-MEMBER4=site2-sessionmgr04:27720
DATA_PATH=/var/data/sessions.4
[SPR-SET1-END]

Note	You can have only maximum seven voting members including arbiter which can be defined as MEMBERn and ARBITERn and all other member must be defined as NON-VOTING-MEMBERn.

The session cache can be scaled by adding an additional sessionmgr VM (additional session replica-set). You must create separate administration database and the hostname and port should be defined in Policy Builder (cluster) as defined in the following sections:

• Service Restart

• Create Session Shards

From Cluster Manager, run /var/qps/bin/diag/diagnostics.sh script.

Note	Currently, running diagnostics.sh --ha_proxy with qns-admin or qns-su user is not supported, It's only supported with root user.

From Cluster Manager, verify that you are able to ping all the hosts in the /etc/hosts file.

The following commands can be used to verify whether the installation was successful or not:

• diagnostics.sh
• about.sh
• list_installed_features.sh
• statusall.sh

Note	For more information on other CPS administrative commands, refer to CPS Operations Guide.

To verify that the CPS web interfaces are running navigate to the following URLs where <lbvip01> is the virtual IP address you defined for the lb01 VM.

Note	Run the about.sh command from the Cluster Manager to display the actual addresses as configured in your deployment.

• Policy Builder: https://<lbvip01>:7443/pb

  Default credentials: qns-svn/cisco123

• Control Center: https://<lbvip01>:443

  Default credentials: qns/cisco123

• Grafana: https://<lbvip01>:9443/grafana

  Default credentials: —

  Note	You must create at least one Grafana user to access the web interface. Refer to the Prometheus and Grafana chapter of the CPS Operations Guide for steps to configure User Authentication for Grafana.

• Unified API: http://<lbvip01>:8443/ua/soap

• CRD REST API: http://<lbvip01>:8443/custrefdata

For more information related to CPS interfaces, refer to CPS Operations Guide.