The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
In this chapter, Active/Standby Geographic Redundancy model has been used to describe the database and configuration changes required to modify the current installed HA system into Geo-HA.
If you want to deploy historical Active/Active model, just deploy additional flipped pair of this active/standby model.
An overview of active/standby model has been provided in this section.
Active/Standby solution has only one CPS cluster at each site, CA-PRI (referenced as ClusterA Primary henceforth) at S1 site (referenced as Geo-site-1/site-1 henceforth) and CA-SEC (referenced as ClusterA secondary henceforth) at S2 site (referenced as Geo-site-2/site-2 henceforth).
In the above figure, you have primary cluster (Geo Site 1/S1), secondary cluster (Geo Site 2/S2) and arbiter (Geo Site 3/S3).
For Site1 PCEF, there are two CPS clusters. One is primary, CA-PRI on S1 and other is secondary, CA-SEC on S2. They are geographically redundant.
Upon failure of primary CPS Cluster, secondary CPS cluster would seamlessly serve the subscriber's sessions. For that, session replication is enabled between primary and secondary clusters and for session replication high bandwidth is expected. For more information, contact your Cisco technical representative.
We recommend to use Replication interface for Database replication between Geo sites (that is, S1 and S2) to segregate Network traffic with Database replication traffic. For example, setting up separate VLAN's for segregating Network and Database traffic.
The secondary CPS cluster is not configured as passive.
We recommend to place the arbiter on site-3.
We recommend the SPR MongoDB and balance databases to be on SSD and session MongoDB to be on tmpfs for optimized performance.
Base install (CPS-HA) has been completed on both sites and verified basic validation on both sites.
Call model has been validated on both HA sites as per your TPS/traffic.
CPS VMs should have Replication IP address.
Familiarity with CPS Installation Guide for 9.0.0 and prior releases or CPS Installation Guide for VMware for 9.1.0 and later releases.
Familiarity with CPS Release Notes.
For third site, Arbiter must be deployed and running the same build (The ISO used to prepare the Geo- Redundant Setup).
The database configuration is planned.
![]() Note | This section is for reference only. You need to follow the nomenclature based on your network requirements. As a prerequisite, HA must be already deployed. |
For better usability of the system, install the HA system according to the following nomenclature:
In order to know the exact geo site details, we recommend to have the following entries in VMSpecification sheet of CPS_deployment_config_template.xlsm or VMSpecification.csv.
Host Name Prefix field value as Sx:
Cluster Name |
Recommended Value |
---|---|
CA-PRI |
S1- |
CA-SEC |
S2- |
In order to know the exact cluster name and role (primary/secondary) details, we recommend to have the following entries in Hosts sheet of CPS_deployment_config_template.xlsm or Hosts.csv:
We recommend to distribute session manager VMs equally between primary and secondary clusters, example:
sessionmgr01, sessionmgr02, sessionmgr03, sessionmgr04 on CA-PRI and
sessionmgr01, sessionmgr02, sessionmgr03, sessionmgr04 on CA-SEC
The following convention must be used while creating cross site replica-set for the session database:
You must create the session database replica-set members on same VM and same port on both sites. For example, among four replica-set members (except arbiter), if sessionmgr01:27717 and sessionmgr02:27717 are two members of replica-set from SITE1 then choose sessionmgr01:27717 and sessionmgr02:27717 of SITE2 as other two replica-set members as shown in following example:
[SESSION-SET] SETNAME=set01 OPLOG_SIZE=5120 ARBITER=SITE-ARB-sessionmgr05:27717 ARBITER_DATA_PATH=/var/data/sessions.1/set1 PRIMARY-MEMBERS MEMBER1=SITE1-sessionmgr01:27717 MEMBER2=SITE1-sessionmgr02:27717 SECONDARY-MEMBERS MEMBER1=SITE2-sessionmgr01:27717 MEMBER2=SITE2-sessionmgr02:27717 DATA_PATH=/var/data/sessions.1/set1 [SESSION-SET-END]
pcrfclient01 and pcrfclient02 of each site require Management/Public IP
Site1 HA Blade naming conventions of VMs looks like (This information is for reference only):
Blade |
Virtual Machines |
---|---|
CC Blade 1 |
S1-CA-PRI-cm S1-CA-PRI-lb01 S1-CA-PRI-pcrfclient01 |
CC Blade 2 |
S1-CA-PRI-lb02 S1-CA-PRI-pcrfclient02 |
CPS Blade 1 |
S1-CA-PRI-qns01 S1-CA-PRI-sessionmgr01 |
CPS Blade 2 |
S1-CA-PRI-qns02 S1-CA-PRI-sessionmgr02 |
CPS Blade 3 |
S1-CA-PRI-qns03 S1-CA-PRI-sessionmgr03 |
CPS Blade 4 |
S1-CA-PRI-qns04 S1-CA-PRI-sessionmgr04 |
Site2 HA configuration looks like (This information is for reference only):
Blade |
Virtual Machines |
---|---|
CC Blade 1 |
S1-CA-SEC-cm S1-CA-SEC-lb01 S1-CA-SEC-pcrfclient01 |
CC Blade 2 |
S1-CA-SEC-lb02 S1-CA-SEC-pcrfclient02 |
CPS Blade 1 |
S1-CA-SEC-qns01 S1-CA-SEC-sessionmgr01 |
CPS Blade 2 |
S1-CA-SEC-qns02 S1-CA-SEC-sessionmgr02 |
CPS Blade 3 |
S1-CA-SEC-qns03 S1-CA-SEC-sessionmgr03 |
CPS Blade 4 |
S1-CA-SEC-qns04 S1-CA-SEC-sessionmgr04 |
Do not install Arbiter if third site is not there or Arbiter is already installed on primary site.
Additionally, if third site blades are accessible from one of the GR sites, you can spawn the Arbiter VM from one of the sites, say, Site1, and installer will sit on third site blades. In that case also, this section is not applicable. Just have appropriate configurations done (Install Arbiter on Primary Site) so that destination VM is on a third site's blade.
The automatic GR site failover happens only when arbiters are placed on third site thus we recommend the MongoDB arbiter to be on third site that is, S3.
![]() Note | Arbiter VM name should be sessionmgrxx. |
Site3 HA configuration looks like (This information is for reference only):
Blade |
Virtual Machines |
vCPU |
Memory (GB) |
---|---|---|---|
CPS Blade 1 |
S3-ARB-cm S3-CA-ARB-sessionmgr01 |
1 4 |
8 8 |
For more information about deploying VMs, refer to:
![]() Note | Optional: Do not perform the following steps if Arbiter is installed on third site. |
If third site is not available then deploy arbiter VM on Primary Cluster that is, CA-PRI.
![]() Note | Arbiter VM name should be sessionmgrXX only. The “XX” should be replaced with a digit higher than the last used digit of the session manager. For example, if there are a total of six sessionmgrs (sessionmgr01-sessionmgr06) then, the Arbiter session manager must be sessionmgr07. |
To deploy arbiter on primary site, perform the following steps:
Step 1 | Configure system
parameters for deployment.
Add the following arbiter entry in Hosts sheet of deployment template sheet or Hosts.csv file. An example entry is shown below m: ![]() | ||
Step 2 | Import modified CSV files using the following command: /var/qps/install/current/scripts/import/import_deploy.sh | ||
Step 3 | Execute the following command to validate the imported data:
cd /var/qps/install/current/scripts/deployer/support/ python jvalidate.py
| ||
Step 4 | For each host
that is defined in the Hosts sheet of the excel document perform the manual
deployment (Refer to the
Manual
Deployment section in the
CPS Installation Guide for 9.0.0 and
prior releases or
CPS Installation Guide for VMware for 9.1.0 and
later releases).
An example is shown below: /var/qps/install/current/scripts/deployer ./deploy.sh sessionmgr07 |
To install Arbiter on VM, perform the following steps:
The following external ports need to be opened from firewall:
Step 1 | Convert the
Cluster Manager VM to an Arbiter (VMware).
| ||
Step 2 | Run
install.sh from ISO directory.
cd /mnt/iso ./install.sh Please enter install type [mobile|wifi|mog|arbiter]: arbiter ----> Select arbiter for this option Would you like to initialize the environment... [y|n]: y ----> Enter y to continue | ||
Step 3 | When prompted
for
Please pick
an option for this setup:. The following are the options for arbiter:
1) New Deployment 2) Upgrade from existing 9.0 system 3) In-Service Upgrade from 7.0.5 onwards (eg: 7.0.5 to 9.x) Select 1 for new Arbiter deployment. For further steps, refer to Example: New Deployment of Arbiter. | ||
Step 4 | To enable the
firewall, it is required to add the following configuration in
/etc/facter/facts.d/qps_firewall.txt file:
firewall_disabled=0 internal_address=XX.XX.XX.XX ---> update XX.XX.XX.XX to your internal IP address internal_device=0 ---> update 0 to your device ID internal_guest_nic=eth0 ---> update eth0 to other port if it is not using default NIC for internal address | ||
Step 5 | Configure
database host names in the
/etc/hosts file, which are defined in
mongoConfig.cfg file.
echo "XX.XX.XX.XX hostname1" >> /etc/hosts --> edit this line echo "XX.XX.XX.XX hostname2" >> /etc/hosts --> edit this line echo "XX.XX.XX.XX hostname3" >> /etc/hosts --> edit this line For example: /etc/broadhop/mongoConfig.cfg: [SESSION-SET1] SETNAME=set01 OPLOG_SIZE=5120 ARBITER=arbiter-site3:27717 ARBITER_DATA_PATH=/var/data/sessions.1/set01 PRIMARY-MEMBERS MEMBER1=sessionmgr01-site1:27717 MEMBER2=sessionmgr02-site1:27717 SECONDARY-MEMBERS MEMBER1=sessionmgr01-site2:27717 MEMBER2=sessionmgr02-site2:27717 DATA_PATH=/var/data/sessions.1/set01 [SESSION-SET1-END] For above mongoConfig.cfg file, below host entries are needed in /etc/hosts file: echo "192.168.1.1 arbiter-site3" >> /etc/hosts echo "192.168.1.2 sessionmgr01-site1" >> /etc/hosts echo "192.168.1.3 sessionmgr02-site1" >> /etc/hosts echo "192.168.1.4 sessionmgr01-site2" >> /etc/hosts echo "192.168.1.5 sessionmgr02-site2" >> /etc/hosts | ||
Step 6 | When install.sh finishes its run, execute the reinit.sh script to apply the appropriate configurations to the system: /var/qps/install/current/scripts/upgrade/reinit.sh | ||
Step 7 | After performing
the upgrade/new installation, unmount the ISO image. This prevents any “device
is busy” errors when a subsequent upgrade/new installation is performed.
cd /root umount /mnt/iso | ||
Step 8 | (Optional) After
unmounting the ISO, delete the ISO image to free the system space.
rm xxxx.iso where, xxxx.iso is the name of the ISO image used. | ||
Step 9 | (Optional) Change the host name of the Arbiter. |
Step 1 | Install arbiter
VM.
cd /mnt/iso ./install.sh Please enter install type [mobile|wifi|mog|arbiter]: arbiter -------------------------------- Install Configuration type: arbiter version: 8.9.9 dir: /var/qps/install/8.9.9 -------------------------------- Extracting Installation Scripts... Extracting Puppet Files... Extracting RPM Files... Bootstrapping yum repository... Copying qps software... Done copying file to: /var/qps/install/8.9.9 Would you like to initialize the environment... [y|n]: y Initializing vmware specific configuration ... Attempting to upgrade java with yum Attempting to upgrade httpd with yum Attempting to upgrade vim with yum Attempting to upgrade genisoimage with yum Attempting to upgrade svn with yum Attempting to upgrade mongo with yum Attempting to upgrade mongorestore with yum Attempting to upgrade nc with yum Attempting to upgrade socat with yum Attempting to upgrade bc with yum Attempting to install bash with yum Attempting to upgrade pystache with yum Attempting to upgrade fab with yum Attempting to upgrade sshpass with yum Attempting to upgrade openssl with yum Attempting to install nss with yum Attempting to upgrade telnet with yum Attempting to upgrade traceroute with yum Attempting to upgrade unzip with yum Attempting to upgrade createrepo with yum Attempting to upgrade python with yum Attempting to upgrade wget with yum iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Flushing firewall rules: [ OK ] iptables: Unloading modules: [ OK ] Starting httpd: [DONE] vmware specific configuration ... Done adding /etc/profile.d/broadhop.sh Please select the type of installation to complete: 1) New Deployment 2) Upgrade from existing 9.0 system 3) In-Service Upgrade from 7.0.5 onwards (eg: 7.0.5 to 9.x) 1 Deploying... Copying /etc/puppet to /var/qps/images/puppet.tar.gz... Creating MD5 Checksum... Updating tar from: /var/qps/env_config/ to /var/www/html/images/ Creating MD5 Checksum... Building /var/qps/bin... Copying /var/qps/bin to /var/qps/images/scripts_bin.tar.gz... Creating MD5 Checksum... Building local facter ... Creating facter file /etc/facter/facts.d/qps_facts.txt... |
Step 2 | Re-initiate the feature files by executing the following command: /var/qps/install/current/scripts/upgrade/reinit.sh |
Latest ISO Image
Latest base VMDK
Glance images
Cinder Volumes, only for ISO (SVN and mongo are not needed) are created
Access and Security (22 and mongo port 27717 to 27720 are opened as per deployment)
![]() Note | For more information on the above mentioned prerequisites, refer to CPS Dynamic Orchestration Guide for 9.0.0 and prior releases or CPS Installation Guide for OpenStack for 9.1.0 and later releases. |
Step 1 | Create flavors
by executing the following command:
nova flavor-create --ephemeral 0 arbiter auto 4096 0 2 | ||||||||||||||||||||||||||||||||||||||||||||||||||
Step 2 | Cloud init
configuration for Arbiter: When Arbiter is launched,
arbiter-cloud.cfg file needs to be passed via
user-data. In order to pass
arbiter-cloud.cfg file, it should be placed in the
directory where the user will execute
nova
boot command (likely the path will be
/root/cps-install directory).
Create arbiter-cloud.cfg file with the following content: #cloud-config write_files: - path: /etc/sysconfig/network-scripts/ifcfg-eth0 encoding: ascii content: | DEVICE=eth0 BOOTPROTO=none NM_CONTROLLED=none IPADDR=172.20.38.251 ---> update with your internal address NETMASK=255.255.255.0 ---> update with your netmask GATEWAY=172.20.38.1 ---> update with your gateway NETWORK=172.20.38.0 ---> update with your network owner: root:root permissions: '0644' - path: /var/lib/cloud/instance/payload/launch-params encoding: ascii owner: root:root permissions: '0644' - path: /root/.autoinstall.sh encoding: ascii content: | #!/bin/bash if [[ -d /mnt/iso ]] && [[ -f /mnt/iso/install.sh ]]; then /mnt/iso/install.sh << EOF arbiter y 1 EOF fi /root/.enable_firewall.sh /root/.add_db_hosts.sh if [[ -x "/var/qps/install/current/scripts/upgrade/reinit.sh" ]]; then /var/qps/install/current/scripts/upgrade/reinit.sh fi permissions: '0755' - path: /root/.enable_firewall.sh encoding: ascii content: | #!/bin/bash mkdir -p /etc/facter/facts.d/ cat <<EOF >/etc/facter/facts.d/qps_firewall.txt firewall_disabled=0 ---> change it to 1 if you do not want firewall enabled on this setup and remove below fields internal_address=172.20.38.251 ---> update with your internal address internal_device=0 EOF permissions: '0755' - path: /root/.add_db_hosts.sh ---> update db hosts IP as per requirement encoding: ascii content: | #!/bin/bash #Example if /etc/broadhop/mongoConfig.cfg: #[SESSION-SET1] #SETNAME=set01 #OPLOG_SIZE=5120 #ARBITER=arbiter-site3:27717 #ARBITER_DATA_PATH=/var/data/sessions.1/set01 #PRIMARY-MEMBERS #MEMBER1=sessionmgr01-site1:27717 #MEMBER2=sessionmgr02-site1:27717 #SECONDARY-MEMBERS #MEMBER1=sessionmgr01-site2:27717 #MEMBER2=sessionmgr02-site2:27717 #DATA_PATH=/var/data/sessions.1/set01 #[SESSION-SET1-END] #For above mongoConfig.cfg below hosts entries are needed in /etc/hosts, edit below list as per your requirement cat <<EOF >> /etc/hosts 192.168.1.1 arbiter-site3 192.168.1.2 sessionmgr01-site1 192.168.1.3 sessionmgr02-site1 192.168.1.4 sessionmgr01-site2 192.168.1.5 sessionmgr02-site2 EOF permissions: '0755' mounts: - [ /dev/vdb, /mnt/iso, iso9660, "auto,ro", 0, 0 ] runcmd: - ifdown eth0 - echo 172.20.38.251 installer arbiter >> /etc/hosts ---> update this IP - ifup eth0 - /root/.autoinstall.sh
| ||||||||||||||||||||||||||||||||||||||||||||||||||
Step 3 | Create Arbiter
VM:
Before executing nova boot command, confirm that the cloud configuration file (arbiter-cloud.cfg) exists in the right directory. Execute the following command to create arbiter VM with two NICs: source ~/keystonerc_core nova boot --config-drive true --user-data=arbiter-cloud.cfg --file /root/keystonerc_user=/root/keystonerc_core --image "base_vm" --flavor "arbiter" --nic net-id="9c89df81-90bf-45bc-a663-e8f80a8c4543,v4-fixed-ip=172.16.2.19" --nic net-id="dd65a7ee-24c8-47ff-8860-13e66c0c966e,v4-fixed-ip=172.18.11.101" --block-device-mapping "/dev/vdb=eee05c17-af22-4a33-a6d9-cfa994fecbb3:::0" --availability-zone "az-2:os24-compute-2.cisco.com" arbiter For example, nova boot --config-drive true --user-data=arbiter-cloud.cfg --file /root/keystonerc_user=/root/keystonerc_core --image "base_vm" --flavor "arbiter" --nic net-id="<Internal n/w id>,v4-fixed-ip=<Interanl n/w private ip>" --nic net-id="<Management n/w id>,v4-fixed-ip=<Management n/w public ip>" --block-device-mapping "/dev/vdb=<Volume id of iso>:::0" --availability-zone "<availability zone:Host info>" arbiter The following examples can be used to get the internal and management IP addresses and volume IDs which are used to spin arbiter VM. source ~/keystonerc_core neutron net-list
nova volume-list
|
![]() Note | The following steps need to be performed on other sites as well. |
![]() Note | In this section, to configure remote/peer site VM in local site, sessionmgr has been taken as an example. You can use this section to add peer policy server (qns) and peer policy directors (lbs) entries in AdditionalHosts file. |
Step 1 | Add the
following entry in
AdditionalHosts sheet of CPS deployment template or
AdditionalHosts.csv on CA-PRI-cm: Objective of this
section is for primary cluster to add other cluster (that is, secondary
cluster) session manager's details.
Example:Example of /var/qps/config/deploy/csv/AdditionalHosts.csv (on CA-PRI-cm): Host,Alias,IP Address ----- CA-SEC-sessionmgr01,psessionmgr01,xx.xx.xx.xx CA-SEC-sessionmgr02,psessionmgr02, xx.xx.xx.xx CA-ARB-sessionmgr01,CA-ARB-sessionmgr01,xx.xx.xx.xx ----- | ||
Step 2 | Import modified
CSV files by executing the following command:
/var/qps/install/current/scripts/import/import_deploy.sh | ||
Step 3 | Execute the
following command to validate the imported data:
cd /var/qps/install/current/scripts/deployer/support/ python jvalidate.py
| ||
Step 4 | Execute the
following command in Cluster Manager to copy updated
/etc/hosts file to all deployed VMs:
SSHUSER_PREFERROOT=true copytoall.sh /etc/hosts /etc/hosts | ||
Step 5 | Validate setup using diagnostics.sh script. |
Step 1 | Log in to pcrfclient01 as root user. |
Step 2 | Modify the
/etc/broadhop/gr_cluster.conf file.
These entries need to match with site name entries without suffix _PRI/_SBY given in qns.conf file. File contents will look like:cat /etc/broadhop/gr_cluster.conf #<site name>:<pcrfclient01 IP address>:<pcrfclient02 IP address> #Primary sites clusterA:xx.xx.xx.xx:xx.xx.xx.xx #Secondary sites clusterA:xx.xx.xx.xx:xx.xx.xx.xx |
Step 3 |
Verify MongoConfig: Do not miss to add
#SITEx_START and
#SITEx_END tags to the block of replica set
entries in
/etc/broadhop/mongoConfig.cfg file, where
x is the site number. To add these tags at proper
location, refer to sample configuration file (geo_mongoconfig_template) present in
/etc/broadhop directory.
|
Step 4 | After configuring mongoConfig.cfg file, install databases using build_set.sh script. |
Step 5 | From
pcrfclient01, set priority 2 for primary site replica-set members, which are
replicated across site(s) using the following mentioned example commands:
Example: cd /var/qps/bin/support/mongo/; ./set_priority.sh --db session cd /var/qps/bin/support/mongo/; ./set_priority.sh --db spr cd /var/qps/bin/support/mongo/; ./set_priority.sh --db admin cd /var/qps/bin/support/mongo/; ./set_priority.sh --db balance |
Step 6 | Verify replica
set status and priority is set correctly using the following command from
pcrfclient01:
diagnostics.sh --get_replica_status |
Step 7 | When the Session
replication is configured then the Host collection of the Cluster database
should have all the “admin replica-set” members, entries with Internal and
Replication VLAN IP's.
By default, db.hosts file gets populated if we configure /etc/broadhop/gr_cluster.conf file. If the entries are not present then use the following commands to add these entries (XX is internal/replication IP address of “admin replica-set” and YY is siteId (defined in qns.conf): mongo --host <admin DB primary host> --port <admin DB port> clusters > db.hosts.update({"ip" : "XX"},{"siteName" : "YY", "ip" : "XX"}, true) Example: [root@CA-PRI-pcrfclient01 mongo]# mongo CA-PRI-sessionmgr02:27721/clusters MongoDB shell version: 2.6.3 connecting to: CA-PRI-sessionmgr02:27721/clusters set05:PRIMARY> db.hosts.find() { "_id" : ObjectId("545e0596f4ce7b3cc119027d"), "siteName" : "clusterA_PRI", "ip" : "192.168.109.127" } { "_id" : ObjectId("545e0596f4ce7b3cc119027e"), "siteName" : "clusterA_PRI", "ip" : "192.168.109.128" } { "_id" : ObjectId("545e0596f4ce7b3cc1190281"), "siteName" : "clusterA_SBY", "ip" : "192.168.109.227" } { "_id" : ObjectId("545e0596f4ce7b3cc1190282"), "siteName" : "clusterA_SBY", "ip" : "192.168.109.228" } { "_id" : ObjectId("545e0596f4ce7b3cc119027d"), "siteName" : "clusterA_PRI", "ip" : "11.11.11.127" } { "_id" : ObjectId("545e0596f4ce7b3cc119027e"), "siteName" : "clusterA_PRI", "ip" : "11.11.11.128" } { "_id" : ObjectId("545e0596f4ce7b3cc1190281"), "siteName" : "clusterA_SBY", "ip" : "11.11.11.227" } { "_id" : ObjectId("545e0596f4ce7b3cc1190282"), "siteName" : "clusterA_SBY", "ip" : "11.11.11.228" }
Example: For sessionmgr01 SITE-A, in /etc/hosts file, if the entry is: 10.10.10.1 sessionmgr01 sessionmgr01-SITE-A and for SITE-B on sessionmgr01, in /etc/hosts, if the entry is: 172.20.20.1 sessionmgr01-SITE-A As, IP addresses of sessionmgr VMs are different in this case, user needs to run the following scripts on both SITEs. cd /var/qps/bin/support/mongo/; ./set_clusterinfo_in_admindb.sh |
Step 8 | From primary pcrfclient01 copy mongoConfig.cfg and gr_cluster.conf files to both primary and secondary Cluster Managers (CM). |
Step 9 | From both CMs,
execute the following commands, where first one will build “etc” and next two
will copy these files to all other deployed VMs:
/var/qps/install/current/scripts/build/build_etc.sh SSHUSER_PREFERROOT=true copytoall.sh /etc/broadhop/gr_cluster.conf /etc/broadhop/gr_cluster.conf SSHUSER_PREFERROOT=true copytoall.sh /etc/broadhop/mongoConfig.cfg /etc/broadhop/mongoConfig.cfg |
CPS runs a distributed database called MongoDB. MongoDB uses a replication concept for high availability called replica-sets. A replica-set is made up of independent MongoDB instances that run in one of the following three modes:
Primary: A primary database is the only database available that can accept writes.
Secondary: A secondary database is a database that is read only and is actively synchronizing to a primary database by replaying the primary's oplog (operations log) on the local node.
Recovering: A secondary database that is currently synchronizing to the primary and has not caught up to the primary.
Session data is highly concurrent, the application always reads and writes from the primary database. The secondary database(s) provide HA for the primary in the event of VM shutdown or process shutdown. Hot standby session cache replica set is configured to take over the load while primary database is failing over to secondary session cache database. In this fail-over process, it minimizes the call failures and provides high system availability.
Step 1 | Verify CPS
application is running on both the sites (pcrfclient01 and pcrfclient02) and
without any application errors.
Example: By executing diagnostics.sh script you can get the diagnostics of application. The diagnostics.sh output should not contain any application errors. |
Step 2 | Verify whether
shard command is available in OSGi console or not. From pcrfclient01, login as
root user into the OSGi console and run the help.
You will
find the following shard command:
telnet qns01 9091 osgi> help ---QNS Commands--- reload - Reload reference data genpassword <db password> ---Sharing Commands--- addshard seed1[,seed2] port db-index [backup] rebalance migrate ---Controlling the Console--- more - More prompt for console output disconnect - Disconnects from telnet session help <commmand> - Display help for the specified command. |
Step 3 | To configure hot
standby session management, execute the following commands:
telnet qns01 9091 addshard sessionmgr03,sessionmgr04 27717 1 backup addshard sessionmgr03,sessionmgr04 27717 2 backup addshard sessionmgr03,sessionmgr04 27717 3 backup addshard sessionmgr03,sessionmgr04 27717 4 backup rebalance migrate disconnect y |
Step 4 | To verify the
configuration:
|
There are three possible ways a MongoDB node can fail and trigger a fail over to another node:
Replica set step down: This scenario is the cleanest method since it disconnects all client sockets and immediately initiates a new election for a master node.
Process abort: This scenario occurs when a node aborts due to an internal failure. Since this is unplanned, the other replica nodes will not request a new election for a master node until a majority of the nodes have detected the master as down.
VM power off: This scenario occurs when a node is powered off without a proper shutdown. In this case sockets are usually hung on all clients and the other replica nodes will not request a new election for a master node until a majority of the nodes have detected the master as down.
The Cisco Policy Server detects client failure by:
We can configure only one backup database. Thus, in GR Active/Standby configuration, if we configure backup database on the standby site, during local primary to local secondary database fail over on active site, the sessions would be saved on the backup database which is on secondary site. This might increase cross-site traffic temporarily.
Step 1 | Configure and
publish Policy Builder changes from each site. Use
about.sh command to find out Policy Builder URL.
Cisco recommends to configure and publish Policy Builder data separately from each site. But if the user wants to publish Policy Builder data from single site to all other sites then it is difficult to access Policy Builder data from other sites when the primary site goes down. To access Policy Builder data from other sites when primary site is down, refer Access Policy Builder from Standby Site when Primary Site is Down. | ||||||
Step 2 | Set appropriate Primary Database IP address, Secondary Database IP address and Port numbers for the following plug-ins: | ||||||
Step 3 | Set
SecondaryPreferred for all databases except balance
database.
as
![]() An example Cluster configuration is given: ![]() | ||||||
Step 4 | It is
recommended to publish Policy Builder changes from each site. If a user is
using primary site to publish Policy Builder changes then publishing into all
the following cluster repositories is not recommended:
| ||||||
Step 5 | Add all the
above repositories. Repository and Publish screen looks like:
![]() ![]() | ||||||
Step 6 | Validate both setups using diagnostics.sh, after publishing the repository (wait for five minutes) OR follow the Validate VM Deployment section in the Cisco Policy Suite Installation Guide for this release. |
The following changes in qns.conf are to be done when session replication is required.
Step 1 | Add the
following GR related parameters in
/etc/broadhop/qns.conf file of Cluster A Primary
cluster manager VM that is, CA-PRI-cm:
-DGeoSiteName=clusterA_PRI -DSiteId=clusterA_PRI -DRemoteSiteId=clusterA_SBY -DheartBeatMonitorThreadSleepMS=500 -Dcom.mongodb.updaterConnectTimeoutMS=1000 -Dcom.mongodb.updaterSocketTimeoutMS=1000 -DdbConnectTimeout=1200 -Dmongo.client.thread.maxWaitTime=1200 -DdbSocketTimeout=600 -DclusterFailureDetectionMS=2000 |
Step 2 | Add the
following GR related parameters in
/etc/broadhop/qns.conf file of Cluster A Secondary
cluster manager VM that is, CA-SEC-cm:
-DGeoSiteName=clusterA_SBY -DSiteId=clusterA_SBY -DRemoteSiteId=clusterA_PRI -DheartBeatMonitorThreadSleepMS=500 -Dcom.mongodb.updaterConnectTimeoutMS=1000 -Dcom.mongodb.updaterSocketTimeoutMS=1000 -DdbConnectTimeout=1200 -Dmongo.client.thread.maxWaitTime=1200 -DdbSocketTimeout=600 -DclusterFailureDetectionMS=2000 |
Step 3 | Add the
following line in
qns.conf file present under
/etc/broadhop/diameter_endpoint directory.
-DGeoSiteName=<site
id>
Example:For example,-DGeoSiteName=ClusterA_PRI |
Step 4 | Create etc directory on each cluster using /var/qps/install/current/scripts/build/build_etc.sh script. |
Step 5 | Copy the changes in qns.conf to other VMs: copytoall.sh /etc/broadhop/qns.conf /etc/broadhop/qns.conf |
Step 6 | Restart all software components on the target VMs: restartall.sh |
Step 7 | Validate setup using diagnostics.sh OR follow the Validate VM Deployment section in the Cisco Policy Suite Installation Guide for this release. |
The following setting should be present only for GR (multi-cluster) CPS deployments:
-DclusterFailureDetectionMS=1000
![]() Note | In an HA or GR deployment with local chassis redundancy, the following setting should be set to true. By default, this is set to false. |
-Dremote.locking.off
Step 1 | This is recommended only when primary site is down and secondary is only for reading/viewing purpose. It is applicable only where user publishes policy builder data from single site that is, primary site to all other sites. |
Step 2 | Open Policy Builder from secondary site (use about.sh command to find out PB URL). |
Step 3 | Create new data
repository SEC-RUN-RO using URL ‘http://< Management interface public IP
address of secondary pcrfclient01>/repos/run’, screen looks like:
![]() |
Step 4 | Access Policy Builder from secondary site using newly created repository. |
The following configurations are needed to enable qns_hb.sh script. This script stops Policy Server (QNS) processes from lb01/lb02 when all Policy Server (QNS) are down (that is, qns01,02..n).
![]() Note | To understand traffic switchover, refer to Load Balancer VIP Outage. |
Step 1 | To enable
script, add the following configuration in
/var/qps/config/deploy/csv/Configuration.csv file:
mon_qns_lb,true,
For more information on how to add the parameter in Configuration.csv file, refer to CPS Installation Guide for 9.0.0 and prior releases or CPS Installation Guide for VMware for 9.1.0 and later releases.
| ||
Step 2 | To disable script, add the following configuration in /var/qps/config/deploy/csv/Configuration.csv file or remove mon_qns_lb tag from this CSV file: mon_qns_lb,, | ||
Step 3 | Import CSV to JSON by executing the following command: /var/qps/install/current/scripts/import/import_deploy.sh | ||
Step 4 | Execute the
following command to validate the imported data:
cd /var/qps/install/current/scripts/deployer/support/ python jvalidate.py
| ||
Step 5 | Reinitiate lb01
and lb02 by executing the following command:
/etc/init.d/vm-init
For more information on configuring GR features, refer to CPS Mobile Configuration Guide. |
![]() Note | To understand traffic switchover, refer to Load Balancer VIP Outage. |
Step 1 | To enable
switchover on primary GR site only, add the following entry in the
/var/qps/config/deploy/csv/Configuration.csv file in
Cluster Manager:
mon_db_for_callmodel,true,
For more information on how to add the parameter in Configuration.csv file, refer to CPS Installation Guide for 9.0.0 or prior releases and CPS Installation Guide for VMware for 9.1.0 and later releases. | ||||
Step 2 | Add the list of
databases that needs to be monitored in
mon_db_for_callmodel.conf file (/etc/broadhop/mon_db_for_callmodel.conf) in Cluster
Manager.
Add the following content in the configuration file (mon_db_for_callmodel.conf):
#this file contains set names that are available in mongoConfig.cfg. Add set names one below other. #Refer to README in the scripts folder. SESSION-SET1 SESSION-SET2 BALANCE-SET1 SPR-SET1 | ||||
Step 3 | To enable
switch-off of UAPI traffic when replicated (inter-site) configured databases
are not primary on this site, add
STOP_UAPI=1 in
/etc/broadhop/mon_db_for_callmodel.conf file.
When we recover GR site, we have to manually start UAPI interface (if it is disabled) by executing the following command as a root user on lb01 and lb02: echo "enable frontend https-api" | socat stdio /tmp/haproxy | ||||
Step 4 | Rebuild etc directory on cluster by executing the following command: /var/qps/install/current/scripts/build/build_etc.sh | ||||
Step 5 | Import CSV to JSON by executing the following command: /var/qps/install/current/scripts/import/import_deploy.sh | ||||
Step 6 | Execute the following command to validate the imported data:
cd /var/qps/install/current/scripts/deployer/support/ python jvalidate.py
| ||||
Step 7 | Execute the following command on pcrfclient01/02 to re-initialize the VM nodes so that the changes done in Cluster Manager are reflected in pcrfclient01/02: /etc/init.d/vm-init |
![]() Note | To understand traffic switch over, refer to Load Balancer VIP Outage. |
Step 1 | To enable
switch over on Primary GR site only, add the following entry in the
/var/qps/config/deploy/csv/Configuration.csv file in
Cluster Manager:
mon_db_for_lb_failover,true,
For more information on how to add the parameter in Configuration.csv file, refer to CPS Installation Guide for 9.0.0 and prior releases or CPS Installation Guide for VMware for 9.1.0 and later releases. | ||||
Step 2 | Add the list of
databases that needs to be migrated to primary on other site after traffic
switch over in
mon_db_for_lb_failover.conf file (/etc/broadhop/mon_db_for_lb_failover.conf) in Cluster
Manager.
Add the following content in the configuration file (mon_db_for_lb_failover.conf):
#this file contains set names that are available in mongoConfig.cfg. Add set names one below other. #Refer to README in the scripts folder. SESSION-SET1 SESSION-SET2 BALANCE-SET1 SPR-SET1 | ||||
Step 3 | Rebuild etc directory on cluster by executing the following command: /var/qps/install/current/scripts/build/build_etc.sh | ||||
Step 4 | Import CSV to JSON by executing the following command: /var/qps/install/current/scripts/import/import_deploy.sh | ||||
Step 5 | Execute the following command to validate the imported data:
cd /var/qps/install/current/scripts/deployer/support/ python jvalidate.py
| ||||
Step 6 | Execute the
following command on pcrfclient01/02 to reinitialize the VM nodes so that the
changes done in Cluster Manager are reflected in pcrfclient01/02:
/etc/init.d/vm-init
Before running stopall.sh or restartall.sh, execute the following commands: /var/qps/bin/control/runonone.sh pcrfclient01 'sudo /usr/bin/monit stop mon_db_for_lb_failover' /var/qps/bin/control/runonone.sh pcrfclient02 'sudo /usr/bin/monit stop mon_db_for_lb_failover' Verify monit stopped monitoring mon_db_for_lb_failover script by executing the following command: /var/qps/bin/control/runonone.sh pcrfclient01 'sudo /usr/bin/monit summary | grep mon_db_for_lb_failover' Program 'mon_db_for_lb_failover' Not monitored /var/qps/bin/control/runonone.sh pcrfclient02 'sudo /usr/bin/monit summary | grep mon_db_for_lb_failover' Program 'mon_db_for_lb_failover' Not monitored After running startall.sh or restartall.sh, execute the following commands: /var/qps/bin/control/runonone.sh pcrfclient01 'sudo /usr/bin/monit start mon_db_for_lb_failover' /var/qps/bin/control/runonone.sh pcrfclient02 'sudo /usr/bin/monit start mon_db_for_lb_failover' Verify monit started monitoring mon_db_for_lb_failover script by executing the following commands: /var/qps/bin/control/runonone.sh pcrfclient01 'sudo /usr/bin/monit summary | grep mon_db_for_lb_failover' Program 'mon_db_for_lb_failover' Status ok (some time it shows status failed also, we can ignore it) /var/qps/bin/control/runonone.sh pcrfclient02 'sudo /usr/bin/monit summary | grep mon_db_for_lb_failover' Program 'mon_db_for_lb_failover' Status ok (some time it shows status failed also, we can ignore it) |