The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
In this chapter, Active/Standby Geographic Redundancy model has been used to describe the database and configuration changes required to modify the current installed HA system into Geo-HA.
If you want to deploy historical Active/Active model, just deploy additional flipped pair of this active/standby model.
An overview of active/standby model has been provided in this section.
Active/Standby solution has only one CPS cluster at each site, CA-PRI (referenced as ClusterA Primary henceforth) at S1 site (referenced as Geo-site-1/site-1 henceforth) and CA-SEC (referenced as ClusterA secondary henceforth) at S2 site (referenced as Geo-site-2/site-2 henceforth).
In the above figure, you have primary cluster (Geo Site 1/S1), secondary cluster (Geo Site 2/S2) and arbiter (Geo Site 3/S3).
Geo site 1/S1 could be any site (for example, Mumbai)
Geo site 2/S2 could be any site (for example, Chennai)
Geo site 3/S3 could be any site (for example, Kolkata)
For Site1 PCEF, there are two CPS clusters. One is primary, CA-PRI on S1 and other is secondary, CA-SEC on S2. They are geographically redundant.
Upon failure of primary CPS Cluster, secondary CPS cluster would seamlessly serve the subscriber's sessions. For that, session replication is enabled between primary and secondary clusters and for session replication high bandwidth is expected. For more information, contact your Cisco technical representative.
We recommend to use Replication interface for Database replication between Geo sites (that is, S1 and S2) to segregate Network traffic with Database replication traffic. For example, setting up separate VLAN's for segregating Network and Database traffic.
The secondary CPS cluster is not configured as passive.
We recommend to place the arbiter on site-3.
We recommend the SPR and balance databases to be on SSD and session database to be on tmpfs for optimized performance.
Base install (CPS-HA) has been completed on both sites and verified basic validation on both sites.
Call model has been validated on both HA sites as per your TPS/traffic.
CPS VMs should have Replication IP address.
Familiarity with CPS Installation Guide for VMware.
Familiarity with CPS Release Notes.
For third site, Arbiter must be deployed and running the same build (The ISO used to prepare the Geo- Redundant Setup).
The database configuration is planned.
 Note |
This section is for reference only. You need to follow the nomenclature based on your network requirements. As a prerequisite, HA must be already deployed. |
For better usability of the system, install the HA system according to the following nomenclature:
In order to know the exact geo site details, we recommend to have the following entries in VMSpecification sheet of CPS_deployment_config_template.xlsm or VMSpecification.csv.
Host Name Prefix field value as Sx:
|
Cluster Name |
Recommended Value |
|---|---|
|
CA-PRI |
S1- |
|
CA-SEC |
S2- |
In order to know the exact cluster name and role (primary/secondary) details, we recommend to have the following entries in Hosts sheet of CPS_deployment_config_template.xlsm or Hosts.csv:
Guest Name field value as:
CA-PRI-XXX for primary cluster (like CA-PRI-lb01, CA-PRI-qns01, and so on.) and CA-SEC-XXX for secondary cluster (like CA-SEC-qsn01, CA-SEC-lb01, and so on.)
We recommend to distribute session manager VMs equally between primary and secondary clusters, example:
sessionmgr01, sessionmgr02, sessionmgr03, sessionmgr04 on CA-PRI and
sessionmgr01, sessionmgr02, sessionmgr03, sessionmgr04 on CA-SEC
The following convention must be used while creating cross site replica-set for the session database:
You must create the
session database replica-set members on same VM and same port on both sites.
For example, among four replica-set members (except arbiter), if
sessionmgr01:27717 and
sessionmgr02:27717 are two members of replica-set from
SITE1 then choose
sessionmgr01:27717 and
sessionmgr02:27717 of SITE2 as other two replica-set
members as shown in following example:
[SESSION-SET]
SETNAME=set01
OPLOG_SIZE=5120
ARBITER1=SITE-ARB-sessionmgr05:27717
ARBITER_DATA_PATH=/var/data/sessions.1/set1
PRIMARY-MEMBERS
MEMBER1=SITE1-sessionmgr01:27717
MEMBER2=SITE1-sessionmgr02:27717
SECONDARY-MEMBERS
MEMBER1=SITE2-sessionmgr01:27717
MEMBER2=SITE2-sessionmgr02:27717
DATA_PATH=/var/data/sessions.1/set1
[SESSION-SET-END]
pcrfclient01 and pcrfclient02 of each site require Management/Public IP
Site1 HA Blade naming conventions of VMs looks like (This information is for reference only):
|
Blade |
Virtual Machines |
|---|---|
|
CC Blade 1 |
S1-CA-PRI-cm S1-CA-PRI-lb01 S1-CA-PRI-pcrfclient01 |
|
CC Blade 2 |
S1-CA-PRI-lb02 S1-CA-PRI-pcrfclient02 |
|
CPS Blade 1 |
S1-CA-PRI-qns01 S1-CA-PRI-sessionmgr01 |
|
CPS Blade 2 |
S1-CA-PRI-qns02 S1-CA-PRI-sessionmgr02 |
|
CPS Blade 3 |
S1-CA-PRI-qns03 S1-CA-PRI-sessionmgr03 |
|
CPS Blade 4 |
S1-CA-PRI-qns04 S1-CA-PRI-sessionmgr04 |
Site2 HA configuration looks like (This information is for reference only):
|
Blade |
Virtual Machines |
|---|---|
|
CC Blade 1 |
S1-CA-SEC-cm S1-CA-SEC-lb01 S1-CA-SEC-pcrfclient01 |
|
CC Blade 2 |
S1-CA-SEC-lb02 S1-CA-SEC-pcrfclient02 |
|
CPS Blade 1 |
S1-CA-SEC-qns01 S1-CA-SEC-sessionmgr01 |
|
CPS Blade 2 |
S1-CA-SEC-qns02 S1-CA-SEC-sessionmgr02 |
|
CPS Blade 3 |
S1-CA-SEC-qns03 S1-CA-SEC-sessionmgr03 |
|
CPS Blade 4 |
S1-CA-SEC-qns04 S1-CA-SEC-sessionmgr04 |
Note |
If you want to add the MongoDB authentication on Arbiter, refer to General Configuration section in CPS Installation Guide for VMware. You need to mention password for all the sites separately using CSV file and that should be same for all the sites. |
Important |
Currently, SNMP and statistics are not supported on third site arbiter. |
Do not install Arbiter if third site is not there or Arbiter is already installed on primary site.
Additionally, if third site blades are accessible from one of the GR sites, you can spawn the Arbiter VM from one of the sites, say, Site1, and installer will sit on third site blades. In that case also, this section is not applicable. Just have appropriate configurations done (On Primary Site) so that destination VM is on a third site's blade.
The automatic GR site failover happens only when arbiters are placed on third site thus we recommend the MongoDB arbiter to be on third site that is, S3.
Note |
Arbiter VM name should be sessionmgrxx. |
Site3 HA configuration looks like (This information is for reference only):
|
Blade |
Virtual Machines |
vCPU |
Memory (GB) |
|---|---|---|---|
|
CPS Blade 1 |
S3-ARB-cm S3-CA-ARB-sessionmgr01 |
1 4 |
8 8 |
For more information about deploying VMs, refer to CPS Installation Guide for VMware.
| Step 1 |
Configure system parameters for deployment for new Arbiter VM. We need the following CSV files to deploy and configure arbiter VM: They are:
|
||||||||||||||||||||||||||||||||||||||
| Step 2 |
Convert the
Excel spreadsheet into a CSV file and upload the file to the Cluster Manager VM
in
|
||||||||||||||||||||||||||||||||||||||
| Step 3 |
For each host that is defined in the Hosts sheet of the deployment spreadsheet, perform the manual deployment (Refer to the Manual Deployment section in the CPS Installation Guide for VMware). Example: |
||||||||||||||||||||||||||||||||||||||
| Step 4 |
If you want to enable mongo authentication on Arbiter, create the file /etc/facter/facts.d/mongo_auth.txt using the following data. The password should be same with all other sites. where, XXXXX and YYYYY are encrypted passwords. |
Note |
Optional: Do not perform the following steps if Arbiter is installed on third site. |
If third site is not available then deploy arbiter VM on Primary Cluster that is, CA-PRI.
Note |
Arbiter VM name should be sessionmgrXX only. XX should be replaced with a digit higher than the last used digit of the session manager. For example, if there are a total of six sessionmgrs (sessionmgr01-sessionmgr06) then, the Arbiter session manager must be sessionmgr07. |
To deploy arbiter on primary site, perform the following steps:
| Step 1 |
Configure system parameters for deployment. Add the following arbiter entry in Hosts sheet of deployment template sheet or Hosts.csv file. An example entry is shown below m:  |
||
| Step 2 |
Import modified CSV files using the following command: /var/qps/install/current/scripts/import/import_deploy.sh
|
||
| Step 3 |
Execute the following command to validate the imported data:
|
||
| Step 4 |
For each host that is defined in the Hosts sheet of the excel document perform the manual deployment (Refer to the Manual Deployment section in the CPS Installation Guide for VMware). An example is shown below:
|
Note |
If you want to add the MongoDB authentication on Arbiter, refer to General Configuration section in CPS Installation Guide for VMware. You need to mention password for all the sites separately using CSV file and that must be same for all the sites. |
To install Arbiter on VM, perform the following steps:
| Step 1 |
Convert the Cluster Manager VM to an Arbiter (VMware).
|
||||
| Step 2 |
Run
|
||||
| Step 3 |
When prompted
for
Select 1 for new Arbiter deployment. |
||||
| Step 4 |
To enable the firewall, it is required to add the following configuration in /etc/facter/facts.d/qps_firewall.txt file: |
||||
| Step 5 |
When
/var/qps/install/current/scripts/upgrade/reinit.sh
|
||||
| Step 6 |
If you want to enable mongo authentication on Arbiter, create the file /etc/facter/facts.d/mongo_auth.txt using the following data. The password should be same with all other sites. where, XXXXX and YYYYY are encrypted passwords. For encrypted passwords, you need to SSH to a Cluster Manager and execute the following command:
|
||||
| Step 7 |
Edit /etc/hosts/ and add the information related to all the replica members entries as per your requirement (replica members in mongoConfig.cfg file). Example: |
||||
| Step 8 |
After performing the upgrade/new installation, unmount the ISO image. This prevents any “device is busy” errors when a subsequent upgrade/new installation is performed.
|
||||
| Step 9 |
(Optional) After unmounting the ISO, delete the ISO image to free the system space.
where, xxxx.iso is the name of the ISO image used. |
||||
| Step 10 |
(Optional) Change the host name of the Arbiter.
|
| Step 1 |
Update the arbiter member information in /etc/broadhop/mongoConfig.cfg file. Example:
|
||
| Step 2 |
Rebuild etc directory on cluster with the updated mongoConfig.cfg file.
|
||
| Step 3 |
Copy
mongoConfig.cfg file to all the nodes using
|
Note |
The following steps need to be performed on other sites as well. |
Note |
In this section, to configure remote/peer site VM in local site, sessionmgr has been taken as an example. You can use this section to add peer policy server (qns) and peer policy directors (lbs) entries in AdditionalHosts file. |
| Step 1 |
Add the following entry in AdditionalHosts sheet of CPS deployment template or AdditionalHosts.csv on CA-PRI-cm: Objective of this section is for primary cluster to add other cluster (that is, secondary cluster) session manager's details. |
||
| Step 2 |
Import modified CSV files by executing the following command:
|
||
| Step 3 |
Execute the following command to validate the imported data:
|
||
| Step 4 |
Execute the following command in Cluster Manager to copy updated /etc/hosts file to all deployed VMs:
|
||
| Step 5 |
Validate setup
using
|
Redis must be enabled as IPC. For more information on how to enable REDIS, refer to CPS Installation Guide for VMware.
| Step 1 |
Add the following entry in AdditionalHosts sheet of CPS deployment template or AdditionalHosts.csv on CA-Site1-cm: Objective of this section is for local site to add other site (that is, remote clusters) policy director (lb) VM details. |
| Step 2 |
Add the number of remote redis instances in Configuration.csv with key as remote_redis_server_count and value as the number of redis instances running on remote site: Example:
For more information in remote_redis_server_count, refer to CPS Installation Guide for VMware. |
| Step 3 |
Import modified CSV files by executing the following command:
|
| Step 4 |
Execute the following commands in Cluster Manager to copy updated /etc/hosts file to all deployed VMs:
|
| Step 5 |
Verify that the /etc/broadhop/redisTopology.ini contains the remote policy director (lb) redis instances entry as follows:
|
If the number of redis instances/lb instances are to be increased/decreased on the remote cluster(s), the same should first be updated in all other clusters in the CSV files as mentioned in Step 1 and Step 2.
Repeat the steps from Step 3 to Step 5 after changing the CSV files so as to update the redisTopology file on all the VMs.
Note |
While configuring mongo ports in a GR environment, there should be a difference of 100 ports between two respective sites. For example, consider there are two sites: Site1 and Site2. For Site1, if the port number used is 27717, then you can configure 27817 as the port number for Site2. This is helpful to identify a mongo member’s site. By looking at first three digits, one can decide where the mongo member belongs to. However, this is just a guideline. You should avoid having mongo ports of two different sites close to each other (for example, 27717 on Site-1 and 27718 on Site2). Reason: The reason is
that the
|
| Step 1 |
Log in to Cluster Manager as a root user. |
||
| Step 2 |
Modify the /etc/broadhop/gr_cluster.conf file. For example, if /etc/broadhop/qns.conf file has the following entries for Site ID:.
These entries need to match with site name entries given in qns.conf file. |
||
| Step 3 |
Verify
MongoConfig: Do not miss to add
Example:Example:-DSiteID. For marking/separating the two sites
replica-sets, identify the start and end of the replica-sets.
For example, one site is having SiteId as Site1 (clusterA_PRI) and other has Site2 (clusterA_SBY). So, the mongoconfig.cfg looks like: |
||
| Step 4 |
To install the database and synchronize the mongoConfig.cfg file across the cluster execute the following commands:
|
||
| Step 5 |
Set priority
using
The primary member of individual replica-sets are on respective sites. |
||
| Step 6 |
Verify replica set status and priority is set correctly using the following command:
|
||
| Step 7 |
When the Session replication is configured then the Host collection of the Cluster database should have all the “admin replica-set” members, entries with Internal and Replication VLAN IP's. By default,
Example:
Example:For sessionmgr01
SITE-A, in
and for SITE-B
on sessionmgr01, in
As, IP addresses of sessionmgr VMs are different in this case, user needs to run the following scripts on both SITEs.
|
||
| Step 8 |
From this Cluster Manager, copy mongoConfig.cfg and gr_cluster.conf files to peer Cluster Managers (CM). |
CPS provides extra high availability for balance database during failover. During failover or switchover, or when primary is not available due to network reachability, balance writes happen in the backup database. After primary database is available, the records in backup database are reconciled with the primary.
The balance backup database configuration in mongoConfig appears like any other balance database with two members of replica-set on a given site. The replica-set must be created in the same manner in which regular balance database replica-sets are created.
| Step 1 |
In Policy Builder, click and select Balance Configuration from right side. In Balance Configuration, configure the primary database information. For parameter description, refer to CPS Mobile Configuration Guide. An example configuration is shown:  |
| Step 2 |
In Policy Builder, click and select Balance Configuration from right side. In Balance Configuration, configure the backup database information. For parameter description, refer to CPS Mobile Configuration Guide. An example configuration is shown:  |
| Step 3 |
The following is an example output for balance backup database: diagnostics.sh
--get_re
The balance database replica-sets for Site1 and Site2 are displayed in the example output. |
Important |
Cisco recommends to configure standby session for GR. |
CPS runs a distributed database called MongoDB. MongoDB uses a replication concept for high availability called replica-sets. A replica-set is made up of independent MongoDB instances that run in one of the following three modes:
Primary: A primary database is the only database available that can accept writes.
Secondary: A secondary database is a database that is read only and is actively synchronizing to a primary database by replaying the primary's oplog (operations log) on the local node.
Recovering: A secondary database that is currently synchronizing to the primary and has not caught up to the primary.
Session data is highly concurrent, the application always reads and writes from the primary database. The secondary database(s) provide HA for the primary in the event of VM shutdown or process shutdown. Hot standby session cache replica set is configured to take over the load while primary database is failing over to secondary session cache database. In this fail-over process, it minimizes the call failures and provides high system availability.
Hot standby replica-set must be created on different blades (for maximum protection).
Admin database and session cache databases must be separate replica-sets.
Hot standby replica-set should be added to shard configuration as backup database true.
| Step 1 |
The hotstandby database must be configured just like any other session cache database in mongo config and a replica-set needs to be created.
This needs to be created for VMware. For more information, refer to CPS Installation Guide for VMware. For OpenStack, user |
||
| Step 2 |
Verify CPS application is running on both the sites (pcrfclient01 and pcrfclient02) and without any application errors. Example:diagnostics.sh script you can get the diagnostics of
application. The
diagnostics.sh output should not contain any
application errors.
|
||
| Step 3 |
Verify whether shard command is available in OSGi console or not. From pcrfclient01, login as root user into the OSGi console and run the help. |
||
| Step 4 |
To configure hot standby session management, execute the following commands: |
||
| Step 5 |
To verify the configuration:
|
There are three possible ways a MongoDB node can fail and trigger a fail over to another node:
Replica set step down: This scenario is the cleanest method since it disconnects all client sockets and immediately initiates a new election for a master node.
Process abort: This scenario occurs when a node aborts due to an internal failure. Since this is unplanned, the other replica nodes will not request a new election for a master node until a majority of the nodes have detected the master as down.
VM power off: This scenario occurs when a node is powered off without a proper shutdown. In this case sockets are usually hung on all clients and the other replica nodes will not request a new election for a master node until a majority of the nodes have detected the master as down.
The Cisco Policy Server detects client failure by:
Utilizing the pre-packaged MongoDB Java driver software to detect failure.
Detecting server power off via server pings to rapidly detect power off situations.
You can configure only one backup database. Thus, in GR Active/Standby configuration, if you configure backup database on the standby site, during local primary to local secondary database fail over on active site, the sessions would be saved on the backup database which is on secondary site. This might increase cross-site traffic temporarily.
| Step 1 |
Configure and publish Policy Builder changes from each site. Use about.sh command to find out Policy Builder URL. Cisco recommends to configure and publish Policy Builder data separately from each site. But if the user wants to publish Policy Builder data from single site to all other sites then it is difficult to access Policy Builder data from other sites when the primary site goes down. To access Policy Builder data from other sites when primary site is down, refer Access Policy Builder from Standby Site when Primary Site is Down. |
||||||
| Step 2 |
Set appropriate Primary Database IP address, Secondary Database IP address and Port numbers for the following plug-ins:
|
||||||
| Step 3 |
Set as SecondaryPreferred for all databases except balance database.  An example Cluster configuration is given:  Also update Lookaside Key Prefixes and Admin Database sections. For more information, refer to CPS Mobile Configuration Guide. |
||||||
| Step 4 |
It is recommended to publish Policy Builder changes from each site. If a user is using primary site to publish Policy Builder changes then publishing into all the following cluster repositories is not recommended:
|
||||||
| Step 5 |
Add all the above repositories. Repository and Publish screen looks like:   |
||||||
| Step 6 |
Validate both setups using diagnostics.sh, after publishing the repository (wait for five minutes) OR follow the Validate VM Deployment section in the Cisco Policy Suite Installation Guide for this release. |
| Step 1 |
This is recommended only when primary site is down and secondary is only for reading/viewing purpose. It is applicable only where user publishes policy builder data from single site that is, primary site to all other sites. |
| Step 2 |
Open Policy Builder from secondary site (use about.sh command to find out PB URL). |
| Step 3 |
Create new data repository SEC-RUN-RO using URL ‘http://< Management interface public IP address of secondary pcrfclient01>/repos/run’, screen looks like:  |
| Step 4 |
Access Policy Builder from secondary site using newly created repository. |
The following changes are required in qns.conf file when session replication is required for active/active or active/standby GR deployments.
For active/active GR deployment, Geo HA feature needs to be enabled. For more information, refer to Active/Active Geo HA - Multi-Session Cache Port Support.
| Step 1 |
Add the
following GR related parameters in
|
||
| Step 2 |
Add the
following GR related parameters in
|
||
| Step 3 |
For multi-cluster, the following setting should be present only for GR (multi-cluster) CPS deployments:
|
||
| Step 4 |
Create
etc directory on each cluster using
|
||
| Step 5 |
Copy the changes in qns.conf to other VMs: copytoall.sh
/etc/broadhop/qns.conf /etc/broadhop/qns.conf
|
||
| Step 6 |
Restart all software components on the target VMs: restartall.sh
|
||
| Step 7 |
Validate setup
using
|
Note |
To understand traffic switch over, refer to Load Balancer VIP Outage. |
| Step 1 |
Add the list of
databases that needs to be migrated to primary on other site after traffic
switch over in
Add the
following content in the configuration file ( The following is an example and needs to be changed based on your requirement. |
||
| Step 2 |
Rebuild etc directory on cluster by executing the following command: /var/qps/install/current/scripts/build/build_etc.sh
|
Feedback