This event is raised when exception occurs while converting policies to policy rules.

Message: 1001 Policies not configured.

Corrective action needs to be taken as per the log message and corresponding configuration error needs to be corrected as mentioned in the logs.

This event is raised when policy engine is not initialized.

Message: Last policy configuration failed with the message: Policy engine is not initialized

Log file is logged with the warning message: Policy engine is not initialized

Make sure that policy engine is initialized.

This event occurs when non policy root object exists.

Message: Last policy configuration failed with the message: Policy XMI file contains non policy root object

Log file is logged with the error message: Policy XML file contains non policy root object.

To add policy root object in Policies.

This event occurs when policy does not contain a root blueprint.

Message: Last policy configuration failed with the message: Policy Builder configurations does not have any Policies configured under Policies Tab.

Log file is logged with the error message: Policy does not contain a root blueprint. Please add one under the policies tab.

To add configures in Policies tab.

The event occurs when configured blueprint is missing.

Message: Last policy configuration failed with the message: There is a configured blueprint <configuredBlueprintId> for which the original blueprint is not found <originalBluePrintId>. You are missing software on your server that is installed in Policy Builder.

Log file is logged with the error message: There is a configured blueprint <configuredBlueprintId> for which the original blueprint is not found <originalBluePrintId>. You are missing software on your server that is installed in Policy Builder.

Make sure that the blueprints are installed.

This event occurs when error was detected while converting Policy Builder configuration to CPS Rrules when the server restarts or when Publish happens.

Message: Last policy configuration failed with the message: exception stack trace.

Log file is logged with the error message: Exception stack trace is logged.

Correct policy configuration based on the exception.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of the peer actually being down.

Check the status of the Diameter Peer, and if found down, troubleshoot the peer to return it to service.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of a network connectivity issue.

Check the status of the Diameter Peer, and if found UP, check the network connectivity between CPS and the Diameter Peer. It should be reachable from both sides.

In case of a down alarm getting generated intermittently followed by a clear alarm, there could be a possibility of an intermittent network connectivity issue.

Check the network connectivity between CPS and the Diameter Peer for intermittent issues and troubleshoot the network connection.

In case of an alarm raised after any recent PB configuration change, there may be a possibility of the PB configurations related to the Diameter Peer being accidently not configured correctly.

1. Verify the changes recently made in PB by taking the SVN diff.

2. Review all PB configurations related to the Diameter Peer (port number, realm, and so on) for any incorrect data and errors.

3. Make sure that the application on Diameter Peer is listening on the port configured in PB.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of all the peer actually being down.

Check the status of each Diameter Peer, and if found down, troubleshoot each peer to return it to service.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of a network connectivity issue.

Check the status of the each Diameter Peer, and if found up, check the network connectivity between CPS and each Diameter Peer. It should be reachable from each side.

In case of a down alarm getting generated intermittently followed by a clear alarm, there could be a possibility of an intermittent network connectivity issue.

Check the network connectivity between CPS and the Diameter Peers for intermittent issues and troubleshoot the network connection.

In case of an alarm raised after any recent PB configuration change, there may be a possibility of the PB configurations related to the Diameter Peers being incorrect.

1. Verify the changes recently made in PB by taking the SVN diff.

2. Review all PB configurations related to each peer (port number, realm, and so on) for any incorrect data and errors.

3. Make sure that the application on each Diameter Peer is listening on the port configured in PB.

In case of a down alarm being generated but no clear alarm being generated, Diameter stack is not configured properly or some configuration is missing.

Check the Policy Builder configuration. Specifically check for local endpoints configuration under Diameter stack.

1. Verify localhost name defined is matching the actual hostname of the policy director (load balancer) VMs.

2. Verify instance number given matches with the policy director instance running on the policy director (load balancer) VM.

3. Verify all the policy director (load balancer) VMs are added in local endpoint configuration.

In case of an alarm raised after a recent PB configuration change, there may be a possibility that the PB configurations related to the Diameter Stack has been accidently misconfigured.

1. Verify the changes recently made in PB by taking the SVN diff.

2. Review all PB configurations related to the Diameter Stack (local hostname, advertise fqdn, and so on) for any incorrect data and errors.

3. Make sure that the application is listening on the port configured in PB in CPS.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of the SMSC Server actually being down.

Check the status of the SMSC Server, and if found down, troubleshoot the server to return it to service.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of a network connectivity issue.

Check the status of the SMSC Server, and if found up, check the network connectivity between CPS and the Server. It should be reachable from both sides.

In case of a down alarm getting generated intermittently followed by a clear alarm, there could be a possibility of an intermittent network connectivity issue.

Check the network connectivity between CPS and the SMSC Server for intermittent issues and troubleshoot the network connection.

In case of an alarm raised after any recent PB configuration change, there may be a possibility of the PB configurations related to the SMSC Server being incorrect.

1. Verify the changes recently made in PB by taking the SVN diff.

2. Review all PB configurations related to SMSC Server (port number, realm, and so on) for any incorrect data and errors.

3. Make sure that the application on SMSC Server is listening on the port configured in PB.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of all the SMSC Servers actually being down.

Check the status of each SMSC Server, and if found down, troubleshoot the servers to return them to service.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of a network connectivity issue.

Check the status of each SMSC Server, and if found up, check the network connectivity between CPS and each SMSC Server. It should be reachable from each side.

In case of a down alarm getting generated intermittently followed by a clear alarm, there could be a possibility of an intermittent network connectivity issue.

Check the network connectivity between CPS and the SMSC Servers for intermittent issues and troubleshoot the network connection.

In case of an alarm raised after any recent PB configuration change, there may be a possibility of the PB configurations related to the SMSC Servers being incorrect.

1. Verify the changes recently made in PB by taking the SVN diff.

2. Review all PB configurations related to SMSC Servers (port number, realm, and so on) for any incorrect data and errors.

3. Make sure that the application on each SMSC Server is listening on the respective port configured in PB.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of the Email Server actually being down.

Check the status of the Email Server, and if found down, troubleshoot the server to return it to service.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of a network connectivity issue.

Check the status of Email Server, and if found up, check the network connectivity between CPS and the Email Server. It should be reachable from both sides.

In case of a down alarm getting generated intermittently followed by a clear alarm, there could be a possibility of an intermittent network connectivity issue.

Check the network connectivity between CPS and the Email Server for intermittent issues and troubleshoot the network connection.

In case of an alarm raised after any recent PB configuration change, there may be a possibility of the PB configurations related to the Email Server being incorrect.

1. Verify the changes recently made in PB by taking the SVN diff.

2. Review all PB configurations related to Email Server (port number, realm, and so on) for any incorrect data and errors.

3. Make sure that the application on Email Server is listening on the port configured in PB.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of all the Email Servers actually being down.

Check the status of each Email Server, and if found down, troubleshoot the server to return it to service.

In case of a down alarm being generated but no clear alarm being generated, there could be a possibility of a network connectivity issue.

Check the status of the each Email Server, and if found up, check the network connectivity between CPS and each Email Server. It should be reachable from each side.

In case of a down alarm getting generated intermittently followed by a clear alarm, there could be a possibility of an intermittent network connectivity issue.

Check the network connectivity between CPS and the Email Servers for intermittent issues and troubleshoot the network connection.

In case of an alarm raised after any recent PB configuration change, there may be a possibility of the PB configurations related to the Email Servers being incorrect.

1. Verify the changes recently made in PB by taking the SVN diff.

2. Review all PB configurations related to Email Servers (port number, realm, and so on) for any incorrect data and errors.

3. Make sure that the application on each Email Server is listening on the respective port configured in Policy Builder.

The memcached process is down on lbvip02.

Check the memcached process on lbvip02. If the process is stopped, start the process using the command monit start memcached assuming the monit service is already started.

The Policy Server VMs fail to reach/connect to lbvip02 or lbvip02:11211.

Check for connectivity issues from Policy Server (QNS) to lbvip02 using ping/telnet command. If the network connectivity issue is found, fix the connectivity.

The test operation to check memcached server timed out. This can happen if the memcached server is slow to respond/network delays OR if the application pauses due to GC. If the error is due to application pause due to GC, it will mostly get resolved when the next diagnostics is run.

1. Check the parameter -DmemcacheClientTimeout in qns.conf file. If the parameter is not present, the default timeout is 50 ms. So if the application pause is >= 50 ms, this issue can be seen. The pause can be monitored in service-qns-x.log file. The error should subside in the next diagnostics run if it was due to application GC pause.

2. Check for network delays for RTT from Policy Server to lbvip02.

The test operation to check memcached server health failed with exception.

Check the exception message and if an exception is caused, during that time only, the diagnostics for memcached should pass in the next run. Check if the memcached process is up on lbvip02. Also check for network connectivity issues.

Internal services cannot connect to a required Java ZeroMQ queue. Although retry logic and recovery is available, and core system functions should continue, investigate and remedy the root cause.

1. Login to the IP mentioned in the alarm and check if the Policy Server (QNS) process is up on that VM. If it is not up, start the process.

2. Login to the IP mentioned in the alarm and check if the port mentioned in the alarm is listening using the netstat command).

   netstat -apn | grep <port>

   If not, check the Policy Server logs for any errors.

3. Check if the VM which raised the alarm is able to connect to the mentioned socket using the telnet command.

   telnet <ip> <port>

   If it is a network issue, fix it.

All LDAP servers are down.

Check if the external LDAP servers are up and if the LDAP server processes are up. If not, bring the servers and the respective server processes up.

Connectivity issues from the LB to LDAP servers.

Check the connectivity from Policy Director (LB) to LDAP server. Check (using ping/telnet) if LDAP server is reachable from Policy Director (LB) VM. If not, fix the connectivity issues.

The mentioned LDAP server in the alarm message is down.

Check if the mentioned external LDAP server is up and if the LDAP server process is up on that server. If not, bring the server and the server processes up.

Connectivity issues from the Policy Director (LB) to the mentioned LDAP server address in the alarm.

Check the connectivity from Policy Director (LB) to mentioned LDAP server. Check (using ping/telnet) if LDAP server is reachable from Policy Director (LB) VM. If not, fix the connectivity issues.

This alarm is generated if installed feature cannot start.

1. Check which images are installed on which CPS hosts by reading /var/qps/images/image-map.

2. Check which features are part of which images by reading /etc/broadhop/<image-name>/features file.

   Note	A feature which cannot start must be in at least one of images.

3. Check if feature which cannot start has its jar in compressed image archive of all images found in above steps.

4. If jar is missing contact Cisco support for required feature. If jar is present, collect logs from /var/log/broadhop on VM where feature cannot start for further analysis.

This alarm is generated when the internal Policy Director (LB) VIP virtual interface does not respond to a ping. Corosync detects this and moves the VIP interface to another Policy Director (LB). The alarm then clears when the other node takes over and a ViritualInterface Up trap is sent.

No action is required since the alarm is cleared automatically as long as a working Policy Director (LB) node gets the VIP address.

This alarm is generated when the internal Policy Director (LB) VIP virtual interface does not respond to a ping and selection of a new VIP hosts fails.

1. Run diagnostics.sh on Cluster Manager as root user to check for any failures on the Policy Director (LB) nodes..

2. Make sure that both policy director nodes are running. If problems are noted, refer to CPS Troubleshooting Guide for further steps required to restore policy director node function problem.

3. After all the policy directors are up, if the trap still does not clear, restart corosync on all policy directors using the monit restart corosync command.

This alarm is generated when a VM listed in the /etc/hosts does not respond to a ping.

1. Run diagnostics.sh on Cluster Manager as root user to check for any failures.

2. For all VMs with FAIL, refer to CPS Troubleshooting Guide for further steps required to restore the VM function.

This alarm is generated during mongo failover or when majority of replica-set members are not available.

1. Login to pcrfclient01/02 VM and verify the replica-set status

   diagnostics.sh --get_replica_status

   Note	If a member is shown in an unknown state, it is likely that the member is not accessible from one of other members, mostly an arbiter. In that case, you must go to that member and check its connectivity with other members. Also, you can login to mongo on that member and check its actual status.

2. If the member is not running start the mongo process on each sessionmgr/arbiter VM

   For example, /usr/bin/systemctl start sessionmgr-port

   Note	Change the port number (port) according to your deployment.

3. Verify the mongo process, if the process does not come UP then verify the mongo logs for further debugging log.

   For example, /var/log/mongodb-port.log

   Note	Change the port number (port) according to your deployment.

This alarm is generate in the event of abrupt failure of arbiter VM and does not come up due to some unspecified reason (In HA - arbiter VM is pcrfclient01/02 and for GR - third site or based on deployment model).

1. Login to pcrfclient01/02 VM and verify the replica-set status

   diagnostics.sh --get_replica_status

   Note	If a member is shown in an unknown state, it is likely that the member is not accessible from one of other members, mostly an arbiter. In that case, you must go to that member and check its connectivity with other members. Also, you can login to mongo on that member and check its actual status.

2. Login to arbiter VM for which the alarm has generated.

3. Check the status of mongo port for which alarm has generated.

   For example, ps –ef | grep 27720

4. If the member is not running, start the mongo process.

   For example, /usr/bin/systemctl start sessionmgr-27720

5. Verify the mongo process, if the process does not come UP then verify the mongo logs for further debugging log.

   For example, /var/log/mongodb-port.log

   Note	Change the port number (port) according to your deployment.

This alarm is generated in the event of abrupt failure of configServer VM (when mongo sharding is enabled) and does not come up due to some unspecified reasons.

1. Login to pcrfclient01/02 VM and verify the shard health status

   diagnostics.sh --get_shard_health <dbname>

2. Check the status of mongo port for which alarm has generated.

   For example, ps –ef | grep 27720

3. If the member is not running, start the mongo process.

   For example, /usr/bin/systemctl start sessionmgr-27720

4. Verify the mongo process, if the process does not come UP then verify the mongo logs for further debugging log.

   For example, /var/log/mongodb-port.log

   Note	Change the port number (port) according to your deployment.

This alarm is generated in the event of abrupt failure of all sessionmgr VMs and does not come up due to some unspecified reason or all members are down.

1. Login to pcrfclient01/02 VM and verify the replica-set status

   diagnostics.sh --get_replica_status

   Note	If a member is shown in an unknown state, it is likely that the member is not accessible from one of other members, mostly an arbiter. In that case, you must go to that member and check its connectivity with other members. Also, you can login to mongo on that member and check its actual status.

2. If the member is not running start the mongo process on each sessionmgr/arbiter VM

   For example, /usr/bin/systemctl start sessionmgr-port

   Note	Change the port number (port) according to your deployment.

3. Verify the mongo process, if the process does not come UP then verify the mongo logs for further debugging log.

   For example, /var/log/mongodb-port.log

   Note	Change the port number (port) according to your deployment.

This alarm is generated whenever a secondary member of replica-set of mongo database does not recover automatically after failure. For example, if sessionmgr VM is down for longer time and after recovery the secondary member does not recover.

1. Login to pcrfclient01/02 VM and verify the replica-set status

   diagnostics.sh --get_replica_status

   Note	If a member is shown in an unknown state, it is likely that the member is not accessible from one of other members, mostly an arbiter. In that case, you must go to that member and check its connectivity with other members. Also, you can login to mongo on that member and check its actual status.

2. Check which member is in recovering/fatal or startup2 state.

3. Login to that sessionmgr VM and check for mongo logs.

   Refer to CPS Troubleshooting Guide for recover procedure.

This alarm is generated if Policy Server (QNS) process on one of the CPS VMs is down.

1. Run diagnostics.sh on Cluster Manager as root user to check for any failures..

2. On VM where qns is down, run monit summary to check if "monit" is monitoring policy server (QNS) process.

3. Analyze logs in /var/log/broadhop directory for exceptions and errors.

1. Gx traffic to the CPS system is beyond system capacity.

2. CPU utilization is very high on qnsXX VM.

3. Mongo database performance is not optimal.

1. Login via Grafana dashboard and check for any Gx message processing trend.

2. Check CPU utilization on all the Policy Server (QNS) VMs via grafana dashboard.

3. Login to pcrfclient01/02 VM and check the mongo database health.

   diagnostics.sh --get_replica_status

   Note	If a member is shown in an unknown state, it is likely that the member is not accessible from one of other members, mostly an arbiter. In that case, you must go to that member and check its connectivity with other members. Also, you can login to mongo on that member and check its actual status.

4. Check for any unusual exceptions in consolidated policy server (qns) and mongo logs.

1. Gx traffic to the CPS system is beyond system capacity.

2. CPU utilization is very high on qnsXX VM.

3. Mongo database performance is not optimal.

1. Login via Grafana dashboard and check for any Gx message processing trend.

2. Check CPU utilization on all the Policy Server (QNS) VMs via grafana dashboard.

3. Login to pcrfclient01/02 VM and check the mongo database health.

   diagnostics.sh --get_replica_status

   Note	If a member is shown in an unknown state, it is likely that the member is not accessible from one of other members, mostly an arbiter. In that case, you must go to that member and check its connectivity with other members. Also, you can login to mongo on that member and check its actual status.

4. Check for any unusual exceptions in consolidated policy server (qns) and mongo logs.

Multiple LDAP servers are configured and LDAP servers are down.

1. Check connectivity between CPS and all LDAP servers configured in Policy Builder.

2. Check latency between CPS to all LDAP servers and LDAP server response time should be normal.

3. Restore connectivity if any LDAP server is down.

1. Gx traffic to the CPS system is beyond system capacity.

2. CPU utilization is very high on qnsXX VM.

3. Mongo database performance is not optimal.

1. Check connectivity between CPS and all LDAP servers configured in Policy Builder.

2. Check latency between CPS to all LDAP servers and LDAP server response time should be normal.

3. Check policy server (qns) logs on policy director (lb) VM for which alarm has been generated.

Multiple LDAP servers are configured and LDAP servers are down.

1. Check connectivity between CPS and all LDAP servers configured in Policy Builder.

2. Check latency between CPS to all LDAP servers and LDAP server response time should be normal.

3. Restore connectivity if any LDAP server is down.

Gx traffic to the CPS system is increased beyond system capacity.

1. Check connectivity between CPS and all LDAP servers configured in Policy Builder.

2. Check latency between CPS to all LDAP servers and LDAP server response time should be normal.

3. Check policy server (qns) logs on policy director (lb) VM for which alarm has been generated.

Binding Not Available at Policy DRA

This alarm is generated whenever binding database at Policy DRA is down.

This alarm gets cleared automatically after the time configured in Policy Builder (Diameter Configuration > PolicyDRA Health Check > Alarm Config > Alarm Clearance Interval is reached.

A network issue/latency in establishing connection to the remote SPR databases.

Check the network connection/latency and adjust the qns.conf parameter -DserverSelectionTimeout.remoteSpr in consultation with Cisco Technical Representative.

qns.node.warmup.hostname.substring parameter is not configured in qns.conf file.

GeoSiteName is not configured if it is a GR setup

• If alarm contains ‘didn’t start node num/SITE_ID not parsed’, make sure that qns.node.warmup.hostname.substring and GeoSiteName (if it is GR setup) is configured in qns.conf file. Policy Server (QNS) VMs hostname must only contain number after substring parameter is configured.

• If alarm contains ‘didn't start due to exception’, please consult with Cisco Technical Representative.

SPR node is not available

When the member(s) of the replica-set are manually recovered and a master node is available for the SPR replica-set, the alarm automatically clears.

GC State

Restart the Policy Server (qns) application for which alarm was reported. After gc_alarm_trigger_interval is reached, if there is no GC triggered, the alarm gets cleared.

OldGen State

Restart the Policy Server (qns) application for which alarm was reported. On restart, if oldGen value is less than configured oldgen_clear_ trigger_thr_per value, the alarm gets cleared.

SessionLimitOverload

ProtectionNotSet

Go to System configuration in Policy Builder and set the value for Session limit Overload Protection to recommended value and publish it. This will clear the alarm within 30 seconds.

SessionLimitOverload

ProtectionExceeded

Increase the database capacity after consulting with Cisco representative or clear the sessions in the session database so that 'n' becomes less than 'm' (n<m). This should clear the alarm within 30 seconds.