Introduction
This document describes the recovery of the PS VM in Policy and Charging Rules Function (PCRF) Cluster.
Troubleshoot
Power on any Instance from SHUTOFF State
If any instance is in SHUTOFF state due to a planned shutdown or some other reason, please use this procedure to start the instance and enable it’s monitoring in Elastic Service Controllor (ESC).
Step 1. Check the state of instance via OpenStack.
source /home/stack/destackovsrc-Pcrf
nova list --fields name,host,status | grep qns-s1
| c5e4ebd4-803d-45c1-bd96-fd6e459b7ed6 | SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d | destackovs-compute-2 | SHUTOFF|
Step 2. Check if the compute is available and ensure that the state is up.
source /home/stack/destackovsrc
nova hypervisor-show destackovs-compute-2 | egrep ‘status|state’
| state | up |
| status | enabled
Step 3. Login to ESC Master as admin user and check the state of instance in opdata.
echo "show esc_datamodel opdata tenants tenant Pcrf deployments * state_machine | tab" | /opt/cisco/esc/confd/bin/confd_cli -u admin –C | grep qns-s1
SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d VM_ERROR_STATE
Step 4. Power on the instance from openstack.
source /home/stack/destackovsrc-Pcrf
nova start SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d
Step 5. Wait five minutes for the instance to boot up and come to active state.
source /home/stack/destackovsrc-Pcrf
nova list –fields name,status | grep qns-s1
| c5e4ebd4-803d-45c1-bd96-fd6e459b7ed6 | SVS1-tmo_qns-s2_0_2899f208-4c27-4296-bb8b-1342cf555e5d | ACTIVE |
Step 6. Enable VM Monitor in ESC after instance is in active state.
/opt/cisco/esc/esc-confd/esc-cli/esc_nc_cli vm-action ENABLE_MONITOR SVS1-tmo_qns-s2_0_2899f208-4c27-4296-bb8b-1342cf555e5d
For Further recovery of instance configurations, refer instance type specific procedures provided in next section
Recover any Instance from ERROR State
This procedure can be used if state of CPS instance in openstack is ERROR:
Step 1. Check the state of instance in OpenStack.
source /home/stack/destackovsrc-Pcrf
nova list --fields name,host,status | grep qns-s1
| c5e4ebd4-803d-45c1-bd96-fd6e459b7ed6 | SVS1-tmo_qns-s2_0_2899f208-4c27-4296-bb8b-1342cf555e5d | destackovs-compute-2 | ERROR|
Step 2. Check if the compute is available and runs fine.
source /home/stack/destackovsrc
nova hypervisor-show destackovs-compute-2 | egrep ‘status|state’
| state | up |
| status | enabled
Step 3. Login to ESC Master as admin user and check the state of instance in opdata.
echo "show esc_datamodel opdata tenants tenant Pcrf deployments * state_machine | tab" | /opt/cisco/esc/confd/bin/confd_cli -u admin –C | grep qns-s1
SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d VM_ERROR_STATE
Step 4. Reset the state of instance to force the instance back to an active state instead of an error state, once done, reboot your instance.
source /home/stack/destackovsrc-Pcrf
nova reset-state –active SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d
nova reboot –-hard SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d
Step 5. Wait five minutes for the instance to boot up and come to active state.
source /home/stack/destackovsrc-Pcrf
nova list –fields name,status | grep qns-s1
| c5e4ebd4-803d-45c1-bd96-fd6e459b7ed6 | SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d | ACTIVE |
Step 6. If, Cluster Manager changes state to ACTIVE after reboot, Enable VM Monitor in ESC after Cluster Manager instance is in active state.
/opt/cisco/esc/esc-confd/esc-cli/esc_nc_cli vm-action ENABLE_MONITOR SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d
Post recovery to running/active state, refer instance type specific procedure to recover config/data from backup.
QNS Recovery
If a load balancer is recently recovered use this procedure to restore haproxy and network setting:
Step 1. If it is required to import the restore QNS configuration data on the Cluster Manager, run this command:
config_br.py -a import --users --haproxy /mnt/backup/<filename.tgz>
Step 2. In order to generate the VM archive files on the Cluster Manager through the latest configurations, run this command:
/var/qps/install/current/scripts/build/build_all.sh
Step 3. In order to update QNS with the latest configuration, login to QNS and run this command
ssh qnsxx
/etc/init.d/vm-init
Verify
Run diagnostics from cluster manager diagnostics.sh
Feedback