Contents

Introduction

Troubleshoot

Power on any Instance from SHUTOFF State

Recover any Instance from ERROR State

QNS Recovery

Verify

Introduction

This document describes the recovery of the PS VM in Policy and Charging Rules Function (PCRF) Cluster.

Troubleshoot

Power on any Instance from SHUTOFF State

If any instance is in SHUTOFF state due to a planned shutdown or some other reason, please use this procedure to start the instance and enable it’s monitoring in Elastic Service Controllor (ESC).

Step 1. Check the state of instance via OpenStack.

source /home/stack/destackovsrc-Pcrf

nova list --fields name,host,status | grep qns-s1

| c5e4ebd4-803d-45c1-bd96-fd6e459b7ed6 | SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d | destackovs-compute-2  | SHUTOFF|

Step 2. Check if the compute is available and ensure that the state is up.

source /home/stack/destackovsrc

nova hypervisor-show destackovs-compute-2 | egrep ‘status|state’

| state                     | up                                       |

| status                    | enabled

Step 3. Login to ESC Master as admin user and check the state of instance in opdata.

echo "show esc_datamodel opdata tenants tenant Pcrf deployments * state_machine | tab" | /opt/cisco/esc/confd/bin/confd_cli -u admin –C | grep qns-s1

SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d VM_ERROR_STATE

Step 4. Power on the instance from openstack.

source /home/stack/destackovsrc-Pcrf

nova start SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d

Step 5. Wait five minutes for the instance to boot up and come to active state.

source /home/stack/destackovsrc-Pcrf

nova list –fields name,status | grep qns-s1

| c5e4ebd4-803d-45c1-bd96-fd6e459b7ed6 | SVS1-tmo_qns-s2_0_2899f208-4c27-4296-bb8b-1342cf555e5d  | ACTIVE |

Step 6. Enable VM Monitor in ESC after instance is in active state.

/opt/cisco/esc/esc-confd/esc-cli/esc_nc_cli vm-action ENABLE_MONITOR SVS1-tmo_qns-s2_0_2899f208-4c27-4296-bb8b-1342cf555e5d

For Further recovery of instance configurations, refer instance type specific procedures provided in next section

Recover any Instance from ERROR State

This procedure can be used if state of CPS instance in openstack is ERROR:

Step 1. Check the state of instance in OpenStack.

source /home/stack/destackovsrc-Pcrf

nova list --fields name,host,status | grep qns-s1

| c5e4ebd4-803d-45c1-bd96-fd6e459b7ed6 | SVS1-tmo_qns-s2_0_2899f208-4c27-4296-bb8b-1342cf555e5d  | destackovs-compute-2  | ERROR|

Step 2. Check if the compute is available and runs fine.

source /home/stack/destackovsrc

nova hypervisor-show destackovs-compute-2 | egrep ‘status|state’

| state                     | up                                       |

| status                    | enabled

Step 3. Login to ESC Master as admin user and check the state of instance in opdata.

echo "show esc_datamodel opdata tenants tenant Pcrf deployments * state_machine | tab" | /opt/cisco/esc/confd/bin/confd_cli -u admin –C | grep qns-s1

SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d VM_ERROR_STATE

Step 4. Reset the state of instance to force the instance back to an active state instead of an error state, once done, reboot your instance.

source /home/stack/destackovsrc-Pcrf

nova reset-state –active SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d

nova reboot –-hard  SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d

Step 5. Wait five minutes for the instance to boot up and come to active state.

source /home/stack/destackovsrc-Pcrf

nova list –fields name,status | grep qns-s1

| c5e4ebd4-803d-45c1-bd96-fd6e459b7ed6 | SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d | ACTIVE |

Step 6. If, Cluster Manager changes state to ACTIVE after reboot, Enable VM Monitor in ESC after Cluster Manager instance is in active state.

/opt/cisco/esc/esc-confd/esc-cli/esc_nc_cli vm-action ENABLE_MONITOR SVS1-tmo_qns-s1_0_2899f208-4c27-4296-bb8b-1342cf555e5d

Post recovery to running/active state, refer instance type specific procedure to recover config/data from backup.

QNS Recovery

If a load balancer is recently recovered use this procedure to restore haproxy and network setting:

Step 1. If it is required to import the restore QNS configuration data on the Cluster Manager, run this command:

config_br.py -a import --users --haproxy /mnt/backup/<filename.tgz>

Step 2. In order to generate the VM archive files on the Cluster Manager through the latest configurations, run this command:

/var/qps/install/current/scripts/build/build_all.sh

Step 3. In order to update QNS with the latest configuration, login to QNS and run this command

ssh qnsxx

/etc/init.d/vm-init

Verify

Run diagnostics from cluster manager diagnostics.sh