Contents

Introduction

Pre-Checks

AutoVNF Backup

AutoVFN Redeploy

Backup Restoration

Post-Checks

Introduction

This document describes the steps required in order to redeploy an AutoVNF in an Ultra-M. The AutoVNF is responsible in order to bring up individual Virtual Network Function Manager (VNFM) and Virtual Network Function (VNF).

Pre-Checks

1. Log in to the OpenStack Platform Director (OSPD) and verify the status of Virtual Machine (VMs) in the health report.

[stack@labucs300-ospd ~]$ cat /var/log/cisco/ultram-health/*.report | grep -i xxx

[stack@labucs300-ospd ~]$ cat /var/log/cisco/ultram-health/ultram_health_uas.report

---------------------------------------------------------------------------------------------------------

VNF-ID/VNFD-ID           | UAS Node | Status| Error Info, if any

---------------------------------------------------------------------------------------------------------

10.10.10.40/LABPGW300-UAS | autovnf  | :-)   | LABPGW300-UAS:(alive)

|          |       |   labucs300-UAS-LABPGW300-UAS-core-UAS2-2:(alive)

|          |       |   labucs300-UAS-LABPGW300-UAS-core-UAS2-1:(alive)

10.10.10.40/LABPCF300-UAS | autovnf  | :-)   | LABPCF300-UAS:(alive)

|          |       |   labucs300-UAS-LABPCF300-UAS-core-UAS1-2:(alive)

|          |       |   labucs300-UAS-LABPCF300-UAS-core-UAS1-1:(alive)

10.10.10.45/LABPCF300-UGP | vnf-em   | :-)   | LABPCF300-UGP:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-EM1-3:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-EM1-2:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-EM1-1:(alive)

10.10.10.45/LABPCF300-ESC | esc      | :-)   | LABPCF300-ESC:(alive)

|          |       |   LABPCF300-LABPCF300-ESC-core-ESC1-1:(alive)

|          |       |   LABPCF300-LABPCF300-ESC-core-ESC1-2:(alive)

10.10.10.45/LABPCF300-UGP | vnf      | :-)   | LABPCF300-UGP:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-CF-VDU-1:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-CF-VDU-0:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-3:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-2:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-1:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-0:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-6:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-5:(alive)

|          |       |   LABPCF300-LABPCF300-UGP-core-LABPCF300-SF-VDU-4:(alive)

10.10.10.48/LABPGW300-UGP | vnf-em   | :-)   | LABPGW300-UGP:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-EM2-2:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-EM2-3:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-EM2-1:(alive)

10.10.10.48/LABPGW300-ESC | esc      | :-)   | LABPGW300-ESC:(alive)

|          |       |   LABPGW300-LABPGW300-ESC-core-ESC2-1:(alive)

|          |       |   LABPGW300-LABPGW300-ESC-core-ESC2-2:(alive)

10.10.10.48/LABPGW300-UGP | vnf      | :-)   | LABPGW300-UGP:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-4:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-5:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-6:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-0:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-1:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-2:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-SF-VDU-3:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-CF-VDU-0:(alive)

|          |       |   LABPGW300-LABPGW300-UGP-core-LABPGW300-CF-VDU-1:(alive)

---------------------------------------------------------------------------------------------------------

2. Verify the AutoVNF status.

[stack@labucs300-ospd ~]$ source *core

[stack@labucs300-ospd ~]$ nova list | grep LABPGW300-UAS-core-UAS2

| 8608fda4-b763-4753-95ff-2e07852098e3 | labucs300-UAS-LABPGW300-UAS-core-UAS2-1 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.15; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.7 |

| 19f4496c-3907-4ea5-84c9-e5a6ef222392 | labucs300-UAS-LABPGW300-UAS-core-UAS2-2 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.17; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.16

labucs300-UAS-LABPGW300-UAS-core-UAS2-2 -> VNFM2-UAS-VIP Primary

labucs300-UAS-LABPGW300-UAS-core-UAS2-1 -> Secondary

3. Log in to the AutoIT and check the heartbeat logs.

ubuntu@labucs300-autoit-2:~$ grep "'restarting'" /var/log/cisco/uas/heartbeat.log

2021-02-22 01:41:42,808 - 192.0.2.15: Notify Event: {'action': 'restart', 'source': 'heartbeat', 'event': 'restarting', 'ip': '192.0.2.15'}

2021-02-22 01:45:42,251 - 192.0.2.15: Notify Event: {'action': 'restart', 'source': 'heartbeat', 'event': 'restarting', 'ip': '192.0.2.15'}

2021-02-23 01:43:36,013 - 192.0.2.15: Notify Event: {'action': 'restart', 'source': 'heartbeat', 'event': 'restarting', 'ip': '192.0.2.15'}

2021-02-23 01:45:55,785 - 192.0.2.15: Notify Event: {'action': 'restart', 'source': 'heartbeat', 'event': 'restarting', 'ip': '192.0.2.15'}

2021-02-24 01:45:19,680 - 192.0.2.15: Notify Event: {'action': 'restart', 'source': 'heartbeat', 'event': 'restarting', 'ip': '192.0.2.15'}

ubuntu@labucs300-autoit-2:~$ cd /var/log/cisco/uas

ubuntu@labucs300-autoit-2:/var/log/cisco/uas$ grep "Rebooting Instance" uas_USPCHBWorker.log

2019-06-26 18:26:13,088 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392

2019-06-29 03:45:12,710 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392

2020-07-17 00:46:25,800 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392

2020-07-18 00:47:13,347 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392

2020-07-18 05:11:11,133 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392

2020-07-18 5:16:07,333 - Rebooting Instance: 19f4496c-3907-4ea5-84c9-e5a6ef222392

AutoVNF Backup

AutoVNF is responsible to bring up individual VNFM and VNF. AutoDeploy sends the configuration required to instantiate both VNFM and VNF to AutoVNF and AutoVNF does this operation. In order to bring up VNFM,

AutoVNF talks directly to VIM/openstack and once VNFM is up, AutoVNF uses VNFM to bring up VNF.

AutoVNF has 1:1 redundancy and in UltraM setup, 2 AutoVNF VMs running in HA mode.

AutoVNF backup details:

- Running configuration

- ConfD CDB DB

- AutoVNF logs (from each instance of AutoVNF)

- Syslog configuration

Important: The backups must be performed before any activation/deactivation on the given POD/site and uploaded to the backup server.

1. Set ha_debug to ON on both Primary and Secondary AutoIT in /opt/uas_baseconfig.txt.

Note: The ha_debug flag is all capital. It is either ON or OFF.

ubuntu@labucs300-autoit-2:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 172.16.181.7

ha-secret: f99d04acb84807c4c6c6c0eaad392b5c

orch-ips: 172.16.181.5,172.16.181.8

orch-intf: eth0

prov-ha-vip: 172.16.181.13

prov-intf: eth0

profile: AUTOIT

ha_debug: ON

ubuntu@labucs300-autoit-1:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 172.16.181.7

ha-secret: f99d04acb84807c4c6c6c0eaad392b5c

orch-ips: 172.16.181.5,172.16.181.8

orch-intf: eth0

prov-ha-vip: 172.16.181.13

prov-intf: eth0

profile: AUTOIT

ha_debug: ON

2. Stop AutoIT service on Primary AutoIT with these commands:

ssh   ubuntu@ < AutoIT Floating IP>

sudo -i

service autoit stop

This is to prevent AutoIT from triggering automatic recovery of Ultra Automation Services (UAS) when the shutdown is initiated on later steps.

ubuntu@labucs300-autoit-2:~$ sudo -i

root@labucs300-autoit-2:~#

root@labucs300-autoit-2:~# service autoit status

autoit start/running, process 25001

root@labucs300-autoit-2:~# service autoit stop

3. Set ha_debug to ON on both Primary and Secondary AutoVNF (UAS) in the file /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 192.0.2.8

ha-secret: d5a9fee60ddae4fe357677dcf1320e51

orch-ips: 192.0.2.15,192.0.2.17

orch-intf: eth0

profile: AUTOVNF

ha_debug: ON

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ ssh ubuntu@192.0.2.15

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-1:~$ cat /opt/uas_baseconfig.txt

ha: true

ha-vip: 192.0.2.8

ha-secret: d5a9fee60ddae4fe357677dcf1320e51

orch-ips: 192.0.2.15,192.0.2.17

orch-intf: eth0

profile: AUTOVNF

ha_debug: ON

4. Shutdown Secondary UAS from OSPD with commends.

. corerc ; openstack server stop <VMName>

labucs300-UAS-LABPGW300-UAS-core-UAS2-2 -> VNFM2-UAS-VIP Primary

labucs300-UAS-LABPGW300-UAS-core-UAS2-1 -> Secondary

[stack@labucs300-ospd ~]$ nova list | grep LABPGW300-UAS-core-UAS2

| 8608fda4-b763-4753-95ff-2e07852098e3 | labucs300-UAS-LABPGW300-UAS-core-UAS2-1 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.15; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.7 |

| 19f4496c-3907-4ea5-84c9-e5a6ef222392 | labucs300-UAS-LABPGW300-UAS-core-UAS2-2 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.17; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.16

[stack@labucs300-ospd ~]$. corerc ; openstack server stop labucs300-UAS-LABPGW300-UAS-core-UAS2-1

5. Stop uas-confd and autovnf services on UAS with these commands:

service uas-confd stop

service autovnf stop

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ sudo -i

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd status

uas-confd start/running, process 1305

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service autovnf status

autovnf start/running, process 24208

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd stop

uas-confd stop/waiting

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service autovnf stop

autovnf stop/waiting

6. Take a backup of the UAS conf database and copy it to a backup server with these commands:

cd /opt/cisco/usp/uas/confd-latest/var/confd/

tar -cvf <pod>_<VNF>_UAS_cdb_backup.tar cdb/

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# cd /opt/cisco/usp/uas/confd-latest/var/confd/

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# tar -cvf Autovnf_cdb_backup.tar cdb/

cdb/

cdb/O.cdb

cdb/C.cdb

cdb/aaa_init.xml

cdb/A.cdb

root@labucs300-uas-LABPGW300-uas-core-uas2-2:/opt/cisco/usp/uas/confd-latest/var/confd# ll Autovnf_cdb_backup_cdb_backup.tar

total 1612

drwxr-xr-x 3 root root 4096 Jan 24 2017 ..

drwxr-xr-x 2 root root 4096 Jan 24 2017 log

drwxr-xr-x 8 root root 4096 Oct 11 11:30 webui

drwxr-xr-x 2 root root 4096 Oct 19 19:18 candidate

drwxr-xr-x 2 root root 4096 Oct 23 13:00 rollback

drwxr-xr-x 2 root root 4096 Oct 28 17:00 cdb

drwxr-xr-x 3 root root 4096 Oct 28 17:00 state

drwxr-xr-x 8 root root 4096 Oct 31 18:00 .

-rw-r--r-- 1 root root 1617920 Oct 31 18:00 Autovnf_cdb_backup.tar

7. Restart uas-confd and autovnf services on UAS with these commands:

service uas-confd start

service autovnf start

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd start

uas-confd start/running, process 13852

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service autovnf start

autovnf start/running, process 13853

8. Restart Secondary UAS from OSPD with these commands:

. corerc ; openstack server start <VMName>

[stack@labucs300-ospd ~]$. corerc ; openstack server start labucs300-UAS-LABPGW300-UAS-core-UAS2-1

9. Verify both Primary and Secondary UAS show up as alive in show uas command.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ sudo -i

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# confd_cli -u admin -C

Welcome to the ConfD CLI

admin connected from 127.0.0.1 using console on labucs300-uas-LABPGW300-uas-core-uas2-2

labucs300-uas-LABPGW300-uas-core-uas2-2#show uas

uas version 6.2.0

uas state active

uas external-connection-point 192.0.2.8

INSTANCE IP STATE ROLE

-----------------------------------

0.0.0.0 error CONFD-Secondary

192.0.2.15 alive CONFD-Secondary

192.0.2.17 alive CONFD-Primary

10. Change ha_debug to OFF on both Primary and Secondary AutoVNF (UAS) in the file /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 192.0.2.8

ha-secret: d5a9fee60ddae4fe357677dcf1320e51

orch-ips: 192.0.2.15,192.0.2.17

orch-intf: eth0

profile: AUTOVNF

ha_debug: OFF

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ ssh ubuntu@192.0.2.15

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-1:~$ cat /opt/uas_baseconfig.txt

ha: true

ha-vip: 192.0.2.8

ha-secret: d5a9fee60ddae4fe357677dcf1320e51

orch-ips: 192.0.2.15,192.0.2.17

orch-intf: eth0

profile: AUTOVNF

ha_debug: OFF

11. Restart autoit service on Primary AutoIT with service autoit start command.

ubuntu@labucs300-autoit-2:~$ sudo -i

root@labucs300-autoit-2:~# service autoit start

12. Verify that both Primary and Secondary AutoIT show up as alive with show uas command.

ubuntu@labucs300-autoit-2:~$ sudo -i

root@labucs300-autoit-2:~# confd_cli -u admin -C

Welcome to the ConfD CLI

admin connected from 127.0.0.1 using console on labucs300-autoit-2

labucs300-autoit-2#show uas

uas version 6.2.0

uas state active

uas external-connection-point 172.16.181.7

INSTANCE IP STATE ROLE

-----------------------------------

172.16.181.5 alive CONFD-Secondary

172.16.181.8 alive CONFD-Primary

13. Set ha_debug to OFF on both Primary and Secondary AutoIT in the file /opt/uas_baseconfig.txt.

ubuntu@labucs300-autoit-2:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 172.16.181.7

ha-secret: f99d04acb84807c4c6c6c0eaad392b5c

orch-ips: 172.16.181.5,172.16.181.8

orch-intf: eth0

prov-ha-vip: 172.16.181.13

prov-intf: eth0

profile: AUTOIT

ha_debug: OFF

ubuntu@labucs300-autoit-1:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 172.16.181.7

ha-secret: f99d04acb84807c4c6c6c0eaad392b5c

orch-ips: 172.16.181.5,172.16.181.8

orch-intf: eth0

prov-ha-vip: 172.16.181.13

prov-intf: eth0

profile: AUTOIT

ha_debug: OFF

14. Collect logs from the UAS and transfer them to a backup server.

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# cd /opt/cisco/usp/uas/confd-latest/var/confd/

root@labucs300-uas-LABPGW300-uas-core-uas2-2:/opt/cisco/usp/uas/confd-latest/var/confd# cd /opt/cisco/usp/uas/scripts/

root@labucs300-uas-LABPGW300-uas-core-uas2-2:/opt/cisco/usp/uas/scripts# sudo ./collect-uas-logs.sh

Dumping output for show transaction in file /tmp/uas-logs/transactions.txt

Dumping output for show log in file /tmp/uas-logs/transactions.txt

Dumping output for show running-config in file /tmp/uas-logs/confd_output.txt

Dumping output for show uas in file /tmp/uas-logs/confd_output.txt

Dumping output for show usp in file /tmp/uas-logs/confd_output.txt

..........

15. Log in to the secondary AutoVNF and repeat previous step in order to collect the logs and transfer them to the backup server.

16. Backup the syslog configuration on the Primary and Secondary AutoVNF VMs and transfer them to the backup server. The files are located in these directories:

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# ls /etc/rsyslog.d/00-autovnf.conf

/etc/rsyslog.d/00-autovnf.conf

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# ls /etc/rsyslog.conf

/etc/rsyslog.conf

17. Enable autoit service with service autoit start command on Primary AutoIT.

ubuntu@labucs300-autoit-2:~$ sudo -i

root@labucs300-autoit-2:~#

root@labucs300-autoit-2:~# service autoit start

autoit start/running, process 25001

18. Set the ha_debug flag mode to OFF in the /opt/uas_baseconfig.txt on Primary AutoVNF and AutoIT.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 192.0.2.8

ha-secret: d5a9fee60ddae4fe357677dcf1320e51

orch-ips: 192.0.2.15,192.0.2.17

orch-intf: eth0

profile: AUTOVNF

ha_debug: OFF

ubuntu@labucs300-autoit-2:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 172.16.181.7

ha-secret: f99d04acb84807c4c6c6c0eaad392b5c

orch-ips: 172.16.181.5,172.16.181.8

orch-intf: eth0

prov-ha-vip: 172.16.181.13

prov-intf: eth0

profile: AUTOIT

ha_debug: OFF

19. Confirm uas-confd and autovnf services running on UAS.

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd status

uas-confd start/running, process 1305

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service autovnf status

autovnf start/running, process 24208

AutoVFN Redeploy

1. Log into AutoDeploy and note the UAS instance.

ubuntu@labucs300-autodeploy-2:~$ sudo su

root@labucs300-autodeploy-2:/home/ubuntu# confd_cli -u admin -C

Welcome to the ConfD CLI

admin connected from 127.0.0.1 using console on labucs300-autodeploy-2

labucs300-autodeploy-2#show nsr

nsr         LABSGW300-instance

nsd         LABSGW300

vnfr        [ LABPCF300-LABPCF300-ESC LABPCF300-LABPCF300-UGP ]

vnf-package [ usp_6_2_b8 ]

vim-artifact vim_art_rack

nsr LABPGW300-instance

nsd         LABPGW300

vnfr        [ LABPGW300-LABPGW300-ESC LABPGW300-LABPGW300-UGP ]

vnf-package [ usp_6_2_b8 ]

vim-artifact vim_art_rack

nsr labucs300-UAS-instance

nsd         labucs300-UAS

vnfr        [ labucs300-UAS-LABPCF300-UAS labucs300-UAS-LABPGW300-UAS ]

vnf-package [ usp_6_2_b8 ]

vim-artifact vim_art_rack

2. Deactivate AutoVNF from AutoDeploy with deactivate nsd-id <nsd-id> vnfd <vnfd-id> command.

ubuntu@labucs300-autodeploy-2:~$ /opt/cisco/usp/uas/confd-6.3.1/bin/confd_cli -u admin -C

Welcome to the ConfD CLI

admin connected from 10.10.10.10 using ssh on labucs300-autodeploy-2

labucs300-autodeploy-2#nsd:deactivate nsd-id labucs300-UAS vnfd [LABPGW300-UAS]

transaction-id 1560431372-357328

3. Confirm the status of the transaction.

labucs300-autodeploy-2#show transaction

DEPLOYMENT                                                 STATUS

TX ID                               TX TYPE                ID            TIMESTAMP                        STATUS      DETAIL

------------------------------------------------------------------------------------------------------------------------------

1560431372-357328                   activate-ns-deployment labucs300-UAS 2019-06-13T13:09:32.357355-00:00 in-progress -

1560431372-357328/1560431373-102024 activate-ns-deployment labucs300-UAS 2019-06-13T13:09:33.102041-00:00 in-progress -

4. Check logs of the transaction. In this case labucs300-UAS transactions: 1560431372-357328 and 1560431372-357328/1560431373-102024.

labucs300-autodeploy-2#show log 1560431372-357328 | display xml

<config xmlns="http://tail-f.com/ns/config/1.0">

<log xmlns="http://www.cisco.com/usp/nfv/usp-transaction">

<tx-id>1560431372-357328</tx-id>

<log>

2019-06-13 13:09:33,367 - Send Deployment notification for: labucs300-UAS-instance

2019-06-13 13:09:33,375 - Deployment activate-ns-deployment: labucs300-UAS started

2019-06-13 13:09:33,378 - Adding NSR: labucs300-UAS-instance

2019-06-13 13:09:33,385 - Start pipeline of 1 tasks

2019-06-13 13:09:33,390 - Scheduling Task: labucs300-UAS

2019-06-13 13:09:33,400 - Waiting for all workers to finish the transactions

2019-06-13 13:15:00,006 - Deployment activate-ns-deployment: labucs300-UAS succeeded

2019-06-13 13:15:00,020 - Send Deployment notification for: labucs300-UAS-instance

2019-06-13 13:09:33,437 - Send Deployment notification for: labucs300-UAS-instance-deploy

2019-06-13 13:09:33,441 - Deployment activate-ns-deployment: labucs300-UAS started

……………

labucs300-autodeploy-2#show log 1560431372-357328/1560431373-102024 | display xml

<config xmlns="http://tail-f.com/ns/config/1.0">

<log xmlns="http://www.cisco.com/usp/nfv/usp-transaction">

<tx-id>1560431372-357328/1560431373-102024</tx-id>

<log>

2019-06-13 13:09:33,437 - Send Deployment notification for: labucs300-UAS-instance-deploy

2019-06-13 13:09:33,441 - Deployment activate-ns-deployment: labucs300-UAS started

2019-06-13 13:09:33,446 - Adding NSR: labucs300-UAS-instance, VNFR: labucs300-UAS-LABPCF300-UAS, vlrs: None

2019-06-13 13:09:33,453 - Adding NSR: labucs300-UAS-instance, VNFR: labucs300-UAS-LABPGW300-UAS, vlrs: None

2019-06-13 13:09:33,463 - VNF deployment pre-check success(all-not-present)

2019-06-13 13:09:33,472 - VNF-Package deployment pre-check success(all-not-present)

2019-06-13 13:09:33,481 - VIM-Artifact deployment pre-check success

2019-06-13 13:09:33,487 - Skipping VIM-Orch pre-deployment, since VIM-Orch is not defined

2019-06-13 13:09:33,496 - Skipping VIM pre-deployment, since VIM is not defined

2019-06-13 13:09:33,499 - NS pre-check success

2019-06-13 13:09:33,503 - Copying '/home/ubuntu/usp-6_2_b8.iso' to '/var/cisco/isos/labucs300-UAS_usp_6_2_b8'

2019-06-13 13:09:53,359 - Updated path to URL 'http://172.16.181.14:5000/isos/labucs300-UAS_usp_6_2_b8'

5. Wait until the transaction completed and confirm the state.

labucs300-autodeploy-2#show transaction

DEPLOYMENT                                            STATUS

TX ID                               TX TYPE                ID            TIMESTAMP                        STATUS DETAIL

------------------------------------------------------------------------------------------------------------------------------

1560431372-357328                   activate-ns-deployment labucs300-UAS 2019-06-13T13:09:32.357355-00:00 success -

1560431372-357328/1560431373-102024 activate-ns-deployment labucs300-UAS 2019-06-13T13:09:33.102041-00:00 success -

6. Activate AutoVNF VNFD from AutoDeploy with activate nsd-id <nsd-id> vnfd <vnfd-id> command.

ubuntu@labucs300-autodeploy-2:~$ /opt/cisco/usp/uas/confd-6.3.1/bin/confd_cli -u admin -C

Welcome to the ConfD CLI

admin connected from 10.253.110.47 using ssh on labucs300-autodeploy-2

labucs300-autodeploy-2#nsd:activate nsd-id labucs300-UAS vnfd [LABPGW300-UAS]

transaction-id 1560431371-357330

7. Check status of the transaction and collect logs from the transaction with these commands:

show transaction

show log <transaction-id> | display xml

show log <transaction-id> | display xml

8. Wait till the transactions are completed. The show transaction command shows the state of transactions.

Backup Restoration

1. On Primary AutoIT set the ha_debug flag mode to ON in /opt/uas_baseconfig.txt.

ubuntu@labucs300-autoit-2:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 172.16.181.7

ha-secret: f99d04acb84807c4c6c6c0eaad392b5c

orch-ips: 172.16.181.5,172.16.181.8

orch-intf: eth0

prov-ha-vip: 172.16.181.13

prov-intf: eth0

profile: AUTOIT

ha_debug: ON

2. On Primary AutoIT stop service autoit. This step is to prevent AutoIT to automatically recover UAS.

ubuntu@labucs300-autoit-2:~$ sudo -i

root@labucs300-autoit-2:~#

root@labucs300-autoit-2:~# service autoit status

autoit start/running, process 25001

root@labucs300-autoit-2:~# service autoit stop

3. On Primary UAS set the ha_debug flag mode to ON in /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 192.0.2.8

ha-secret: d5a9fee60ddae4fe357677dcf1320e51

orch-ips: 192.0.2.15,192.0.2.17

orch-intf: eth0

profile: AUTOVNF

ha_debug: ON

4. On Secondary UAS set the ha_debug flag mode to ON in /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-1:~$ cat /opt/uas_baseconfig.txt

ha: true

ha-vip: 192.0.2.8

ha-secret: d5a9fee60ddae4fe357677dcf1320e51

orch-ips: 192.0.2.15,192.0.2.17

orch-intf: eth0

profile: AUTOVNF

ha_debug: ON

5. On OSPD stop Secondary UAS server with the OpenStack command.

labucs300-UAS-LABPGW300-UAS-core-UAS2-2 -> VNFM2-UAS-VIP Primary

labucs300-UAS-LABPGW300-UAS-core-UAS2-1 -> Secondary

[stack@labucs300-ospd ~]$. corerc ; openstack server stop labucs300-UAS-LABPGW300-UAS-core-UAS2-1

6. On Primary UAS stop uas-confd service.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ sudo -i

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd status

uas-confd start/running, process 1305

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# service uas-confd stop

uas-confd stop/waiting

7. On Primary UAS, copy the backuped archive of CDB to directory /opt/cisco/usp/uas/confd-latest/var/confd/.

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# cp Autovnf_cdb_backup.tar to /opt/cisco/usp/uas/confd-latest/var/confd/

8. On Primary UAS delete files under CBD directory.

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# cd /opt/cisco/usp/uas/confd-latest/var/confd/ ; rm cdb/*

9. On Primary UAS extract files from the CDB backup file.

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# cd /opt/cisco/usp/uas/confd-latest/var/confd/ ; tar -xvf <archive_backup_tar_file>

10. On OSPD reboot Primary UAS with OpenStack commands.

[stack@labucs300-ospd ~]$ source *core

[stack@labucs300-ospd ~]$ nova list | grep LABPGW300-UAS-core-UAS2

| 8608fda4-b763-4753-95ff-2e07852098e3 | labucs300-UAS-LABPGW300-UAS-core-UAS2-1 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.15; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.7 |

| 19f4496c-3907-4ea5-84c9-e5a6ef222392 | labucs300-UAS-LABPGW300-UAS-core-UAS2-2 | ACTIVE | - | Running | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.17; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.16

labucs300-UAS-LABPGW300-UAS-core-UAS2-2 -> VNFM2-UAS-VIP Primary

labucs300-UAS-LABPGW300-UAS-core-UAS2-1 -> Secondary

[stack@labucs300-ospd ~]$ nova reboot --hard 19f4496c-3907-4ea5-84c9-e5a6ef222392

Request to reboot server <Server: auto-testautovnf1-uas-2> has been accepted.

11. Wait till the Primary UAS comes up. Check the status os UAS on Primary UAS after the reboot. The Primary state is alive while secondary one shows state unknown.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ sudo -i

root@labucs300-uas-LABPGW300-uas-core-uas2-2:~# confd_cli -u admin -C

Welcome to the ConfD CLI

admin connected from 127.0.0.1 using console on labucs300-uas-LABPGW300-uas-core-uas2-2

labucs300-uas-LABPGW300-uas-core-uas2-2#show uas

uas version 6.2.0

uas state active

uas external-connection-point 192.0.2.8

INSTANCE IP STATE ROLE

-----------------------------------

192.0.2.15 unknown CONFD-Secondary

192.0.2.17 alive CONFD-Primary

12. On OSPD, start Secondary UAS with the OpenStack command.

[stack@labucs300-ospd ~]$. corerc ; openstack server start labucs300-UAS-LABPGW300-UAS-core-UAS2-1

13. On OSPD, verify the states of the Primary and Secondary UAS are Active.

[stack@labucs300-ospd ~]$ openstack server list | grep labucs300-UAS-LABPGW300

| 19f4496c-3907-4ea5-84c9-e5a6ef222392 | labucs300-UAS-LABPGW300-UAS-core-UAS2-2 | ACTIVE | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.17; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.16 | labucs300-UAS-usp_6_2_b8-core-uas |

| 8608fda4-b763-4753-95ff-2e07852098e3 | labucs300-UAS-LABPGW300-UAS-core-UAS2-1 | ACTIVE | labucs300-UAS-LABPGW300-ORCH-NW=192.0.2.15; labucs300-UAS-LABPGW300-MGMT-NW=192.0.2.7 | labucs300-UAS-usp_6_2_b8-core-uas |

14. On Primary UAS verify the states of the Primary and Secondary UAS are alive.

labucs300-uas-LABPGW300-uas-core-uas2-2#show uas

uas version 6.2.0

uas state active

uas external-connection-point 192.0.2.8

INSTANCE IP STATE ROLE

-----------------------------------

192.0.2.15 alive CONFD-Secondary

192.0.2.17 alive CONFD-Primary

15. On Primary AutoIT start autoit service.

ubuntu@labucs300-autoit-2:~$ sudo -i

root@labucs300-autoit-2:~# service autoit start

16. Verify that the Secure Shell (SSH) session stays up for a couple of minutes to Primary and Secondary UAS.

17. On Primary UAS set the ha_debug flag mode to OFF in /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-2:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 192.0.2.8

ha-secret: d5a9fee60ddae4fe357677dcf1320e51

orch-ips: 192.0.2.15,192.0.2.17

orch-intf: eth0

profile: AUTOVNF

ha_debug: OFF

18.  On Secondary UAS set the ha_debug flag mode to OFF in /opt/uas_baseconfig.txt.

ubuntu@labucs300-uas-LABPGW300-uas-core-uas2-1:~$ cat /opt/uas_baseconfig.txt

ha: true

ha-vip: 192.0.2.8

ha-secret: d5a9fee60ddae4fe357677dcf1320e51

orch-ips: 192.0.2.15,192.0.2.17

orch-intf: eth0

profile: AUTOVNF

ha_debug: OFF

19. On Primary AutoIT set the ha_debug flag mode to OFF in /opt/uas_baseconfig.txt.

ubuntu@labucs300-autoit-2:~$ vi /opt/uas_baseconfig.txt

ha: true

ha-vip: 172.16.181.7

ha-secret: f99d04acb84807c4c6c6c0eaad392b5c

orch-ips: 172.16.181.5,172.16.181.8

orch-intf: eth0

prov-ha-vip: 172.16.181.13

prov-intf: eth0

profile: AUTOIT

ha_debug: OFF

20. On AutoVNF check the 00-autovnf.conf and rsyslog.conf files and restore them from previous backup.

ubuntu@autoit-tb1-autovnf1-core-avf-1:~#sudo su

root@autoit-tb1-autovnf1-core-avf-1:~#ls /etc/rsyslog.d/00-autovnf.conf

00-autovnf.conf

root@autoit-tb1-autovnf1-core-avf-1:~#/home/ubuntu#ls /etc/rsyslog.conf

rsyslog.conf

Post-Checks

On OSPD verify that both AutoVNFs are active and check Ultra-M health check report.

[stack@labucs300-ospd ~]$ cat /var/log/cisco/ultram-health/*.report | grep -i xxx

[stack@labucs300-ospd ~]$ cat /var/log/cisco/ultram-health/ultram_health_uas.report