Contents

Introduction

Prerequisites

Requirements

Components used

Background Information

UAME

Problem

Procedure for the Re-deployment

To Delete

Introduction

This document describes the procedure to deploy or re-deploy Ultra Automation and Monitoring Engine (UAME).

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Cisco Ultra Virtual Packet Core solutions components

- UAME

- Openstack

Components used

The information in this document is based on these software and hardware versions:

- USP 6.9.0

- UAME

- Cloud - Openstack 13 (Queens)

Background Information

UAME

The UAME is a new Ultra Automation Services (UAS) software module introduced to:

- Support the integrated deployment of 4G or 5G Virtualized Network Functions (VNFs) and 5G Cloud-Native Network Functions (CNFs).

- Reduce the number of Virtual Machines (VMs) typically required by USP and UAS, replacing the UEM, AutoIT, AutoDeploy, and AutoVNF components.

The UAME provides deployment orchestration for these:

1. 4GVNFs:

1.1. CUPS-based VNFs: The UAME works with the Virtual Network Function Manager (VNFM) to deploy VPC SI-based control plane (CP) and user plane (UP) VNFs in support of the Control and User Plane Separation (CUPS) architecture.

1.2. Non-CUPS-based VNFs: The UAME is backwards-compatible, working with the VNFM to provide deployment support for non-CUPS 4G gateways (based on VPC-DI) and the 4G Policy and Charging Rules Function (PCRF).

2. 5G NFs:

2.1. VNF-based NFs: The UAME works with the VNFM to deploy VPC-SI-based Network Functions (NFs).

2.2. Cloud-native-based NFs: The UAME interacts with the VNFM to deploy the Ultra Cloud Core Subscriber Microservices Infrastructure (SMI). The SMI then works with the VNFM to deploy NFs in a VM-based Kubernetes (also known as K8s) cluster.

Problem

One or both of the UAME VMs are corrupted. You are unable to recover UAME even after you reboot the hosting compute server from OpenStack.

nova start/nova rebooot --hard also fails even after setting the state of the VM active from OpenStack with this command.

nova reset-state --active <UAME_Vm_NAME>.

Procedure for the Re-deployment

1. Log in to OpenStack Platform Director (OSPD) and verify mount configuration.

[stack@<POD-NAME>-ospd usp-images]$ df -h

Filesystem Size Used Avail Use% Mounted on

devtmpfs 189G 0 189G 0% /dev

tmpfs 189G 80K 189G 1% /dev/shm

tmpfs 189G 2.7M 189G 1% /run

tmpfs 189G 0 189G 0% /sys/fs/cgroup

/dev/sda2 1.1T 109G 930G 11% /

/dev/loop0 543M 543M 0 100% /mnt/ucs-c220m5-huu-4.1.1g

/dev/sda1 477M 102M 346M 23% /boot

tmpfs 38G 0 38G 0% /run/user/0

tmpfs 38G 0 38G 0% /run/user/1000

/dev/loop1 4.0G 4.0G 0 100% /home/stack/usp-6_9_8/usp-6_9_8-mount

2. If the mount is not available, you can manually mount it with these steps.

# cd /home/stack

# mkdir /home/stack/usp-6_9_8/usp-6_9_8-mount

# sudo mount -t iso9660 -o loop /home/stack/usp-6_9_8/usp-images/usp-6_9_8.iso /home/stack/usp-6_9_8/usp-6_9_8-mount

mount: /dev/loop0 is write-protected, mounting read-only

3. Verify the contents of the ISO file.

(undercloud) [stack@<POD-NAME>-ospd ~]$ ll /home/stack/usp-6_9_8/usp-6_9_8-mount/tools

total 22

-r--r--r--. 1 root root 8586 Sep 1 2020 cisco_openpgp_verify_release.py

-r-xr-xr-x. 1 root root 1955 Sep 1 2020 uas-certs.sh

-r-xr-xr-x. 1 root root 5534 Sep 1 2020 usp-csar-installer.sh

-r-xr-xr-x. 1 root root 2546 Sep 1 2020 usp-gpg-key.sh

-r-xr-xr-x. 1 root root 3354 Sep 1 2020 usp-uas-installer.sh

(undercloud) [stack@<POD-NAME>-ospd ~]$

4. Verify that these files are available.

(undercloud) [stack@<POD-NAME>-ospd ~]$ ll /opt/cisco/usp/uas-installer/images/

total 909544

-rw-r--r--. 1 root root 931367936 Aug 20 2020 usp-uas-6.9.0-9247.qcow2

(undercloud) [stack@<POD-NAME>-ospd ~]$ ll /opt/cisco/usp/uas-installer/scripts/

total 180

-rwxr-xr-x. 1 root root 806 Jun 24 18:28 auto-deploy-booting.sh

-rwxr-xr-x. 1 root root 5460 Jun 24 18:28 autoit-user.py

-rwxr-xr-x. 1 root root 811 Jun 24 18:28 auto-it-vnf-staging.sh

-r-xr-x---. 1 root root 102209 Jun 24 18:28 boot_uas.py

-rwxr-xr-x. 1 root root 4762 Jun 24 18:28 encrypt_account.sh

-rwxr-xr-x. 1 root root 3945 Jun 24 18:28 encrypt_credentials.sh

-rwxr-xr-x. 1 root root 16019 Jun 24 18:28 start-ultram-vm.py

-rwxr-xr-x. 1 root root 15315 Jun 24 18:28 uas-boot.py

-rwxr-xr-x. 1 root root 5384 Jun 24 18:28 uas-check.py

-rwxr-xr-x. 1 root root 11283 Jun 24 18:28 usp-tenant.py

(undercloud) [stack@<POD-NAME>-ospd ~]$

5. If not, launch the usp-uas-installer.sh script. It will extract and make the previously mentioned files available.

[stack@<POD-NAME>-ospd tools]$ sudo ./usp-uas-installer.sh

6. List the qcow2 image and boot_uas.py script.

[root@<POD-NAME>-ospd ~]# cd /opt/cisco/usp/bundles/uas-bundle

[root@<POD-NAME>-ospd uas-bundle]# ll

total 909572

drwxr-xr-x. 3 root root 4096 Jun 24 18:28 models

drwxr-xr-x. 2 root root 4096 Jun 24 18:28 tools

-rw-r--r--. 1 root root 649 Aug 20 2020 usp-build-info.json

-rw-r--r--. 1 root root 97 Aug 20 2020 usp-bundle-manifest.yml

-rw-r--r--. 1 root root 931367936 Aug 20 2020 usp-uas-6.9.0-9247.qcow2

-rw-r--r--. 1 root root 32 Aug 20 2020 usp-uas-6.9.0-9247.qcow2.md5

-rw-r--r--. 1 root root 40 Aug 20 2020 usp-uas-6.9.0-9247.qcow2.sha1

-rw-r--r--. 1 root root 128 Aug 20 2020 usp-uas-6.9.0-9247.qcow2.sha512

[root@<POD-NAME>-ospd uas-bundle]# cd tools/

[root@<POD-NAME>-ospd tools]# ll

total 100

-rwxr-xr-x. 1 root root 102209 Aug 20 2020 boot_uas.py

[root@<POD-NAME>-ospd tools]#

[root@<POD-NAME>-ospd tools]# ll /opt/cisco/usp/uas-installer/images/

total 909544

-rw-r--r--. 1 root root 931367936 Aug 20 2020 usp-uas-6.9.0-9247.qcow2

7. Extract the usp bundle with the use of this step if the previously mentioned files are not available.

(undercloud) [stack@<POD-NAME>-ospd ~]$ sudo su -

Last login: Tue Sep 7 02:20:36 UTC 2021 from 10.255.143.5 on pts/0

[root@<POD-NAME>-ospd ~]# ll /home/stack/usp-6_9_8/usp-6_9_8-mount/repo/

total 4142608

-r--r--r--. 1 root root 623 Sep 1 2020 rel.gpg

-r--r--r--. 1 root root 87783720 Sep 1 2020 usp-auto-it-bundle-5.8.0-1.x86_64.rpm

-r--r--r--. 1 root root 1008975328 Sep 1 2020 usp-em-bundle-6.9.0-1.x86_64.rpm

-r--r--r--. 1 root root 1168 Sep 1 2020 USP_RPM_CODE_REL_KEY-CCO_RELEASE.cer

-r--r--r--. 1 root root 918264637 Sep 1 2020 usp-uas-bundle-6.9.0-1.x86_64.rpm

-r--r--r--. 1 root root 886391928 Sep 1 2020 usp-ugp-bundle-21.15.47-1.x86_64.rpm

-r--r--r--. 1 root root 1340535896 Sep 1 2020 usp-vnfm-bundle-4.5.0.112-1.x86_64.rpm

-r--r--r--. 1 root root 74725 Sep 1 2020 usp-yang-bundle-1.0.0-1.x86_64.rpm

[root@<POD-NAME>-ospd ~]#

[root@<POD-NAME>-ospd tools]# cd /

[root@<POD-NAME>-ospd /]# rpm2cpio /home/stack/usp-6_9_8/usp-6_9_8-mount/repo/usp-uas-bundle-6.9.0-1.x86_64.rpm | cpio -idmv

8. Delete the UAME deployment and then re-deploy

To Delete

./boot_uas.py --autovnf --delete 1624559350-098061

The deployment ID will be available as mentioned highlighted above or in the file mentioned “/home/stack/UAME_$(date +"%Y%m%d-%H%M").log”. Refer the latest file.

(undercloud) [stack@<POD-NAME>-ospd ~]$ cat UAME_0624Jun061624559462.log

2021-06-24 18:30:30,392 - Deployment: 1624559350-098061 instantiated successfully

1. Deploy the UAME.

(undercloud) [stack@<POD-NAME>-ospd ~]$ sudo -s

[root@<POD-NAME>-ospd stack]# source *core

(<POD-NAME>) (<POD-NAME>) [root@<POD-NAME>-ospd stack]# cd /opt/cisco/usp/uas-installer/scripts

(<POD-NAME>) (<POD-NAME>) [root@<POD-NAME>-ospd scripts]# ./boot_uas.py --openstack --uame --image /opt/cisco/usp/uas-installer/images/usp-uas-6.9.0-9247.qcow2 --flavor <PODNAME>-UAME-FLAVOR --net <PODNAME>-MGMT-NW --net <PODNAME>-ORCH-NW --ha --hostname <PODNAME>-UAME --ha-net <PODNAME>-MGMT-NW --password password --gateway 172.168.10.1 --floating-ip <floating ip of UAME> --external-network <PODNAME>-EXTERNAL-MGMT --admin password--oper  password --security password

Note: "password" has to be replaced with the UAME login password and "PODNAME" with the current POD. "floating-ip" should be obtained from the IP matrix

2021-06-24 18:28:52,225 - Uploading image '<POD-NAME>-UAME-usp-uas-6.9.0-9247' from '/opt/cisco/usp/uas-installer/images/usp-uas-6.9.0-9247.qcow2'

2021-06-24 18:29:06,945 - Uploaded image '<POD-NAME>-UAME-usp-uas-6.9.0-9247' successfully

2021-06-24 18:29:09,987 - Creating Server Group to enforce anti-affinity

2021-06-24 18:29:10,098 - Deployment started with transaction id --- 1624559350-098061

2021-06-24 18:29:11,766 - Created HA VIP, IP: 172.168.20.40

2021-06-24 18:29:17,125 - Allocating/Associating floating-ip

2021-06-24 18:29:17,125 - Acquire Lock : floating_ip

2021-06-24 18:29:17,125 - Lock floating_ip acquired

2021-06-24 18:29:20,474 - Assigned floating IP '10.250.100.198' to IP '172.168.20.40'

2021-06-24 18:29:20,475 - Released lock: floating_ip

2021-06-24 18:29:26,206 - Server: <POD-NAME>-UAME instantiated, waiting for server to be active

2021-06-24 18:30:01,415 - Server: <POD-NAME>-UAME instantiated, waiting for server to be active

2021-06-24 18:30:30,392 - Deployment: 1624559350-098061 instantiated successfully

2021-06-24 18:30:30,393 -

+--------------------------------------------------------------------+

Deployment ID | Instances

----------------------------------------------------------------------

1624559350-098061 | e71616e8-bf01-4561-bdd6-4e3bf3ed1d5e

VIP: 172.168.20.40 | eth0: 172.168.10.6/24

Floating IP: 10.250.100.198 | eth1: 172.168.11.23/24

|

| 3d956097-16b1-4909-b539-c6a90e01678c

| eth0: 172.168.10.18/24

| eth1: 172.168.11.8/24

+--------------------------------------------------------------------+

2. Save deployment id manually in the file /home/stack/uame_(date).log.

grep -i "deployment:" /var/log/autovnf.log | tail -1 >> /home/stack/UAME_$(date +"%Y%m%d-%H%M").log

3. Check if the UAME is active and running.

(<POD-NAME>) [stack@<POD-NAME>-ospd ~]$ nova list | grep -i uame

| e71616e8-bf01-4561-bdd6-4e3bf3ed1d5e | <POD-NAME>-UAME-1 | ACTIVE | - | Running | <POD-NAME>-MGMT-NW=172.168.10.6; <POD-NAME>-ORCH-NW=172.168.11.23 |

| 3d956097-16b1-4909-b539-c6a90e01678c | <POD-NAME>-UAME-2 | ACTIVE | - | Running | <POD-NAME>-MGMT-NW=172.168.10.18; <POD-NAME>-ORCH-NW=172.168.11.8 |

4. Login into both the UAME and perform these checks.

[root@adi-tmo Downloads]# ssh ubuntu@10.250.100.198

ubuntu@10.250.100.198's password:

Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-187-generic x86_64)

* Documentation: https://help.ubuntu.com

* Management: https://landscape.canonical.com

* Support: https://ubuntu.com/advantage

Cisco Ultra Services Platform (USP)

Build Date: Thu Aug 20 09:11:07 EDT 2020

Description: UAS build assemble-uas#9247

sha1: 557151c

ubuntu@<POD-NAME>-uame-1:~$

ubuntu@<POD-NAME>-uame-1:~$ sudo su -

root@<POD-NAME>-uame-1:~# confd_cli -u admin -C

Enter Password for 'admin':

elcome to the ConfD CLI

admin connected from 127.0.0.1 using console on <POD-NAME>-uame-1

<POD-NAME>-uame-1#

<POD-NAME>-uame-1#show uas

uas version 6.9.0

uas state active

uas external-connection-point 172.168.20.40

INSTANCE IP STATE ROLE

------------------------------------

172.168.10.6 alive CONFD-MASTER

172.168.10.18 alive CONFD-SLAVE

NAME LAST HEARTBEAT

----------------------------------------------------------------

AutoVNF-MASTER 2021-09-07 05:11:03

ESCHeartBeatMonitor-<POD-NAME>-VNF-NEPCF300 2021-09-07 05:11:26

ESCHeartBeatMonitor-<POD-NAME>-VNF-NEPGW300 2021-09-07 05:11:22

USPCFMWorker 2021-09-07 05:11:06

USPCHBWorker 2021-09-07 05:11:06

USPCWorker 2021-09-07 05:11:02

<POD-NAME>-uame-1#