Deploy/Redeploy UAME in Case of a Corruption

Available Languages

Download Options

PDF (46.8 KB)
View with Adobe Reader on a variety of devices
ePub (83.4 KB)
View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle) (70.9 KB)
View on Kindle device or Kindle app on multiple devices

Updated:September 20, 2021

Document ID:217374

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Background Information

UAME

Problem

Procedure for the Re-deployment

To Delete

Introduction

This document describes the procedure to deploy or re-deploy Ultra Automation and Monitoring Engine (UAME).

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

Cisco Ultra Virtual Packet Core solutions components
UAME
Openstack

Components used

The information in this document is based on these software and hardware versions:

USP 6.9.0
UAME
Cloud - Openstack 13 (Queens)

Background Information

UAME

The UAME is a new Ultra Automation Services (UAS) software module introduced to:

Support the integrated deployment of 4G or 5G Virtualized Network Functions (VNFs) and 5G Cloud-Native Network Functions (CNFs).
Reduce the number of Virtual Machines (VMs) typically required by USP and UAS, replacing the UEM, AutoIT, AutoDeploy, and AutoVNF components.

The UAME provides deployment orchestration for these:

1. 4GVNFs:

1.1. CUPS-based VNFs: The UAME works with the Virtual Network Function Manager (VNFM) to deploy VPC SI-based control plane (CP) and user plane (UP) VNFs in support of the Control and User Plane Separation (CUPS) architecture.

1.2. Non-CUPS-based VNFs: The UAME is backwards-compatible, working with the VNFM to provide deployment support for non-CUPS 4G gateways (based on VPC-DI) and the 4G Policy and Charging Rules Function (PCRF).

2. 5G NFs:

2.1. VNF-based NFs: The UAME works with the VNFM to deploy VPC-SI-based Network Functions (NFs).

2.2. Cloud-native-based NFs: The UAME interacts with the VNFM to deploy the Ultra Cloud Core Subscriber Microservices Infrastructure (SMI). The SMI then works with the VNFM to deploy NFs in a VM-based Kubernetes (also known as K8s) cluster.

Problem

One or both of the UAME VMs are corrupted. You are unable to recover UAME even after you reboot the hosting compute server from OpenStack.

nova start/nova rebooot --hard also fails even after setting the state of the VM active from OpenStack with this command.

nova reset-state --active <UAME_Vm_NAME>.

Procedure for the Re-deployment

1. Log in to OpenStack Platform Director (OSPD) and verify mount configuration.

[stack@<POD-NAME>-ospd usp-images]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 189G 0 189G 0% /dev
tmpfs 189G 80K 189G 1% /dev/shm
tmpfs 189G 2.7M 189G 1% /run
tmpfs 189G 0 189G 0% /sys/fs/cgroup
/dev/sda2 1.1T 109G 930G 11% /
/dev/loop0 543M 543M 0 100% /mnt/ucs-c220m5-huu-4.1.1g
/dev/sda1 477M 102M 346M 23% /boot
tmpfs 38G 0 38G 0% /run/user/0
tmpfs 38G 0 38G 0% /run/user/1000
/dev/loop1 4.0G 4.0G 0 100% /home/stack/usp-6_9_8/usp-6_9_8-mount

2. If the mount is not available, you can manually mount it with these steps.

# cd /home/stack   

# mkdir /home/stack/usp-6_9_8/usp-6_9_8-mount

# sudo mount -t iso9660 -o loop /home/stack/usp-6_9_8/usp-images/usp-6_9_8.iso /home/stack/usp-6_9_8/usp-6_9_8-mount            

mount: /dev/loop0 is write-protected, mounting read-only

3. Verify the contents of the ISO file.

(undercloud) [stack@<POD-NAME>-ospd ~]$ ll /home/stack/usp-6_9_8/usp-6_9_8-mount/tools
total 22
-r--r--r--. 1 root root 8586 Sep 1 2020 cisco_openpgp_verify_release.py
-r-xr-xr-x. 1 root root 1955 Sep 1 2020 uas-certs.sh
-r-xr-xr-x. 1 root root 5534 Sep 1 2020 usp-csar-installer.sh
-r-xr-xr-x. 1 root root 2546 Sep 1 2020 usp-gpg-key.sh
-r-xr-xr-x. 1 root root 3354 Sep 1 2020 usp-uas-installer.sh
(undercloud) [stack@<POD-NAME>-ospd ~]$

4. Verify that these files are available.

(undercloud) [stack@<POD-NAME>-ospd ~]$ ll /opt/cisco/usp/uas-installer/images/
total 909544
-rw-r--r--. 1 root root 931367936 Aug 20 2020 usp-uas-6.9.0-9247.qcow2

(undercloud) [stack@<POD-NAME>-ospd ~]$ ll /opt/cisco/usp/uas-installer/scripts/
total 180
-rwxr-xr-x. 1 root root 806 Jun 24 18:28 auto-deploy-booting.sh
-rwxr-xr-x. 1 root root 5460 Jun 24 18:28 autoit-user.py
-rwxr-xr-x. 1 root root 811 Jun 24 18:28 auto-it-vnf-staging.sh
-r-xr-x---. 1 root root 102209 Jun 24 18:28 boot_uas.py
-rwxr-xr-x. 1 root root 4762 Jun 24 18:28 encrypt_account.sh
-rwxr-xr-x. 1 root root 3945 Jun 24 18:28 encrypt_credentials.sh
-rwxr-xr-x. 1 root root 16019 Jun 24 18:28 start-ultram-vm.py
-rwxr-xr-x. 1 root root 15315 Jun 24 18:28 uas-boot.py
-rwxr-xr-x. 1 root root 5384 Jun 24 18:28 uas-check.py
-rwxr-xr-x. 1 root root 11283 Jun 24 18:28 usp-tenant.py
(undercloud) [stack@<POD-NAME>-ospd ~]$

5. If not, launch the usp-uas-installer.sh script. It will extract and make the previously mentioned files available.

[stack@<POD-NAME>-ospd tools]$ sudo ./usp-uas-installer.sh

6. List the qcow2 image and boot_uas.py script.

[root@<POD-NAME>-ospd ~]# cd /opt/cisco/usp/bundles/uas-bundle
[root@<POD-NAME>-ospd uas-bundle]# ll
total 909572
drwxr-xr-x. 3 root root 4096 Jun 24 18:28 models
drwxr-xr-x. 2 root root 4096 Jun 24 18:28 tools
-rw-r--r--. 1 root root 649 Aug 20 2020 usp-build-info.json
-rw-r--r--. 1 root root 97 Aug 20 2020 usp-bundle-manifest.yml
-rw-r--r--. 1 root root 931367936 Aug 20 2020 usp-uas-6.9.0-9247.qcow2
-rw-r--r--. 1 root root 32 Aug 20 2020 usp-uas-6.9.0-9247.qcow2.md5
-rw-r--r--. 1 root root 40 Aug 20 2020 usp-uas-6.9.0-9247.qcow2.sha1
-rw-r--r--. 1 root root 128 Aug 20 2020 usp-uas-6.9.0-9247.qcow2.sha512
[root@<POD-NAME>-ospd uas-bundle]# cd tools/
[root@<POD-NAME>-ospd tools]# ll
total 100
-rwxr-xr-x. 1 root root 102209 Aug 20 2020 boot_uas.py
[root@<POD-NAME>-ospd tools]#

[root@<POD-NAME>-ospd tools]# ll /opt/cisco/usp/uas-installer/images/
total 909544
-rw-r--r--. 1 root root 931367936 Aug 20 2020 usp-uas-6.9.0-9247.qcow2

7. Extract the usp bundle with the use of this step if the previously mentioned files are not available.

(undercloud) [stack@<POD-NAME>-ospd ~]$ sudo su -
Last login: Tue Sep 7 02:20:36 UTC 2021 from 10.255.143.5 on pts/0
[root@<POD-NAME>-ospd ~]# ll /home/stack/usp-6_9_8/usp-6_9_8-mount/repo/
total 4142608
-r--r--r--. 1 root root 623 Sep 1 2020 rel.gpg
-r--r--r--. 1 root root 87783720 Sep 1 2020 usp-auto-it-bundle-5.8.0-1.x86_64.rpm
-r--r--r--. 1 root root 1008975328 Sep 1 2020 usp-em-bundle-6.9.0-1.x86_64.rpm
-r--r--r--. 1 root root 1168 Sep 1 2020 USP_RPM_CODE_REL_KEY-CCO_RELEASE.cer
-r--r--r--. 1 root root 918264637 Sep 1 2020 usp-uas-bundle-6.9.0-1.x86_64.rpm
-r--r--r--. 1 root root 886391928 Sep 1 2020 usp-ugp-bundle-21.15.47-1.x86_64.rpm
-r--r--r--. 1 root root 1340535896 Sep 1 2020 usp-vnfm-bundle-4.5.0.112-1.x86_64.rpm
-r--r--r--. 1 root root 74725 Sep 1 2020 usp-yang-bundle-1.0.0-1.x86_64.rpm
[root@<POD-NAME>-ospd ~]#

[root@<POD-NAME>-ospd tools]# cd /

[root@<POD-NAME>-ospd /]# rpm2cpio /home/stack/usp-6_9_8/usp-6_9_8-mount/repo/usp-uas-bundle-6.9.0-1.x86_64.rpm | cpio -idmv

8. Delete the UAME deployment and then re-deploy

To Delete

./boot_uas.py --autovnf --delete 1624559350-098061


The deployment ID will be available as mentioned highlighted above or in the file mentioned “/home/stack/UAME_$(date +"%Y%m%d-%H%M").log”. Refer the latest file.


(undercloud) [stack@<POD-NAME>-ospd ~]$ cat UAME_0624Jun061624559462.log
2021-06-24 18:30:30,392 - Deployment: 1624559350-098061 instantiated successfully

1. Deploy the UAME.

(undercloud) [stack@<POD-NAME>-ospd ~]$ sudo -s
[root@<POD-NAME>-ospd stack]# source *core
(<POD-NAME>) (<POD-NAME>) [root@<POD-NAME>-ospd stack]# cd /opt/cisco/usp/uas-installer/scripts 
(<POD-NAME>) (<POD-NAME>) [root@<POD-NAME>-ospd scripts]# ./boot_uas.py --openstack --uame --image /opt/cisco/usp/uas-installer/images/usp-uas-6.9.0-9247.qcow2 --flavor <PODNAME>-UAME-FLAVOR --net <PODNAME>-MGMT-NW --net <PODNAME>-ORCH-NW --ha --hostname <PODNAME>-UAME --ha-net <PODNAME>-MGMT-NW --password password --gateway 172.168.10.1 --floating-ip <floating ip of UAME> --external-network <PODNAME>-EXTERNAL-MGMT --admin password--oper  password --security password

Note: "password" has to be replaced with the UAME login password and "PODNAME" with the current POD. "floating-ip" should be obtained from the IP matrix


2021-06-24 18:28:52,225 - Uploading image '<POD-NAME>-UAME-usp-uas-6.9.0-9247' from '/opt/cisco/usp/uas-installer/images/usp-uas-6.9.0-9247.qcow2'
2021-06-24 18:29:06,945 - Uploaded image '<POD-NAME>-UAME-usp-uas-6.9.0-9247' successfully
2021-06-24 18:29:09,987 - Creating Server Group to enforce anti-affinity
2021-06-24 18:29:10,098 - Deployment started with transaction id --- 1624559350-098061
2021-06-24 18:29:11,766 - Created HA VIP, IP: 172.168.20.40
2021-06-24 18:29:17,125 - Allocating/Associating floating-ip
2021-06-24 18:29:17,125 - Acquire Lock : floating_ip
2021-06-24 18:29:17,125 - Lock floating_ip acquired
2021-06-24 18:29:20,474 - Assigned floating IP '10.250.100.198' to IP '172.168.20.40'
2021-06-24 18:29:20,475 - Released lock: floating_ip
2021-06-24 18:29:26,206 - Server: <POD-NAME>-UAME instantiated, waiting for server to be active
2021-06-24 18:30:01,415 - Server: <POD-NAME>-UAME instantiated, waiting for server to be active
2021-06-24 18:30:30,392 - Deployment: 1624559350-098061 instantiated successfully
2021-06-24 18:30:30,393 -
+--------------------------------------------------------------------+
Deployment ID | Instances
----------------------------------------------------------------------
1624559350-098061 | e71616e8-bf01-4561-bdd6-4e3bf3ed1d5e
VIP: 172.168.20.40 | eth0: 172.168.10.6/24
Floating IP: 10.250.100.198 | eth1: 172.168.11.23/24
|
| 3d956097-16b1-4909-b539-c6a90e01678c
| eth0: 172.168.10.18/24
| eth1: 172.168.11.8/24
+--------------------------------------------------------------------+

2. Save deployment id manually in the file /home/stack/uame_(date).log.

grep -i "deployment:" /var/log/autovnf.log | tail -1 >> /home/stack/UAME_$(date +"%Y%m%d-%H%M").log

3. Check if the UAME is active and running.


(<POD-NAME>) [stack@<POD-NAME>-ospd ~]$ nova list | grep -i uame
| e71616e8-bf01-4561-bdd6-4e3bf3ed1d5e | <POD-NAME>-UAME-1 | ACTIVE | - | Running | <POD-NAME>-MGMT-NW=172.168.10.6; <POD-NAME>-ORCH-NW=172.168.11.23 |
| 3d956097-16b1-4909-b539-c6a90e01678c | <POD-NAME>-UAME-2 | ACTIVE | - | Running | <POD-NAME>-MGMT-NW=172.168.10.18; <POD-NAME>-ORCH-NW=172.168.11.8 |

4. Login into both the UAME and perform these checks.

[root@adi-tmo Downloads]# ssh ubuntu@10.250.100.198
ubuntu@10.250.100.198's password: 
Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-187-generic x86_64)

* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage

Cisco Ultra Services Platform (USP) 
Build Date: Thu Aug 20 09:11:07 EDT 2020
Description: UAS build assemble-uas#9247
sha1: 557151c
ubuntu@<POD-NAME>-uame-1:~$

ubuntu@<POD-NAME>-uame-1:~$ sudo su -
root@<POD-NAME>-uame-1:~# confd_cli -u admin -C
Enter Password for 'admin': 
elcome to the ConfD CLI
admin connected from 127.0.0.1 using console on <POD-NAME>-uame-1
<POD-NAME>-uame-1#

<POD-NAME>-uame-1#show uas
uas version 6.9.0
uas state active
uas external-connection-point 172.168.20.40
INSTANCE IP STATE ROLE 
------------------------------------
172.168.10.6 alive CONFD-MASTER 
172.168.10.18 alive CONFD-SLAVE

NAME LAST HEARTBEAT 
----------------------------------------------------------------
AutoVNF-MASTER 2021-09-07 05:11:03 
ESCHeartBeatMonitor-<POD-NAME>-VNF-NEPCF300 2021-09-07 05:11:26 
ESCHeartBeatMonitor-<POD-NAME>-VNF-NEPGW300 2021-09-07 05:11:22 
USPCFMWorker 2021-09-07 05:11:06 
USPCHBWorker 2021-09-07 05:11:06 
USPCWorker 2021-09-07 05:11:02

<POD-NAME>-uame-1#

Revision History

Revision	Publish Date	Comments
1.0	20-Sep-2021	Initial Release

Contributed by Cisco Engineers

Adhitian Arathi
Cisco TAC Engineer

Was this Document Helpful?

Feedback

Contact Cisco

Open a Support Case
(Requires a Cisco Service Contract)

Deploy/Redeploy UAME in Case of a Corruption

Available Languages

Download Options

Bias-Free Language

Contents

Introduction

Prerequisites

Requirements

Components used

Background Information

UAME

Problem

Procedure for the Re-deployment

To Delete

Revision History

Contributed by Cisco Engineers

Was this Document Helpful?

Contact Cisco

This Document Applies to These Products