PDF(85.8 KB) View with Adobe Reader on a variety of devices
ePub(86.3 KB) View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle)(76.7 KB) View on Kindle device or Kindle app on multiple devices
Updated:December 9, 2020
Document ID:216509
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
The information in this document is based on these software and hardware versions:
USP 6.9.0
UAME
ESC: 4.5.0(112
StarOS : 21.15.28 (74825)
Cloud - Openstack 13 (Queens)
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
Ultra Automation and Monitoring Engine (UAME)
The UAME is a new Ultra Automation Services (UAS) software module introduced to:
Support the integrated deployment of 4G or 5G Virtualized Network Functions (VNFs) and 5G Cloud-Native Network Functions (CNFs).
Reduce the number of Virtual Machines (VMs) typically required by USP and UAS, replacing the UEM, AutoIT, AutoDeploy, and AutoVNF components.
The UAME provides deployment orchestration for the following:
4GVNFs :
CUPS-based VNFs: The UAME works with the Virtual Network Function Manager (VNFM) to deploy VPC SI-based control plane (CP) and user plane (UP) VNFs in support of the Control and User Plane Separation (CUPS) architecture.
Non-CUPS-based VNFs: The UAME is backward-compatible, working with the VNFM to provide deployment support for non-CUPS 4G gateways (based on VPC-DI) and the 4G Policy and Charging Rules Function (PCRF).
5G NFs:
VNF-based NFs:The UAME works with the VNFM to deploy VPC-SI-based Network Functions (NFs).
Cloud-native-based NFs:The UAME interacts with the VNFM to deploy the Ultra Cloud Core Subscriber Microservices Infrastructure (SMI). The SMI then works with the VNFM to deploy NFs in a VM-based Kubernetes (also known as K8s) cluster.
Elastic Services Controller (ESC)
ESC is the VNFM mentioned in this article which is the only supported platform currently.
Problem
VMs hosting cloud-native 5G SMI VMs are in ERROR state in ESC.
Check for VM status in UAME and ESC. start the recovery process from ESC. If ESC is unable to recover the VM, proceed with re-deploy from UAME.
Step 1. Check the VM status from UAME.
Log in to the UAME, navigate to the confd cli, and check the state as shown here.
ubuntu@crucs502-uame-1:~$ /opt/cisco/usp/uas/confd-6.3.8/bin/confd_cli -u admin -C Enter Password for 'admin': elcome to the ConfD CLI admin connected from 10.249.80.137 using ssh on crucs502-uame-1 crucs502-uame-1#
crucs502-uame-1#show vnfr state VNFR ID STATE --------------------------------- crucs502-4g-CRPCF504 alive crucs502-4g-CRPCF505 alive crucs502-4g-CRPCF506 alive crucs502-4g-CRPCF507 error crucs502-4g-CRPCF604 alive crucs502-cnat-cnat error
Note: Recovery might take up to 900s (15 min) to complete.
bootup_time 300 recovery_wait_time 600
Log in to master ESC, check the health, and then execute the recovery commands as shown here.
Last login: Wed May 13 02:07:42 2020 from 10.x.x.x
####################################################################
# ESC on crucs502-esc-vnf-esc-core-esc-1 is in MASTER state.
####################################################################
[admin@crucs502-esc-vnf-esc-core-esc-1 ~]$ health.sh
============== ESC HA (MASTER) with DRBD =================
vimmanager (pgid 14643) is running
monitor (pgid 14712) is running
mona (pgid 14768) is running
drbd (pgid 0) is master
snmp is disabled at startup
etsi is disabled at startup
pgsql (pgid 15119) is running
keepalived (pgid 14070) is running
portal is disabled at startup
confd (pgid 15016) is running
filesystem (pgid 0) is running
escmanager (pgid 15254) is running
=======================================
ESC HEALTH PASSED
/opt/cisco/esc/esc-confd/esc-cli/esc_nc_cli recovery-vm-action DO crucs502-cnat-cn_oam1_0_d7f90c1e-4401-4be9-87f6-f39ecf04ea3a
Check the yangesc logs (tail -50f /var/log/esc/yangesc.log) and look for Status and Recovery as shown above. If it is successful, navigate to the confd cli and verify.
admin connected from 10.249.80.137 using ssh on crucs502-esc-vnf-esc-core-esc-1
crucs502-esc-vnf-esc-core-esc-1# show esc_datamodel opdata tenants tenant | select deployments state_machine NAME DEPLOYMENT NAME STATE VM NAME STATE ------------------------------------------------------------------------------------------------------------------------------------------- <trucated output> crucs502-cnat-cn_etcd2_0_7263c87c-ee62-4b81-8e1e-a0f5c463a5b5 VM_ALIVE_STATE crucs502-cnat-cn_etcd3_0_512ef3c0-96a2-4a10-83b0-4c7d13805856 VM_ALIVE_STATE crucs502-cnat-cn_master_0_05487525-c86f-47e1-a07e-fd33720d114f VM_ALIVE_STATE crucs502-cnat-cn_master_0_8cf66daa-9dfe-4c7e-817e-36624f9c98c2 VM_ALIVE_STATE crucs502-cnat-cn_master_0_dff4ad36-7982-4131-a737-ccb6c8eae348 VM_ALIVE_STATE crucs502-cnat-cn_oam1_0_d7f90c1e-4401-4be9-87f6-f39ecf04ea3a VM_ALIVE_STATE
Successful Recovery
When ESC shows VM_ALIVE_STATE, verify the status in UAME
In ESC, the recovery method is reboot only. This shows that the VM could not be brought back with a reboot, Need to redeploy.
crucs502-esc-vnf-esc-core-esc-1# show running-config | include recovery_policy recovery_policy recovery_type AUTO recovery_policy action_on_recovery REBOOT_ONLY recovery_policy max_retries 1
Important Note: Redeploy comes with day0 config. Day1 config needs to be loaded separately.
ubuntu@crucs502-uame-1:~$ /opt/cisco/usp/uas/confd-6.3.8/bin/confd_cli -u admin -C Enter Password for 'admin': elcome to the ConfD CLI admin connected from 10.249.80.137 using ssh on crucs502-uame-1 crucs502-uame-1#
crucs502-uame-1# crucs502-uame-1#show vnfr state VNFR ID STATE --------------------------------- crucs502-4g-CRPCF504 alive crucs502-4g-CRPCF505 alive crucs502-4g-CRPCF506 alive crucs502-4g-CRPCF507 error crucs502-4g-CRPCF604 alive