This document describes the issue of SMF NF PODs that does not come up after the Day 1 configuration is loaded in SMF ops-center.
Cisco recommends that you have knowledge of these topics:
Subscriber Microservices Infrastructure (SMI)
The information in this document is based on these software and hardware versions:
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
In the customer setup, they have two SMF NF that runs with the same version. Both these SMF NFs were upgraded to the latest version last night. Before the upgrade, both the NF were having PODs in running state. The issue is seen only with one SMF i.e. SMF-IMS. The other POD SMF-DATA is upgraded and has all PODs in running state.
SMF version before upgrade: smf.2020.01.0-12
SMF version after upgrade: smf.2020.01.0-18
Session Management Function
Common Execution Environment
It is the smallest possible unit in the Kubernetes environment i.e. at least one container.
IP Multimedia Subsystem
Subscriber Microservices Infrastructure
Cluster Sync shows Deployment Successful.
Kubernetes Master shows the PODS in running state with Day zero configuration.
When the Day-1 config is loaded, the new PODS does not come up.
Inside SMF ops-center, you would see the helm chards in the deleted state.
Change system mode runs to shut down and vice versa did not help.
Add a new day-1 configuration also did not help.
SMF-IMS NF shows the PODs with Day-0 configuration.
Ops-center is allowing us to log in.
CEE ops-center is up and running.
SMF-DATA ops-center is up and running with day-1 config - This one is the another NF with working PODs.
~ubuntu@crucs501-cnat-cnat-core-master1:~$ kubectl get pods -n smf-ims
As per the deployments call flow, it is SMI Deployer who takes care of the extraction of the images for PODs from the package that is stored in it.
Normally, the downloaded Software package of SMF are stored local directory, from which SMI deployer extracts and shift them under this directory: /data/software/packages/</strong >
If the list of packages available under this directory is checked, you can see all the older packages as well available in it along with the new package list.
ubuntu@xxxxx501-cnat-smi-cm-core-cm1:/data/software/packages$ ls -lrt
drwxrwxr-x 3 root root 4096 Mar 23 13:15 sample
drwxrwxr-x 3 root root 4096 Mar 24 05:48 smf.2020.01.0-12 >>> Older version of SMF
drwxrwxr-x 3 root root 4096 Mar 24 05:48 cee.2020.01.0-1
drwxrwxr-x 3 root root 4096 Apr 13 19:48 smf.2020.01.0-18 >>> Newer version of SMF
drwxr-xr-x 3 root root 4096 May 4 10:10 smf.2020.02.0.i66 >>> Older version os SMF
drwxr-xr-x 3 root root 4096 May 8 12:02 cee.2020.02.0
In this output, you can see there are three different SMF packages available. Even though the correct SMF version (i.e. smf.2020.01.0-18) is defined in SMI-Deployer running configuration, still, somehow the SMI-Deployer is not able to get the correct image files for that package.
After the workaround mentioned in the Solution section is performed, the issue was resolved.
Note: Similar issue is observed with CEE PODs as well, for which similar workaround is applied that is mentioned in the Solution section.