Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Problem

Analysis

Solution

Related Information

Introduction

This document describes problem and provides solution regards to SMF system sync fail.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Session Management Function (SMF) fails to start the service, when issue occurs there is raised alert on Common Execution Environment (CEE).

Problem

The SMF-RCDN continues to cycle through Ops Center System Upgrade and then fail.

On CEE you see this alert:

[smf-rcdn/cee-rcdn] cee# show alerts active summary | inc ops

ops-system-sync-runni 687ca7b9266c minor 09-07T17:59:36 smf-rcdn-mas ops center system upgrade for smf-rcdn is in progress

ops-latest-sync-faile 31531915bf54 major 09-07T10:52:26 smf-rcdn-mas ops center latest system sync for smf-rcdn failed

On SMF you see this error:

[smf-rcdn/smf-rcdn] smf#

Message from confd-api-manager at 2022-09-07 17:49:32...

Helm update is STARTING. Trigger for update is STARTUP.

[smf-rcdn/smf-rcdn] smf#

Message from confd-api-manager at 2022-09-07 17:49:51...

Helm update is ERROR. Trigger for update is STARTUP. Message is:

InterruptedException: one or multiple helm chart installations failed

javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error

at com.broadhop.confd.config.proxy.dao.HelmRepositoryDAO.sendConfiguration(HelmRepositoryDAO.java:272)

at com.broadhop.confd.config.proxy.service.ConfigurationSynchManager.run(ConfigurationSynchManager.java:233)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.InterruptedException: one or multiple helm chart installations failed

at com.broadhop.confd.config.proxy.dao.HelmRepositoryDAO.sendConfiguration(HelmRepositoryDAO.java:266)

... 8 more

Analysis

To troubleshoot this, you have to look inside logs from SMF ops center pod.

In this scenario, smf-rcdn did not start the smf related pods.

cloud-user@smf-rcdn-master-1:~$ kubectl get pods -n smf-rcdn

NAME READY STATUS RESTARTS AGE

documentation-69768456cb-klq8d 1/1 Running 0 102d

ops-center-smf-rcdn-ops-center-85899d6b90-9kx6h 5/5 Running 1 40m

smart-agent-smf-rcdn-ops-center-6b9cd64f85-8f8cz 1/1 Running 0 22h

cloud-user@smf-rcdn-master-1:~$

Note down the name of the ops center pod and collect logs for the container confd-api-bridge.

cloud-user@smf-rcdn-master-1:~$ kubectl logs ops-center-smf-rcdn-ops-center-85899d6b90-9kx6h -n smf-rcdn -c confd-api-bridge

Preparing upgrade logic for helm ...

Inside the logs is the reason why system failed to start. In this example, the issue occurred due to sgw-service configuration. The profile does not have configured interfaces.

WARN [2022-09-13 19:44:55,860] com.broadhop.confd.config.proxy.dao.helm.ReleaseInstallCallable: [436] Install or upgrade failure for chart: sgw-service,

release-name: smf-rcdn-sgw-service, command: [/usr/local/bin/helm, upgrade, smf-rcdn-sgw-service, /tmp/chart1014799367411807494.tgz,

--install, -f, /tmp/override1205042274924409625.yaml, -f, /tmp/values4318819924777544020.yaml, --namespace, smf-rcdn, --dry-run]

WARN [2022-09-13 19:44:55,860] com.broadhop.confd.config.proxy.dao.helm.ReleaseInstallCallable: Command result:

Release "smf-rcdn-sgw-service" does not exist. Installing it now.

Error: template: sgw-service/templates/sgw-service.yaml:14:30: executing "sgw-service/templates/sgw-service.yaml" at

<$endpoint.service.nodeCount>: nil pointer evaluating interface {}.nodeCount

INFO [2022-09-13 19:44:55,860] com.broadhop.confd.config.proxy.dao.helm.ReleaseInstallCallable: Command result:

Release "smf-rcdn-udp-proxy" does not exist. Installing it now.

NAME: smf-rcdn-udp-proxy

LAST DEPLOYED: Tue Sep 13 19:44:55 2022

NAMESPACE: smf-rcdn

STATUS: pending-install

REVISION: 1

TEST SUITE: None

HOOKS:

MANIFEST:

On SMF, check show running-configuration. This configuration contains the profile for sgw-service but mandatory parameters are not defined.

profile smf smfprof

mode offline

locality LOC1

allowed-nssai [ slice1 ]

instances 1 fqdn xxx

instances 2 fqdn xxx

plmn-list mcc 123 mnc 456

exit

service name nsmf-pdu

type pdu-session

schema http

version 1.0.2

http-endpoint base-url http://smf-service

icmpv6-profile icmpprf1

compliance-profile June19

priority 20

access-profile idft

subscriber-policy polsub

exit

exit

profile sgw cn-sgw

exit

Solution

The solution is to remove the configuration mistake.

Related Information

- Cisco Technical Support & Downloads