Troubleshoot Standby UPF Does Not Get NSO Config Push

Introduction

This document describes RCM  based UPF unable to get the config push(host/ecs/apn) due to an issue in NSO side 

Problem

Upon RCM (Redundancy Configuration Manager) initiated reboot of UPF (User Plane Function), once RCM receives configpushcomplete from UPF, RCM sends notification message to NSO(Network Services Orchestrator), to proceed to next step - to push host,ecs,apn config files in that order.

But in NSO issue is identified when the timestamp of the RCM sent notification is getting parsed.

# NSO does not proceed for config push to UPF when timestamp of the notification is in the format (Observe microsecond field NOT present)

netconf-notifications received-notifications notification 2024-01-25T04:59:30+00:00 0

//Sample notification - does not work on NSO

netconf-notifications received-notifications notification 2024-01-25T04:59:30+00:00 0

user          <NSO username>

subscription  alert-notification

stream        alert-notification

received-time 2024-01-25T04:59:30.042866+00:00

data smi-alert-notification starts-at ""

data smi-alert-notification ends-at ""

data smi-alert-notification alert-status Config-Push

NAME           VALUE          

-------------------------------

DeviceID                      

Event          Config-Push    

GroupID        2              

ManagementIP   <IPv4 address>  

NameSpace      <rcm-namespace>   

RCMVIP         <IPv4 address> 

RecoveryState                 

SSHIP          <IPv4 address>  

UPFState       STANDBY        

# NSO proceeds for configpush when timestamp of the notification is in the format(Observe microsecond field is present)

netconf-notifications received-notifications notification 2024-01-25T04:43:30.045+00:00 0

//Sample notification - working on NSO

netconf-notifications received-notifications notification 2024-01-25T04:43:30.045+00:00 0

user          <NSO username>

subscription  alert-notification

stream        alert-notification

received-time 2024-01-25T04:43:30.087872+00:00

data smi-alert-notification starts-at ""

data smi-alert-notification ends-at ""

data smi-alert-notification alert-status Config-Push

NAME           VALUE          

-------------------------------

DeviceID                      

Event          Config-Push    

GroupID        2              

ManagementIP  <IPv4 address>  

NameSpace      <rcm-namespace>   

RCMVIP        <IPv4 address>

RecoveryState                 

SSHIP          <IPv4 address>  

UPFState       STANDBY 

Note : NSO notification mesages can be requested from production environment from the NSO logs.

Solution

Currently RCM has a timer of 100 minutes wait time after which it reboots the stuck UPF, waiting for config push from NSO.

NSO side fix is coming via Cisco bug ID CSCwi70899