Before you perform health check, ensure that all the services are running and the nodes are in ready state. To perform health check, log on to master node and use the following configuration:

kubectl get pods -n smi 
kubectl get nodes 
kubectl get pod --all-namespaces -o wide 
kubectl get pods -n smf-wsp -o wide 
kubectl get pods -n cee-wsp -o wide 
kubectl get pods -n smi-vips -o wide 
helm list 
kubectl get pods -A | wc -l 

This section describes the procedure for creating a backup configuration, logs, and deployment files. To backup the files:

1. Log on to the SMI Cluster Manager Node as an ubuntu user.

2. Create a new directory for deployment.

   Example:

   test@smismf-cm01:~$ mkdir -p "temp_$(date +'%m%d%Y_T%H%M')" && cd "$_"

3. Move all the working files into the newly created deployment directory.

4. Untar the smf deployment file.

   Example:

   test@smi1smf01-cm01:~/temp_08072019_T1651$ tar -xzvf smf.2020.01.0-1.SPA.tgz
   ./
   ./smf_REL_KEY-CCO_RELEASE.cer
   ./cisco_x509_verify_release.py
   ./smf.2020.01.0-1.tar
   ./smf.2020.01.0-1.tar.signature.SPA
   ./smf.2020.01.0-1.tar.SPA.README
   

5. Verify the downloaded image.

   Example:

   test@smi1smf01-cm01:~/temp_08072019_T1651$ cat smf.2020.01.0-1.tar.SPA.README

   Important

   Follow the procedure mentioned in the SPA.README file to verify the build before proceeding to the Back Up Ops Center Configuration section.

This section describes the procedure for creating a backup of the Ops Center configurations.

To perform a backup of the Ops Center configurations, use the following steps:

1. Log on to SMI Cluster Manager node as an ubuntu user.

2. Run the following command to backup the SMI Ops Center configuration to /home/ubuntu/smiops.backup file.

   ssh -p  port_number admin@$(kubectl get svc -n smi | grep '.*netconf.*<port_number>' | awk '{ print $4 }') "show run | nomore" > smiops.backup_$(date +'%m%d%Y_T%H%M') 

3. Run the following command to backup the CEE Ops Center configuration to /home/ubuntu/ceeops.backup file.

   ssh admin@<cee-vip> "show run | nomore" > ceeops.backup_$(date +'%m%d%Y_T%H%M') 

4. Run the following command to backup the SMF Ops Center configuration to /home/ubuntu/smfops.backup file.

   ssh admin@<smf-vip> "show run | nomore" > smfops.backup_$(date +'%m%d%Y_T%H%M') 

This section describes the procedure to create a backup of CEE and Ops Center configuration from the master node.

To perform a backup of CEE and Ops Center configuration, , use the following steps:

1. Log in to the master node as an ubuntu user.

2. Create a directory to backup the configuration files.

   mkdir backups_$(date +'%m%d%Y_T%H%M') && cd "$_"

3. Backup the SMF Ops Center configuration and verify the line count of the backup files.

   ssh -p port_number admin@$(kubectl get svc -n $(kubectl get namespaces | grep -oP 'smf-(\d+|\w+)') | grepport_number | awk '{ print $3 }') "show run | nomore" > smfops.backup_$(date +'%m%d%Y_T%H%M') && wc -l smfops.backup_$(date +'%m%d%Y_T%H%M')

   Example:

   ubuntu@posmf-mas01:~/backups_09182019_T2141$ ssh -p 2024 admin@$(kubectl get svc -n $(kubectl get namespaces | grep -oP 'smf-(\d+|\w+)') | grep <port_number> | awk '{ print $3 }') "show run | nomore" > smfops.backup_$(date +'%m%d%Y_T%H%M') && wc -l smfops.backup_$(date +'%m%d%Y_T%H%M')
   admin@<ipv4address>'s password: smf-OPS-PASSWORD
   334 smfops.backup
   

4. Backup the CEE Ops Center configuration and verify the line count of the backup files.

   ssh -p port_number admin@$(kubectl get svc -n $(kubectl get namespaces | grep -oP 'cee-(\d+|\w+)') | grep port_number | awk '{ print $3 }') "show run | nomore" > ceeops.backup_$(date +'%m%d%Y_T%H%M') && wc -l ceeops.backup_$(date +'%m%d%Y_T%H%M')

   Example:

   ubuntu@posmf-mas01:~/backups_09182019_T2141$ ssh -p <port_number> admin@$(kubectl get svc -n $(kubectl get namespaces | grep -oP 'cee-(\d+|\w+)') | grep <port_number> | awk '{ print $3 }') "show run | nomore" > ceeops.backup_$(date +'%m%d%Y_T%H%M') && wc -l ceeops.backup_$(date +'%m%d%Y_T%H%M')
   admin@<ipv4address>'s password: CEE-OPS-PASSWORD
   233 ceeops.backup
   

5. Move the SMI Ops Center backup file from the SMI Cluster Manager to the backup directory.

   scp $(grep cm01 /etc/hosts | awk '{ print $1 }'):/home/ubuntu/smiops.backup_$(date +'%m%d%Y_T%H%M') .

   Example:

   ubuntu@posmf-mas01:~/backups_09182019_T2141$ scp $(grep cm01 /etc/hosts | awk '{ print $1 }'):/home/ubuntu/smiops.backup_$(date +'%m%d%Y_T%H%M') .
   ubuntu@<ipv4address>'s password: SMI-CM-PASSWORD
   smiops.backup                                                       100% 9346    22.3MB/s   00:00                                                                            
   

6. Verify the line count of the backup files.

   Example:

   ubuntu@posmf-mas01:~/backups_09182019_T2141$ wc -l *
     233 ceeops.backup
     334 smfops.backup
     361 smiops.backup
     928 total
   

This section describes the procedure for staging a new SMF image before initiating the upgrade.

To stage the new SMF image:

1. Download and verify the new SMF image.

2. Log in to the SMI Cluster Manager node as an ubuntu user.

3. Copy the images to Uploads directory.

   sudo mv smf_new_image.tar /data/software/uploads 

   Note	The SMI uses the new image available in the Uploads directory to upgrade.

4. Verify whether the image is picked up by the SMI for processing from the Uploads directory.

   sleep 30; ls /data/software/uploads 

   Example:

   ubuntu@posmf-cm01:~/temp_08072019_T1651$ sleep 30; ls /data/software/uploads
   ubuntu@posmf-cm01:~/temp_08072019_T1651$
   

5. Verify whether the images were successfully picked up and processed.

   Example:

   auser@unknown:$ sudo du -sh /data/software/packages/*
   1.6G	/data/software/packages/cee.2019.07
   5.3G	/data/software/packages/smf.2019.08-04
   16K	/data/software/packages/sample
   

   The SMI must extract the images into the packages directory to complete the staging.

Peer pods are made aware of the upgrade or restart of a particular pod so that the transactions from/to that pod are handled gracefully.

The following section describes the handling of incoming and outgoing messages on a service pod:

Incoming Messages

• During an upgrade, the state of a service pod is communicated to other pods through the topology update. If no affinity exists for the session, the pod is not selected for forwarding new messages.

• For a session, if a service pod doesn't have a context, which is the first message for a user after a cache timeout, an Application Stop message is communicated to the protocol pod. This pod redirects messages to other service pod instances.

• If a service pod has a context for the session with no pending procedures or transactions for the session or user, an Application Stop message is communicated to the protocol pod. This pod redirects messages to other service pod instances. However, before communication to the protocol pod, the forceful Sync of Session State is done toward CDL. In addition, the affinity entry is removed for the session or user.

Outgoing Messages

• After completion of a call flow or a procedure for a session or user, if a service pod receives an upgrade or restart indication, then synchronization of the session or PDU state with CDL is performed. In addition, the affinity entry is removed for the service or user to disallow the triggering of further messages toward the service pod instance.

• For the existing call flows, which are in progress, the transactions are handled on the best-effort basis for call completion.

This section describes the handling of messages on the REST endpoint, GTPC endpoint, and protocol (PFCP) endpoint pods.

REST Endpoint Pod

The following section describes the handling of incoming and outgoing messages on the REST endpoint pod.

Incoming Messages

• For new TCP connections the ingress K8 service doesn't select a specific REST endpoint pod during an upgrade or restart. These requests are forwarded to other instances.

• After receiving an upgrade or restart indication, a GOAWAY frame is sent on the existing connections. By sending this frame, the new messages are sent on a new connection from the peer node.

Outgoing Messages

• After receiving the outgoing request messages from service pods, the Application Stop indication is communicated back to the service pod. With this communication, a service pod can select another REST endpoint instance to trigger the messages.

• For outgoing responses for the existing transient messages, the best effort is made to complete the transactions.

GTPC Endpoint Pod

Incoming Messages

Session level response messages cache is added at service pod to support handling of incoming request messages during GTPC endpoint pod restart. Service pods store response messages buffers based on the sequence number, source IP address, source port, and request message type.

• If there is a response message loss on wire due to GTPC endpoint pod restart, then peer retries the request message. This message is responded using a session level response message cache.

• Even if a response message isn’t generated at a service pod, the message is detected as retransmission at the service and handled accordingly.

Outgoing Messages

The outgoing request messages during GTPC endpoint pod restart are handled in the following way:

• The service pod sets the request timeout interval and the number of retransmissions while doing BGIPC. The timeout interval and the number of transmissions are based on the N3 T3 defined for S5, S11, and S5E interfaces.

• Instead of the GTPC endpoint pod, the service pod sets the source port and sequence number.

• This mechanism helps in retransmission of the outgoing message when a response message is lost during pod restart. The service pod generates the sequence number and port. In this case, there's no ambiguity of messages on the wire for the peer also to detect the message as retransmission.

Protocol (PFCP) Endpoint Pod

Incoming Messages

Session level response messages cache is added at service pod to support handling of incoming request messages during the PFCP endpoint pod restart. Service pods store response messages buffers based on the sequence number, source IP address, source port, and request message type.

• If a response message is lost on wire due to protocol endpoint pod restart, then a peer retries the request message. This message is responded using a session level response message cache.

• Even if a response message is not generated at a service pod, the message is detected as retransmission at the service and handled accordingly.

Outgoing Messages

The outgoing request messages during protocol endpoint pod restart are handled in the following way:

• The service pod sets the request timeout interval and the number of retransmissions while doing BGIPC. The timeout interval and the number of transmissions are based on the N3 T3 timer defined for the PFCP interfaces.

• Instead of the protocol endpoint pod, the service pod sets the source port and sequence number.

• This mechanism helps in retransmission of the outgoing message when a response message is lost during pod restart. The service pod generates the sequence number and port. In this case, there is no ambiguity of messages on the wire for the peer to detect the message as retransmission.

One instance of a node manager pod is available to serve the calls during upgrades. The readiness probe and the timer are also configured for the upgrade scenario. In case a pod is inactive, the service pods retry the other node manager pod instances.

CDL pods have multiple replicas for the service continuity during the upgrade process. Multiple connection streams are available towards CDL endpoints to minimize failures during upgrade and restart processes. In case of errors for these processes, the retry mechanism is also implemented towards another CDL endpoint connection.