The cnAAA software deploy or in-service procedure utilizes the K8s rolling strategy to update the pod images. In K8s rolling update strategy, the pods of a StatefulSet updates sequentially to ensure that the ongoing process remains unaffected. Initially, a rolling update on a StatefulSet causes a single pod instance to terminate. This process continues until all the replicas of the StatefulSet are updated. The terminating pods exit gracefully after completing all the ongoing processes.

Note	Run the SMI sync operation for the cnAAA Ops Center and Cloud Native Common Execution Environment (CN-CEE). For more information, see the SMI Cluster Manager - Deployment guide.

The following figure illustrates a cnAAA rolling update for cnAAA RADIUS endpoint pods (two replicas) on Protocol worker nodes along with cnAAA Service pods (three replicas) on Service worker nodes.

The prerequisites for upgrading cnAAA are:

• All the nodes – including all the pods in the node – are up and running.

• A patch version of the cnAAA software.

Note	Currently, major versions do not support the rolling upgrade. The major version represents the release year, release number, and maintenance number. Cisco follows the versioning format as YYYY.RN.MN such as 2025.02.0.

Important

Trigger rolling upgrade only when the CPU usage of the nodes is less than 50%.

Perform a health check to ensure that all the services are running and nodes are in ready state. To perform a health check:

1. Log in to master node.

2. Use these commands to view pod and node configurations:

   • View pods in the SMI namespace:

     kubectl get pods -n smi

   • View Kubernetes nodes:

     kubectl get nodes

   • View pods in all namespaces with wide output:

     kubectl get pod --all-namespaces -o wide

   • View pods in the CPC namespace with wide output:

     kubectl get pods -n <cpc namespace> -o wide

   • View pods in the CEE namespace with wide output:

     kubectl get pods -n <cpc namespace> -o wide

   • List helm releases:

     helm list

   • Count all pods across all namespaces:

     kubectl get pods -A | wc -l

     Note	If any pod is not in a running state, verify the status of the nodes using this command: kubectl get nodes --show-labels Start the cnAAA Ops-Center with 100% functionality before starting traffic on it.

Important

Ensure that all the nodes are in the ready state before you proceed further. Use the kubectl get nodes command to display the node states.

Check cnAAA Ops-Center Status

Use this command to view the status of the cnAAA Ops-Center:

• Use this command to view the system status:

  [m13-cnaaa/m13] cpc# show system

Sample output:

Tue Apr  15 13:13:03.130 UTC+00:00
system uuid 6301d096-a1d7-442d-8b6b-9577143dd47f
system status deployed true
system status percent-ready 100.0
system ops-center repository https://charts.<Master_VIP>.nip.io/cpc.2025.02.0.i64
system ops-center-debug status true
system synch running true
system synch pending false

Verify non-running Pods

Use this command to list all pods and filter for those not in a running status.

kubectl get pods -A | grep -iv running

Sample output:

NAMESPACE         NAME                                                              READY   STATUS             RESTARTS         AGE
<namespace>           crd-api-<namespace>-engine-app-production-rjio-7689c94786-kvd26   1/2     CrashLoopBackOff   17 (4m28s ago)   76m

This section describes the procedure involved creating a backup configuration, logs, and deployment files. To back up the files:

1. Log in to the SMI Cluster Manager Node as an ubuntu user.

2. Create a new directory for deployment.

   Example:

   test@smicpc-cm01:~$ mkdir -p "temp_$(date +'%m%d%Y_T%H%M')" && cd "$_"

3. Move all the cnAAA deployment file into the newly created deployment directory.

4. Untar the cnAAA deployment file.

   Example:

   test@smi1cpc01-cm01:~/temp_08072019_T1651$ tar -xzvf cpc.2025.01.SPA.tgZ
   ./
   ./cpc_REL_KEY-CCO_RELEASE.cer
   ./cisco_x509_verify_release.py
   ./cpc.2020.01.0-1.tar
   ./cpc.2020.01.0-1.tar.signature.SPA
   ./cpc.2020.01.0-1.tar.SPA.README
   

5. Verify the downloaded image.

   Example:

   test@smi1cnAAA01-cm01:~/temp_08072019_T1651$ cat cpc.2025.01.SPA.tgZ

   Important

   Follow the procedure mentioned in the SPA.README file to verify the build before proceeding to the next step.

This section describes the procedure involved in creating a backup of SVN, Policy, and CRD data. To perform a backup of SVN and Policy files:

1. Log in to the master node as an ubuntu user.

2. Use the following command to retrieve the Policy Builder URL.

   kubectl get ing -n $( kubectl get namespaces | grep -oP 'cnAAA-(\d+|\w+)' | cut -d\  -f1) | grep policy-builder | awk '{ print $2 }'
   pb.cnAAA-02-cnAAA-engine-app-blv02.ipv4address.nip.io 
   

   Example:

   cloud-user@m13-cnaaa-master-2:~$ kubectl get ing -n $( kubectl get namespaces | grep -oP 'pcf-(\d+|\w+)') |grep policy-builder | awk '{ print $3 }'
   pb.pcf-m13-pcf-engine-app-production-rjio.10.84.16.218.nip.io
   cloud-user@m13-cnaaa-master-2:~$
   
   

   Sample output:

   pb.cnAAA-02-cnAAA-engine-app-blv02.ipv4address.nip.io
   

3. Navigate to the Policy Builder home page.

4. Click Import/Export.

5. Click All Data.

   • Export URL—Specify the export URL.

   • Export File Prefix—Specify an appropriate name for the export file.

6. Click Export.

   Important

   You can find the exported file in your local Downloads directory.

To perform a backup of CRD data:

1. Navigate to the Policy Builder Home page.

2. Click Custom Reference Data.

3. Click Import/Export CRD data.

4. Click Export.

   Important

   You can find the CRD data in your Web browser's Downloads directory.

This section describes the procedures involved in upgrading cnAAA.

• Staging a new cnAAA Image

• Triggering the Rolling Software Upgrade

• Monitoring the Upgrade

This section describes the procedure involved in verifying the helm status. You need to determine whether the deployed helm chart is listed in the helm list successfully.

To determine the helm status:

1. Run the following on the master node to view the list of deployed helm charts.

   helm list -A 

   Sample output:

   cloud-user@m13-cnaaa-master-3:~$ helm ls -A| grep cpc
   cpc-m13-cnat-cps-infrastructure         cpc-m13         1               2025-10-29 06:24:22.68282772 +0000 UTC  deployed        cnat-cps-infrastructure-0.6.10-dev-cpc-2025-03-0051-250811004416-4daacd0    BUILD_2025.03.1.i54
   cpc-m13-cps-radius-ep                   cpc-m13         1               2025-10-29 06:24:22.681193924 +0000 UTC deployed        cps-radius-ep-0.6.43-dev-cpc-2025-03-0317-250812071800-2db928a              BUILD_2025.03.1.i54
   cpc-m13-network-query                   cpc-m13         1               2025-10-29 06:24:22.681469304 +0000 UTC deployed        network-query-0.5.4-dev-cpc-2025-03-0095-250716052025-833caab               BUILD_2025.03.1.i54
   cpc-m13-ops-center                      cpc-m13         10              2025-10-29 06:18:10.184343814 +0000 UTC deployed        cpc-ops-center-0.6.32-dev-cpc-2025-03-0558-250716054741-99e44f1             BUILD_2025.03.1.i54
   cpc-m13-cpc-config                      cpc-m13         1               2025-10-29 06:24:22.68490608 +0000 UTC  deployed        cpc-config-0.6.3-dev-cpc-2025-03-0030-250716051310-045be1e                  BUILD_2025.03.1.i54
   cpc-m13-cpc-dashboard                   cpc-m13         1               2025-10-29 06:24:22.686188032 +0000 UTC deployed        cpc-dashboard-0.2.17-dev-cpc-2025-03-0196-250716051353-871e51f              BUILD_2025.03.1.i54
   cpc-m13-cpc-engine-app-production-rjio  cpc-m13         1               2025-10-29 06:24:22.681441532 +0000 UTC deployed        cpc-engine-app-0.9.2-dev-cpc-2025-03-0788-250811003555-7830316              BUILD_2025.03.1.i54 
   cpc-m13-cpc-oam-app                     cpc-m13         1               2025-10-29 06:24:22.681257099 +0000 UTC deployed        cpc-oam-app-0.6.2-dev-cpc-2025-03-0022-250716051638-582c2b0                 BUILD_2025.03.1.i54
   cpc-m13-cpc-services                    cpc-m13         1               2025-10-29 06:24:22.681354868 +0000 UTC deployed        cpc-services-0.6.17-dev-cpc-2025-03-0089-250716051336-a203202               BUILD_2025.03.1.i54
   cloud-user@m13-cnaaa-master-3:~$
   

2. If the helm chart is not found, run the following in the operational mode to view the charts irrespective of their deployment status.

   show helm charts 

   Sample output:

   [m13-cnaaa/m13] cpc# show running-config helm
   Wed Oct  29 06:37:41.475 UTC+00:00
   helm default-repository base-repos
   helm repository base-repos
    url https://charts.2002-10-192-1--21.sslip.io/cpc.2025.03.1.i54
   exit
   [m13-cnaaa/m13] cpc#
   
   
   [m13-cnaaa/m13] cpc# show helm charts version
   Wed Oct  29 06:36:48.650 UTC+00:00
   CHART                    INSTANCE                                VERSION
   --------------------------------------------------------------------------------------
   cpc-ops-center           cpc-m13-ops-center                      BUILD_2025.03.1.i54
   cnat-cps-infrastructure  cpc-m13-cnat-cps-infrastructure         BUILD_2025.03.1.i54
   cps-radius-ep            cpc-m13-cps-radius-ep                   BUILD_2025.03.1.i54
   etcd-cluster             cpc-m13-etcd-cluster                    BUILD_2025.03.1.i54
   network-query            cpc-m13-network-query                   BUILD_2025.03.1.i54
   ngn-datastore            cpc-m13-ngn-datastore                   BUILD_2025.03.1.i54
   cpc-config               cpc-m13-cpc-config                      BUILD_2025.03.1.i54
   cpc-dashboard            cpc-m13-cpc-dashboard                   BUILD_2025.03.1.i54
   cpc-engine-app           cpc-m13-cpc-engine-app-production-rjio  BUILD_2025.03.1.i54
   cpc-oam-app              cpc-m13-cpc-oam-app                     BUILD_2025.03.1.i54
   cpc-services             cpc-m13-cpc-services                    BUILD_2025.03.1.i54
   
   [m13-cnaaa/m13] cpc# show helm charts status
   Wed Oct  29 06:36:54.252 UTC+00:00
   CHART                    INSTANCE                                STATUS
   ---------------------------------------------------------------------------
   cpc-ops-center           cpc-m13-ops-center                      deployed
   cnat-cps-infrastructure  cpc-m13-cnat-cps-infrastructure         deployed
   cps-radius-ep            cpc-m13-cps-radius-ep                   deployed
   etcd-cluster             cpc-m13-etcd-cluster                    deployed
   network-query            cpc-m13-network-query                   deployed
   ngn-datastore            cpc-m13-ngn-datastore                   deployed
   cpc-config               cpc-m13-cpc-config                      deployed
   cpc-dashboard            cpc-m13-cpc-dashboard                   deployed
   cpc-engine-app           cpc-m13-cpc-engine-app-production-rjio  deployed
   cpc-oam-app              cpc-m13-cpc-oam-app                     deployed
   cpc-services             cpc-m13-cpc-services                    deployed
   [m13-cnaaa/m13] cpc#

Follow these steps to determine the pod and container status after upgrading cnAAA. Ensure that all pods and containers are up and running.

• Use this command to view the cnAAA pod logs:

  kubectl describe pod pod_name -n namespace 

• Use this command to list all pods and filter for those not in a Running status.

  kubectl get pods -A | grep -v Running

Note	If the Status column displays the state as Running, and the Ready column has the same number of containers on both sides of the forward-slash (/), then the pod is healthy and operational.

The upgrade can be rolled back if issues are encountered during the upgrade process. This section describes the procedure for rolling back the upgrade..