You can deploy a Kubernetes Cluster when a Cluster Manager (HA or AIO or Inception) is already available. To deploy a Kubernetes Cluster:

1. Setup the cluster configuration.

   • The following is a sample UCS Configuration for deploying a Kubernetes Cluster:

     Important

     When upgrading to Ubuntu 22.04, use routes gateway_address instead of gateway4 gateway_address in your network configuration.

     software cnf cee 
      url <repo_url> 
      user <user_name> 
      password <password> 
      sha256 <sha256_hash> 
     exit 
     
      # associating to Bare Metal environment
     environments bare-metal 
      ucs-server 
      exit 
     
     # General cluster configuration
     clusters <cluster_name> 
       environment bare-metal 
       addons ingress bind-ip-address <bind_ip_address> 
       addons cpu-partitioner enabled 
       configuration master-virtual-ip <master_vip> 
       configuration master-virtual-ip-interface <master_vip_interface_name> #For example, eno1 
       configuration allow-insecure-registry true 
      node-defaults initial-boot default-user <username> #For example, cloud-user 
      node-defaults initial-boot default-user-ssh-public-key  
       "<SSH_Public_Key>"
      node-defaults initial-boot default-user-password <password> 
      node-defaults netplan template  
      node-defaults initial-boot netplan ethernets eno1
       dhcp4    false
       dhcp6    false
       gateway4 <gateway_ipv4address>
       nameservers search [ <domain_name> ]
       nameservers addresses [ <ipv4address>...<ipv4address> ]
      exit 
      node-defaults k8s ssh-username <username> 
      node-defaults k8s ssh-connection-private-key 
      "<SSH_Private_Key>" 
      #initial-boot section of node-defaults
        node-defaults ucs-server host initial-boot networking static-ip netmask <ipv4_address> 
        node-defaults ucs-server host initial-boot networking static-ip gateway <ipv4_address> 
        node-defaults ucs-server host initial-boot networking static-ip dns <ipv4_address> 
        node-defaults ucs-server cimc user <username> 
        node-defaults ucs-server cimc password <password> 
        node-defaults ucs-server cimc remote-management sol enabled 
        node-defaults ucs-server cimc remote-management sol baud-rate <baud_rate> 
        node-defaults ucs-server cimc remote-management sol comport <com_port>  
        node-defaults ucs-server cimc remote-management sol ssh-port <ssh_port> 
        node-defaults ucs-server cimc networking ntp enabled 
        node-defaults ucs-server cimc networking ntp servers <ntp_server_url> 
        exit 
     
      node-defaults os proxy https-proxy <proxy_server> 
      node-defaults os proxy no-proxy <proxy_servers> 
      node-defaults os ntp enabled 
      node-defaults os ntp servers <ntp_server_url> #For exmaple, ntp.esl.cisco.com node-defaults os proxy https-proxy http://proxy-wsa.esl.cisco.com:80
      exit 
       
     #node configuration
       ucs-server host initial-boot networking static-ip ipv4-address <ipv4address> 
       ucs-server cimc ip-address <ipv4address> 
       ucs-server cimc storage-adaptor create-virtual-drive true 
       exit 
      
     # control plane node configuration
     nodes <control_plane_node_name> #For example, control-plane-1 
       k8s node-type control-plane 
       k8s ssh-ip <ipv4address>  
       k8s node-labels <node_label/node_type> #For example, smi.cisco.com/oam 
       exit 
       ucs-server host initial-boot networking static-ip ipv4-address <ipv4address> 
       ucs-server cimc ip-address <ipv4address> 
       ucs-server cimc storage-adaptor create-virtual-drive true 
       exit 
      nodes <control_plane_node_name> #For example, control-plane-2 
       k8s node-type control-plane 
       k8s ssh-ip <ipv4address> 
       k8s node-labels <node_label/node_type> #For example, smi.cisco.com/oam 
       exit 
       ucs-server host initial-boot networking static-ip ipv4-address <ipv4address>  
       ucs-server cimc ip-address <ipv4address> 
       ucs-server cimc storage-adaptor create-virtual-drive true 
       exit 
      nodes <control_plane_node_name> #For example, control-plane-3 
       k8s node-type control-plane 
       k8s ssh-ip <ipv4address> \ 
       k8s node-labels <node_label/node_type>  #For example, smi.cisco.com/node-type oam 
       exit 
       ucs-server host initial-boot networking static-ip ipv4-address <ipv4address>  
       ucs-server cimc ip-address <ipv4address> 
       ucs-server cimc storage-adaptor create-virtual-drive true 
       exit 
      ops-centers cee <ops_center_name> #For example, cee 
       repository-local <repo_name> cee-2020-02-0-i04 
      exit 
     exit 
     

2. Login to the Cluster Manager CLI and enter the configuration mode

   • Add the Kubernetes Cluster configuration to deploy the Kubernetes Cluster.

     Note	A sample Kubernetes Cluster Configuration is provided here.

3. Commit the configuration.

4. Run the synchronization.

   clusters cluster_name actions sync run debug true 

5. Monitor the progress of the synchronization.

   monitor sync-logs cluster_name 

   Note	The synchronization completes after 30 minutes approximately. The time taken for synchronization is based on network speed, VM power, and so on.

The node names are added to /etc/host as part of the sync process. You can connect to the nodes using the node name from the control plane node.

The SMI Cluster Manager uses the Kernel Based Virtual Machine (KVM) to deploy the User Plane Function (UPF) on Bare Metal Servers. The subsequent sections describe the procedures involved in installing the KVM and UPF.

Provisioning a Virtual Machine involves creating and configuring a VM based on the requirements. In Bare Metal deployments, the VM is provisioned using the KVM technology. For more details on provisioning the VM using KVM, see Installing the Kernel Based Virtual Machine and User Plane VM section.

You can remove the VM through the KVM cluster from the ConfD CLI. To remove the VM, use the following configuration:

1. Login to the ConfD CLI.

2. Remove the VM using the following configuration:

   cluster <kvm_cm_name> 
   nodes <node_name> 
   no vms <vm_name> 
   commit 

3. Run the following command to trigger the cluster synchronization and remove the VM from cluster.

   clusters cluster_name actions sync run debug true 

NOTES:

• cluster <kvm_cm_name> - Specifies the KVM cluster.

• nodes <node_name> - Specifies the VM node.

• no vms <vm_name> - Removes the specified VM.

systemctl status staros_monitor 

The staros_monitor service restarts the VM in the following conditions:

• When the VM is stopped.

• When the VM is running but not responding to the ping command.

Note

For troubleshooting any issues, you can view the logs captured in the /var/log/staros_monitor.log file. For UPF VM, you can configure the failure occurrence and Ping interval through /data/upf/monitor/staros_monitor.cfg file. You must restart the staros_monitor service for the changes to take effect.

You can add a new KVM node to the existing cluster when required. To add a new KVM node, see Installing the Kernel Based Virtual Machine and User Plane VM section.

Similarly, you can remove an existing KVM node when required. To remove an existing node, see Removing the Virtual Machine section.

For Return Material Authorization (RMA) servers, you can replace the old node and replace it with a new one. When the hardware is fixed, you can synchronize the cluster and deploy the VM. For more information on replacing the Bare Metal nodes, see SMI Cluster RMA section.

The SMI Cluster Manager allows updating the UCS server firmware remotely or locally in a data center. This section describes the procedures involved in updating the UCS Server firmware using the SMI Cluster Manager.

To update the UCS firmware, use the following configurations:

1. Setup the UCS configuration.

   The following is a sample UCS configuration:

   software ucs software_version 
    url repo_url 
    sha512 file_checksum 
   exit 
   software ucs software_version 
    url repo_url 
    sha512 ssh_key 
   exit 
   software upf software_version 
    url repo_url 
    sha256 ssh_key 
   exit 
   environments ucs 
    ucs-server 
   exit 
   ... 
    nodes node_name 
     ssh-ip ipv4address 
     type kvm 
     vms node_name 
      upf networking management ip ipv4address 
     exit 
     ucs-server cimc ip-address ipv4address 
     ucs-server software software_version 
     initial-boot netplan vlans vlan_name 
      addresses ipv4address/subnet 
     exit 
    exit 
    nodes node_name 
     ssh-ip ipv4address 
     type kvm 
     vms node_name 
      upf networking management ip ipv4address 
     exit 
     ucs-server cimc ip-address ipv4address 
     initial-boot netplan vlans vlan_name 
      addresses ipv4address/subnet 
     exit 
    exit 

   Note	Cisco Integrated Management Controller (CIMC) manages the life cycle of all the components - including firmware updates - in UCS C Series servers. The UCS firmware is updated through the CIMC API.

2. Log in to SMI Cluster Manager CLI and enter the configuration mode.

   1. Add the UCS firmware configuration.

      Note	A sample UCS configuration is provided here.

   2. Commit and exit the configuration.

3. Configure the UCS firmware.

   1. To configure the firmware version for UCS servers inside a cluster, use the following configuration:

      configure 
        node-defaults 
          ucs-server software software_version 
          exit 

   2. To configure the firmware version for UCS servers at the node level, use the following configuration:

      configure 
        nodes node_name 
          ucs-server software software_version 
          exit 

   Note

   All the UCS firmware configuration parameters are applicable only for UCS based environments. You can add multiple versions of the firmware software under the software section in the UCS configuration. It is recommended to have a minimum of two versions (new and old) provisioned. You can download the UCS firmware software from https://software.cisco.com/

4. Trigger the cluster synchronization, using the following command.

   clusters cluster_name actions sync run debug true 

   Example:

   SMI-CM# clusters test actions sync run debug true
   This will run sync.  Are you sure? [no,yes] yes

5. Monitor the progress of the cluster synchronization, using the following command.

   monitor sync-logs cluster_name 

   Note

   The cluster synchronization process time depends on the number of UCS servers planned for an upgrade. The average time taken for updating a single UCS server ranges from 30-50 minutes. You can select the upgrade strategy manually using the following command: clusters cluster_name actions sync run upgrade-strategy { rolling | concurrent }

NOTES:

• node-defaults ucs-server software software_version - Specifies the default version of UCS firmware to be applied to all UCS servers inside a cluster.

• nodes node_name ucs-server software software_version - Specifies the version of UCS firmware to be applied to all UCS servers at the node level.

• clusters cluster_name - Specifies the name of the cluster.

• actions sync run - Triggers the cluster synchronization

• upgrade-strategy - Specifies the upgrade strategy. rolling specifies the rolling upgrade strategy. concurrent specifies the concurrent upgrade strategy. The default upgrade strategy is auto.

The UCS firmware upgrade process may fail for multiple reasons. The following are some of the most common reasons:

1. Hardware Issues - Faulty components, power, and cabling issues. It most often requires Return Merchandise Authorization (RMA), component re-seating or visual inspection of the hardware.

2. Software Failure - This happens when certain components fail to apply the new software version. These components will skip the upgrade process and fall back to the previous or backup version.

3. Unsupported Hardware - The upgrade fails, if the new firmware does not support some of the components.

4. Software Misconfiguration - If the inappropriate firmware package is used or an attempt to upgrade to an incorrect or unsupported firmware version, the upgrade process fails.

5. Timeout - If the upgrade takes more than 2 hours to complete, the sync process time outs and fails.

Note	The CIMC performs a list of prerequisites before proceeding with the firmware installation. When all the prerequisites are successful, the CIMC proceeds with the upgrade. There is no dry-run option available for the firmware upgrade.

The impact of the firmware upgrade failure falls into the following three categories:

1. Non-Service Impacting - The server continues to work. This is the most common behavior. The reason for the failure are software misconfiguration, software failure or a failed prerequisites verification. In such a scenario, it’s recommended to troubleshoot the failure, fix the problem and upgrade firmware again either automatically (using the same sync process) or manually. A restart of the CIMC may also reset and unblock the issue.

2. Partially Service Impacting – The server continues to work with some performance, connectivity or stability issues. An unsupported hardware, incorrect firmware version, or an issue introduced with the new firmware may have caused this. In such a scenario, it’s recommended to perform a BUG scrub or validation to ensure that the OS version, drivers, firmware and hardware combination is officially supported by Cisco. You can perform the validation here:

   https://ucshcltool.cloudapps.cisco.com/public/

3. Service Impacting – The server fails to boot or becomes inaccessible. If the firmware upgrade failure affects the CIMC access, then the server either must be sent for a RMA or physical access may be required. A hardware issue or a fatal error (needs further investigation by TAC team) might have caused this.

Note	When you are unable to identify the root cause, it is recommended to open a TAC case and involve a UCS expert to help further troubleshoot the issue.

If one or multiple servers fail to upgrade the firmware, you must follow these steps to isolate and identify the origin or root cause of the issue:

1. Verify the output of the sync process. It contains the high-level details of the root cause and exact step where of the firmware upgrade failed.

2. You can also use the CLI for the status of the last firmware for a node or multiple nodes in a cluster.

3. Based on the error, the root cause may have caused by either of the following reasons:

   1. An incorrect UCS CIMC password was provided

   2. An incorrect ISO was provided or the ISO was corrupt

   3. The CIMC failed to download the ISO image. A firewall, security-group, or Cluster-Manager (based on the deployment) may have caused this issue

   4. The CIMC failed to trigger, run, or validate the firmware upgrade

4. Access the CIMC server and verify the server console, SEL and alerts or issues for more details (if required).

5. Based on the sync logs and the CIMC data, identify the problem and its resolution. If required, perform a CIMC reboot

6. If you can identify the error and resolve the issue, re-run the upgrade either automatically (through sync process) or manually. You must have the ISO image to perform this operation

7. Open a TAC case and involve an UCS expert, if you are unable to find the root cause or resolve the issue.

SMI allows you to perform extended heath check and monitoring of the deployed UCS servers. The method is capable of communicating with supported UCS C-series servers.

The server health check is a steadfast method to check and report the server health before executing a cluster-sync operation. This check reduces the disruptions caused by cluster-sync failures.

The existing health reporting system within the REST (Redfish) API of the UCS servers is leveraged to obtain information from multiple hardware platforms.

The sync-log reports the following information:

• TASK section displays the following details:

  • Host information—Hostname, Serial Number, Model, Firmware version, HTTPS Certificate validity, and Current Faults

  • Summary information—Processors, Memory, Fans, Storage, PCI Devices, Power Supplies, and MLOM

• Current System Faults section lists the faults present on the server at sync time.

  The format of the fault entries is "Date Raised Severity (fault identifier number) Component: Description: What action is required". Note that all faults have an action.

  For example: "2024-03-06 12:11:21 Critical (F0185) DDR4_P1_E1_ECC: DIMM 9 is inoperable : Check or replace DIMM"

• Checking for recent POST events section in the log helps with debugging. The System Event Log (SEL) is used to display host reboots and uptimes.

  The POST Time column displays the time taken for the host to complete the Power On Self Test (POST). The short uptimes indicate host reboot loops, and longer uptimes indicate firmware upgrades or hardware component replacements.

• System Warnings section accumulates the data returned from the server.

  The warnings provide additional information on faults or other issues. It will also contain the current firmware recommendations to use. If the versions do not match, it will indicate future upgrades and does not necessarily require immediate action.

• Warnings in the following scenarios:

  • Any health check such as thermal health and memory check that is not OK

  • Mismatched disk sizes

  • HDD found when SSD is expected

  • Power Policy or Fan Policy is not as expected

  • The following warnings are displayed for firmware checks on M5 servers only:

    • UCS team recommends firmware versions of at least 4.3.2.230270 for M5 servers

    • If the firmware version is less than or equal to 4.1.2: Firmware version 4.0(4h) is not supported

    • If potential compatibility issues are identified with the firmware version: Firmware Version: 4.0(4h) - Potential compatibility issue

• Failures in the following scenarios when:

  • The server reports Critical health status

  • The CIMC reports Critical health status

  • The Host is powered off

  • No storage controllers are found

  • No drives are found

  • Any DIMM failure

  • The server cannot be reached for reasons such as bad IP/credentials and so on

Overriding Health Check Sync Failure

The health check is performed prior to the software download tasks to provide a quick exit if there are actionable issues. The check is non-intrusive and you can perform only read operations.

If you want to proceed with a sync knowing there are health issues with the server, you can execute the following CLI command to override the check failure:

If ignore-health is configured, then cluster sync may fail.

Configuring Server-Check Synchronization Phase

SMI supports an additional server-check synchronization phase to review the condition of the servers anytime.

To execute the server-check sync phase, use the following command in Cluster Configuration mode:

To view the sync-log report, use the following command:

For GR deployment, the node-monitor pods starts automatically. During RMA procedure, the node-monitor pod automatically shutdown the rack if multi-compute failure is detected when the node is drain and deleted.

Before starting RMA process, perform the following:

1. Switch the role for both the instance to other rack using geo switch-role role command and make sure the target rack for RMA is in STANDBY_ERROR role for both the instances.

2. Disable the node-monitor pod.

   1. Take the backup of daemonsets.

      kubectl get daemonsets node-monitor -n cn -o yaml > node-monitor.yaml

   2. Delete node-monitor pods.

      kubectl delete daemonsets node-monitor -n cn

3. Continue with RMA procedure. For more information, see the link.

4. Once RMA procedure is complete, check if the node-monitor pods are already spawned.

   kubectl get pods -n cn -o wide | grep node-monitor

   If the node-monitor pods have not started, restart them.

   kubectl create -f node-monitor.yaml

   Note	node-monitor.yaml file is same as in Step 2.a.

5. Correct the role for the instances accordingly.

Note

For both earlier and current SMI versions: If you are replacing hardware components during an RMA procedure that contain firmware, such as an mLOM card, before adding the repaired or replaced node back to the cluster, you must run the HUU (Host Upgrade Utility) to ensure that the component is compatible with the system before syncing the node back into service. As part of RMA, if you remove a node from the cluster and before you return it to the manufacturer, you must purge all data on the device as per instructions provided by the hardware vendor.

For the SMI version 2020.02.2.41 and later, the RMA (Return Merchandise Authorization) procedure for planned maintenance and any unplanned node failure events is unified for the SMI Bare Metal stacked cluster. The same procedure is applicable for both primary and worker nodes. This feature simplifies the automation and MOP requirements for both NSO and user.

Note	For information on the RMA procedures for all previous versions of SMI Bare Metal, refer to the following sections in this guide: Control Plane Bare Metal Node Failure - Unplanned Control Plane Bare Metal Node Maintenance - Planned

In SMI Bare Metal deployment, a stacked cluster is defined as a cluster containing three Bare Metal control plane nodes with a primary control plane, OAM, and other applications hosted on the same three nodes. A healthy stacked cluster needs at least two working control plane nodes to perform all the functions. When the two control plane nodes fail, the entire cluster fails and you must redeploy the entire cluster. You can run the K8s cluster synchronization from the current Active node of the Cluster Manager HA.

The subsequent sections provide information about handling the stacked cluster in the event of a Bare Metal node failure (unplanned) or planned maintenance.

For scaling the NFs, you can add a number of worker nodes along with the stacked control plane nodes. After adding the worker nodes to the stacked control plane nodes, you can distribute the NFs to the worker nodes.

This section describes procedures involved in replacing a Bare Metal worker node within a SMI Cluster in the event of node failure or planned maintenance.

For KVM Bare Metal node failures, you must replace the failed or faulty KVM Bare Metal node and run a cluster synchronization. When the cluster synchronizes successfully, the SMI Cluster Manager retains its database for Day 0 configuration. However, for Day 1 configuration, the SMI Cluster Manager retains its database from the Network Services Orchestrator (NSO).

Note	The SMI does not support any other data backup or restore methods currently.

Unlike a typical a Kubernetes cluster, the SMI Bare Metal HA consists of two AIO Bare Metal nodes running independently but synchronized in data using an underlying Distributed Replicated Block Device (DRBD) driver. The data synchronization happens in a specific manner with restrictions. Also, the DRBD utilizes the master keepalived for switchovers. You can run the synchronization for the SMI Cluster Manager HA from the Inception Server.

The subsequent sections provide more information about replacing failed nodes in SMI Bare Metal Cluster Manager HA mode.

When the SMI Cluster Manager is deployed in All-In-One (AIO) mode, the CEE replicates the data across all the three nodes (control plane, worker and etcd).

Also, the SMI Cluster Manager AIO model does not support the back up and recovery of operational data. Therefore, you must ensure the following to prevent the loss of monitoring data when a double node or triple node failure occurs:

1. Back up the CEE configuration for the Ops Center using the standard Ops Center backup and restore functionality.

2. Back up the Cluster Manager configuration for the Ops Center (remote cluster configuration) using the standard Ops Center backup and restore functionality.

3. For planned maintenance or unplanned outage in the SMI AIO mode, you can perform a cluster synchronization and restore the backed-up configuration.

4. Export the metrics (KPIs) from bulk stats regularly.

5. Gather Alerts in an external system.

This section describes the procedure to redeploy the SMI Cluster Manager HA, when both the Active and Standby nodes fail or when the entire site or cluster is down.

The CEE replicates the data across all the three nodes (control plane, worker, and etcd). Also, the SMI Cluster Manager AIO model does not support the back up and recovery of operational data. Therefore, you must ensure the following to prevent the loss of monitoring data when a double node or triple node failure occurs:

1. Back up the CEE configuration for the Ops Center using the standard Ops Center backup and restore functionality.

2. Back up the Cluster Manager configuration for the Ops Center (remote cluster configuration) using the standard Ops Center backup and restore functionality.

3. For planned maintenance or unplanned outage in the SMI AIO mode, you can perform a cluster synchronization and restore the backed-up configuration.

4. Export the metrics (KPIs) from bulk stats regularly.

5. Gather Alerts in an external system.

The SMI comes equipped with different notification streams to verify the event status (cluster or node synchronization) or synchronization state of the current cluster or nodes in it.

The show notification stream provides notification details for alert notifications, license status and system status.

show notification stream ? 

Possible completions:
  alert-notification   Notifications for alerts
  license-status       Notifications for smart agent licensing will be generated here
  system-status        Notifications for system status

The NSO must subscribe to the alert-notification notification stream to receive alerts.

You can access the notification stream using the following command show command in the SMI Cluster Manager CLI:

show notification stream {node-state | node-status | sync-state | sync-status} 

Sync-State Notification

You can access the synchronization state of a given cluster using the sync-state notification. After every cluster synchronization, a sync-state notification is registered as sent.

The following example displays the deployment of a stacked cluster:

SMI Cluster Deployer# show notification stream sync-state
notification 
 eventTime 2020-10-07T18:58:27.368+00:00
 sync-state-notification 
  cluster <stacked_cluster_name>
  state DEPLOYED
 !
!
notification 
 eventTime 2020-10-07T19:07:56.114+00:00
 sync-state-notification 
  cluster <stacked_cluster_name>
  state DEPLOYED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.531+00:00
 sync-state-notification 
  cluster <stacked_cluster_name>
  state DEPLOYED
 !
!
notification 
 eventTime 2020-10-08T15:59:56.697+00:00
 sync-state-notification 
  cluster <stacked_cluster_name>
  state DEPLOYED
 !
!
The node-state-notification could be used to assess the state of the node in a cluster deloyment sync process. Below example shows the state of control plane and worker nodes are joined in <stacked_cluster_name> cluster. Before reaching JOINED state if the join process is still running they would show up as JOINING.
[installer-controlplane] SMI Cluster Deployer# show notification stream node-state
notification 
 eventTime 2020-10-07T19:46:54.531+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker3
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.532+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane2
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.534+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker2
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.535+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane3
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.535+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane1
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.54+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker1
  state JOINED
 !
!

Node-State Notification

You can view the node state in a cluster deployment synchronization process using the node-state notification.

The following example displays the state of control plane and worker nodes that have joined the stacked cluster. The node state is displayed as JOINING , if the process is running. Otherwise, the node state is displayed as JOINED .

SMI Cluster Deployer# show notification stream node-state
notification 
 eventTime 2020-10-07T19:46:54.531+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker3
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.532+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane2
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.534+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker2
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.535+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane3
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.535+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane1
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.54+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker1
  state JOINED
 !
!

Sync-Status Notification

You view the progression of the cluster synchronization process using the sync-status notification.

The following example displays the different states (SYNC-START, ERROR and DONE) of the cluster synchronization process:

SMI Cluster Deployer# show notification stream sync-status
notification 
 eventTime 2020-10-08T16:05:53.326+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status ERROR
 !
!
notification 
 eventTime 2020-10-08T16:12:48.042+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status SYNC-START
 !
!
notification 
 eventTime 2020-10-08T16:26:23.444+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status ERROR
 !
!
notification 
 eventTime 2020-10-08T16:26:23.446+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status ERROR
 !
!
notification 
 eventTime 2020-10-08T16:31:42.551+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status SYNC-START
 !
!
notification 
 eventTime 2020-10-08T16:43:33.498+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status DONE
 !
!
notification 
 eventTime 2020-10-08T20:04:21.386+00:00
 sync-status-notification 
  cluster kali-stacked
  status SYNC-START
 !
!
notification 
 eventTime 2020-10-08T20:04:21.386+00:00
 sync-status-notification 
  cluster kali-stacked
  status SYNC-START
 !
!
notification 
 eventTime 2020-10-08T20:27:32.155+00:00
 sync-status-notification 
  cluster kali-stacked
  status DONE
 !
!

Node-Status Notification

You can view the status of the node synchronization using the node-status notification. The following example displays the status of the nodes in a stacked cluster:

SMI Cluster Deployer# show notification stream node-status
notification 
 eventTime 2020-09-25T19:59:56.728+00:00
 node-status-notification 
cluster kali-vm3
node controlplane
status DONE
 !
!
notification 
 eventTime 2020-09-29T16:36:05.962+00:00
 node-status-notification 
cluster kali
node etcd1
status DONE
 !
!
notification 
 eventTime 2020-09-29T20:54:18.415+00:00
 node-status-notification 
cluster kali-stacked
node controlplane1
status DONE
 !
!
notification 
 eventTime 2020-09-30T01:28:57.355+00:00
 node-status-notification 
cluster kali-stacked
node controlplane1
status DONE
 !
!
notification 
 eventTime 2020-09-30T01:46:53.034+00:00
 node-status-notification 
cluster kali-stacked
node controlplane1
status DONE
 !
!
notification 
 eventTime 2020-09-30T04:17:25.701+00:00
 node-status-notification 
cluster kali-stacked
node controlplane1
status ERROR
 !
!
notification 
 eventTime 2020-10-06T20:02:30.108+00:00
 node-status-notification 
cluster kali-stacked
node cmts-worker1
status DONE
 !
!

VM Status Alerts

The NSO needs to know the state of the NF (Network Function) that has been deployed and notifications for the following states:

• DEPLOYED

• ALIVE

• UNDEPLOYED

• ERROR

• RECOVERING

• RECOVERY_FAILED

Note	The SMI supports VM status notification for UPF only.

The following parameters are introduced in the notification streams for the VM status notifications:

Depending on the life cycle stage of the VM, notifications are generated from either the Cluster Manager or CEE Ops-Center and sent to the NSO. When VM gets deleted or redeployed, the UNDEPLOYED notification is sent to the Cluster Manager notification stream. All other notifications are generated by the Alert Manager and then sent to the CEE Ops-Center notification stream.

Cluster Manager Notification

The Cluster Manager sends vm-state-notification for the UNDEPLOYED VM state.

CEE Ops-Center Notification

The CEE Ops-Center alert-notification sends the following alerts for different VM states:

• vm-deployed: minor - DEPLOYED

• vm-alive: minor – ALIVE (alert lasts for a short time and disappears automatically)

• vm-error: major - ERROR

• vm-recovering: warning - RECOVERING

• vm-recovery_failed: critical - RECOVERY_FAILED

All required fields are included in alert labels for notification from alert-notification. All VM alerts are viewable on the Grafana dashboard.

VM Action Notifications

Delete Action: When delete VM action is triggered, CM sends notifications that the VM is deleted. The VM states are UNDEPLOYED and ERROR for vm delete action.

clusters abc-cluster-15 nodes kvm-1 vms upf1 actions delete

Redeploy Action: When VM is in RECOVERY_FAILED state, NSO sends a request to redeploy the VM. A redeploy action does both delete action and sync action.

clusters abc-cluster-15 nodes kvm-1 vms upf1 actions redeploy

Redeploy Action Notification: The redeploy action sends a notification to the CM. The redeploy vm action has the following 4 states: UNDEPLOYED, ERROR, REDEPLOYED, REDEPLOY_ERROR.

show notification stream vm-state 

notification
eventTime 2021-02-23T21:27:28.692+00:00
vm-state-notification
cluster_name cndp-testbed
node_name kvm-1
vm_name upf2
state UNDEPLOYED
message
!
!
notification
eventTime 2021-02-23T21:29:18.699+00:00
vm-state-notification
cluster_name cndp-testbed
node_name kvm-1
vm_name upf2
state REDEPLOYED
message
!
!

Configuring the Alert Notification in CEE

The user must configure alert notifications when they deploy the UPF VMs. Log in to the CEE cli to add the following configuration:

config
bulk-stats prune-interval-days 3
prometheus kvm-metrics defaults private-key "-----BEGIN OPENSSH PRIVATE KEY-----LGXtil23N4YV=\n-----END OPENSSH PRIVATE KEY-----\n"
prometheus kvm-metrics defaults user cloud-user
prometheus kvm-metrics monitor-server 10.194.62.41
hostname abc-bm-15-controlplane
exit

Note	The user must replace the IP, hostname, private key and user details.

Sample Notification from the Alert Notification Stream

notification
eventTime 2021-01-08T03:28:54.501+00:00
smi-alert-notification
starts-at 2021-01-08T03:28:24.493874101Z
ends-at 0001-01-01T00:00:00Z
alert-status firing
smi-alert-notification alert-label
name alertname
value vm-recovery-failed
!
smi-alert-notification alert-label
name cluster
value test-cee-kvm_cee-voice
!
smi-alert-notification alert-label
name hostname
value test-bm-15-controlplane
!
smi-alert-notification alert-label
name instance
value metrics-proxy-test-bm-15-controlplane:9100
!
smi-alert-notification alert-label
name job
value metrics-proxy
!
smi-alert-notification alert-label
name message
value 10.1.1.3
!
smi-alert-notification alert-label
name monitor
value prometheus
!
smi-alert-notification alert-label
name node_name
value controlplane
!
smi-alert-notification alert-label
name replica
value test-cee-kvm_cee-voice
!
smi-alert-notification alert-label
name severity
value critical
!
smi-alert-notification alert-label
name state
value RECOVERY_FAILED
!
smi-alert-notification alert-label
name vm_name
value upf2
!
smi-alert-notification alert-annotation
name summary
value upf2 failed to recover.
!
smi-alert-notification alert-annotation
name type
value Equipment Alarm
!
!
!

This section describes the sequence of operation about how to add a host-profile to the software configuration configured on the CM.

1. To use host profiles, add the .tgz file to the software configuration on the CM by using the following CLI configuration and sync.

   config 
       software 
           host-profile host-profile_name url software_url user user_name password passowrd 

   NOTES:

   • host-profile host-profile_name - Specify the name of the host profile, which is used to configure the BIOS settings for a UCS server.

   • url software_url - Specify the HTTP, HTTPS, or file URL of the software. File format must be "file:///absolute/path/to/file".

     Must be a URI in the pattern (http:|https:|file:/)//.*.

   • user user_name - Specify the user name for HTTP/HTTPS authentication.

     Must be a string.

   • password passowrd - Specify the password for downloading software package.

     Must be an aes-cfb-128-encrypted string.

   The following CLI commands form a sample configuration.

   software host-profile cndp-upf
    url             https://www.abc.com/ucs/profile/cndp-upf-profile.tgz
    user            smi-readonly.gen
    password        $8$abc
    sha256          b01dc4d47926e7a35e352c2f2d1c9f8b280fd44a89d0657281ff858387669753
   exit

2. To add the host-profile name in the cluster configuration, use the following CLI configuration.

   config 
       clusters cluster_name 
           nodes node_name 
               host-profile host-profile_name 
               commit 
               exit 

   NOTES:

   • clusters cluster_name - Specify the name used to uniquely identify the cluster. Must be an alphanumeric string, and can contain the hyphen (-) character, however it must not start with a hyphen.

     Must be a string.

   • nodes node_name - Specify the name of the node. name can be an alphanumeric string containing the hyphen (-). A host name cannot start with a hyphen (-). For example, kashaio-123.

     Must be a string.

   • host-profile host-profile_name - Specify the name of the host profile, which is used to configure the BIOS settings for a UCS server.

   The following CLI commands form a sample configuration.

   config
   clusters example-upf nodes kvm-1 host-profile cndp-upf
   commit
   top
   exit

   To return to the default settings, remove host-profile setting from the node. The following CLI commands form a sample configuration.

   config
   no clusters example-upf nodes kvm-1 host-profile
   commit        
   exit

CPU isolation utilizes a containerd Node Resource Interface (NRI) plugin (v0.3.0) that subscribes for pod or container lifecycle events.

NRI is a common framework for plugging extensions into OCI-compatible container runtimes. It provides basic mechanisms for plugins to track the state of containers and to make limited changes to their configuration.

Using the NRI Plugin

The following built-in rules apply for the plugin:

• The CPU isolator ignores containers in a pod with prefix name as "vpc-"

• If the annotation smi.cisco.com/cpuset exists, CPU isolator adjusts the CPU set using its value

• Otherwise the value of environment "CPUSET_KUBEPODS" is used.

The following steps describe how to start the NRI plugin:

• Retain the CPU set for K8s to all CPUs

• During a CreateContainer event, the plugin adjusts the container's CPU set based on the following conditions:

  • if it is a VPC container, it does nothing so that VPP workers can be pinned to isolated CPUs

  • for other non-VPC containers, it creates a customized CPU set to exclude the isolated CPU

CM calculates the CPU set of kubepods and passes it as the environment value "CPUSET_KUBEPODS" to create a static pod for CPU isolator.

To define isolcpu from host profile, the following is a sample host profile configuration:

---
profile_version: 2
profiles:
  bios:
    name: test_bios_settings
    description: "CIMC BIOS settings"
    pids:
      UCSC-C220-M5SX:
        description: "Copy of Default CIMC BIOS settings for UCSC-C220-M5SX"
        tokens:
          cpuPerformance: hpc
          cpuEnergyPerformance: balanced-performance
          eppProfile: Performance
          packageCstateLimit: C0 C1 State
          usbPortInternal: disabled
          usbPortKvm: enabled
          usbPortRear: disabled
          usbPortSdCard: disabled
  linux:
    cisco_c220_m5_dual_numa:
      # name: no longer used
      description: "Cisco M5 Dual Numa Test Profle"
      grub:
        dedicate-cpus:
          numa-0: '0%'
          numa-1: '26-39'
        hugepages:
          numa-0:
            2m: 0%
            1g: 0%
          numa-1:
            2m: 20%
            1g: 50
        addition-grub-cmd-line: "intel_iommu=on intel_pstate=disable intel_idle.max_cstate=1 pcie_aspm.policy=performance idle=poll clocksource=tsc tsc=reliable skew_tick=1 nosoftlockup"
      sysctl:
        net.ipv4.neigh.default.gc_thresh1: 8192
        net.ipv4.neigh.default.gc_thresh2: 32768
        net.ipv4.neigh.default.gc_thresh3: 65536
        net.ipv6.neigh.default.gc_thresh1: 8192
        net.ipv6.neigh.default.gc_thresh2: 32768
        net.ipv6.neigh.default.gc_thresh3: 65536
      sysfs:
        /sys/module/nvme_core/parameters/io_timeout: 4294967295
      sriov:
      - match:
          pf-name:  enp94s*
        vfs-per-pf: 4
        dpdk-vfs-per-pf: 2
        dpdk-bind: vfio-pci
      - match:
          pf-name:  enp216s*
        vfs-per-pf: 16
 
    cisco_c220_m5_single_numa:
      description: "Cisco M5 Single Numa Test Profle"
      grub:
        dedicate-cpus:
          numa-0: '26-39'
        hugepages:
          numa-0:
            2m: 20%
            1g: 0%
        # single core needs intel_iommu=off
        addition-grub-cmd-line: "intel_iommu=off intel_pstate=disable intel_idle.max_cstate=1 pcie_aspm.policy=performance idle=poll clocksource=tsc tsc=reliable skew_tick=1 nosoftlockup"
      sysctl:
        net.ipv4.neigh.default.gc_thresh1: 8192
        net.ipv4.neigh.default.gc_thresh2: 32768
        net.ipv4.neigh.default.gc_thresh3: 65536
        net.ipv6.neigh.default.gc_thresh1: 8192
        net.ipv6.neigh.default.gc_thresh2: 32768
        net.ipv6.neigh.default.gc_thresh3: 65536
      sysfs:
        /sys/module/nvme_core/parameters/io_timeout: 4294967295
      sriov:
      - match:
          pf-name:  enp94s*
        vfs-per-pf: 4
        dpdk-vfs-per-pf: 2
        dpdk-bind: vfio-pci
      - match:
          pf-name:  enp216s*
        vfs-per-pf: 16

The Single Root I/O Virtualization (SR-IOV) specification is a Peripheral Component Interconnect (PCI) passthrough hardware standard that regulates the device assignment. The PCI passthrough natively shares a single resource with multiple guests. PCI passthrough enables the virtual machines to bypass the hypervisor and virtual switch layer, and communicate directly with the PCI devices residing on the host. SR-IOV is an extension of the PCI passthrough specification. In the PCI configuration space, a single physical device that has SR-IOV capability enabled is virtualized as multiple devices. Each device is a secured and distinct unit with isolation at the resource level, which means it has its own set of storage and configurations. This mechanism provides data transfer at the near wire-speed along with low latency.

The SMI on Bare Metal platform supports both, PCF passthrough and the SR-IOV standards in the form of plugins. In the CNDP-KVM-UPF implementation, the VM and network devices are by default configured to support SR-IOV Virtual Functions (VFs). If you want to pass the network resources to network function (UPF) through the PCI passthrough standard, then enable the PCI passthrough for the network device.

This section describes how to enable a VM to support PCI Passthrough.

If the VM uses the SR-IOV mechanism, then ensure that the VF number for the associated PF and MAC address is available.

Use the following configuration to configure the PCF passthrough option on the VM:

cluster cluster_name 
   nodes node_name 
      type kvm 
         os enable-passthrough  {true|false} 
         os num-vfs-per-pf number_of_vf_per_pf 
   exit 
exit 

NOTES:

• cluster cluster_name – Specify the name of the cluster.

• nodes node_name – Specify the name of the node.

• os enable-passthrough {true|false} – Configures the PCI passthrough capability for the VM. Default value is "false".

• os num-vfs-per-pf number_of_vf_per_pf – Specifies the number of the VF for each physical function. Default value is 16.

This feature introduces strongSwan, a keying daemon, which uses the Internet Key Exchange (IKE) protocols, IKEv1 and IKEv2, to establish security associations (SA) between two peers in a network. Such an IKE session is denoted as IKE_SA in this chapter. The IKE provides strong authentication for both peers and derives unique cryptographic session keys. Besides authentication and key material, IKE also provides the means to exchange configuration information and to negotiate IPsec SAs, which are often called as CHILD_SAs. IPsec SAs define which network traffic is to be secured and how it has to be encrypted and authenticated.

The strongSwan feature is available as an add-on from the Cluster Manager (CM). Use the CM Ops-Center to configure this add-on. In the current release, the SMI uses strongSwan version 5.9.3.

SMI allows monitoring of IPSec certificates—sends certificate expiry alerts and updates certificate through strongSwan configuration.

Configuration Parameters

In this section, see the description for different configuration parameters available for the strongSwan add-on feature. Use the CM Ops-Center to configure these parameters.

• name : Specifies the name of the connection, which can be used for connection specific operations, for example, up or down.

• auto { ignore | add | route | start } : Specifies the operation, if any, that should be automatically performed at IPsec startup. The add option loads a connection without starting it, whereas route loads a connection and installs kernel traps. If traffic is detected between the leftsubnet and rightsubnet, a connection is established. The start option loads a connection and brings it up immediately. The ignore option ignores the connection and is the same as deleting a connection from the config file.

  The default value is ignore.

• keyexchange { ikev1 | ikev2 } : Specifies the method of key exchange and the protocol to use to initialize the connection.

• type { tunnel | transport | transport_proxy | passthrough | drop } : Specifies the type of the connection. Currently, the accepted values are tunnel, signifying a host-to-host, host-to-subnet, or subnet-to-subnet tunnel. The transport option signifies a host-to-host transport mode, whereas the transport_proxy option signifies the special Mobile IPv6 transport proxy mode. The passthrough option signifies that no IPsec processing should be done at all and drop signifies that packets must be discarded.

• left or right { ip address ip_address | fqdn fqdn | %any | %any4 | %any6 | range | subnet } : Specifies the IP address or FQDN of the participant public-network interface. The value %any for the local endpoint signifies an address to be filled in (by automatic keying) during negotiation. If the local peer initiates the connection setup, then the routing table is queried to determine the correct local IP address. If the local peer is responding to a connection setup, then any IP address that is assigned to a local interface is accepted. The value %any4 restricts address selection to IPv4 addresses and %any6 restricts address selection to IPv6 addresses.

• leftsubnet or rightsubnet ip subnet : Specifies the private subnet behind the left participant, expressed as either network or netmask.

• leftid or rightid id : Specifies how the left or right participant must be identified for authentication. The default values are left or right or the subject of the certificate configured. It must match the full subject DN or one of the subjectAltName extensions contained in the certificate.

• leftsendcert { never | no | ifasked | always | yes } : Defines whether a peer must send a certificate request (CR) payload in order to get a certificate in return.

• leftauth or rightauth { pubkey | psk | eap | xauth } : Specifies the authentication method to use locally (left) or require from the remote (right) side. The acceptable values are pubkey for public key encryption (RSA/ECDSA), psk for pre-shared key authentication, eap to use the Extensible Authentication Protocol, and xauth for IKEv1 eXtended Authentication.

  Pubkey is the default option.

• psk pre-shared key : Specifies the required setting if leftauth or rightauth is configured as psk.

• esp { cipher suites | aes128-sha256 } : A comma-separated list of ESP encryption or authentication algorithms is used for the connection, for example, aes128-sha256. The notation is encryption-integrity[-dhgroup][-esnmode]. For IKEv2, multiple algorithms (separated by -) of the same type can be included in a single proposal. IKEv1 only includes the first algorithm in a proposal.

  aes128-sha256 is the default option.

• ike { cipher suites | aes128-sha256-modp3072 } : A comma-separated list of IKE/ISAKMP SA encryption or authentication algorithms is used, for example, aes128-sha256-modp3072. The notation is encryption-integrity[-prf]-dhgroup. In IKEv2, multiple algorithms and proposals might be included, such as aes128-aes256-sha1-modp3072-modp2048 or 3des-sha1-md5-modp1024.

• ikelifetime { time time | 3h } : Specifies how long the keying channel of a connection (ISAKMP or IKE SA) must last before being renegotiated.

• lifetime { time time | 1h } : Specifies how long a particular instance of a connection should last, from successful negotiation to expiry.

• dpdaction { none | clear | hold | restart } : Specifies the action to be taken when dead peer is detected.

  none is the default value.

• dpddelay { time time | 30s } : Defines the period time interval with which INFORMATIONAL exchanges are sent to the peer. These are only sent if no other traffic is received.

• dpdtimeout { time time | 150s } : Defines the timeout interval after which, all the connections to a peer are deleted in case of inactivity.

• inactivity time time : Defines the timeout interval after which, a CHILD_SA is closed if it did not send or receive any traffic.

• closeaction { none | clear | hold | restart } : Defines the action to take if the remote peer unexpectedly closes a CHILD_SA (see dpdaction for the description of different options). If the peer uses reauthentication or uniqueids checking, closeaction must not be used, these events might trigger the defined action when it's not desired.

• nodes list_of_node_names : Specifies the node names on which IPSec connection must be established.

• serverCert server_certificate : Specifies the content of Server certificate in the pem format to be used for this connection.

  Note	This keyword is not supported under strongSwan configuration.

• serverPrivKey server_private_key : Specifies the content of server private key in the pem format to be used for this connection.

  Note	This keyword is not supported under strongSwan configuration.

• serverPrivKeyPassphrase passphrase : Specifies the passphrase used to encrypt the server-priv-key value.

• server-secret : Pass an existing TLS secret for this connection.

This section describes how to install the strongSwan feature.

Install strongSwan as an Add-on from the CM

Use the following steps to install strongSwan as an add-on from the CM Ops-Center:

1. Use the following CLI commands to enable the strongSwan add-on:

   clusters cluster_name addons strongswan enabled

2. Set all the strongSwan parameters for connection (refer to the Configuration Parameters section for more details on available parameters).

3. Trigger the cluster sync operation.

   Note	The strongSwan pods run on all the nodes, however traffic is accepted only on those nodes, which are configured by using the "nodes" parameter in the CM Ops-Center. strongSwan does not accept or send any traffic on non-configured nodes.

Configuring IPSec Certificates

To configure IPSec certificates under strongSwan configuration, use the following procedure:

1. Create TLS associated secret for server and CA certificate.

   Note: Create strongSwan-related secrets inside the smi-strongswan namespace.

   Example:

   [test-cm-controlplane] SMI Cluster Deployer# show running-config clusters secrets ca-cert
   clusters test-aio
    secrets ca-cert smi-strongswan 134-ca
     certificate "-----BEGIN CERTIFICATE-----\nMIIDqzCzQubm..................1Ac1L+s4M3ug==\n-----END CERTIFICATE-----\n"
    exit
    secrets ca-cert smi-strongswan 135-ca
     certificate "-----BEGIN CERTIFICATE-----\nMIIFqzCCA5Og..................9XdMDiQANHgf7w\n-----END CERTIFICATE-----\n"
    exit
    secrets ca-cert smi-strongswan ca-1
     certificate "-----BEGIN CERTIFICATE-----\nMIID0TCCArmg..................UNvF0nAmIX0qxg4\n-----END CERTIFICATE-----\n"
    exit
    secrets ca-cert smi-strongswan ca-2
     certificate "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADAN..................tbNDzGAnF29nus=\n-----END PRIVATE KEY-----\n"
    exit
   exit

2. Refer the secrets in strongSwan configuration. The strongSwan configuration shows the available TLS and certificates.

   Example:

   [test-cm-controlplane] SMI Cluster Deployer# show running-config clusters karan-aio strongswan connections server-secret
   clusters test-aio
    strongswan connections a-to-b
     server-secret a-to-b
    exit
   exit
    
   [test-cm-controlplane] SMI Cluster Deployer# show running-config clusters karan-aio strongswan ca-certs
   clusters test-aio
   strongswan ca-certs [ 134-ca 135-ca ]
   exit

To deploy the product, perform the following:

1. Use the following configuration to install the product

   configure 
     software cnf software_name 
       url HTTP_HTTPS_File_URL 
       user username 
       password password 
       sha256 sha256_hash 
       exit 

2. Link the CEE into the desired cluster in the ops-centers section.

   configure 
     clusters cluster_name ops-center app_name instance_name 
     repository-local cnf_repo 
     exit 

3. Download the TAR ball from the URL.

   software-packages download URL 

   Example:

   SMI Cluster Manager# software-packages download http://<ipv4address>:<port_number>/packages/cee-2019-08-21.tar 
   

4. Verify whether the TAR balls are loaded.

   software-packages list 

   Example:

   SMI Cluster Manager# software-packages list 
   [ cee-2019-08-21 ] 
   [ sample ] 
   

5. Configure the necessary Ops Center parameters in the required cluster to deploy the product.

   configure 
      cluster cluster_name 
         ops-centers app_name instance_name 
            repository url 
            netconf-ip ipv4_address 
            netconf-port port 
            ssh-ip ipv4_address 
            ssh-port port 
            ingress-hostname  <ipv4_address>.<customer_specific_domain_name> 
            initial-boot-parameters use-volume-claims true/false 
            initial-boot-parameters first-boot-password password 
            initial-boot-parameters auto-deploy true/false 
            initial-boot-parameters single-node true/false 
            initial-boot-parameters image-pull-secrets 
            exit 
      exit 
   

   Example:

   SMI Cluster Manager# config 
   SMI Cluster Manager(config)# clusters test2 
   SMI Cluster Manager(config-clusters-test2)# ops-centers cee data 
   SMI Cluster Manager(config-ops-centers-cee/data)# repository http://charts.<ipv4address>.<domain_name>/cee-2019-08-21/ 
   SMI Cluster Manager(config-ops-centers-cee/data)# initial-boot-parameters use-volume-claims false 
   SMI Cluster Manager(config-ops-centers-cee/data)# initial-boot-parameters first-boot-password Cisco@123 
   SMI Cluster Manager(config-ops-centers-cee/data)# initial-boot-parameters auto-deploy true 
   SMI Cluster Manager(config-ops-centers-cee/data)# initial-boot-parameters single-node false 
   SMI Cluster Manager(config-ops-centers-cee/data)# exit 
   SMI Cluster Manager(config-clusters-test2)# exit 
   SMI Cluster Manager(config)# 
   

6. Configure the secrets, if your local registry contains secrets.

   configure 
      cluster cluster_name 
         secrets docker-registry secret_name 
            docker-server server_name 
            docker-username username 
            docker-password password 
            docker-email email 
            namespace k8s namespace 
            commit 
            exit 
         exit 
   

   Example:

   SMI Cluster Manager# config 
   SMI Cluster Manager(config)# clusters test2 
   SMI Cluster Manager(config-clusters-test2)# secrets docker-registry sec1 
   SMI Cluster Manager(config-docker-registry-sec1)# docker-server serv1 
   SMI Cluster Manager(config-docker-registry-sec1)# docker-username user1 
   SMI Cluster Manager(config-docker-registry-sec1)# docker-password Cisco@123 
   SMI Cluster Manager(config-docker-registry-sec1)# docker-email reg@cisco.com 
   SMI Cluster Manager(config-docker-registry-sec1)# namespace ns1 
   SMI Cluster Manager(config-docker-registry-sec1)# exit 
   SMI Cluster Manager(config-clusters-test2)# exit 
   SMI Cluster Manager(config)# 
   

7. Run the cluster synchronization.

   clusters cluster_name actions sync run 
   

   Example:

   SMI Cluster Manager# clusters test2 actions sync run 
   

Notes:

• software-packages download url – Specifies the software packages to be downloaded through HTTP/HTTPS.

• software-packages list – Specifies the list of available software packages.

• ops-centers app_name instance_name – Specifies the product Ops Center and instance. app_name is the application name. instance_name is the name of the instance.

• repository url - Specifies the local registry URL for downloading the charts.

• netconf-ip ipv4_address – Specifies the Ops Center netconf IPv4 address.

• netconf-port port – Specifies the Ops Center netconf port number.

• ssh-ip ipv4_address – Specifies the SSH IPv4 address for the Ops Center.

• ssh-port port - Specifies the SSH port number for the Ops Center.

• ingress-hostname <ipv4_address>.<customer_specific_domain_name> – Specifies the ingress hostname to be set to the Ops Center. <customer_specific_domain_name> specifies the customer's domain name.

• initial-boot-parameters – Specifies the initial boot parameters for deploying the helm charts.

  • use-volume-claims true/false – Specifies the usage of persistent volumes. Set this option to True to use persistent volumes. The default value is true.

  • first-boot-password password – Specifies the first boot password for the product's Ops Center.

  • auto-deploy true/false – Auto deploys all the services of the product. Set this option to false to deploy only the product's Ops Center.

  • single-node true/false – Specifies the product deployment on a single node. Set this option to false for multi node deployments.

  • image-pull-secrets – Specifies the docker registry secret name to be used.

• secrets docker-registry secret_name – Specifies the secret name for your docker registry.

  • docker-server server_name – Specifies the docker server name.

  • docker-username username – Specifies the docker registry user name.

  • docker-password password – Specifies the docker registry password.

  • docker-email email – Specifies the docker registry email.

  • namespace namespace – Specifies the docker registry namespace.

NOTES:

• software cnf software_name - Specifies the Cisco's Cloud Native software. software_name is the name of the Cloud Native software.

  • url HTTP_HTTPS_File_URL - Specifies the repository URL.

  • user username - Specifies the username for HTTP/HTTPS authentication.

  • password password - Specifies the password used for downloading the software package.

  • sha256 sha256_hash - Specifies the SHA256 hash of the software download.

To deploy the product, perform the following:

1. Login to the SMI Cluster Manager CLI using the ingress URL.

   
   https://cli.smi-cluster-manager.<IP_address>.<customer_specific_domain_name> 
   

   NOTES:

   • customer_specific_domain_name - Specifies the customer's domain name.

2. Use the following configuration to install the CEE

   configure 
     software cnf software_name 
       url HTTP_HTTPS_File_URL 
       user username 
       password password 
       sha256 sha256_hash 
       exit 

   NOTES:

   • software cnf software_name - Specifies the Cisco's Cloud Native software. software_name is the name of the Cloud Native software.

     • url HTTP_HTTPS_File_URL - Specifies the repository URL.

     • user username - Specifies the username for HTTP/HTTPS authentication.

     • password password - Specifies the password used for downloading the software package.

     • sha256 sha256_hash - Specifies the SHA256 hash of the software download.

3. Link the product (CEE or Network Functions) into the desired cluster in the ops-centers .

   configure 
     clusters cluster_name ops-center app_name instance_name 
     repository-local cnf_repo 
     exit 

   NOTES:

   • repository-local cnf_repo - Specifies the CNF repository.

4. Run the cluster synchronization to deploy the CEE Ops Center and wait for the synchronization to complete.

   clusters cluster_name actions sync run 

   NOTES:

   • clusters cluster_name actions sync run - Synchronizes the committed changes to the cluster.

5. Verify the cluster synchronization through cluster sync status or log commands.

   clusters cluster_name actions sync status 
   clusters cluster_name actions sync logs 

   NOTES:

   • clusters cluster_name actions sync status - Displays the status of the cluster synchronization.

   • clusters cluster_name actions sync logs - Displays the logs generated during the cluster synchronization process.

You can verify the status of the product deployment through the product CLI. To verify, use the following commands:

1. Log in to the SMI product CLI. For example, CEE.

2. Verify whether the charts are loaded in the specific instance (verify the namespace).

   show helm charts 
   

   Example:

   cee# show helm charts 
   CHART         INSTANCE                          STATUS    VERSION     REVISION RELEASE                                   NAMESPACE 
   ------------------------------------------------------------------------------------------------------------------------------------------- 
   cee-ops-center         cee-global-ops-center             deployed  2023.02.1.d249  1   0.7.0-2023-02-1-0513-230331051211-dec612f  cee-global 
   cnat-monitoring        cee-global-cnat-monitoring        deployed  2023.02.1.d249  1   0.7.0-2023-02-1-0031-230331183330-58ec41c  cee-global 
   product-documentation  cee-global-product-documentation  deployed  2023.02.1.d249  1   0.8.0-2023-02-1-0131-230321085503-2699cb5  cee-global 
   pv-manager             cee-global-pv-manager             deployed  2023.02.1.d249  1   0.3.0-2023-02-1-0029-230320155437-e484272  cee-global 
   smi-autoheal           cee-global-smi-autoheal           deployed  2023.02.1.d249  1   0.2.0-2023-02-1-0030-230330084451-99684bf  cee-global 
   smi-show-tac           cee-global-smi-show-tac           deployed  2023.02.1.d249  1   0.4.0-2023-02-1-0189-230331050005-81130f1  cee-global 
   storage-provisioner    cee-global-storage-provisioner    deployed  2023.02.1.d249  1   0.3.0-2023-02-1-0120-230320160505-1597fdb  cee-global 
   telegraf-monitoring    cee-global-telegraf-monitoring    deployed  2023.02.1.d249  1   0.1.0-2023-02-1-0048-230330084426-9b02da0  cee-global 

3. Verify the status of the system.

   show system status 
   

   Example:

   cee# show system status 
   system status deployed true 
   system status percent-ready 100.0 
   

Notes:

• show helm charts – Displays the helm release details.

• show system status – Displays the status of the system.

SMI supports the sequential helm chart deployment feature in the base ops-center through op-center configuration. NFs should use the specific ops-center branch and a minimum version of 7.71400.7 as the ops-center-yang-compiler.

To deploy the helm charts in a sequential manner, use the following steps:

1. Before upgrading NF product through cluster-sync, disable the config sync at NF ops-center.

   [tb2-smi-blr-s1-c1/global] cee# system synch stop 
   Wed Mar  27 04:05:54.301 UTC+00:00
   [tb2-smi-blr-s1-c1/global] cee#
   Message from confd-api-manager at 2024-03-27 04:05:57...
   Synchronization process is not running - config changes are not processed.  Enable synch with: system synch start
   [tb2-smi-blr-s1-c1/global] cee# 

   Note	You are required to perform config sync stop or start operation only when the sequential deployment feature is not available in the existing ops-center. the helm charts are added or deleted in the new NF software.

2. Upgrade the NF software through cluster sync. Note that the cluster sync will upgrade only ops-center. Rest of the charts will not be upgraded due to config sync stop.

   [tb2-smi-blr-s1-c1/global] cee# show helm charts

   Wed Mar  27 04:08:14.528 UTC+00:00

   CHART          INSTANCE         STATUS    VERSION   REVISION  RELEASE      NAMESPACE
   ------------------------------------------------------------------------------------------------------
   cee-ops-center    cee-global-ops-center      deployed  2024.02.1.d181 10  0.7.0-2024-02-1 cee-global
   cnat-monitoring   cee-global-cnat-monitoring deployed  2024.02.1.i05  9   0.7.0-2024-02-1 cee-global
   product-documentation cee-global-product-documentation deployed 2024.02.1.i05  9  0.8.0-2024-02-1 cee-global
   pv-manager        cee-global-pv-manager      deployed  2024.02.1.i05  10  0.3.0-2024-02-1 cee-global
   smi-autoheal      cee-global-smi-autoheal    deployed  2024.02.1.i05  10  0.2.0-2024-02-1 cee-global
   smi-show-tac      cee-global-smi-show-tac    deployed  2024.02.1.i05  9   0.4.0-2024-02-1 cee-global
   storage-provisioner cee-global-storage-provisioner deployed 2024.02.1.i05  9  0.3.0-2024-02-1 cee-global
   telegraf-monitoring cee-global-telegraf-monitoring deployed 2024.02.1.i05  9  0.1.0-2024-02-1 cee-global

3. Configure the parameters that are used to sequentialize the deployment of helm charts.

   [tb2-smi-blr-s1-c1/global] cee(config)# helm ordered_deployment
   Possible completions:
   chart                Chart specific deployment configuration
   continue_on_timeout  Continue/Stop after Chart bringup timeout
   enable               Enable/Disable Ordered Helm Chart Deployment On Upgrade
   timeout              Wait interval (in seconds) for Chart bringup (all pods Ready) 

   [tb2-smi-blr-s1-c1/global] cee(config)# helm ordered_deployment enable true
   Wed Mar  27 04:23:10.852 UTC+00:00 

   [tb2-smi-blr-s1-c1/global] cee(config)# helm ordered_deployment chart
   Possible completions:
   cnat-monitoring kvm-metrics      product-documentation      pv-manager      smi-autoheal      smi-logs-forwarder      smi-show-tac      storage-provisioner      telegraf-monitoring 

   [tb2-smi-blr-s1-c1/global] cee(config)# helm ordered_deployment chart cnat-monitoring priority_group 20
   Wed Mar  27 04:24:10.852 UTC+00:00
   [tb2-smi-blr-s1-c1/global] cee(config)# helm ordered_deployment chart storage-provisioner priority_group 10
    

   NOTES:

   • helm ordered_deployment enable { false | true } : Enable or disable the sequential deployment of helm charts. Default: false

   • helm ordered_deployment chart chart_name priority_group priority_id : Specify the chart name and the chart priority group ID. The priority group ID decides the helm chart deployment order. The priority group configuration is specific to the helm charts and not the sub-charts.

     The priority_id is an integer ranging from 1 to 99 with 1 being the highest priority and 99 being the lowest priority. Default: 99

     The chart assigned the priority group ID of 1 gets deployed first before the other charts assigned the priority group ID of 2,3,4, and so on. Note that the other charts will not be deployed until all the pods associated with the first chart are deployed. Charts with the same priority group will be deployed simultaneously.

     Note	You must use helm ordered_deployment enable true command before configuring helm ordered_deployment chart name priority_group priority_id command.

     If the ordered_deployment is not enabled or if the priority_group is not defined for charts, NF follows the default deployment that is bringing up all the charts in parallel.

   • helm ordered_deployment continue_on_timeout : Set this configuration to true or false to continue or stop the helm chart deployment after the chart bring-up timeout. Default: true

   • helm ordered_deployment timeout time_interval : Specify the timeout interval, in seconds, until then the chart waits for all its pods to get deployed.

     The time_interval is an integer ranging from 0 to 3600. Default: 600

     If any failure is encountered in higher priority group chart deployment, the deployment stops or continues depending on the continue_after_timeout parameter.

   • If the sequential deployment feature is not available in any of the ops-centers, enable this feature and perform the NF software upgrade.

   • If there are any new helm charts available through the NF software upgrade, update the ordered deployment configuration parameters for the new charts. Similarly, if there are any charts to be removed as part of software upgrade, make sure to first remove the chart from ordered-deployment configuration and then proceed with the software upgrade through cluster sync.

4. Enable the config sync. It will upgrade the chart as per the ordered deployment configuration.

   [tb2-smi-blr-s1-c1/global] cee# system synch start
   Wed Mar  27 04:23:10.852 UTC+00:00
   [tb2-smi-blr-s1-c1/global] cee# 

   [tb2-smi-blr-s1-c1/global] cee# show helm charts

   Wed Mar  27 04:24:54.336 UTC+00:00

   CHART          INSTANCE         STATUS    VERSION   REVISION  RELEASE      NAMESPACE
   ------------------------------------------------------------------------------------------------------
   cee-ops-center    cee-global-ops-center      deployed  2024.02.1.d181 10  0.7.0-2024-02-1 cee-global
   cnat-monitoring   cee-global-cnat-monitoring deployed  2024.02.1.i05  9   0.7.0-2024-02-1 cee-global
   product-documentation cee-global-product-documentation deployed 2024.02.1.i05  9  0.8.0-2024-02-1 cee-global
   pv-manager        cee-global-pv-manager      deployed  2024.02.1.i05  10  0.3.0-2024-02-1 cee-global
   smi-autoheal      cee-global-smi-autoheal    deployed  2024.02.1.i05  10  0.2.0-2024-02-1 cee-global
   smi-show-tac      cee-global-smi-show-tac    deployed  2024.02.1.i05  9   0.4.0-2024-02-1 cee-global
   storage-provisioner cee-global-storage-provisioner deployed 2024.02.1.i05  9  0.3.0-2024-02-1 cee-global
   telegraf-monitoring cee-global-telegraf-monitoring deployed 2024.02.1.i05  9  0.1.0-2024-02-1 cee-global

   [tb2-smi-blr-s1-c1/global] cee#

   cloud-user@tb2-smi-blr-s1-c1-master1:~$ helm ls -n cee-global

   NAME         NAMESPACE     REVISION  UPDATED               STATUS     CHART                           APP VERSION
   cee-global-cnat-monitoring  cee-global   10  2024-03-27 04:24:17.178670318 +0000 UTC   deployed  cnat-monitoring-0.7.0-2024-02-1-0032-240321173925-2c8a968   2024.02.1.d181
   cee-global-ops-center       cee-global   10  2024-03-27 04:07:39.542317449 +0000 UTC   deployed  cee-ops-center-0.7.0-2024-02-1-0547-240326113917-f8d2d25    2024.02.1.d181
   cee-global-product-documentation cee-global  10  2024-03-27 04:24:33.537482349 +0000 UTC  deployed product-documentation-0.8.0-2024-02-1-0149-240216180844-dd04f43 2024.02.1.d181
   cee-global-pv-manager       cee-global  11    2024-03-27 04:24:33.491414622 +0000 UTC   deployed  pv-manager-0.3.0-2024-02-1-0036-240216180715-0677f38       2024.02.1.d181
   cee-global-smi-autoheal     cee-global  11    2024-03-27 04:24:33.495848154 +0000 UTC   deployed  smi-autoheal-0.2.0-2024-02-1-0040-240216180748-13dd903     2024.02.1.d181
   cee-global-smi-show-tac     cee-global  10    2024-03-27 04:24:33.550657024 +0000 UTC   deployed  smi-show-tac-0.4.0-2024-02-1-0207-240321173848-84f417c     2024.02.1.d181
   cee-global-storage-provisioner cee-global  10 2024-03-27 04:24:11.514250682 +0000 UTC   deployed  storage-provisioner-0.3.0-2024-02-1-0127-240216180827-d0e9159 2024.02.1.d181
   cee-global-telegraf-monitoring cee-global  10 2024-03-27 04:24:33.494447625 +0000 UTC   deployed  telegraf-monitoring-0.1.0-2024-02-1-0064-240216180755-97cf3c1 2024

   See the updated timestamp to confirm the sequential deployment of charts.

To upgrade the product, perform the following:

1. Download the latest TAR ball from the URL.

   software-packages download URL 

   Example:

   SMI Cluster Manager# software-packages download http://<ipv4address>:<port_number>/packages/cee-2019-08-21.tar 
   

2. Verify whether the TAR balls are loaded.

   software-packages list 

   Example:

   SMI Cluster Manager# software-packages list 
   [ cee-2019-08-21 ] 
   [ sample ] 
   

3. Update the repository URL to point the correct product chart release to upgrade the product.

   configure 
      cluster cluster_name 
         ops-centers app_name instance_name 
            repository url 
            exit 
         exit 
   

   Example:

   SMI Cluster Manager# config 
   SMI Cluster Manager(config)# clusters test2 
   SMI Cluster Manager(config-clusters-test2)# ops-centers cee data 
   SMI Cluster Manager(config-ops-centers-cee/data)# repository http://charts.<ipv4address>.<domain_name>/cee-2019-08-21/ 
   SMI Cluster Manager(config-ops-centers-cee/data)# exit 
   SMI Cluster Manager(config-clusters-test2)# exit 
   

4. Configure the secrets, if your local registry contains secrets.

   configure 
      cluster cluster_name 
         secrets docker-registry secret_name 
            docker-server server_name 
            docker-username username 
            docker-password password 
            docker-email email 
            namespace k8s namespace 
            commit 
            exit 
         exit 
   

   Example:

   SMI Cluster Manager# config 
   SMI Cluster Manager(config)# clusters test2 
   SMI Cluster Manager(config-clusters-test2)# secrets docker-registry sec1 
   SMI Cluster Manager(config-docker-registry-sec1)# docker-server serv1 
   SMI Cluster Manager(config-docker-registry-sec1)# docker-username user1 
   SMI Cluster Manager(config-docker-registry-sec1)# docker-password Cisco@123 
   SMI Cluster Manager(config-docker-registry-sec1)# docker-email reg@cisco.com 
   SMI Cluster Manager(config-docker-registry-sec1)# namespace ns1 
   SMI Cluster Manager(config-docker-registry-sec1)# exit 
   SMI Cluster Manager(config-clusters-test2)# exit 
   SMI Cluster Manager(config)# 
   

5. Run the cluster synchronization.

   clusters cluster_name actions sync run 
   

   Example:

   SMI Cluster Manager# clusters test2 actions sync run 
   

You can verify the status of the product upgrade through the product CLI. To verify, use the following commands:

1. Log in to the SMI product CLI. For example, CEE.

2. Verify whether the charts are loaded in the specific instance (verify the namespace).

   show helm charts 
   

   Example:

   cee# show helm charts 
   CHART         INSTANCE                          STATUS    VERSION     REVISION RELEASE                                   NAMESPACE 
   ------------------------------------------------------------------------------------------------------------------------------------------- 
   cee-ops-center         cee-global-ops-center             deployed  2023.02.1.d249  1   0.7.0-2023-02-1-0513-230331051211-dec612f  cee-global 
   cnat-monitoring        cee-global-cnat-monitoring        deployed  2023.02.1.d249  1   0.7.0-2023-02-1-0031-230331183330-58ec41c  cee-global 
   product-documentation  cee-global-product-documentation  deployed  2023.02.1.d249  1   0.8.0-2023-02-1-0131-230321085503-2699cb5  cee-global 
   pv-manager             cee-global-pv-manager             deployed  2023.02.1.d249  1   0.3.0-2023-02-1-0029-230320155437-e484272  cee-global 
   smi-autoheal           cee-global-smi-autoheal           deployed  2023.02.1.d249  1   0.2.0-2023-02-1-0030-230330084451-99684bf  cee-global 
   smi-show-tac           cee-global-smi-show-tac           deployed  2023.02.1.d249  1   0.4.0-2023-02-1-0189-230331050005-81130f1  cee-global 
   storage-provisioner    cee-global-storage-provisioner    deployed  2023.02.1.d249  1   0.3.0-2023-02-1-0120-230320160505-1597fdb  cee-global 
   telegraf-monitoring    cee-global-telegraf-monitoring    deployed  2023.02.1.d249  1   0.1.0-2023-02-1-0048-230330084426-9b02da0  cee-global 

3. Verify the status of the system.

   show system status 
   

   Example:

   cee# show system status 
   system status deployed true 
   system status percent-ready 100.0 
   

Notes:

• show helm charts – Displays the helm release details.

• show system status – Displays the status of the system.