SMI Cluster Manager - Deployment

Introduction to Deploying SMI Cluster Manager

This chapter provides information about deploying (orchestrating) and upgrading the SMI Cluster Manager in a VMware vCenter environment using Docker. The deploying process involves creating a working SMI Cluster Manager Virtual Machine (VM) used for deploying the SMI Kubernetes Clusters. For instantiating the Kubernetes Clusters on the SMI Cluster Manager, see the Ultra Cloud Core Subscriber Microservices Infrastructure Operations Guide.

VMware Requirements

This section provides information about the VMware vCenter Server requirements for deploying the SMI Cluster Manager.

The VMware vCenter Server requirements for deploying the SMI Cluster Manager includes:

  • VMware vCenter Server.

    The following version of VMware vCenter is supported:

    • VMware vSphere Hypervisor (ESXi) 6.5 and later versions. The VMware vSphere Hypervisor (ESXi) 6.5 and 6.7 has been fully tested and meets performance benchmarks.

    • The VMware vCenter Server must be ready for deploying the VMs. The setup must include:

      • Creating the Datacenter, Cluster, and Host(s).

      • Setting up the Datastore on the Solid State Drive (SSD).


        Note

        Deploying on the Hard Disk Drive (HDD) and Network Attached Storage (NAS) datastores is the best effort support. It can lead to timeouts and slow performance.

      • Network setup configuration:

        • The SMI Cluster Manager requires at least one static IPv4 address. The IPv4 address must be routable from the machine on which the Docker is being run.

        • Authentication to the VMware vCenter Server.

        • Administrative access to the VMware vCenter Server (user name and password).


          Note

          It is possible to setup a limited permission user to perform the deploy operations. Due to the complexity involved in troubleshooting permissions issues in ESXi, only best effort support is provided for non-administrative users.

    • A valid IPv4 address for the ESXi host.


      Note
      The default Docker installation does not enable IPv6, which is the cause of this limitation. This limitation will be addressed in a future release.

  • The Docker software package.


    Note
    To install docker on your laptop or an existing virtual machine (VM), see Docker software package in Prerequisites section.

    The Docker must have network access to:

    • The VMware vCenter Server.

    • The deployed SMI Cluster IP address. An SSH connection is used to configure the SMI Cluster Manager after booting it in the vCenter.

  • The SMI Deployer artifacts.

    • For information on downloading the artifacts, see Prerequisites section.

Deploying the SMI Cluster Manager on VMware vCenter

The SMI Cluster Manager VM (virtual machine) is deployed from a Virtual Machine Disk (vmdk) file. This vmdk file contains all the required airgap images to make the SMI Cluster Manager functional. For deploying, the SMI Cluster Manager also uses a Docker Deployer helper image. Moreover, the SMI Cluster Manager supports offline environments wherein all the required product charts and images reside within the local registries.

Prerequisites

The following are the prerequisites for deploying SMI Cluster Manager on a VMware environment:

  • The airgap vmdk release.


    Note
    The vmdk file (with the file name cluster-manager.vmdk) is downloaded to the root/cluster-manager-docker-deployer/example-work-directory folder.

  • The Docker Deployer image release.

    Example:

    docker pull 
<docker_deployer_image_directory>

  • The Docker software package (required for running the Docker Deployer image) for installing it on your laptop or an existing VM.

  • The product release tarball. For instance, the latest version of CEE product release tarball.


    Note

    You have to place the downloaded tarball in the offline-products subdirectory.

  • deploy.yaml (configuration file) – The deploy.yaml file contains the configuration required to bring up the SMI Cluster Manager. Also, the information that is related to K8s cluster (managed by the SMI Cluster Manager) first boot configuration resides in the deploy.yaml file.

Requirements

This feature is supported on all the Docker Deployer images and SMI Cluster Manager releases. You need to create the following directory and subdirectories for loading the Docker Deployer images on to the Cluster Manager:

  • A working directory as shown below:

    [cloud-user@master-01 cluster-manager-docker-deployer]# ls -altr example-work-directory/ 
total 550752 
drwxr-xr-x 2 root root 24 Jun 26 23:53 docker-images 
drwxr-xr-x 2 root root 23 Aug 29 20:38 offline-products 
-rw-r--r-- 1 root root 14283 Sep 8 17:32 deploy.yaml 
drwxr-xr-x 4 root root 99 Sep 9 07:07 . 
drwxr-xr-x 7 root root 4096 Sep 10 18:18 ..

    Important
    The working directory comprises of the following:

    • docker-images (subdirectory) - The Docker image tarballs that are needed for loading onto the Deployer, if any, resides in this subdirectory. The Docker Deployer helper (customization) image tarballs from the network function (SMF or PCF) teams, if any, can also reside in this subdirectory.

    • offline-products (subdirectory) - The product release tarballs that must exist in the local registries, if any, reside in this subdirectory.

Components Used

The Docker Deployer image for deploying the SMI Cluster Manager.

Deploying the SMI Cluster Manager

The subsequent section describe the procedures involved in deploying the SMI Cluster Manager on VMware vCenter.

Deploying the Docker Deployer Image

To deploy the Docker Deployer image, use the following configuration:

  1. Load the Docker Deployer image.

    docker load -i <docker_deployer_image>

  2. Verify whether the expected image is loaded or not.

    docker images | grep -i cluster-manager

  3. Initiate the deployment using the docker run command. 

    docker run --rm -v <path_to_current_working_directory:/opt/deployer/work -it <docker_deployer_helper_image> ./deploy

    Important
    The deploy.yaml file contains the configuration required to bring up the SMI Cluster Manager. A clock server – recommended for the K8s to function properly – is provided for setting and configuring the NTP server. The following example deploys the Docker Deployer image using the docker run command.

    Example: 

    docker run --rm -v 
<path_to_current_working_directory>:/opt/deployer/work -it <docker_deployer_image> ./deploy
2019-09-10 20:53:09.219 INFO deploy: Parsing config file: deploy.yaml
2019-09-10 20:53:09.241 WARNING deploy: Please define your NTP servers to configure on the deployer ... Without this k8s will run into issues!!!
Enter the NTP server url to set for the k8s cluster (e.g. clock.cisco.com):clock.cisco.com

  4. You can also use the following (additional) options while deploying the SMI Cluster Manager:

    • -–work-directory – Specifies the location of the default working directory that contains the configuration file. The downloaded vmdk file must reside in this directory.

    • –config-file-name – Specifies the name (other than deploy.yaml) of the configuration file. This allows you to store multiple configuration files within a single working directory.

    • --vmdk-name – Specifies the name of the vmdk file (other than the default cluster-manager.vmdk) that must be downloaded. This allows you to store multiple vmdk files within a single working directory.

    • -–vmdk-url-md5 – Specifies the vmdk hash to verify the downloaded vmdk file. Also, you can use this option to determine if the vmdk file must be downloaded again.

    • -–skip-vmdk-hash-check – You can set this option to ‘True' to skip the hash check. This prevents downloading the vmdk image again.

    • -–log-level – Specifies the logging level supported. The supported options include: CRITICAL, ERROR, WARNING, INFO, DEBUG, and NOTSET.

    • -–overwrite – Set this option to ‘True' to overwrite an existing SMI Cluster Manager with the latest vmdk release.

Sample Configuration File

The following sample configuration file (deploy.yaml) contains the configuration parameters necessary for orchestrating the SMI Cluster Manager. 


#How to access vcenter to deploy cluster manager VM 
vmware: 
  server: <vcenter_server_ip> 
  port: <vcenter_port> 
  #Allows self-signed certs 
  allow-self-signed-cert: <true>  
  user: <vcenter_username> 
  password: <vcenter_password> 
  datacenter: <vcenter_datacenter> 
  host: <vcenter_host_ip_address> 
  #Specifies the corresponding vcenter host datastore 
  datastore: <vcenter_host_datastore>  
  # Specifies the vcenter cluster containing the above host 
  cluster: <vcenter_cluster> 
  nics: 
    #Network IDs (needed for netplan) will start with ens192 and increase by 32 for each list entry. 
    #Example: ens192, ens224, and ens256. 
    - network-name: <network_id> 
  sizing: 
    diskDataGb: <data_disk_size_in_gb> 
    diskHomeGb: <home_disk_size_in_gb> 
    cpu: <number_of_virtual_cpu> 
    ramMb: <memory_in_mb> 
 
#Cloud-init configuration 
#These configuration will only take effect the first time the VM is booted 
initial-boot: 
  #Hostname to be assigned to the VM. For instance, cluster-manager. 
  hostname: <hostname> 
  default-user: <default_username> 
  default-user-ssh-public-key: <public_ssh_key> Example: ssh-rsa    
AAAAB3NzaC1yc2EAAAADAQABAAACAQDBUUTU9xre1S3HY7YyKEsDa2shLhqyIjFwJKSAmYEpBWGUj
aijCR/ArGHhUC63QWc2GekyvTpJeenKTLV+KUDuX383jxA+8MrBul5GJfpAXOAyYqPMQxCO7Nhqsi
/T6n1S7qWfy1kuJQocF1G/jUEFIQdxKYuD/D/W3FjxKoZVA8vao9C73KfMOORFOUlbL0O5KfAFI4E
TAGVnvD+vZdZXMtv5kFUoPwr5MDHoYmmhtYDxMZDB5OAD6tj3a6E4+eoEEAMIBR6HSa68JKhbngxz
wmtUekkorJH+yeYvhWCKFSbirwsrVI1J2rfskE99a6HwihyCrRH23axGg3InY474jZvu/RgfmDU/k
Ow5I/pa9wDYRfo5b9EqOQ==>  
  default-user-password: <first_boot_password> 
   #Password is only enabled for console by default (not SSH) 
   #For enabling SSH through password 
  allow-ssh-password-login: <true>   
#Provides networking information to the VM through a cloud-init netplan. 
# See: https://netplan.io/ and https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v2.html 
netplan: 
# Example of a netplan is provided below 
  template: | 
    network: 
      ethernets: 
        ens192: 
          addresses: <ipv4_address/subnet> 
          gateway4: <gateway_ipv4_address> 
          nameservers: 
            addresses: <ipv4_address> 
            search: <cisco.com> 
          dhcp4: <no> 
      version: <2> 
 
#Configure your ntp servers seprated by commas, as shown in the following example: 
os: 
  ntp: 
    servers:  
      - url: <clock.cisco.com> 
      - url: <ntp.esl.cisco.com> 
 
#Ansible SSH connection information 
ssh-connection: 
  ssh-ip: <vm_ip_address> 
  ssh-username: <vm_login_username>  
  ssh-private-key: | 
-----BEGIN RSA PRIVATE KEY----- 
MIIJKQIBAAKCAgEAwVFE1Pca3tUtx2O2MihLA2trIS4asiIxcCSkgJmBKQVhlI2o
owkfwKxh4VAut0FnNhnpMr06SXnpyky1filA7l9/N48QPvDKwbpeRiX6QFzgMmKj
zEMQjuzYarIv0+p9Uu6ln8tZLiUKHBdRv41BBSEHcSmLg/w/1txY8SqGVQPL2qPQ
u9ynzDjkRTlJWy9DuSnwBSOBHFIyVAW6Iw4l+F7W2Lc5nvQAONlOiyEegrYdlXC+
pnl++Dx/HjiT4OtljUsrvVyTYeShMEWZ/5xtbhBs5pkC0y55bKRtLfOsCUeqGRXO
X2BKCMIsl1dbb6do+BwUHvFbukzjFPtBR4X6vZBafFE3X0Xy8lh9D7cSCqcV7sCl
UJyjaS4SVRrf9WUIGrKB7++pyiMy54e8xkU4nTjjkDHnve6+e43hsUVtT1EkA+rP
BvtpmglV3Z1xrM7fCx6ZGCnSf2mjA6L4t1rWxJXhXi27sWo/E8UpXpwGY2FkySpT
HClQqi60yVU8kwBlZ7w/r2XWVzLb+ZBVKD8K+TAx6GJpobWA8TGQweTgA+rY92uh
OPnqBBADCAUeh0muvCSoW54Mc8JrVHpJKKyR/snmL4VgihUm4q8LK1SNSdq37JBP
fWuh8Iocgq0R9t2sRoNyJ2OO+I2b7v0YH5g1P5DsOSP6WvcA2EX6OW/RKjkCAwEA
o5aRFREBbAaRY0BPdkjoNqbzBCUf592oznefFvVem1P49Nmfu00BWZMnTrGb0tBw
2n9qwgkjc9/OkMXGHFugaMivxGNoUb37dhO1RhFCZRIeoiV1VxnQNl89KVok1hVK
UTyuhbXCBgJnYSaleTQJNIhAe8V4Pn2VdjueDkiNTKfa6psOrqNJC9cfotwJ66o/
ieKJiO8vCRQXpORWrzBu9NboJF+VXgi18OZkRmcvLL/LM5yAZrtkcYd58wZ9GfyV
fvuGUkgWbmA6HEzEZHUa3JhMJJIrFobWE/adVTC/7vhuyKnogAMX29sLMlHI 
-----END RSA PRIVATE KEY----- 
 
#Config to initialize k3s 
init-k3s: 
  external-ip: <vm_external_facing_ip_address> 
  #internal-ip will default to external-ip if not set 
  #internal-ip: 
  external-ingress: <ipv4_address>.<customer_specific_domain_name> 
  #internal-ingress will default to external-ingress if not set 
  #internal-ingress: 
  first-boot-password: <first_boot_password> 
 
#First boot config provides initial configuration to deployer. 'show run' format can be used 
#NOTE: This will only take effect if the database is unitialized. You may need to remove /data and reinitialize if you would like it to take effect 
first-boot-config:

Important
The Kubernetes Cluster configuration (yaml) file must be defined in the first-boot-config fields in the deploy.yaml file. For more information about Kubernetes Cluster configuration, see the Ultra Cloud Core Subscriber Microservices Infrastructure Operations Guide..

Upgrading the SMI Cluster Manager

You can upgrade an existing SMI Cluster Manager with the latest vmdk file using the -–overwrite option. To upgrade the SMI Cluster Manager to the latest vmdk image, perform the following:

  1. Set the --overwrite option to True while deploying the SMI Cluster Manager using docker run command.

    Example: 

    docker run --rm -v <path_to_current_working_directory>:/opt/deployer/work -it <docker_deployer_image> ./deploy --vmdk-url 
<path_to_cluster_manager_airgap.vmdk> --name cluster-manager-20190904.vmdk 
--config-file-name deploy-config-file.yaml --log-level DEBUG --overwrite True

Deploying the SMI Cluster Manager on OpenStack

For deploying the SMI Cluster Manager on OpenStack environment, see the UAME-based VNF Deployment section in the Ultra Cloud Core UAME-based VNF and CNF Deployment Automation Guide. To obtain the UAME documentation, contact your Cisco Account representative.