• Cisco Container Platform Installer OVA

• Latest two versions of the tenant OVA

• vCenter cluster with High Availability (HA) and Distributed Resource Scheduler (DRS) enabled

• A DHCP server that provides IP addresses to the Cisco Container Platform installer VMs

• A shared datastore that is mounted on all the ESXi hosts in the cluster

• Cisco Container Platform Control Plane VMs need to have network access to the vCenter appliance API

• Cisco Container Platform 1.3.0 and later requires hypervisor hosts to be running CPUs with an Ivy Bridge or newer microarchitecture.

• Kubectl version 1.11+

• Support for Kubernetes 1.12 tenant clusters

  • Supports ACI 3.2(4e).

    Note: This version of ACI CNI supports Kubernetes 1.12.

• Controller and tenant cluster node IP addresses are automatically allocated by Cisco Container Platform

• Removal of support for Kubernetes 1.10

• For deployments or SSH access, supports the use of Ed25519 or ECDSA digital signature algorithms

• Removal of support for less secure RSA and DSA digital signature algorithms

• Password authentication for SSH access to VMs is no longer allowed

• User passphrase is required for passphrase changes

• In an ACI deployment, the Cisco Container Platform cluster name is the same as the ACI tenant name

  For example, if the Cisco Container Platform tenant name is ccp-tenant-1, the ACI tenant name is ccp_tenant_1.

• Support configuring LDAP/AD during installation and upgrade of Cisco Container Platform

• Support configuring NTP servers for the installer VM using the OVF property

For step by step instructions on installing Cisco Container Platform, refer to the Cisco Container Platform Installation Guide.

• Upgrading Cisco Container Platform is supported from the 1.5.0 release for deployments using Calico or ACI for CNI.

• If an existing deployment uses Contiv for CNI, then upgrades to the current version are not supported.

It is required to open a support case for assistance in creating a complete backup and for performing a restore.

• Fixed list of AMI options for EKS clusters

• Cisco Container Platform web interface bug fixes

• When upgrading Cisco Container Platform or a tenant cluster from 3.0.x to 3.1.x, the managed IP addresses for nodes are not properly released. They remain allocated though not in use. This consumes four IP addresses for the Control Plane and one IP address each for the master and worker nodes of the tenant clusters. For this reason, you may have to expand the IP pool to support future upgrades.

  Note: Upgrading to Cisco Container Platform 3.2.0 (when available) or later versions will not consume extra IP addresses.

• Cisco Container Platform upgrade from a version earlier than 2.2.2 fails when the cluster name contains uppercase letters.

  Workaround

  1. SSH to the Cisco Container Platform Control Plane master VM and change the cluster name to lowercase in the `ccp-appdata` table:

     sudo apt-get update
      sudo apt-get install -y jq
      kubectl exec -it mysql-0 -- mysql -p$(kubectl get secret mysql -o json | jq -r '.data["mysql-root-password"]' | base64 -d) ccp-appdata -e "update keyvalues_keyvalue set value = replace(value, 'CCP-CLUSTER-NAME', lower('CCP-CLUSTER-NAME')) where instr(value, 'CCP-CLUSTER-NAME') > 0;"
      kubectl exec -it mysql-0 -- mysql -p$(kubectl get secret mysql -o json | jq -r '.data["mysql-root-password"]' | base64 -d) ccp-appdata -e "select * from keyvalues_keyvalue;"

  2. If you are using a localized version of vSphere, follow these steps to rename the datastore folder for the cluster data:

     1. In the vSphere web client, click vCenter.

     2. Click the Storage tab.

     3. From the left pane, choose the datastore that is used to create the cluster.

     4. Select the folder with the cluster name that you want to change.

        For example: CCP-CLUSTER-NAME

     5. Rename the folder to the lowercase of the same name.

        For example: ccp-cluster-name

  3. Follow these steps to ensure that any existing disk path uses lowercase names:

     1. Click the Virtual Machines tab, choose the VM named ccp-cluster-name-masterxxxxx, and then click Edit settings.

     2. Remove Harddisk 2.

     3. Click the Manage Other Disks tab and remove Harddisk.

     4. Click Add Existing Hard Disk and choose the disk from datastore/<your cluster name>/etcd.disk.

     5. Click Add Existing Hard Disk and choose the disk from datastore/<your cluster name>/cert.disk.

  4. Start the upgrade of Cisco Container Platform using the same cluster name in lowercase.

• On Upgrading to HyperFlex 3.5.2, volume traffic disruption occurs.

  Note: This section is applicable only if you are using the HyperFlex Flex Volume plugin for Kubernetes.

  In HyperFlex 3.5.1 or earlier, the IP address used by the vSwitch on ESXi hosts was 169.254.1.1. The HyperFlex clusters whose Storage Hypervisor Network addresses are in the range 169.254.1.0/24 conflicted with 169.254.1.1. To work around this IP conflict issue, in HyperFlex 3.5.2, the default IP address is changed to 169.254.254.1. Due to this change, the Flex Volume configuration on the Kubernetes nodes will no longer be correct after an upgrade.

  Workarounds

  Note: You must use only one of the following two options to workaround this issue.

  Option 1: Change Configuration on HyperFlex Controller VMs

  You can use this option when there are no existing HyperFlex clusters that use the 169.154.1.0/24 range on ESXi. This avoids the need to change the Kubernetes node configuration for these clusters.

  After upgrading HyperFlex to 3.5.2, follow these steps to change the default IP address to 169.254.1.1:

  1. Run the following command to find iscsiTargetAddress = "169.254.254.1" and replace it with iscsiTargetAddress = "169.254.1.1" in the application.conf file:

       sed -i -e 's/iscsiTargetAddress*169.254.1.1/iscsiTargetAddress*169.254.254.1/g' /opt/springpath/storfs-mgmt/hxSvcMgr-1.0/conf/application.conf 

  2. Run the following command to find istgtConfTargetAddress = "169.254.254.1" and replace it with istgtConfTargetAddress = "169.254.1.1" in the application.conf file:

      sed -i -e 's/istgtConfTargetAddress*169.254.254.1/istgtConfTargetAddress*169.254.1.1/g' /opt/springpath/storfs-mgmt/hxSvcMgr-1.0/conf/application.conf

  3. Run the following commands to restart the following services:

      restart hxSvcMgr
      restart stMgr

  Option 2: Change Configuration on all Kubernetes VMs

  You can use this option when there are existing HyperFlex clusters that use the 169.154.1.0/24 range on ESXi. After a Kubernetes cluster operation such as scale up or upgrade, this step must be repeated on the new VMs. For this reason, we recommend option 1 as the preferred solution.

  After upgrading HyperFlex to 3.5.2, run the following command for every Kubernetes VM to find "targetIp": "169.254.1.1" and replace it with "targetIp": "169.254.254.1" in the hxflexvolume.json file:

  ssh -l <ssh user> -i <private key file> <VM IP> -- sed -i -e 's/169.254.1.1/169.254.254.1/g' /etc/kubernetes/hxflexvolume.json

  Note:

  The <ssh user> must match the ssh user that you specified during cluster creation.

  The <private key file> must correspond to the public key that you specified during cluster creation.

• The Upgrade button on the Cluster Details screen of the Cisco Container Platform web interface is not working currently.

  Workaround

  You can upgrade a tenant cluster from the Clusters screen.

• During a Control Plane upgrade, if you change the SUBNET CIDR field on the Verify Network screen, the IP ADDRESS RANGE is updated.

  Workaround

  Note: You must use only one of the following two options to workaround this issue.

  Option 1:

  Go to the Authenticate CCP screen, enter the necessary data, and then click NEXT.

  The original IP address range is restored.

  Option 2:

  In a Contiv or Calico deployment, find the original IP address range from the Network Editing screen.

  Note:

  • In an ACI deployment earlier than the 2.2.x, the original start and end IP address is the existing Control Plane IP address.

  • In an ACI deployment 2.2.x and later, the original start and end IP address is the same as that which is configured during the Cisco Container Platform installation.

• In an ACI deployment with Kubernetes 1.12, Cisco Container Platform configures the NGINX controller and Fluentd pod during the deployment. If the pods restart post-deployment, the pod configurations are lost.

  Workaround

  Run the following command to configure the pods again:

  kubectl annotate pod <POD NAME> -n ccp opflex.cisco.com/computed-endpoint-group='{"policy-space":"<ACI TENANT NAME>","name":"kubernetes|kube-default"}' --overwrite=True

• During an ACI tenant upgrade, you can safely ignore the Subnet field.

• Cisco Container Platform must use a Kubernetes 1.11 or 1.12 image that is associated with this release. Older versions of the tenant base image are not supported.

• ACI tenant cluster does not work with a link-local interface with Kubernetes 1.11.3.

• You can use only the latest two versions of the tenant image that are associated with the current release. Use of older versions of the tenant image is not supported.

• You will get errors when you scale up tenant clusters or add new node pools to clusters that were created using an older version of Cisco Container Platform.

  Workaround

  You must upgrade Cisco Container Platform before attempting to scale up tenant clusters or add new node pools.

• Kubernetes 1.11.3 API server has a known memory leak issue

• When using HyperFlex as the dynamic provisioner, mounting volumes may fail with the following error message:

  MountVolume.SetUp failed for volume "xxxxx" : mount command failed, status: Failed to mount volume xxxxx, reason:

  Workaround

  1. Restart the scvmclient on the esx server using the following command:

     /etc/init.d/scvmclient restart

  2. Ensure that the status is running.

• Contiv as the CNI for tenant clusters is only supported as Tech Preview, and upgrading to a newer version of Cisco Container Platform is not supported.

• In an ACI environment, the link to a tenant cluster Kubernetes Dashboard from the Cisco Container Platform dashboard is not supported. To view the tenant cluster in the Kubernetes Dashboard, you need to obtain the Ingress IP of external IP address using kubectl get svc.

• The Cisco Container Platform web interface displays links to external pages such as Smart Licensing. You cannot launch these pages if you do not have access to them.

• Virtual IP address is not released when cluster creation fails.

• In a Contiv deployment, you should not use matchExpressions for a NetworkPolicy.

• In a Contiv deployment, network policy does not work with the hostnetwork pod.

• In a Contiv deployment, the pod CIDR must be at least a /14 network.

• In a Calico deployment:

  • The network policy matching on labels will not block hostnetwork access to pods or services.

  • Host IP change may impact pod networking. To resolve the issue, you need to restart the Calico pods.

• istioctl is not installed when you enable Istio.

  Workaround

  Run the following command to deploy istioctl:

  export ISTIO_VERSION=1.0 
  curl -L https://git.io/getLatestIstio | sh - 
  chmod +x istio-${ISTIO_VERSION}/bin/istioctl 
  sudo mv istio-${ISTIO_VERSION}/bin/istioctl /usr/local/bin/ 
  istioctl version  
  

  For more information, refer to the Monitoring Istio Service Meshes section of the Cisco Container Platform User Guide.

• When upgrading Istio from the 0.8 version to the 1.0 version, the backend services stop responding and you need to manually restart them.

• When you upgrade tenant clusters, the Prometheus and EFK components are purged before installing the new versions. If you want to save history, a manual backup and migration are required before a tenant cluster upgrade.

• Taking a snapshot of the VMs managed by Cisco Container Platform is currently unsupported and results in failures during upgrades.

• ACI deployments are only supported in online mode.

• ACI deployments do not support Kubernetes security context.

• cert-manager is now deployed in tenant clusters. It is supported as Tech Preview.