New and Changed Information

The following table provides an overview of the significant changes up to this current release. The table does not provide an exhaustive list of all changes or of the new features up to this release.

Cisco APIC Release Version

Feature

Description

5.0(1)

Support for Red Hat OpenShift 4.3 on Amazon Web Services (AWS) clouds.

You can use OpenShift on AWS clouds to achieve better visibility and consistent policy application and to speed development.

Using OpenShift on Cisco Cloud APIC

Beginning with release 5.0(1), the Cisco Application Centric Infrastructure (ACI) Container Network Interface (CNI) plug-in supports Red Hat OpenShift 4.3 integrated with Cisco Cloud Application Policy Infrastructure Controller (APIC).

With this support, you can use OpenShift with Cisco Cloud APIC to enforce security policies on Amazon Web Services (AWS) clouds the same way as you use OpenShift with Cisco APIC. Using Cisco ACI CNI with OpenShift on AWS clouds provides better visibility and consistent segmentation policy for your applications.

To use OpenShift on an AWS cloud, you first set up Cisco Cloud APIC as you ordinarily would. You then provision the Cisco ACI CNI plug-in to work with Cisco Cloud APIC. Finally, you install OpenShift using the AWS Installer Provisioned Infrastructure (IPI). IPI creates all elements of the networking, machines, and operating systems that are needed to support the cluster.

This document provides information for integrating OpenShift with Cisco Cloud APIC, including provisioning the network and associates the networking with the OpenShift upstream installer.


Note

Using OpenShift 4.3 with Cisco Cloud APIC is not compatible with multi-site policies. It is meant for use only with local policies—that is, on the same cloud.

Benefits of Using OpenShift with Cisco Cloud APIC

Integrating OpenShift into Cisco Cloud Application Policy Infrastructure Controller (APIC) for use in Amazon Web Services (AWS) clouds enables you to use containers in the cloud in a way similar to how you use containers in on-premises deployments.

The solution's key elements provide a number of advantages:

  • Red Hat's IPI

    Red Hat's Installer Provisioned Infrastructure (IPI) deployment methodology enables you to deploy the cloud environment with a single click.

  • Cisco ACI CNI plug-in

    The Cisco Application Centric Infrastructure(ACI) Container Network Interface (CNI) plug-in enables you to get visibility into the AWS cloud and containers and tie the network in the cloud to high-level policies.

    The Cisco ACI CNI plug-in for AWS makes cloud containers first-class citizens by enabling consistent networking and security policies across all AWS workloads.

  • Cisco Cloud APIC

    Cisco Cloud APIC helps you manage virtual private cloud (VPC) networks, connectivity, and security policies for generic cloud workflows.

Using OpenShift in AWS clouds also has advantages for application development. Moving development to the cloud speeds up processes and does not require the purchase and installation of dedicated hardware or new software.

OpenShift in AWS Clouds

The Cisco Application Centric Infrastructure (ACI) Container Network Interface (CNI) plug-in is automatically installed on the OpenShift nodes as part of the deployment process.

The following diagram depicts the Red Hat OpenShift Installer Provisioned Infrastructure (IPI) architecture in Amazon Web Services with the Cisco ACI CNI plug-in.

Figure 1. OpenShift IPI Architecture


The diagram highlights the following network configuration:

  • The CIDR allocated to the OpenShift Virtual Private Cloud (VPC) is divided into in multiple subnets.

  • The OpenShift nodes and the bootstrap server require dedicated subnets.

  • A NAT gateway must be created and attached to the bootstrap subnet.

  • The NAT gateway must be set as the default gateway within the OpenShift nodes routing table.

  • A VXLAN overlay is created for containers.

  • Containers are deployed in the OpenShift cluster within a single subnet.

  • The Cisco ACI CNI plug-in acts as an IPAM for containers.

Workflow for Integrating OpenShift with Cisco Cloud APIC

This section provides a high-level overview of the tasks that you need to do to integrate Red Hat OpenShift with Cisco Cloud Application Policy Infrastructure Controller (APIC) for use in Amazon Web Service (AWS) clouds.

  1. Set up Cisco Cloud APIC.

    Follow the instructions in the Cisco Cloud APIC for AWS Installation Guide.

  2. Use the acc-provision tool.

  3. Use OpenShift Installer Provisioned Infrastructure (IPI) to integrate OpenShift with the AWS cloud.


    Note

    For step 2 and step 3, follow instructions in Integrate OpenShift with Cisco Cloud APIC.
  4. Use the Cisco Cloud APIC GUI to work with OpenShift in the AWS cloud.

    See the Cisco Cloud APIC for AWS Installation Guide and the Cisco Cloud APIC for AWS User Guide.

Prerequisites for Integrating OpenShift with Cisco Cloud APIC

You must perform the following tasks before you integrate OpenShift with Cisco Cloud Application Policy Infrastructure Controller (APIC).

  1. Install and set up Cisco Cloud APIC.

    Follow the instructions in the Cisco Cloud APIC for AWS Installation Guide.

  2. Configure a tenant account through the Cloud APIC, that is compliant with OpenShift requirements.

    Follow the instructions in Configuring an AWS account for OpenShift 4.3 on the Red Hat website. The following text is an example of the tenant POST:
    
    <polUni>
    <fvTenant name="kube" status="">
    <cloudAwsProvider accountId="XXXXXXXX" providerId="awsprov8" isTrusted="true"/>
    </fvTenant>
    </polUni>

    Note

    You also need to provision an identity access management (IAM) role.
  3. Determine the following data for OpenShift:


    Note

    Do not configure the following options manually. The acc-provision tool needs the information but deploys the configuration automatically.
    • A CIDR IP address to be used for the OpenShift virtual machines (VMs) within the virtual private cloud (VPC); the default is 10.0.0.0/16.

    • Two subnets from the CIDR IP address; the defaults are 10.0.0.0/24 and 10.0.1.0/24.

    • A cluster name for the OpenShift cluster.

  4. Determine the tenant name corresponding to the tenant account for the Cisco Application Centric Infrastructure (ACI) Container Network Interface (CNI) plug-in.

Integrate OpenShift with Cisco Cloud APIC


Note

You can execute the acc-provision and openshift-installer commands from any machine with an internet connection, such as your laptop.

Before you begin

Perform the tasks in the section Prerequisites for Integrating OpenShift with Cisco Cloud APIC.

Procedure


Step 1

Download the provisioning tool from cisco.com.

Step 2

Provision the Cisco Cloud Application Policy Infrastructure Controller (APIC) fabric:

acc-provision--sample -a -c acc_provision_input.yaml -f cloud -u user -p pass -o aci_deployment.yaml

This step generates the aci_deployment.yaml file and the aci_deployment.tar.yaml.gz file. The --sample option generates a configuration file that you should save; you will edit the file in this procedure.

An example of the acc_provision_input.yaml is shown in the following example:

#
# Configuration for ACI Fabric
#
aci_config:
system_id: <cluster_name> # Every opflex cluster must have a distinct ID
apic_hosts: # List of cAPIC hosts to connect for APIC API
- <capic_ip>
tenant:
name: <tenant_name>
vrf: # Information about the VPC. If the specified VRF does not exist,
name: <VRF of ctx profile # a VPC and a VRF will be created for you
tenant: kuber
region: <aws region> # REQUIRED -- your aws region, e.g. us-east-2
#custom_epgs: # List of additional endpoint-group names
# - epg-a # to configure for use with annotations
#
# Networks used by ACI containers
#
net_config:
machine_cidr: 10.0.0.0/16 # cidr used for openshift VMs, must be unique in VPC
bootstrap_subnet: 20.0.0.1/24
node_subnet: 20.0.1.1/24
pod_subnet: 192.168.0.1/16 # Subnet to use for Kubernetes Pods
extern_dynamic: 10.3.0.1/24 # Subnet to use for dynamic external IPs
extern_static: 10.4.0.1/24 # Subnet to use for static external IPs
node_svc_subnet: 10.5.0.1/24
opflex_server_port: 19999
#
# Configuration for container registry
# Update if a custom container registry has been setup
#
# The <tag> values in the registry example represent default values from your system.
registry:
image_prefix: noirolabs # e.g: registry.example.com/noiro
aci_containers_host_version: <tag>
opflex_agent_version: <tag>
opflex_server_version: <tag>
openvswitch_version: <tag>
gbp_version: <tag>
aci_containers_controller_version: <tag>
# image_pull_secret: secret_name # (if needed)
kube_config:
run_snat_container: false
run_gbp_container: true
ep_registry: k8s
opflex_mode: overlay
# ovs_memory_limit: "20Gi" # override if needed, default is "1Gi"
# reboot_opflex_with_ovs: "false" # override if needed, default is "true"
istio_config:
install_istio: False

The step also generates a .tar file containing the Cisco Application Centric Infrastructure (ACI) Container Network Interface (CNI) plug-in manifests with the name aci_deployment.yaml.tar.gz. Note the location of the .tar file, which you will use later in the procedure. Also, note the OpenShift information printed by the installer, as shown in the following example:

Openshift Info
----------------
networking:
  clusterNetwork:
  - cidr: 192.168.0.0/16
    hostPrefix: 23
  machineCIDR: 10.0.0.0/16
  networkType: CiscoAci
  serviceNetwork:
  - 172.30.0.0./16
platform:
   aws:
     region: us-east-2
     subnets:
     - subnet-0bce44864d58fe062
     - subnet-0a31686397f8d7372
Step 3

Use AWS console to make the following configuration:

  1. Create a NAT gateway attached to the bootstrap_subnet.

  2. Create a route table, and add a route to 0.0.0.0/0 pointing to the NAT gateway that you created in the previous step.

  3. Edit the route table association of the node_subnet and replace the route table with the route table that you created in the previous step.

  4. Verify that the bootstrap_subnet has a route table entry pointing to the internet gateway (IGW).

Step 4

Download the Red Hat installer from the Red Hat website: https://cloud.redhat.com/openshift/install/aws/installer-provisioned

Step 5

Create the install-config.yaml file:

./openshift-install create install-config --dir=install_dir
Step 6

Edit the networking and platform entries in the install-config.yaml file, using information from the console output in step 1:

Example:

networking:
clusterNetwork:
- cidr: 14.3.0.0/16 <= match the pod-subnet
hostPrefix: 23
machineCIDR: 21.0.0.0/16 <= match machineCIDR
networkType: CiscoAci <= set to CiscoACI
serviceNetwork:
- 172.30.0.0/16
platform:
aws:
region: us-east-2
subnets:
- subnet-0bce44864d58fe062 <= Update subnets
- subnet-0a31686397f8d7372 <=
Step 7

Generate the manifests directory by completing one of the following actions:

The manifests directory enables you to override the Red Hat OpenShift default CNI plug-in with Cisco ACI CNI plug-in.

Option Description
Run a script (recommended)
  1. Run the following script:

    setup.sh
    #!/bin/sh
    rm -rf ipi
    mkdir ipi
    cp install-config.yaml ipi/
    ./openshift-install create manifests --dir=./ipi
    cp manifests/cluster-network* ipi/manifests/
    ./openshift-install create cluster --dir=ipi --log-level=debug
    
  2. Skip step 8 through step 11 and resume the procedure at step 12.

Enter individual commands Resume the procedure at step 8.
Step 8

Update the manifests with the .tar file that was created in step 1:

cd manifests; tar xvfz <aci_deployment.yaml.tar.gz>
Step 9

Run the installer to create the OpenShift cluster.

./openshift-install create cluster --log-level debug
Step 10

(Optional) Monitor the progress of the cluster creation from a different terminal:

./openshift-install wait-for bootstrap-complete --log-level=debug

./openshift-install wait-for install-complete --log-level=debug
Step 11

Check the cluster status by examining the installer output.

The output looks similar to the following example:



You can also check the status by checking the status of the configuration operators.


What to do next

Verify that the OpenShift integration with Cisco Cloud APIC was successful. See Verify the OpenShift Integration.

Verify the OpenShift Integration

You must verify the OpenShift integration with Cisco Cloud Application Policy Infrastructure Controller (APIC) in Cisco Cloud APIC and Amazon Web Services (AWS).

Before you begin

You must have completed the procedure Integrate OpenShift with Cisco Cloud APIC.

Procedure


Step 1

Log in to Cisco Cloud APIC and verify that you can see the Kubernetes cluster components.

The following screen capture shows the cluster as Prov-Kubernetes.



Step 2

Log into AWS and verify that the cluster components are deployed in the AWS virtual public cloud (VPC) and that the OpenShift portal is reachable.

Also ensure that the aci-containers-system is visible, as shown in the following screen capture.