Download the provisioning tool from cisco.com.

1. Click APIC OpenStack and Container Plugins.

2. Choose the package that you want to download.

3. Click Download.

Generate a sample configuration file that you can edit.

terminal$ acc-provision--sample > aci-containers-config.yaml

#
# Configuration for ACI Fabric
#
aci_config:
  system_id: mykube             # Every opflex cluster must have a distict ID
  apic_hosts:                   # List of APIC hosts to connect for APIC API
  - 10.1.1.101
  vmm_domain:                   # Kubernetes VMM domain configuration
    encap_type: vxlan           # Encap mode: vxlan or vlan
    mcast_range:                # Every opflex VMM must use a distinct range
      start: 225.20.1.1
      end: 225.20.255.255

  # The following resources must already exist on the APIC,
  # they are used, but not created by the provisioning tool.
  aep: kube-cluster             # The AEP for ports/VPCs used by this cluster
  vrf:                          # This VRF used to create all kubernetes EPs
    name: mykube-vrf
    tenant: common              # This can be system-id or common
  l3out:
    name: mykube_l3out          # Used to provision external IPs
    external_networks:
    - mykube_extepg             # Used for external contracts

#
# Networks used by Kubernetes
#
net_config:
  node_subnet: 10.1.0.1/16      # Subnet to use for nodes
  pod_subnet: 10.2.0.1/16       # Subnet to use for Kubernetes Pods
  extern_dynamic: 10.3.0.1/24   # Subnet to use for dynamic external IPs
  extern_static: 10.4.0.1/24    # Subnet to use for static external IPs
  node_svc_subnet: 10.5.0.1/24  # Subnet to use for service graph--This is not the same as the Kubernetes service-cluster-ip-range: Use different subnets.
  kubeapi_vlan: 4001            # The VLAN used by the physdom for nodes
  service_vlan: 4003            # The VLAN used by LoadBalancer services
  infra_vlan: 4093              # The VLAN used by ACI infra

#
# Configuration for container registry
# Update if a custom container registry has been setup
#
registry:
  image_prefix: noiro                   # e.g: registry.example.com/noiro
  # image_pull_secret: secret_name      # (if needed)

Edit and save the sample configuration file, providing information from your network.

Obtain the package which provides acc-provision.

1. Refer the Virtualization Matrix to get the appropriate releases and packages. To check the recommended plugin, hover your mouse over the text marked as Note in the corresponding cell of the matrix table.

2. To get the packages for the plugin, go to Cisco.com. Scroll down and click Downloads.

3. On the Software Downloads page, for the Select a Product field, enter Cisco APIC.

4. On the Software Downloads page, for the Select a Software Type field, click the APIC OpenStack and Containers Plugins option.

5. The latest release software packages are displayed. To get to your release, scroll down and click All Release and select the required release.

6. The required packages and images are displayed. The package(s) for installing acc-provision is indicated explicitly, similar to as displayed below.

   Example: 
   openshift_installer_4.11-5.2.3.5-2.src.tar.gz      29-Jan-2023	164.26 MB
   
   Debian package for ACI CNI Tools 		    26-Jan-2023	11.72 MB
   (acc-provision and acikubectl)
   dist-debs-5.2.3.5.tar.gz
   
   ACI CNI Container Images				26-Jan-2023	2783.95 MB
   dist-generics-5.2.3.5.tar.gz
   
   RPM package for ACI CNI Tools (acc-provision	26-Jan-2023 	6.95 MB
   and acikubectl)
   dist-rpms-5.2.3.5.tar.gz
   

7. Instead of following the above sub-steps (a to f), you can install acc-provision using the package posted in pypi.

   Use the pip install acc_provision command to install from pypi.

   Note	ACI-CNI support is generally backward compatible with the ACI, OpenShift, Kubernetes and Rancher releases and it is recommended to always install the latest packages (however if required, use the pip install acc_provision command to deploy a specific release.

Provision the Cisco ACI fabric.

acc-provision -c aci-containers-config.yaml -o aci-containers.yaml -f kubernetes-<version> -a -u [apic username] -p [apic password]

(Optional): Advanced optional parameters can be configured to adjust to custom parameters other than the ACI default values or base provisioning assumptions:

Configure your uplink interface with NIC bonding or not, depending on how your AEP is configured.

Create a subinterface on your uplink interface on your infra VLAN.

Configure a static route for the multicast subnet 224.0.0.0/4 through the uplink interface used for VXLAN traffic.

Create a subinterface on your uplink interface on your node VLAN. For example, which is called kubeapi_vlan in the configuration file.

Create the /etc/dhcp/dhclient-eth0.4093.conf file with the following content, inserting the MAC address of the Ethernet interface for each server on the first line of the file:

send dhcp-client-identifier 01:<mac-address of infra VLAN interface>;
request subnet-mask, domain-name, domain-name-servers, host-name;
send host-name gethostname();
 
option rfc3442-classless-static-routes code 121 = array of unsigned integer 8;
option ms-classless-static-routes code 249 = array of unsigned integer 8;
option wpad code 252 = string;
 
also request rfc3442-classless-static-routes;
also request ms-classless-static-routes;
also request static-routes;
also request wpad;
also request ntp-servers;

If you have a separate management interface for the node being configured, configure any static routes that you need to access your management network on the management interface.

Ensure that Open vSwitch (OVS) is not running on the node.

Informational: Here is an example of the interface configuration (/etc/network/interfaces):

# Management network interface (not connected to ACI)
auto ens160
iface ens160 inet static
    address  192.168.66.17
    netmask 255.255.255.0
    up route add -net 10.0.0.0/8 gw 192.168.66.1
    dns-nameservers  192.168.66.1

# Interface connected to ACI
auto ens192
iface ens192 inet manual
    mtu 1600

# ACI Infra VLAN
auto ens192.3095
iface ens192.3095 inet dhcp
    mtu 1600
    up route add -net 224.0.0.0/4 dev ens192.3095
    vlan-raw-device ens192

# Node Vlan 
auto ens192.4001
iface ens192.4001 inet static
    address  12.1.0.101
    netmask 255.255.0.0
    mtu 1600
    gateway 12.1.0.1
    vlan-raw-device ens192

Tune the igmp_max_memberships kernel parameter.

1. Check the current configured maximum with the following command:

   sysctl net.ipv4.igmp_max_memberships

2. Edit the /etc/sysctl.d/99-sysctl.conf file and add the following line:

   net.ipv4.igmp_max_memberships = Max_Number_Of_EPGs

3. Restart the node or issue the following command:

   sysctl -p

4. Verify that the new limit is correctly configured with the following command:

   sysctl net.ipv4.igmp_max_memberships

Install the CNI plug-in using the following command: kubectl apply -f aci-containers.yaml

Log in to the Cisco APIC.

Go to Tenants > tenant.

In the tenant navigation pane, expand the following: tenant > Application Profiles > application profile > Application EPGs.

In the tenant navigation pane, expand the Networking and Bridge Domains folders.

If you deploy Kubernetes with a load balancer, go to Tenants > common, expand L4-L7 Services, and perform the following steps:

1. Open the L4-L7 Service Graph Templates folder; you should see a template for Kubernetes.

2. Open the L4-L7 Devices folder; you should see a device for Kubernetes.

3. Open the Deployed Graph Instances folder; you should see an instance for Kubernetes.

Go to VM Networking > Inventory.

In the Inventory navigation pane, expand the Kubernetes folder.

Add routes to null to the subnets:

1. Log in to Cisco Application Policy Infrastructure Controller (APIC).

2. Go to Tenant > Networking > External Routed Networks > your_L3-Out > Logical Node Profile > your_node_profile.

3. In the node profile work pane, double-click a node.

4. In the Node Association dialog box, in the Static Routes area, click the + (plus) icon.

5. In the Create Static Route dialog box, in the Prefix field, enter the static route IP address and network mask that is assigned to the outside network.

   Note	Add a static route for external dynamic and external static subnets. Leave the Next Hop Addresses field empty. A null interface is automatically created.

6. Click Submit.

   In the Node Association dialog box, click Close.

7. In the Node Association dialog box, in the Static Routes area, repeat steps 1d through 1e for each node.

Create match rules for the route map and add the subnet.

1. Go to Tenant > Networking > External Routed Networks > Match Rules for Route Map.

2. Right-click Match Rules for Route Map and choose Create Match Rule for a Route Map.

3. In the Create Match Rule dialog box, in the Name field, enter a name for the match rule.

4. In the Match Prefix area, click the + (plus) icon.

5. In the IP field, enter the static route IP address and network mask that you entered in step 1e.

6. Click Update and then click Submit.

Create a route map for the L3 Out, add a context to it, and choose the match rules.

1. Go to Tenant > Networking > External Routed Networks > your_L3-Out > Route Maps/Profiles.

2. Right-click Route Maps/Profiles, and choose Create Route Maps/Profiles.

3. In the Create Route Map dialog box, from the Name drop-down list, choose default-export.

4. In the Type area, make sure that Match Prefix AND Routing Policy is chosen.

5. In the Contexts area, click the + (plus) icon.

6. In the Create Route Control Context dialog box, with the Order selector, choose 0.

7. In the Name field, enter a name for the policy context.

8. In the Action area, make sure that Permit is chosen.

9. From the Match Rule drop-down list, choose the rule that you created in Step 2c.

10. Click OK and then click Submit.

Check the route maps applied to your dynamic routing protocol that permits the advertisement of the subnets.

1. Find your route map for static routes:

   Example:

   fabric 203 show ip ospf vrf common:k8s  | grep route-map
    Table-map using route-map exp-ctx-2981889-deny-external-tag
      static route-map exp-ctx-st-2981889
      direct route-map exp-ctx-st-2981889
      bgp route-map exp-ctx-proto-2981889
      eigrp route-map exp-ctx-proto-2981889
      coop route-map exp-ctx-st-2981889
   

Find the specific route for each of the nodes, looking for entries that match the name of the match rule:

fabric 203 show route-map exp-ctx-st-2981889 | grep k8s-svc-export
    ip address prefix-lists: IPv4-st19-2981889-exc-ext-out-k8s-svc-export2k8s-svc-export0k8s-svc-export-dst

fabric 204 show route-map  exp-ctx-proto-2981889 | grep k8s-svc-export
    ip address prefix-lists: IPv4-proto19-2981889-exc-ext-out-k8s-svc-export2k8s-svc-export0k8s-svc-export-dst

Verify that the IP addresses are correct for each of the nodes:

fab2-apic1# fabric 203 show ip prefix-list IPv4-st19-2981889-exc-ext-out-k8s-svc-export2k8s-svc-export0k8s-svc-export-dst
----------------------------------------------------------------
 Node 203 (Leaf203)
----------------------------------------------------------------
ip prefix-list IPv4-st19-2981889-exc-ext-out-k8s-svc-export2k8s-svc-export0k8s-svc-export-dst: 2 entries
   seq 1 permit 10.3.0.1/24
   seq 2 permit 10.4.0.1/24

fab2-apic1# fabric 204 show ip prefix-list IPv4-proto19-2981889-exc-ext-out-k8s-svc-export2k8s-svc-export0k8s-svc-export-dst
----------------------------------------------------------------
 Node 204 (Leaf204)
----------------------------------------------------------------
ip prefix-list IPv4-proto19-2981889-exc-ext-out-k8s-svc-export2k8s-svc-export0k8s-svc-export-dst: 2 entries
   seq 1 permit 10.3.0.1/24
   seq 2 permit 10.4.0.1/24

Log in to Cisco Application Policy Infrastructure Controller (APIC).

Go to Admin > AAA > Users.

Click the username associated with your cluster.

Scroll to the security domain and verify the following:

Log in to the Cisco Application Policy Infrastructure Controller (APIC).

Create a new security domain by completing the following steps:

1. Go to Admin > AAA > Security.

2. Right-click Security and choose Create Security Domain.

3. In the Create Security Domain dialog box, enter a name for the security name and click Submit.

   We recommend that you use the Kubernetes system_id name. Entering a description is optional.

Go to Admin > AAA > Users and complete the following steps:

1. Double-click the system_id username.

2. In the Local User dialog box, scroll to the Security Domains area.

3. Expand Security Domain all, right-click Role Admin, and then click Delete from the drop-down list.

4. Click the + (plus) icon.

5. In the Add User Domain dialog box, choose the container system_id domain.

   You now add two new roles.

6. Click the + (plus) icon.

7. From the Name drop-down list, choose Admin, from the Access Type drop-down list, choose Write Privilege, and then click Update.

   Note	Complete the following three steps only if the cluster resources—such as the bridge domain, virtual routing and forwarding (VRF), and Layer 3 outside (L3Out)—are in the common tenant.

8. From the drop-down list, choose common.

9. Click the + (plus) icon.

10. From the Name drop-down list, choose Admin, from the Access Type drop-down list, choose Write Privilege, and then click Update

11. Click Submit.

Create a custom role-based access control (RBAC) rule to allow the container account to write information into the container domain by completing the following steps:

1. Go to Admin > Security and in the User Management - Security central pane, choose RBAC Rules and Explicit Rules.

2. Click the tools icon and choose Create RBAC Rule from the drop-down list.

3. In the Create RBAC Rule dialog box, in the DN field, enter comp/prov-Kubernetes/ctrlr-[system_id]-system_id.

   Example:

   If your system_id is "k8s," the DN is comp/prov-Kubernetes/ctrlr-[k8s]-k8s

4. From the Domain drop-down list, choose the security domain that you created in Step 2.

5. Set the Allow Writes option to Yes.

6. Click Submit.

Map the security domain to the cluster tenant by completing the following steps:

1. Go to Tenants, choose the tenant for the Kubernetes system_id, and then in the Tenant central pane, choose Policy.

2. In the Security Domains area, click the + (plus) icon.

3. From the Name drop-down list, choose the newly created security name, click Update, and then click Submit.

Check for faults on the fabric and resolve any that are relevant.

Check that the API server advertisement addresses use the node subnet, and that the nodes are configured to route all Kubernetes subnets over the node uplink.

Check the logs for the container aci-containers-controller for errors using the following command on the Kubernetes master node: acikubectl debug logs controller acc

Check these node-specific logs for errors using the following commands on the Kubernetes master node:

1. Host agent: acikubectl debug logs node -n [nodename] host-agent

2. OpFlex agent: acikubectl debug logs node -n [nodename] opflex-agent

3. Open vSwitch: acikubectl debug logs node -n [nodename] openvswitch

Enter the following command to collect and export Kubernetes logs: acikubectl debug cluster-report -o cluster-report.tar.gz

Check configuration of the next-hop router.

When contracts are not enforced by the fabric, you cannot access external services from the next-hop router. You can access external services from an IP address that is not in the subnet configured on the next-hop router interface.

Check the contracts between the pod EPGs.

Verify that there is a contract that allows ARP traffic. All pods are in the same subnet so Address Resolution Protocol (ARP) is required.

Ensure that the EPG name, tenant, and application are spelled correctly.

Make sure that the VMM domain is mapped to an EPG.

Ensure that the health check contract exists between the pod EPG and the node EPG.

Check the mcast-daemon log messages:

kubectl -n kube-system logs aci-containers-host-[xxxxx] mcast-daemon