Log in to Cisco APIC.

Create the EPG and add it to the bridge domain kube-pod-bd.

Attach the EPG to the VMM domain.

Configure the EPG to consume contracts in the OpenShift tenant:

Configure the EPG to consume contracts in the common tenant:

Create any contracts you need for your application and provide and consume them as needed.

Apply annotations to the namespaces or deployments.

Add routes to null to the subnets:

1. Log in to Cisco Application Policy Infrastructure Controller (APIC).

2. Go to Tenant > Networking > External Routed Networks > your_L3-Out > Logical Node Profile > your_node_profile.

3. In the node profile work pane, double-click a node.

4. In the Node Association dialog box, in the Static Routes area, click the + (plus) icon.

5. In the Create Static Route dialog box, in the Prefix field, enter the static route IP address and network mask that is assigned to the outside network.

   Note	Add a static route for external dynamic subnets. Leave the Next Hop Addresses field empty. A null interface is automatically created.

6. Click Submit.

   In the Node Association dialog box, click Close.

7. In the Node Association dialog box, in the Static Routes area, repeat steps 1.d/1.4 through 1.f/1.6 for each node.

Create match rules for the route map and add the subnet.

1. Go to Tenant > Networking > External Routed Networks > Match Rules for Route Map.

2. Right-click Match Rules for Route Map and choose Create Match Rule for a Route Map.

3. In the Create Match Rule dialog box, in the Name field, enter a name for the match rule.

4. In the Match Prefix area, click the + (plus) icon.

5. In the IP field, enter the static route IP address and network mask that you entered in step 1e.

6. Click Update and then click Submit.

Create a route map for the L3 Out, add a context to it, and choose the match rules.

1. Go to Tenant > Networking > External Routed Networks > your_L3-Out > Route Maps/Profiles.

2. Right-click Route Maps/Profiles, and choose Create Route Maps/Profiles.

3. In the Create Route Map dialog box, from the Name drop-down list, choose default-export.

4. In the Type area, make sure that Match Prefix AND Routing Policy is chosen.

5. In the Contexts area, click the + (plus) icon.

6. In the Create Route Control Context dialog box, with the Order selector, choose 0.

7. In the Name field, enter a name for the policy context.

8. In the Action area, make sure that Permit is chosen.

9. From the Match Rule drop-down list, choose the rule that you created in Step 2c.

10. Click OK and then click Submit.

Check the route maps applied to your dynamic routing protocol that permits the advertisement of the subnets.

1. Find your route map for static routes:

   Example:

   fabric 203 show ip ospf vrf common:k8s  | grep route-map
    Table-map using route-map exp-ctx-2981889-deny-external-tag
      static route-map exp-ctx-st-2981889
      direct route-map exp-ctx-st-2981889
      bgp route-map exp-ctx-proto-2981889
      eigrp route-map exp-ctx-proto-2981889
      coop route-map exp-ctx-st-2981889
   

Find the specific route for each of the nodes, looking for entries that match the name of the match rule:

fabric 203 show route-map exp-ctx-st-2981889 | grep os-svc-export
    ip address prefix-lists: IPv4-st19-2981889-exc-ext-out-os-svc-export2os-svc-export0os-svc-export-dst

fabric 204 show route-map  exp-ctx-proto-2981889 | grep os-svc-export
    ip address prefix-lists: IPv4-proto19-2981889-exc-ext-out-os-svc-export2os-svc-export0os-svc-export-dst

Verify that the IP addresses are correct for each of the nodes:

fab2-apic1# fabric 203 show ip prefix-list IPv4-st19-2981889-exc-ext-out-os-svc-export2os-svc-export0os-svc-export-dst
----------------------------------------------------------------
 Node 203 (Leaf203)
----------------------------------------------------------------
ip prefix-list IPv4-st19-2981889-exc-ext-out-os-svc-export2os-svc-export0os-svc-export-dst: 2 entries
   seq 1 permit 10.3.0.1/24

fab2-apic1# fabric 204 show ip prefix-list IPv4-proto19-2981889-exc-ext-out-os-svc-export2os-svc-export0os-svc-export-dst
----------------------------------------------------------------
 Node 204 (Leaf204)
----------------------------------------------------------------
ip prefix-list IPv4-proto19-2981889-exc-ext-out-os-svc-export2os-svc-export0os-svc-export-dst: 2 entries

Log in to Cisco Application Policy Infrastructure Controller (APIC).

Go to Admin > AAA > Users.

Click the username associated with your cluster.

Scroll to the security domain and verify the following:

Log in to the Cisco Application Policy Infrastructure Controller (APIC).

Create a new security domain by completing the following steps:

1. Go to Admin > AAA > Security.

2. Right-click Security and choose Create Security Domain.

3. In the Create Security Domain dialog box, enter a name for the security name and click Submit.

   We recommend that you use the OpenShift system_id name. Entering a description is optional.

Go to Admin > AAA > Users and complete the following steps:

1. Double-click the system_id username.

2. In the Local User dialog box, scroll to the Security Domains area.

3. Expand Security Domain all, right-click Role Admin, and then click Delete from the drop-down list.

4. Click the + (plus) icon.

5. In the Add User Domain dialog box, choose the container system_id domain.

   You now add two new roles.

6. Click the + (plus) icon.

7. From the Name drop-down list, choose Admin, from the Access Type drop-down list, choose Write Privilege, and then click Update.

   Note	Complete the following three steps only if the cluster resources—such as the bridge domain, virtual routing and forwarding (VRF), and Layer 3 outside (L3Out)—are in the common tenant.

8. From the drop-down list, choose common.

9. Click the + (plus) icon.

10. From the Name drop-down list, choose Admin, from the Access Type drop-down list, choose Write Privilege, and then click Update

11. Click Submit.

Create a custom role-based access control (RBAC) rule to allow the container account to write information into the container domain by completing the following steps:

1. Go to Admin > Security and in the User Management - Security central pane, choose RBAC Rules and Explicit Rules.

2. Click the tools icon and choose Create RBAC Rule from the drop-down list.

3. In the Create RBAC Rule dialog box, in the DN field, enter comp/prov-OpenShift/ctrlr-[system_id]-system_id.

   Example:

   If your system_id is "OS," the DN is comp/prov-OpenShift/ctrlr-[OS]-OS

4. From the Domain drop-down list, choose the security domain that you created in Step 2.

5. Set the Allow Writes option to Yes.

6. Click Submit.

Map the security domain to the cluster tenant by completing the following steps:

1. Go to Tenants, choose the tenant for the OpenShift system_id, and then in the Tenant central pane, choose Policy.

2. In the Security Domains area, click the + (plus) icon.

3. From the Name drop-down list, choose the newly created security name, click Update, and then click Submit.

Check for faults on the fabric and resolve any that are relevant.

Check that the API server advertisement addresses use the node subnet, and that the nodes are configured to route all OpenShift subnets over the node uplink.

Check the logs for the container aci-containers-controller for errors using the following command on the OpenShift master node: acikubectl debug logs controller acc

Check these node-specific logs for errors using the following commands on the OpenShift master node:

1. Host agent: acikubectl debug logs node -n [nodename] host-agent

2. OpFlex agent: acikubectl debug logs node -n [nodename] opflex-agent

3. Open vSwitch: acikubectl debug logs node -n [nodename] openvswitch