Step 1

On the menu bar click Fabric > Inventory.

Step 2

In the Navigation pane, expand Quick Start and click Node or Pod Setup.

Step 3

In the Remote Leaf pane of the working pane, click Setup Remote Leaf or right-click Node or Pod Setup and click Setup Remote Leaf.

Step 4

Follow the instructions to configure the following:

Step 5

On the menu bar click System > System Settings.

Step 6

In the Navigation pane, choose System Global GIPo.

Step 7

For Use Infra GIPo as System GIPo, choose Enabled.

Step 1

On the menu bar click Fabric > Inventory.

Step 2

In the Navigation pane, expand Quick Start and click Add Remote Leaf.

Step 3

In the Remote Leaf pane of the working pane, click Add Remote Leaf.

Step 4

Configure the interpod connectivity before adding the remote leaf switch, if necessary.

Step 5

At the end of the process for configuring interpod connectivity, click Add Remote Leaf in the Summary page.

Step 6

In the Add Remote Leaf wizard, review the information in the Overview page.

Step 7

When you have finished reviewing the information in the Overview panel, click Get Started at the bottom right corner of the page.

Step 8

In the External TEP page, configure the necessary parameters.

1. Leave the Use Defaults checkbox checked, or uncheck it, if necessary.

   When checked, the wizard automatically allocates data plane and unicast TEP addresses. Those fields are not displayed when the Use Defaults box is checked. Uncheck the Use Defaults box to view or modify the proposed addresses, if necessary.

2. In the External TEP Pool field, enter the external TEP for the physical pod.

   The external TEP pool must not overlap the internal TEP pool.

3. In the Unicast TEP IP field, change the value that is automatically populated in this field, if necessary.

   This address is automatically allocated by Cisco APIC from the External TEP Pool, and will be used for sending traffic from the remote leaf switch to the local leaf switches on that pod.

   Cisco APIC automatically configures the unicast TEP IP address when you enter the External TEP Pool address.

4. Repeat these steps for each pod, if you have a multi-pod configuration.

5. When you have entered all of the necessary information in this page, click the Next button at the bottom right corner of the page.

   The Pod Selection page appears.

Step 9

In the Pod Selection page, configure the necessary parameters.

1. In the Pod ID field, select the pod ID of the pod where the remote leaf switches will be associated.

2. In the Remote Leaf TEP Pool field, select an existing remote leaf TEP pool or enter a remote leaf TEP pool to allocate IP addresses to the remote leaf switches.

   Click the View existing TEP Pools link underneath the Remote Leaf TEP Pool field to see the existing TEP pools (internal TEP pools, remote leaf TEP pools, and external TEP pools). Use this information to avoid creating duplicate or overlapping pools.

3. When you have entered all of the necessary information in this page, click the Next button at the bottom right corner of the page.

   The Routing Protocol page appears.

Step 10

In the Routing Protocol page, configure the necessary parameters.

1. Under the L3 Outside Configuration section, in the L3 Outside field, create or select an existing L3Out to represent the connection between the remote leaf switches and the upstream router.

   For the remote leaf switch configuration, we recommend that you use or create an L3Out that is different from the L3Out used in the multi-pod configuration.

2. Under the OSPF section, leave the Use Defaults checkbox checked, or uncheck it, if necessary.

   When the checkbox is checked, the Cisco APIC GUI conceals the optional fields for configuring OSPF.

   The checkbox is checked by default. Uncheck it to reveal the optional fields.

3. Gather the configuration information from the IPN, if necessary.

   For example, from the IPN, you might enter the following command to gather certain configuration information:

   IPN# show running-config interface ethernet slot/chassis-number
   
   

   For example:

   IPN# show running-config interface ethernet 1/5.11
   ...
   ip router ospf infra area 0.0.0.59
   ...

4. In the Area ID field, enter the OSPF area ID.

   Looking at the OSPF area 59 information shown in the output in the previous step, you could enter a different area in the Area ID field (for example, 0) and have a different L3Out. If you are using a different area for the remote leaf switch, you must create a different L3Out. You can also create a different L3Out, even if you are using the same OSPF area ID.

5. In the Area Type field, select the OSPF area type.

   You can choose one of the following OSPF types:

   • NSSA area

   • Regular area

   Note	You might see Stub area as an option in the Area Type field; however, stub area will not advertise the routes to the IPN, so stub area is not a supported option for infra L3Outs.

   Regular area is the default.

6. In the Area Cost field, select the appropriate OSPF value.

7. In the Authentication Type field, select the appropriate OSPF authentication type.

8. In the Authentication Key field, select the appropriate OSPF authentication key. Re-enter the OSPF authentication key in the Confirm Key field.

9. In the Interface Policy field, enter or select the OSPF interface policy.

   You can choose an existing policy or create a new one using the Create OSPF Interface Policy dialog box.

10. When you have entered all of the necessary information in this page, click the Next button at the bottom right corner of the page.

    The Remote Leafs page appears.

Step 11

In the Remote Leafs page, configure the necessary parameters.

1. In the Serial field, enter the serial number for the remote leaf switch or select a discovered remote leaf switch from the dropdown menu.

2. In the Node ID field, assign a node ID to the remote leaf switch.

3. In the Name field, assign a name to the remote leaf switch.

4. In the Router ID field, enter a router ID that will be used to establish the control-plane communication with the upstream router and the rest of the Cisco ACI fabric.

5. In the Loopback Address field, enter the IPN router loopback IP address, if necessary.

   Leave this field blank if you use a router ID address.

6. Under the Interfaces section, in the Interface field, enter interface information for this remote leaf switch.

7. Under the Interfaces section, in the IPv4 Address field, enter the IPv4 IP address for the interface.

8. Enter information on additional interfaces, if necessary.

   Click + within the Interfaces box to enter information for multiple interfaces.

9. When you have entered all of the necessary information for this remote leaf switch, enter information for additional remote leaf switches, if necessary.

   Click + to the right of the Interfaces box to enter information for multiple remote leaf switches.

10. When you have entered all of the necessary information in this page, click the Next button at the bottom right corner of the page.

    The Confirmation page appears.

Step 12

In the Confirmation page, review the list of policies that the wizard will create and change the names of any of the policies, if necessary, then click Finish at the bottom right corner of the page.

Step 13

In the Remote Leaf Summary page, click the appropriate button.

Step 14

In the Navigation pane, click Fabric Membership, then click the Nodes Pending Registration tab to view the status of the remote leaf switch configuration.

Step 15

Log into the spine switch connected to the IPN and enter the following command:

switch# show nattable

––––––– NAT TABLE ––––––––--
Private Ip   Routeable Ip
10.0.0.1     192.0.2.100
10.0.0.2     192.0.2.101
10.0.0.3     192.0.2.102

Step 16

On the IPN sub-interfaces connecting the remote leaf switches, configure the DHCP relays for each interface.

switch# configure terminal
switch(config)# interface ethernet 1/5.11
switch(config-subif)# ip dhcp relay address 192.0.2.100
switch(config-subif)# ip dhcp relay address 192.0.2.101
switch(config-subif)# ip dhcp relay address 192.0.2.102
switch(config-subif)# exit
switch(config)# interface ethernet 1/7.11
switch(config-subif)# ip dhcp relay address 192.0.2.100
switch(config-subif)# ip dhcp relay address 192.0.2.101
switch(config-subif)# ip dhcp relay address 192.0.2.102
switch(config-subif)# exit
switch(config)# exit
switch#

Step 17

In the Navigation pane, click Fabric Membership, then click the Registered Nodes tab to view the status of the remote leaf switch configuration.

Step 18

On the menu bar click System > System Settings.

Step 19

In the Navigation pane, choose System Global GIPo.

Step 20

For Use Infra GIPo as System GIPo, choose Enabled.

Step 1

Configure the TEP pool for the remote leaf switches, with the following steps:

1. On the menu bar, click Fabric > Inventory.

2. In the Navigation pane, click Pod Fabric Setup Policy.

3. On the Fabric Setup Policy panel, double-click the pod where you want to add the pair of remote leaf switches.

4. Click the + on the Remote Pools table.

5. Enter the remote ID and a subnet for the remote TEP pool and click Submit.

6. On the Fabric Setup Policy panel, click Submit.

Step 2

Configure the L3Out for the spine switch connected to the IPN router, with the following steps:

1. On the menu bar, click Tenants > infra.

2. In the Navigation pane, expand Networking, right-click External Routed Networks, and choose Create Routed Outside.

3. Enter a name for the L3Out.

4. Click the OSPF checkbox to enable OSPF, and configure the OSPF details the same as on the IPN and WAN routers.

5. Only check the Enable Remote Leaf check box, if the pod where you are adding the remote leaf switches is part of a multipod fabric.

   This option enables a second OSPF instance using VLAN-5 for multipod, which ensures that routes for remote leaf switches are only advertised within the pod they belong to.

6. Choose the overlay-1 VRF.

Step 3

Configure the details for the spine and the interfaces used in the L3Out, with the following steps:

1. Click the + on the Nodes and Interfaces Protocol Profiles table.

2. Enter the node profile name.

3. Click the + on the Nodes table, enter the following details.

   • Node ID—ID for the spine switch that is connected to the IPN router.

   • Router ID—IP address for the IPN router

   • External Control Peering—disable if the pod where you are adding the remote leaf switches is in a single-pod fabric

4. Click OK.

5. Click the + on the OSPF Interface Profiles table.

6. Enter the name of the interface profile and click Next.

7. Under OSPF Profile, click OSPF Policy and choose a previously created policy or click Create OSPF Interface Policy.

8. Click Next.

9. Click Routed Sub-Interface, click the + on the Routed Sub-Interfaces table, and enter the following details:

   • Node—Spine switch where the interface is located.

   • Path—Interface connected to the IPN router

   • Encap—Enter 4 for the VLAN

10. Click OK and click Next.

11. Click the + on the External EPG Networks table.

12. Enter the name of the external network, and click OK.

13. Click Finish.

Step 4

To complete the fabric membership configuration for the remote leaf switches, perform the following steps:

1. Navigate to Fabric > Inventory > Fabric Membership.

   At this point, the new remote leaf switches should appear in the list of switches registered in the fabric. However, they are not recognized as remote leaf switches until you configure the Node Identity Policy, with the following steps.

2. For each remote leaf switch, double-click on the node in the list, configure the following details, and click Update:

   • Node ID—Remote leaf switch ID

   • RL TEP Pool—Identifier for the remote leaf TEP pool, that you previously configured

   • Node Name—Name of the remote leaf switch

   After you configure the Node Identity Policy for each remote leaf switch, it is listed in the Fabric Membership table with the role remote leaf.

Step 5

Configure the L3Out for the remote leaf location, with the following steps:

1. Navigate to Tenants > infra > Networking.

2. Right-click External Routed Networks, and choose Create Routed Outside.

3. Enter a name for the L3Out.

4. Click the OSPF checkbox to enable OSPF, and configure the OSPF details the same as on the IPN and WAN router.

5. For releases prior to release 4.1(2), check the Enable Remote Leaf check box if the pod where you are adding the remote leaf switches is part of a multipod fabric.

   Note

   Do not check the Enable Remote Leaf check box if you are deploying new remote leaf switches running release 4.1(2) or later and you are enabling direct traffic forwarding on those remote leaf switches. This option enables an OSPF instance using VLAN-5 for multipod, which is not needed in this case. See About Direct Traffic Forwarding for more information.

6. Choose the overlay-1 VRF.

Step 6

Configure the nodes and interfaces leading from the remote leaf switches to the WAN router, with the following steps:

1. In the Create Routed Outside panel, click the + on the Nodes and Interfaces Protocol Profiles table.

2. Click the + on the Nodes table and enter the following details:

   • Node ID—ID for the remote leaf that is connected to the WAN router

   • Router ID—IP address for the WAN router

   • External Control Peering—only enable if the remote leaf switches are being added to a pod in a multipod fabric

3. Click OK.

4. Click on the + on OSPF Interface Profiles, and configure the following details for the routed sub-interface used to connect a remote leaf switch with the WAN router.

   • Identity—Name of the OSPF interface profile

   • Protocol Profiles—A previously configured OSPF profile or create one

   • Interfaces—On the Routed Sub-Interface tab, the path and IP address for the routed sub-interface leading to the WAN router

Step 7

Configure the Fabric External Connection Profile, with the following steps:

1. Navigate to Tenants > infra > Policies > Protocol.

2. Right-click Fabric Ext Connection Policies and choose Create Intrasite/Intersite Profile.

3. Enter the mandatory Community value in the format provided in the example.

4. Click the + on Fabric External Routing Profile.

5. Enter the name of the profile and add uplink interface subnets for all of the remote leaf switches.

6. Click Update and click Submit.

Step 8

To verify that the remote leaf switches are discovered by the APIC, navigate to Fabric > Inventory > Fabric Membership, or Fabric > Inventory > Pod > Topology.

Step 9

To view the status of the links between the fabric and the remote leaf switches, enter the show ip ospf neighbors vrf overlay-1 command on the spine switch that is connected to the IPN router.

Step 10

To view the status of the remote leaf switches in the fabric, enter the acidiag fnvread NX-OS style command on the APIC using the CLI.

Step 1

Upgrade Cisco APIC and all the nodes in the fabric to Release 4.1(2) or later.

Step 2

Verify that the routes for the Routable Subnet that you wish to configure will be reachable in the Inter-Pod Network (IPN), and that the subnet is reachable from the remote leaf switches.

Step 3

Configure Routable Subnets in all the pods in the fabric:

1. On the menu bar, click Fabric > Inventory.

2. In the Navigation pane, click Pod Fabric Setup Policy.

3. On the Fabric Setup Policy panel, double-click the pod where you want to configure routable subnets.

4. Access the information in the subnets or TEP table, depending on the release of your APIC software:

   • For releases prior to 4.2(3), click the + on the Routable Subnets table.

   • For 4.2(3) only, click the + on the External Subnets table.

   • For 4.2(4) and later, click the + on the External TEP table.

5. Enter the IP address and Reserve Address, if necessary, and set the state to Active or Inactive.

   • The IP address is the subnet prefix that you wish to configure as the routeable IP space.

   • The Reserve Address is a count of addresses within the subnet that must not be allocated dynamically to the spine switches and remote leaf switches. The count always begins with the fist IP in the subnet and increments sequentially. If you wish to allocate the Unicast TEP (covered later in these procedures) from this pool, then it must be reserved.

6. On the Fabric Setup Policy panel, click Submit.

   Note	If you find that you have to make changes to the information in the subnets or TEP table after you've made these configurations, follow the procedures provided in "Changing the External Routable Subnet" in the Cisco APIC Getting Started Guide to make those changes successfully.

Step 4

Add Routable Ucast for each pod:

1. On the menu bar, click Tenants > infra > Policies > Protocol > Fabric Ext Connection Policies > intrasite-intersite_profile_name.

2. In the properties page for this intrasite/intersite profile, click + in the Pod Connection Profile area.

   The Create Pod Connection Profile window appears.

3. Select a pod and enter the necessary information in the Create Pod Connection Profile window.

   In the Unicast TEP field, enter a routable TEP IP address, including the bit-length of the prefix, to be used for unicast traffic over the IPN. This IP address is used by the spine switches in their respective pod for unicast traffic in certain scenarios. For example, a unicast TEP is required for remote leaf switch direct deployments.

Step 5

Click Submit.

Step 6

Verify that COOP is configured correctly.

# show coop internal info global
# netstat -anp | grep 5000

Step 7

Verify that the BGP route reflector session in the remote leaf switch is configured correctly.

 
remote-leaf# show bgp vpnv4 unicast summary vrf all | grep 14.0.0
 14.0.0.227 4 100 1292 1164 395 0 0 19:00:13 52 
 14.0.0.228 4 100 1296 1164 395 0 0 19:00:10 52 

Step 8

Enable direct traffic forwarding on the remote leaf switches.

1. On the menu bar, click System > System Settings.

2. Click Fabric Wide Setting.

3. Click the check box on Enable Remote Leaf Direct Traffic Forwarding.

   When this is enabled, the spine switches will install Access Control Lists (ACLs) to prevent traffic coming from remote leaf switches from being sent back, since the remote leaf switches will now send directly between each remote leaf switches' TEPs. There may be a brief disruption in service while the tunnels are built between the remote leaf switches.

4. Click Submit.

5. To verify that the configuration was set correctly, on the spine switch, enter the following command:

   spine# cat /mit/sys/summary

   You should see the following highlighted line in the output, which is verification that the configuration was set correctly (full output truncated):

   
   ...
   podId : 1
   remoteNetworkId : 0
   remoteNode : no
   rldirectMode : yes
   rn : sys
   role : spine
   ...
   

Step 9

Add the Routable IP address of Cisco APIC as DHCP relay on the IPN interfaces connecting the remote leaf switches.

remote-leaf# moquery -c infraWiNode | grep routable

Step 10

Decommission and recommission each remote leaf switch one at a time to get it discovered on the routable IP address for the Cisco APIC.

Step 1

For a multipod configuration, configure a multipod-internal L3Out using VLAN-5.

Step 2

Provision back private network reachability if it was removed when you enabled the direct traffic forwarding feature on the remote leaf switches.

Step 3

Disable remote leaf switch direct traffic forwarding for all remote leaf switches by posting the following policy:

POST URL : https://<ip address>/api/node/mo/uni/infra/settings.xml
<imdata>
    <infraSetPol dn="uni/infra/settings" enableRemoteLeafDirect="no" />
</imdata>

Step 4

Remove the Routable Subnets and Routable Ucast from the pods in the fabric.

Step 5

Decommission and recommission each remote leaf switch to get it discovered on the non-routable internal IP address of the Cisco APIC.

Step 6

Downgrade the Cisco APIC and all the nodes in the fabric to a release prior to 4.1(2).