Route peering is a special case of the more generic Cisco Application Centric Infrastructure (ACI) fabric as a transit use case, in which route peering enables the ACI fabric to serve as a transit domain for Open Shortest Path First (OSPF) or Border Gateway Protocol (BGP) protocols. A common use case for route peering is route health injection, in which the server load balancing virtual IP is advertised over OSPF or internal BGP (iBGP) to clients that are outside of the ACI fabric. You can use route peering to configure OSPF or BGP peering on a service device so that the device can peer and exchange routes with the ACI leaf node to which it is connected.

The following protocols are supported for route peering:

• OSPF

• OSPFv3

• iBGPv4

• iBGPv6

• Static routes

The following figure shows how route peering is commonly deployed:

Figure 1. Common Route Peering Topology

As shown in the figure, a Web server's public IP address is advertised to an external router through a firewall by deploying a service graph with route peering configured. You must deploy OSPF routing policies on each leg of the firewall. This is typically done by deploying l3extOut policies. This enables the Web server reachability information to be advertised over OSPF through the firewall to the border leaf and to the external router.

Route distribution between leaves in the fabric is internally accomplished over Multi-Protocol Border Gateway Protocol (MP-BGP).

For a more detailed example of the route peering topology, see Route Peering End-to-End Flow.

For more information about configuring l3extOut policies, see the Cisco Application Centric Infrastructure Fundamentals Guide.

Note	Point-to-point non-broadcast mode is not supported on an Adaptive Security Appliance (ASA). You must remove the point-to-point non-broadcast mode configuration from the Application Policy Infrastructure Controller (APIC) if the configuration exists.

You can configure route peering using internal Border Gateway Protocol (iBGP) on the device’s external interface and static routes on the internal interface. You cannot configure iBGP on both the internal and external interfaces of the device without extra configuration, as the interfaces must be in different autonomous systems and inter-autonomous system redistribute policies do not get pushed down.

The following figure illustrates the route peering object relations:

Figure 3. iBGP Route Peering Object Relations

1. vnsLDevCtx—Device selection policy.

2. l3extOut—Contains all BGP policies for a single autonomous system.

3. bgpCtxPol—Per-context BGP timers.

4. bgpExtP—BGP per ASN policy.

5. l3extLIfP/l3extLNodeP—Controls to which nodes or ports these endpoint groups (EPGs) are deployed.

6. l3extSubnet—Subnets to export from and import into the fabric.

7. l3extInstP—Prefix-based EPG.

The following policy configures iBGPv4/v6 on the external interface:

<polUni>
  <fvTenant name="common">
    <fvCtx name="commonctx">
      <fvRsBgpCtxPol tnBgpCtxPolName="timer-3-9"/>
      <fvRsCtxToExtRouteTagPol tnL3extRouteTagPolName="myTagPol"/>
    </fvCtx>
    <l3extRouteTagPol tag="212" name="myTagPol"/>
    <bgpCtxPol grCtrl="helper" holdIntvl="9" kaIntvl="3" name="timer-3-9" staleIntvl="30"/>
    <l3extOut name="BgpExternal" status="created,modified">
      <l3extLNodeP name="bLeaf-101">
        <!-- <bgpPeerP addr="40.40.40.102/32 "ctrl="send-com"/> -->
        <l3extRsNodeL3OutAtt tDn="topology/pod-1/node-101" rtrId="180.0.0.8/28>"
          <l3extLoopBackIfP addr="50.50.50.100/32"/>
        </l3extRsNodeL3OutAtt>
        <l3extLIfP name="portIf">
          <l3extRsPathL3OutAtt tDn="topology/pod-1/paths-101/pathep-[eth1/23]"
            ifInstT="ext-svi" encap="vlan-3843" addr="40.40.40.100/28 "mtu="1500">
            <bgpPeerP addr="40.40.40.102/32 "ctrl="send-com"/>
          </l3extRsPathL3OutAtt>
        </l3extLIfP>
      </l3extLNodeP>
      <bgpExtP/>
      <l3extInstP name="ExtInstP">
        <l3extSubnet ip="40.40.40.100/28 "scope="import-security"/>
        <l3extSubnet ip="10.10.10.0/24 "scope="import-security"/>
        <l3extSubnet ip="20.20.20.0/24 "scope="export-rtctrl"/>
      </l3extInstP>
      <l3extRsEctx tnFvCtxName="commonctx"/>
    </l3extOut>
  </fvTenant>
</polUni>

iBGP peers can be configured at the physical interface level or the loopback level. The following example shows a iBGP peer configured at the physical interface level:

<l3extLIfP name="portIf">
          <l3extRsPathL3OutAtt tDn="topology/pod-1/paths-101/pathep-[eth1/23]"
            ifInstT="ext-svi" encap="vlan-3843" addr="40.40.40.100/28 "mtu="1500">
            <bgpPeerP addr="40.40.40.102/32 "ctrl="send-com"/>
          </l3extRsPathL3OutAtt>
</l3extLIfP>

In this case, the iBGP process that is running on the fabric uses the switch virtual interface (SVI) IP address 40.40.40.100/28 to peer with its neighbor. The neighbor is the service device at IP address 40.40.40.102/32.

In the following example, the iBGP peer definition has been moved to the logical node level (under l3extLNodeP) and a loopback interface has been configured:

      <l3extLNodeP name="bLeaf-101">
        <bgpPeerP addr="40.40.40.102/32 "ctrl="send-com"/>
        <l3extRsNodeL3OutAtt tDn="topology/pod-1/node-101" rtrId="180.0.0.8/28>"
          <l3extLoopBackIfP addr="50.50.50.100/32"/>
        </l3extRsNodeL3OutAtt>
        <l3extLIfP name="portIf">
          <l3extRsPathL3OutAtt tDn="topology/pod-1/paths-101/pathep-[eth1/23]"
            ifInstT="ext-svi" encap="vlan-3843" addr="40.40.40.100/28 "mtu="1500">
          </l3extRsPathL3OutAtt>
        </l3extLIfP>
      </l3extLNodeP>

In this case, the iBGP process uses the loopback address to peer with its neighbor. If no loopback is configured, then the fabric uses the IP address that is specified by rtrId to peer with the neighbor.

In such cases, the device needs a route to reach the SVI. This is typically configured using graph parameters, as shown by the following example for ASA, where IP address 50.50.50.0 is reachable from IP address 40.40.40.100:

<vnsAbsFolder name="ExtRouteCfg" key="StaticRoute">
              <vnsAbsFolder name="route1" key="route">
                <vnsAbsParam name="network" key="network" value="50.50.50.0"/>
                <vnsAbsParam name="netmask" key="netmask" value="255.255.255.0"/>
                <vnsAbsParam name="gateway" key="gateway" value="40.40.40.100"/>
              </vnsAbsFolder>
              <vnsAbsFolder name="route2" key="ipv6_route">
                <vnsAbsParam name="prefix" key="prefix" value="2005::/64"/>
                <vnsAbsParam name="gateway" key="gateway" value="2004::2828:2866"/>
              </vnsAbsFolder>
</vnsAbsFolder>

The following example shows static route configuration on the fabric for the internal interface of the device:

<polUni>
  <fvTenant name="tenant11">
    <l3extOut name="StaticInternal" status="created,modified">
      <l3extLNodeP name="bLeaf-201">
        <l3extRsNodeL3OutAtt tDn="topology/pod-1/node-101" rtrId="180.0.0.11>"
          <ipRouteP ip="20.20.20.0/24>"
            <ipNexthopP nhAddr="30.30.30.102/32"/>
          </ipRouteP>
        </l3extRsNodeL3OutAtt>
        <l3extLIfP name="portIf">
          <l3extRsPathL3OutAtt tDn="topology/pod-1/paths-101/pathep-[eth1/25]"
            ifInstT="ext-svi" encap="vlan-3844" addr="30.30.30.100/28 "mtu="1500"/>
        </l3extLIfP>
      </l3extLNodeP>
      <l3extInstP name="IntInstP">
        <l3extSubnet ip="20.20.20.0/24 "scope="import-security"/>
      </l3extInstP>
      <l3extRsEctx tnFvCtxName="tenant1ctx1"/>
    </l3extOut>
  </fvTenant>
</polUni>

The following figure shows how route peering works end-to-end.

Figure 4. Route Peering End-to-End Flow

The figure shows an example two leaf, single spine topology where a Linux web server's IP address is advertised to an external router using route peering. The Linux web server is at IP address 10.10.10.101/24 and is hosted on an ESX server that is connected to leaf1. A regular bridge domain-based endpoint group (EPG) is deployed to represent traffic that originates from the web server.

You deploy a service graph that comprises a two-arm routable firewall, with both arms being connected to leaf1. There is a virtual routing and forwarding (VRF)-split on the firewall device, meaning that each arm of the firewall is connected to the leaf in a different VRF (context). The VRF-split is necessary to ensure that traffic is routed through the service device, rather than being short-circuited by the leaf. The external traffic is represented by an l3extOut (L3OutInternet) that is deployed on leaf2. leaf2 can be viewed as a fabric border-leaf in this scenario. You deploy a contract between L3OutInternet and the web server EPG. This contract is associated with a service graph that encompasses the firewall device.

To publish the web server route to the external world, you deploy two l3extOuts—L3OutExternal and L3OutInternal—to the leaf ports to which the service device is connected. As a result, Open Shortest Path First (OSPF) peering sessions are established between the leaf and the firewall in both of the contexts (commonctx and tenant1ctx1). The export attribute on these l3extOuts control how the routing information is advertised to the border leaf. Routes are exchanged internally between the fabric leaves using Multiprotocol Border Gateway Protocol (MP-BGP) redistribution.

Ultimately, the web server route is advertised to the external router (IP address 20.20.20.102) using a separate OSPF session. This results in the external router being able to ping the web server without any manual static route configuration.

You can deploy the Cisco Application Centric Infrastructure (ACI) fabric as a transit routing domain, which is useful when the ACI point of delivery (POD) serves as a transit routing domain between other PODs. In the following illustration, two external l3extOuts—L3OutInternet and L3OutInternet2—are deployed on two border leaves. There is a contract associated between these l3extOuts, and the contract is attached to a single node service graph containing a firewall service device.

Figure 5. ACI Fabric Serving As a Transit Routing Domain

Two additional l3extOuts are deployed on the external and internal legs of the firewall device to establish Open Shortest Path First (OSPF) peering sessions between them. By appropriately configuring the import security control (the import-security attribute), you can control which routes are allowed to transit the ACI fabric to the border leaves.

You must perform the following tasks to configure route peering:

1. Create a static VLAN pool that will be used for the encapsulation VLAN between the device and the Cisco Application Centric Infrastructure (ACI) fabric.

   See Creating a Static VLAN Pool Using the GUI.

2. Create an external routed domain that will tie together the location (leaf node/path) of the device and the VLAN pool.

   See Creating an External Routed Domain Using the GUI.

3. Create an external routed network, which is used to specify the routing configuration in the ACI fabric for route peering.

   See Creating an External Routed Network Using the GUI.

4. Create a new router configuration to specify the router ID that will be used on the device.

   See Creating a Router Configuration Using the GUI.

5. Create a service graph association, which involves associating the external routed network policy and router configuration with a device selection policy.

   See Creating a Service Graph Association Using the GUI.

apic1# configure

apic1(config)# tenant 101

apic1(config-tenant)# l4l7 graph g1 contract c1

apic1(config-graph)# service ASA_FW device-cluster-tenant 101 device-cluster ASA_FW1

apic1(config-service)# connector consumer cluster-interface provider

apic1(config-connector)# l4l7-peer tenant 101 out l101 epg e101 redistribute bgp
apic1(config-connector)# exit

apic1(config-service)# connector provider cluster-interface consumer
apic1(config-connector)# l4l7-peer tenant 101 out l101 epg e101 redistribute bgp
apic1(config-connector)# exit

apic1(config-connector)# no l4l7-peer tenant 101 out l101 epg e101 redistribute bgp

apic1(config)# tenant 102
apic1(config-tenant)# rtr-cfg bgp1
apic1(config-router)# router-id 1.2.3.5
apic1(config-router)# exit

apic1(config-tenant)# l4l7 graph g2 contract c2 subject http
apic1(config-graph)# service ASA_FW device-cluster-tenant 102 device-cluster ASA_FW2
apic1(config-service)# rtr-cfg bgp1
apic1(config-service)# exit
apic1(config-graph)# exit

apic1(config-tenant)# external-l3 epg e101 l3out l101
apic1(config-tenant-l3ext-epg)# vrf member v101
apic1(config-tenant-l3ext-epg)# match ip 101.101.1.0/24
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant 101 vrf v101 l3out l101
apic1(config-leaf-vrf)# ip route 101.101.1.0/24 99.1.1.2
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# interface ethernet 1/10
apic1(config-leaf-if)# vrf member tenant 101 vrf v101 l3out l101
apic1(config-leaf-if)# vlan-domain member dom101
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# ip address 99.1.1.1/24
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

For the complete configuration for Layer 3 external connectivity (Layer 3 outside) using the named mode, including routing protocols (BGP, OSPF) and route maps, see the Cisco APIC NX-OS Style CLI Command Reference document.

Note	The external Layer 3 configuration in the CLI is available in two modes: basic mode and named mode. For a given tenant or VRF, user only one of these modes for all external Layer 3 configuration. Route peering is supported only in the named mode.

If your Cisco Application Centric Infrastructure (ACI) fabric has a route peering or data traffic issue, there are several commands that you can run on ACI fabric leaves to troubleshoot the issue.

The following table provides troubleshooting commands that you can run in the switch shell on the fabric leaf.

The following table provides a troubleshooting command that you can run in the vsh_lc shell.

In addition to the shell commands, you can check the following things to help with troubleshooting:

• Health count on the device

• All of the faults and NwIssues under a particular tenant