Routed Connectivity to External Networks

This chapter contains the following sections:

About Routed Connectivity to Outside Networks

A Layer 3 outside network configuration (L3Out) defines how traffic is forwarded outside of the fabric. Layer 3 is used to discover the addresses of other nodes, select routes, select quality of service, and forward the traffic that is entering, exiting, and transiting the fabric.


Note

For guidelines and cautions for configuring and maintaining Layer 3 outside connections, see Guidelines for Routed Connectivity to Outside Networks.

For information about the types of L3Outs, see External Layer 3 Outside Connection Types.

Layer 3 Out for Routed Connectivity to External Networks

Routed connectivity to external networks is enabled by associating a fabric access (infraInfra) external routed domain (l3extDomP) with a tenant Layer 3 external instance profile (l3extInstP or external EPG) of a Layer 3 external outside network (l3extOut), in the hierarchy in the following diagram:

Figure 1. Policy Model for Layer 3 External Connections

A Layer 3 external outside network (l3extOut object) includes the routing protocol options (BGP, OSPF, or EIGRP or supported combinations) and the switch-specific and interface-specific configurations. While the l3extOut contains the routing protocol (for example, OSPF with its related Virtual Routing and Forwarding (VRF) and area ID), the Layer 3 external interface profile contains the necessary OSPF interface details. Both are needed to enable OSPF.

The l3extInstP EPG exposes the external network to tenant EPGs through a contract. For example, a tenant EPG that contains a group of web servers could communicate through a contract with the l3extInstP EPG according to the network configuration contained in the l3extOut. The outside network configuration can easily be reused for multiple nodes by associating the nodes with the L3 external node profile. Multiple nodes that use the same profile can be configured for fail-over or load balancing. Also, a node can be added to multiple l3extOuts resulting in VRFs that are associated with the l3extOuts also being deployed on that node. For scalability information, refer to the current Verified Scalability Guide for Cisco ACI.

Advertise Host Routes

Enabling Advertise Host Routes on the BD, individual host-routes (/32 and /128 prefixes) are advertised from the Border-Leaf switches (BL). The BD must be associated to the L3out or an explicit prefix list matching the host routes. The host routes must be configured to advertise host routes out of the fabric.

Border-Leaf switches along with the subnet advertise the individual end-point(EP) prefixes. The route information is advertised only if the host is connected to the local POD. If the EP is moved away from the local POD or once the EP is removed from EP database (even if the EP is attached to a remote leaf), the route advertisement is then withdrawn.

Advertise Host Route configuration guidelines and limitations are:

  • When host routes are advertised, the VRF Transit Route Tag is set in order to prevent them from being advertised back into the fabric and installed. In order for this loop protection to work properly, external routers must preserve this route-tag if advertising to another L3Out.

  • If a bridge domain is tied to an EPG that has the same subnet configured for internal leaking, you must also enable the "Advertised Externally" flag on the EPG subnet.

  • The Advertise Host Routes feature is supported on Generation 2 switches or later (Cisco Nexus N9K switches with "EX", "FX", or "FX2" on the end of the switch model name or later; for example, N9K-93108TC-EX).

  • Enabling PIMv4 (Protocol-Independent Multicast, version 4) and Advertise Host routes on a BD is not supported.

  • Host route advertisement supports both BD to L3out Association and the explicit route map configurations. We recommend using explicit route map configuration which allows you greater control in selecting individual or a range of host routes to configure.

  • EPs/Host routes in SITE-1 will not be advertised out through Border Leafs in other SITEs.

  • When EPs is aged out or removed from the database, Host routes are withdrawn from the Border Leaf.

  • When EP is moved across SITEs or PODs, Host routes should be withdrawn from first SITE/POD and advertised in new POD/SITE.

  • EPs learned on a specific BD, under any of the BD subnets are advertised from the L3out on the border leaf in the same POD.

  • EPs are advertised out as Host Routes only in the local POD through the Border Leaf.

  • Host routes are not advertised out from one POD to another POD.

  • In the case of Remote Leaf, if EPs are locally learned in the Remote Leaf, they are then advertised only through a L3out deployed in Remote Leaf switches in same POD.

  • EPs/Host routes in a Remote Leaf are not advertised out through Border Leaf switches in main POD or another POD.

  • EPs/Host routes in the main POD are not advertised through L3out in Remote Leaf switches of same POD or another POD.

  • The BD subnet must have the Advertise Externally option enabled.

  • The BD must be associated to an L3out or the L3out must have explicit route-map configured matching BD subnets.

  • There must be a contract between the EPG in the specified BD and the External EPG for the L3out.


    Note

    If there is no contract between the BD/EPG and the External EPG the BD subnet and host routes will not be installed on the border leaf.

  • Advertise Host Route is supported for shared services. For example: epg1/BD1 deployed is in VRF-1 and L3out in another VRF-2. By providing shared contract between EPG and L3out host routes are pulled from one VRF-1 to another VRF-2.

  • When Advertise Host Route is enabled on BD custom tag cannot be set on BD Subnet using route-map.

  • When Advertise Host Route is enabled on a BD and the BD is associated with an L3Out, BD subnet is marked public. If there's a rogue EP present under the BD, that EP is advertised out on L3Out.

Guidelines for Routed Connectivity to Outside Networks

Use the following guidelines when creating and maintaining Layer 3 outside connections.

Topic

Caution or Guideline

Issue where a border leaf switch in a vPC pair forwards a BGP packet with an incorrect VNID to an on-peer learned endpoint

If the following conditions exist in your configuration:

  • Two leaf switches are part of a vPC pair

  • For the two leaf switches connected behind the L3Out, the destination endpoint is connected to the second (peer) border leaf switch, and the endpoint is on-peer learned on that leaf switch

If the endpoint is on-peer learned on the ingress leaf switch that receives a BGP packet that is destined to the on-peer learned endpoint, an issue might arise where the transit BGP connection fails to establish between the first layer 3 switch behind the L3Out and the on-peer learned endpoint on the second leaf switch in the vPC pair. This might happen in this situation because the transit BGP packet with port 179 is forwarded incorrectly using the bridge domain VNID instead of the VRF VNID.

To resolve this issue, move the endpoint to any other non-peer leaf switch in the fabric so that it is not learned on the leaf switch.

Border leaf switches and GIR (maintenance) mode

If a border leaf switch has a static route and is placed in Graceful Insertion and Removal (GIR) mode, or maintenance mode, the route from the border leaf switch might not be removed from the routing table of switches in the ACI fabric, which causes routing issues.

To work around this issue, either:

  • Configure the same static route with the same administrative distance on the other border leaf switch, or

  • Use IP SLA or BFD for track reachability to the next hop of the static route

Updates through CLI

For Layer 3 external networks created through the API or GUI and updated through the CLI, protocols need to be enabled globally on the external network through the API or GUI, and the node profile for all the participating nodes needs to be added through the API or GUI before doing any further updates through the CLI.

Loopbacks for Layer 3 networks on same node

When configuring two Layer 3 external networks on the same node, the loopbacks need to be configured separately for both Layer 3 networks.

Ingress-based policy enforcement

Starting with Cisco APIC release 1.2(1), ingress-based policy enforcement enables defining policy enforcement for Layer 3 Outside (L3Out) traffic for both egress and ingress directions. The default is ingress. During an upgrade to release 1.2(1) or higher, existing L3Out configurations are set to egress so that the behavior is consistent with the existing configuration. You do not need any special upgrade sequence. After the upgrade, you change the global property value to ingress. When it has been changed, the system reprograms the rules and prefix entries. Rules are removed from the egress leaf and installed on the ingress leaf, if not already present. If not already configured, an Actrl prefix entry is installed on the ingress leaf. Direct server return (DSR), and attribute EPGs require ingress based policy enforcement. vzAny and taboo contracts ignore ingress based policy enforcement. Transit rules are applied at ingress.

Bridge Domains with L3Outs

A bridge domain in a tenant can contain a public subnet that is advertised through an l3extOut provisioned in the common tenant.

Bridge domain route advertisement For OSPF and EIGRP

When both OSPF and EIGRP are enabled on the same VRF on a node and if the bridge domain subnets are advertised out of one of the L3Outs, it will also get advertised out of the protocol enabled on the other L3Out.

For OSPF and EIGRP, the bridge domain route advertisement is per VRF and not per L3Out. The same behavior is expected when multiple OSPF L3Outs (for multiple areas) are enabled on the same VRF and node. In this case, the bridge domain route will be advertised out of all the areas, if it is enabled on one of them.

BGP Maximum Prefix Limit

Starting with Cisco APIC release 1.2(1x), tenant policies for BGP l3extOut connections can be configured with a maximum prefix limit, that enables monitoring and restricting the number of route prefixes received from a peer. Once the maximum prefix limit has been exceeded, a log entry is recorded, and further prefixes are rejected. The connection can be restarted if the count drops below the threshold in a fixed interval, or the connection is shut down. Only one option can be used at a time. The default setting is a limit of 20,000 prefixes, after which new prefixes are rejected. When the reject option is deployed, BGP accepts one more prefix beyond the configured limit, before the APIC raises a fault.

MTU

Note

 

Cisco ACI does not support IP fragmentation. Therefore, when you configure Layer 3 Outside (L3Out) connections to external routers, or Multi-Pod connections through an Inter-Pod Network (IPN), it is recommended that the interface MTU is set appropriately on both ends of a link. On some platforms, such as Cisco ACI, Cisco NX-OS, and Cisco IOS, the configurable MTU value does not take into account the Ethernet headers (matching IP MTU, and excluding the 14-18 Ethernet header size), while other platforms, such as IOS-XR, include the Ethernet header in the configured MTU value. A configured value of 9000 results in a max IP packet size of 9000 bytes in Cisco ACI, Cisco NX-OS, and Cisco IOS, but results in a max IP packet size of 8986 bytes for an IOS-XR untagged interface.

For the appropriate MTU values for each platform, see the relevant configuration guides.

We highly recommend that you test the MTU using CLI-based commands. For example, on the Cisco NX-OS CLI, use a command such as ping 1.1.1.1 df-bit packet-size 9000 source-interface ethernet 1/1.

Layer 4 to Layer 7

When you are using a multinode service graph, you must have the two EPGs in separate VRF instances. For these functions, the system must do a Layer 3 lookup, so the EPGs must be in separate VRFs. This limitation follows legacy service insertion, based on Layer 2 and Layer 3 lookups.

QoS for L3Outs

To configure QoS policies for an L3Out and enable the policies to be enforced on the BL switch where the L3Out is located, use the following guidelines:

  • The VRF Policy Control Enforcement Direction must be set toEgress.

  • The VRF Policy Control Enforcement Preference must be set to Enabled.

  • When configuring the contract that controls communication between the EPGs using the L3Out, include the QoS class or Target DSCP in the contract or subject of the contract.

ICMP settings

ICMP redirect and ICMP unreachable are disabled by default in Cisco ACI to protect the switch CPU from generating these packets.

Configuring Layer 3 Outside for Tenant Networks

Configuring a Tenant Layer 3 Outside Network Connection Overview

This topic provides a typical example of how to configure a Layer 3 Outside for tenant networks when using Cisco APIC.

The examples in this chapter use the following topology:

Figure 2. Layer 3 External Connections Topology

In this example, the Cisco ACI fabric has 3 leaf switches and two spine switches, that are controlled by an APIC cluster. The nonborder leaf switches (101 and 102) are connected to a web server and a database server. The border leaf switch (103) has an L3Out on it providing connection to a router and thus to the Internet. The goal of this example is to enable the web server to communicate through the L3Out on the border leaf switch to an endpoint (EP) on the Internet.

In this example, the tenant that is associated with the L3Out is t1, with VRF v1, and L3Out external EPG, extnw1.

Before configuring an L3Out, configure the node, port, functional profile, AEP, and Layer 3 domain. You must also configure the spine switches 104 and 105 as BGP route reflectors.

Configuring the L3Out includes defining the following components:

  1. Tenant and VRF

  2. Node and interface on leaf 103

  3. Primary routing protocol (used to exchange routes between border leaf switch and external routers; in this example, BGP)

  4. Connectivity routing protocol (provides reachability information for the primary protocol; in this example, OSPF)

  5. External EPG

  6. Route map

  7. Bridge domain

  8. At least one application EPG on node 101

  9. Filters and contracts

  10. Associate the contracts with the EPGs

The following table lists the names that are used in the examples in this chapter:

Property

Node 103 (Border Leaf)

Node 101 (Non-Border Leaf)

Tenant

t1

t1

VRF

v1

v1

Layer 3 Outside

l3out1

--

Bridge domain

--

bd1 with subnet 44.44.44.1/24

Node

Node 103, with profile nodep1 with router ID 11.11.11.103 and path through 12.12.12.3/24

Node 101

Interface

OSPF interface ifp1 at eth/1/3 with IP address 11.11.11.1/24

--

BGP details

Peer address 15.15.15.2/24 and ASN 100

--

OSPF details

OSPF area 0.0.0.0 and type Regular

--

EPG

External EPG extnw1 at 20.20.20.0/24

Application app1 with epg1, with bd1

Route Control Profile

rp1 with a route control context ctxp1

--

Route map

map1 with rule match-rule1 with a route destination 200.3.2.0/24

--

Filter

http-filter

http-filter

Contract

httpCtrct provided by extnw1

httpCtrct consumed by epg1

Configuring Layer 3 Outside for Tenant Networks Using the REST API

The external routed network that is configured in the example can also be extended to support both IPv4 and IPv6. Both IPv4 and IPv6 routes can be advertised to and learned from the external routed network. To configure an L3Out for a tenant network, send a post with XML such as the example.

This example is broken into steps for clarity. For a merged example, see REST API Example: L3Out.

Before you begin

  • Configure the node, port, functional profile, AEP, and Layer 3 domain.

  • Create the external routed domain and associate it to the interface for the L3Out.

  • Configure a BGP route reflector policy to propagate the routes within the fabric.

For an XML example of these prerequisites, see REST API Example: L3Out Prerequisites.

Procedure

Step 1

Configure the tenant, VRF, and bridge domain.

This example configures tenant t1 with VRF v1 and bridge domain bd1. The tenant, VRF, and BD are not yet deployed.

Example:

<fvTenant  name="t1">
    <fvCtx name="v1"/>
    <fvBD name="bd1">
        <fvRsCtx tnFvCtxName="v1"/>
        <fvSubnet ip="44.44.44.1/24" scope="public"/>
        <fvRsBDToOut tnL3extOutName="l3out1"/>
    </fvBD>/>
</fvTenant>

Step 2

Configure an application profile and application EPG.

This example configures application profile app1 (on node 101), EPG epg1, and associates the EPG with bd1 and the contract httpCtrct, as the consumer.

Example:

<fvAp name="app1">
    <fvAEPg name="epg1">
        <fvRsDomAtt instrImedcy="immediate" tDn="uni/phys-dom1"/>
        <fvRsBd tnFvBDName="bd1" />
        <fvRsPathAtt encap="vlan-2011" instrImedcy="immediate" mode="regular" tDn="topology/pod-1/paths-101/pathep-[eth1/3]"/>
        <fvRsCons tnVzBrCPName="httpCtrct"/>
    </fvAEPg>
</fvAp>

Step 3

Configure the node and interface.

This example configures VRF v1 on node 103 (the border leaf switch), with the node profile, nodep1, and router ID 11.11.11.103. It also configures interface eth1/3 as a routed interface (Layer 3 port), with IP address 12.12.12.1/24 and Layer 3 domain dom1.

Example:

<l3extOut name="l3out1">
    <l3extRsEctx tnFvCtxName="v1"/>
    <l3extLNodeP name="nodep1">
        <l3extRsNodeL3OutAtt rtrId="11.11.11.103" tDn="topology/pod-1/node-103"/>
        <l3extLIfP name="ifp1"/>
         <l3extRsPathL3OutAtt addr="12.12.12.3/24" ifInstT="l3-port" tDn="topology/pod-1/paths-103/pathep-[eth1/3]"/>
        </l3extLIfP>
    </l3extLNodeP>
    <l3extRsL3DomAtt tDn="uni/l3dom-dom1"/>
</l3extOut>

Step 4

Configure the routing protocol.

This example configures BGP as the primary routing protocol, with a BGP peer with the IP address, 15.15.15.2 and ASN 100.

Example:

<l3extOut name="l3out1">
    <l3extLNodeP name="nodep1">
        <bgpPeerP addr="15.15.15.2">
            <bgpAsP asn="100"/>
        </bgpPeerP>
    </l3extLNodeP>
    <bgpExtP/>
</l3extOut>

Step 5

Configure the connectivity routing protocol.

This example configures OSPF as the communication protocol, with regular area ID 0.0.0.0.

Example:

<l3extOut name="l3out1">
    <ospfExtP areaId="0.0.0.0" areaType="regular"/>
    <l3extLNodeP name="nodep1">
        <l3extLIfP name="ifp1">
            <ospfIfP/>
        <l3extIfP>
    <l3extLNodeP>
</l3extOut>

Step 6

Configure the external EPG.

This example configures the network 20.20.20.0/24 as external network extnw1. It also associates extnw1 with the route control profile rp1 and the contract httpCtrct, as the provider.

Example:

<l3extOut name="l3out1">
    <l3extInstP name="extnw1">
        <l3extSubnet ip="20.20.20.0/24" scope="import-security"/>
        <fvRsProv tnVzBrCPName="httpCtrct"/>
    </l3extInstP> 
</l3extOut>

Step 7

Optional. Configure a route map.

This example configures a route map for the BGP peer in the outbound direction. The route map is applied for routes that match a destination of 200.3.2.0/24. Also, on a successful match (if the route matches this range) the route AS PATH attribute is updated to 200 and 100.

Example:

<fvTenant name="t1">
    <rtctrlSubjP name="match-rule1">
         <rtctrlMatchRtDest ip="200.3.2.0/24"/>
    </rtctrlSubjP>
    <l3extOut name="l3out1">
        <rtctrlProfile name="rp1">
            <rtctrlCtxP name="ctxp1" action="permit" order="0">
                <rtctrlScope>
                    <rtctrlRsScopeToAttrP tnRtctrlAttrPName="attrp1"/>
                </rtctrlScope>
                <rtctrlRsCtxPToSubjP tnRtctrlSubjPName="match-rule1"/>
            </rtctrlCtxP>
        </rtctrlProfile>
        <l3extInstP name="extnw1">
            <l3extSubnet ip="20.20.20.0/24" scope="import-security"/>
            <l3extRsInstPToProfile direction='export' tnRtctrlProfileName="rp1"/>
            <fvRsProv tnVzBrCPName="httpCtrct"/>
        </l3extInstP>
    </l3extOut>
</fvTenant>

Step 8

This example creates filters and contracts to enable the EPGs to communicate. The external EPG and the application EPG are already associated with the contract httpCtrct as provider and consumer respectively. The scope of the contract (where it is applied) can be within the application profile, the tenant, the VRF, or it can be used globally (throughout the fabric). In this example, the scope is the VRF (context).

Example:

<vzFilter name="http-filter">
    <vzEntry  name="http-e" etherT="ip" prot="tcp"/>
</vzFilter>
<vzBrCP name="httpCtrct" scope="context">
    <vzSubj name="subj1">
        <vzRsSubjFiltAtt tnVzFilterName="http-filter"/>
    </vzSubj>
</vzBrCP>

Step 9

Configure Advertise Host Routes.

Example:

"<fvBD dn="uni/tn-t1/BD-b100" hostBasedRouting="yes"/>”

REST API Example: L3Out Prerequisites

This example configures the node, port, functional profile, AEP, and Layer 3 domain:

<?xml version="1.0" encoding="UTF-8"?>
<!-- api/policymgr/mo/.xml -->
<polUni>
    <infraInfra>
        <!-- Node profile -->
        <infraNodeP name="nodeP1">
            <infraLeafS name="leafS1" type="range">
                <infraNodeBlk name="NodeBlk1" from_="101" to_="103" />
            </infraLeafS>
            <infraRsAccPortP tDn="uni/infra/accportprof-PortP1" />
        </infraNodeP>
        <!-- Port profile -->
        <infraAccPortP name="PortP1">
            <!-- 12 regular ports -->
            <infraHPortS name="PortS1" type="range">
                <infraPortBlk name="portBlk1" fromCard="1" toCard="1" fromPort="3" toPort="32"/>
                <infraRsAccBaseGrp tDn="uni/infra/funcprof/accportgrp-default" />
            </infraHPortS>
        </infraAccPortP>
        <!-- Functional profile -->
        <infraFuncP>
            <!-- Regular port group -->
            <infraAccPortGrp name="default">
                <infraRsAttEntP tDn="uni/infra/attentp-aeP1" />
            </infraAccPortGrp>
        </infraFuncP>
        <infraAttEntityP name="aeP1">
            <infraRsDomP tDn="uni/phys-dom1"/>
            <infraRsDomP tDn="uni/l3dom-dom1/>
        </infraAttEntityP>
        <fvnsVlanInstP name="vlan-1024-2048" allocMode="static">
             <fvnsEncapBlk name="encap" from="vlan-1024" to="vlan-2048" status="created"/>
        </fvnsVlanInstP>
    </infraInfra>
    <physDomP dn="uni/phys-dom1" name="dom1">
         <infraRsVlanNs tDn="uni/infra/vlanns-[vlan-1024-2048]-static"/>
    </physDomP>
    <l3extDomP name="dom1">
        <infraRsVlanNs tDn="uni/infra/vlanns-[vlan-1024-2048]-static" />
    </l3extDomP>
</polUni>

The following example configures the required BGP route reflectors:

 <!-- Spine switches 104 and 105 are configured as route reflectors -->
<?xml version="1.0" encoding="UTF8"?>
<!-- api/policymgr/mo/.xml -->
<polUni>
    <bgpInstPol name="default">
        <bgpAsP asn="100"/>
        <bgpRRP>
            <bgpRRNodePEp id="104"/>
            <bgpRRNodePEp id="105"/>
        </bgpRRP>  
    </bgpInstPol>
    <fabricFuncP>
        <fabricPodPGrp name="bgpRRPodGrp1">
            <fabricRsPodPGrpBGPRRP tnBgpInstPolName="default"/>
        </fabricPodPGrp>
    </fabricFuncP> 
    <fabricPodP name="default">
        <fabricPodS name="default" type="ALL">
            <fabricRsPodPGrp tDn="uni/fabric/funcprof/podpgrp-bgpRRPodGrp1"/>
        </fabricPodS>
    </fabricPodP>
</polUni>

REST API Example: L3Out

The following example provides a merged version of the steps to configure an L3Out using the REST API.

<?xml version="1.0" encoding="UTF8"?>
<!-- api/policymgr/mo/.xml -->
<polUni>
   <fvTenant name="t1">
       <fvCtx name="v1"/>
       <fvBD name="bd1">
           <fvRsCtx tnFvCtxName="v1"/>
           <fvSubnet ip="44.44.44.1/24" scope="public"/>
           <fvRsBDToOut tnL3extOutName="l3out1"/>
       </fvBD>
       <fvAp name="app1">
           <fvAEPg name="epg1">
               <fvRsDomAtt instrImedcy="immediate" tDn="uni/phys-dom1"/>
               <fvRsBd tnFvBDName="bd1" />
               <fvRsPathAtt encap="vlan-2011" instrImedcy="immediate" mode="regular" tDn="topology/pod-1/paths-101/pathep-[eth1/3]"/>
               <fvRsCons tnVzBrCPName="httpCtrct"/>
           </fvAEPg>
       </fvAp>
       <l3extOut name="l3out1">
           <l3extRsEctx tnFvCtxName="v1"/>
           <l3extLNodeP name="nodep1">
               <l3extRsNodeL3OutAtt rtrId="11.11.11.103" tDn="topology/pod-1/node-103"/>
               <l3extLIfP name="ifp1">
                   <l3extRsPathL3OutAtt addr="12.12.12.3/24" ifInstT="l3-port" tDn="topology/pod-1/paths-103/pathep-[eth1/3]"/>
               </l3extLIfP>
               <bgpPeerP addr="15.15.15.2">
                   <bgpAsP asn="100"/>
               </bgpPeerP>
            </l3extLNodeP>
            <l3extRsL3DomAtt tDn="uni/l3dom-dom1"/>
            <bgpExtP/>
            <ospfExtP areaId="0.0.0.0" areaType="regular"/>
            <l3extInstP name="extnw1" >
                <l3extSubnet ip="20.20.20.0/24" scope="import-security"/>
                <l3extRsInstPToProfile direction="export" tnRtctrlProfileName="rp1"/>
                <fvRsProv tnVzBrCPName="httpCtrct"/>
            </l3extInstP>
            <rtctrlProfile name="rp1">
                <rtctrlCtxP name="ctxp1" action="permit" order="0">
                    <rtctrlScope>
                        <rtctrlRsScopeToAttrP tnRtctrlAttrPName="attrp1"/>
                    </rtctrlScope>
                    <rtctrlRsCtxPToSubjP tnRtctrlSubjPName="match-rule1"/>
                </rtctrlCtxP>
            </rtctrlProfile>
        </l3extOut>
        <rtctrlSubjP name="match-rule1">
            <rtctrlMatchRtDest ip="200.3.2.0/24"/>
        </rtctrlSubjP>
        <rtctrlAttrP name="attrp1">
            <rtctrlSetASPath criteria="prepend">
                <rtctrlSetASPathASN asn="100" order="2"/>
                <rtctrlSetASPathASN asn="200" order="1"/>
            </rtctrlSetASPath>
        </rtctrlAttrP>
        <vzFilter name='http-filter'>
            <vzEntry name="http-e" etherT="ip" prot="tcp"/>
        </vzFilter>
        <vzBrCP name="httpCtrct" scope="context">
            <vzSubj name="subj1">
                <vzRsSubjFiltAtt tnVzFilterName="http-filter"/>
            </vzSubj>
        </vzBrCP>
    </fvTenant>
</polUni>

Configuring a Layer 3 Outside for Tenant Networks Using the NX-OS Style CLI

These steps describe how to configure a Layer 3 outside network for tenant networks. This example shows how to deploy a node and L3 port for tenant VRF external L3 connectivity using the NX-OS CLI.

This example is broken into steps for clarity. For a merged example, see NX-OS Style CLI Example: L3Out.

Before you begin

  • Configure the node, port, functional profile, AEP, and Layer 3 domain.

  • Configure a VLAN domain using the vlan-domain domain and vlan vlan-range commands.

  • Configure a BGP route reflector policy to propagate the routed within the fabric.

For an example using the commands for these prerequisites, see NX-OS Style CLI Example: L3Out Prerequisites.

Procedure

Step 1

Configure the tenant and VRF.

This example configures tenant t1 with VRF v1. They are not yet deployed.

Example:

apic1# configure
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(congig-tenant)# exit
apic1(config)#

Step 2

Configure the node and interface for the L3Out.

This example configures VRF v1 on node 103 (the border leaf switch), which is named nodep1, with router ID 11.11.11.103. It also configures interface eth1/3 as a routed interface (Layer 3 port), with IP address 12.12.12.3/24 and Layer 3 domain dom1.

Example:

apic1(config)# leaf 103
apic1(config-leaf)# vrf context tenant t1 vrf v1 
apic1(config-leaf-vrf)# router-id 11.11.11.103
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1 
apic1(config-leaf-if)# no switchport 
apic1(config-leaf-if)# vrf member tenant t1 vrf v1 
apic1(config-leaf-if)# ip address 12.12.12.3/24
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Step 3

Configure the routing protocol.

This example configures BGP as the primary routing protocol, with a BGP peer address, 15.15.15.2 and ASN 100.

Example:


apic1(config)# leaf 103
apic1(config-leaf)# router bgp 100 
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1 
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit

Step 4

Optional. Configure a connectivity routing protocol.

This example configures OSPF as the communication protocol, with regular area ID 0.0.0.0, with loopback address 30.30.30.0.

Example:


apic1(config)# leaf 103
apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant t1 vrf v1 
apic1(config-leaf-ospf-vrf)# area 0.0.0.0 loopback 30.30.30.0
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)# exit

Step 5

Configure the external EPG on node 103.

In this example, the network 20.20.20.0/24 is configured as the external network extnw1.

Example:

apic1(config)# tenant t1 
apic1(config-tenant)# external-l3 epg extnw1
apic1(config-tenant-l3ext-epg)# vrf member v1 
apic1(config-tenant-l3ext-epg)# match ip 20.20.20.0/24
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 103
apic1(config-leaf)# vrf context tenant t1 vrf v1 
apic1(config-leaf-vrf)# external-l3 epg extnw1
apic1(config-leaf-vrf)# exit

Step 6

Optional. Configure Advertise Host Routing.

Example:

apic1# configure
apic1(config)# tenant <Name>
apic1(config-tenant)# bridge-domain <Name>
apic1(config-tenant-bd)# advertise-host-routes
apic1(config-tenant-bd)# end

Step 7

Optional. Configure a route map.

This example configures a route map rp1 for the BGP peer in the outbound direction. The route map is applied for routes that match a destination of 200.3.2.0/24. Also, on a successful match (if the route matches this range) the route AS PATH attribute is updated to 200 and 100.

Example:

apic1(config-leaf)# template route group match-rule1 tenant t1
apic1(config-route-group)# ip prefix permit 200.3.2.0/24
apic1(config-route-group)# exit
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# route-map rp1
apic1(config-leaf-vrf-route-map)# match route group match-rule1 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config)# leaf 103
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp1 in
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)#exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit

Step 8

Add a bridge domain.

Example:

apic1(config)# tenant t1
apic1(config-tenant)# bridge-domain bd1
apic1(config-tenant-bd)# vrf member v1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# interface bridge-domain bd1
apic1(config-tenant-interface)# ip address 44.44.44.1/24 scope public 
apic1(config-tenant-interface)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant t1 vrf v1 
apic1(config-leaf-vrf)# route-map rp1
apic1(config-leaf-vrf-route-map)# match bridge-domain bd1 tenant t1 
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit

Step 9

Create an application EPG on node 101.

Example:

apic1(config)# tenant t1 
apic1(config-tenant)# application app1 
apic1(config-tenant-app)# epg epg1
apic1(config-tenant-app-epg)# bridge-domain member bd1
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 2011 tenant t1 application app1 epg epg1
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)#

Step 10

Create filters (access-lists) and contracts.

Example:

apic1(config)# tenant t1
apic1(config-tenant)# access-list http-filter
apic1(config-tenant-acl)# match ip
apic1(config-tenant-acl)# match tcp dest 80
apic1(config-tenant-acl)# exit
apic1(config-tenant)# contract httpCtrct 
apic1(config-tenant-contract)# scope vrf
apic1(config-tenant-contract)# subject subj1
apic1(config-tenant-contract-subj)# access-group http-filter both 
apic1(config-tenant-contract-subj)# exit
apic1(config-tenant-contract)# exit

Step 11

Configure contracts and associate them with EPGs.

Example:

apic1(config-tenant)# external-l3 epg extnw1 
apic1(config-tenant-l3ext-epg)# vrf member v1 
apic1(config-tenant-l3ext-epg)# contract provider httpCtrct 
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# application app1 
apic1(config-tenant-app)# epg epg1
apic1(config-tenant-app-epg)# contract consumer httpCtrct
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit
apic1(config)#

NX-OS Style CLI Example: L3Out Prerequisites

Before you can configure an L3Out, perform the following steps:

  1. Configure a VLAN domain:

    apic1# configure 
apic1(config)# vlan-domain dom1
apic1(config-vlan)# vlan 1024-2048
apic1(config-vlan)# exit

  2. Configure BGP route reflectors:

    
apic1(config)# bgp-fabric
apic1(config-bgp-fabric)# asn 100
apic1(config-bgp-fabric)# route-reflector spine 104,105

NX-OS Style CLI Example: L3Out

The following example provides a merged version of the steps to configure an L3Out using the NX-OS style CLI. Configure the following prerequisites before configuring the L3Out. 

apic1# configure 
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 103 
apic1(config-leaf)# vrf context tenant t1 vrf v1 
apic1(config-leaf-vrf)# router-id 11.11.11.103
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1 
apic1(config-leaf-if)# no switchport 
apic1(config-leaf-if)# vrf member tenant t1 vrf v1 
apic1(config-leaf-if)# ip address 12.12.12.3/24
apic1(config-leaf-if)# exit
apic1(config-leaf)# router bgp 100 
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1 
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# router ospf default 
apic1(config-leaf-ospf)# vrf member tenant t1 vrf v1 
apic1(config-leaf-ospf-vrf)# area 0.0.0.0 loopback 30.30.30.0
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)# exit
apic1(config)# tenant t1 
apic1(config-tenant)# external-l3 epg extnw1
apic1(config-tenant-l3ext-epg)# vrf member v1 
apic1(config-tenant-l3ext-epg)# match ip 20.20.20.0/24
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 103
apic1(config-leaf)# vrf context tenant t1 vrf v1 
apic1(config-leaf-vrf)# external-l3 epg extnw1
apic(config-leaf-vrf)# exit
apic1(config-leaf)# template route group match-rule1 tenant t1
apic1(config-route-group)# ip prefix permit 200.3.2.0/24
apic1(config-route-group)# exit
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# route-map rp1
apic1(config-leaf-vrf-route-map)# match route group match-rule1 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp1 in
apic1(config-leaf-bgp-vrf-neighbor)#exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit
apic1(config)# tenant t1 
apic1(config-tenant)# bridge-domain bd1
apic1(config-tenant-bd)# vrf member v1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# interface bridge-domain bd1
apic1(config-tenant-interface)# ip address 44.44.44.1/24 scope public 
apic1(config-tenant-interface)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant t1 vrf v1 
apic1(config-leaf-vrf)# route-map map1
apic1(config-leaf-vrf-route-map)# match bridge-domain bd1 tenant t1 
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit
apic1(config)# tenant t1 
apic1(config-tenant)# application app1 
apic1(config-tenant-app)# epg epg1
apic1(config-tenant-app-epg)# bridge-domain member bd1
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 2011 tenant t1 application app1 epg epg1
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# tenant t1 
apic1(config-tenant)# access-list http-filter
apic1(config-tenant-acl)# match ip
apic1(config-tenant-acl)# match tcp dest 80
apic1(config-tenant-acl)# exit
apic1(config-tenant)# contract httpCtrct 
apic1(config-tenant-contract)# scope vrf
apic1(config-tenant-contract)# subject subj1
apic1(config-tenant-contract-subj)# access-group http-filter both 
apic1(config-tenant-contract-subj)# exit
apic1(config-tenant-contract)# exit
apic1(config-tenant)# external-l3 epg extnw1 
apic1(config-tenant-l3ext-epg)# vrf member v1 
apic1(config-tenant-l3ext-epg)# contract provider httpCtrct 
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# application app1 
apic1(config-tenant-app)# epg epg1
apic1(config-tenant-app-epg)# contract consumer httpCtrct
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit
apic1(config)#

Configuring a Layer 3 Outside for Tenant Networks Using the GUI

Perform the following steps to configure a Layer 3 outside (L3Out) connection for the fabric.

Before you begin

  • Configure the node, port, functional profile, AEP, and Layer 3 domain.

  • Create the external routed domain and associate it to the interface for the L3Out.

  • Configure a BGP Route Reflector policy to propagate the routes within the fabric.

Procedure

Step 1

To create the tenant and VRF, on the menu bar, choose Tenants > Add Tenant and in the Create Tenant dialog box, perform the following tasks:

  1. In the Name field, enter the tenant name.

  2. In the VRF Name field, enter the VRF name.

  3. Click Submit.

Step 2

To create a bridge domain, in the Navigation pane, expand Tenant and Networking and perform the following steps:

  1. Right-click Bridge Domains and choose Create Bridge Domain.

  2. In the Name field, enter a name for the bridge domain (BD).

  3. (Optional) Click the box for Advertise Host Routes to enable advertisement to all deployed border leafs.

  4. In the VRF field, from the drop-down list, choose the VRF you created (v1 in this example).

  5. Click Next.

  6. Click the + icon on Subnets.

  7. In the Gateway IP field, enter the subnet for the BD.

  8. In the Scope field, choose Advertised Externally.

    Add the L3 Out for Route Profile later, after you create it.

    Note

     

    If Advertise Host Routes is enabled, the route-map will also match all host routes.

  9. Click OK.

  10. Click Next and click Finish.

Step 3

To create an application EPG, perform the following steps:

  1. Right-click Application Profiles and choose Create Application Profile.

  2. Enter a name for the application.

  3. Click the + icon for EPGs.

  4. Enter a name for the EPG.

  5. From the BD drop-down list, choose the bridge domain you previously created.

  6. Click Update.

  7. Click Submit.

Step 4

To start creating the L3Out, on the Navigation pane, expand Tenant and Networking and perform the following steps:

  1. Right-click External Routed Networks and choose Create Routed Outside.

  2. In the Name field, enter a name for the L3Out.

  3. From the VRF drop-down list, choose the VRF.

  4. From the External Routed Domain drop-down list, choose the external routed domain that you previously created.

  5. In the area with the routing protocol check boxes, check the desired protocols (BGP, OSPF, or EIGRP).

    For the example in this chapter, choose BGP and OSPF.

    Depending on the protocols you choose, enter the properties that must be set.

  6. Enter the OSPF details, if you enabled OSPF.

    For the example in this chapter, use the OSPF area 0 and type Regular area.

  7. Click + to expand Nodes and Interfaces Protocol Profiles.

  8. In the Name field, enter a name.

  9. Click + to expand Nodes.

  10. From the Node IDfield drop-down menu, choose the node for the L3Out.

    For the topology in these examples, use node 103.

  11. In the Router ID field, enter the router ID (IPv4 or IPv6 address for the router that is connected to the L3Out).

  12. (Optional) You can configure another IP address for a loopback address. Uncheck Use Router ID as Loopback Address, expand Loopback Addresses, enter an IP address, and click Update.

  13. In the Select Node dialog box, click OK.

Step 5

If you enabled BGP, click the + icon to expand BGP Peer Connectivity Profiles and perform the following steps:

  1. In the Peer Address field, enter the BGP peer address.

  2. In the Local-AS Number field, enter the BGP AS number.

    For the example in this chapter, use the BGP peer address 15.15.15.2 and ASN number 100.

  3. Click OK.

Step 6

Click + to expand Interface Profiles (OSPF Interface Profiles if you enabled OSPF), and perform the following actions:

  1. In the Name field, enter a name for the interface profile.

  2. Click Next.

  3. In the Protocol Profiles dialog box, in the OSPF Policy field, choose an OSPF policy.

  4. Click Next.

  5. Click the + icon to expand Routed Interfaces.

  6. In the Select Routed Interface dialog box, from the Node drop-down list, choose the node.

  7. From the Path drop-down list, choose the interface path.

  8. In the IPv4 Primary/IPv6 Preferred Address field, enter the IP address and network mask for the interface.

    Note

     

    To configure IPv6, you must enter the link-local address in the Link-local Address field.

  9. Click OK in the Select Routed Interface dialog box.

  10. Click OK in the Create Interface Profile dialog box.

Step 7

In the Create Node Profile dialog box, click OK.

Step 8

In the Create Routed Outside dialog box, click Next.

Step 9

In the External EPG Networks tab, click Create Route Profiles.

Step 10

Click the + icon to expand Route Profiles and perform the following actions:

  1. In the Name field, enter the route map name.

  2. Choose the Type.

    For this example, leave the default, Match Prefix AND Routing Policy.

  3. Click the + icon to expand Contexts and create a route context for the route map.

  4. Enter the order and name of the profile context.

  5. Choose Deny or Permit for the action to be performed in this context.

  6. (Optional) In the Set Rule field, choose Create Set Rules for a Route Map.

    Enter the name for the set rules, click the objects to be used in the rules, and click Finish.

  7. In the Associated Matched Rules field, click + to create a match rule for the route map.

  8. Enter the name for the match rules and enter the Match Regex Community Terms, Match Community Terms, or Match Prefix to match in the rule.

  9. Click the rule name and click Update.

  10. In the Create Match Rule dialog box, click Submit, and then click Update.

  11. In the Create Route Control Context dialog box, click OK.

  12. In the Create Route Map dialog box, click OK.

Step 11

Click the + icon to expand External EPG Networks.

Step 12

In the Name field, enter a name for the external network.

Step 13

Click the + icon to expand Subnet.

Step 14

In the Create Subnet dialog box, perform the following actions:

  1. In the IP address field, enter the IP address and network mask for the external network.

  2. In the Scope field, check the appropriate check boxes to control the import and export of prefixes for the L3Out.

    Note

     

    For more information about the scope options, see the online help for this Create Subnet panel.

  3. (Optional) In the Route Summarization Policy field, from the drop-down list, choose an existing route summarization policy or create a new one as desired. Also click the check box for Export Route Control Subnet.

    The type of route summarization policy depends on the routing protocols that are enabled for the L3Out.

  4. Click the + icon to expand Route Control Profile.

  5. In the Name field, choose the route control profile that you previously created from the drop-down list.

  6. In the Direction field, choose Route Export Policy.

  7. Click Update.

  8. In the Create Subnet dialog box, click OK.

  9. (Optional) Repeat to add more subnets.

  10. In the Create External Network dialog box, click OK.

Step 15

In the Create Routed Outside dialog box, click Finish.

Step 16

In the Navigation pane, under Tenant_name > Networking expand Bridge Domains.

Note

 

If the L3Out is static, you are not required to choose any BD settings.

Step 17

Choose the BD you created.

  1. In the Work pane, click Policy and L3 Configurations.

  2. Click the + icon to expand the Associated L3 Outs field, choose the previously configured L3Out, and click Update.

  3. In the L3Out for Route Profile field, choose the L3Out again.

  4. Click Next and Finish.

Step 18

In the Navigation pane, under External Routed Networks, expand the previously created L3Out and right-click Route Maps/Profiles.

Note

 

To set attributes for BGP, OSPF, or EIGRP for received routes, create a default-import route control profile, with the appropriate set actions and no match actions.

Step 19

Choose Create Route Map/Profile, and in the Create Route Map/Profile dialog box, perform the following actions:

  1. From the drop-down list on the Name field, choose default-import.

  2. In the Type field, you must click Match Routing Policy Only. Click Submit.

Step 20

(Optional) To enable extra communities to use BGP, using the following steps:

  1. Right-click Set Rules for Route Maps, and click Create Set Rules for a Route Map.

  2. In the Create Set Rules for a Route Map dialog box, click the Add Communities field, and follow the steps to assign multiple BGP communities per route prefix.

Step 21

To enable communications between the EPGs consuming the L3Out, create at least one filter and contract, using the following steps:

  1. In the Navigation pane, under the tenant consuming the L3Out, expand Contracts.

  2. Right-click Filters and choose Create Filter.

  3. In the Name field, enter a filter name.

    A filter is essentially an Access Control List (ACL).

  4. Click the + icon to expand Entries, and add a filter entry.

  5. Add the Entry details.

    For example, for a simple web filter, set criteria such as the following:

    • EtherType—IP

    • IP Protocol—tcp

    • Destination Port Range From—Unspecified

    • Destination Port Range To to https

  6. Click Update.

  7. In the Create Filter dialog box, click Submit.

Step 22

To add a contract, use the following steps:

  1. Under Contracts, right-click Standard and choose Create Contract.

  2. Enter the name of the contract.

  3. Click the + icon to expand Subjects to add a subject to the contract.

  4. Enter a name for the subject.

  5. Click the + icon to expand Filters and choose the filter that you previously created, from the drop-down list.

  6. Click Update.

  7. In the Create Contract Subject dialog box, click OK.

  8. In the Create Contract dialog box, click Submit.

Step 23

Associate the EPGs for the L3Out with the contract, with the following steps:

In this example, the L3 external EPG (extnw1) is the provider and the application EPG (epg1) is the consumer.

  1. To associate the contract to the L3 external EPG, as the provider, under the tenant, click Networking, expand External Routed Networks, and expand the L3Out.

  2. Expand Networks, click the L3 external EPG, and click Contracts.

  3. Click the the + icon to expand Provided Contracts.

  4. In the Name field, choose the contract that you previously created from the list.

  5. Click Update.

  6. To associate the contract to an application EPG, as a consumer, under the tenant, navigate to Application Profiles > app-prof-name > Application EPGs > and expand the app-epg-name.

  7. Right-click Contracts, and choose Add Consumed Contract.

  8. On the Contract field, choose the contract that you previously created.

  9. Click Submit.