As shown in the figure below, when the leaf switch interface is configured as a bridged interface, the default gateway for the tenant VNID is the external router.

The ACI fabric is unaware of the presence of the external router and the APIC statically assigns the leaf switch interface to its EPG.

Routed connectivity to external networks is enabled by associating a fabric access (infraInfra) external routed domain (l3extDomP) with a tenant Layer 3 external instance profile (l3extInstP or external EPG) of a Layer 3 external outside network (l3extOut), in the hierarchy in the following diagram:

A Layer 3 external outside network (l3extOut object) includes the routing protocol options (BGP, OSPF, or EIGRP or supported combinations) and the switch-specific and interface-specific configurations. While the l3extOut contains the routing protocol (for example, OSPF with its related Virtual Routing and Forwarding (VRF) and area ID), the Layer 3 external interface profile contains the necessary OSPF interface details. Both are needed to enable OSPF.

The l3extInstP EPG exposes the external network to tenant EPGs through a contract. For example, a tenant EPG that contains a group of web servers could communicate through a contract with the l3extInstP EPG according to the network configuration contained in the l3extOut. The outside network configuration can easily be reused for multiple nodes by associating the nodes with the L3 external node profile. Multiple nodes that use the same profile can be configured for fail-over or load balancing. Also, a node can be added to multiple l3extOuts resulting in VRFs that are associated with the l3extOuts also being deployed on that node. For scalability information, refer to the current Verified Scalability Guide for Cisco ACI.

Advertise Host Routes

Enabling Advertise Host Routes on the BD, individual host-routes (/32 and /128 prefixes) are advertised from the Border-Leaf switches (BL). The BD must be associated to the L3out or an explicit prefix list matching the host routes. The host routes must be configured to advertise host routes out of the fabric.

Border-Leaf switches along with the subnet advertise the individual end-point(EP) prefixes. The route information is advertised only if the host is connected to the local POD. If the EP is moved away from the local POD or once the EP is removed from EP database (even if the EP is attached to a remote leaf), the route advertisement is then withdrawn.

Advertise Host Route configuration guidelines and limitations are:

• When host routes are advertised, the VRF Transit Route Tag is set in order to prevent them from being advertised back into the fabric and installed. In order for this loop protection to work properly, external routers must preserve this route-tag if advertising to another L3Out.

• If a bridge domain is tied to an EPG that has the same subnet configured for internal leaking, you must also enable the "Advertised Externally" flag on the EPG subnet.

• The Advertise Host Routes feature is supported on Generation 2 switches or later (Cisco Nexus N9K switches with "EX", "FX", or "FX2" on the end of the switch model name or later; for example, N9K-93108TC-EX).

• Host route advertisement supports both BD to L3out Association and the explicit route map configurations. We recommend using explicit route map configuration which allows you greater control in selecting individual or a range of host routes to configure.

• EPs/Host routes in SITE-1 will not be advertised out through Border Leafs in other SITEs.

• When EPs is aged out or removed from the database, Host routes are withdrawn from the Border Leaf.

• When EP is moved across SITEs or PODs, Host routes should be withdrawn from first SITE/POD and advertised in new POD/SITE.

• EPs learned on a specific BD, under any of the BD subnets are advertised from the L3out on the border leaf in the same POD.

• EPs are advertised out as Host Routes only in the local POD through the Border Leaf.

• Host routes are not advertised out from one POD to another POD.

• In the case of Remote Leaf, if EPs are locally learned in the Remote Leaf, they are then advertised only through a L3out deployed in Remote Leaf switches in same POD.

• EPs/Host routes in a Remote Leaf are not advertised out through Border Leaf switches in main POD or another POD.

• EPs/Host routes in the main POD are not advertised through L3out in Remote Leaf switches of same POD or another POD.

• The BD subnet must have the Advertise Externally option enabled.

• The BD must be associated to an L3out or the L3out must have explicit route-map configured matching BD subnets.

• There must be a contract between the EPG in the specified BD and the External EPG for the L3out.

  Note	If there is no contract between the BD/EPG and the External EPG the BD subnet and host routes will not be installed on the border leaf.

• Advertise Host Route is supported for shared services. For example: epg1/BD1 deployed is in VRF-1 and L3out in another VRF-2. By providing shared contract between EPG and L3out host routes are pulled from one VRF-1 to another VRF-2.

• When Advertise Host Route is enabled on BD custom tag cannot be set on BD Subnet using route-map.

• When Advertise Host Route is enabled on a BD and the BD is associated with an L3Out, BD subnet is marked public. If there's a rogue EP present under the BD, that EP is advertised out on L3Out.

Static route preference within the ACI fabric is carried in MP-BGP using cost extended community.

The following figure illustrates how the ACI fabric keeps static route preferences intact across leaf switches so that route selection happens based on this preference.

This figure shows a MP-BGP route coming to leaf switch 3 (L3) from leaf switch 4 (L4) that wins over a local static route. A static route is installed in the Unicast Routing Information Base (URIB) with the preference configured by an administrator. On an ACI non-border leaf switch, a static route is installed with leaf switch 4 (L4) as its nexthop. When nexthop on L4 is not available, the L3 static route becomes the best route in fabric.

Note	If a static route in a leaf switch is defined with next hop Null 0, MP-BGP does not advertise that route to other leaf switches in fabric.

Subnet route export or import configuration options can be specified according to the scope and aggregation options described below.

For routed subnets, the following scope options are available:

• Export Route Control Subnet—Controls the export route direction.

• Import Route Control Subnet—Controls the import route direction.

  Note	Import route control is supported for BGP and OSPF, but not EIGRP.

• External Subnets for the External EPG (Security Import Subnet)—Specifies which external subnets have contracts applied as part of a specific External Network Instance Profile (l3extInstP). For a subnet under the l3extInstP to be classified as an External EPG, the scope on the subnet should be set to "import-security". Subnets of this scope determine which IP addresses are associated with the l3extInstP. Once this is determined, contracts determine with which other EPGs that external subnet is allowed to communicate. For example, when traffic enters the ACI switch on the Layer 3 External Outside Network (L3extOut), a lookup occurs to determine which source IP addresses are associated with the l3extInstP. This action is performed based on Longest Prefix Match (LPM) so that more specific subnets take precedence over more general subnets.

• Shared Route Control Subnet— In a shared service configuration, only subnets that have this property enabled will be imported into the consumer EPG Virtual Routing and Forwarding (VRF). It controls the route direction for shared services between VRFs.

• Shared Security Import Subnet—Applies shared contracts to imported subnets. The default specification is External Subnets for the External EPG.

Routed subnets can be aggregated. When aggregation is not set, the subnets are matched exactly. For example, if 11.1.0.0/16 is the subnet, then the policy will not apply to a 11.1.1.0/24 route, but it will apply only if the route is 11.1.0.0/16. However, to avoid a tedious and error prone task of defining all the subnets one by one, a set of subnets can be aggregated into one export, import or shared routes policy. At this time, only 0/0 subnets can be aggregated. When 0/0 is specified with aggregation, all the routes are imported, exported, or shared with a different VRF, based on the selection option below:

• Aggregate Export—Exports all transit routes of a VRF (0/0 subnets).

• Aggregate Import—Imports all incoming routes of given L3 peers (0/0 subnets).

  Note	Aggregate import route control is supported for BGP and OSPF, but not for EIGRP.

• Aggregate Shared Routes—If a route is learned in one VRF but needs to be advertised to another VRF, the routes can be shared by matching the subnet exactly, or can be shared in an aggregate way according to a subnet mask. For aggregate shared routes, multiple subnet masks can be used to determine which specific route groups are shared between VRFs. For example, 10.1.0.0/16 and 12.1.0.0/16 can be specified to aggregate these subnets. Or, 0/0 can be used to share all subnet routes across multiple VRFs.

Note

Routes shared between VRFs function correctly on Generation 2 switches (Cisco Nexus N9K switches with "EX" or "FX" on the end of the switch model name, or later; for example, N9K-93108TC-EX). On Generation 1 switches, however, there may be dropped packets with this configuration, because the physical ternary content-addressable memory (TCAM) tables that store routes do not have enough capacity to fully support route parsing.

Route summarization simplifies route tables by replacing many specific addresses with an single address. For example, 10.1.1.0/24, 10.1.2.0/24, and 10.1.3.0/24 are replaced with 10.1.0.0/16. Route summarization policies enable routes to be shared efficiently among border leaf switches and their neighbor leaf switches. BGP, OSPF, or EIGRP route summarization policies are applied to a bridge domain or transit subnet. For OSPF, inter-area and external route summarization are supported. Summary routes are exported; they are not advertised within the fabric. In the example above, when a route summarization policy is applied, and an EPG uses the 10.1.0.0/16 subnet, the entire range of 10.1.0.0/16 is shared with all the neighboring leaf switches.

Note	When two L3extOut policies are configured with OSPF on the same leaf switch, one regular and another for the backbone, a route summarization policy configured on one L3extOut is applied to both L3extOut policies because summarization applies to all areas in the VRF.

As illustrated in the figure below, route control profiles derive route maps according to prefix-based and community-based matching.

The route control profile (rtctrtlProfile) specifies what is allowed. The Route Control Context specifies what to match, and the scope specifies what to set. The subject profile contains the community match specifications, which can be used by multiple l3extOut instances. The subject profile (SubjP) can contain multiple community terms each of which contains one or more community factors (communities). This arrangement enables specifying the following boolean operations:

• Logical or among multiple community terms

• Logical and among multiple community factors

For example, a community term called northeast could have multiple communities that each include many routes. Another community term called southeast could also include many different routes. The administrator could choose to match one, or the other, or both. A community factor type can be regular or extended. Care should be taken when using extended type community factors, to ensure there are no overlaps among the specifications.

The scope portion of the route control profile references the attribute profile (rtctrlAttrP) to specify what set-action to apply, such as preference, next hop, community, and so forth. When routes are learned from an l3extOut, route attributes can be modified.

The figure above illustrates the case where an l3extOut contains a rtctrtlProfile. A rtctrtlProfile can also exist under the tenant. In this case, the l3extOut has an interleak relation policy (L3extRsInterleakPol) that associates it with the rtctrtlProfile under the tenant. This configuration enables reusing the rtctrtlProfile for multiple l3extOut connections. It also enables keeping track of the routes the fabric learns from OSPF to which it gives BGP attributes (BGP is used within the fabric). A rtctrtlProfile defined under an L3extOut has a higher priority than one defined under the tenant.

The rtctrtlProfile has two modes: combinable, and global. The default combinable mode combines pervasive subnets (fvSubnet) and external subnets (l3extSubnet) with the match/set mechanism to render the route map. The global mode applies to all subnets within the tenant, and overrides other policy attribute settings. A global rtctrtlProfile provides permit-all behavior without defining explicit (0/0) subnets. A global rtctrtlProfile is used with non-prefix based match rules where matching is done using different subnet attributes such as community, next hop, and so on. Multiple rtctrtlProfile policies can be configured under a tenant.

rtctrtlProfile policies enable enhanced default import and default export route control. Layer 3 Outside networks with aggregated import or export routes can have import/export policies that specify supported default-export and default–import, and supported 0/0 aggregation policies. To apply a rtctrtlProfile policy on all routes (inbound or outbound), define a global default rtctrtlProfile that has no match rules.

Note	While multiple l3extOut connections can be configured on one switch, all Layer 3 outside networks configured on a switch must use the same rtctrtlProfile because a switch can have only one route map.

The protocol interleak and redistribute policy controls externally learned route sharing with ACI fabric BGP routes. Set attributes are supported. Such policies are supported per L3extOut, per node, or per VRF. An interleak policy applies to routes learned by the routing protocol in the L3extOut. Currently, interleak and redistribute policies are supported for OSPF v2 and v3. A route control policy rtctrtlProfile has to be defined as global when it is consumed by an interleak policy.

Shared services enable communications across tenants while preserving the isolation and security policies of the tenants. A routed connection to an external network is an example of a shared service that multiple tenants use.

Follow these guidelines when configuring shared services contracts.

• For shared services that export subnets to different Virtual Routing and Forwarding (VRF) instances (also known as contexts or private networks), the subnet must be configured under an EPG, and the scope must be set to Advertised Externally and Shared Between VRFs.

• Contracts are not needed for inter-bridge domain traffic when a VRF is unenforced.

• Contracts are needed for shared service inter-VRF traffic, even when a VRF is unenforced.

• The VRF of a provider EPG cannot be in unenforced mode while providing a shared service.

•  A shared service is supported only with non-overlapping and non-duplicate subnets. When configuring subnets for shared services, follow these guidelines:

  • Configure the subnet for a shared service provider under the EPG, not under the bridge domain.

  • Subnets configured under an EPG that share the same VRF must be disjointed and must not overlap.

  • Subnets leaked from one VRF to another must be disjointed and must not overlap.

  • Subnets leaked from multiple consumer networks into a VRF or vice versa must be disjointed and must not overlap.

  Note	If two consumers are mistakenly configured with the same subnet, recover from this condition by removing the subnet configuration for both then reconfigure the subnets correctly.

• Do not configure a shared service with AnyToProv in the provider VRF. The APIC rejects this configuration and raises a fault.

• When a contract is configured between in-band and out-of-band EPGs, the following restrictions apply:

  •  Both EPGs should be in the same VRF.

  • Ffilters apply only in the incoming direction.

  • Layer 2 filters are not supported.

  • QoS does not apply to in-band Layer 4 to Layer 7 services.

  • Management statistics are not available.

  • Shared services for CPU-bound traffic are not supported.

A shared Layer 3 outside network (L3extOut) configuration provides routed connectivity to external networks as a shared service. An L3extOut profile (l3extInstP) EPG provides routed connectivity to external networks. It can be provisioned as a shared service in any tenant (user, common, infra, or mgmt). Prior to release 1.2(1x), this configuration was only supported in the user and common tenants. An EPG in any tenant can use a shared services contract to connect with an l3extInstP EPG regardless of where in the fabric that l3extInstP EPG is provisioned. This simplifies the provisioning of routed connectivity to external networks; multiple tenants can share a single l3extInstP EPG for routed connectivity to external networks. Sharing an l3extInstP EPG is more efficient because it consumes only one session on the switch regardless of how many EPGs use the single shared l3extInstP EPG.

Note

All switches that will use l3extInstP EPG shared service contracts require the hardware and software support available starting with the APIC 1.2(1x) and switch 11.2(1x) releases. Refer to the Cisco APIC Management, Installation, Upgrade, and Downgrade Guide and Release Notes documentation for more details.

The figure below illustrates the major policy model objects that are configured for a shared l3extInstP EPG.

Take note of the following guidelines and limitations for shared Layer 3 outside network configurations:

• No tenant limitations: Tenants A and B can be any kind of tenant (user, common, infra, mgmt.). The shared l3extInstP EPG does not have to be in the common tenant.

• Flexible placement of EPGs: EPG A and EPG B in the illustration above are in different tenants. EPG A and EPG B could use the same bridge domain and VRF, but they are not required to do so. EPG A and EPG B are in different bridge domains and different VRFs but still share the same l3extInstP EPG.

• A subnet can be private, public, or shared. A subnet that is to be advertised into a consumer or provider EPG of an L3extOut must be set to shared. A subnet that is to be exported to an L3extOut must be set to public.

• The shared service contract is exported from the tenant that contains the l3extInstP EPG that provides shared Layer 3 outside network service. The shared service contract is imported into the tenants that contain the EPGs that consume the shared service.

• Do not use taboo contracts with a shared L3 out; this configuration is not supported.

• The l3extInstP as a shared service provider is supported, but only with non l3extInstP consumers (where the L3extOut EPG is the same as the l3extInstP).

• Traffic Disruption (Flap): When an l3instP EPG is configured with an external subnet of 0.0.0.0/0 with the scope property of the l3instP subnet set to shared route control (shared-rctrl), or shared security (shared-security), the VRF is redeployed with a global pcTag. This will disrupt all the external traffic in that VRF (because the VRF is redeployed with a global pcTag).

• Prefixes for a shared L3extOut must to be unique. Multiple shared L3extOut configurations with the same prefix in the same VRF will not work. Be sure that the external subnets (external prefixes) that are advertised into a VRF are unique (the same external subnet cannot belong to multiple l3instPs). An L3extOut configuration (for example, named L3Out1) with prefix1 and a second Layer 3 outside configuration (for example, named L3Out2) also with prefix1 belonging to the same VRF will not work (because only 1 pcTag is deployed). Different behaviors of L3extOut are possible when configured on the same leaf switch under the same VRF. The two possible scenarios are as follows:

  • Scenario 1 has an L3extOut with an SVI interface and two subnets (10.10.10.0/24 and 0.0.0.0/0) defined. If ingress traffic on the Layer 3 outside network has the matching prefix 10.10.10.0/24, then the ingress traffic uses the External EPG pcTag. If ingress traffic on the Layer 3 Outside network has the matching default prefix 0.0.0.0/0, then the ingress traffic uses the External Bridge pcTag.

  • Scenario 2 has an L3extOut using a routed or routed-sub-interface with two subnets (10.10.10.0/24 and 0.0.0.0/0) defined. If ingress traffic on the Layer 3 outside network has the matching prefix 10.10.10.0/24, then the ingress traffic uses the External EPG pcTag. If ingress traffic on the Layer 3 outside network has the matching default prefix 0.0.0.0/0, then the ingress traffic uses the VRF pcTag.

  • As a result of these described behaviors, the following use cases are possible if the same VRF and same leaf switch are configured with L3extOut-A and L3extOut-B using an SVI interface:

    Case 1 is for L3extOut-A: This External Network EPG has two subnets defined: 10.10.10.0/24 & 0.0.0.0/1. If ingress traffic on L3extOut-A has the matching prefix 10.10.10.0/24, it uses the external EPG pcTag & contract-A which is associated with L3extOut-A. When egress traffic on L3extOut-A has no specific match found, but there is a maximum prefix match with 0.0.0.0/1, it uses the External Bridge Domain (BD) pcTag & contract-A.

    Case 2 is for L3extOut-B: This External Network EPG has one subnet defined: 0.0.0.0/0. When ingress traffic on L3extOut-B has the matching prefix10.10.10.0/24 (which is defined underL3extOut-A), it uses the External EPG pcTag of L3extOut-A and the contract-A which is tied with L3extOut-A. It does not use contract-B which is tied with L3extOut-B.

• Traffic not permitted: Traffic is not permitted when an invalid configuration sets the scope of the external subnet to shared route control (shared-rtctrl) as a subset of a subnet that is set to shared security (shared-security). For example, the following configuration is invalid:

  • shared rtctrl: 10.1.1.0/24, 10.1.2.0/24

  • shared security: 10.1.0.0/16

  In this case, ingress traffic on a non-border leaf with a destination IP of 10.1.1.1 is dropped, since prefixes 10.1.1.0/24 and 10.1.2.0/24 are installed with a drop rule. Traffic is not permitted. Such traffic can be enabled by revising the configuration to use the shared-rtctrl prefixes as shared-security prefixes as well.

• Inadvertent traffic flow: Prevent inadvertent traffic flow by avoiding the following configuration scenarios:

  • Case 1 configuration details:

    • A Layer 3 outside network configuration (for example, named L3extOut-1) with VRF1 is called provider1.

    • A second Layer 3 outside network configuration (for example, named L3extOut-2) with VRF2 is called provider2.

    • L3extOut-1 VRF1 advertises a default route to the Internet, 0.0.0.0/0 which enables both shared-rtctrl and shared-security.

    • L3extOut-2 VRF2 advertises specific subnets to DNS and NTP, 192.0.0.0/8 which enables shared-rtctrl.

    • L3extOut-2 VRF2 has specific subnet 192.1.0.0/16, which enables shared-security.

    • Variation A: EPG Traffic Goes to Multiple VRFs.

      • Communications between EPG1 and L3extOut-1 is regulated by an allow_all contract.

      • Communications between EPG1 and L3extOut-2 is regulated by an allow_all contract.

        Result: Traffic from EPG1 to L3extOut-2 also goes to 192.2.x.x.

    • Variation B: An EPG conforms to the allow_all contract of a second shared Layer 3 outside network.

      • Communications between EPG1 and L3extOut-1 is regulated by an allow_all contract.

      • Communications between EPG1 and L3extOut-2 is regulated by an allow_icmp contract.

        Result: Traffic from EPG1 to L3extOut-2 to 192.2.x.x conforms to the allow_all contract.

  • Case 2 configuration details:

    • A L3extOut profile (l3instP) has one shared prefix and other non-shared prefixes.

    • Traffic coming in with src = non-shared is allowed to go to the EPG

      • Variation A: Unintended traffic goes through an EPG.

        L3extOut (l3instP) EPG traffic goes through a L3extOut that has these prefixes:

        - 192.0.0.0/8 = import-security, shared-rtctrl

        - 192.1.0.0/16 = shared-security

        - The EPG has 1.1.0.0/16 = shared

        Result: Traffic going from 192.2.x.x also goes through to the EPG.

      • Variation B: Unintended traffic goes through an EPG. Traffic coming in a shared L3extOut can go through the EPG.

        - The shared L3extOut VRF has an EPG with pcTag = prov vrf and a contract set to allow_all

        - The EPG <subnet> = shared.

        Result: The traffic coming in on the Layer 3 out can go through the EPG.

Use Bidirectional Forwarding Detection (BFD) to provide sub-second failure detection times in the forwarding path between ACI fabric border leaf switches configured to support peering router connections.

BFD is particularly useful in the following scenarios:

• When the peering routers are connected through a Layer 2 device or a Layer 2 cloud where the routers are not directly connected to each other. Failures in the forwarding path may not be visible to the peer routers. The only mechanism available to control protocols is the hello timeout, which can take tens of seconds or even minutes to time out. BFD provides sub-second failure detection times.

• When the peering routers are connected through a physical media that does not support reliable failure detection, such as shared Ethernet. In this case too, routing protocols have only their large hello timers to fall back on.

• When many protocols are running between a pair of routers, each protocol has its own hello mechanism for detecting link failures, with its own timeouts. BFD provides a uniform timeout for all the protocols, which makes convergence time consistent and predictable.

Observe the following BFD guidelines and limitations:

• Prior to APIC release 5.0, BFD echo packets received on a leaf switch from neighbor routers are classified with the default QoS class (best effort). Due to that classification, BFD drops might result when there is congestion on the interfaces.

• Starting from APIC release 3.1(1), BFD between leaf and spine switches is supported on fabric-interfaces for IS-IS. In addition, BFD feature on spine switch is supported for OSPF and static routes.

• BFD is supported on modular spine switches that have -EX and -FX line cards (or newer versions), and BFD is also supported on the Nexus 9364C non-modular spine switch (or newer versions).

• BFD between VPC peers is not supported.

• Multihop BFD is not supported.

• BFD over iBGP is not supported for loopback address peers.

• BFD sub interface optimization can be enabled in an interface policy. One sub-interface having this flag will enable optimization for all the sub-interfaces on that physical interface.

• BFD for BGP prefix peer not supported.

Note

Cisco ACI does not support IP fragmentation. Therefore, when you configure Layer 3 Outside (L3Out) connections to external routers, or Multi-Pod connections through an Inter-Pod Network (IPN), it is recommended that the interface MTU is set appropriately on both ends of a link. On some platforms, such as Cisco ACI, Cisco NX-OS, and Cisco IOS, the configurable MTU value does not take into account the Ethernet headers (matching IP MTU, and excluding the 14-18 Ethernet header size), while other platforms, such as IOS-XR, include the Ethernet header in the configured MTU value. A configured value of 9000 results in a max IP packet size of 9000 bytes in Cisco ACI, Cisco NX-OS, and Cisco IOS, but results in a max IP packet size of 8986 bytes for an IOS-XR untagged interface. For the appropriate MTU values for each platform, see the relevant configuration guides. We highly recommend that you test the MTU using CLI-based commands. For example, on the Cisco NX-OS CLI, use a command such as ping 1.1.1.1 df-bit packet-size 9000 source-interface ethernet 1/1.