Validated Design for Cisco ACI to SR-MPLS Handoff

Available Languages

Download Options

  • PDF
    (4.7 MB)
    View with Adobe Reader on a variety of devices
Updated:February 18, 2021

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Available Languages

Download Options

  • PDF
    (4.7 MB)
    View with Adobe Reader on a variety of devices
Updated:February 18, 2021

Table of Contents

 

 


 

Introduction

Note:      The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product.

Prior to the introduction of SR/MPLS Handoff, there were two ways to extend a Tenant VPN located in a Cisco ACI fabric to an MPLS network:

      Using IP Handoff, the border leaf is connected to a Provider Edge (PE) router using VRF-lite. The main drawback is that for each VPN that must be extended, a physical or logical interface and a routing protocol session is required which leads to some scalability and automation challenges.

 

 

 

Diagram, applicationDescription automatically generated

 

Figure 1.            

IP Handoff Connectivity to WAN

The following link provides more information on IP Handoff and L3outs.

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/guide-c07-743150.html

 

      Using GOLF, the spine is running EVPN VxLAN to a PE router and the PE router performs the interworking function between EVPNoVxLAN and L3VPNoMPLS. While this option brings advantages in terms of scaling compared to IP Handoff (as there is a single EVPN session between the border leaf and DC-PE), it requires VxLAN to be enabled on the DC-PE, and some service provider platforms may have a limited support of VxLAN.

 

 

 

Graphical user interface, diagramDescription automatically generated

Figure 2.            

GOLF Connectivity to WAN

 

The following link provides more information on GOLF.

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/2-x/L3_config/b_Cisco_APIC_Layer_3_Configuration_Guide/b_Cisco_APIC_Layer_3_Configuration_Guide_chapter_010010.html

 

SR/MPLS Handoff is a new interconnection option that enables you to connect a border leaf or remote leaf to a DC-PE using Segment Routing (SR) MPLS. SR/MPLS is a better solution than others as it is much more common for an SP core. The solution brings the following benefits:

      Unified transport and policies between DC and SP

      Single Control Plane session for multiple VRFs

      Traffic engineering in the SP core controlled from the DC

 

 

 

DiagramDescription automatically generated

Figure 3.            

SR/MPLS Connectivity to WAN

Note:      Although the solution is named SR/MPLS Handoff, it is fully compatible with any existing MPLS LDP or RSVP-TE network deployment. However, to get the full benefit, especially the traffic-engineering control feature, a Segment Routing network is required.

See the SR/MPLS handoff white paper (https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-744107.html) for additional details.

 

Sample Use Cases

This section provides sample use cases where SR/MPLS handoff can be used.

Private Cloud for Enterprise

 

 

DiagramDescription automatically generated

 

Figure 4.            

Private Cloud for Enterprise

 

In this use case, an operator provides a Layer 3 VPN to multiple enterprises but also to Private cloud services by hosting customer dedicated applications into its datacenter. End to end VPN extension from the SP to the DC can easily be achieved by using the SR/MPLS Handoff or MPLS Handoff.

A similar use case is an enterprise having its own datacenter and private WAN network. The enterprise has several applications that must be reachable only from a particular group of users. Segmentation in the DC can easily be extended across the WAN using SR/MPLS handoff.

In addition, based on the application constraints, traffic can be steered over a specific path in the core based on the application requirements.

 

5G Network Slicing

 

 

DiagramDescription automatically generated

Figure 5.            

5G Network Slicing

SR/MPLS handoff adds automation to 5G network slicing. Each slice may use a different VPN and the traffic within a slice may use one or more class of services. SR/MPLS handoff automates both the VPN extension between DC and WAN as well as mapping the class of services on a specific transport path in the WAN.

In the use case described above, two mobile slices are created. One slice is dedicated to end users reaching the Internet and generic services and the other slice is dedicated to an enterprise customer.

As displayed in the example of Figure 5, traffic from an enterprise terminal can be steered seamlessly to the dedicated UPF using a premium routing path, and from the UPF to the enterprise headquarter site, also using a premium routing path. On the other hand, the end-user traffic is steered to another dedicated UPF and then to some Gi services before going to the Internet. Best effort routing in the underlay for the end-user traffic.

ACI as Transit

There are a couple of use cases where the Cisco ACI fabric can be used as transit with SR/MPLS handoff. This means that the traffic comes into the fabric, then exits the fabric and finally reaches its destination.

Related image, diagram or screenshot

Figure 6.            

Cisco ACI Fabric as Transit with Service Chaining

Figure 6 describes a typical GiLAN use case, where the mobile traffic from a user is steered to a service chain within the fabric. The traffic crosses all the necessary services before leaving the fabric and reaching its Internet destination.

 


 

Related image, diagram or screenshot        Related image, diagram or screenshot

         

Figure 7.            

Transit Without Any Services

Figure 7 describes two use cases where a Cisco ACI fabric is used purely as transport between two networks.

The figure on the left describes a user accessing a mainframe. The mainframe is connected using a regular L3out to a border leaf. Traffic comes from an external network using SR/MPLS handoff and is carried through the Cisco ACI fabric to the L3out where the mainframe is attached.

On the right hand side, a residential user is accessing some content located in a main DC. While the regional DC could be used to host mobile core functions for instance, it is used purely as transit for the traffic related to the residential users.

 

Audience

The audience for this document includes, but is not limited to Datacenter IT, Telco cloud, and Enterprise DCs.

 

Purpose of the Document

This document consists of validated designs and configurations to ease and to speed up customer deployments or testing.

The detailed configurations provided (especially related to ACI ) make the document quite long. However, this enables the reader to copy/paste the configurations and easily reproduce the validated designs.

 

Validated Hardware and Software Versions

The version numbers provided below are the ones that have been tested and are provided for information. A customer is may use any later versions without any issues.

 

Service Provider Network

NCS5500 acting as DC-PE and running XR 7.0.2

ASR9K Typhoon and Tomahawk line cards acting as DC-PE and running XR 7.0.2

 

Datacenter Network

ACI APIC Controller version: 5.0(1k)

ACI Switch version: 15.0(1k)

ACI switch hardware tested:

            Leaf switch: N9K-C93360YC-FX2, N9K-C9336C-FX2, N9K-C93240YC-FX2, N9K-C93180YC-FX,   N9K-C9348GC-FXP

            Spine switch: N9K-C9364C

 

The complete list of the supported hardware is available here:

https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/5x/l3-configuration/cisco-apic-layer-3-networking-configuration-guide-51x/m-sr-mpls-v2.html#Cisco_Reference.dita_b951e701-0dd7-4198-ac19-1c4f322d4054__section_l1l_r1m_vkb

 

The configuration will be the same regardless of the hardware combination used.

Segment Routing Considerations

As this document is focused on the SR/MPLS Handoff between ACI and an MPLS core, the segment routing base configuration of the MPLS core will not be detailed. This base configuration is related to IGP Segment Routing configuration (SRGB definition, Prefix-SID configuration).

From a design point of view, the core SR domain may extend to Cisco ACI fabrics when implementing the SR/MPLS Handoff. When this is the case, the Segment Identifier (Node-SID) used for ACI leafs must be unique across the SR domain, including the SR core.

It is recommended that you use the same Segment Routing Global Block (SRGB) value across the SR domain to ease operations. The SRGB considered in this document is [16000, 32000] across the whole network.

Testbed IP Addresses

Table 1.        Loopback Addresses

Node

Loopback IP Address

LEAF3

20.204.103.1/32

LEAF4

20.204.104.1/32

LEAF5

20.204.105.1/32

LEAF6

20.204.106.1/32

LEAF7

20.204.107.1/23

LEAF8

20.204.108.1/32

LEAF11

20.204.111.1/32

LEAF12

20.204.112.1/32

LEAF13

20.204.113.1/32

LEAF14

20.204.114.1/32

PE1

201.201.201.1/32

PE2

201.201.201.2/32

PE3

201.201.201.3/32

PE6

201.201.201.6/32

PE7

201.201.201.7/32

AGG2-PE1

201.221.201.1/32

AGG2-PE2

201.221.201.2/32

AGG2-RR1-EVPN

201.221.202.3/32

AGG2-RR2-EVPN

201.221.202.4/32

AGG3-PE1

201.231.201.1/32

AGG3-PE2

201.231.201.2/32

 

Basic SR-Handoff

Design

 

 

 

DiagramDescription automatically generated

Figure 8.            

Basic Communication Between the ACI Main DC and Remote Leaf

 

In Figure 8, two workloads A and B respectively connected to the ACI main DC with a remote leaf to communicate. There is no transport requirement for the traffic flow.

SR/MPLS handoff is implemented between the border leafs and the DC-PEs. The central Cisco ACI fabric uses Leaf3 as a border leaf directly connected to PE2 acting as a DC-PE. The remote leaf Leaf5 is directly connected to PE6 acting as a DC-PE.

ACI and SP network are using different AS numbers.

Note:      Using iBGP between ACI border leaf and SP core is NOT supported yet.

The design assumes that the same loopback is used for the dataplane and controlplane which is the simplest solution even if ACI supports having a separate loopback for controlplane and dataplane.

From a tenant configuration standpoint, VRFs are not stretched across sites. Each ACI site must use a different VRF with a different set of route targets (RT). In our example, the Cisco ACI fabric has a VRF called TENANT_BASIC_21 which uses the RT 1:2110005 while the remote leaf site has a VRF called TENANT_BASIC_2121 which uses the RT 1:2110006. The SP core will use the RT 1:2110000 in the VPN unicast address-family domain.

Note:      When the VRF is stretched, a VxLAN tunnel is automatically created, and the VxLAN path will be used instead of the SR/MPLS path. More details can be found in the following section.

While the setup is presented as a fabric site communicating with a remote leaf site, other combinations have been tested and are fully supported as detailed below:

 

Graphical user interface, diagramDescription automatically generated

Figure 9.             

Various Validated Traffic Combinations

The configuration remains the same in all the listed cases.

 

Infrastructure Configuration

The infrastructure configuration consists in setting up the SR/MPLS handoff between the DC-PEs and the ACI border leafs. In this simple use case, we assume that there is no multihoming of the border leaf to the DC-PEs and there is no tuning related to fast-convergence. The next section will cover multihoming.

 

PE2 DC-PE Configuration

Interface to BL  Configuration

 

interface TenGigE0/0/0/0/4

 description connected to ifav204-leaf3:1/1 used for SR/MPLS

 ipv4 address 120.1.53.2 255.255.255.0

 load-interval 30

!

 

BGP Labeled Unicast Configuration

The BGP LU (Labeled Unicast) configuration requires several configuration blocks:

1.     First, BGP LU address-family must be configured as well as redistribution of the local loopback address to BGP LU. The filtering of the loopback address is enabled by using a route-policy CONNECTED-TO-BGP-LU leveraging on a prefix-set PFXSET-OWN-LO0 which defines the local loopback address. The route-policy takes the node-SID value associated to the loopback as an input parameter, so the node-SID can be attached as a BGP Prefix-SID attribute to the BGP-LU route. In our configuration, the node-SID “2” is used as a parameter of the route-policy.

 

router bgp 1

 bgp router-id 201.201.201.2

 address-family ipv4 unicast

  redistribute connected route-policy CONNECTED-TO-BGP-LU(2)

  allocate-label all

 !

!

prefix-set PFXSET-OWN-LO0

  201.201.201.2/32

end-set

!

route-policy CONNECTED-TO-BGP-LU($node_sid)

  if destination in PFXSET-OWN-LO0 then

    set label-index $node_sid

    pass

  endif

end-policy

!

 

 

2.     Then, the BGP-LU session with the ACI Border Leaf must be configured:

The proposed configuration of the BGP-LU uses a neighbor-group. The neighbor-group can be reused and applied to multiple BGP-LU peers if necessary. In our example, there is a single peer configured. By default, XR requires some explicit policies to be configured on eBGP peers, otherwise all routes are dropped.

As the DC-PE is directly connected to the ACI border leaf, there is no need to propagate the border leaf loopback addresses received by the DC-PE over the BGP-LU session into the SP network. The inbound policy SET-CT-NO-ADVERTISE ensures that the BGP-LU prefixes will not be propagated to any other BGP peer (if any) by setting the well-known “no-advertise” BGP community.

The outbound policy ADVERTISE-LO0-ONLY ensures that the DC-PE only propagates its local loopback address to the ACI-Leaf (this is again because the leaf and the DC-PE are directly connected).

router bgp 1

 neighbor-group ACI-site-LU

  remote-as 987654321

  address-family ipv4 labeled-unicast

   route-policy SET-CT-NO-ADVERTISE in

   maximum-prefix 10 80 warning-only

   route-policy ADVERTISE-LO0-ONLY out

  !

 !

 neighbor 120.1.53.1

  use neighbor-group ACI-site-LU

 !

!

community-set COMSET-NO-ADVERTISE

  no-advertise

end-set

!

route-policy SET-CT-NO-ADVERTISE

  set community COMSET-NO-ADVERTISE

end-policy

!

route-policy ADVERTISE-LO0-ONLY

  if destination in PFXSET-OWN-LO0 then

    pass

  else

    drop

  endif

end-policy

!

 

 

3.     The last step is to activate MPLS forwarding on the interface towards the leaf.

 

router bgp 1

 mpls activate

  interface TenGigE0/0/0/0/4

!

router static

 address-family ipv4 unicast

  120.1.53.1/32 TenGigE0/0/0/0/4

 

BGP Overlay Configuration

 

The BGP overlay configuration consists in activating the BGP service overlay address families on the DC-PE.

On the SP core side, the DC-PE must run the VPNv4/VPNv6 address families while EVPN address family runs towards the ACI leaf.

Note:      Although the ACI border leaf and the DC-PE are directly connected, the EVPN session runs between loopback addresses. Then eBGP multihop must be implemented on the EVPN session.

In our configurations, constrained route distribution is also activated on the VPNv4/v6 peers but it is not required to be activated.

As for the BGP-LU session, the provided configurations are for a single-homing use case without fast-convergence.

router bgp 1

 address-family vpnv4 unicast

 !

 address-family vpnv6 unicast

 !

 address-family ipv4 rt-filter

 !

 address-family l2vpn evpn

 !

 neighbor-group ACI-site-EVPN

  remote-as 987654321

  ebgp-multihop 255

  update-source Loopback0

  address-family l2vpn evpn

  !

 !

 neighbor-group RR-VPNunicast

  remote-as 1

  update-source Loopback0

  address-family vpnv4 unicast

  !

  address-family vpnv6 unicast

  !

  address-family ipv4 rt-filter

  !

 neighbor 20.204.103.1

  use neighbor-group ACI-site-EVPN

  address-family l2vpn evpn

   route-policy PASS in

   route-policy PASS out

  !

 neighbor 202.202.202.102

  use neighbor-group RR-VPNunicast

 !

 neighbor 202.202.202.103

  use neighbor-group RR-VPNunicast

!

route-policy PASS

 pass

end-policy

!

 

The last step is to enable the EVPN/VPNvX stitching functionality.

On the ACI neighbor-group, the EVPN address-family must enable stitching to VPNvX address-family by using the following commands:

router bgp 1

  neighbor-group ACI-site-EVPN

  address-family l2vpn evpn

   import stitching-rt re-originate

   advertise vpnv4 unicast re-originated stitching-rt

   advertise vpnv6 unicast re-originated stitching-rt

  !

 !

 

The EVPN/VPNvX stitching is based on the concept of stitching and non-stitching RTs. For instance, an EVPN route is learned by DC-PE with an RT configured as stitching RT, the EVPN routes with stitching RTs are imported in a local VRF and re-originated into VPNvX using a non-stitching RT value. Reverse processing happens when a VPNvX route is received with a non-stitching RT, the route is imported in the VRF and re-originated into EVPN using a stitching RT.

On the VPN RR neighbor-group, stitching also must be activated using the following configuration.

router bgp 1

 neighbor-group RR-VPNunicast

  address-family vpnv4 unicast

   import re-originate stitching-rt

   advertise vpnv4 unicast re-originated

  !

  address-family vpnv6 unicast

   import re-originate stitching-rt

   advertise vpnv6 unicast re-originated

 

PE6 DC-PE configuration

 

PE6 configuration is similar to the PE2 configuration.

interface TenGigE0/0/0/12

 description connected to ifav204-leaf5:1/1 used for SR/MPLS

 ipv4 address 120.1.62.2 255.255.255.0

 load-interval 30

!

router static

 address-family ipv4 unicast

  120.1.62.1/32 TenGigE0/0/0/12

!

router bgp 1

 bgp router-id 201.201.201.6

 mpls activate

  interface TenGigE0/0/0/12

 !

 address-family ipv4 unicast

  redistribute connected route-policy CONNECTED-TO-BGP-LU(6)

  allocate-label all

 !

 address-family vpnv4 unicast

 

 !

 address-family vpnv6 unicast

 !

 address-family ipv4 rt-filter

 !

 address-family l2vpn evpn

 !

neighbor-group ACI-site-EVPN

  remote-as 987654321

  ebgp-multihop 255

  update-source Loopback0

  address-family l2vpn evpn

   import stitching-rt re-originate

   advertise vpnv4 unicast re-originated stitching-rt

   advertise vpnv6 unicast re-originated stitching-rt

  !

 !

 neighbor-group RR-VPNunicast

  remote-as 1

  update-source Loopback0

  address-family vpnv4 unicast

   import re-originate stitching-rt

   advertise vpnv4 unicast re-originated

  !

  address-family vpnv6 unicast

   import re-originate stitching-rt

   advertise vpnv6 unicast re-originated

  !

  address-family ipv4 rt-filter

  !

 !

 neighbor 120.1.62.1

  use neighbor-group ACI-site-LU

  address-family ipv4 labeled-unicast

   route-policy SET-CT-NO-ADVERTISE in

   maximum-prefix 10 80 warning-only

   route-policy ADVERTISE-LO0-ONLY out

  !

 !

 neighbor 20.204.105.1

  use neighbor-group ACI-site-EVPN

  address-family l2vpn evpn

   route-policy PASS in

   route-policy PASS out

  !

 !

!

prefix-set PFXSET-OWN-LO0

  201.201.201.6/32

end-set

!

route-policy ADVERTISE-LO0-ONLY

  if destination in PFXSET-OWN-LO0 then

    pass

  else

    drop

  endif

end-policy

!

community-set COMSET-NO-ADVERTISE

  no-advertise

end-set

!

route-policy SET-CT-NO-ADVERTISE

  set community COMSET-NO-ADVERTISE

end-policy

!

route-policy PASS

 pass

end-policy

!

route-policy CONNECTED-TO-BGP-LU($node_sid)

  if destination in PFXSET-OWN-LO0 then

    set label-index $node_sid

    pass

  endif

end-policy

 

ACI Configuration

The SR/MPLS Infra L3out is configured in the “INFRA” tenant on the border leaf.

The configuration contains underlay BGP-LU and overlay BGP EVPN sessions for SR/MPLS Handoff.

Tenant VRFs must then be selectively attached to ACI Infra-L3out(s) to advertise Tenant prefixes to DC-PE routers and import MPLS VPN prefixes from DC-PE.

As the ACI configuration is provided as XML that can easily be imported in Cisco APIC, each block of configuration is associated with a tag comment in the XML. The explanation of the block associated with each tag comment is provided below.

 

<?xml version="1.0" encoding="utf-8"?>

<imdata totalCount="1">

     <fvTenant

               dn="uni/tn-infra"

               name="infra"

               >

               <!-- Tag_1 Default Label Range config in Infra Tenant -->

               <mplsLabelPol

                             maxDynamicLabel="525286"

                             maxStaticLabel="0"

                             minDynamicLabel="16"

                             minStaticLabel="0"

                             name="default"

                             >

                              <mplsSrgbLabelPol

                                                localId="1"

                                                maxSrgbLabel="32000"

                                                minSrgbLabel="16000"

                                                />

               </mplsLabelPol>

               <!-- Tag_2 Default interface config in Infra Tenant -->

               <mplsIfPol

                          name="default"

                          />

               <!-- Tag_3 MPLS Infra L3out towards DC PE Location 2-1 -->

               <l3extOut

                         enforceRtctrl="export"

                         mplsEnabled="yes"

                         name="2-1"

                         targetDscp="unspecified"

                         >

                              <mplsExtP

                                        >

                                             <mplsRsLabelPol

                                                             tDn="uni/tn-infra/mplslabelpol-default"

                                                             />

                              </mplsExtP>

                              <l3extRsL3DomAtt

                                               tDn="uni/l3dom-L3Dom"

                                               />

                              <l3extRsEctx

                                           tnFvCtxName="overlay-1"

                                           />

                              <l3extProvLbl

                                            name="2-1"

                                            tag="yellow-green"

                                            />

                              <l3extLNodeP

                                           name="2-1_nodeProfile"

                                           tag="yellow-green"

                                           targetDscp="unspecified"

                                           >

                                             <l3extRsNodeL3OutAtt

                                                                  rtrId="30.204.103.1"

                                                                  rtrIdLoopBack="no"

                                                                  tDn="topology/pod-1/node-103"

                                                                  >

                                                            <l3extLoopBackIfP addr="20.204.103.1"

                                                                              >

                                                                          <mplsNodeSidP

                                                                                        loopbackAddr="20.204.103.1"

                                                                                        sidoffset="45"

                                                                                        />

                                                            </l3extLoopBackIfP>

                                             </l3extRsNodeL3OutAtt>

                                             <l3extRsLNodePMplsCustQosPol

                                                                          tDn="uni/tn-infra/qosmplscustom-"

                                                                          />

                                             <l3extLIfP

                                                        name="2-1_interfaceProfile"

                                                        prio="unspecified"

                                                        tag="yellow-green"

                                                        >

                                                            <l3extRsPathL3OutAtt addr="120.1.53.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-103/pathep-[eth1/1]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.53.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <mplsIfP

                                                                     >

                                                                          <mplsRsIfPol

                                                                                       tnMplsIfPolName="default"

                                                                                       />

                                                            </mplsIfP>

                                             </l3extLIfP>

                                             <bgpInfraPeerP addr="201.201.201.2"

                                                            addrTCtrl="af-ucast"

                                                            adminSt="enabled"

                                                            allowedSelfAsCnt="3"

                                                            ctrl="allow-self-as,dis-peer-as-check,send-com,send-ext-com"

                                                            dataPlaneAddr="0.0.0.0"

                                                            peerT="SR/MPLS"

                                                            remoteIntersiteRR="no"

                                                            srcIfT="l3out-loopback"

                                                            trustCtrl="untrusted"

                                                            ttl="16"

                                                            weight="0">

                                                            <bgpRsPeerPfxPol

                                                                             />

                                                            <bgpAsP

                                                                    asn="1"

                                                                    />

                                             </bgpInfraPeerP>

                              </l3extLNodeP>

                              <l3extInstP

                                          floodOnEncap="disabled"

                                          matchT="AtleastOne"

                                          name="2-1_mplsInstP"

                                          prefGrMemb="exclude"

                                          prio="unspecified"

                                          targetDscp="unspecified"

                                          >

                                             <fvRsCustQosPol

                                                             />

                              </l3extInstP>

                              <bgpExtP

                                       />

               </l3extOut>

               <!-- Tag_4 MPLS Infra L3out towards DC PE Location 2-1-2-1 -->

               <l3extOut

                         enforceRtctrl="export"

                         mplsEnabled="yes"

                         name="2-1-2-1"

                         targetDscp="unspecified"

                         >

                              <mplsExtP

                                        >

                                             <mplsRsLabelPol

                                                             tDn="uni/tn-infra/mplslabelpol-default"

                                                             />

                              </mplsExtP>

                              <l3extRsL3DomAtt

                                               tDn="uni/l3dom-L3Dom"

                                               />

                              <l3extRsEctx

                                           tnFvCtxName="overlay-1"

                                           />

                              <l3extProvLbl

                                            name="2-1-2-1"

                                            tag="yellow-green"

                                            />

                              <l3extLNodeP

                                           name="2-1-2-1_nodeProfile"

                                           tag="yellow-green"

                                           targetDscp="unspecified"

                                           >

                                             <l3extRsNodeL3OutAtt

                                                                  rtrId="30.204.105.1"

                                                                  rtrIdLoopBack="no"

                                                                  tDn="topology/pod-1/node-105"

                                                                  >

                                                            <l3extLoopBackIfP addr="20.204.105.1"

                                                                              >

                                                                          <mplsNodeSidP

                                                                                        loopbackAddr="20.204.105.1"

                                                                                        sidoffset="47"

                                                                                        />

                                                            </l3extLoopBackIfP>

                                             </l3extRsNodeL3OutAtt>

                                             <l3extRsLNodePMplsCustQosPol

                                                                          tDn="uni/tn-infra/qosmplscustom-"

                                                                          />

                                             <l3extLIfP

                                                        name="2-1-2-1_interfaceProfile"

                                                        prio="unspecified"

                                                        tag="yellow-green"

                                                        >

                                                            <l3extRsPathL3OutAtt addr="120.1.62.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-105/pathep-[eth1/1]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.62.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <mplsIfP

                                                                     >

                                                                          <mplsRsIfPol

                                                                                       tnMplsIfPolName="default"

                                                                                       />

                                                            </mplsIfP>

                                             </l3extLIfP>

                                             <bgpInfraPeerP addr="201.201.201.6"

                                                            addrTCtrl="af-ucast"

                                                            adminSt="enabled"

                                                            allowedSelfAsCnt="3"

                                                            ctrl="allow-self-as,dis-peer-as-check,send-com,send-ext-com"

                                                            dataPlaneAddr="0.0.0.0"

                                                            peerT="SR/MPLS"

                                                            remoteIntersiteRR="no"

                                                            srcIfT="l3out-loopback"

                                                            trustCtrl="untrusted"

                                                            ttl="16"

                                                            weight="0">

                                                            <bgpRsPeerPfxPol

                                                                             />

                                                            <bgpAsP

                                                                    asn="1"

                                                                    />

                                             </bgpInfraPeerP>

                              </l3extLNodeP>

                              <l3extInstP

                                          floodOnEncap="disabled"

                                          matchT="AtleastOne"

                                          name="2-1-2-1_mplsInstP"

                                          prefGrMemb="exclude"

                                          prio="unspecified"

                                          targetDscp="unspecified"

                                          >

                                             <fvRsCustQosPol

                                                             />

                              </l3extInstP>

                              <bgpExtP

                                       />

               </l3extOut>

     </fvTenant>

</imdata>

Tag_1 Default Label Range config in INFRA Tenant:

This block creates an MPLS Label policy.

ACI supports same Segment Routing Global Block (SRGB) across all fabrics. Default SRGB range in ACI is from 16000 to 23999. SRGB range is user configurable with a minimum (16000) and maximum (471804).

Note:      ACI always advertises implicit-null for underlay label (Transport loopback). VRF Aggregate label is advertised from a different range (948576 to 1068576) than the SRGB range.

In our example, we use an SRGB starting from 16000 to 32000.

Tag_2 Default interface Profile config in INFRA Tenant:

This block creates an MPLS Interface profile policy. We have a default policy  and have attached it to the interface connected to DC-PE.

Tag_3 and Tag_4 MPLS Infra L3out towards DC PE:

This block handles the configuration of the SR/MPLS INFRA L3 out.

The container “mplsExtP” attaches the created MPLS Label policy to the SR/MPLS INFRA L3out.

The container “l3extLNodeP” is a NodeProfile where we reference the leaf which is connected to DC-PE.

Note:      In our example, the node already has a Router-id. For instance leaf3 has a router-id of 30.204.103.1.

SR/MPLS is supported on L3 port, L3 Sub-interface, Port-channel, Port-channel sub-interface. SR/MPLS is not supported for vPC and SVI.

The BGP EVPN peer type must be set to SR MPLS. As the EVPN BGP session is based on loopback addresses, the TTL for the session must be at minimum 2 (eBGP multihop). In the provided configuration, we have configured PE02 EVPN loopback as the peer address (201.201.201.2) in location 2-1.

It is important to set the allow-as-in (allow-self-as) attribute in EVPN peer so that the ACI site can receive BGP routes from a remote site with the same AS number.

Example:

<bgpInfraPeerP addr="201.201.201.2" addrTCtrl="af-ucast" adminSt="enabled" allowedSelfAsCnt="3" annotation="" ctrl="allow-self-as,dis-peer-as-check,send-com,send-ext-com"

 

Verifications

XR BGP-LU

The BGP session state can be displayed using the following command. In the output below, the PE02 has received one prefix from the leaf.

RP/0/RSP0/CPU0:PE02#show bgp ipv4 labeled-unicast summary

Mon Aug 31 13:13:29.718 UTC

BGP router identifier 201.201.201.2, local AS number 1

BGP generic scan interval 60 secs

Non-stop routing is enabled

BGP table state: Active

Table ID: 0xe0000000   RD version: 273

BGP main routing table version 273

BGP NSR Initial initsync version 42 (Reached)

BGP NSR/ISSU Sync-Group versions 0/0

BGP scan interval 60 secs

 

BGP is operating in STANDALONE mode.

 

 

Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer

Speaker             273        273        273        273         273           0

 

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd

120.1.53.1        0 987654321    1352    1351      273    0    0 22:27:28          1

 

The received prefixes can be verified using the following command. The DC-PE correctly receives the leaf loopback address 20.204.103.1/32 over the BGP-LU session.

 

RP/0/RSP0/CPU0:PE02#show bgp ipv4 labeled-unicast neighbors 120.1.53.1 routes

Mon Aug 31 13:14:34.270 UTC

BGP router identifier 201.201.201.2, local AS number 1

BGP generic scan interval 60 secs

Non-stop routing is enabled

BGP table state: Active

Table ID: 0xe0000000   RD version: 273

BGP main routing table version 273

BGP NSR Initial initsync version 42 (Reached)

BGP NSR/ISSU Sync-Group versions 0/0

BGP scan interval 60 secs

 

Status codes: s suppressed, d damped, h history, * valid, > best

              i - internal, r RIB-failure, S stale, N Nexthop-discard

Origin codes: i - IGP, e - EGP, ? - incomplete

   Network            Next Hop            Metric LocPrf Weight Path

*> 20.204.103.1/32    120.1.53.1                             0 987654321 i

 

When displaying the details of prefix 20.204.103.1/32, we can see the implicit-null MPLS label, as well as the Node-SID value (45) that has been advertised by the leaf.

 

RP/0/RSP0/CPU0:PE02#show bgp ipv4 unicast 20.204.103.1/32

Thu Sep  3 02:51:28.796 UTC

BGP routing table entry for 20.204.103.1/32

Versions:

  Process           bRIB/RIB  SendTblVer

  Speaker               1867        1867

    Local Label: 16045

Last Modified: Sep  2 16:29:40.144 for 10:21:49

Paths: (1 available, best #1, not advertised to any peer)

  Not advertised to any peer

  Path #1: Received by speaker 0

  Not advertised to any peer

  987654321

    120.1.53.1 from 120.1.53.1 (30.204.103.1)

      Received Label 3

      Origin IGP, localpref 100, valid, external, best, group-best, labeled-unicast

      Received Path ID 0, Local Path ID 1, version 1867

      Community: no-advertise

      Origin-AS validity: (disabled)

      Label-Index: 45

RP/0/RSP0/CPU0:PE02#

 

The local MPLS label value is computed as expected by using:

base SRGB + Index = 16000 + 45 = 16045.

 

ACI BGP-LU:

 

The default route-maps attached to the BGP-LU session can be displayed by using the following command.

ifav204-leaf3# show bgp ipv4 labeled-unicast neighbors 120.1.53.2 vrf overlay-1

BGP neighbor is 120.1.53.2,  remote AS 1, ebgp link,  Peer index 5

 

  For address family: IPv4 Unicast

  BGP table version 7, neighbor version 0

  0 accepted paths consume 0 bytes of memory

  0 sent paths

  0 denied paths

  Maximum prefixes allowed 20000 (action is reject)

  Threshold for warning messages 75%

  Inbound route-map configured is permit-all, handle obtained

  Outbound route-map configured is permit-mpls-cp-loopback, handle obtained

 

  For address family: IPv4 Label Unicast

  BGP table version 10, neighbor version 10

  1 accepted paths consume 112 bytes of memory

  1 sent paths

  0 denied paths

  Inbound route-map configured is permit-all, handle obtained

  Outbound route-map configured is permit-mpls-dp-loopback, handle obtained

 

ifav204-leaf3#

 

----

 

ifav204-leaf3# show route-map permit-all

route-map permit-all, permit, sequence 2

  Match clauses:

  Set clauses:

ifav204-leaf3# show route-map permit-mpls-cp-loopback

route-map permit-mpls-cp-loopback, permit, sequence 1

  Match clauses:

    ip address prefix-lists: infra_mpls_cp_tep

  Set clauses:

ifav204-leaf3# show route-map permit-mpls-dp-loopback

route-map permit-mpls-dp-loopback, permit, sequence 1

  Match clauses:

    ip address prefix-lists: infra_mpls_dp_tep

  Set clauses:

ifav204-leaf3# show ip prefix-list infra_mpls_dp_tep

ip prefix-list infra_mpls_dp_tep: 1 entries

   seq 1 permit 20.204.103.1/32

ifav204-leaf3# show ip prefix-list infra_mpls_cp_tep

ip prefix-list infra_mpls_cp_tep: 1 entries

   seq 1 permit 20.204.103.1/32

ifav204-leaf3#

 

The default route-maps have the following purpose:

      permit-all allows inbound prefixes of ipv4 labeled unicast and ipv4 unicast address family

      permit-mpls-cp-loopback advertises EVPN control plane loopback

      permit-mpls-dp-loopback advertises MPLS transport loopback

 

On LEAF3, the loopback from PE02 is correctly received using BGP-LU. As the DC-PE and the BL are directly connected, the received label is 3 (implicit-null), corresponding to implicit-null. The BGP Prefix SID attribute (Label index 2) is also correctly received.

ifav204-leaf3# show bgp ipv4 labeled-unicast 201.201.201.2/32 vrf overlay-1

BGP routing table information for VRF overlay-1, address family IPv4 Label Unicast

BGP routing table entry for 201.201.201.2/32, version 20 dest ptr 0xa4977ed8

Paths: (1 available, best #1)

Flags: (0x08001a 00000000) on xmit-list, is in urib, is best urib route, is in HW

  label af: version 29, (0x100002) on xmit-list

 

  Advertised path-id 1, Label AF advertised path-id 1

  Path type: external 0x40000028 0x0 ref 0 adv path ref 2, path is valid, is best path

  AS-Path: 1 , path sourced external to AS

    120.1.53.2 (metric 0) from 120.1.53.2 (201.201.201.2)

      Origin incomplete, MED 0, localpref 100, weight 0 tag 0, propagate 0

      Received label 3

      Prefix-SID Attribute: Length: 10

        Label Index TLV: Length 7, Flags 0x0 Label Index 2

 

  Path-id 1 not advertised to any peer

 

  Label AF advertisement

  Path-id 1 not advertised to any peer

 

ifav204-leaf3#

 

XR BGP EVPN Session State

 

RP/0/RSP0/CPU0:PE02#show bgp l2vpn evpn summary

Mon Aug 31 13:15:43.258 UTC

BGP router identifier 201.201.201.2, local AS number 1

BGP generic scan interval 60 secs

Non-stop routing is enabled

BGP table state: Active

Table ID: 0x0   RD version: 0

BGP main routing table version 147

BGP NSR Initial initsync version 147 (Reached)

BGP NSR/ISSU Sync-Group versions 0/0

BGP scan interval 60 secs

 

BGP is operating in STANDALONE mode.

 

 

Process       RcvTblVer   bRIB/RIB   LabelVer  ImportVer  SendTblVer  StandbyVer

Speaker             147        147        147        147         147           0

 

Neighbor        Spk    AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down  St/PfxRcd

20.204.103.1      0 987654321    1429    1556      147    0    0 22:29:19         0

 

 

ACI BGP EVPN Session State

 

ifav204-leaf3# show bgp l2vpn evpn summary vrf overlay-1

BGP summary information for VRF overlay-1, address family L2VPN EVPN

BGP router identifier 30.204.103.1, local AS number 987654321

BGP table version is 2004, L2VPN EVPN config peers 2, capable peers 2

401 network entries and 520 paths using 65452 bytes of memory

BGP attribute entries [81/12960], BGP AS path entries [0/0]

BGP community entries [5/160], BGP clusterlist entries [2/8]

 

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

201.201.201.2   4     1    1203     642     2004    0    0 10:07:21 0

 

ifav204-leaf3#

 

At this stage, no routes are received with the ACI border leaf. Tenant configuration is required to exchange VPN routes.

Tenant Configuration

PE2 DC-PE

 

Each DC-PE requires a VRF to be configured to perform the EVPN/VPNvX stitching function.

The VRF is configured with a stitching RT value (in our design, this is the RT used in the EVPN domain with the ACI DC) and a non-stitching RT value (the RT used in the VPNvX domain of the SP core).

Each DC-PE must use a different route-distinguisher value for the stitching VRF.

Also, the user must ensure that the DC-PE is not configured with the same RD value as the VRF located in the connected ACI site. Using different RDs is required to enable the route re-origination between EVPN and VPNvX. Reorigination cannot happen if a learned route has the same RD as the stitching VRF.

 

vrf TENANT_BASIC

 address-family ipv4 unicast

  import route-target

   1:2110000

   1:2110005 stitching

  !

  export route-target

   1:2110000

   1:2110005 stitching

  !

 !

 address-family ipv6 unicast

  import route-target

   1:2110000

   1:2110005 stitching

  !

  export route-target

   1:2110000

   1:2110005 stitching

  !

 !

 

!

router bgp 1

 vrf TENANT_BASIC

  rd 1:2110002

  address-family ipv4 unicast

  !

  address-family ipv6 unicast

  !

 

PE6 DC-PE

vrf TENANT_BASIC

 address-family ipv4 unicast

  import route-target

   1:2110000

   1:2110006 stitching

  !

  export route-target

   1:2110000

   1:2110006 stitching

  !

 !

 address-family ipv6 unicast

  import route-target

   1:2110000

   1:2110006 stitching

  !

  export route-target

   1:2110000

   1:2110006 stitching

  !

 !

 

!

router bgp 1

 vrf TENANT_BASIC

  rd 1:2110006

  address-family ipv4 unicast

  !

  address-family ipv6 unicast

  !

 

ACI Configuration

<?xml version="1.0" encoding="utf-8"?>

<imdata totalCount="1">

     <!-- Specify Name of Tenant as Required -->

     <fvTenant descr="Tenant with Basic SR MPLS Handoff between 2-1 and 2-1-2-1"

               dn="uni/tn-TENANT-BASIC"

               name="TENANT-BASIC"

               >

               <!-- Tag_1 User L3out Route Control Config -->

               <rtctrlSubjP

                            name="all"

                            >

                              <rtctrlMatchRtDest aggregate="yes"

                                                 fromPfxLen="0"

                                                 ip="100.0.0.0/8"

                                                 toPfxLen="0"

                                                 />

                              <rtctrlMatchRtDest aggregate="yes"

                                                 fromPfxLen="0"

                                                 ip="::/0"

                                                 toPfxLen="0"

                                                 />

               </rtctrlSubjP>

               <rtctrlProfile

                              name="Export-Pol"

                              type="combinable"

                              >

                              <rtctrlCtxP action="permit"

                                          name="Export-Pol"

                                          order="1"

                                          >

                                             <rtctrlRsCtxPToSubjP

                                                                  tnRtctrlSubjPName="all"

                                                                  />

                              </rtctrlCtxP>

               </rtctrlProfile>

               <rtctrlProfile

                              name="Import-Pol"

                              type="combinable"

                              >

                              <rtctrlCtxP action="permit"

                                          name="Import-Pol"

                                          order="1"

                                          >

                                             <rtctrlRsCtxPToSubjP

                                                                  tnRtctrlSubjPName="all"

                                                                  />

                              </rtctrlCtxP>

               </rtctrlProfile>

               <!-- Tag_2 User L3out Config towards DC PE Location 2-1 -->

               <l3extOut

                         enforceRtctrl="export"

                         mplsEnabled="yes"

                         name="LOCATION-2-1-2-1-MPLS-TENANT-L3OUT-2106"

                         targetDscp="unspecified"

                         >

                              <l3extRsEctx tnFvCtxName="TENANT_BASIC_2121"

                                           />

                              <rtctrlProfile

                                             name="Export-Pol"

                                             type="combinable"

                                             >

                                             <rtctrlCtxP action="permit"

                                                         name="Export-Pol"

                                                         order="1"

                                                         >

                                                            <rtctrlRsCtxPToSubjP

                                                                                 tnRtctrlSubjPName="all"

                                                                                 />

                                             </rtctrlCtxP>

                              </rtctrlProfile>

                              <rtctrlProfile

                                             name="Import-Pol"

                                             type="combinable"

                                             >

                                             <rtctrlCtxP action="permit"

                                                         name="Import-Pol"

                                                         order="1"

                                                         >

                                                            <rtctrlRsCtxPToSubjP

                                                                                 tnRtctrlSubjPName="all"

                                                                                 />

                                             </rtctrlCtxP>

                              </rtctrlProfile>

                              <l3extConsLbl

                                            name="2-1-2-1"

                                            owner="infra"

                                            tag="yellow-green"

                                            >

                                             <l3extRsLblToProfile direction="import"

                                                                  tDn="uni/tn-TENANT-BASIC/prof-Import-Pol"

                                                                  />

                                             <l3extRsLblToProfile

                                                                  direction="export"

                                                                  tDn="uni/tn-TENANT-BASIC/prof-Export-Pol"

                                                                  />

                                             <l3extRsLblToInstP tDn="uni/tn-TENANT-BASIC/out-LOCATION-2-1-2-1-MPLS-TENANT-L3OUT-2106/instP-LOCATION-2-1-2-1-MPLS-TENANT-L3OUT-2106-InstP"

                                                                />

                              </l3extConsLbl>

                              <l3extInstP

                                          floodOnEncap="disabled"

                                          matchT="AtleastOne"

                                          name="LOCATION-2-1-2-1-MPLS-TENANT-L3OUT-2106-InstP"

                                          prefGrMemb="exclude"

                                          prio="unspecified"

                                          targetDscp="unspecified"

                                          >

                                             <fvRsProv

                                                       intent="install"

                                                       matchT="AtleastOne"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <l3extSubnet

                                                          ip="::/0"

                                                          name="ipv6All"

                                                          scope="import-security"

                                                          />

                                             <l3extSubnet

                                                          ip="0.0.0.0/0"

                                                          name="ipv4All"

                                                          scope="import-security"

                                                          />

                                             <fvRsCustQosPol

                                                             />

                                             <fvRsCons

                                                       intent="install"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                              </l3extInstP>

                              <bgpExtP

                                       />

               </l3extOut>

               <!-- Tag_3 User L3out Config towards DC PE Location 2-1-2-1 -->

               <l3extOut

                         enforceRtctrl="export"

                         mplsEnabled="yes"

                         name="LOCATION-2-1-MPLS-TENANT-L3OUT-2105"

                         targetDscp="unspecified"

                         >

                              <l3extRsEctx tnFvCtxName="TENANT_BASIC_21"

                                           />

                              <rtctrlProfile

                                             name="Import-pol"

                                             type="combinable"

                                             >

                                             <rtctrlCtxP action="permit"

                                                         name="Import-pol"

                                                         order="1"

                                                         >

                                                            <rtctrlRsCtxPToSubjP

                                                                                 tnRtctrlSubjPName="all"

                                                                                 />

                                             </rtctrlCtxP>

                              </rtctrlProfile>

                              <rtctrlProfile

                                             name="Export-pol"

                                             type="combinable"

                                             >

                                             <rtctrlCtxP action="permit"

                                                         name="Export-pol"

                                                         order="1"

                                                         >

                                                            <rtctrlRsCtxPToSubjP

                                                                                 tnRtctrlSubjPName="all"

                                                                                 />

                                             </rtctrlCtxP>

                              </rtctrlProfile>

                              <l3extConsLbl

                                            name="2-1"

                                            owner="infra"

                                            tag="yellow-green"

                                            >

                                             <l3extRsLblToProfile direction="import"

                                                                  tDn="uni/tn-TENANT-BASIC/prof-Import-Pol"

                                                                  />

                                             <l3extRsLblToProfile direction="export"

                                                                  tDn="uni/tn-TENANT-BASIC/prof-Export-Pol"

                                                                  />

                                             <l3extRsLblToInstP tDn="uni/tn-TENANT-BASIC/out-LOCATION-2-1-MPLS-TENANT-L3OUT-2105/instP-LOCATION-2-1-MPLS-TENANT-L3OUT-2105-InstP"

                                                                />

                              </l3extConsLbl>

                              <l3extInstP

                                          floodOnEncap="disabled"

                                          matchT="AtleastOne"

                                          name="LOCATION-2-1-MPLS-TENANT-L3OUT-2105-InstP"

                                          prefGrMemb="exclude"

                                          prio="unspecified"

                                          targetDscp="unspecified"

                                          >

                                             <fvRsProv

                                                       intent="install"

                                                       matchT="AtleastOne"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <l3extSubnet

                                                          ip="::/0"

                                                          name="ipv6All"

                                                          scope="import-security"

                                                          />

                                             <l3extSubnet

                                                          ip="0.0.0.0/0"

                                                          name="ipv4All"

                                                          scope="import-security"

                                                          />

                                             <fvRsCustQosPol

                                                             />

                                             <fvRsCons

                                                       intent="install"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                              </l3extInstP>

                              <bgpExtP

                                       />

               </l3extOut>

               <!-- Tag_4 User VRF Config towards DC PE Location 2-1-2-1 -->

               <fvCtx bdEnforcedEnable="no"

                      ipDataPlaneLearning="enabled"

                      knwMcastAct="permit"

                      name="TENANT_BASIC_2121"

                      pcEnfDir="ingress"

                      pcEnfPref="enforced"

                      >

                              <fvRsVrfValidationPol

                                                    tnL3

                                                    />

                              <vzAny

                                     matchT="AtleastOne"

                                     prefGrMemb="disabled"

                                     />

                              <fvRsOspfCtxPol

                                              />

                              <fvRsCtxToEpRet

                                              />

                              <fvRsCtxToExtRouteTagPol

                                                       tnL3

                                                       />

                              <fvRsBgpCtxPol

                                             />

                              <bgpRtTargetP af="ipv4-ucast"

                                            >

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110006"

                                                          type="import"

                                                          />

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110006"

                                                          type="export"

                                                          />

                              </bgpRtTargetP>

                              <bgpRtTargetP af="ipv6-ucast"

                                            >

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110006"

                                                          type="import"

                                                          />

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110006"

                                                          type="export"

                                                          />

                              </bgpRtTargetP>

               </fvCtx>

               <!-- Tag_5 User VRF Config towards DC PE Location 2-1 -->

               <fvCtx bdEnforcedEnable="no"

                      ipDataPlaneLearning="enabled"

                      knwMcastAct="permit"

                      name="TENANT_BASIC_21"

                      pcEnfDir="ingress"

                      pcEnfPref="enforced"

                      >

                              <fvRsVrfValidationPol

                                                    tnL3

                                                    />

                              <vzAny

                                     matchT="AtleastOne"

                                     prefGrMemb="disabled"

                                     />

                              <fvRsOspfCtxPol

                                              />

                              <fvRsCtxToEpRet

                                              />

                              <fvRsCtxToExtRouteTagPol

                                                       tnL3

                                                       />

                              <fvRsBgpCtxPol

                                             />

                              <bgpRtTargetP af="ipv4-ucast"

                                            >

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110005"

                                                          type="import"

                                                          />

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110005"

                                                          type="export"

                                                          />

                              </bgpRtTargetP>

                              <bgpRtTargetP af="ipv6-ucast"

                                            >

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110005"

                                                          type="export"

                                                          />

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110005"

                                                          type="import"

                                                          />

                              </bgpRtTargetP>

               </fvCtx>

               <!-- Tag_6 User BD Config towards DC PE Location 2-1 -->

               <fvBD OptimizeWanBandwidth="no"

                     arpFlood="no"

                     epClear="no"

                     hostBasedRouting="no"

                     intersiteBumTrafficAllow="no"

                     intersiteL2Stretch="no"

                     ipLearning="yes"

                     ipv6McastAllow="no"

                     limitIpLearnToSubnets="yes"

                     llAddr="::"

                     mac="00:22:BD:F8:19:FF"

                     mcastAllow="no"

                     multiDstPktAct="bd-flood"

                     name="BD2105"

                     type="regular"

                     unicastRoute="yes"

                     unkMacUcastAct="proxy"

                     unkMcastAct="flood"

                     v6unkMcastAct="flood"

                     vmac="not-applicable">

                              <fvSubnet ctrl="nd"

                                        ip="2001:100:21:5::254/64"

                                        preferred="no"

                                        scope="public"

                                        virtual="no"/>

                              <fvSubnet ctrl="nd"

                                        ip="100.21.5.254/24"

                                        preferred="no"

                                        scope="public"

                                        virtual="no"/>

                              <fvRsMldsn

                                         />

                              <fvRsIgmpsn

                                          />

                              <fvRsCtx

                                       tnFvCtxName="TENANT_BASIC_21"

                                       />

                              <fvRsBdToEpRet

                                             resolveAct="resolve"

                                             />

                              <fvRsBDToOut

                                           tnL3extOutName="LOCATION-2-1-MPLS-TENANT-L3OUT-2105"

                                           />

                              <fvRsBDToNdP

                                           />

               </fvBD>

               <!-- Tag_7 User BD Config towards DC PE Location 2-1-2-1 -->

               <fvBD OptimizeWanBandwidth="no"

                     arpFlood="no"

                     epClear="no"

                     hostBasedRouting="no"

                     intersiteBumTrafficAllow="no"

                     intersiteL2Stretch="no"

                     ipLearning="yes"

                     ipv6McastAllow="no"

                     limitIpLearnToSubnets="yes"

                     llAddr="::"

                     mac="00:22:BD:F8:19:FF"

                     mcastAllow="no"

                     multiDstPktAct="bd-flood"

                     name="BD2106"

                     type="regular"

                     unicastRoute="yes"

                     unkMacUcastAct="proxy"

                     unkMcastAct="flood"

                     v6unkMcastAct="flood"

                     vmac="not-applicable">

                              <fvSubnet ctrl="nd"

                                        ip="2001:100:21:6::254/64"

                                        preferred="no"

                                        scope="public"

                                        virtual="no"/>

                              <fvSubnet ctrl="nd"

                                        ip="100.21.6.254/24"

                                        preferred="no"

                                        scope="public"

                                        virtual="no"/>

                              <fvRsMldsn

                                         />

                              <fvRsIgmpsn

                                          />

                              <fvRsCtx

                                       tnFvCtxName="TENANT_BASIC_2121"

                                       />

                              <fvRsBdToEpRet

                                             resolveAct="resolve"

                                             />

                              <fvRsBDToOut

                                           tnL3extOutName="LOCATION-2-1-2-1-MPLS-TENANT-L3OUT-2106"

                                           />

                              <fvRsBDToNdP

                                           />

               </fvBD>

               <!-- Tag_8 Application Profile Config towards DC PE Location 2-1 -->

               <fvAp

                     name="LOCATION-2-1-2-1-AP2106"

                     prio="unspecified"

                     >

                              <fvAEPg

                                      floodOnEncap="disabled"

                                      hasMcastSource="no"

                                      isAttrBasedEPg="no"

                                      matchT="AtleastOne"

                                      name="EPG106"

                                      pcEnfPref="unenforced"

                                      prefGrMemb="exclude"

                                      prio="unspecified"

                                      shutdown="no"

                                      >

                                             <fvRsProv

                                                       intent="install"

                                                       matchT="AtleastOne"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <fvRsPathAtt

                                                          encap="vlan-2101"

                                                          instrImedcy="lazy"

                                                          mode="regular"

                                                          primaryEncap="unknown"

                                                          tDn="topology/pod-1/paths-105/pathep-[eth1/10]"

                                                          />

                                             <fvRsDomAtt bindingType="none"

                                                         classPref="encap"

                                                         encap="unknown"

                                                         encapMode="auto"

                                                         epgCos="Cos0"

                                                         epgCosPref="disabled"

                                                         instrImedcy="lazy"

                                                         netflowDir="both"

                                                         netflowPref="disabled"

                                                         numPorts="0"

                                                         portAllocation="none"

                                                         primaryEncap="unknown"

                                                         primaryEncapInner="unknown"

                                                         resImedcy="lazy"

                                                         secondaryEncapInner="unknown"

                                                         switchingMode="native"

                                                         tDn="uni/phys-phys"

                                                         untagged="no"

                                                         />

                                             <fvRsCons

                                                       intent="install"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <fvRsCustQosPol

                                                             />

                                             <fvRsBd

                                                     tnFvBDName="BD2106"

                                                     />

                              </fvAEPg>

               </fvAp>

               <!-- Tag_9 Application Profile Config towards DC PE Location 2-1-2-1 -->

               <fvAp

                     name="LOCATION-2-1-AP2105"

                     prio="unspecified"

                     >

                              <fvAEPg

                                      floodOnEncap="disabled"

                                      hasMcastSource="no"

                                      isAttrBasedEPg="no"

                                      matchT="AtleastOne"

                                      name="EPG105"

                                      pcEnfPref="unenforced"

                                      prefGrMemb="exclude"

                                      prio="unspecified"

                                      shutdown="no"

                                      >

                                             <fvRsProv

                                                       intent="install"

                                                       matchT="AtleastOne"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <fvRsPathAtt

                                                          encap="vlan-2101"

                                                          instrImedcy="immediate"

                                                          mode="regular"

                                                          primaryEncap="unknown"

                                                          tDn="topology/pod-1/paths-101/pathep-[eth1/10]"

                                                          />

                                             <fvRsDomAtt bindingType="none"

                                                         classPref="encap"

                                                         encap="unknown"

                                                         encapMode="auto"

                                                         epgCos="Cos0"

                                                         epgCosPref="disabled"

                                                         instrImedcy="lazy"

                                                         netflowDir="both"

                                                         netflowPref="disabled"

                                                         numPorts="0"

                                                         portAllocation="none"

                                                         primaryEncap="unknown"

                                                         primaryEncapInner="unknown"

                                                         resImedcy="immediate"

                                                         secondaryEncapInner="unknown"

                                                         switchingMode="native"

                                                         tDn="uni/phys-phys"

                                                         untagged="no"

                                                         />

                                             <fvRsCons

                                                       intent="install"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <fvRsCustQosPol

                                                             />

                                             <fvRsBd

                                                     tnFvBDName="BD2105"

                                                     />

                              </fvAEPg>

               </fvAp>

     </fvTenant>

</imdata>

Tag_1 User L3out Route Control Configuration:

 

Outbound Route Policy:

An outbound route policy is required to advertise any prefix including BD subnets. The default outbound route policy is to not advertise any prefix. Explicit outbound route-map can be configured with the following functionalities but are not limited to them.

      Match prefixes to advertise to SR/MPLS network

      Match prefixes and community to advertise prefixes to SR/MPLS network

      Set community including color community based on prefix and/or community match

 

Inbound Route Policy:

By default, the inbound route policy is to accept all prefixes. Explicit inbound route-map can be configured to match prefixes to selectively deny in the fabric if required.

 

Tag_2 and Tag_3 User L3out Configuration towards DC PE

 

This specifies the SR MPLS Tenant L3out for each site.

Each VRF that needs to be advertised towards SR MPLS core must be associated to an SR/MPLS-Infra L3out. Import and export route-maps can be optionally configured to apply route-policies based on prefixes and/or communities to advertise prefixes into SR network or receive prefixes from SR network.

Detailed explanation of above XML configuration:

      “rtctrlProfile”:  Route Control Profile in the above configuration is used to apply the inbound and outbound route-policies.

      “l3extConsLbl”: Layer 3 Consumer Label in the above configuration is used to map SR MPLS Infra Location to User Tenant SR MPLS location from where the route needs to be leaked.

      “l3extInstP”:  External EPG is defined on SR/MPLS Tenant L3out.

      User can define subnet and those subnets will be used to apply ACI security policies (contract)

      External EPG subnet is used to leak prefixes in another VRF using flags. If route-leak and security flag is enabled on external EPG subnet, then subnet can be leaked to another VRF.

      User can also configure external EPG subnet with aggregated flag to leak prefixes to another VRF. Contract must be defined to leaf prefixes and allow communication across VRFs.

      External EPG on SR/MPLS is not used for routing policies such as applying route-map to advertise/deny prefix advertisement.

 

Tag_4 and Tag_5 User VRF Config towards DC

 

Specify the VRF and the corresponding EVPN RTs. In our case, the RT 1:2110006 is used both for importing and exporting routes for the VRF in site 2-1-2-1 and the RT 1:210005 is used for the VRF in site 2-1.

 

Tag_6 and Tag_7 User BD Config towards DC PE

 

Specify the Bridge Domain (BD) subnet where the host is attached. Make sure you make the scope of the BD subnet public so that the route can be advertised externally. This can be achieved by setting the tag “scope = public” as displayed in the example configuration.

<fvSubnet  ctrl="nd" descr="" ip="100.21.6.254/24" scope="public" virtual="no"/>

 

Subnets used in the above example:

      100.21.5.0/24 is the subnet in Location 2-1, with a gateway being 100.21.5.254.

      100.21.6.0/24 is the subnet 2-1-2-1, with a gateway being 100.21.6.254.

      These subnets will be exchanged via SR MPLS across these two locations.

 

Tag_8 and Tag_9 Application Profile Config towards DC PE

 

Specify the end point group and the static port information where your host is attached. There is no change with respect to Application profile and EPG configuration when it comes to SR MPLS. It is configured like any regular tenant configuration.

 

Verifications

Diagram, timelineDescription automatically generated

 

Figure 10.         

End to End Tenant VPN Route Propagation

This section will provide the necessary operation commands to verify the controlplane and dataplane for inter-site communications.

It will focus on how site 2-1 can reach the destination subnet 100.21.6.0/24 located in site 2-1-2-1.

 

XR: EVPN Route Received from BL

 

Although the route that is received from the BL is an EVPN route, the route reception should be checked using VPN unicast commands. This is related to how the stitching between EVPN and VPN unicast works. The route is received as EVPN but is automatically translated to a VPN unicast route.

The route 100.21.6.0 is correctly received by PE6 from the BL as displayed below. The route is received with an MPLS EVPN label of 948594. One of the important points to note in the command is that the displayed received route-target is not actually the one which is received. Again, the EVPN to VPN unicast stitching performs some action on the RTs, and the matching stitching RT (1:2110006 received from the BL) is replaced by the non-stitching RT (1:2110000) during the translation to VPN unicast.

The route is installed in the routing and forwarding table.

 

RP/0/RP0/CPU0:PE06#show bgp vpnv4 unicast vrf TENANT_BASIC 100.21.6.0

Fri Sep  4 08:44:42.653 UTC

BGP routing table entry for 100.21.6.0/24, Route Distinguisher: 1:2110006

Versions:

  Process           bRIB/RIB  SendTblVer

  Speaker             164147      164147

    Local Label: 32074

Last Modified: Sep  3 15:59:43.382 for 16:44:59

Paths: (3 available, best #1)

  Advertised to update-groups (with more than one peer):

    0.5

  Path #1: Received by speaker 0

  Advertised to update-groups (with more than one peer):

    0.5

  987654321

    20.204.105.1 from 20.204.105.1 (30.204.105.1)

      Received Label 948594

      Origin incomplete, metric 0, localpref 100, valid, external, best, group-best, import-candidate, imported, reoriginated

      Received Path ID 0, Local Path ID 1, version 164147

      Extended community: RT:1:2110000

      EVPN Gateway Address : 0.0.0.0

      Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 105:3014666

RP/0/RP0/CPU0:PE06#show route vrf TENANT_BASIC 100.21.6.0

Fri Sep  4 12:08:16.932 UTC

 

Routing entry for 100.21.6.0/24

  Known via "bgp 1", distance 20, metric 0

  Tag 987654321, type external

  Installed Sep  3 15:59:42.931 for 20:08:34

  Routing Descriptor Blocks

    20.204.105.1, from 20.204.105.1, BGP external

      Nexthop in Vrf: "default", Table: "default", IPv4 Unicast, Table Id: 0xe0000000

      Route metric is 0

  No advertising protos.

 

RP/0/RP0/CPU0:PE06#show cef vrf TENANT_BASIC 100.21.6.0

Fri Sep  4 12:07:57.268 UTC

100.21.6.0/24, version 232, internal 0x1000001 0x30 (ptr 0xa9a60978) [1], 0x0 (0x0), 0x208 (0x8aeb5318)

 Updated Sep  3 15:59:42.934

 Prefix Len 24, traffic index 0, precedence n/a, priority 3

   via 20.204.105.1/32, 5 dependencies, recursive, bgp-ext [flags 0x6020]

    path-idx 0 NHID 0x0 [0xa9d1f830 0x0]

    recursion-via-/32

    next hop VRF - 'default', table - 0xe0000000

    next hop 20.204.105.1/32 via 16047/0/21

     next hop 120.1.62.1/32 Te0/0/0/12   labels imposed {ImplNull ImplNull 948594}

 

XR: VPN Unicast Route Received from Remote PE

 

PE6 has advertised the route 100.21.6.0 using VPNv4 to the SP core and this route is received and imported by PE2 in the VRF TENANT_BASIC.

Similarly, when PE2 imports the VPN unicast route using the non-stitching RT 1:2110000, it automatically translates it to the stitching RT value 1:2110005.

 

RP/0/RSP0/CPU0:PE02#show bgp vpnv4 unicast vrf TENANT_BASIC 100.21.6.0

Fri Sep  4 08:37:21.850 UTC

BGP routing table entry for 100.21.6.0/24, Route Distinguisher: 1:2110002

Versions:

  Process           bRIB/RIB  SendTblVer

  Speaker             287129      287129

    Local Label: 32068

Last Modified: Sep  3 16:01:33.144 for 16:35:49

Paths: (2 available, best #1)

  Advertised to update-groups (with more than one peer):

    0.2

  Path #1: Received by speaker 0

  Advertised to update-groups (with more than one peer):

    0.2

  987654321

    201.201.201.6 (metric 20150) from 202.202.202.102 (201.201.201.6)

      Received Label 32074

      Origin incomplete, metric 0, localpref 100, valid, internal, best, group-best, import-candidate, imported, reoriginated with stitching-rt

      Received Path ID 1, Local Path ID 1, version 287128

      Extended community: RT:1:2110005

      Originator: 201.201.201.6, Cluster list: 0.0.0.2

      

      Source AFI: VPNv4 Unicast, Source VRF: default, Source Route Distinguisher: 1:2110006

 

RP/0/RSP0/CPU0:PE02#show route  vrf TENANT_BASIC 100.21.6.0

Fri Sep  4 12:12:20.875 UTC

 

Routing entry for 100.21.6.0/24

  Known via "bgp 1", distance 200, metric 0

  Tag 987654321, type internal

  Installed Sep  3 16:01:32.983 for 20:10:48

  Routing Descriptor Blocks

    201.201.201.6, from 202.202.202.102

      Nexthop in Vrf: "default", Table: "default", IPv4 Unicast, Table Id: 0xe0000000

      Route metric is 0

  No advertising protos.

 

In the VRF forwarding table, the route uses the BGP VPN received label (32074), as well as the MPLS label associated with the Node-SID to reach PE6 (16006).

 

RP/0/RSP0/CPU0:PE02#show cef vrf TENANT_BASIC 100.21.6.0

Fri Sep  4 12:12:14.667 UTC

100.21.6.0/24, version 202, internal 0x1000001 0x0 (ptr 0x78f29bbc) [1], 0x0 (0x0), 0x208 (0x8a652e28)

 Updated Sep  3 16:01:32.984

 Prefix Len 24, traffic index 0, precedence n/a, priority 3

   via 201.201.201.6/32, 6 dependencies, recursive [flags 0x6000]

    path-idx 0 NHID 0x0 [0x89a98a78 0x0]

    recursion-via-/32

    next hop VRF - 'default', table - 0xe0000000

    next hop 201.201.201.6/32 via 16006/0/21

     next hop 200.200.200.2/32 BE1          labels imposed {16006 32074}

 

XR: VPN Unicast Route Advertised to BL

 

When checking the routes advertised to the border leaf, VPN unicast commands have to be used. Again this is because the translation of the route to EVPN happens late in the BGP update processing.

RP/0/RSP0/CPU0:PE02#show bgp vpnv4 unicast advertised neighbor 20.204.103.1 | be 1:2110002

Fri Sep  4 12:18:59.518 UTC

Route Distinguisher: 1:2110002

100.21.6.0/24 is advertised to 20.204.103.1

  Path info:

    neighbor: 202.202.202.102 neighbor router id: 201.201.201.6

    valid  internal  best  import-candidate  imported  reoriginated  with stitching-rt

Received Path ID 1, Local Path ID 1, version 287128

  Attributes after inbound policy was applied:

    next hop: 201.201.201.6

    MET ORG AS LOCAL EXTCOMM

    origin: incomplete  neighbor as: 987654321  metric: 0  local pref: 100

    aspath: 987654321

    extended community: RT:1:2110005

    originator: 201.201.201.6    cluster list: 0.0.0.2

 

  Attributes after outbound policy was applied:

    next hop: 201.201.201.2

    ORG AS LOCAL EXTCOMM

    origin: incomplete  neighbor as: 987654321  local pref: 100

    aspath: 1 987654321

    extended community: RT:1:2110005

 

RP/0/RSP0/CPU0:PE02#show bgp l2vpn evpn neighbors 20.204.103.1 advertised-routes

Fri Sep  4 12:23:51.024 UTC

RP/0/RSP0/CPU0:PE02#

 

It is expected that EVPN commands do not display anything as routes are still considered as VPN unicast route.

 

ACI: EVPN Route Received from PE

 

The subnet 100.21.6.0/24, coming from the remote leaf site 2-1-2-1, is correctly received by the border leaf located in site 2-1. Although there is a loop in the ASPATH, the route is accepted because of the allowas-in parameter.

 

ifav204-leaf3# show bgp l2vpn evpn 100.21.6.0 vrf overlay-1

Route Distinguisher: 1:2110002

BGP routing table entry for [5]:[0]:[0]:[24]:[100.21.6.0]:[0.0.0.0]/224, version 507 dest ptr 0xacfd07fa

Paths: (1 available, best #1)

Flags: (0x000002 00000000) on xmit-list, is not in rib/evpn, is not in HW, is locked

Multipath: eBGP iBGP

 

  Advertised path-id 1

  Path type: external 0x40000028 0x0 ref 2 adv path ref 1, path is valid, is best path

             Imported to 2 destination(s)

  AS-Path: 1 987654321 , path sourced external to AS

    201.201.201.2 (metric 0) from 201.201.201.2 (201.201.201.2)

      Origin incomplete, MED not set, localpref 100, weight 0 tag 4294966257, propagate 0

      Received label 32068

      Extcommunity:

          RT:1:2110005

 

  Path-id 1 not advertised to any peer

 

Route Distinguisher: 103:2392067    (L3VNI 2392067)

BGP routing table entry for [5]:[0]:[0]:[24]:[100.21.6.0]:[0.0.0.0]/224, version 2182 dest ptr 0xacfd21e6

Paths: (1 available, best #1)

Flags: (0x000002 00000000) on xmit-list, is not in rib/evpn, is not in HW

Multipath: eBGP iBGP

 

  Advertised path-id 1

  Path type: external 0xc0000028 0x0 ref 0 adv path ref 1, path is valid, is best path

             Imported from 1:2110002:[5]:[0]:[0]:[24]:[100.21.6.0]:[0.0.0.0]/120

  AS-Path: 1 987654321 , path sourced external to AS

    201.201.201.2 (metric 0) from 201.201.201.2 (201.201.201.2)

      Origin incomplete, MED not set, localpref 100, weight 0 tag 4294966257, propagate 0

      Received label 32068

      Extcommunity:

          RT:1:2110005

          

  Path-id 1 not advertised to any peer

 

ifav204-leaf3#

 

The received route is correctly installed in the tenant routing table.

The route uses a recursive nexthop (PE2 loopback) and an outgoing MPLS label value of 32068 which corresponds to the label received from BGP EVPN.

ifav204-leaf3# show ip route 100.21.6.0/24 vrf TENANT-BASIC:TENANT_BASIC_21

IP Route Table for VRF "TENANT-BASIC:TENANT_BASIC_21"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

 

100.21.6.0/24, ubest/mbest: 1/0

    *via 201.201.201.2%overlay-1, [20/0], 10:26:14, bgp-987654321, external, tag 1, Mpls Label 32068

         recursive next hop: 201.201.201.2/32%overlay-1

ifav204-leaf3#

 

ACI: Tenant Connectivity Check

 

In order to check the connectivity between the sites, a ping can be issued from a leaf in the fabric 2-1 where the Bridge Domain is deployed towards the remote leaf site. In our case, LEAF1 connects the tenant in the fabric.

 

ifav204-leaf1# iping 100.21.6.254 -V TENANT-BASIC:TENANT_BASIC_21

PING 100.21.6.254 (100.21.6.254): 56 data bytes

64 bytes from 100.21.6.254: icmp_seq=0 ttl=61 time=0.46 ms

64 bytes from 100.21.6.254: icmp_seq=1 ttl=61 time=0.281 ms

64 bytes from 100.21.6.254: icmp_seq=2 ttl=61 time=0.273 ms

64 bytes from 100.21.6.254: icmp_seq=3 ttl=61 time=0.296 ms

^C

--- 100.21.6.254 ping statistics ---

4 packets transmitted, 4 packets received, +1 duplicates, 0.00% packet loss

round-trip min/avg/max = 0.273/0.405/0.72 ms

ifav204-leaf1#

 


 

Basic SR-Handoff with Multihoming

Design

 

Related image, diagram or screenshot

Figure 11.         

Basic Communication between ACI Fabric and Remote Leaf with Multihoming

The use case is similar to the previous one where two workloads located respectively in the Cisco ACI fabric and a remote leaf site are required to communicate. However, from an infrastructure perspective, the network is fully redundant, and a pair of remote leaf switches is connected to a pair of DC-PEs on each ACI site.

To get the best convergence, it is recommended that you connect each border leaf to two DC-PEs. In the provided configurations, BFD will be implemented both on the BGP-LU session and the EVPN session to speed up the failure detection between the DC-PE and the ACI border leaf.

When multihoming is implemented, some routes learned from a site may be reinjected into the same site. Some loop prevention mechanisms will be configured to prevent such loops from occuring. For instance, when PE2 learns a route from the Cisco ACI fabric, it propagates the route to the SP CORE and PE1 will receive it and may propagate the route back to the Cisco ACI fabric.

 

Configuration

PE1 DC-PE configuration

 

The configuration below is very similar to the previous one. The differences are as follows:

      BFD is activated using the “bfd fast-detect” keyword on the LU session as well as on the EVPN session. The globally configured timers (50ms x 3) are used. However, the multihop EVPN BFD session will not be using these values as ACI Border Leaf minimum values are 250ms x 3.

      Loop prevention policies are configured on the EVPN BGP sessions using the route-policies MARK-ACI-ROUTES (inbound) and DROP-ACI-ROUTES (outbound). Loops may happen in both directions. The ACI route may be learned by SP core and injected back to the original ACI site, and an SP route may be learned by an ACI site and injected back to the SP core. As multiple ACI sites may use the same AS number (in our case the remote leaf is using the same AS number as the Fabric), then ASPATH loop check has been disabled and cannot be used any longer to prevent loops. Site of Origin is used in both directions (one SOO value per direction) to prevent loops. MARK-ACI-ROUTES route-policy mark the routes with an SOO associated to the pair of border leaf switches and drops routes with an SOO value associated to the pair of DC-PEs connected to the ACI site. DROP-ACI-ROUTES drops the routes with an SOO associated to the pair of border leaf switches and marks the routes with an SOO associated to the pair of DC-PEs connected to the ACI site. To allow reusability of the policies, the policies use an ACI site number as a variable.

      The stitching VRF is configured to perform iBGP and eBGP multipath to get the benefit of loadbalancing (maximum-paths ebgp 16 / maximum-paths ibgp 16).

 

 

DiagramDescription automatically generated 

Figure 12.         

Loop Prevention using Site of Origin in Multihoming Scenario

 

 

vrf TENANT_BASIC

 address-family ipv4 unicast

  import route-target

   1:2110000

   1:2110005 stitching

  !

  export route-target

   1:2110000

   1:2110005 stitching

  !

 !

 address-family ipv6 unicast

  import route-target

   1:2110000

   1:2110005 stitching

  !

  export route-target

   1:2110000

   1:2110005 stitching

  !

 !

!

interface TenGigE0/0/0/9

 description connected to ifav204-leaf3:1/18 used for SR/MPLS

 ipv4 address 120.1.51.2 255.255.255.0

 load-interval 30

!

interface TenGigE0/0/0/11

 description connected to ifav204-leaf4:1/14 used for SR/MPLS

 ipv4 address 120.1.52.2 255.255.255.0

 load-interval 30

!

router static

 address-family ipv4 unicast

  120.1.51.1/32 TenGigE0/0/0/9

  120.1.52.1/32 TenGigE0/0/0/11

 !

!

router bgp 1

 bfd minimum-interval 50

 bfd multiplier 3

 bgp router-id 201.201.201.1

 mpls activate

  interface TenGigE0/0/0/9

  interface TenGigE0/0/0/11

 !

 address-family ipv4 unicast

  redistribute connected route-policy CONNECTED-TO-BGP-LU(1)

  allocate-label all

 !

 address-family vpnv4 unicast

 !

 address-family vpnv6 unicast

 !

 address-family ipv4 rt-filter

 !

 address-family l2vpn evpn

 !

 neighbor-group ACI-site-LU

  remote-as 987654321

  bfd fast-detect

  address-family ipv4 labeled-unicast

   route-policy SET-CT-NO-ADVERTISE in

   maximum-prefix 10 80 warning-only

   route-policy ADVERTISE-LO0-ONLY out

  !

 !

 neighbor-group ACI-site-EVPN

  remote-as 987654321

  bfd fast-detect

  ebgp-multihop 255

  update-source Loopback0

  address-family l2vpn evpn

   import stitching-rt re-originate

   allowas-in 5

   advertise vpnv4 unicast re-originated stitching-rt

   advertise vpnv6 unicast re-originated stitching-rt

  !

 !

 neighbor-group RR-VPNunicast

  remote-as 1

  update-source Loopback0

  address-family vpnv4 unicast

   import re-originate stitching-rt

   advertise vpnv4 unicast re-originated

  !

  address-family vpnv6 unicast

   import re-originate stitching-rt

   advertise vpnv6 unicast re-originated

  !

  address-family ipv4 rt-filter

  !

 !

neighbor 120.1.51.1

  use neighbor-group ACI-site-LU

 !

 neighbor 120.1.52.1

  use neighbor-group ACI-site-LU

 !

 neighbor 20.204.103.1

  use neighbor-group ACI-site-EVPN

  address-family l2vpn evpn

   route-policy MARK-ACI-ROUTES(5) in

   route-policy DROP-ACI-ROUTES(5) out

  !

 !

 neighbor 20.204.104.1

  use neighbor-group ACI-site-EVPN

  address-family l2vpn evpn

   route-policy MARK-ACI-ROUTES(5) in

   route-policy DROP-ACI-ROUTES(5) out

  !

 !

 neighbor 202.202.202.100

  use neighbor-group RR-VPNunicast

 !

 neighbor 202.202.202.101

  use neighbor-group RR-VPNunicast

 !

 vrf TENANT_BASIC

  rd 1:2110005

  address-family ipv4 unicast

   maximum-paths ebgp 16

   maximum-paths ibgp 16

  !

  address-family ipv6 unicast

   maximum-paths ebgp 16

   maximum-paths ibgp 16

  !

!

route-policy MARK-ACI-ROUTES($site)

  if extcommunity soo matches-any (987654321:$site) then

    drop

  else

    set extcommunity soo (1:$site)

  endif

end-policy

!

route-policy DROP-ACI-ROUTES($site)

  if extcommunity soo matches-any (1:$site) then

    drop

  else

    set extcommunity soo (987654321:$site)

  endif

end-policy

!

prefix-set PFXSET-OWN-LO0

  201.201.201.1/32

end-set

!

route-policy ADVERTISE-LO0-ONLY

  if destination in PFXSET-OWN-LO0 then

    pass

  else

    drop

  endif

end-policy

!

community-set COMSET-NO-ADVERTISE

  no-advertise

end-set

!

route-policy SET-CT-NO-ADVERTISE

  set community COMSET-NO-ADVERTISE

end-policy

!

route-policy CONNECTED-TO-BGP-LU($node_sid)

  if destination in PFXSET-OWN-LO0 then

    set label-index $node_sid

    pass

  endif

end-policy

 

ACI Configuration

 

INFRA Tenant Configuration

 

<?xml version="1.0" encoding="utf-8"?>

<imdata totalCount="1">

     <fvTenant

               dn="uni/tn-infra"

               name="infra"

               >

               <!-- Tag_1 Default Label Range config in Infra Tenant -->

               <mplsLabelPol

                             maxDynamicLabel="525286"

                             maxStaticLabel="0"

                             minDynamicLabel="16"

                             minStaticLabel="0"

                             name="default"

                             >

                              <mplsSrgbLabelPol

                                                localId="1"

                                                maxSrgbLabel="32000"

                                                minSrgbLabel="16000"

                                                />

               </mplsLabelPol>

               <!-- Tag_2 Default interface config in Infra Tenant -->

               <mplsIfPol

                          name="default"

                          />

               <!-- Tag_3 MPLS Infra L3out towards DC PE Location 2-1 -->

               <l3extOut

                         enforceRtctrl="export"

                         mplsEnabled="yes"

                         name="2-1"

                         targetDscp="unspecified"

                         >

                              <mplsExtP

                                        >

                                             <mplsRsLabelPol

                                                             tDn="uni/tn-infra/mplslabelpol-default"

                                                             />

                              </mplsExtP>

                              <l3extRsL3DomAtt

                                               tDn="uni/l3dom-L3Dom"

                                               />

                              <l3extRsEctx

                                           tnFvCtxName="overlay-1"

                                           />

                              <l3extProvLbl

                                            name="2-1"

                                            tag="yellow-green"

                                            />

                              <l3extLNodeP

                                           name="2-1_nodeProfile"

                                           tag="yellow-green"

                                           targetDscp="unspecified"

                                           >

                                             <l3extRsNodeL3OutAtt

                                                                  rtrId="30.204.103.1"

                                                                  rtrIdLoopBack="no"

                                                                  tDn="topology/pod-1/node-103"

                                                                  >

                                                            <l3extLoopBackIfP addr="20.204.103.1"

                                                                              >

                                                                          <mplsNodeSidP

                                                                                        loopbackAddr="20.204.103.1"

                                                                                        sidoffset="45"

                                                                                        />

                                                            </l3extLoopBackIfP>

                                             </l3extRsNodeL3OutAtt>

                                             <l3extRsNodeL3OutAtt

                                                                  rtrId="30.204.104.1"

                                                                  rtrIdLoopBack="no"

                                                                  tDn="topology/pod-1/node-104"

                                                                  >

                                                            <l3extLoopBackIfP addr="20.204.104.1"

                                                                              >

                                                                          <mplsNodeSidP

                                                                                        loopbackAddr="20.204.104.1"

                                                                                        sidoffset="46"

                                                                                        />

                                                            </l3extLoopBackIfP>

                                             </l3extRsNodeL3OutAtt>

                                             <l3extRsLNodePMplsCustQosPol

                                                                          tDn="uni/tn-infra/qosmplscustom-"

                                                                          />

                                             <l3extLIfP

                                                        name="2-1_interfaceProfile"

                                                        prio="unspecified"

                                                        tag="yellow-green"

                                                        >

                                                            <l3extRsPathL3OutAtt addr="120.1.51.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-103/pathep-[eth1/18]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.51.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.53.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-103/pathep-[eth1/1]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.53.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.52.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-104/pathep-[eth1/14]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.52.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                           </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.54.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-104/pathep-[eth1/2]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.54.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <mplsIfP

                                                                     >

                                                                          <mplsRsIfPol

                                                                                       tnMplsIfPolName="default"

                                                                                       />

                                                            </mplsIfP>

                                                            <bfdIfP

                                                                    keyId="1"

                                                                    type="none"

                                                                    >

                                                                          <bfdRsIfPol

                                                                                       tnBfdIfPolName="BFD_Pol"

                                                                                      />

                                                            </bfdIfP>

                                             </l3extLIfP>

                                             <bgpInfraPeerP addr="201.201.201.2"

                                                            addrTCtrl="af-ucast"

                                                            adminSt="enabled"

                                                            allowedSelfAsCnt="3"

                                                            ctrl="allow-self-as,dis-peer-as-check,send-com,send-ext-com"

                                                            dataPlaneAddr="0.0.0.0"

                                                            peerCtrl="bfd"

                                                            peerT="SR/MPLS"

                                                            remoteIntersiteRR="no"

                                                            srcIfT="l3out-loopback"

                                                            trustCtrl="untrusted"

                                                            ttl="16"

                                                            weight="0">

                                                            <bgpRsPeerPfxPol

                                                                             />

                                                            <bgpAsP

                                                                    asn="1"

                                                                    />

                                             </bgpInfraPeerP>

                                             <bgpInfraPeerP addr="201.201.201.1"

                                                            addrTCtrl="af-ucast"

                                                            adminSt="enabled"

                                                            allowedSelfAsCnt="3"

                                                            ctrl="allow-self-as,dis-peer-as-check,send-com,send-ext-com"

                                                            dataPlaneAddr="0.0.0.0"

                                                            peerCtrl="bfd"

                                                            peerT="SR/MPLS"

                                                            remoteIntersiteRR="no"

                                                            srcIfT="l3out-loopback"

                                                            trustCtrl="untrusted"

                                                            ttl="16"

                                                            weight="0">

                                                            <bgpRsPeerPfxPol

                                                                             />

                                                            <bgpAsP

                                                                    asn="1"

                                                                    />

                                             </bgpInfraPeerP>

                              </l3extLNodeP>

                              <l3extInstP

                                          floodOnEncap="disabled"

                                          matchT="AtleastOne"

                                          name="2-1_mplsInstP"

                                          prefGrMemb="exclude"

                                          prio="unspecified"

                                          targetDscp="unspecified"

                                          >

                                             <fvRsCustQosPol

                                                             />

                              </l3extInstP>

                              <bgpExtP

                                       />

               </l3extOut>

               <!-- Tag_4 MPLS Infra L3out towards DC PE Location 2-1-2-1 -->

               <l3extOut

                         enforceRtctrl="export"

                         mplsEnabled="yes"

                         name="2-1-2-1"

                         targetDscp="unspecified"

                         >

                              <mplsExtP

                                        >

                                             <mplsRsLabelPol

                                                             tDn="uni/tn-infra/mplslabelpol-default"

                                                             />

                              </mplsExtP>

                              <l3extRsL3DomAtt

                                               tDn="uni/l3dom-L3Dom"

                                               />

                              <l3extRsEctx

                                           tnFvCtxName="overlay-1"

                                           />

                              <l3extProvLbl

                                            name="2-1-2-1"

                                            tag="yellow-green"

                                            />

                              <l3extLNodeP

                                           name="2-1-2-1_nodeProfile"

                                           tag="yellow-green"

                                           targetDscp="unspecified"

                                           >

                                             <l3extRsNodeL3OutAtt

                                                                  rtrId="30.204.105.1"

                                                                  rtrIdLoopBack="no"

                                                                  tDn="topology/pod-1/node-105"

                                                                  >

                                                            <l3extLoopBackIfP addr="20.204.105.1"

                                                                              >

                                                                          <mplsNodeSidP

                                                                                        loopbackAddr="20.204.105.1"

                                                                                        sidoffset="47"

                                                                                        />

                                                            </l3extLoopBackIfP>

                                             </l3extRsNodeL3OutAtt>

                                             <l3extRsNodeL3OutAtt

                                                                  rtrId="30.204.106.1"

                                                                  rtrIdLoopBack="no"

                                                                  tDn="topology/pod-1/node-106"

                                                                  >

                                                            <l3extLoopBackIfP addr="20.204.106.1"

                                                                              >

                                                                          <mplsNodeSidP

                                                                                        loopbackAddr="20.204.106.1"

                                                                                        sidoffset="48"

                                                                                        />

                                                            </l3extLoopBackIfP>

                                             </l3extRsNodeL3OutAtt>

                                             <l3extRsLNodePMplsCustQosPol

                                                                          tDn="uni/tn-infra/qosmplscustom-"

                                                                          />

                                             <l3extLIfP

                                                        name="2-1-2-1_interfaceProfile"

                                                        prio="unspecified"

                                                        tag="yellow-green"

                                                        >

                                                            <l3extRsPathL3OutAtt addr="120.1.63.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-106/pathep-[eth1/2]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.63.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.62.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-105/pathep-[eth1/1]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.62.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.64.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-106/pathep-[eth1/3]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.64.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.61.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-105/pathep-[eth1/3]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.61.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                     adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <mplsIfP

                                                                     >

                                                                          <mplsRsIfPol

                                                                                       tnMplsIfPolName="default"

                                                                                       />

                                                            </mplsIfP>

                                                            <bfdIfP

                                                                    keyId="1"

                                                                    type="none"

                                                                    >

                                                                          <bfdRsIfPol

                                                                                      tnBfdIfPolName="BFD_Pol"

                                                                                      />

                                                            </bfdIfP>

                                             </l3extLIfP>

                                             <bgpInfraPeerP addr="201.201.201.7"

                                                            addrTCtrl="af-ucast"

                                                            adminSt="enabled"

                                                            allowedSelfAsCnt="3"

                                                            ctrl="allow-self-as,dis-peer-as-check,send-com,send-ext-com"

                                                            dataPlaneAddr="0.0.0.0"

                                                            peerCtrl="bfd"

                                                            peerT="SR/MPLS"

                                                            remoteIntersiteRR="no"

                                                            srcIfT="l3out-loopback"

                                                            trustCtrl="untrusted"

                                                            ttl="16"

                                                            weight="0">

                                                            <bgpRsPeerPfxPol

                                                                             />

                                                            <bgpAsP

                                                                    asn="1"

                                                                    />

                                             </bgpInfraPeerP>

                                             <bgpInfraPeerP addr="201.201.201.6"

                                                            addrTCtrl="af-ucast"

                                                            adminSt="enabled"

                                                            allowedSelfAsCnt="3"

                                                            ctrl="allow-self-as,dis-peer-as-check,send-com,send-ext-com"

                                                            dataPlaneAddr="0.0.0.0"

                                                            peerCtrl="bfd"

                                                            peerT="SR/MPLS"

                                                            remoteIntersiteRR="no"

                                                            srcIfT="l3out-loopback"

                                                            trustCtrl="untrusted"

                                                            ttl="16"

                                                            weight="0">

                                                            <bgpRsPeerPfxPol

                                                                             />

                                                            <bgpAsP

                                                                    asn="1"

                                                                    />

                                             </bgpInfraPeerP>

                              </l3extLNodeP>

                              <l3extInstP

                                          floodOnEncap="disabled"

                                          matchT="AtleastOne"

                                          name="2-1-2-1_mplsInstP"

                                          prefGrMemb="exclude"

                                          prio="unspecified"

                                          targetDscp="unspecified"

                                          >

                                             <fvRsCustQosPol

                                                             />

                              </l3extInstP>

                              <bgpExtP

                                       />

               </l3extOut>

     </fvTenant>

</imdata>

User Tenant Configuration

 

There is no change in the user tenant configuration for multihoming as we add additional links to INFRA tenant in same location.

 

<?xml version="1.0" encoding="utf-8"?>

<imdata totalCount="1">

     <!-- Specify Name of Tenant as Required -->

     <fvTenant descr="Tenant with Basic SR MPLS Handoff between 2-1 and 2-1-2-1"

               dn="uni/tn-TENANT-BASIC"

               name="TENANT-BASIC"

               >

               <!-- Tag_1 User L3out Route Control Config -->

               <rtctrlSubjP

                            name="all"

                            >

                              <rtctrlMatchRtDest aggregate="yes"

                                                 fromPfxLen="0"

                                                 ip="100.0.0.0/8"

                                                 toPfxLen="0"

                                                 />

                              <rtctrlMatchRtDest aggregate="yes"

                                                 fromPfxLen="0"

                                                 ip="::/0"

                                                 toPfxLen="0"

                                                 />

               </rtctrlSubjP>

               <rtctrlProfile

                              name="Export-Pol"

                              type="combinable"

                              >

                              <rtctrlCtxP action="permit"

                                          name="Export-Pol"

                                          order="1"

                                          >

                                             <rtctrlRsCtxPToSubjP

                                                                  tnRtctrlSubjPName="all"

                                                                  />

                              </rtctrlCtxP>

               </rtctrlProfile>

               <rtctrlProfile

                              name="Import-Pol"

                              type="combinable"

                              >

                              <rtctrlCtxP action="permit"

                                          name="Import-Pol"

                                          order="1"

                                          >

                                             <rtctrlRsCtxPToSubjP

                                                                  tnRtctrlSubjPName="all"

                                                                  />

                              </rtctrlCtxP>

               </rtctrlProfile>

               <!-- Tag_2 User L3out Config towards DC PE Location 2-1 -->

               <l3extOut

                         enforceRtctrl="export"

                         mplsEnabled="yes"

                         name="LOCATION-2-1-2-1-MPLS-TENANT-L3OUT-2106"

                         targetDscp="unspecified"

                         >

                              <l3extRsEctx tnFvCtxName="TENANT_BASIC_2121"

                                           />

                              <rtctrlProfile

                                             name="Export-Pol"

                                             type="combinable"

                                             >

                                             <rtctrlCtxP action="permit"

                                                         name="Export-Pol"

                                                         order="1"

                                                         >

                                                            <rtctrlRsCtxPToSubjP

                                                                                 tnRtctrlSubjPName="all"

                                                                                 />

                                             </rtctrlCtxP>

                              </rtctrlProfile>

                              <rtctrlProfile

                                             name="Import-Pol"

                                             type="combinable"

                                             >

                                             <rtctrlCtxP action="permit"

                                                         name="Import-Pol"

                                                         order="1"

                                                         >

                                                            <rtctrlRsCtxPToSubjP

                                                                                 tnRtctrlSubjPName="all"

                                                                                 />

                                             </rtctrlCtxP>

                              </rtctrlProfile>

                              <l3extConsLbl

                                            name="2-1-2-1"

                                            owner="infra"

                                            tag="yellow-green"

                                            >

                                             <l3extRsLblToProfile direction="import"

                                                                  tDn="uni/tn-TENANT-BASIC/prof-Import-Pol"

                                                                  />

                                             <l3extRsLblToProfile

                                                                  direction="export"

                                                                  tDn="uni/tn-TENANT-BASIC/prof-Export-Pol"

                                                                  />

                                             <l3extRsLblToInstP tDn="uni/tn-TENANT-BASIC/out-LOCATION-2-1-2-1-MPLS-TENANT-L3OUT-2106/instP-LOCATION-2-1-2-1-MPLS-TENANT-L3OUT-2106-InstP"

                                                                />

                              </l3extConsLbl>

                              <l3extInstP

                                          floodOnEncap="disabled"

                                          matchT="AtleastOne"

                                          name="LOCATION-2-1-2-1-MPLS-TENANT-L3OUT-2106-InstP"

                                          prefGrMemb="exclude"

                                          prio="unspecified"

                                          targetDscp="unspecified"

                                          >

                                             <fvRsProv

                                                       intent="install"

                                                       matchT="AtleastOne"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <l3extSubnet

                                                          ip="::/0"

                                                          name="ipv6All"

                                                          scope="import-security"

                                                          />

                                             <l3extSubnet

                                                          ip="0.0.0.0/0"

                                                          name="ipv4All"

                                                          scope="import-security"

                                                          />

                                             <fvRsCustQosPol

                                                             />

                                             <fvRsCons

                                                       intent="install"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                              </l3extInstP>

                              <bgpExtP

                                       />

               </l3extOut>

               <!-- Tag_3 User L3out Config towards DC PE Location 2-1-2-1 -->

               <l3extOut

                         enforceRtctrl="export"

                         mplsEnabled="yes"

                         name="LOCATION-2-1-MPLS-TENANT-L3OUT-2105"

                         targetDscp="unspecified"

                         >

                              <l3extRsEctx tnFvCtxName="TENANT_BASIC_21"

                                           />

                              <rtctrlProfile

                                             name="Import-pol"

                                             type="combinable"

                                             >

                                             <rtctrlCtxP action="permit"

                                                         name="Import-pol"

                                                         order="1"

                                                         >

                                                            <rtctrlRsCtxPToSubjP

                                                                                 tnRtctrlSubjPName="all"

                                                                                 />

                                             </rtctrlCtxP>

                              </rtctrlProfile>

                              <rtctrlProfile

                                             name="Export-pol"

                                             type="combinable"

                                             >

                                             <rtctrlCtxP action="permit"

                                                         name="Export-pol"

                                                         order="1"

                                                         >

                                                            <rtctrlRsCtxPToSubjP

                                                                                 tnRtctrlSubjPName="all"

                                                                                 />

                                             </rtctrlCtxP>

                              </rtctrlProfile>

                              <l3extConsLbl

                                            name="2-1"

                                            owner="infra"

                                            tag="yellow-green"

                                            >

                                             <l3extRsLblToProfile direction="import"

                                                                  tDn="uni/tn-TENANT-BASIC/prof-Import-Pol"

                                                                  />

                                             <l3extRsLblToProfile direction="export"

                                                                  tDn="uni/tn-TENANT-BASIC/prof-Export-Pol"

                                                                  />

                                             <l3extRsLblToInstP tDn="uni/tn-TENANT-BASIC/out-LOCATION-2-1-MPLS-TENANT-L3OUT-2105/instP-LOCATION-2-1-MPLS-TENANT-L3OUT-2105-InstP"

                                                                />

                              </l3extConsLbl>

                              <l3extInstP

                                          floodOnEncap="disabled"

                                          matchT="AtleastOne"

                                          name="LOCATION-2-1-MPLS-TENANT-L3OUT-2105-InstP"

                                          prefGrMemb="exclude"

                                          prio="unspecified"

                                          targetDscp="unspecified"

                                          >

                                             <fvRsProv

                                                       intent="install"

                                                       matchT="AtleastOne"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <l3extSubnet

                                                          ip="::/0"

                                                          name="ipv6All"

                                                          scope="import-security"

                                                          />

                                             <l3extSubnet

                                                          ip="0.0.0.0/0"

                                                          name="ipv4All"

                                                          scope="import-security"

                                                          />

                                             <fvRsCustQosPol

                                                             />

                                             <fvRsCons

                                                       intent="install"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                              </l3extInstP>

                              <bgpExtP

                                       />

               </l3extOut>

               <!-- Tag_4 User VRF Config towards DC PE Location 2-1-2-1 -->

               <fvCtx bdEnforcedEnable="no"

                      ipDataPlaneLearning="enabled"

                      knwMcastAct="permit"

                      name="TENANT_BASIC_2121"

                      pcEnfDir="ingress"

                      pcEnfPref="enforced"

                      >

                              <fvRsVrfValidationPol

                                                    tnL3

                                                    />

                              <vzAny

                                     matchT="AtleastOne"

                                     prefGrMemb="disabled"

                                     />

                              <fvRsOspfCtxPol

                                              />

                              <fvRsCtxToEpRet

                                              />

                              <fvRsCtxToExtRouteTagPol

                                                       tnL3

                                                       />

                              <fvRsBgpCtxPol

                                             />

                              <bgpRtTargetP af="ipv4-ucast"

                                            >

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110006"

                                                          type="import"

                                                          />

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110006"

                                                          type="export"

                                                          />

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110007"

                                                          type="import"

                                                          />

                              </bgpRtTargetP>

                              <bgpRtTargetP af="ipv6-ucast"

                                            >

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110006"

                                                          type="import"

                                                          />

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110006"

                                                          type="export"

                                                          />

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110007"

                                                          type="import"

                                                          />

                              </bgpRtTargetP>

               </fvCtx>

               <!-- Tag_5 User VRF Config towards DC PE Location 2-1 -->

               <fvCtx bdEnforcedEnable="no"

                      ipDataPlaneLearning="enabled"

                      knwMcastAct="permit"

                      name="TENANT_BASIC_21"

                      pcEnfDir="ingress"

                      pcEnfPref="enforced"

                      >

                              <fvRsVrfValidationPol

                                                    tnL3

                                                    />

                              <vzAny

                                     matchT="AtleastOne"

                                     prefGrMemb="disabled"

                                     />

                              <fvRsOspfCtxPol

                                              />

                              <fvRsCtxToEpRet

                                              />

                              <fvRsCtxToExtRouteTagPol

                                                       tnL3

                                                       />

                              <fvRsBgpCtxPol

                                             />

                              <bgpRtTargetP af="ipv4-ucast"

                                            >

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110005"

                                                          type="import"

                                                          />

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110005"

                                                          type="export"

                                                          />

                              </bgpRtTargetP>

                              <bgpRtTargetP af="ipv6-ucast"

                                            >

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110005"

                                                          type="export"

                                                          />

                                             <bgpRtTarget

                                                          rt="route-target:as2-nn4:1:2110005"

                                                          type="import"

                                                          />

                              </bgpRtTargetP>

               </fvCtx>

               <!-- Tag_6 User BD Config towards DC PE Location 2-1 -->

               <fvBD OptimizeWanBandwidth="no"

                     arpFlood="no"

                     epClear="no"

                     hostBasedRouting="no"

                     intersiteBumTrafficAllow="no"

                     intersiteL2Stretch="no"

                     ipLearning="yes"

                     ipv6McastAllow="no"

                     limitIpLearnToSubnets="yes"

                     llAddr="::"

                     mac="00:22:BD:F8:19:FF"

                     mcastAllow="no"

                     multiDstPktAct="bd-flood"

                     name="BD2105"

                     type="regular"

                     unicastRoute="yes"

                     unkMacUcastAct="proxy"

                     unkMcastAct="flood"

                     v6unkMcastAct="flood"

                     vmac="not-applicable">

                              <fvSubnet ctrl="nd"

                                        ip="2001:100:21:5::254/64"

                                        preferred="no"

                                        scope="public"

                                        virtual="no"/>

                              <fvSubnet ctrl="nd"

                                        ip="100.21.5.254/24"

                                        preferred="no"

                                        scope="public"

                                        virtual="no"/>

                              <fvRsMldsn

                                         />

                              <fvRsIgmpsn

                                          />

                              <fvRsCtx

                                       tnFvCtxName="TENANT_BASIC_21"

                                       />

                              <fvRsBdToEpRet

                                             resolveAct="resolve"

                                             />

                              <fvRsBDToOut

                                           tnL3extOutName="LOCATION-2-1-MPLS-TENANT-L3OUT-2105"

                                           />

                              <fvRsBDToNdP

                                           />

               </fvBD>

               <!-- Tag_7 User BD Config towards DC PE Location 2-1-2-1 -->

               <fvBD OptimizeWanBandwidth="no"

                     arpFlood="no"

                     epClear="no"

                     hostBasedRouting="no"

                     intersiteBumTrafficAllow="no"

                     intersiteL2Stretch="no"

                     ipLearning="yes"

                     ipv6McastAllow="no"

                     limitIpLearnToSubnets="yes"

                     llAddr="::"

                     mac="00:22:BD:F8:19:FF"

                     mcastAllow="no"

                     multiDstPktAct="bd-flood"

                     name="BD2106"

                     type="regular"

                     unicastRoute="yes"

                     unkMacUcastAct="proxy"

                     unkMcastAct="flood"

                     v6unkMcastAct="flood"

                     vmac="not-applicable">

                              <fvSubnet ctrl="nd"

                                        ip="2001:100:21:6::254/64"

                                        preferred="no"

                                        scope="public"

                                        virtual="no"/>

                              <fvSubnet ctrl="nd"

                                        ip="100.21.6.254/24"

                                        preferred="no"

                                        scope="public"

                                        virtual="no"/>

                              <fvRsMldsn

                                         />

                              <fvRsIgmpsn

                                          />

                              <fvRsCtx

                                       tnFvCtxName="TENANT_BASIC_2121"

                                       />

                              <fvRsBdToEpRet

                                             resolveAct="resolve"

                                             />

                              <fvRsBDToOut

                                           tnL3extOutName="LOCATION-2-1-2-1-MPLS-TENANT-L3OUT-2106"

                                           />

                              <fvRsBDToNdP

                                           />

               </fvBD>

               <!-- Tag_8 Application Profile Config towards DC PE Location 2-1 -->

               <fvAp

                     name="LOCATION-2-1-2-1-AP2106"

                     prio="unspecified"

                     >

                              <fvAEPg

                                      floodOnEncap="disabled"

                                      hasMcastSource="no"

                                      isAttrBasedEPg="no"

                                      matchT="AtleastOne"

                                      name="EPG106"

                                      pcEnfPref="unenforced"

                                      prefGrMemb="exclude"

                                      prio="unspecified"

                                      shutdown="no"

                                      >

                                             <fvRsProv

                                                       intent="install"

                                                       matchT="AtleastOne"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <fvRsPathAtt

                                                          encap="vlan-2101"

                                                          instrImedcy="lazy"

                                                          mode="regular"

                                                          primaryEncap="unknown"

                                                          tDn="topology/pod-1/paths-105/pathep-[eth1/10]"

                                                          />

                                             <fvRsDomAtt bindingType="none"

                                                         classPref="encap"

                                                         encap="unknown"

                                                         encapMode="auto"

                                                         epgCos="Cos0"

                                                         epgCosPref="disabled"

                                                         instrImedcy="lazy"

                                                         netflowDir="both"

                                                         netflowPref="disabled"

                                                         numPorts="0"

                                                         portAllocation="none"

                                                         primaryEncap="unknown"

                                                         primaryEncapInner="unknown"

                                                         resImedcy="lazy"

                                                         secondaryEncapInner="unknown"

                                                         switchingMode="native"

                                                         tDn="uni/phys-phys"

                                                         untagged="no"

                                                         />

                                             <fvRsCons

                                                       intent="install"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <fvRsCustQosPol

                                                             />

                                             <fvRsBd

                                                     tnFvBDName="BD2106"

                                                     />

                              </fvAEPg>

               </fvAp>

               <!-- Tag_9 Application Profile Config towards DC PE Location 2-1-2-1 -->

               <fvAp

                     name="LOCATION-2-1-AP2105"

                     prio="unspecified"

                     >

                              <fvAEPg

                                      floodOnEncap="disabled"

                                      hasMcastSource="no"

                                      isAttrBasedEPg="no"

                                      matchT="AtleastOne"

                                      name="EPG105"

                                      pcEnfPref="unenforced"

                                      prefGrMemb="exclude"

                                      prio="unspecified"

                                      shutdown="no"

                                      >

                                             <fvRsProv

                                                       intent="install"

                                                       matchT="AtleastOne"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <fvRsPathAtt

                                                          encap="vlan-2101"

                                                          instrImedcy="immediate"

                                                          mode="regular"

                                                          primaryEncap="unknown"

                                                          tDn="topology/pod-1/paths-101/pathep-[eth1/10]"

                                                          />

                                             <fvRsDomAtt bindingType="none"

                                                         classPref="encap"

                                                         encap="unknown"

                                                         encapMode="auto"

                                                         epgCos="Cos0"

                                                         epgCosPref="disabled"

                                                         instrImedcy="lazy"

                                                         netflowDir="both"

                                                         netflowPref="disabled"

                                                         numPorts="0"

                                                         portAllocation="none"

                                                         primaryEncap="unknown"

                                                         primaryEncapInner="unknown"

                                                         resImedcy="immediate"

                                                         secondaryEncapInner="unknown"

                                                         switchingMode="native"

                                                         tDn="uni/phys-phys"

                                                         untagged="no"

                                                         />

                                             <fvRsCons

                                                       intent="install"

                                                       prio="unspecified"

                                                       tnVzBrCPName="default"

                                                       />

                                             <fvRsCustQosPol

                                                             />

                                             <fvRsBd

                                                     tnFvBDName="BD2105"

                                                     />

                              </fvAEPg>

               </fvAp>

     </fvTenant>

</imdata>

Verifications

XR: Site of Origin setting

 

On PE2, when receiving the VPN unicast route from PE6 and PE7, the path contains the SoO that has been set by the remote PEs (1:6). As the site connected to PE1/PE2 uses a different SoO, PE2 advertises the route to the BL by setting the new SoO value 987654321:5.

 

RP/0/RSP0/CPU0:PE02#show bgp vpnv4 uni vrf TENANT_BASIC 100.21.6.0

Fri Sep  4 12:33:17.060 UTC

BGP routing table entry for 100.21.6.0/24, Route Distinguisher: 1:2110002

Versions:

  Process           bRIB/RIB  SendTblVer

  Speaker             287129      287129

    Local Label: 32068

Last Modified: Sep  3 16:01:33.144 for 20:31:44

Paths: (2 available, best #1)

  Advertised to update-groups (with more than one peer):

    0.2

  Path #1: Received by speaker 0

  Advertised to update-groups (with more than one peer):

    0.2

  987654321

    201.201.201.6 (metric 20150) from 202.202.202.102 (201.201.201.6)

      Received Label 32074

      Origin incomplete, metric 0, localpref 100, valid, internal, best, group-best, import-candidate, imported, reoriginated with stitching-rt

      Received Path ID 1, Local Path ID 1, version 287128

      Extended community: SoO:1:6 RT:1:2110005

      Originator: 201.201.201.6, Cluster list: 0.0.0.2

  

      Source AFI: VPNv4 Unicast, Source VRF: default, Source Route Distinguisher: 1:2110006

  Path #2: Received by speaker 0

  Not advertised to any peer

  987654321

    201.201.201.7 (metric 20150) from 202.202.202.102 (201.201.201.7)

      Received Label 32109

      Origin incomplete, metric 0, localpref 100, valid, internal, add-path, import-candidate, imported, reoriginated with stitching-rt

      Received Path ID 1, Local Path ID 2, version 287129

      Extended community: SoO:1:6 RT:1:2110005

      Originator: 201.201.201.7, Cluster list: 0.0.0.2, 0.0.0.1

     

      Source AFI: VPNv4 Unicast, Source VRF: default, Source Route Distinguisher: 1:2110007

 

RP/0/RSP0/CPU0:PE02#show bgp vpnv4 unicast advertised neighbor 20.204.103.1 | be 1:2110002

Fri Sep  4 12:31:39.918 UTC

Route Distinguisher: 1:2110002

100.21.6.0/24 is advertised to 20.204.103.1

  Path info:

    neighbor: 202.202.202.102 neighbor router id: 201.201.201.6

    valid  internal  best  import-candidate  imported  reoriginated  with stitching-rt

Received Path ID 1, Local Path ID 1, version 287128

  Attributes after inbound policy was applied:

    next hop: 201.201.201.6

    MET ORG AS LOCAL EXTCOMM

    origin: incomplete  neighbor as: 987654321  metric: 0  local pref: 100

    aspath: 987654321

    extended community: SoO:1:6 RT:1:2110005

    originator: 201.201.201.6    cluster list: 0.0.0.2

 

  Attributes after outbound policy was applied:

    next hop: 201.201.201.2

    ORG AS LOCAL EXTCOMM

    origin: incomplete  neighbor as: 987654321  local pref: 100

    aspath: 1 987654321

    extended community: RT:1:2110005 SoO:987654321:5

 

On PE2, the route 100.21.5.0/24 received from the locally connected ACI site are also tagged with an SoO value of 1:5. The second path associated to this prefix is received from PE1 and also has the SoO value 1:5 which prevents PE2 to advertise the path from PE1 to the BL in case it becomes the best path.

 

RP/0/RSP0/CPU0:PE02#show bgp vpnv4 uni vrf TENANT_BASIC 100.21.5.0

Fri Sep  4 12:36:21.805 UTC

BGP routing table entry for 100.21.5.0/24, Route Distinguisher: 1:2110002

Versions:

  Process           bRIB/RIB  SendTblVer

  Speaker             292940      292940

    Local Label: 32068

Last Modified: Sep  4 07:58:54.144 for 04:37:28

Paths: (3 available, best #1)

  Advertised to update-groups (with more than one peer):

    0.3

  Path #1: Received by speaker 0

  Advertised to update-groups (with more than one peer):

    0.3

  987654321

    20.204.103.1 from 20.204.103.1 (30.204.103.1)

      Received Label 949935

      Origin incomplete, metric 0, localpref 100, valid, external, best, group-best, import-candidate, imported, reoriginated

      Received Path ID 0, Local Path ID 1, version 292940

      Extended community: SoO:1:5 Color:500 RT:1:2110000

      EVPN Gateway Address : 0.0.0.0

      Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 103:2392067

  Path #2: Received by speaker 0

  Advertised to update-groups (with more than one peer):

    0.3

  987654321

    20.204.104.1 from 20.204.104.1 (30.204.104.1)

      Received Label 949935

      Origin incomplete, metric 0, localpref 100, valid, external, add-path, import-candidate, imported, reoriginated

      Received Path ID 0, Local Path ID 5, version 292940

      Extended community: SoO:1:5 RT:1:2110000

      EVPN Gateway Address : 0.0.0.0

      Source AFI: L2VPN EVPN, Source VRF: default, Source Route Distinguisher: 104:2392067

  Path #3: Received by speaker 0

  Not advertised to any peer

  987654321

    201.201.201.1 (metric 1000) from 202.202.202.102 (201.201.201.1)

      Received Label 32062

      Origin incomplete, metric 0, localpref 100, valid, internal, add-path, import-candidate, imported, reoriginated with stitching-rt

      Received Path ID 1, Local Path ID 4, version 292895

      Extended community: SoO:1:5 RT:1:2110005

      Originator: 201.201.201.1, Cluster list: 0.0.0.2, 0.0.0.1

 

      Source AFI: VPNv4 Unicast, Source VRF: default, Source Route Distinguisher: 1:2110001

 

 


 

ACI Routes Received and Advertised

 

As displayed below, leaf3 has two BGP-LU and EVPN sessions where it receives prefixes.

 

ifav204-leaf3# show bgp ipv4 labeled-unicast summary vrf overlay-1

BGP summary information for VRF overlay-1, address family IPv4 Label Unicast

BGP router identifier 30.204.103.1, local AS number 987654321

BGP table version is 13, IPv4 Label Unicast config peers 2, capable peers 2

3 network entries and 3 paths using 612 bytes of memory

BGP attribute entries [1/160], BGP AS path entries [0/0]

BGP community entries [5/160], BGP clusterlist entries [2/8]

 

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

120.1.51.2      4     1     652     656       13    0    0 00:17:36 1

120.1.53.2      4     1     686     687       13    0    0 11:22:21 1

 

ifav204-leaf3# show bgp l2vpn evpn summary vrf overlay-1

BGP summary information for VRF overlay-1, address family L2VPN EVPN

BGP router identifier 30.204.103.1, local AS number 987654321

BGP table version is 2777, L2VPN EVPN config peers 2, capable peers 2

401 network entries and 520 paths using 65452 bytes of memory

BGP attribute entries [82/13120], BGP AS path entries [0/0]

BGP community entries [5/160], BGP clusterlist entries [2/8]

 

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

201.201.201.1   4     1    1467     722     2777    0    0 00:17:31 119

201.201.201.2   4     1    1302     744     2777    0    0 11:22:23 119

 

ifav204-leaf3#

 

We may notice that the BD subnet is now advertised to both the EVPN peers (PE01 and PE02). The prefix 100.21.5.0/24 is advertised to both 201.201.201.1 (PE01) and 201.201.201.2 (PE2).

 

ifav204-leaf3# show bgp l2vpn evpn 100.21.5.0 vrf overlay-1

Route Distinguisher: 103:2392067    (L3VNI 2392067)

BGP routing table entry for [5]:[0]:[0]:[24]:[100.21.5.0]:[0.0.0.0]/224, version 2786 dest ptr 0xacfdf810

Paths: (1 available, best #1)

Flags: (0x00000a 00000000) on xmit-list, is not in rib/evpn

Multipath: eBGP iBGP

 

  Advertised path-id 1

  Path type: local 0x4000008c 0x0 ref 0 adv path ref 1, path is valid, is best path

  AS-Path: NONE, path locally originated

    0.0.0.0 (metric 0) from 0.0.0.0 (30.204.103.1)

      Origin incomplete, MED 0, localpref 100, weight 32768 tag 4294966001, propagate 0

      Received label 949935

      Extcommunity:

          RT:1:2110005

          VNID:2392067

 

  Path-id 1 advertised to peers:

    201.201.201.1      201.201.201.2

 

The leaf receives the subnet 100.21.6.0/24 (located in site 2-1-2-1) from the two DC-PE.

 

ifav204-leaf3# show bgp l2vpn evpn 100.21.6.0 vrf overlay-1

Route Distinguisher: 1:2110001

BGP routing table entry for [5]:[0]:[0]:[24]:[100.21.6.0]:[0.0.0.0]/224, version 2782 dest ptr 0xacfd41fe

Paths: (1 available, best #1)

Flags: (0x000002 00000000) on xmit-list, is not in rib/evpn, is not in HW, is locked

Multipath: eBGP iBGP

 

  Advertised path-id 1

  Path type: external 0x40000028 0x0 ref 2 adv path ref 1, path is valid, is best path

             Imported to 2 destination(s)

  AS-Path: 1 987654321 , path sourced external to AS

    201.201.201.1 (metric 0) from 201.201.201.1 (201.201.201.1)

      Origin incomplete, MED not set, localpref 100, weight 0 tag 4294966257, propagate 0

      Received label 32065

      Extcommunity:

          RT:1:2110005

          SOO:987654321:5

 

  Path-id 1 not advertised to any peer

 

Route Distinguisher: 1:2110002

BGP routing table entry for [5]:[0]:[0]:[24]:[100.21.6.0]:[0.0.0.0]/224, version 2778 dest ptr 0xacfd07fa

Paths: (1 available, best #1)

Flags: (0x000002 00000000) on xmit-list, is not in rib/evpn, is not in HW, is locked

Multipath: eBGP iBGP

 

  Advertised path-id 1

  Path type: external 0x40000028 0x0 ref 2 adv path ref 1, path is valid, is best path

             Imported to 2 destination(s)

  AS-Path: 1 987654321 , path sourced external to AS

    201.201.201.2 (metric 0) from 201.201.201.2 (201.201.201.2)

      Origin incomplete, MED not set, localpref 100, weight 0 tag 4294966257, propagate 0

      Received label 32068

      Extcommunity:

          RT:1:2110005

          SOO:987654321:5

 

  Path-id 1 not advertised to any peer

 

Route Distinguisher: 103:2392067    (L3VNI 2392067)

BGP routing table entry for [5]:[0]:[0]:[24]:[100.21.6.0]:[0.0.0.0]/224, version 2784 dest ptr 0xacfd21e6

Paths: (2 available, best #2)

Flags: (0x000002 00000000) on xmit-list, is not in rib/evpn, is not in HW

Multipath: eBGP iBGP

 

  Path type: external 0xc0020028 0x0 ref 0 adv path ref 0, path is valid, not best reason: newer EBGP path, multipath

             Imported from 1:2110001:[5]:[0]:[0]:[24]:[100.21.6.0]:[0.0.0.0]/120

  AS-Path: 1 987654321 , path sourced external to AS

    201.201.201.1 (metric 0) from 201.201.201.1 (201.201.201.1)

      Origin incomplete, MED not set, localpref 100, weight 0 tag 4294966257, propagate 0

      Received label 32065

      Extcommunity:

          RT:1:2110005

          SOO:987654321:5

 

  Advertised path-id 1

  Path type: external 0xc0000028 0x0 ref 0 adv path ref 1, path is valid, is best path

             Imported from 1:2110002:[5]:[0]:[0]:[24]:[100.21.6.0]:[0.0.0.0]/120

  AS-Path: 1 987654321 , path sourced external to AS

    201.201.201.2 (metric 0) from 201.201.201.2 (201.201.201.2)

      Origin incomplete, MED not set, localpref 100, weight 0 tag 4294966257, propagate 0

      Received label 32068

      Extcommunity:

          RT:1:2110005

          SOO:987654321:5

 

  Path-id 1 not advertised to any peer

 

Load Balancing Between DC-PE and BL/RL

 

As described earlier, the enablement of BGP multipath enables the load balancing of traffic between multiple BGP paths.

In the described example, there is one path per DC-PE. However, it is also perfectly fine to have multiple parallel IP links between an RL/BL and a DC-PE and then create BGP multipath. When using multiple parallel IP links, a mix of interface types can be used for each link (physical interface, sub-interface, port-channel). There will be one BGP-LU session per IP link.

To simplify, it is also possible to use bundles (port-channels) between a DC-PE and a BL/RL. In this case, there is a single BGP-LU session per bundle interface.

 

Related image, diagram or screenshot

Figure 13.         

Load Balancing

SR-Handoff with Multihoming and Multiple ACI Sites Connected to the Same DC-PEs

Design

 

 

DiagramDescription automatically generated

Figure 14.         

Basic Communication Between ACI Fabric and Multiple Remote Leafs Connected to the Same DC-PE

The design above is the same as the previous one, except that there are two remote leaf sites connected to the same pair of DC-PEs. To use SR/MPLS handoff between the two remote leaf sites, each site must use a different VRF. If the VRF is stretched across the two remote leaf sites, VXLAN path will be used.

This design requires a special feature to be activated on the DC-PE to enable the propagation of EVPN routes from one ACI site to another one.

While this example uses two remote leaf switches connected to the same DC-PE, a similar design and configuration has been tested with two PODs of an Cisco ACI fabric connecting to the same pair of DC-PE and will also apply for multiple Cisco ACI fabrics connecting to the same pair of DC-PE (as displayed in Figure 14).

 

 

 

DiagramDescription automatically generated

Figure 15.         

Multiple PODs and Multiple Fabrics Connected to the Same DC-PEs

Configuration

PE7 DC-PE Configuration

As a reminder, the EVPN neighbor-group “ACI-site-EVPN” is used to configure the EVPN session parameters. When multiple sites/pods are connected to the same PE, the “next-hop-unchanged” knob must be configured. When an EVPN route comes from site 6 on PE6 or PE7, it is propagated to site 7 without changing the BGP nexthop (and vice versa). Having the XR DC-PE changing the EVPN nexthop and MPLS label (ASBR option B behavior) is currently not supported.

 

router bgp 1

 neighbor-group ACI-site-EVPN

  remote-as 987654321

  bfd fast-detect

  bfd multiplier 3

  bfd minimum-interval 50

  ebgp-multihop 255

  update-source Loopback0

  address-family l2vpn evpn

   import stitching-rt re-originate

   allowas-in 5

   advertise vpnv4 unicast re-originated stitching-rt

   advertise vpnv6 unicast re-originated stitching-rt

   next-hop-unchanged

  !

 !

 

 

Consequently, the remote leaf from site 7 must then have reachability to the loopback address from the remote leaf located in site 6. This is achieved by allowing the propagation of the BGP LU prefixes from Site 6 to Site 7 and vice versa. BGP communities (1:52121 and 1:52122) are used to mark LU routes coming from each ACI site and for proper filtering and loop prevention.

The route-policies SET-CT-FROM-ACI-2-1-2-1 and SET-CT-FROM-ACI-2-1-2-2 are used as inbound policies on the BGP LU sessions respectively to site 6 and 7 and they respectively set the community 1:52121 and 1:52122.

The route-policies ADVERTISE-LU-TO-ACI-2-1-2-1 and ADVERTISE-LU-TO -2-1-2-2 are used as outbound policies on the BGP LU sessions respectively to site 6 and 7 and they have the following behavior:

      For ADVERTISE-LU-TO-ACI-2-1-2-1 (used to site 6): authorize the local loopback and the routes with community 1:52122 (routes from site 7) and drop everything else.

      For ADVERTISE-LU-TO-ACI-2-1-2-2 (used to site 7): authorize the local loopback and the routes with community 1:52121 (routes from site 6) and drop everything else.

 

Finally, the “as-override” keyword is also required on the BGP-LU session as we must exchange LU routes coming from different ACI sites that may use the same AS number. In our example, the two remote leaf sites are part of the same fabric and consequently are using the same AS number.

 

router bgp 1

 neighbor-group ACI-site-LU

  remote-as 987654321

  bfd fast-detect

  bfd multiplier 3

  bfd minimum-interval 50

  address-family ipv4 labeled-unicast

   maximum-prefix 10 80 warning-only

   as-override

  !

 !

 neighbor 120.1.61.1

  use neighbor-group ACI-site-LU

  address-family ipv4 labeled-unicast

   route-policy SET-CT-FROM-ACI-2-1-2-1 in

   route-policy ADVERTISE-LU-TO-ACI-2-1-2-1 out

  !

 !

 neighbor 120.1.64.1

  use neighbor-group ACI-site-LU

  address-family ipv4 labeled-unicast

   route-policy SET-CT-FROM-ACI-2-1-2-1 in

   route-policy ADVERTISE-LU-TO-ACI-2-1-2-1 out

  !

 !

 neighbor 120.1.73.1

  use neighbor-group ACI-site-LU

  address-family ipv4 labeled-unicast

   route-policy SET-CT-FROM-ACI-2-1-2-2 in

   route-policy ADVERTISE-LU-TO-ACI-2-1-2-2 out

  !

 !

 neighbor 120.1.74.1

  use neighbor-group ACI-site-LU

  address-family ipv4 labeled-unicast

   route-policy SET-CT-FROM-ACI-2-1-2-2 in

   route-policy ADVERTISE-LU-TO-ACI-2-1-2-2 out

  !

 !

!

route-policy SET-CT-FROM-ACI-2-1-2-1

  set community (1:52121)

end-policy

!

route-policy ADVERTISE-LU-TO-ACI-2-1-2-1

  if destination in PFXSET-OWN-LO0 then

    pass

  elseif community matches-any (1:52122) then

    pass

  else

    drop

  endif

end-policy

!

route-policy SET-CT-FROM-ACI-2-1-2-2

  set community (1:52122)

end-policy

!

route-policy ADVERTISE-LU-TO-ACI-2-1-2-2

  if destination in PFXSET-OWN-LO0 then

    pass

  elseif community matches-any (1:52121) then

    pass

  else

    drop

  endif

end-policy

!

 

ACI Configuration

INFRA Tenant Configuration

 

The configuration is the same as the previous multihoming case, except that the site 2-1-2-2 configuration is added.

 

<?xml version="1.0" encoding="utf-8"?>

<imdata totalCount="1">

     <fvTenant

               dn="uni/tn-infra"

               name="infra"

               >

               <!-- Tag_1 Default Label Range config in Infra Tenant -->

               <mplsLabelPol

                             maxDynamicLabel="525286"

                             maxStaticLabel="0"

                             minDynamicLabel="16"

                             minStaticLabel="0"

                             name="default"

                             >

                              <mplsSrgbLabelPol

                                                localId="1"

                                                maxSrgbLabel="32000"

                                                minSrgbLabel="16000"

                                                />

               </mplsLabelPol>

               <!-- Tag_2 Default interface config in Infra Tenant -->

               <mplsIfPol

                          name="default"

                          />

               <!-- Tag_3 MPLS Infra L3out towards DC PE Location 2-1 -->

               <l3extOut

                         enforceRtctrl="export"

                         mplsEnabled="yes"

                         name="2-1"

                         targetDscp="unspecified"

                         >

                              <mplsExtP

                                        >

                                             <mplsRsLabelPol

                                                             tDn="uni/tn-infra/mplslabelpol-default"

                                                             />

                              </mplsExtP>

                              <l3extRsL3DomAtt

                                               tDn="uni/l3dom-L3Dom"

                                               />

                              <l3extRsEctx

                                           tnFvCtxName="overlay-1"

                                           />

                              <l3extProvLbl

                                            name="2-1"

                                            tag="yellow-green"

                                            />

                              <l3extLNodeP

                                           name="2-1_nodeProfile"

                                           tag="yellow-green"

                                           targetDscp="unspecified"

                                           >

                                             <l3extRsNodeL3OutAtt

                                                                  rtrId="30.204.103.1"

                                                                  rtrIdLoopBack="no"

                                                                  tDn="topology/pod-1/node-103"

                                                                  >

                                                            <l3extLoopBackIfP addr="20.204.103.1"

                                                                              >

                                                                          <mplsNodeSidP

                                                                                        loopbackAddr="20.204.103.1"

                                                                                        sidoffset="45"

                                                                                        />

                                                            </l3extLoopBackIfP>

                                             </l3extRsNodeL3OutAtt>

                                             <l3extRsNodeL3OutAtt

                                                                  rtrId="30.204.104.1"

                                                                  rtrIdLoopBack="no"

                                                                  tDn="topology/pod-1/node-104"

                                                                  >

                                                            <l3extLoopBackIfP addr="20.204.104.1"

                                                                              >

                                                                          <mplsNodeSidP

                                                                                        loopbackAddr="20.204.104.1"

                                                                                        sidoffset="46"

                                                                                        />

                                                            </l3extLoopBackIfP>

                                             </l3extRsNodeL3OutAtt>

                                             <l3extRsLNodePMplsCustQosPol

                                                                          tDn="uni/tn-infra/qosmplscustom-"

                                                                          />

                                             <l3extLIfP

                                                        name="2-1_interfaceProfile"

                                                        prio="unspecified"

                                                        tag="yellow-green"

                                                        >

                                                            <l3extRsPathL3OutAtt addr="120.1.51.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-103/pathep-[eth1/18]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.51.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.53.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-103/pathep-[eth1/1]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.53.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                     ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.52.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-104/pathep-[eth1/14]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.52.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.54.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-104/pathep-[eth1/2]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.54.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <mplsIfP

                                                                     >

                                                                          <mplsRsIfPol

                                                                                       tnMplsIfPolName="default"

                                                                                       />

                                                            </mplsIfP>

                                                            <bfdIfP

                                                                    keyId="1"

                                                                    type="none"

                                                                    >

                                                                          <bfdRsIfPol

                                                                                      tnBfdIfPolName="BFD_Pol"

                                                                                      />

                                                            </bfdIfP>

                                             </l3extLIfP>

                                             <bgpInfraPeerP addr="201.201.201.2"

                                                            addrTCtrl="af-ucast"

                                                            adminSt="enabled"

                                                            allowedSelfAsCnt="3"

                                                            ctrl="allow-self-as,dis-peer-as-check,send-com,send-ext-com"

                                                            dataPlaneAddr="0.0.0.0"

                                                            peerCtrl="bfd"

                                                            peerT="SR/MPLS"

                                                            remoteIntersiteRR="no"

                                                            srcIfT="l3out-loopback"

                                                            trustCtrl="untrusted"

                                                            ttl="16"

                                                            weight="0">

                                                            <bgpRsPeerPfxPol

                                                                             />

                                                            <bgpAsP

                                                                    asn="1"

                                                                    />

                                             </bgpInfraPeerP>

                                             <bgpInfraPeerP addr="201.201.201.1"

                                                            addrTCtrl="af-ucast"

                                                            adminSt="enabled"

                                                            allowedSelfAsCnt="3"

                                                            ctrl="allow-self-as,dis-peer-as-check,send-com,send-ext-com"

                                                            dataPlaneAddr="0.0.0.0"

                                                            peerCtrl="bfd"

                                                            peerT="SR/MPLS"

                                                            remoteIntersiteRR="no"

                                                            srcIfT="l3out-loopback"

                                                            trustCtrl="untrusted"

                                                            ttl="16"

                                                            weight="0">

                                                            <bgpRsPeerPfxPol

                                                                             />

                                                            <bgpAsP

                                                                    asn="1"

                                                                    />

                                             </bgpInfraPeerP>

                              </l3extLNodeP>

                              <l3extInstP

                                          floodOnEncap="disabled"

                                          matchT="AtleastOne"

                                          name="2-1_mplsInstP"

                                          prefGrMemb="exclude"

                                          prio="unspecified"

                                          targetDscp="unspecified"

                                          >

                                             <fvRsCustQosPol

                                                             />

                              </l3extInstP>

                              <bgpExtP

                                       />

               </l3extOut>

               <!-- Tag_4 MPLS Infra L3out towards DC PE Location 2-1-2-1 -->

               <l3extOut

                         enforceRtctrl="export"

                         mplsEnabled="yes"

                         name="2-1-2-1"

                         targetDscp="unspecified"

                         >

                              <mplsExtP

                                        >

                                             <mplsRsLabelPol

                                                             tDn="uni/tn-infra/mplslabelpol-default"

                                                             />

                              </mplsExtP>

                              <l3extRsL3DomAtt

                                               tDn="uni/l3dom-L3Dom"

                                               />

                              <l3extRsEctx

                                           tnFvCtxName="overlay-1"

                                           />

                              <l3extProvLbl

                                            name="2-1-2-1"

                                            tag="yellow-green"

                                            />

                              <l3extLNodeP

                                           name="2-1-2-1_nodeProfile"

                                           tag="yellow-green"

                                           targetDscp="unspecified"

                                           >

                                             <l3extRsNodeL3OutAtt

                                                                  rtrId="30.204.105.1"

                                                                  rtrIdLoopBack="no"

                                                                  tDn="topology/pod-1/node-105"

                                                                  >

                                                            <l3extLoopBackIfP addr="20.204.105.1"

                                                                              >

                                                                          <mplsNodeSidP

                                                                                        loopbackAddr="20.204.105.1"

                                                                                        sidoffset="47"

                                                                                        />

                                                            </l3extLoopBackIfP>

                                             </l3extRsNodeL3OutAtt>

                                             <l3extRsNodeL3OutAtt

                                                                  rtrId="30.204.106.1"

                                                                  rtrIdLoopBack="no"

                                                                  tDn="topology/pod-1/node-106"

                                                                  >

                                                            <l3extLoopBackIfP addr="20.204.106.1"

                                                                              >

                                                                          <mplsNodeSidP

                                                                                        loopbackAddr="20.204.106.1"

                                                                                        sidoffset="48"

                                                                                        />

                                                            </l3extLoopBackIfP>

                                             </l3extRsNodeL3OutAtt>

                                             <l3extRsLNodePMplsCustQosPol

                                                                          tDn="uni/tn-infra/qosmplscustom-"

                                                                          />

                                             <l3extLIfP

                                                        name="2-1-2-1_interfaceProfile"

                                                        prio="unspecified"

                                                        tag="yellow-green"

                                                        >

                                                            <l3extRsPathL3OutAtt addr="120.1.63.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-106/pathep-[eth1/2]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.63.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.62.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-105/pathep-[eth1/1]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.62.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.64.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-106/pathep-[eth1/3]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.64.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <l3extRsPathL3OutAtt addr="120.1.61.1/24"

                                                                                 autostate="disabled"

                                                                                 encap="unknown"

                                                                                 encapScope="local"

                                                                                 ifInstT="l3-port"

                                                                                 ipv6Dad="enabled"

                                                                                 llAddr="::"

                                                                                 mac="00:22:BD:F8:19:FF"

                                                                                 mode="regular"

                                                                                 mtu="inherit"

                                                                                 tDn="topology/pod-1/paths-105/pathep-[eth1/3]"

                                                                                 targetDscp="unspecified"

                                                                                 >

                                                                          <bgpPeerP addr="120.1.61.2"

                                                                                    addrTCtrl="af-label-ucast,af-ucast"

                                                                                    adminSt="enabled"

                                                                                    allowedSelfAsCnt="3"

                                                                                    peerCtrl="bfd"

                                                                                    ttl="1"

                                                                                    weight="0">

                                                                                    <bgpRsPeerPfxPol

                                                                                                     />

                                                                                    <bgpAsP

                                                                                            asn="1"

                                                                                            />

                                                                          </bgpPeerP>

                                                            </l3extRsPathL3OutAtt>

                                                            <mplsIfP

                                                                     >

                                                                          <mplsRsIfPol

                                                                                       tnMplsIfPolName="default"

                                                                                       />

                                                            </mplsIfP>

                                                            <bfdIfP

                                                                    keyId="1"

                                                                    type="none"

                                                                    >

                                                                          <bfdRsIfPol

                                                                                      tnBfdIfPolName="BFD_Pol"

                                                                                      />

                                                            </bfdIfP>

                                             </l3extLIfP>

                                             <bgpInfraPeerP addr="201.201.201.7"

                                                            addrTCtrl="af-ucast"

                                                            adminSt="enabled"

                                                            allowedSelfAsCnt="3"

                                                            ctrl="allow-self-as,dis-peer-as-check,send-com,send-ext-com"

                                                            dataPlaneAddr="0.0.0.0"

                                                            peerCtrl="bfd"

                                                            peerT="SR/MPLS"

                                                            remoteIntersiteRR="no"

                                                            srcIfT="l3out-loopback"

                                                            trustCtrl="untrusted"

                                                            ttl="16"

                                                            weight="0">

                                                            <bgpRsPeerPfxPol

                                                                             />

                                                            <bgpAsP

                                                                    asn="1"

                                                                    />

                                             </bgpInfraPeerP>

                                             <bgpInfraPeerP addr="201.201.201.6"

                                                            addrTCtrl="af-ucast"

                                                            adminSt="enabled"

                                                            allowedSelfAsCnt="3"

                                                            ctrl="allow-self-as,dis-peer-as-check,send-com,send-ext-com"

                                                            dataPlaneAddr="0.0.0.0"

                                                            peerCtrl="bfd"

                                                            peerT="SR/MPLS"

                                                            remoteIntersiteRR="no"

                                                            srcIfT="l3out-loopback"

                                                            trustCtrl="untrusted"

                                                            ttl="16"

                                                            weight="0">

                                                            <bgpRsPeerPfxPol

                                                                             />

                                                            <bgpAsP

                                                                    asn="1"

                                                                    />

                                             </bgpInfraPeerP>

                              </l3extLNodeP>

                              <l3extInstP

                                          floodOnEncap="disabled"

                                          matchT="AtleastOne"

                                          name="2-1-2-1_mplsInstP"

                                          prefGrMemb="exclude"

                                          prio="unspecified"

                                          targetDscp="unspecified"

                                          >

                                             <fvRsCustQosPol

                                                             />

                              </l3extInstP>

                              <bgpExtP

                                       />

               </l3extOut>

               <!-- Tag_5 MPLS Infra L3out towards DC PE Location 2-1-2-2 -->

               <l3extOut

                         enforceRtctrl="export"

                         mplsEnabled="yes"

                         name="2-1-2-2"

                         targetDscp="unspecified"

                         >

                              <mplsExtP

                                        >

                                             <mplsRsLabelPol

            &nbs