Configuring Seamless Integration of EVPN with L3VPN (MPLS SR)

This chapter contains these sections:

Information About Configuring Seamless Integration of EVPN with L3VPN (MPLS SR)

Data Center (DC) deployments have adopted VXLAN EVPN for its benefits such as EVPN control-plane learning, multi-tenancy, seamless mobility, redundancy, and easier horizontal scaling. Similarly, the Core network transitions to different technologies with their respective capabilities. MPLS with Label Distribution Protocol (LDP) and Layer-3 VPN (L3VPN) is present in many Core networks interconnecting Data Centers. With the technology evolution, a transformation from the traditional MPLS L3VPN with LDP-based underlay to MPLS-based Segment Routing (SR) with L3VPN became available. Segment Routing is adopted for its benefits such as:

  • Unified IGP and MPLS control planes

  • Simpler traffic engineering methods

With the Data Center (DC) established on VXLAN EVPN and the Core network requiring multi-tenant capable transport, there is a natural necessity for seamless integration. To provide this integration between different control-plane protocols and encapsulations—from VXLAN to an MPLS-based Core network—the Cisco Nexus 9000 Series Switch provides the Border Provider Edge (Border PE) capability by interfacing the Data Center and the Core routers (Provider Routers or Provider Edge-Routers).

Deployment Scenarios and Integration Details

There are multiple deployment scenarios for integrating VXLAN EVPN Data Center fabrics with MPLS-based Segment Routing (SR) Core networks. The following sections describe typical topologies and operational details.

In the following scenario, a single Data Center Fabric running VXLAN EVPN is depicted. The VRFs (VRF_A, VRF_B) present in the Data Center require to be extended over a WAN/Core running MPLS-based Segment Routing (MPLS-SR). The Data Center Fabric’s Border switches act as Border Provider Edge (Border PE1, Border PE2) interconnecting VXLAN BGP EVPN with MPLS-SR with L3VPN (VPNv4/VPNv6). The BPEs are interconnected with the Provider Router (P-Router) via eBGP using the IPv4 Labeled-Unicast as well as the VPNv4/VPNv6 Address-Family (AF). The P-Router acts as BGP Route-Reflector for the mentioned AF and relays the necessary routes to the MPLS-SR Provider Edge (PE3, PE4) via iBGP. Beyond the usage of BGP as the control plane, the MPLS-SR nodes within the same Autonomous System (AS) use an IGP (OSPF or ISIS) for label distribution. From the PEs shown in the above figure (PE3, PE4), Inter-AS Option A can be used to extend the Data Center or Core network VRFs to another external network. Even though this diagram shows only one Data Center, the MPLS-SR network can interconnect multiple Data Center Fabrics.

Figure 1. Topology with DC to Core Network Domain Separation

An alternative deployment scenario is when the Core network is separated into multiple Administrative Domains or Autonomous Systems (AS). In the above figure, a single Data Center Fabric running VXLAN EVPN is depicted. The VRFs (VRF_A, VRF_B) present in the Data Center require to be extended over a WAN/Core running MPLS-based Segment Routing (MPLS-SR). The Data Center Fabric’s Border switches act as Border Provider Edge (Border PE1, Border PE2) interconnecting VXLAN BGP EVPN with MPLS-SR with L3VPN (VPNv4/VPNv6). The BPEs are interconnected with the Provider Router (P-Router) via eBGP using the IPv4 Labeled-Unicast as well as the VPNv4/VPNv6 Address-Family (AF). The P-Router acts as BGP Route Server for the mentioned AF and relays the necessary routes to the MPLS-SR Provider Edge (PE3, PE4) via eBGP; no other control-plane protocol is used between the MPLS-SR nodes. Similar to the previous scenario, the PEs (PE3, PE4) can operate with Inter-AS Option A to extend the Data Center or Core network VRFs to an external network. Even though this diagram shows only one Data Center, the MPLS-SR network can interconnect multiple Data Center Fabrics.

Figure 2. Multiple Administrative Domains within the Core Network

For additional information on MPLS SR, see the Cisco Nexus 9000 Series NX-OS Label Switching Configuration Guide.

Guidelines and Limitations for Configuring Seamless Integration of EVPN with L3VPN (MPLS SR)

Supported Features and Platforms

The following Cisco Nexus platform switches support seamless integration of EVPN with L3VPN (MPLS SR):

  • 9336C-FX2 switches

  • 93240YC-FX2 switches

  • 9300-FX3 platform switches

  • 9300-GX platform switches

  • 9504 and 9508 platform switches with 96136YC-R and 9636C-RX line cards (The 9636C-R and 9636Q-R line cards are not supported.)

The following features are supported with seamless integration of EVPN with L3VPN (MPLS SR):

  • Host Facing (Downlinks towards)

    • Individual Layer-3 interfaces (orphan ports)

    • Layer-3 Port-Channel

    • Layer-3 Sub-interfaces

    • Inter-AS Option A (often also called VRF-lite)

  • Core Facing (Uplinks towards VXLAN)

    • Individual Layer-3 interfaces

    • Layer-3 Port-Channel

  • Core Facing (Uplinks towards MPLS SR)

    • Individual Layer-3 interface

    • Per-VRF labels

    • VPN label statistics

  • End-to-EndTime to Live (TTL) and Explicit Congestion Notification (ECN) with pipe-mode only.

  • MPLS SegmentRouting and MPLS LDP cannot be configured at the same time on a Cisco Nexus 9504 and 9508 platform switches with Cisco Nexus 96136YC-R and Cisco Nexus 9636C-RX line cards.

Feature

Cisco Nexus 9300-FX2, FX3, GX Platform Switches

Cisco Nexus 9504 and 9508 switches with -R Line Cards

Comments

VXLAN EVPN to SR-L3VPN

Yes

Yes

Extend Layer 3 connectivity between different DC pods Underlay IGP/BGP with SR extensions.

VXLAN EVPN to SR-L3VPN

Yes

Yes

Extend Layer 3 connectivity between DC POD running VXLAN and any domain (DC or CORE) running SR.
VXLAN EVPN to MPLS L3VPN (LDP)

No

Yes

Underlay is LDP.

Platform Limitations and Unsupported Features

The following features are not supported with seamless integration of EVPN with L3VPN (MPLS SR):

  • Distributed Anycast Gateway or First-Hop Redundancy Protocol like HSRP, VRRP or GLBP.

  • vPC for redundant Host or Network Service attachment.

  • SVI/Sub-interfaces for Core facing uplinks (MPLS or VXLAN).

  • SVI/Sub-interfaces with configured MAC addresses.

  • MPLS Segment Routing and Border Gateway (BGW for VXLAN Multi-Site) cannot be configured at the same time.

  • Layer-2 for stretched Subnet across the MPLS-SR domain

  • No-drop for VXLAN/SR and SR/VXLAN handoff, for Cisco Nexus 9336C-FX2, 93240YC-FX2, and 9300-FX3 platform switches

  • Statistics, for Cisco Nexus 9504 and 9508 platform switches with 96136YC-R and 9636C-RX line cards

  • Priority flow control (PFC), for Cisco Nexus 9336C-FX2, 93240YC-FX2, 9300-FX3, and 9300-GX platform switches

  • When using MPLS Segment Routing (SR) L2VPN, packets with an inner destination MAC address that begins with 4 or 6 are dropped. This is a platform limitation. Ensure inner destination MAC addresses do not start with 4 or 6 to avoid packet drops.

VXLAN-to-SR Handoff QoS Value Preservation

The VXLAN-to-SR handoff QoS value is preserved during handoff and propagated from VXLAN tunnel packets to SR-tunneled packets for Cisco Nexus 9336C-FX2, 93240YC-FX2, 9300-FX3, and 9300-GX platform switches.

Feature-Specific Guidelines and Limitations

Configuring Seamless Integration of EVPN with L3VPN (MPLS SR)

The following procedure for Border Provider Edge (Border PE) imports and reoriginates the routes from the VXLAN domain to the MPLS domain and in the other direction.

Procedure

Step 1

configure terminal

Example:

switch# configure terminal

Enters global configuration mode.

Step 2

Enable platform features

  1. feature-set mpls

    Example:

    switch(config)# feature-set mpls

    Enables the MPLS feature set.

  2. nv overlay evpn

    Example:

    switch(config)# nv overlay evpn

    Enables VXLAN.

  3. feature bgp

    Example:

    switch(config)# feature bgp

    Enables BGP.

  4. feature mpls l3vpn

    Example:

    switch(config)# feature mpls l3vpn

    Enables Layer 3 VPN.

    Note

     

    Feature mpls l3vpn requires feature mpls segment-routing.

  5. feature mpls segment-routing

    Example:

    switch(config)# feature mpls segment-routing

    Enables Segment Routing.

  6. feature interface-vlan

    Example:

    switch(config)# feature interface-vlan

    Enables the interface VLAN.

  7. feature vn-segment-vlan-based

    Example:

    switch(config)# feature vn-segment-vlan-based

    Enables the VLAN-based VN segment.

  8. feature nv overlay

    Example:

    switch(config)# feature nv overlay

    Enables VXLAN.

Step 3

Configure BGP for MPLS domain

  1. router bgp autonomous-system-number

    Example:

    switch(config)# router bgp 65095

    Configures BGP. The value of autonomous-system-number is from 1 to 4294967295.

  2. address-family ipv4 unicast

    Example:

    switch(config-router)# address-family ipv4 unicast

    Configures the address family for IPv4.

  3. network address

    Example:

    switch(config-router-af)# network 10.51.0.51/32

    Injects prefixes into BGP for the MPLS-SR domain.

    Note

     

    All viable next-hops for MPLS-SR tunnel deposition on the Border PE must be advertised via the network statement (/32 only).

  4. allocate-label all

    Example:

    switch(config-router-af)# allocate-label all

    Configures label allocation for every prefix injected via the network statement.

  5. exit

    Example:

    switch(config-router-af)# exit

    Exits command mode.

Step 4

Configure iBGP neighbor (Route Reflector)

  1. neighbor address remote-as number

    Example:

    switch(config-router)# neighbor 10.95.0.95 remote-as 65095

    Defines the iBGP neighbor IPv4 address and remote Autonomous-System (AS) number towards the Route-Reflector.

  2. update-source type/id

    Example:

    switch(config-router)# update-source loopback0

    Defines the interface for eBGP peering.

  3. address-family l2vpn evpn

    Example:

    switch(config-router)# address-family l2vpn evpn

    Configures the L2VPN EVPN address family.

  4. send-community both

    Example:

    switch(config-router-af)# send-community both

    Configures the community for BGP neighbors.

  5. import vpn unicast reoriginate

    Example:

    switch(config-router-af)# import vpn unicast reoriginate

    Reoriginates the route with a new Route-Target. It can be extended to use an optional route-map.

  6. exit

    Example:

    switch(config-router-af)# exit

    Exits command mode.

Step 5

Configure eBGP neighbor (P-Router)

  1. neighbor address remote-as number

    Example:

    switch(config-router)# neighbor 10.51.131.131 remote-as 65013

    Defines the eBGP neighbor IPv4 address and remote Autonomous-System (AS) number towards the P-Router.

  2. update-source type/id

    Example:

    switch(config-router)# update-source Ethernet1/1

    Defines the interface for eBGP peering.

  3. address-family ipv4 labeled-unicast

    Example:

    switch(config-router)# address-family ipv4 labeled-unicast

    Configures the address family for IPv4 labeled-unicast.

  4. send-community both

    Example:

    switch(config-router-af)# send-community both

    Configures the community for BGP neighbors.

  5. exit

    Example:

    switch(config-router-af)# exit

    Exits command mode.

Step 6

Configure eBGP neighbor (remote AS)

  1. neighbor address remote-as number

    Example:

    switch(config-router)# neighbor 10.131.0.131 remote-as 65013

    Defines the eBGP neighbor IPv4 address and remote Autonomous-System (AS) number.

  2. update-source type/id

    Example:

    switch(config-router)# update-source loopback0

    Defines the interface for eBGP peering.

  3. ebgp-multihop number

    Example:

    switch(config-router)# ebgp-multihop 5

    Specifies multihop TTL for the remote peer. The range of number is from 2 to 255.

  4. address-family vpnv4 unicast

    Example:

    switch(config-router)# address-family vpnv4 unicast

    Configures the address family for VPNv4 or VPNv6.

  5. send-community both

    Example:

    switch(config-router-af)# send-community both

    Configures the community for BGP neighbors.

  6. import l2vpn evpn reoriginate

    Example:

    switch(config-router-af)# import l2vpn evpn reoriginate

    Reoriginates the route with a new Route-Target. It can be extended to use an optional route-map.

  7. exit

    Example:

    switch(config-router-af)# exit

    Exits command mode.

Example Configuration for Configuring Seamless Integration of EVPN with L3VPN (MPLS SR)

This reference provides sample CLI configurations for scenarios involving route import and reorigination between VXLAN and MPLS domains, including both DC-to-Core and Core network domain separation topologies.

Scenario 1: DC to Core Network Domain Separation and IGP within MPLS-SR network

Scenario - 1 with DC to Core Network Domain Separation and IGP within MPLS-SR network.

Figure 3. Topology with DC to Core Network Domain Separation

The following is a sample CLI configuration that is required to import and reoriginate the routes from the VXLAN domain to the MPLS domain and in the reverse direction. The sample CLI configuration represents only the necessary configuration for the respective roles.

Border PE 


hostname BL51-N9336FX2
install feature-set mpls

feature-set mpls

feature bgp
feature mpls l3vpn
feature mpls segment-routing
feature ospf
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay

nv overlay evpn

mpls label range 16000 23999 static 6000 8000

segment-routing
  mpls
    connected-prefix-sid-map
      address-family ipv4
        10.51.0.51/32 index 51

vlan 2000
  vn-segment 50000

vrf context VRF_A
  vni 50000
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
    route-target import 50000:50000
    route-target export 50000:50000
  address-family ipv6 unicast
    route-target both auto
    route-target both auto evpn
    route-target import 50000:50000
    route-target export 50000:50000

interface Vlan2000
  no shutdown
  vrf member VRF_A
  no ip redirects
  ip forward
  ipv6 address use-link-local-only
  no ipv6 redirects

interface nve1
  no shutdown
  host-reachability protocol bgp
  source-interface loopback1
  member vni 50000 associate-vrf

interface Ethernet1/1
  description TO_P-ROUTER
  ip address 10.51.131.51/24
  mpls ip forwarding
  no shutdown

interface Ethernet1/36
  description TO_SPINE
  ip address 10.95.51.51/24
  ip router ospf 10 area 0.0.0.0
  no shutdown

interface loopback0
  description ROUTER-ID & SR-LOOPBACK
  ip address 10.51.0.51/32
  ip router ospf UNDERLAY area 0.0.0.0

interface loopback1
  description NVE-LOOPBACK
  ip address 10.51.1.51/32
  ip router ospf UNDERLAY area 0.0.0.0

router ospf UNDERLAY
  router-id 10.51.0.51

router bgp 65095
  address-family ipv4 unicast
    network 10.51.0.51/32
    allocate-label all
!
  neighbor 10.95.0.95
    remote-as 65095
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
      import vpn unicast reoriginate
!
  neighbor 10.51.131.131
    remote-as 65013
    update-source Ethernet1/1
    address-family ipv4 labeled-unicast
      send-community
      send-community extended
!
  neighbor 10.131.0.131
    remote-as 65013
    update-source loopback0
    ebgp-multihop 5
    address-family vpnv4 unicast
      send-community
      send-community extended
      import l2vpn evpn reoriginate
    address-family vpnv6 unicast
      send-community
      send-community extended
      import l2vpn evpn reoriginate

!
  vrf VRF_A
    address-family ipv4 unicast
      redistribute direct route-map fabric-rmap-redist-subnet

P-Router 

hostname P131-N9336FX2
install feature-set mpls

feature-set mpls

feature bgp
feature isis
feature mpls l3vpn
feature mpls segment-routing

mpls label range 16000 23999 static 6000 8000

segment-routing
  mpls
    connected-prefix-sid-map
      address-family ipv4
        10.131.0.131/32 index 131

route-map RM_NH_UNCH permit 10
  set ip next-hop unchanged

interface Ethernet1/1
  description TO_BORDER-PE
  ip address 10.51.131.131/24
  ip router isis 10
  mpls ip forwarding
  no shutdown

interface Ethernet1/11
  description TO_PE
  ip address 10.52.131.131/24
  ip router isis 10
  mpls ip forwarding
  no shutdown

interface loopback0
  description ROUTER-ID & SR-LOOPBACK
  ip address 10.131.0.131/32
  ip router isis 10

router isis 10
  net 49.0000.0000.0131.00
  is-type level-2
  address-family ipv4 unicast
    segment-routing mpls

router bgp 65013
  event-history detail
  address-family ipv4 unicast
    allocate-label all
!
  neighbor 10.51.131.51
    remote-as 65095
    update-source Ethernet1/1
    address-family ipv4 labeled-unicast
      send-community
      send-community extended
!
  neighbor 10.51.0.51
    remote-as 65095
    update-source loopback0
    ebgp-multihop 5
    address-family vpnv4 unicast
      send-community
      send-community extended
      route-map RM_NH_UNCH out
    address-family vpnv6 unicast
      send-community
      send-community extended
      route-map RM_NH_UNCH out
!
  neighbor 10.52.131.52
    remote-as 65013
    update-source Ethernet1/11
    address-family ipv4 labeled-unicast
      send-community
      send-community extended
!
  neighbor 10.52.0.52
    remote-as 65013
    update-source loopback0
    address-family vpnv4 unicast
      send-community
      send-community extended
      route-reflector-client
      route-map RM_NH_UNCH out
    address-family vpnv6 unicast
      send-community
      send-community extended
      route-reflector-client
      route-map RM_NH_UNCH out

Provider Edge (PE) 

hostname L52-N93240FX2
install feature-set mpls

feature-set mpls

feature bgp
feature isis
feature mpls l3vpn
feature mpls segment-routing

mpls label range 16000 23999 static 6000 8000

segment-routing
  mpls
    connected-prefix-sid-map
      address-family ipv4
        10.52.0.52/32 index 52

vrf context VRF_A
  rd auto
  address-family ipv4 unicast
    route-target import 50000:50000
    route-target export 50000:50000
  address-family ipv6 unicast
    route-target import 50000:50000
    route-target export 50000:50000

interface Ethernet1/49
  description TO_P-ROUTER
  ip address 10.52.131.52/24
  ip router isis 10
  mpls ip forwarding
  no shutdown

interface loopback0
  description ROUTER-ID & SR-LOOPBACK
  ip address 10.52.0.52/32
  ip router isis 10

router isis 10
  net 49.0000.0000.0052.00
  is-type level-2
  address-family ipv4 unicast
    segment-routing mpls

router bgp 65013
  address-family ipv4 unicast
    network 10.52.0.52/32
    allocate-label all
!
  neighbor 10.52.131.131
    remote-as 65013
    update-source Ethernet1/49
    address-family ipv4 labeled-unicast
      send-community
      send-community extended
!
  neighbor 10.131.0.131
    remote-as 65013
    update-source loopback0
    address-family vpnv4 unicast
      send-community
      send-community extended
    address-family vpnv6 unicast
      send-community
      send-community extended
!
  vrf VRF_A
    address-family ipv4 unicast
      redistribute direct route-map fabric-rmap-redist-subnet

Scenario 2: DC to Core and within Core Network Domain Separation (eBGP within MPLS-SR network)

Scenario - 2 with DC to Core and within Core Network Domain Separation (eBGP within MPLS-SR network).

Figure 4. Multiple Administrative Domains within the Core network

The following is a sample CLI configuration that is required to import and reoriginate the routes from the VXLAN domain to the MPLS domain and in the reverse direction. The sample CLI configuration represents only the nodes that are different from Scenario #1, which are the P-Router and the Provider Edge (PE) roles. The Border PE remains the same for both scenarios.

P-Router 

hostname P131-N9336FX2
install feature-set mpls

feature-set mpls

feature bgp
feature mpls l3vpn
feature mpls segment-routing

mpls label range 16000 23999 static 6000 8000

segment-routing
  mpls
    connected-prefix-sid-map
      address-family ipv4
        10.131.0.131/32 index 131

route-map RM_NH_UNCH permit 10
  set ip next-hop unchanged

interface Ethernet1/1
  description TO_BORDER-PE
  ip address 10.51.131.131/24
  mpls ip forwarding
  no shutdown

interface Ethernet1/11
  description TO_PE
  ip address 10.52.131.131/24
  mpls ip forwarding
  no shutdown

interface loopback0
  description ROUTER-ID & SR-LOOPBACK
  ip address 10.131.0.131/32
  ip router isis 10

router bgp 65013
  event-history detail
  address-family ipv4 unicast
    network 10.131.0.131/32
    allocate-label all
!
  address-family vpnv4 unicast
    retain route-target all
  address-family vpnv6 unicast
    retain route-target all
!
  neighbor 10.51.131.51
    remote-as 65095
    update-source Ethernet1/1
    address-family ipv4 labeled-unicast
      send-community
      send-community extended
!
  neighbor 10.51.0.51
    remote-as 65095
    update-source loopback0
    ebgp-multihop 5
    address-family vpnv4 unicast
      send-community
      send-community extended
      route-map RM_NH_UNCH out
    address-family vpnv6 unicast
      send-community
      send-community extended
      route-map RM_NH_UNCH out
!
  neighbor 10.52.131.52
    remote-as 65003
    update-source Ethernet1/11
    address-family ipv4 labeled-unicast
      send-community
      send-community extended
!
  neighbor 10.52.0.52
    remote-as 65003
    update-source loopback0
    ebgp-multihop 5
    address-family vpnv4 unicast
      send-community
      send-community extended
      route-map RM_NH_UNCH out
    address-family vpnv6 unicast
      send-community
      send-community extended
      route-map RM_NH_UNCH out

Provider Edge (PE) 

hostname L52-N93240FX2
install feature-set mpls

feature-set mpls

feature bgp
feature mpls l3vpn
feature mpls segment-routing

mpls label range 16000 23999 static 6000 8000

segment-routing
  mpls
    connected-prefix-sid-map
      address-family ipv4
        10.52.0.52/32 index 52

vrf context VRF_A
  rd auto
  address-family ipv4 unicast
    route-target import 50000:50000
    route-target export 50000:50000
  address-family ipv6 unicast
    route-target import 50000:50000
    route-target export 50000:50000

interface Ethernet1/49
  description TO_P-ROUTER
  ip address 10.52.131.52/24
  mpls ip forwarding
  no shutdown

interface loopback0
  description ROUTER-ID & SR-LOOPBACK
  ip address 10.52.0.52/32
  ip router isis 10

router bgp 65003
  address-family ipv4 unicast
    network 10.52.0.52/32
    allocate-label all
!
  neighbor 10.52.131.131
    remote-as 65013
    update-source Ethernet1/49
    address-family ipv4 labeled-unicast
      send-community
      send-community extended
!
  neighbor 10.131.0.131
    remote-as 65013
    update-source loopback0
    ebgp-multihop 5
    address-family vpnv4 unicast
      send-community
      send-community extended
    address-family vpnv6 unicast
      send-community
      send-community extended
!
  vrf VRF_A
    address-family ipv4 unicast
      redistribute direct route-map fabric-rmap-redist-subnet