Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Conventions

Topology

Troubleshooting

Initial Information

Verification

Cisco IOS XE Verification Commands

Cisco IOS XR Verification Commands

Related Information

Introduction

This document describes the process to verify end-to-end connectivity across an MPLS Layer 3 VPN core network.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Knowledge of Basic IP Routing

- Knowledge of Cisco IOS® XE and Cisco IOS® XR command line

Components Used

The information in this document is based on these software and hardware versions:

- Router with Cisco IOS XR software

- Router with Cisco IOS XE software

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The purpose of this document is to demonstrate the basic verification and troubleshooting steps to check the connectivity and forwarding between two CE (Customer Edge) routers interconnected with BGP (Border Gateway Protocol) by an MPLS Layer 3 VPN core network with a mix of Cisco IOS XE and Cisco IOS XR routers acting as PE (Provider Edge) and P (Provider) routers.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Topology

MPLS Topology Diagram

Troubleshooting

Initial Information

Source Network: 192.168.1.0/24

Source CE Router: CE-EAST

Destination Network: 172.16.1.0/24

Destination CE Router: CE-WEST

Based on the initial information and topology, reachability must be successful between source address 192.168.1.10 represented by Loopback1 on router CE-EAST and destination address 172.16.1.10 represented by Loopback1 on router CE-WEST:

CE-EAST#show run interface loopback1

Building configuration...

Current configuration : 66 bytes

!

interface Loopback1

ip address 192.168.1.10 255.255.255.0

end

CE-WEST#show run interface loopback 1

Building configuration...

Current configuration : 65 bytes

!

interface Loopback1

ip address 172.16.1.10 255.255.255.0

end

The ICMP reachability and a traceroute were used to start checking the connectivity between these source and destination addresses, however from the next outputs it can be seen that this was not successful:

CE-EAST#ping 172.16.1.10 source loopback1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.1.10, timeout is 2 seconds:

Packet sent with a source address of 192.168.1.10

.....

Success rate is 0 percent (0/5)

CE-EAST#traceroute 172.16.1.10 source loop1 probe 1 numeric

Type escape sequence to abort.

Tracing the route to 172.16.1.10

VRF info: (vrf in name/id, vrf out name/id)

1 10.11.0.2 2 msec

2  *

3 10.10.0.2 [MPLS: Label 16 Exp 0] 9 msec

4  *

5  *

6  *

7  *

8  *

9  *

10  *

11  *

12  *

13  *

14  *

15  *

16  *

17  *

18  *

19  *

20  *

21  *

22  *

23  *

24  *

25  *

26  *

27  *

28  *

29  *

30  *

CE-EAST#

Note: While troubleshooting, the use of a traceroute when connected to an MPLS network can be less effective as some service providers tend to configure the no mpls ip propagate-ttl forward command in Cisco IOS XE or mpls ip-ttl-propagate disable forwarded command in Cisco IOS XR to hide all LSRs (Label Switch Router) in the core (this however with the exception of the ingress and egress PE routers).

While reviewing the status of the source CE router, as this router does not have any VRF (Virtual Route Forwarding) and is not MPLS aware, you need to verify the RIB (Routing Information Base), CEF (Cisco Express Forwarding) and BGP. At the next outputs, it can be observed that there is a routing entry known via BGP to the destination subnet 172.16.1.0/24 and is reachable via interface GigabitEthernet0/0:

CE-EAST#show ip route 172.16.1.10

Routing entry for 172.16.1.0/24

Known via "bgp 65001", distance 20, metric 0          <<<<<

Tag 65500, type external

Last update from 10.11.0.2 3d01h ago

Routing Descriptor Blocks:

* 10.11.0.2, from 10.11.0.2, 3d01h ago

Route metric is 0, traffic share count is 1

AS Hops 2

Route tag 65500

MPLS label: none

CE-EAST#show ip cef 172.16.1.10

172.16.1.0/24

nexthop 10.11.0.2 GigabitEthernet0/0          <<<<<

CE-EAST#

As the source CE-EAST router has a route to the destination installed in the RIB, its time to look at the provider edge router PE4 (ingress PE), as shown in the topology. At this point, the VRF and route distinguishers are configured as well as the route target import and export, as seen on the next outputs:

RP/0/0/CPU0:PE4#show run vrf EAST

Mon Sep 11 20:01:54.454 UTC

vrf EAST

address-family ipv4 unicast

import route-target

65000:1

65001:1

65001:2

!

export route-target

65001:1

!

!

!

RP/0/0/CPU0:PE4#show run router bgp

Mon Sep 11 20:06:48.164 UTC

router bgp 65500

address-family ipv4 unicast

!

address-family vpnv4 unicast

!

neighbor 10.10.10.6

remote-as 65500

update-source Loopback0

address-family vpnv4 unicast

!

!

vrf EAST

rd 65001:1

address-family ipv4 unicast

!

neighbor 10.11.0.1

remote-as 65001

address-family ipv4 unicast

route-policy PASS in

route-policy PASS out

!

!

!

!

RP/0/0/CPU0:PE4#

From the previous output it can be seen that the VRF name "EAST" was defined with the route-target import for 65000:1, now the VRF routing table can be checked and this helps to determine if PE4 has a route to the destination IP address 172.16.1.10:

RP/0/0/CPU0:PE4#show route vrf EAST 172.16.1.10

Mon Sep 11 19:58:28.128 UTC

Routing entry for 172.16.1.0/24

Known via "bgp 65500", distance 200, metric 0

Tag 65000, type internal

Installed Sep  8 18:28:46.303 for 3d01h

Routing Descriptor Blocks

10.10.10.1, from 10.10.10.6

Nexthop in Vrf: "default", Table: "default", IPv4 Unicast, Table Id: 0xe0000000

Route metric is 0

No advertising protos.

RP/0/0/CPU0:PE4#

As this PE is a Cisco IOS XR device, the "detail" keyword can be used at the end of the show route vrf <name> command to look at some additional information such as the VPNv4 label imposed by MP-BGP (Multiprotocol BGP) and the source RD (Route Distinguisher) from the prefix:

RP/0/0/CPU0:PE4#show route vrf EAST 172.16.1.10 detail

Mon Sep 11 20:21:48.492 UTC

Routing entry for 172.16.1.0/24

Known via "bgp 65500", distance 200, metric 0

Tag 65000, type internal

Installed Sep  8 18:28:46.303 for 3d01h

Routing Descriptor Blocks

10.10.10.1, from 10.10.10.6

Nexthop in Vrf: "default", Table: "default", IPv4 Unicast, Table Id: 0xe0000000

Route metric is 0

Label: 0x10 (16)          <<<<<

Tunnel ID: None

Binding Label: None

Extended communities count: 0

Source RD attributes: 0x0000:65000:1          <<<<<

NHID:0x0(Ref:0)

Route version is 0x5 (5)

No local label

IP Precedence: Not Set

QoS Group ID: Not Set

Flow-tag: Not Set

Fwd-class: Not Set

Route Priority: RIB_PRIORITY_RECURSIVE (12) SVD Type RIB_SVD_TYPE_REMOTE

Download Priority 3, Download Version 36

No advertising protos.

RP/0/0/CPU0:PE4#

Now, lets take a look at the BGP VPNv4 prefix that was imported into the VRF, observe that this is the same label 16 from the previous output and it also has the extended community 65000:1. Also it is important to notice that 10.10.10.1 is the next-hop that PE4 needs to be able to perform a route recursion to it, the next address "from 10.10.10.6" is the BGP peer that PE4 used to learn this prefix (in this scenario it is the Route Reflector P6):

RP/0/0/CPU0:PE4#show bgp vpnv4 unicast vrf EAST 172.16.1.10

Mon Sep 11 22:42:28.114 UTC

BGP routing table entry for 172.16.1.0/24, Route Distinguisher: 65001:1

Versions:

Process           bRIB/RIB  SendTblVer

Speaker                 48          48

Last Modified: Sep  8 18:28:46.314 for 3d04h

Paths: (1 available, best #1)

Not advertised to any peer

Path #1: Received by speaker 0

Not advertised to any peer

65000

10.10.10.1 (metric 20) from 10.10.10.6 (10.10.10.1)         <<<<<

Received Label 16

Origin IGP, metric 0, localpref 100, valid, internal, best, group-best, import-candidate, imported

Received Path ID 0, Local Path ID 0, version 48

Extended community: RT:65000:1         <<<<<

Originator: 10.10.10.1, Cluster list: 10.10.10.6

Source AFI: VPNv4 Unicast, Source VRF: default, Source Route Distinguisher: 65000:1  <<<<<

By reviewing CEF with the exact-route keyword at the VRF level you can get an idea of the exit interface for the packets. This command can also give some important details, as it shows the two labels imposed to the prefix, 24001 and 16, the reason being that label 16 is coming from BGP VPNv4 and label 24001 is coming from LDP (Label Distribution Protocol):

RP/0/0/CPU0:PE4#show cef vrf EAST exact-route 192.168.1.10 172.16.1.10

Mon Sep 11 22:48:15.241 UTC

172.16.1.0/24, version 36, internal 0x5000001 0x0 (ptr 0xa12dc74c) [1], 0x0 (0x0), 0x208 (0xa155b1b8)

Updated Sep  8 18:28:46.323

local adjacency 10.0.0.16

Prefix Len 24, traffic index 0, precedence n/a, priority 3

via GigabitEthernet0/0/0/4

via 10.10.10.1/32, 3 dependencies, recursive [flags 0x6000]

path-idx 0 NHID 0x0 [0xa15c3f54 0x0]

recursion-via-/32

next hop VRF - 'default', table - 0xe0000000

next hop 10.10.10.1/32 via 24010/0/21

next hop 10.0.0.16/32 Gi0/0/0/4    labels imposed {24001 16}         <<<<<

As a next step, the show bgp vpnv4 unicast command is used to check the VPNv4 routes that are being learned by this PE. This output shows the information before the VPNv4 prefix is imported into the VRF, remember that the RT (Route Target) configured (for this example the imported RTs are 65000:1, 65001:1, 65001:2) indicates what routes and to which VRF are imported:

RP/0/0/CPU0:PE4#show bgp vpnv4 unicast

Fri Sep 15 02:15:15.463 UTC

BGP router identifier 10.10.10.4, local AS number 65500

BGP generic scan interval 60 secs

Non-stop routing is enabled

BGP table state: Active

Table ID: 0x0   RD version: 0

BGP main routing table version 85

BGP NSR Initial initsync version 1 (Reached)

BGP NSR/ISSU Sync-Group versions 0/0

BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best

i - internal, r RIB-failure, S stale, N Nexthop-discard

Origin codes: i - IGP, e - EGP, ? - incomplete

Network            Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 65000:1

*>i172.16.1.0/24      10.10.10.1               0    100      0 65000 i         <<<<<

*>i172.16.2.0/24      10.10.10.1               0    100      0 65000 i

Route Distinguisher: 65001:1 (default for vrf EAST)

* i0.0.0.0/0          10.10.10.3               0    100      0 65001 i

*>                    10.11.0.1                0             0 65001 i

*>i172.16.1.0/24      10.10.10.1               0    100      0 65000 i

*>i172.16.2.0/24      10.10.10.1               0    100      0 65000 i

*> 192.168.1.0/24     10.11.0.1                0             0 65001 i

*>i192.168.2.0/24     10.10.10.3               0    100      0 65001 i

*> 192.168.3.0/24     10.11.0.1                0             0 65001 i

Route Distinguisher: 65001:2

*>i0.0.0.0/0          10.10.10.3               0    100      0 65001 i

*>i192.168.2.0/24     10.10.10.3               0    100      0 65001 i

Processed 10 prefixes, 11 paths

In this example, the VPNv4 table can be small, but in a production environment instead of looking at all the VPNv4 prefixes, you can narrow the verification down to the specific RD and prefix with the next command:

RP/0/0/CPU0:PE4#show bgp vpnv4 unicast rd 65000:1 172.16.1.10

Mon Sep 11 22:54:04.967 UTC

BGP routing table entry for 172.16.1.0/24, Route Distinguisher: 65000:1

Versions:

Process           bRIB/RIB  SendTblVer

Speaker                 46          46

Last Modified: Sep  8 18:28:46.314 for 3d04h

Paths: (1 available, best #1)

Not advertised to any peer

Path #1: Received by speaker 0

Not advertised to any peer

65000

10.10.10.1 (metric 20) from 10.10.10.6 (10.10.10.1)

Received Label 16

Origin IGP, metric 0, localpref 100, valid, internal, best, group-best, import-candidate, not-in-vrf

Received Path ID 0, Local Path ID 0, version 46

Extended community: RT:65000:1

Originator: 10.10.10.1, Cluster list: 10.10.10.6

At this point, the MP-BGP control plane has the destination prefix and the LDP and VPNv4 labels {24001 16} respectively, the exit interface for this traffic seems to be Gi0/0/0/4 and the next hop where the traffic needs to be forwarded is 10.10.10.1. But, Is there another option to verify the preferred exit interface? It is time to take a look at the MPLS forwarding table or LFIB (Label Forwarding Information Base). With the use of command show mpls forwarding two entries are displayed towards 10.10.10.1 destination (Loopbback0 from PE1), one path with an Outgoing Interface of Gi0/0/0/4 and a next-hop 10.0.0.16 (router P5) where the imposed Outgoing Label is 24001 and another path through Gi0/0/0/3 with a next-hop 10.0.0.13 (router P6) and an Outgoing Label of 23:

RP/0/0/CPU0:PE4#show mpls forwarding

Mon Sep 11 23:28:33.425 UTC

Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes

Label  Label       or ID              Interface                    Switched

------ ----------- ------------------ ------------ --------------- ------------

24000  Unlabelled  192.168.1.0/24[V]  Gi0/0/0/0    10.11.0.1       1096

24001  Unlabelled  192.168.3.0/24[V]  Gi0/0/0/0    10.11.0.1       56056

24002  Unlabelled  0.0.0.0/0[V]       Gi0/0/0/0    10.11.0.1       0

24003  Pop         10.10.10.6/32      Gi0/0/0/3    10.0.0.13       7778512

24004  Pop         10.0.0.4/31        Gi0/0/0/3    10.0.0.13       0

24005  Pop         10.0.0.8/31        Gi0/0/0/3    10.0.0.13       0

24006  Pop         10.10.10.5/32      Gi0/0/0/4    10.0.0.16       3542574

24007  Pop         10.0.0.10/31       Gi0/0/0/3    10.0.0.13       0

Pop         10.0.0.10/31       Gi0/0/0/4    10.0.0.16       0

24008  Pop         10.0.0.6/31        Gi0/0/0/4    10.0.0.16       0

24009  Pop         10.0.0.0/31        Gi0/0/0/4    10.0.0.16       0

24010  23          10.10.10.1/32      Gi0/0/0/3    10.0.0.13       22316         <<<<<

24001       10.10.10.1/32      Gi0/0/0/4    10.0.0.16       42308         <<<<<

24011  18          10.10.10.2/32      Gi0/0/0/3    10.0.0.13       0

24003       10.10.10.2/32      Gi0/0/0/4    10.0.0.16       0

24012  17          10.0.0.2/31        Gi0/0/0/3    10.0.0.13       0

24005       10.0.0.2/31        Gi0/0/0/4    10.0.0.16       0

24013  Pop         10.10.10.3/32      Gi0/0/0/1    10.0.0.20       3553900

24014  Pop         10.0.0.14/31       Gi0/0/0/1    10.0.0.20       0

Pop         10.0.0.14/31       Gi0/0/0/4    10.0.0.16       0

24015  Pop         10.0.0.18/31       Gi0/0/0/1    10.0.0.20       0

Pop         10.0.0.18/31       Gi0/0/0/3    10.0.0.13       0

RP/0/0/CPU0:PE4#show mpls forwarding prefix 10.10.10.1/32

Mon Sep 11 23:30:54.685 UTC

Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes

Label  Label       or ID              Interface                    Switched

------ ----------- ------------------ ------------ --------------- ------------

24010  23          10.10.10.1/32      Gi0/0/0/3    10.0.0.13       3188

24001       10.10.10.1/32      Gi0/0/0/4    10.0.0.16       6044

RP/0/0/CPU0:PE4#show mpls forwarding prefix 10.10.10.1/32 detail hardware egress

Mon Sep 11 23:36:06.504 UTC

Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes

Label  Label       or ID              Interface                    Switched

------ ----------- ------------------ ------------ --------------- ------------

24010  23          10.10.10.1/32      Gi0/0/0/3    10.0.0.13       N/A

Updated: Sep  8 20:27:26.596

Version: 39, Priority: 3

Label Stack (Top -> Bottom): { 23 }

NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0

MAC/Encaps: 14/18, MTU: 1500

Outgoing Interface: GigabitEthernet0/0/0/3 (ifhandle 0x000000a0)

Packets Switched: 0

24001       10.10.10.1/32      Gi0/0/0/4    10.0.0.16       N/A

Updated: Sep  8 20:27:26.596

Version: 39, Priority: 3

Label Stack (Top -> Bottom): { 24001 }

NHID: 0x0, Encap-ID: N/A, Path idx: 1, Backup path idx: 0, Weight: 0

MAC/Encaps: 14/18, MTU: 1500

Outgoing Interface: GigabitEthernet0/0/0/4 (ifhandle 0x000000c0)

Packets Switched: 0

From the previous outputs it is clear that there are two path options where the traffic can be load balanced, but there are a couple of ways that can help to determine which one is the preferred path. One way is with the use of the show cef exact-route <source IP> <destination IP> command by adding the Loopback0 from Source PE and Loopback0 from Destination PE. As shown in the next output, the preferred path is through Gi0/0/0/4:

RP/0/0/CPU0:PE4#show cef exact-route 10.10.10.4 10.10.10.1

Mon Sep 11 23:49:44.558 UTC

10.10.10.1/32, version 39, internal 0x1000001 0x0 (ptr 0xa12dbdbc) [1], 0x0 (0xa12c18c0), 0xa28 (0xa185307c)

Updated Sep  8 20:27:26.596

local adjacency 10.0.0.16

Prefix Len 32, traffic index 0, precedence n/a, priority 3

via GigabitEthernet0/0/0/4

via 10.0.0.16/32, GigabitEthernet0/0/0/4, 9 dependencies, weight 0, class 0 [flags 0x0]         <<<<<

path-idx 1 NHID 0x0 [0xa16765bc 0x0]

next hop 10.0.0.16/32

local adjacency

local label 24010      labels imposed {24001}

Another option is to first verify the LIB (Label Information Base) and get the LDP binding of the destination Loopback0 (10.10.10.1 that belongs to the egress PE) by using the command show mpls ldp bindings <prefix/mask>, and once that the local binding label is found from that output, use that label value in the command show mpls forwarding exact-route label <label> ipv4 <source IP> <destination IP> detail to find the preferred path:

RP/0/0/CPU0:PE4#show mpls ldp bindings 10.10.10.1/32

Wed Sep 13 17:18:43.007 UTC

10.10.10.1/32, rev 29

Local binding: label: 24010         <<<<<

Remote bindings: (3 peers)

Peer                Label

-----------------   ---------

10.10.10.3:0        24

10.10.10.5:0        24001

10.10.10.6:0        23

RP/0/0/CPU0:PE4#show mpls forwarding exact-route label 24010 ipv4 10.10.10.4 10.10.10.1 detail

Wed Sep 13 17:20:06.342 UTC

Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes

Label  Label       or ID              Interface                    Switched

------ ----------- ------------------ ------------ --------------- ------------

24010  24001       10.10.10.1/32      Gi0/0/0/4    10.0.0.16       N/A                  <<<<<

Updated: Sep 12 14:15:37.009

Version: 198, Priority: 3

Label Stack (Top -> Bottom): { 24001 }

NHID: 0x0, Encap-ID: N/A, Path idx: 1, Backup path idx: 0, Weight: 0

Hash idx: 1

MAC/Encaps: 14/18, MTU: 1500

Outgoing Interface: GigabitEthernet0/0/0/4 (ifhandle 0x000000c0)

Packets Switched: 0

Via: Gi0/0/0/4, Next Hop: 10.0.0.16

Label Stack (Top -> Bottom): { 24001 }

NHID: 0x0, Encap-ID: N/A, Path idx: 1, Backup path idx: 0, Weight: 0

Hash idx: 1

MAC/Encaps: 14/18, MTU: 1500

Outgoing Interface: GigabitEthernet0/0/0/4 (ifhandle 0x000000c0)

Next, it is important to check the next hop router that is in the dataplane, for this particular example the router to verify is P5 (that has the interface 10.0.0.16). The first place to start looking into is the MPLS forwarding table, where the Local Label for prefix 10.10.10.1 must be 24001:

RP/0/0/CPU0:P5#show mpls forwarding

Thu Sep 14 20:07:16.455 UTC

Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes

Label  Label       or ID              Interface                    Switched

------ ----------- ------------------ ------------ --------------- ------------

24000  Pop         10.10.10.6/32      Gi0/0/0/2    10.0.0.11       361906

24001  Pop         10.10.10.1/32      Gi0/0/0/1    10.0.0.0        361002               <<<<<

24002  Pop         10.0.0.4/31        Gi0/0/0/1    10.0.0.0        0

Pop         10.0.0.4/31        Gi0/0/0/2    10.0.0.11       0

24003  Pop         10.10.10.2/32      Gi0/0/0/0    10.0.0.6        360940

24004  Pop         10.0.0.8/31        Gi0/0/0/0    10.0.0.6        0

Pop         10.0.0.8/31        Gi0/0/0/2    10.0.0.11       0

24005  Pop         10.0.0.2/31        Gi0/0/0/0    10.0.0.6        0

Pop         10.0.0.2/31        Gi0/0/0/1    10.0.0.0        0

24006  Pop         10.10.10.4/32      Gi0/0/0/4    10.0.0.17       361230

24007  Pop         10.0.0.12/31       Gi0/0/0/2    10.0.0.11       0

Pop         10.0.0.12/31       Gi0/0/0/4    10.0.0.17       0

24008  Pop         10.10.10.3/32      Gi0/0/0/3    10.0.0.15       361346

24009  Pop         10.0.0.20/31       Gi0/0/0/3    10.0.0.15       0

Pop         10.0.0.20/31       Gi0/0/0/4    10.0.0.17       0

24010  Pop         10.0.0.18/31       Gi0/0/0/2    10.0.0.11       0

Pop         10.0.0.18/31       Gi0/0/0/3    10.0.0.15       0

RP/0/0/CPU0:P5#show mpls forwarding labels 24001

Thu Sep 14 20:07:42.584 UTC

Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes

Label  Label       or ID              Interface                    Switched

------ ----------- ------------------ ------------ --------------- ------------

24001  Pop         10.10.10.1/32      Gi0/0/0/1    10.0.0.0        361060

RP/0/0/CPU0:P5#

From the previous output it can be seen that the LFIB entry for prefix 10.10.10.1/32 shows "Pop" as the outgoing label, meaning that this router is the Penultimate Hop Popping (PHP). It also shows that traffic must be sent through Gi0/0/0/1 based on LFIB information, and this can be verified as well while looking at CEF. The next CEF exact-route output shows the Implicit Null Label as the imposed label, this again, is due to the fact that the next-hop connected at Gi0/0/0/1 is the last router in the label switch path and is also the PE facing the destination site (CE-WEST). This is also the reason why router P5 is removing and not imposing another label to the packet, thanks to this process the egress router PE1 is going to receive a packet without an LDP label:

RP/0/0/CPU0:P5#show cef exact-route 10.10.10.4 10.10.10.1

Thu Sep 14 20:25:57.269 UTC

10.10.10.1/32, version 192, internal 0x1000001 0x0 (ptr 0xa1246394) [1], 0x0 (0xa122b638), 0xa20 (0xa155b550)

Updated Sep 12 14:15:38.009

local adjacency 10.0.0.0

Prefix Len 32, traffic index 0, precedence n/a, priority 3

via GigabitEthernet0/0/0/1

via 10.0.0.0/32, GigabitEthernet0/0/0/1, 9 dependencies, weight 0, class 0 [flags 0x0]

path-idx 0 NHID 0x0 [0xa166e280 0xa166e674]

next hop 10.0.0.0/32

local adjacency

local label 24001      labels imposed {ImplNull}         <<<<<

The last point to verify the label switch path is PE1. While looking at the MPLS forwarding table it can be noticed that there is no entry for the prefix 10.10.10.1/32 in the LFIB:

PE1#show mpls forwarding-table

Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop

Label      Label      or Tunnel Id     Switched      interface

16         No Label   172.16.1.0/24[V] 12938         Gi3        10.10.0.1

17         No Label   172.16.2.0/24[V] 0             Gi3        10.10.0.1

18         Pop Label  10.0.0.6/31      0             Gi1        10.0.0.1

Pop Label  10.0.0.6/31      0             Gi2        10.0.0.3

19         Pop Label  10.0.0.8/31      0             Gi2        10.0.0.3

Pop Label  10.0.0.8/31      0             Gi4        10.0.0.5

20         Pop Label  10.0.0.10/31     0             Gi1        10.0.0.1

Pop Label  10.0.0.10/31     0             Gi4        10.0.0.5

21         Pop Label  10.0.0.12/31     0             Gi4        10.0.0.5

22         Pop Label  10.0.0.14/31     0             Gi1        10.0.0.1

23         Pop Label  10.0.0.16/31     0             Gi1        10.0.0.1

24         Pop Label  10.0.0.18/31     0             Gi4        10.0.0.5

25         24009      10.0.0.20/31     0             Gi1        10.0.0.1

22         10.0.0.20/31     0             Gi4        10.0.0.5

26         Pop Label  10.10.10.2/32    0             Gi2        10.0.0.3

27         24008      10.10.10.3/32    0             Gi1        10.0.0.1

24         10.10.10.3/32    0             Gi4        10.0.0.5

28         24006      10.10.10.4/32    0             Gi1        10.0.0.1

25         10.10.10.4/32    0             Gi4        10.0.0.5

29         Pop Label  10.10.10.5/32    0             Gi1        10.0.0.1

Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop

Label      Label      or Tunnel Id     Switched      interface

30         Pop Label  10.10.10.6/32    0             Gi4        10.0.0.5

31    [T]  Pop Label  1/1[TE-Bind]     0             drop

[T]     Forwarding through a LSP tunnel.

View additional labelling info with the 'detail' option

As you have figured out, the reason of this behavior is that prefix (10.10.10.1/32) belongs to PE1, and router has also assigned an implicit null label to this connected prefix. This can be verified with the use of the show mpls ldp bindings command:

PE1#show run interface loopback 0

Building configuration...

Current configuration : 66 bytes

!

interface Loopback0

ip address 10.10.10.1 255.255.255.255

end

PE1#show mpls ldp bindings 10.10.10.1 32

lib entry: 10.10.10.1/32, rev 24

local binding:  label: imp-null

remote binding: lsr: 10.10.10.6:0, label: 23

remote binding: lsr: 10.10.10.5:0, label: 24001

remote binding: lsr: 10.10.10.2:0, label: 24000

As PE1 is a Cisco IOS XE router, the use of the command show bgp vpnv4 unicast all or show bgp vpnv4 unicast rd <value> <destination IP> can help to identify and confirm that the destination prefix 172.16.1.0/24 is being learned correctly through MP-BGP. The output of these commands shows the prefix after being exported:

PE1#show bgp vpnv4 unicast all

BGP table version is 61, local router ID is 10.10.10.1

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,

x best-external, a additional-path, c RIB-compressed,

t secondary path, L long-lived-stale,

Origin codes: i - IGP, e - EGP, ? - incomplete

RPKI validation codes: V valid, I invalid, N Not found

Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 65000:1 (default for vrf WEST)

*>i  0.0.0.0          10.10.10.3               0    100      0 65001 i

*bi                   10.10.10.4               0    100      0 65001 i

*>   172.16.1.0/24    10.10.0.1                0             0 65000 i         <<<<<

*>   172.16.2.0/24    10.10.0.1                0             0 65000 i

*>i  192.168.1.0      10.10.10.4               0    100      0 65001 i

*>i  192.168.2.0      10.10.10.3               0    100      0 65001 i

*>i  192.168.3.0      10.10.10.4               0    100      0 65001 i

Route Distinguisher: 65001:1

*>i  0.0.0.0          10.10.10.4               0    100      0 65001 i

*>i  192.168.1.0      10.10.10.4               0    100      0 65001 i

*>i  192.168.3.0      10.10.10.4               0    100      0 65001 i

Route Distinguisher: 65001:2

Network          Next Hop            Metric LocPrf Weight Path

*>i  0.0.0.0          10.10.10.3               0    100      0 65001 i

*>i  192.168.2.0      10.10.10.3               0    100      0 65001 i

PE1#show bgp vpnv4 unicast rd 65000:1 172.16.1.10

BGP routing table entry for 65000:1:172.16.1.0/24, version 2

Paths: (1 available, best #1, table WEST)

Additional-path-install

Advertised to update-groups:

6

Refresh Epoch 2

65000

10.10.0.1 (via vrf WEST) from 10.10.0.1 (172.16.2.10)         <<<<<

Origin IGP, metric 0, localpref 100, valid, external, best

Extended Community: RT:65000:1 , recursive-via-connected         <<<<<

mpls labels in/out 16/nolabel

rx pathid: 0, tx pathid: 0x0

Updated on Sep 15 2023 18:27:23 UTC

In a similar way, looking at the BGP VPNv4 prefix at the VRF which is the prefix received by CE-WEST, with the use of the command show bgp vpnv4 unicast vrf <name> <prefix>, the output shows the MP-BGP label 16 that was carried all the way to the ingress PE4 as well as the RT export configured 65000:1:

PE1#show bgp vpnv4 unicast vrf WEST 172.16.1.10

BGP routing table entry for 65000:1:172.16.1.0/24, version 2

Paths: (1 available, best #1, table WEST)

Additional-path-install

Advertised to update-groups:

6

Refresh Epoch 2

65000

10.10.0.1 (via vrf WEST) from 10.10.0.1 (172.16.2.10)

Origin IGP, metric 0, localpref 100, valid, external, best

Extended Community: RT:65000:1 , recursive-via-connected         <<<<<

mpls labels in/out 16/nolabel         <<<<<

rx pathid: 0, tx pathid: 0x0

Updated on Sep 15 2023 18:27:23 UTC

PE1#show run vrf WEST

Building configuration...

Current configuration : 478 bytes

vrf definition WEST

rd 65000:1

route-target export 65000:1         <<<<<

route-target import 65000:1

route-target import 65001:1

route-target import 65001:2

!

address-family ipv4

exit-address-family

!

!

interface GigabitEthernet3

vrf forwarding WEST

ip address 10.10.0.2 255.255.255.252

negotiation auto

no mop enabled

no mop sysid

!

router bgp 65500

!

address-family ipv4 vrf WEST

neighbor 10.10.0.1 remote-as 65000

neighbor 10.10.0.1 activate

exit-address-family

!

end

The last information to check at this PE is the RIB and CEF entries at the VRF level to the destination IP, as opposed to the entry seen at PE4 there is no Label on the RIB for prefix 172.16.1.0/24, the reason is that this is the route coming in from the CE and this is learned via eBGP and inserted into the VRF routing table before this prefix is being exported into VPNv4. This can be verified with the use of the commands show ip route vrf <name> <prefix> and show ip cef vrf <name> <prefix> shown next:

PE1#show ip route vrf WEST 172.16.1.10

Routing Table: WEST

Routing entry for 172.16.1.0/24

Known via "bgp 65500", distance 20, metric 0

Tag 65000, type external

Last update from 10.10.0.1 1w0d ago

Routing Descriptor Blocks:

* 10.10.0.1, from 10.10.0.1, 1w0d ago, recursive-via-conn

opaque_ptr 0x7F8B4E3E1D50

Route metric is 0, traffic share count is 1

AS Hops 1

Route tag 65000

MPLS label: none

PE1#show ip cef vrf WEST 172.16.1.10

172.16.1.0/24

nexthop 10.10.0.1 GigabitEthernet3

At this point, it has been confirmed that the destination prefix 172.16.1.0/24 was learned correctly by the source of the traffic CE (CE-EAST), it was correctly propagated through MP-BGP and also the labels from PEs and Ps loopbacks were learned across the label switch path. But still, the reachability between source/destination is not successful, and there is still one last router to verify CE-WEST. The first thing to check in this router is the routing table, remember that the source IP prefix 192.168.1.0/24 is expected to appear in there:

CE-WEST#show ip route 192.168.1.10

% Network not in table

CE-WEST#

The "Network not in table" is clearly the problem, the BGP table can also be verified but after looking for the prefix it is not there either:

CE-WEST#show ip bgp

BGP table version is 41, local router ID is 172.16.2.10

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,

x best-external, a additional-path, c RIB-compressed,

t secondary path,

Origin codes: i - IGP, e - EGP, ? - incomplete

RPKI validation codes: V valid, I invalid, N Not found

Network          Next Hop            Metric LocPrf Weight Path

*>   172.16.1.0/24    0.0.0.0                  0         32768 i

*>   172.16.2.0/24    0.0.0.0                  0         32768 i

CE-WEST#

Going a step back, you can verify if this provider edge router (PE1) is advertising the prefix to the eBGP neighbor CE-WEST, this can be done with the use of the command show bgp vpnv4 unicast vrf <name> neighbors <neighbor IP> advertised-routes shown next:

PE1#show bgp vpnv4 unicast vrf WEST neighbors 10.10.0.1 advertised-routes

BGP table version is 61, local router ID is 10.10.10.1

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,

x best-external, a additional-path, c RIB-compressed,

t secondary path, L long-lived-stale,

Origin codes: i - IGP, e - EGP, ? - incomplete

RPKI validation codes: V valid, I invalid, N Not found

Network          Next Hop            Metric LocPrf Weight Path

Route Distinguisher: 65000:1 (default for vrf WEST)

*>i  0.0.0.0          10.10.10.3               0    100      0 65001 i

*>i  192.168.1.0      10.10.10.4               0    100      0 65001 i         <<<<<

*>i  192.168.2.0      10.10.10.3               0    100      0 65001 i

*>i  192.168.3.0      10.10.10.4               0    100      0 65001 i

Total number of prefixes 4

Based on the previous step, it can be confirmed that PE1 router is advertising the prefix correctly to CE-WEST, hence it is time to look at the BGP neighbors on the CE side:

CE-WEST#show ip bgp neighbors

BGP neighbor is 10.10.0.2,  remote AS 65500, external link

BGP version 4, remote router ID 10.10.10.1

BGP state = Established, up for 1w4d

Last read 00:00:40, last write 00:00:43, hold time is 180, keepalive interval is 60 seconds

Neighbor sessions:

1 active, is not multisession capable (disabled)

Neighbor capabilities:

Route refresh: advertised and received(new)

Four-octets ASN Capability: advertised and received

Address family IPv4 Unicast: advertised and received

Enhanced Refresh Capability: advertised and received

Multisession Capability:

Stateful switchover support enabled: NO for session 1

Message statistics:

InQ depth is 0

OutQ depth is 0

Sent       Rcvd

Opens:                  1          1

Notifications:          0          0

Updates:                3         17

Keepalives:         19021      18997

Route Refresh:          2          0

Total:              19029      19019

Do log neighbor state changes (via global configuration)

Default minimum time between advertisement runs is 30 seconds

For address family: IPv4 Unicast

Session: 10.10.0.2

BGP table version 41, neighbor version 41/0

Output queue size : 0

Index 3, Advertise bit 0

3 update-group member

Inbound path policy configured

Route map for incoming advertisements is FILTER         <<<<<

Slow-peer detection is disabled

Slow-peer split-update-group dynamic is disabled

Sent       Rcvd

Prefix activity:               ----       ----

Prefixes Current:               2          0

Prefixes Total:                 4         23

Implicit Withdraw:              2         13

Explicit Withdraw:              0         10

Used as bestpath:             n/a          0

Used as multipath:            n/a          0

Used as secondary:            n/a          0

Outbound    Inbound

Local Policy Denied Prefixes:    --------    -------

route-map:                            0          4

Bestpath from this peer:             18        n/a

Total:                               18          4

Number of NLRIs in the update sent: max 2, min 0

Last detected as dynamic slow peer: never

Dynamic slow peer recovered: never

Refresh Epoch: 3

Last Sent Refresh Start-of-rib: 4d23h

Last Sent Refresh End-of-rib: 4d23h

Refresh-Out took 0 seconds

Last Received Refresh Start-of-rib: 4d23h

Last Received Refresh End-of-rib: 4d23h

Refresh-In took 0 seconds

Sent       Rcvd

Refresh activity:              ----       ----

Refresh Start-of-RIB          1          2

Refresh End-of-RIB            1          2

Address tracking is enabled, the RIB does have a route to 10.10.0.2

Route to peer address reachability Up: 1; Down: 0

Last notification 1w5d

Connections established 3; dropped 2

Last reset 1w4d, due to Peer closed the session of session 1

External BGP neighbor configured for connected checks (single-hop no-disable-connected-check)

Interface associated: GigabitEthernet0/3 (peering address in same link)

Transport(tcp) path-mtu-discovery is enabled

Graceful-Restart is disabled

SSO is disabled

Connection state is ESTAB, I/O status: 1, unread input bytes: 0

Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1

Local host: 10.10.0.1, Local port: 179

Foreign host: 10.10.0.2, Foreign port: 39410

Connection tableid (VRF): 0

Maximum output segment queue size: 50

Enqueued packets for retransmit: 0, input: 0  mis-ordered: 0 (0 bytes)

Event Timers (current time is 0x4D15FD56):

Timer          Starts    Wakeups            Next

Retrans         19027          1             0x0

TimeWait            0          0             0x0

AckHold         19012      18693             0x0

SendWnd             0          0             0x0

KeepAlive           0          0             0x0

GiveUp              0          0             0x0

PmtuAger            0          0             0x0

DeadWait            0          0             0x0

Linger              0          0             0x0

ProcessQ            0          0             0x0

iss: 1676751051  snduna: 1677112739  sndnxt: 1677112739

irs: 2109012892  rcvnxt: 2109374776

sndwnd:  16061  scale:      0  maxrcvwnd:  16384

rcvwnd:  15890  scale:      0  delrcvwnd:    494

SRTT: 1000 ms, RTTO: 1003 ms, RTV: 3 ms, KRTT: 0 ms

minRTT: 0 ms, maxRTT: 1000 ms, ACK hold: 200 ms

uptime: 1036662542 ms, Sent idletime: 40725 ms, Receive idletime: 40925 ms

Status Flags: passive open, gen tcbs

Option Flags: nagle, path mtu capable

IP Precedence value : 6

Datagrams (max data segment is 1460 bytes):

Rcvd: 37957 (out of order: 0), with data: 19014, total data bytes: 361883

Sent: 37971 (retransmit: 1, fastretransmit: 0, partialack: 0, Second Congestion: 0), with data: 19027, total data bytes: 361687

Packets received in fast path: 0, fast processed: 0, slow path: 0

fast lock acquisition failures: 0, slow path: 0

TCP Semaphore      0x0F3194AC  FREE

The previous output reveals that there is a route map applied for incoming advertisements with the name "FILTER", after looking at the route map configuration it shows a match clause pointing to a prefix-list with a permit statement for 192.168.0.0/16, however this is incorrect as the prefix-list is only permitting that specific prefix and not all the ones that can be included in this range:

CE-WEST#show route-map FILTER

route-map FILTER, permit, sequence 10

Match clauses:

ip address prefix-lists: FILTER

Set clauses:

Policy routing matches: 0 packets, 0 bytes

CE-WEST#show ip prefix-list FILTER

ip prefix-list FILTER: 1 entries

seq 5 permit 192.168.0.0/16         <<<<<

CE-WEST#show run | i ip prefix-list

ip prefix-list FILTER seq 5 permit 192.168.0.0/16

With a small change to the prefix-list configuration, the route towards 192.168.1.10 is now installed in the RIB:

CE-WEST#show run | i ip prefix-list

ip prefix-list FILTER seq 5 permit 192.168.0.0/16 le 32         <<<<<

CE-WEST#show ip bgp

BGP table version is 44, local router ID is 172.16.2.10

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,

x best-external, a additional-path, c RIB-compressed,

t secondary path,

Origin codes: i - IGP, e - EGP, ? - incomplete

RPKI validation codes: V valid, I invalid, N Not found

Network          Next Hop            Metric LocPrf Weight Path

*>   172.16.1.0/24    0.0.0.0                  0         32768 i

*>   172.16.2.0/24    0.0.0.0                  0         32768 i

*>   192.168.1.0      10.10.0.2                              0 65500 65001 i         <<<<<

*>   192.168.2.0      10.10.0.2                              0 65500 65001 i

*>   192.168.3.0      10.10.0.2                              0 65500 65001 i

CE-WEST#show ip route 192.168.1.10

Routing entry for 192.168.1.0/24         <<<<<

Known via "bgp 65000", distance 20, metric 0

Tag 65500, type external

Last update from 10.10.0.2 00:00:37 ago

Routing Descriptor Blocks:

* 10.10.0.2, from 10.10.0.2, 00:00:37 ago

Route metric is 0, traffic share count is 1

AS Hops 2

Route tag 65500

MPLS label: none

Verification

Now, the reachability between source and destination is successful and it can be confirmed that the traceroute goes through the same label switch path that was tracked across the MPLS network:

Forwarding Path

CE-EAST#ping 172.16.1.10 source loopback 1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.1.10, timeout is 2 seconds:

Packet sent with a source address of 192.168.1.10

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 7/7/9 ms         <<<<<

CE-EAST#traceroute 172.16.1.10 source loop1 probe 1 numeric

Type escape sequence to abort.

Tracing the route to 172.16.1.10

VRF info: (vrf in name/id, vrf out name/id)

1 10.11.0.2 2 msec

2 10.0.0.16 [MPLS: Labels 24001/16 Exp 0] 9 msec

3 10.10.0.2 [MPLS: Label 16 Exp 0] 8 msec

4 10.10.0.1 9 msec

RP/0/0/CPU0:P5#show ipv4 interface brief

Wed Sep 20 18:23:47.158 UTC

Interface                      IP-Address      Status          Protocol Vrf-Name

Loopback0                      10.10.10.5      Up              Up       default

MgmtEth0/0/CPU0/0              unassigned      Shutdown        Down     default

GigabitEthernet0/0/0/0         10.0.0.7        Up              Up       default

GigabitEthernet0/0/0/1         10.0.0.1        Up              Up       default         <<<<<

GigabitEthernet0/0/0/2         10.0.0.10       Up              Up       default

GigabitEthernet0/0/0/3         10.0.0.14       Up              Up       default

GigabitEthernet0/0/0/4         10.0.0.16       Up              Up       default         <<<<<

RP/0/0/CPU0:P5#

Cisco IOS XE Verification Commands

MPLS/LDP

show mpls interfaces

show mpls forwarding-table

show mpls ldp bindings [destination prefix]

show mpls ldp neighbor [neighbor address]

clear mpls ldp neighbor [neighbor address|*]

RIB and CEF

show ip vrf [detail]

show run vrf

show ip route [destination prefix]

show ip route vrf <name> [destination prefix]

show ip cef vrf <name> [destination prefix]

show ip cef exact-route <source IP> <destination IP>

show ip cef vrf <name> exact-route <source IP> <destination IP>

BGP/VPNv4

show ip bgp [neighbors] <neighbor address>

show bgp vpnv4 unicast all [summary|destination prefix]

show bgp vpnv4 unicast all neighbor <neighbor address> advertised-routes

show bgp vpnv4 unicast vrf <name> neighbors <neighbor IP> advertised-routes

show bgp vpnv4 unicast vrf <name> <prefix>

show bgp vpnv4 unicast rd <value> <destination IP>

Cisco IOS XR Verification Commands

MPLS/LDP

show mpls interfaces

show mpls forwarding

show mpls ldp bindings [destination prefix/mask]

show mpls ldp neighbor [neighbor address]

show mpls forwarding prefix [destination prefix/mask]

show mpls forwarding prefix [destination prefix/mask] detail hardware egress

clear mpls ldp neighbor [neighbor address]

RIB and CEF

show vrf [name|all]

show run vrf [name]

show route [destination prefix]

show route vrf <name> [destination prefix]

show cef vrf <name> [destination prefix]

show cef exact-route <source IP> <destination IP>

show cef vrf <name> exact-route <source IP> <destination IP>

BGP/VPNv4

show bgp vpnv4 unicast [summary|destination prefix/mask]

show bgp vpnv4 unicast neighbors <neighbor address> advertised-routes

show bgp vpnv4 unicast vrf <name> [prefix]

show bgp vrf <name> neighbors <neighbor IP> advertised-routes

show bgp vpnv4 unicast rd [value|all] [destination IP]

Related Information

- MPLS Basic MPLS Configuration Guide

- Configure a Basic MPLS VPN Network

- How to Troubleshoot the MPLS VPN

- Verify End-to-End Connectivity across a Segment Routing SP