Verify MPLS Layer 3 VPN Forwarding
Available Languages
Introduction
This document describes the process to verify end-to-end connectivity across an MPLS Layer 3 VPN core network.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- Knowledge of Basic IP Routing
- Knowledge of Cisco IOS® XE and Cisco IOS® XR command line
Components Used
The information in this document is based on these software and hardware versions:
- Router with Cisco IOS XR software
- Router with Cisco IOS XE software
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
The purpose of this document is to demonstrate the basic verification and troubleshooting steps to check the connectivity and forwarding between two CE (Customer Edge) routers interconnected with BGP (Border Gateway Protocol) by an MPLS Layer 3 VPN core network with a mix of Cisco IOS XE and Cisco IOS XR routers acting as PE (Provider Edge) and P (Provider) routers.
Conventions
For more information on document conventions, refer to the Cisco Technical Tips Conventions.
Topology
MPLS Topology Diagram
Troubleshooting
Initial Information
Source Network: 192.168.1.0/24
Source CE Router: CE-EAST
Destination Network: 172.16.1.0/24
Destination CE Router: CE-WEST
Based on the initial information and topology, reachability must be successful between source address 192.168.1.10 represented by Loopback1 on router CE-EAST and destination address 172.16.1.10 represented by Loopback1 on router CE-WEST:
CE-EAST#show run interface loopback1
Building configuration...
Current configuration : 66 bytes
!
interface Loopback1
ip address 192.168.1.10 255.255.255.0
end
CE-WEST#show run interface loopback 1
Building configuration...
Current configuration : 65 bytes
!
interface Loopback1
ip address 172.16.1.10 255.255.255.0
endThe ICMP reachability and a traceroute were used to start checking the connectivity between these source and destination addresses, however from the next outputs it can be seen that this was not successful:
CE-EAST#ping 172.16.1.10 source loopback1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.1.10, timeout is 2 seconds:
Packet sent with a source address of 192.168.1.10
.....
Success rate is 0 percent (0/5)
CE-EAST#traceroute 172.16.1.10 source loop1 probe 1 numeric
Type escape sequence to abort.
Tracing the route to 172.16.1.10
VRF info: (vrf in name/id, vrf out name/id)
1 10.11.0.2 2 msec
2 *
3 10.10.0.2 [MPLS: Label 16 Exp 0] 9 msec
4 *
5 *
6 *
7 *
8 *
9 *
10 *
11 *
12 *
13 *
14 *
15 *
16 *
17 *
18 *
19 *
20 *
21 *
22 *
23 *
24 *
25 *
26 *
27 *
28 *
29 *
30 *
CE-EAST#
Note: While troubleshooting, the use of a traceroute when connected to an MPLS network can be less effective as some service providers tend to configure the no mpls ip propagate-ttl forward command in Cisco IOS XE or mpls ip-ttl-propagate disable forwarded command in Cisco IOS XR to hide all LSRs (Label Switch Router) in the core (this however with the exception of the ingress and egress PE routers).
While reviewing the status of the source CE router, as this router does not have any VRF (Virtual Route Forwarding) and is not MPLS aware, you need to verify the RIB (Routing Information Base), CEF (Cisco Express Forwarding) and BGP. At the next outputs, it can be observed that there is a routing entry known via BGP to the destination subnet 172.16.1.0/24 and is reachable via interface GigabitEthernet0/0:
CE-EAST#show ip route 172.16.1.10
Routing entry for 172.16.1.0/24
Known via "bgp 65001", distance 20, metric 0 <<<<<
Tag 65500, type external
Last update from 10.11.0.2 3d01h ago
Routing Descriptor Blocks:
* 10.11.0.2, from 10.11.0.2, 3d01h ago
Route metric is 0, traffic share count is 1
AS Hops 2
Route tag 65500
MPLS label: none
CE-EAST#show ip cef 172.16.1.10
172.16.1.0/24
nexthop 10.11.0.2 GigabitEthernet0/0 <<<<<
CE-EAST#As the source CE-EAST router has a route to the destination installed in the RIB, its time to look at the provider edge router PE4 (ingress PE), as shown in the topology. At this point, the VRF and route distinguishers are configured as well as the route target import and export, as seen on the next outputs:
RP/0/0/CPU0:PE4#show run vrf EAST
Mon Sep 11 20:01:54.454 UTC
vrf EAST
address-family ipv4 unicast
import route-target
65000:1
65001:1
65001:2
!
export route-target
65001:1
!
!
!
RP/0/0/CPU0:PE4#show run router bgp
Mon Sep 11 20:06:48.164 UTC
router bgp 65500
address-family ipv4 unicast
!
address-family vpnv4 unicast
!
neighbor 10.10.10.6
remote-as 65500
update-source Loopback0
address-family vpnv4 unicast
!
!
vrf EAST
rd 65001:1
address-family ipv4 unicast
!
neighbor 10.11.0.1
remote-as 65001
address-family ipv4 unicast
route-policy PASS in
route-policy PASS out
!
!
!
!
RP/0/0/CPU0:PE4#From the previous output it can be seen that the VRF name "EAST" was defined with the route-target import for 65000:1, now the VRF routing table can be checked and this helps to determine if PE4 has a route to the destination IP address 172.16.1.10:
RP/0/0/CPU0:PE4#show route vrf EAST 172.16.1.10
Mon Sep 11 19:58:28.128 UTC
Routing entry for 172.16.1.0/24
Known via "bgp 65500", distance 200, metric 0
Tag 65000, type internal
Installed Sep 8 18:28:46.303 for 3d01h
Routing Descriptor Blocks
10.10.10.1, from 10.10.10.6
Nexthop in Vrf: "default", Table: "default", IPv4 Unicast, Table Id: 0xe0000000
Route metric is 0
No advertising protos.
RP/0/0/CPU0:PE4#As this PE is a Cisco IOS XR device, the "detail" keyword can be used at the end of the show route vrf <name> command to look at some additional information such as the VPNv4 label imposed by MP-BGP (Multiprotocol BGP) and the source RD (Route Distinguisher) from the prefix:
RP/0/0/CPU0:PE4#show route vrf EAST 172.16.1.10 detail
Mon Sep 11 20:21:48.492 UTC
Routing entry for 172.16.1.0/24
Known via "bgp 65500", distance 200, metric 0
Tag 65000, type internal
Installed Sep 8 18:28:46.303 for 3d01h
Routing Descriptor Blocks
10.10.10.1, from 10.10.10.6
Nexthop in Vrf: "default", Table: "default", IPv4 Unicast, Table Id: 0xe0000000
Route metric is 0
Label: 0x10 (16) <<<<<
Tunnel ID: None
Binding Label: None
Extended communities count: 0
Source RD attributes: 0x0000:65000:1 <<<<<
NHID:0x0(Ref:0)
Route version is 0x5 (5)
No local label
IP Precedence: Not Set
QoS Group ID: Not Set
Flow-tag: Not Set
Fwd-class: Not Set
Route Priority: RIB_PRIORITY_RECURSIVE (12) SVD Type RIB_SVD_TYPE_REMOTE
Download Priority 3, Download Version 36
No advertising protos.
RP/0/0/CPU0:PE4#Now, lets take a look at the BGP VPNv4 prefix that was imported into the VRF, observe that this is the same label 16 from the previous output and it also has the extended community 65000:1. Also it is important to notice that 10.10.10.1 is the next-hop that PE4 needs to be able to perform a route recursion to it, the next address "from 10.10.10.6" is the BGP peer that PE4 used to learn this prefix (in this scenario it is the Route Reflector P6):
RP/0/0/CPU0:PE4#show bgp vpnv4 unicast vrf EAST 172.16.1.10
Mon Sep 11 22:42:28.114 UTC
BGP routing table entry for 172.16.1.0/24, Route Distinguisher: 65001:1
Versions:
Process bRIB/RIB SendTblVer
Speaker 48 48
Last Modified: Sep 8 18:28:46.314 for 3d04h
Paths: (1 available, best #1)
Not advertised to any peer
Path #1: Received by speaker 0
Not advertised to any peer
65000
10.10.10.1 (metric 20) from 10.10.10.6 (10.10.10.1) <<<<<
Received Label 16
Origin IGP, metric 0, localpref 100, valid, internal, best, group-best, import-candidate, imported
Received Path ID 0, Local Path ID 0, version 48
Extended community: RT:65000:1 <<<<<
Originator: 10.10.10.1, Cluster list: 10.10.10.6
Source AFI: VPNv4 Unicast, Source VRF: default, Source Route Distinguisher: 65000:1 <<<<< By reviewing CEF with the exact-route keyword at the VRF level you can get an idea of the exit interface for the packets. This command can also give some important details, as it shows the two labels imposed to the prefix, 24001 and 16, the reason being that label 16 is coming from BGP VPNv4 and label 24001 is coming from LDP (Label Distribution Protocol):
RP/0/0/CPU0:PE4#show cef vrf EAST exact-route 192.168.1.10 172.16.1.10
Mon Sep 11 22:48:15.241 UTC
172.16.1.0/24, version 36, internal 0x5000001 0x0 (ptr 0xa12dc74c) [1], 0x0 (0x0), 0x208 (0xa155b1b8)
Updated Sep 8 18:28:46.323
local adjacency 10.0.0.16
Prefix Len 24, traffic index 0, precedence n/a, priority 3
via GigabitEthernet0/0/0/4
via 10.10.10.1/32, 3 dependencies, recursive [flags 0x6000]
path-idx 0 NHID 0x0 [0xa15c3f54 0x0]
recursion-via-/32
next hop VRF - 'default', table - 0xe0000000
next hop 10.10.10.1/32 via 24010/0/21
next hop 10.0.0.16/32 Gi0/0/0/4 labels imposed {24001 16} <<<<<As a next step, the show bgp vpnv4 unicast command is used to check the VPNv4 routes that are being learned by this PE. This output shows the information before the VPNv4 prefix is imported into the VRF, remember that the RT (Route Target) configured (for this example the imported RTs are 65000:1, 65001:1, 65001:2) indicates what routes and to which VRF are imported:
RP/0/0/CPU0:PE4#show bgp vpnv4 unicast
Fri Sep 15 02:15:15.463 UTC
BGP router identifier 10.10.10.4, local AS number 65500
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 0
BGP main routing table version 85
BGP NSR Initial initsync version 1 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65000:1
*>i172.16.1.0/24 10.10.10.1 0 100 0 65000 i <<<<<
*>i172.16.2.0/24 10.10.10.1 0 100 0 65000 i
Route Distinguisher: 65001:1 (default for vrf EAST)
* i0.0.0.0/0 10.10.10.3 0 100 0 65001 i
*> 10.11.0.1 0 0 65001 i
*>i172.16.1.0/24 10.10.10.1 0 100 0 65000 i
*>i172.16.2.0/24 10.10.10.1 0 100 0 65000 i
*> 192.168.1.0/24 10.11.0.1 0 0 65001 i
*>i192.168.2.0/24 10.10.10.3 0 100 0 65001 i
*> 192.168.3.0/24 10.11.0.1 0 0 65001 i
Route Distinguisher: 65001:2
*>i0.0.0.0/0 10.10.10.3 0 100 0 65001 i
*>i192.168.2.0/24 10.10.10.3 0 100 0 65001 i
Processed 10 prefixes, 11 paths
In this example, the VPNv4 table can be small, but in a production environment instead of looking at all the VPNv4 prefixes, you can narrow the verification down to the specific RD and prefix with the next command:
RP/0/0/CPU0:PE4#show bgp vpnv4 unicast rd 65000:1 172.16.1.10
Mon Sep 11 22:54:04.967 UTC
BGP routing table entry for 172.16.1.0/24, Route Distinguisher: 65000:1
Versions:
Process bRIB/RIB SendTblVer
Speaker 46 46
Last Modified: Sep 8 18:28:46.314 for 3d04h
Paths: (1 available, best #1)
Not advertised to any peer
Path #1: Received by speaker 0
Not advertised to any peer
65000
10.10.10.1 (metric 20) from 10.10.10.6 (10.10.10.1)
Received Label 16
Origin IGP, metric 0, localpref 100, valid, internal, best, group-best, import-candidate, not-in-vrf
Received Path ID 0, Local Path ID 0, version 46
Extended community: RT:65000:1
Originator: 10.10.10.1, Cluster list: 10.10.10.6At this point, the MP-BGP control plane has the destination prefix and the LDP and VPNv4 labels {24001 16} respectively, the exit interface for this traffic seems to be Gi0/0/0/4 and the next hop where the traffic needs to be forwarded is 10.10.10.1. But, Is there another option to verify the preferred exit interface? It is time to take a look at the MPLS forwarding table or LFIB (Label Forwarding Information Base). With the use of command show mpls forwarding two entries are displayed towards 10.10.10.1 destination (Loopbback0 from PE1), one path with an Outgoing Interface of Gi0/0/0/4 and a next-hop 10.0.0.16 (router P5) where the imposed Outgoing Label is 24001 and another path through Gi0/0/0/3 with a next-hop 10.0.0.13 (router P6) and an Outgoing Label of 23:
RP/0/0/CPU0:PE4#show mpls forwarding
Mon Sep 11 23:28:33.425 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
24000 Unlabelled 192.168.1.0/24[V] Gi0/0/0/0 10.11.0.1 1096
24001 Unlabelled 192.168.3.0/24[V] Gi0/0/0/0 10.11.0.1 56056
24002 Unlabelled 0.0.0.0/0[V] Gi0/0/0/0 10.11.0.1 0
24003 Pop 10.10.10.6/32 Gi0/0/0/3 10.0.0.13 7778512
24004 Pop 10.0.0.4/31 Gi0/0/0/3 10.0.0.13 0
24005 Pop 10.0.0.8/31 Gi0/0/0/3 10.0.0.13 0
24006 Pop 10.10.10.5/32 Gi0/0/0/4 10.0.0.16 3542574
24007 Pop 10.0.0.10/31 Gi0/0/0/3 10.0.0.13 0
Pop 10.0.0.10/31 Gi0/0/0/4 10.0.0.16 0
24008 Pop 10.0.0.6/31 Gi0/0/0/4 10.0.0.16 0
24009 Pop 10.0.0.0/31 Gi0/0/0/4 10.0.0.16 0
24010 23 10.10.10.1/32 Gi0/0/0/3 10.0.0.13 22316 <<<<<
24001 10.10.10.1/32 Gi0/0/0/4 10.0.0.16 42308 <<<<<
24011 18 10.10.10.2/32 Gi0/0/0/3 10.0.0.13 0
24003 10.10.10.2/32 Gi0/0/0/4 10.0.0.16 0
24012 17 10.0.0.2/31 Gi0/0/0/3 10.0.0.13 0
24005 10.0.0.2/31 Gi0/0/0/4 10.0.0.16 0
24013 Pop 10.10.10.3/32 Gi0/0/0/1 10.0.0.20 3553900
24014 Pop 10.0.0.14/31 Gi0/0/0/1 10.0.0.20 0
Pop 10.0.0.14/31 Gi0/0/0/4 10.0.0.16 0
24015 Pop 10.0.0.18/31 Gi0/0/0/1 10.0.0.20 0
Pop 10.0.0.18/31 Gi0/0/0/3 10.0.0.13 0
RP/0/0/CPU0:PE4#show mpls forwarding prefix 10.10.10.1/32
Mon Sep 11 23:30:54.685 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
24010 23 10.10.10.1/32 Gi0/0/0/3 10.0.0.13 3188
24001 10.10.10.1/32 Gi0/0/0/4 10.0.0.16 6044
RP/0/0/CPU0:PE4#show mpls forwarding prefix 10.10.10.1/32 detail hardware egress
Mon Sep 11 23:36:06.504 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
24010 23 10.10.10.1/32 Gi0/0/0/3 10.0.0.13 N/A
Updated: Sep 8 20:27:26.596
Version: 39, Priority: 3
Label Stack (Top -> Bottom): { 23 }
NHID: 0x0, Encap-ID: N/A, Path idx: 0, Backup path idx: 0, Weight: 0
MAC/Encaps: 14/18, MTU: 1500
Outgoing Interface: GigabitEthernet0/0/0/3 (ifhandle 0x000000a0)
Packets Switched: 0
24001 10.10.10.1/32 Gi0/0/0/4 10.0.0.16 N/A
Updated: Sep 8 20:27:26.596
Version: 39, Priority: 3
Label Stack (Top -> Bottom): { 24001 }
NHID: 0x0, Encap-ID: N/A, Path idx: 1, Backup path idx: 0, Weight: 0
MAC/Encaps: 14/18, MTU: 1500
Outgoing Interface: GigabitEthernet0/0/0/4 (ifhandle 0x000000c0)
Packets Switched: 0
From the previous outputs it is clear that there are two path options where the traffic can be load balanced, but there are a couple of ways that can help to determine which one is the preferred path. One way is with the use of the show cef exact-route <source IP> <destination IP> command by adding the Loopback0 from Source PE and Loopback0 from Destination PE. As shown in the next output, the preferred path is through Gi0/0/0/4:
RP/0/0/CPU0:PE4#show cef exact-route 10.10.10.4 10.10.10.1
Mon Sep 11 23:49:44.558 UTC
10.10.10.1/32, version 39, internal 0x1000001 0x0 (ptr 0xa12dbdbc) [1], 0x0 (0xa12c18c0), 0xa28 (0xa185307c)
Updated Sep 8 20:27:26.596
local adjacency 10.0.0.16
Prefix Len 32, traffic index 0, precedence n/a, priority 3
via GigabitEthernet0/0/0/4
via 10.0.0.16/32, GigabitEthernet0/0/0/4, 9 dependencies, weight 0, class 0 [flags 0x0] <<<<<
path-idx 1 NHID 0x0 [0xa16765bc 0x0]
next hop 10.0.0.16/32
local adjacency
local label 24010 labels imposed {24001}Another option is to first verify the LIB (Label Information Base) and get the LDP binding of the destination Loopback0 (10.10.10.1 that belongs to the egress PE) by using the command show mpls ldp bindings <prefix/mask>, and once that the local binding label is found from that output, use that label value in the command show mpls forwarding exact-route label <label> ipv4 <source IP> <destination IP> detail to find the preferred path:
RP/0/0/CPU0:PE4#show mpls ldp bindings 10.10.10.1/32
Wed Sep 13 17:18:43.007 UTC
10.10.10.1/32, rev 29
Local binding: label: 24010 <<<<<
Remote bindings: (3 peers)
Peer Label
----------------- ---------
10.10.10.3:0 24
10.10.10.5:0 24001
10.10.10.6:0 23
RP/0/0/CPU0:PE4#show mpls forwarding exact-route label 24010 ipv4 10.10.10.4 10.10.10.1 detail
Wed Sep 13 17:20:06.342 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
24010 24001 10.10.10.1/32 Gi0/0/0/4 10.0.0.16 N/A <<<<<
Updated: Sep 12 14:15:37.009
Version: 198, Priority: 3
Label Stack (Top -> Bottom): { 24001 }
NHID: 0x0, Encap-ID: N/A, Path idx: 1, Backup path idx: 0, Weight: 0
Hash idx: 1
MAC/Encaps: 14/18, MTU: 1500
Outgoing Interface: GigabitEthernet0/0/0/4 (ifhandle 0x000000c0)
Packets Switched: 0
Via: Gi0/0/0/4, Next Hop: 10.0.0.16
Label Stack (Top -> Bottom): { 24001 }
NHID: 0x0, Encap-ID: N/A, Path idx: 1, Backup path idx: 0, Weight: 0
Hash idx: 1
MAC/Encaps: 14/18, MTU: 1500
Outgoing Interface: GigabitEthernet0/0/0/4 (ifhandle 0x000000c0)Next, it is important to check the next hop router that is in the dataplane, for this particular example the router to verify is P5 (that has the interface 10.0.0.16). The first place to start looking into is the MPLS forwarding table, where the Local Label for prefix 10.10.10.1 must be 24001:
RP/0/0/CPU0:P5#show mpls forwarding
Thu Sep 14 20:07:16.455 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
24000 Pop 10.10.10.6/32 Gi0/0/0/2 10.0.0.11 361906
24001 Pop 10.10.10.1/32 Gi0/0/0/1 10.0.0.0 361002 <<<<<
24002 Pop 10.0.0.4/31 Gi0/0/0/1 10.0.0.0 0
Pop 10.0.0.4/31 Gi0/0/0/2 10.0.0.11 0
24003 Pop 10.10.10.2/32 Gi0/0/0/0 10.0.0.6 360940
24004 Pop 10.0.0.8/31 Gi0/0/0/0 10.0.0.6 0
Pop 10.0.0.8/31 Gi0/0/0/2 10.0.0.11 0
24005 Pop 10.0.0.2/31 Gi0/0/0/0 10.0.0.6 0
Pop 10.0.0.2/31 Gi0/0/0/1 10.0.0.0 0
24006 Pop 10.10.10.4/32 Gi0/0/0/4 10.0.0.17 361230
24007 Pop 10.0.0.12/31 Gi0/0/0/2 10.0.0.11 0
Pop 10.0.0.12/31 Gi0/0/0/4 10.0.0.17 0
24008 Pop 10.10.10.3/32 Gi0/0/0/3 10.0.0.15 361346
24009 Pop 10.0.0.20/31 Gi0/0/0/3 10.0.0.15 0
Pop 10.0.0.20/31 Gi0/0/0/4 10.0.0.17 0
24010 Pop 10.0.0.18/31 Gi0/0/0/2 10.0.0.11 0
Pop 10.0.0.18/31 Gi0/0/0/3 10.0.0.15 0
RP/0/0/CPU0:P5#show mpls forwarding labels 24001
Thu Sep 14 20:07:42.584 UTC
Local Outgoing Prefix Outgoing Next Hop Bytes
Label Label or ID Interface Switched
------ ----------- ------------------ ------------ --------------- ------------
24001 Pop 10.10.10.1/32 Gi0/0/0/1 10.0.0.0 361060
RP/0/0/CPU0:P5#From the previous output it can be seen that the LFIB entry for prefix 10.10.10.1/32 shows "Pop" as the outgoing label, meaning that this router is the Penultimate Hop Popping (PHP). It also shows that traffic must be sent through Gi0/0/0/1 based on LFIB information, and this can be verified as well while looking at CEF. The next CEF exact-route output shows the Implicit Null Label as the imposed label, this again, is due to the fact that the next-hop connected at Gi0/0/0/1 is the last router in the label switch path and is also the PE facing the destination site (CE-WEST). This is also the reason why router P5 is removing and not imposing another label to the packet, thanks to this process the egress router PE1 is going to receive a packet without an LDP label:
RP/0/0/CPU0:P5#show cef exact-route 10.10.10.4 10.10.10.1
Thu Sep 14 20:25:57.269 UTC
10.10.10.1/32, version 192, internal 0x1000001 0x0 (ptr 0xa1246394) [1], 0x0 (0xa122b638), 0xa20 (0xa155b550)
Updated Sep 12 14:15:38.009
local adjacency 10.0.0.0
Prefix Len 32, traffic index 0, precedence n/a, priority 3
via GigabitEthernet0/0/0/1
via 10.0.0.0/32, GigabitEthernet0/0/0/1, 9 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 NHID 0x0 [0xa166e280 0xa166e674]
next hop 10.0.0.0/32
local adjacency
local label 24001 labels imposed {ImplNull} <<<<<The last point to verify the label switch path is PE1. While looking at the MPLS forwarding table it can be noticed that there is no entry for the prefix 10.10.10.1/32 in the LFIB:
PE1#show mpls forwarding-table
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
16 No Label 172.16.1.0/24[V] 12938 Gi3 10.10.0.1
17 No Label 172.16.2.0/24[V] 0 Gi3 10.10.0.1
18 Pop Label 10.0.0.6/31 0 Gi1 10.0.0.1
Pop Label 10.0.0.6/31 0 Gi2 10.0.0.3
19 Pop Label 10.0.0.8/31 0 Gi2 10.0.0.3
Pop Label 10.0.0.8/31 0 Gi4 10.0.0.5
20 Pop Label 10.0.0.10/31 0 Gi1 10.0.0.1
Pop Label 10.0.0.10/31 0 Gi4 10.0.0.5
21 Pop Label 10.0.0.12/31 0 Gi4 10.0.0.5
22 Pop Label 10.0.0.14/31 0 Gi1 10.0.0.1
23 Pop Label 10.0.0.16/31 0 Gi1 10.0.0.1
24 Pop Label 10.0.0.18/31 0 Gi4 10.0.0.5
25 24009 10.0.0.20/31 0 Gi1 10.0.0.1
22 10.0.0.20/31 0 Gi4 10.0.0.5
26 Pop Label 10.10.10.2/32 0 Gi2 10.0.0.3
27 24008 10.10.10.3/32 0 Gi1 10.0.0.1
24 10.10.10.3/32 0 Gi4 10.0.0.5
28 24006 10.10.10.4/32 0 Gi1 10.0.0.1
25 10.10.10.4/32 0 Gi4 10.0.0.5
29 Pop Label 10.10.10.5/32 0 Gi1 10.0.0.1
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
30 Pop Label 10.10.10.6/32 0 Gi4 10.0.0.5
31 [T] Pop Label 1/1[TE-Bind] 0 drop
[T] Forwarding through a LSP tunnel.
View additional labelling info with the 'detail' optionAs you have figured out, the reason of this behavior is that prefix (10.10.10.1/32) belongs to PE1, and router has also assigned an implicit null label to this connected prefix. This can be verified with the use of the show mpls ldp bindings command:
PE1#show run interface loopback 0
Building configuration...
Current configuration : 66 bytes
!
interface Loopback0
ip address 10.10.10.1 255.255.255.255
end
PE1#show mpls ldp bindings 10.10.10.1 32
lib entry: 10.10.10.1/32, rev 24
local binding: label: imp-null
remote binding: lsr: 10.10.10.6:0, label: 23
remote binding: lsr: 10.10.10.5:0, label: 24001
remote binding: lsr: 10.10.10.2:0, label: 24000As PE1 is a Cisco IOS XE router, the use of the command show bgp vpnv4 unicast all or show bgp vpnv4 unicast rd <value> <destination IP> can help to identify and confirm that the destination prefix 172.16.1.0/24 is being learned correctly through MP-BGP. The output of these commands shows the prefix after being exported:
PE1#show bgp vpnv4 unicast all
BGP table version is 61, local router ID is 10.10.10.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65000:1 (default for vrf WEST)
*>i 0.0.0.0 10.10.10.3 0 100 0 65001 i
*bi 10.10.10.4 0 100 0 65001 i
*> 172.16.1.0/24 10.10.0.1 0 0 65000 i <<<<<
*> 172.16.2.0/24 10.10.0.1 0 0 65000 i
*>i 192.168.1.0 10.10.10.4 0 100 0 65001 i
*>i 192.168.2.0 10.10.10.3 0 100 0 65001 i
*>i 192.168.3.0 10.10.10.4 0 100 0 65001 i
Route Distinguisher: 65001:1
*>i 0.0.0.0 10.10.10.4 0 100 0 65001 i
*>i 192.168.1.0 10.10.10.4 0 100 0 65001 i
*>i 192.168.3.0 10.10.10.4 0 100 0 65001 i
Route Distinguisher: 65001:2
Network Next Hop Metric LocPrf Weight Path
*>i 0.0.0.0 10.10.10.3 0 100 0 65001 i
*>i 192.168.2.0 10.10.10.3 0 100 0 65001 i
PE1#show bgp vpnv4 unicast rd 65000:1 172.16.1.10
BGP routing table entry for 65000:1:172.16.1.0/24, version 2
Paths: (1 available, best #1, table WEST)
Additional-path-install
Advertised to update-groups:
6
Refresh Epoch 2
65000
10.10.0.1 (via vrf WEST) from 10.10.0.1 (172.16.2.10) <<<<<
Origin IGP, metric 0, localpref 100, valid, external, best
Extended Community: RT:65000:1 , recursive-via-connected <<<<<
mpls labels in/out 16/nolabel
rx pathid: 0, tx pathid: 0x0
Updated on Sep 15 2023 18:27:23 UTCIn a similar way, looking at the BGP VPNv4 prefix at the VRF which is the prefix received by CE-WEST, with the use of the command show bgp vpnv4 unicast vrf <name> <prefix>, the output shows the MP-BGP label 16 that was carried all the way to the ingress PE4 as well as the RT export configured 65000:1:
PE1#show bgp vpnv4 unicast vrf WEST 172.16.1.10
BGP routing table entry for 65000:1:172.16.1.0/24, version 2
Paths: (1 available, best #1, table WEST)
Additional-path-install
Advertised to update-groups:
6
Refresh Epoch 2
65000
10.10.0.1 (via vrf WEST) from 10.10.0.1 (172.16.2.10)
Origin IGP, metric 0, localpref 100, valid, external, best
Extended Community: RT:65000:1 , recursive-via-connected <<<<<
mpls labels in/out 16/nolabel <<<<<
rx pathid: 0, tx pathid: 0x0
Updated on Sep 15 2023 18:27:23 UTC
PE1#show run vrf WEST
Building configuration...
Current configuration : 478 bytes
vrf definition WEST
rd 65000:1
route-target export 65000:1 <<<<<
route-target import 65000:1
route-target import 65001:1
route-target import 65001:2
!
address-family ipv4
exit-address-family
!
!
interface GigabitEthernet3
vrf forwarding WEST
ip address 10.10.0.2 255.255.255.252
negotiation auto
no mop enabled
no mop sysid
!
router bgp 65500
!
address-family ipv4 vrf WEST
neighbor 10.10.0.1 remote-as 65000
neighbor 10.10.0.1 activate
exit-address-family
!
endThe last information to check at this PE is the RIB and CEF entries at the VRF level to the destination IP, as opposed to the entry seen at PE4 there is no Label on the RIB for prefix 172.16.1.0/24, the reason is that this is the route coming in from the CE and this is learned via eBGP and inserted into the VRF routing table before this prefix is being exported into VPNv4. This can be verified with the use of the commands show ip route vrf <name> <prefix> and show ip cef vrf <name> <prefix> shown next:
PE1#show ip route vrf WEST 172.16.1.10
Routing Table: WEST
Routing entry for 172.16.1.0/24
Known via "bgp 65500", distance 20, metric 0
Tag 65000, type external
Last update from 10.10.0.1 1w0d ago
Routing Descriptor Blocks:
* 10.10.0.1, from 10.10.0.1, 1w0d ago, recursive-via-conn
opaque_ptr 0x7F8B4E3E1D50
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 65000
MPLS label: none
PE1#show ip cef vrf WEST 172.16.1.10
172.16.1.0/24
nexthop 10.10.0.1 GigabitEthernet3At this point, it has been confirmed that the destination prefix 172.16.1.0/24 was learned correctly by the source of the traffic CE (CE-EAST), it was correctly propagated through MP-BGP and also the labels from PEs and Ps loopbacks were learned across the label switch path. But still, the reachability between source/destination is not successful, and there is still one last router to verify CE-WEST. The first thing to check in this router is the routing table, remember that the source IP prefix 192.168.1.0/24 is expected to appear in there:
CE-WEST#show ip route 192.168.1.10
% Network not in table
CE-WEST#The "Network not in table" is clearly the problem, the BGP table can also be verified but after looking for the prefix it is not there either:
CE-WEST#show ip bgp
BGP table version is 41, local router ID is 172.16.2.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 172.16.1.0/24 0.0.0.0 0 32768 i
*> 172.16.2.0/24 0.0.0.0 0 32768 i
CE-WEST#Going a step back, you can verify if this provider edge router (PE1) is advertising the prefix to the eBGP neighbor CE-WEST, this can be done with the use of the command show bgp vpnv4 unicast vrf <name> neighbors <neighbor IP> advertised-routes shown next:
PE1#show bgp vpnv4 unicast vrf WEST neighbors 10.10.0.1 advertised-routes
BGP table version is 61, local router ID is 10.10.10.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 65000:1 (default for vrf WEST)
*>i 0.0.0.0 10.10.10.3 0 100 0 65001 i
*>i 192.168.1.0 10.10.10.4 0 100 0 65001 i <<<<<
*>i 192.168.2.0 10.10.10.3 0 100 0 65001 i
*>i 192.168.3.0 10.10.10.4 0 100 0 65001 i
Total number of prefixes 4Based on the previous step, it can be confirmed that PE1 router is advertising the prefix correctly to CE-WEST, hence it is time to look at the BGP neighbors on the CE side:
CE-WEST#show ip bgp neighbors
BGP neighbor is 10.10.0.2, remote AS 65500, external link
BGP version 4, remote router ID 10.10.10.1
BGP state = Established, up for 1w4d
Last read 00:00:40, last write 00:00:43, hold time is 180, keepalive interval is 60 seconds
Neighbor sessions:
1 active, is not multisession capable (disabled)
Neighbor capabilities:
Route refresh: advertised and received(new)
Four-octets ASN Capability: advertised and received
Address family IPv4 Unicast: advertised and received
Enhanced Refresh Capability: advertised and received
Multisession Capability:
Stateful switchover support enabled: NO for session 1
Message statistics:
InQ depth is 0
OutQ depth is 0
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 3 17
Keepalives: 19021 18997
Route Refresh: 2 0
Total: 19029 19019
Do log neighbor state changes (via global configuration)
Default minimum time between advertisement runs is 30 seconds
For address family: IPv4 Unicast
Session: 10.10.0.2
BGP table version 41, neighbor version 41/0
Output queue size : 0
Index 3, Advertise bit 0
3 update-group member
Inbound path policy configured
Route map for incoming advertisements is FILTER <<<<<
Slow-peer detection is disabled
Slow-peer split-update-group dynamic is disabled
Sent Rcvd
Prefix activity: ---- ----
Prefixes Current: 2 0
Prefixes Total: 4 23
Implicit Withdraw: 2 13
Explicit Withdraw: 0 10
Used as bestpath: n/a 0
Used as multipath: n/a 0
Used as secondary: n/a 0
Outbound Inbound
Local Policy Denied Prefixes: -------- -------
route-map: 0 4
Bestpath from this peer: 18 n/a
Total: 18 4
Number of NLRIs in the update sent: max 2, min 0
Last detected as dynamic slow peer: never
Dynamic slow peer recovered: never
Refresh Epoch: 3
Last Sent Refresh Start-of-rib: 4d23h
Last Sent Refresh End-of-rib: 4d23h
Refresh-Out took 0 seconds
Last Received Refresh Start-of-rib: 4d23h
Last Received Refresh End-of-rib: 4d23h
Refresh-In took 0 seconds
Sent Rcvd
Refresh activity: ---- ----
Refresh Start-of-RIB 1 2
Refresh End-of-RIB 1 2
Address tracking is enabled, the RIB does have a route to 10.10.0.2
Route to peer address reachability Up: 1; Down: 0
Last notification 1w5d
Connections established 3; dropped 2
Last reset 1w4d, due to Peer closed the session of session 1
External BGP neighbor configured for connected checks (single-hop no-disable-connected-check)
Interface associated: GigabitEthernet0/3 (peering address in same link)
Transport(tcp) path-mtu-discovery is enabled
Graceful-Restart is disabled
SSO is disabled
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 1
Local host: 10.10.0.1, Local port: 179
Foreign host: 10.10.0.2, Foreign port: 39410
Connection tableid (VRF): 0
Maximum output segment queue size: 50
Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)
Event Timers (current time is 0x4D15FD56):
Timer Starts Wakeups Next
Retrans 19027 1 0x0
TimeWait 0 0 0x0
AckHold 19012 18693 0x0
SendWnd 0 0 0x0
KeepAlive 0 0 0x0
GiveUp 0 0 0x0
PmtuAger 0 0 0x0
DeadWait 0 0 0x0
Linger 0 0 0x0
ProcessQ 0 0 0x0
iss: 1676751051 snduna: 1677112739 sndnxt: 1677112739
irs: 2109012892 rcvnxt: 2109374776
sndwnd: 16061 scale: 0 maxrcvwnd: 16384
rcvwnd: 15890 scale: 0 delrcvwnd: 494
SRTT: 1000 ms, RTTO: 1003 ms, RTV: 3 ms, KRTT: 0 ms
minRTT: 0 ms, maxRTT: 1000 ms, ACK hold: 200 ms
uptime: 1036662542 ms, Sent idletime: 40725 ms, Receive idletime: 40925 ms
Status Flags: passive open, gen tcbs
Option Flags: nagle, path mtu capable
IP Precedence value : 6
Datagrams (max data segment is 1460 bytes):
Rcvd: 37957 (out of order: 0), with data: 19014, total data bytes: 361883
Sent: 37971 (retransmit: 1, fastretransmit: 0, partialack: 0, Second Congestion: 0), with data: 19027, total data bytes: 361687
Packets received in fast path: 0, fast processed: 0, slow path: 0
fast lock acquisition failures: 0, slow path: 0
TCP Semaphore 0x0F3194AC FREE The previous output reveals that there is a route map applied for incoming advertisements with the name "FILTER", after looking at the route map configuration it shows a match clause pointing to a prefix-list with a permit statement for 192.168.0.0/16, however this is incorrect as the prefix-list is only permitting that specific prefix and not all the ones that can be included in this range:
CE-WEST#show route-map FILTER
route-map FILTER, permit, sequence 10
Match clauses:
ip address prefix-lists: FILTER
Set clauses:
Policy routing matches: 0 packets, 0 bytes
CE-WEST#show ip prefix-list FILTER
ip prefix-list FILTER: 1 entries
seq 5 permit 192.168.0.0/16 <<<<<
CE-WEST#show run | i ip prefix-list
ip prefix-list FILTER seq 5 permit 192.168.0.0/16With a small change to the prefix-list configuration, the route towards 192.168.1.10 is now installed in the RIB:
CE-WEST#show run | i ip prefix-list
ip prefix-list FILTER seq 5 permit 192.168.0.0/16 le 32 <<<<<
CE-WEST#show ip bgp
BGP table version is 44, local router ID is 172.16.2.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 172.16.1.0/24 0.0.0.0 0 32768 i
*> 172.16.2.0/24 0.0.0.0 0 32768 i
*> 192.168.1.0 10.10.0.2 0 65500 65001 i <<<<<
*> 192.168.2.0 10.10.0.2 0 65500 65001 i
*> 192.168.3.0 10.10.0.2 0 65500 65001 i
CE-WEST#show ip route 192.168.1.10
Routing entry for 192.168.1.0/24 <<<<<
Known via "bgp 65000", distance 20, metric 0
Tag 65500, type external
Last update from 10.10.0.2 00:00:37 ago
Routing Descriptor Blocks:
* 10.10.0.2, from 10.10.0.2, 00:00:37 ago
Route metric is 0, traffic share count is 1
AS Hops 2
Route tag 65500
MPLS label: noneVerification
Now, the reachability between source and destination is successful and it can be confirmed that the traceroute goes through the same label switch path that was tracked across the MPLS network:
Forwarding Path
CE-EAST#ping 172.16.1.10 source loopback 1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 172.16.1.10, timeout is 2 seconds: Packet sent with a source address of 192.168.1.10 !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 7/7/9 ms <<<<< CE-EAST#traceroute 172.16.1.10 source loop1 probe 1 numeric Type escape sequence to abort. Tracing the route to 172.16.1.10 VRF info: (vrf in name/id, vrf out name/id) 1 10.11.0.2 2 msec 2 10.0.0.16 [MPLS: Labels 24001/16 Exp 0] 9 msec 3 10.10.0.2 [MPLS: Label 16 Exp 0] 8 msec 4 10.10.0.1 9 msecRP/0/0/CPU0:P5#show ipv4 interface brief Wed Sep 20 18:23:47.158 UTC Interface IP-Address Status Protocol Vrf-Name Loopback0 10.10.10.5 Up Up default MgmtEth0/0/CPU0/0 unassigned Shutdown Down default GigabitEthernet0/0/0/0 10.0.0.7 Up Up default GigabitEthernet0/0/0/1 10.0.0.1 Up Up default <<<<< GigabitEthernet0/0/0/2 10.0.0.10 Up Up default GigabitEthernet0/0/0/3 10.0.0.14 Up Up default GigabitEthernet0/0/0/4 10.0.0.16 Up Up default <<<<< RP/0/0/CPU0:P5#
Cisco IOS XE Verification Commands
MPLS/LDP
show mpls interfaces
show mpls forwarding-table
show mpls ldp bindings [destination prefix]
show mpls ldp neighbor [neighbor address]
clear mpls ldp neighbor [neighbor address|*]
RIB and CEF
show ip vrf [detail]
show run vrf
show ip route [destination prefix]
show ip route vrf <name> [destination prefix]
show ip cef vrf <name> [destination prefix]
show ip cef exact-route <source IP> <destination IP>
show ip cef vrf <name> exact-route <source IP> <destination IP>
BGP/VPNv4
show ip bgp [neighbors] <neighbor address>
show bgp vpnv4 unicast all [summary|destination prefix]
show bgp vpnv4 unicast all neighbor <neighbor address> advertised-routes
show bgp vpnv4 unicast vrf <name> neighbors <neighbor IP> advertised-routes
show bgp vpnv4 unicast vrf <name> <prefix>
show bgp vpnv4 unicast rd <value> <destination IP>Cisco IOS XR Verification Commands
MPLS/LDP
show mpls interfaces
show mpls forwarding
show mpls ldp bindings [destination prefix/mask]
show mpls ldp neighbor [neighbor address]
show mpls forwarding prefix [destination prefix/mask]
show mpls forwarding prefix [destination prefix/mask] detail hardware egress
clear mpls ldp neighbor [neighbor address]
RIB and CEF
show vrf [name|all]
show run vrf [name]
show route [destination prefix]
show route vrf <name> [destination prefix]
show cef vrf <name> [destination prefix]
show cef exact-route <source IP> <destination IP>
show cef vrf <name> exact-route <source IP> <destination IP>
BGP/VPNv4
show bgp vpnv4 unicast [summary|destination prefix/mask]
show bgp vpnv4 unicast neighbors <neighbor address> advertised-routes
show bgp vpnv4 unicast vrf <name> [prefix]
show bgp vrf <name> neighbors <neighbor IP> advertised-routes
show bgp vpnv4 unicast rd [value|all] [destination IP]Related Information
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
21-Sep-2023 |
Initial Release |
Feedback