Contents

Introduction

Problem

Solution

Introduction

This document describes how to troubleshoot when the MPLS forwarding path is broken due to L2 header corruption on NCS4K node (6.5.26).

Problem

In this scenario, the Traffic Engineering (TE) tunnel is up, but Multiprotocol Label Switching (MPLS) ping does not work via MPLS tunnel:

tunnel-te5180                  10.38.101.62    Up              Up       default

#ping mpls traffic-eng tunnel-te 5180

Thu Jan  5 21:30:29.245 UTC

Sending 5, 100-byte MPLS Echos to tunnel-te5180,

timeout is 2 seconds, send interval is 0 msec:

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,

'L' - labeled output interface, 'B' - unlabeled output interface,

'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,

'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label,

'P' - no rx intf label prot, 'p' - premature termination of LSP,

'R' - transit router, 'I' - unknown upstream index,

'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.

.....

Success rate is 0 percent (0/5)

You see some hops in the MPLS traceroute:

#traceroute mpls traffic-eng tunnel-te 5180

Thu Jan  5 21:30:49.405 UTC

Tracing MPLS TE Label Switched Path on tunnel-te5180, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,

'L' - labeled output interface, 'B' - unlabeled output interface,

'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,

'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label,

'P' - no rx intf label prot, 'p' - premature termination of LSP,

'R' - transit router, 'I' - unknown upstream index,

'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.

0 172.16.61.78 MRU 9582 [Labels: 27769 Exp: 0]

L 1 172.16.61.79 MRU 9582 [Labels: 28136 Exp: 0] 7 ms

. 2 *

. 3 *

. 4 *

. 5 *^C

When you check MPLS tunnel, you see two more hops are signalized for the Explicit Router (ERO):

#show mpls traffic-eng tunnels 5180

Thu Jan  5 21:31:11.958 UTC

Name: tunnel-te5180  Destination: 10.38.96.1  Ifhandle:0x80002c4

Signalled-Name: MIVLPAMI-0112003A_t5180

Status:

Admin:    up Oper:   up   Path:  valid   Signalling: connected

path option 10,  type dynamic  (Basis for Setup, path weight 3000)

Accumulative metrics: TE 3000 IGP 30 Delay 900000

Path-option attribute: eline-any

Number of affinity constraints: 1

Include bit map           : 0x2

Include ext bit map       :

Length: 256 bits

Value : 0x::2

Include affinity name     : eline(1)

G-PID: 0x0800 (derived from egress interface properties)

Bandwidth Requested: 7 kbps  CT0

Creation Time: Thu Nov 10 22:17:55 2022 (7w6d ago)

Config Parameters:

Bandwidth:        0 kbps (CT0) Priority:  5  5 Affinity: 0x0/0xffff

Metric Type: TE (interface)

Path Selection:

Tiebreaker: Min-fill (default)

Hop-limit: disabled

Cost-limit: disabled

Delay-limit: disabled

Path-invalidation timeout: 10000 msec (default), Action: Tear (default)

AutoRoute:  enabled  LockDown: disabled   Policy class: not set

Forward class: 0 (not enabled)

Forwarding-Adjacency: disabled

Autoroute Destinations: 0

Loadshare:          0 equal loadshares

Auto-bw: enabled

Last BW Applied: 7 kbps CT0   BW Applications: 29

Last Application Trigger: Periodic Application

Bandwidth Min/Max: 0-4294967295 kbps

Application Frequency: 60 min   Jitter: 0s   Time Left: 46m 48s

Collection Frequency: 5 min

Samples Collected: 2   Next: 1m 3s

Highest BW: 0 kbps   Underflow BW: 0 kbps

Adjustment Threshold: 10%   10 kbps

Overflow Detection disabled

Underflow Detection disabled

Resignal Last-bandwidth Disabled

Auto-Capacity: Disabled:

Fast Reroute: Enabled, Protection Desired: Any

Path Protection: Not Enabled

BFD Fast Detection: Disabled

Reoptimization after affinity failure: Enabled

Soft Preemption: Disabled

History:

Tunnel has been up for: 7w6d (since Thu Nov 10 22:17:55 UTC 2022)

Current LSP:

Uptime: 15:09:12 (since Thu Jan 05 06:22:00 UTC 2023)

Reopt. LSP:

Last Failure:

LSP not signalled, identical to the [CURRENT] LSP

Date/Time: Thu Jan 05 19:03:33 UTC 2023 [02:27:39 ago]

Prior LSP:

ID: 32 Path Option: 10

Removal Trigger: reoptimization completed

Path info (IS-IS 1 level-2):

Node hop count: 3

Hop0: 172.16.61.79

Hop1: 172.16.57.244

Hop2: 172.16.6.59

Hop3: 10.38.96.1

When you go to the first hop along the path, you see correct MPLS Label Forwarding Information Base (LFIB) entry for this tunnel:

#show mpls forwarding labels 27769

Fri Jan  6 06:13:04.220 UTC

Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes

Label  Label       or ID              Interface                    Switched

------ ----------- ------------------ ------------ --------------- ------------

27769  28136       TE: 5180           Hu0/10/0/11/2.4001 172.16.57.244   0

28136       TE: 5180           tt60409      point2point     0            (!)

The egress interface on this node uses this MAC address, so this one can be used as a Source (SRC) MAC in L2 header for the L2 frames:

#show interfaces hundredGigE 0/10/0/11/2.4001

Fri Jan  6 06:14:45.773 UTC

HundredGigE0/10/0/11/2.4001 is up, line protocol is up

Interface state transitions: 79

Hardware is VLAN sub-interface(s), address is 0c11.67c8.2041

Description: To HundredGigE1/3/0/10/2.PHLAPALO-12121302A:CID:I1001/GE100/PHLAPALO/SLTNPAST

Internet address is 172.16.57.245

MTU 9600 bytes, BW 100000000 Kbit (Max: 100000000 Kbit)

reliability 255/255, txload 0/255, rxload 0/255

Encapsulation 802.1Q Virtual LAN, VLAN Id 4001,  loopback not set,

Last link flapped 1w6d

ARP type ARPA, ARP timeout 04:00:00

Last input 00:00:00, output 00:00:00

Last clearing of "show interface" counters never

5 minute input rate 64000 bits/sec, 62 packets/sec

5 minute output rate 2198000 bits/sec, 699 packets/sec

4529877895 packets input, 2267795435148 bytes, 6 total input drops

0 drops for unrecognized upper-level protocol

Received 124 broadcast packets, 0 multicast packets

3926978895 packets output, 1611587340639 bytes, 0 total output drops

Output 0 broadcast packets, 0 multicast packets

But on the neighbor side in show captured packets ingress location <active LC VM>, you see the correct MPLS label, but totally wrong L2 SRC and Destination (DST) MAC addresses:

[200] Jan   6 06:10:12.449, len: 103, hits: 1, buffhdr type: 1

i/p i/f: HundredGigE1/3/0/10/2

punt reason: DROP_PACKET

Ingress Headers:

port_ifh: 0x8001ae4, sub_ifh: 0x0, bundle_ifh: 0x0

logical_port: 0x6c1, pd_pkt_type: 3

punt_reason: DROP_PACKET (0)

payload_offset: 21, l3_offset: 21

FTMH:

pkt_size: 0x7e, tc: 0, tm_act_type: 0, ssp: 0x981

PPH:

pph_fwd_code: CPU Trap (7), fwd_hdr_offset: 0

inlif: 0x0, vrf: 0x0, rif: 0x0

FHEI:

trap_code: Rx_UNKNOWN_PACKET (63), trap_qual: 193

[ether dst: 0000.0000.0000 src: 0c11.67c8.2000 type/len: 0x8847]

[MPLS  label: 28136, exp 0x6, eos 0, ttl 255]

DST Mac address is all-0, and SRC MAC address matches the management interface on NCS4K node:

#show interfaces mgmtEth 0/rp1/emS/0

Fri Jan  6 06:15:59.141 UTC

MgmtEth0/RP1/EMS/0 is down, line protocol is down

Interface state transitions: 0

Hardware is Management Ethernet, address is 0c11.67c8.2000 (bia 0c11.67c8.2000)

Internet address is 10.230.192.86

MTU 1514 bytes, BW 100000 Kbit (Max: 100000 Kbit)

reliability 255/255, txload 0/255, rxload 0/255

Encapsulation ARPA,

Full-duplex, 100Mb/s, unknown, link type is autonegotiation

loopback not set,

ARP type ARPA, ARP timeout 04:00:00

Last input never, output never

Last clearing of "show interface" counters never

5 minute input rate 0 bits/sec, 0 packets/sec

5 minute output rate 0 bits/sec, 0 packets/sec

0 packets input, 0 bytes, 0 total input drops

0 drops for unrecognized upper-level protocol

Received 0 broadcast packets, 0 multicast packets

0 runts, 0 giants, 0 throttles, 0 parity

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

0 packets output, 0 bytes, 0 total output drops

Output 0 broadcast packets, 0 multicast packets

0 output errors, 0 underruns, 0 applique, 0 resets

0 output buffer failures, 0 output buffers swapped out

0 carrier transitions

Solution

The root cause is this DDTS - Cisco bug ID CSCvz99253 Cisco bug ID CSCwa11748 duplicate of Cisco bug ID CSCvz99253 which is fixed in 6.5.32 release:

++++++++

Dec 23 23:24:58.214 ofa_ipnhgroup_event 0/LC1 3235382# t4839 TP3147224,dnxsdk_l3_fec_create,enter,trans_id,357633649,npu_id,0,is_modify,1,use_eei_encoding,0,dest,0x6052,encap_id,0x4000304a,fec_id,0x2001fe98,

Dec 23 15:10:16.787 ofa_ipnh_event 0/LC1 210586# t5614 TP2061,client_ipnh_create,grid_res_id_alloc_req_success,trans_id,357386085,encap_id,0x304a,alloc_sz,2

Dec 23 15:10:16.787 ofa_ipnh_event 0/LC1 286920# t4857 TP9909,dispatch_ipnh,resolve_refhdl_success,ref_l3intf_trans_id,357386081,hdl,0x87a75238

Dec 23 15:10:16.787 ofa_ipnh_event 0/LC1 172418# t4857 TP9913,dummy_block,trans_id,357386085,wait,duration,time,0.1742

Dec 23 15:10:16.787 ofa_ipnh_event 0/LC1 153352# t4857 TP3149344,srv_ipnh_create,entry,trans_id,357386085,npu_mask,0x100000,l3a_mac_addr,00af.1f18.0043,l3a_intf_id,28,port_id,0

Dec 23 15:10:16.780 ofa_ipnh_event 0/LC1 258284# t5614 TP2061,client_ipnh_create,grid_res_id_alloc_req_success,trans_id,357386077,encap_id,0x3048,alloc_sz,2

Dec 23 15:10:16.780 ofa_ipnh_event 0/LC1 105614# t4857 TP9909,dispatch_ipnh,resolve_refhdl_success,ref_l3intf_trans_id,357383451,hdl,0x87a75238

Dec 23 15:10:16.780 ofa_ipnh_event 0/LC1 172408# t4857 TP9913,dummy_block,trans_id,357386077,wait,duration,time,0.1947

Dec 23 15:10:16.780 ofa_ipnh_event 0/LC1 286912# t4857 TP3149344,srv_ipnh_create,entry,trans_id,357386077,npu_mask,0x100000,l3a_mac_addr,00af.1f18.0043,l3a_intf_id,28,port_id,0

++++++++

As a recovery method, you can reconfigure the affected egress subinterface.