Introduction
This document describes the symptoms observed on a NCS4K platform when affected by Cisco bug ID CSCwb69873
Background Information
There are multiple cases where Bidirectional Forwarding Detection (BFD) session is flapping between two NCS4K nodes (SW 6.5.26), and finally put into dampened state.
At the same time, both nodes are able to ping each other without packet loss.
Problem
On a problem node, you see BFD session is in INIT state on active and standby LC VM:
Hu0/4/0/11/2.4001 10.55.6.180 0s 6s(2s*3) INIT
Yes 0/LC0
Hu0/4/0/11/2.4001 10.55.6.180 0s 6s(2s*3) INIT
Yes 0/LC1
In show logging output, you see these tracebacks:
LC/0/LC1:2022 Nov 24 11:29:37.098 UTC: fib_mgr[198]: %ROUTING-FIB-3-PLATF_UPD_FAIL : FIB platform update failed: Obj=DATA_TYPE_NHINFO[ptr=0x87c654c0,refc=0,flags=0x10000008] Action=CREATE Proto=ipv4. Cerr='Subsystem(8165)' detected the 'resource not available' condition 'Code(0)' : fib_mgr : (PID=5045) : -Traceback= 7f648e9733c7 7f648f8d546e 7f648f74cb3f 7f648f75449c 7f648f75a3c4 7f648f7cee95 7f648f8c7e58 7f648f7c623f 7f648f7ca7f4 7f648f8c841f 7f648f8fc3f7 7f648f8c7e58 7f648f7c623f 7f648f7ca7f4 7f648f8c8465 7f648f7d3028
LC/0/LC1:2022 Nov 24 11:29:52.163 UTC: fib_mgr[198]: %ROUTING-FIB-3-PLATF_UPD_FAIL : FIB platform update failed: Obj=DATA_TYPE_NHINFO[ptr=0x87c652d0,refc=0,flags=0x10000008] Action=CREATE Proto=ipv4. Cerr='Subsystem(8165)' detected the 'resource not available' condition 'Code(0)' : fib_mgr : (PID=5045) : -Traceback= 7f648e9733c7 7f648f8d546e 7f648f74cb3f 7f648f75449c 7f648f75a3c4 7f648f7cee95 7f648f8c7e58 7f648f7c623f 7f648f7ca7f4 7f648f8c841f 7f648f8fc3f7 7f648f8c7e58 7f648f7c623f 7f648f7ca7f4 7f648f8c8465 7f648f7d3028
In the command, show cef platform trace all all location <active LC VM> output you see:
Nov 23 21:03:03.659 dnx_fib_retry_event 0/LC1 t5851 TP764,ppinfo_send_encap_gcc,ppindex_alloc_failed,trans_id,716207282,ifh,0x800087c,rc,-806191104:'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
And finally, you are able to see these errors in command show ppinfo producer trac all location <active LCVM> inc resource output:
Nov 28 07:47:20.819 ppinfo/prod/fib/ipv4/err 0/LC1 517384# t5851 ppindex alloc Cerr:'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:20.819 ppinfo/prod/fib/ipv4/err 0/LC1 452746# t5851 indx alloc failed: Cerr:'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:20.819 ppinfo/prod/fib/ipv4/err 0/LC1 508753# t5851 ppindex alloc Cerr: 'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:20.827 ppinfo/prod/fib/ipv4/err 0/LC1 504601# t5851 ppindex alloc Cerr:'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:20.827 ppinfo/prod/fib/ipv4/err 0/LC1 619759# t5851 indx alloc failed: Cerr:'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:20.827 ppinfo/prod/fib/ipv4/err 0/LC1 1067176# t5851 ppindex alloc Cerr: 'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:20.835 ppinfo/prod/fib/mpls/err 0/LC1 517116# t5851 ppindex alloc Cerr:'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:20.835 ppinfo/prod/fib/mpls/err 0/LC1 2096661# t5851 indx alloc failed: Cerr:'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:20.835 ppinfo/prod/fib/mpls/err 0/LC1 617742# t5851 ppindex alloc Cerr: 'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:20.843 ppinfo/prod/fib/mpls/err 0/LC1 1298595# t5851 ppindex alloc Cerr:'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:20.843 ppinfo/prod/fib/mpls/err 0/LC1 190161# t5851 indx alloc failed: Cerr:'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:20.843 ppinfo/prod/fib/mpls/err 0/LC1 1299219# t5851 ppindex alloc Cerr: 'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:35.845 ppinfo/prod/fib/ipv4/err 0/LC1 621013# t5851 ppindex alloc Cerr:'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Nov 28 07:47:35.845 ppinfo/prod/fib/ipv4/err 0/LC1 512881# t5851 indx alloc failed: Cerr:'Subsystem(8165)' detected the 'resource not available' condition 'Code(0)'
Solution
All these symptoms together match to this DDTS:
Cisco bug ID CSCwb69873ppinfo allocation failed:
Summary Defect
Fixed in 6.5.33.
Symptom:
Link BFD session is flapping or stuck in INIT state.
Conditions:
Flapping of next hop address causes this issue.
Workaround:
RP card switch over (basically RP and LC VM switchover). This clears the PPIndex allocation failure. To resolve the LINK BFD issue, do either of the these, delete and recreate the impacted sub-interface (OR) shut / no shut of the impacted sub-interface.
Further Problem Description:
ppindex entries leak and cause all free entries to be consumed and that causes the resource exhaustion issue. PPindex is freed when NH is deleted and ppindex is allocated when NH is created. There is a bug that causes NH delete not to free ppindex that causes the leak. Once all ppindex are used up, there are no calls to make them free and that makes the system persistently fail and creates any new NHs. There is a check for encap_id tha is allocated or not, but the code does not get encap_id, and causes the check to fail and bails out early without calling ppindex_free.
Feedback