Troubleshoot Multiple Data Processing Cards Shutdown Due too Many npumgr Crashes

Introduction

This document describes how to troubleshoot a problem that occurs when multiple Data Processing Card (DPCs) shutdown in a very short period due to npumgr crashes.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Hardware knowledge of ASR5000/5500
• StarOS
• Basic knowledge of Routing

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command

Abbreviations

Problem 

As a part of a planned activity, new interfaces are bound in VLANs under ports done. The second part of the activity is to create static routes via these interfaces. As soon as VLAN is opened for traffic, npumgr crashes begun and subsequently made all DPC cards to shut down multiple times.

Troubleshoot 

This section provides information in order to troubleshoot the problem of multiple DPC card shutdown in a very short period due to npumgr crashes.

Collected Show Support Details (SSD), activity logs and syslogs that cover logs of the problem are shown here. Firstly, the rct stats are checked in order to see the reason for these shutdowns. It could be seen here that they are shut down due to too many npumgr crashes.

******** show rct stats verbose *******
Thursday September 19 03:57:04 IST 2019
RCT stats details (Last 18 Actions)
 #  Action        Type      From  To  Start Time                Duration     Status 
--- ------------- --------- ---- ---- ------------------------  -----------  -------
 7  Shutdown      N/A         2   10  2019-Sep-19+00:09:51.587    2.322 sec  Success 
 8  Shutdown      N/A         1    0  2019-Sep-19+00:10:14.541    0.005 sec  Success 
 9  Shutdown      N/A         3    0  2019-Sep-19+00:10:44.625    0.005 sec  Success 
10  Shutdown      N/A         4    0  2019-Sep-19+00:11:03.428    0.005 sec  Success 
11  Shutdown      N/A         7    0  2019-Sep-19+00:11:34.771    0.478 sec  Success 
12  Shutdown      N/A         8    0  2019-Sep-19+00:11:54.328    0.005 sec  Success 
13  Shutdown      N/A         9    0  2019-Sep-19+00:12:19.656    0.005 sec  Success 
14  Shutdown      N/A        10    0  2019-Sep-19+00:12:39.706    0.004 sec  Success 
15  Shutdown      N/A         1    9  2019-Sep-19+00:32:30.567    0.005 sec  Success 
16  Shutdown      N/A         2    0  2019-Sep-19+00:32:36.282    0.031 sec  Success 
17  Shutdown      N/A         3    0  2019-Sep-19+00:32:56.456    0.005 sec  Success 
18  Shutdown      N/A         4    0  2019-Sep-19+00:33:30.426    0.005 sec  Success 

RCT stats summary
-----------------
Migrations        =   2,  Average time = 10.890 sec
  Management Card =   2,  Average time = 10.890 sec
  Packet Card     =   0
Switchovers       =   2,  Average time = 18.526 sec

RCT stats verbose
-----------------
Stats  7:
  Action            : Shutdown
  Type              : N/A
  From              : 2
  To                : 10
  Start Time        : 2019-Sep-19+00:09:51.587
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 2.322 sec
  Graceful          : Enabled

Stats  8:
  Action            : Shutdown
  Type              : N/A
  From              : 1
  To                : 0
  Start Time        : 2019-Sep-19+00:10:14.541
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 0.005 sec
  Graceful          : Enabled

Stats  9:
  Action            : Shutdown
  Type              : N/A
  From              : 3
  To                : 0
  Start Time        : 2019-Sep-19+00:10:44.625
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 0.005 sec
  Graceful          : Enabled

Stats 10:
  Action            : Shutdown
  Type              : N/A
  From              : 4
  To                : 0
  Start Time        : 2019-Sep-19+00:11:03.428
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 0.005 sec
  Graceful          : Enabled

Stats 11:
  Action            : Shutdown
  Type              : N/A
  From              : 7
  To                : 0
  Start Time        : 2019-Sep-19+00:11:34.771
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 0.478 sec
  Graceful          : Enabled

Stats 12:
  Action            : Shutdown
  Type              : N/A
  From              : 8
  To                : 0
  Start Time        : 2019-Sep-19+00:11:54.328
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 0.005 sec
  Graceful          : Enabled

Stats 13:
  Action            : Shutdown
  Type              : N/A
  From              : 9
  To                : 0
  Start Time        : 2019-Sep-19+00:12:19.656
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 0.005 sec
  Graceful          : Enabled

Stats 14:
  Action            : Shutdown
  Type              : N/A
  From              : 10
  To                : 0
  Start Time        : 2019-Sep-19+00:12:39.706
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 0.004 sec
  Graceful          : Enabled

Stats 15:
  Action            : Shutdown
  Type              : N/A
  From              : 1
  To                : 9
  Start Time        : 2019-Sep-19+00:32:30.567
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 0.005 sec
  Graceful          : Enabled

Stats 16:
  Action            : Shutdown
  Type              : N/A
  From              : 2
  To                : 0
  Start Time        : 2019-Sep-19+00:32:36.282
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 0.031 sec
  Graceful          : Enabled

Stats 17:
  Action            : Shutdown
  Type              : N/A
  From              : 3
  To                : 0
  Start Time        : 2019-Sep-19+00:32:56.456
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 0.005 sec
  Graceful          : Enabled

Stats 18:
  Action            : Shutdown
  Type              : N/A
  From              : 4
  To                : 0
  Start Time        : 2019-Sep-19+00:33:30.426
  Failure Reason    : NPUMGR_TOO_MANY_CRASHES
  Failure Device    : CARD
  Is Card Usable    : Yes
  Recovery Status   : Success
  Facility          : N/A
  Instance          : N/A
  Duration          : 0.005 sec
  Graceful          : Enabled

Then, check the details of npumgr crashes. Here, you see that npumgr gets crashed at function nexthop_get. Hence, you see an indication of some issues when you try to get the next hop.

********************* CRASH #09 ***********************
SW Version          : 21.9.7
Similar Crash Count : 16
Time of First Crash : 2019-Sep-19+00:08:16

Assertion failure at npu/npumgr/ares_npumgr_forwarding_handler.c:1829
  Function: ares_npumgr_nexthop_get()
  Expression: (nh_id) >= 0 && (nh_id) < ares_npumgr_db_get_count(SN_NPUSHM_TABREC_NH, (ares_inst)->profile)
  Proclet: npumgr (f=103000,i=30)
  Process: card=3 cpu=0 arch=X pid=7066 cpu=~0% argv0=npumgr
  Crash time: 2019-Sep-18+19:01:11 UTC
  Recent errno: 11 Resource temporarily unavailable
  Build_number: 71001
  Stack (18024@0x0xffff0000):
    [ffffe430/X] __kernel_vsyscall() sp=0xffff0428
    [0c7df834/X] sn_assert() sp=0xffff0468
    [002fcedb/X] ares_npumgr_nexthop_get() sp=0xffff04b8
    [002feb23/X] ares_npumgr_fwd_ddf2_tcam_entry_update() sp=0xffff0948
    [00301896/X] ares_npumgr_lpm_add() sp=0xffff0e98
    [003c4345/X] ares_npumgr_fwd_add() sp=0xffff1768
    [003e38fa/X] ares_npumgr_fwd_func() sp=0xffff1bf8
    [003e444a/X] ares_sn_npumgr_forwarding_add_del_mod_handler() sp=0xffff2048
    [0c892918/X] sn_msg_arriving_handle() sp=0xffff4138
    [0c8713a6/X] sn_loop_run() sp=0xffff45e8
    [0c55a3b5/X] main() sp=0xffff4658
*******************************************************

********************* CRASH #10 ***********************
SW Version          : 21.9.7
Similar Crash Count : 1
Time of First Crash : 2019-Sep-19+00:31:22

Assertion failure at npu/npumgr/ares_npumgr_port_handler.c:8409
  Note: failed to find index of created lport 5/11#11-65: status=SN_STATUS_FAILURE[1]
  Function: ares_sn_npumgr_port_lp_create_func()
  Expression: 0
  Code: CRASH
  Proclet: npumgr (f=103000,i=11)
  Process: card=1 cpu=1 arch=X pid=7181 argv0=npumgr
  Crash time: 2019-Sep-18+19:01:22 UTC
  Recent errno: 11 Resource temporarily unavailable
  Build_number: 71001
  Stack (14728@0x0xffcb8000):
    [ffffe430/X] __kernel_vsyscall() sp=0xffcb8a48
    [0c7df834/X] sn_assert() sp=0xffcb8a88
    [003bd590/X] ares_sn_npumgr_port_lp_create_func() sp=0xffcb8f18
    [003c10d4/X] ares_sn_npumgr_port_lp_create_handler() sp=0xffcb9368
    [0c892918/X] sn_msg_arriving_handle() sp=0xffcbb458
    [0c8713a6/X] sn_loop_run() sp=0xffcbb908
    [0c55a3b5/X] main() sp=0xffcbb978
*******************************************************

********************* CRASH #11 ***********************
SW Version          : 21.9.7
Similar Crash Count : 107
Time of First Crash : 2019-Sep-19+00:09:03

Assertion failure at npu/npumgr/ares_npumgr_forwarding_handler.c:1829
  Function: ares_npumgr_nexthop_get()
  Expression: (nh_id) >= 0 && (nh_id) < ares_npumgr_db_get_count(SN_NPUSHM_TABREC_NH, (ares_inst)->profile)
  Proclet: npumgr (f=103000,i=80)
  Process: card=8 cpu=0 arch=X pid=9130 cpu=~98% argv0=npumgr
  Crash time: 2019-Sep-18+19:03:35 UTC
  Recent errno: 115 Operation now in progress
  Build_number: 71001
  Stack (10360@0x0xffe58000):
    [ffffe430/X] __kernel_vsyscall() sp=0xffe58618
    [0c7df834/X] sn_assert() sp=0xffe58658
    [002fcedb/X] ares_npumgr_nexthop_get() sp=0xffe586a8
    [002feb23/X] ares_npumgr_fwd_ddf2_tcam_entry_update() sp=0xffe58b38
    [00301896/X] ares_npumgr_lpm_add() sp=0xffe59088
    [003c4345/X] ares_npumgr_fwd_add() sp=0xffe59958
    [003e1191/X] fwddb_import_add_entry() sp=0xffe59dd8
    [003e2452/X] ares_npumgr_fwddb_import() sp=0xffe5a2c8
    [0025e4ea/X] npumgr_rx_db_evt() sp=0xffe5a2f8
    [0c8660d4/X] sn_epoll_run_events() sp=0xffe5a348
    [0c872bca/X] sn_loop_run() sp=0xffe5a7f8
    [0c55a3b5/X] main() sp=0xffe5a868
*******************************************************

You can check the activity logs and here is the chronology of events that happened. As part of an activity, interfaces are created followed by static routes.

show ipv6 interface summary

Thursday September 19 00:09:16 IST 2019
Interface Name                 Address/Mask        Port               Status
============================== =================== ================== ======
SGi_LAG100_vlan50              2401:4900:c:f::201/126 5/10 vlan 50       UP           [sec]
SGi_LAG100_vlan64_VO4G_SBC     2401:4900:c:10::1/126  5/10 vlan 64       UP    
SGi_LAG200_vlan51              2401:4900:c:f::205/126 5/11 vlan 51       UP           [sec]
SGi_LAG200_vlan65_VO4G_SBC     2401:4900:c:10::5/126  5/11 vlan 65       UP      

Total interface count: 4

(config-ctx)# ipv6 route a:b:c:d:1/128 next-hop x:y:z:w::2 interface A
Thursday September 19 00:07:13 IST 2019
(config-ctx)#
(config-ctx)# ipv6 route a:b:c:d:1/128 next-hop x:y:z:w::2 interface B
Thursday September 19 00:07:21 IST 2019
Failure: Invalid Nexthop address!
(config-ctx)#
(config-ctx)# ipv6 route a:b:c:d:1/128 next-hop x:y:z:w::6 interface C
Thursday September 19 00:07:36 IST 2019
(config-ctx)# exit
Thursday September 19 00:07:50 IST 2019
[SGi]MOH-C25-SPG-04(config)#

VLAN is then configured inside the port and opened for traffic around September 19 00:08:16.

(config)# port ethernet 5/10
Thursday September 19 00:08:01 IST 2019
(config-port-5/10)# vla
(config-port-5/10)# vlan 64
Thursday September 19 00:08:05 IST 2019
(config-port-5/10-vlan-64)# bind interface C SGi
Thursday September 19 00:08:14 IST 2019
(config-port-5/10-vlan-64)# no shu
(config-port-5/10-vlan-64)# no shutdown
Thursday September 19 00:08:17 IST 2019
(config-port-5/10-vlan-64)# exit
Thursday September 19 00:08:19 IST 2019
(config-port-5/10)# exit
Thursday September 19 00:08:21 IST 2019

Here, the steps and configurations of planned activity for creating interfaces and static routes are followed by bind inside VLAN looks fine. But soon after this, it could be seen that the npumgr started to crash followed by DPC card shutdowns due to too many npumgr crashes.

show snmp trap history verbose | grep -i mgr
Thursday September 19 00:20:22 IST 2019
Thu Sep 19 00:08:18 2019 Internal trap notification 73 (ManagerFailure) facility npumgr instance 30 card 3 cpu 0
Thu Sep 19 00:08:18 2019 Internal trap notification 150 (TaskFailed) facility npumgr instance 30 on card 3 cpu 0
Thu Sep 19 00:08:18 2019 Internal trap notification 73 (ManagerFailure) facility npumgr instance 40 card 4 cpu 0
Thu Sep 19 00:08:18 2019 Internal trap notification 150 (TaskFailed) facility npumgr instance 40 on card 4 cpu 0

As an immediate workaround, VLANs are removed from the port. Soon, after you remove VLANs, the npumgr crashes stop.

configure
Thursday September 19 00:29:31 IST 2019
(config)# port eth
(config)# port ethernet 5/10
Thursday September 19 00:33:13 IST 2019
(config-port-5/10)# no vlan 64
Thursday September 19 00:33:23 IST 2019
(config-port-5/10)# exit
Thursday September 19 00:33:38 IST 2019
(config)# port ethernet 5/11
Thursday September 19 00:33:42 IST 2019
(config-port-5/11)# no vlan 65
Thursday September 19 00:33:50 IST 2019
(config-port-5/11)# end
Thursday September 19 00:33:52 IST 2019


******** show crash list *******
Thursday September 19 03:54:39 IST 2019
=== ==================== ======== ========== =============== =======================
# Time Process Card/CPU/ SW HW_SER_NUM
PID VERSION MIO / Crash Card
=== ==================== ======== ========== =============== =======================
 9 2019-Sep-19+00:31:11 npumgr 03/0/07066 21.9.7 FLM221503A5/FLM221404FF
10 2019-Sep-19+00:31:22 npumgr 01/1/07181 21.9.7 FLM221503A5/FLM221404FH
11 2019-Sep-19+00:33:35 npumgr 08/0/09130 21.9.7 FLM221503A5/FLM221404FU

When you further check syslogs, it can be seen that the system tried to get next-hop but it was unsuccessful at Sep 19 00:08:16. i.e. soon after VLAN was opened for traffic.

Sep 19 00:08:16 10.107.211.36 evlogd: [local-60sec16.758] [npumgr-fwd 168001 error] [3/2/7024 <npumgr:32> rding_handler.c:7739] [software internal system syslog] ECMP create fail - no nexthop ARP: vpnid=6, type=ip[0], action=add[1], ip=2401:4900:c:f::408/128, tb_ver=3 ifentry-flag=0, ifroute-flag=0, updatettl-flag=1, nh=2401:4900:c:10::1 dpti=0 priority=0, tos=0, ttl-decr=1, nhlfe_ix=0, lport=0x050a000a
Sep 19 00:08:18 10.107.211.36 evlogd: [local-60sec18.448] [sitmain 4103 warning] [1/0/7008 <sitmain:10> sitproc.c:2653] [software internal system critical-info syslog] [npumgr pid 7102] async core transfer and proclet recovery is not set!
Sep 19 00:08:18 10.107.211.36 evlogd: [local-60sec18.852] [sitmain 4027 critical] [2/0/6993 <sitmain:20> crash_mini.c:602] [software internal system callhome-crash] Process Crash Info: time 2019-Sep-18+18:38:16(hex time 5d827998) card 02 cpu 00 pid 07146 procname npumgr   crash_details  Assertion failure at npu/npumgr/ares_npumgr_forwarding_handler.c:1829   Function: ares_npumgr_nexthop_get()   Expression: (nh_id) >= 0 && (nh_id) < ares_npumgr_db_get_count(SN_NPUSHM_TABREC_NH, (ares_inst)->profile)   Proclet: npumgr (f=103000,i=20)   Process: card=2 cpu=0 arch=X pid=7146 cpu=~0% argv0=npumgr   Crash time: 2019-Sep-18+18:38:16 UTC   Recent errno: 11 Resource temporarily unavailable   Build_number: 71001   Stack (20600@0x0xffce5000):     [ffffe430/X] __kernel_vsyscall() sp=0xffce5e38     [0c7df834/X] sn_assert() sp=0xffce5e78     [002fcedb/X] ares_npumgr_nexthop_get() sp=0xffce5ec8     [002feb23/X] ares_npumgr_fwd_ddf2_tcam_entry_update() sp=0xffce6358     [00301896/X] ares_npumgr_lpm_add() sp=0xffce68a8     [003c4345
Sep-18+18:38:16(hex time 5d827998) card 02 cpu 00 pid 07146 procname npumgr   crash_details  Assertion failure at npu/npumgr/ares_npumgr_forwarding_handler.c:1829   Function: ares_npumgr_nexthop_get()   Expression: (nh_id) >= 0 && (nh_id) < ares_npumgr_db_get_count(SN_NPUSHM_TABREC_NH, (ares_inst)->profile)   Proclet: npumgr (f=103000,i=20)   Process: card=2 cpu=0 arch=X pid=7146 cpu=~0% argv0=npumgr   Crash time: 2019-Sep-18+18:38:16 UTC   Recent errno: 11 Resource temporarily unavailable   Build_number: 71001   Stack (20600@0x0xffce5000):     [ffffe430/X] __kernel_vsyscall() sp=0xffce5e38     [0c7df834/X] sn_assert() sp=0xffce5e78     [002fcedb/X] ares_npumgr_nexthop_get() sp=0xffce5ec8     [002feb23/X] ares_npumgr_fwd_ddf2_tcam_entry_update() sp=0xffce6358     [00301896/X] ares_npumgr_lpm_add() sp=0xffce68a8     [003c4345

When you further check the configuration from SSD, it can also be seen that one static route already exists before the start of the planned activity (interface and static route configurations).

context SGi
  ipv6 route a:b:c:d:1/128 next-hop x:y:z:w::1 interface C
#exit

From the configuration, it can be seen that there was already a static route present for IP a:b:c:d:1/128 via interface C next-hop as hop x:y:z:w::1. But as part of the activity, one more static route is defined with next-hop as hop x:y:z:w::2.

So, when the VLANs are opened for traffic, the system can not get the next hop x:y:z:w::1 as it was defined first. Also, there were logs that indicated Equal-Cost Multi-Path (ECMP) routing to next-hop not successful because it is not reachable. As a result, it was not able to forward packets of these VLANs traffic which eventually lead to npumgr crashes.

Multiple Card switchover is a by-product of too many npumgr crashes on the system.

Solution

There are multiple static routes to the same destination via the same interface but different next-hop that leads to the npumgr is not able to forward packets followed by npumgr crashes.

Hence, the incorrect static route is removed from the configuration. The same configurations are then successfully applied in another maintenance window without any issues.