Contents

Introduction

Prerequisites

Requirements

Components Used

Problem

Troubleshoot

Solution

Automatic Recovery Mechanism

Introduction

This document describes how to troubleshoot Fabric and storage Card (FSC) card when "ThreshFabricEGQDiscards" SNMP trap.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- StarOs

- ASR5500 Platform

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Problem

Error "ThreshFabricEGQDiscards" is noticed when Single Event Upset (SEU) on the fabric edge (FE) chipset on one FSC card in an ASR5500 chassis. Due to this bit flip in FE tables, the FE chip starts to corrupt packets (cells) in the fabric causing Egress Queue Discards which leads to heartbeat failures between the cards.

You can see example of this problem using the Command line interface (CLI) command show snmp trap history verbose.

Sat Jan 02 03:59:30 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 9 device 2 threshold 50 measured value 2430 interval 30

Sat Jan 02 03:59:30 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 9 device 1 threshold 50 measured value 2096 interval 30

Sat Jan 02 03:59:40 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 5 device 4 threshold 50 measured value 481 interval 30

Sat Jan 02 03:59:40 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 4 device 2 threshold 50 measured value 3761 interval 30

Sat Jan 02 03:59:40 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 4 device 1 threshold 50 measured value 3660 interval 30

Sat Jan 02 03:59:40 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 5 device 2 threshold 50 measured value 173 interval 30

Sat Jan 02 03:59:40 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 5 device 1 threshold 50 measured value 133 interval 30

Sat Jan 02 03:59:42 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 8 device 2 threshold 50 measured value 2977 interval 30

Sat Jan 02 03:59:42 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 8 device 1 threshold 50 measured value 4310 interval 30

Sat Jan 02 03:59:44 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 3 device 1 threshold 50 measured value 4499 interval 30

Sat Jan 02 03:59:44 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 3 device 2 threshold 50 measured value 4091 interval 30

Sat Jan 02 03:59:45 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 10 device 1 threshold 50 measured value 2796 interval 30

Sat Jan 02 03:59:45 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 10 device 2 threshold 50 measured value 5418 interval 30

Sat Jan 02 03:59:47 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 1 device 2 threshold 50 measured value 4747 interval 30

Sat Jan 02 03:59:47 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 1 device 1 threshold 50 measured value 5243 interval 30

Sat Jan 02 03:59:49 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 7 device 2 threshold 50 measured value 4644 interval 30

Sat Jan 02 03:59:49 2016 Internal trap notification 523 (ThreshFabricEGQDiscards) slot 7 device 1 threshold 50 measured value 5017 interval 30

This line is seen under multiple cards cpu consoles:

Note: command debug console card is hidden/test command. This command is also collected every time for all cards on ASR5500 when show support details command is run on StarOs node.

******** debug console card 1 cpu 0 tail 10000 only *******

Saturday January 02 05:45:38 EST 2016

[...]

2016-Jan-02+03:59:47.479 card 1-cpu0: afio [1/0/2701] [2862193.674] afio/afio_petrab_egress.c:121: #1: petrab=1=1/1, PetraB EGQ Egress drop threshold exceeded, drop count=5243, interval=30 secs, threshold=50

Troubleshoot

Check if egress drops are incrementing.

Note: If fabric errors are incrementing and you are running StarOs node on version release 19.0 or higher then proceed to the Solution section in this article.

Note: If fabric errors are incrementing and you are running StarOs node version bellow version release.19.0 please raise Service Reuqest towards TAC.

Step 1. Enter test mode, here is documentation how to enable it on StarOs node.

cli test-commands [encrypted] password password

Step 2. Check fabric health.

show fabric health | grep -i -E "^Petra-B|EGQ"

Example of output when problem is not present:

[local]#show fabric health | grep -i -E "^Petra-B|EGQ"

Petra-B 1=1/1

Petra-B 2=1/2

Petra-B 3=2/1

Petra-B 4=2/2

Petra-B 5=3/1

Petra-B 6=3/2

[...]

Example of output where you see increese in EGQ Discard Packets:

[local]#show fabric health | grep -i -E "^Petra-B|EGQ"

Petra-B 1=1/1

EGQ.RqpDiscardPacketCounter                         1143278

EGQ.EhpDiscardPacketCounter                         1143278

EGQ.PqpDiscardUnicastPacketCounter                  1143278

Petra-B 2=1/2

EGQ.RqpDiscardPacketCounter                         1068491

EGQ.EhpDiscardPacketCounter                         1068491

EGQ.PqpDiscardUnicastPacketCounter                  1068491

[local]#show fabric health | grep -i -E "^Petra-B|EGQ"

Petra-B 1=1/1

EGQ.RqpDiscardPacketCounter                         1346022 <<<

EGQ.EhpDiscardPacketCounter                         1346022 <<<

EGQ.PqpDiscardUnicastPacketCounter                  1346022 <<<

Petra-B 2=1/2

EGQ.RqpDiscardPacketCounter                         1271360 <<<

EGQ.EhpDiscardPacketCounter                         1271360 <<<

EGQ.PqpDiscardUnicastPacketCounter                  1271360 <<<

Solution

Automatic Recovery Mechanism

Type of behavior change:

New CLI command to enable FSC auto recovery/reset procedure upon detecting excessive fabric egress discards

Release introduced:

19.0

Old behavior:

Manual recovery process to reset FSCs.

New behavior:

New CLI configuration commands, please check documentation:

fabric fsc auto-recovery enable max-attempts <X> to enable this feature.

max-attempts is the number of times it resets each FSCs. By default, max-attempts is unlimited.

fabric fsc auto-recovery disable to disable this feature.

show afctrl fsc-auto-recovery displays details about FSC auto recovery, including devices yet to be reset, reset count, max attempts, Egress Drop Threshold state, and FSC auto recovery history.

Caution: Impact on customer: FSC FE devices are reset and any packets in flight are lost.

Note: All values except the history are replicated when the MIO fails over.