Troubleshoot SERDES Lane in ASR 5500
Available Languages
Download Options
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
Contents
Introduction
This document describes troubleshoot commands for the Serializer Deserializer (SERDES) lane(link) in the ASR 5500.
Background Information
The ASR 5500 contains SERDES links between the cards in order to facilitate communication and data paths between the Fabric and Storage Card (FSC), Data Processing Card (DPC), and Management Input/Output (MIO) cards. At times, these SERDES links can go down due to errors or hardware failure.
SERDES Lane Troubleshooting Commands
Commands to investigate the SERDES Lanes of the ASR 5500 chassis:
- Collect the
show support details, look in the "debug console..." portion for the lines of output:
1397273780.205 card 5-cpu0: afio [5/0/7808] [ 80616.933] afio/afio_fe600_serdes.c:3297: #1: fe600=47=16/1, Fabric SERDES lane transitioned from up to down, serdes=29, devid=25=7/1
- Access test mode of the chassis CLI
cli test-commands password <password>.
Caution: Use of this mode can cause significant service interruption
- Issue
show fabric healthcommand for an overall picture of the switching fabric.
Tip: The same information can be obtained from the show fabric support details part of the show support details
In the example, there is an issue between DPC card 2 and FSC card 14.
In the output, the fault is reported from source DPC in slot 2 towards FSC in slot 14:
Command: petra-b system-device-id 3
Command: show health
Petra-B 3=2/1
Fabric Status:
Status OK(+)------------------------+
Topology fault(T)------------------+|
Far side not expected(*)----------+||
Logically not connected(L)-------+|||
Physically not connected(P)-----+|||| NIF Status:
Rx Down(*)---------------------+||||| +----------NIF powered off(*)
Tx Down(*)--------------------+|||||| |+---------SERDES powered off(*)
Code Group(G)----------------+||||||| ||+--------Local side down(l)
Misalignment(M)-------------+|||||||| |||+-------Remote side down(r)
Cell Size(C)---------------+||||||||| ||||+------Rx activity(r)
Internally fixed(I)-------+|||||||||| |||||+-----Tx activity(t)
Not Accept Cells(A)------+||||||||||| ||||||+----Status OK(+)
|||||||||||| |||||||
SERDES Status: |||||||||||| |||||||
Status OK(+)-----------+ |||||||||||| |||||||
Rx power off(*)-------+| |||||||||||| |||||||
Tx power off(*)------+|| |||||||||||| |||||||
Sig not locked(S)---+||| |||||||||||| |||||||
Rx signal loss(*)--+|||| |||||||||||| |||||||
Modified Parms(m)-+||||| |||||||||||| |||||||
Admin down(D)----+|||||| |||||||||||| |||||||
||||||| |||||||||||| |||||||
Fabric lane-----+ ||||||| |||||||||||| |||||||
SERDES lane--+ | ||||||| |||||||||||| ||||||| Config
Source Dev SL FL vvvvvvv vvvvvvvvvvvv vvvvvvv Rate Topology CRC Errs Remote Dev SL FL Last Change
------- --- -- -- ------- ------------ ------- ------------ -------- -------- ------- --- -- -- ----------------------
3= 2/1 FAP 47 15 + A M L 6250.00 Mbps - - 43=14/1 FE 82 82 FAULT_DETECTED ***
In the output for the same link in the other direction from FSC card in slot 14 to DPC card in slot 2 the same error is reported:
Command: fe600 system-device-id 43
Command: show health
FE600 43=14/1
Fabric Status:
Status OK(+)------------------------+
Topology fault(T)------------------+|
Far side not expected(*)----------+||
Logically not connected(L)-------+|||
Physically not connected(P)-----+|||| NIF Status:
Rx Down(*)---------------------+||||| +----------NIF powered off(*)
Tx Down(*)--------------------+|||||| |+---------SERDES powered off(*)
Code Group(G)----------------+||||||| ||+--------Local side down(l)
Misalignment(M)-------------+|||||||| |||+-------Remote side down(r)
Cell Size(C)---------------+||||||||| ||||+------Rx activity(r)
Internally fixed(I)-------+|||||||||| |||||+-----Tx activity(t)
Not Accept Cells(A)------+||||||||||| ||||||+----Status OK(+)
|||||||||||| |||||||
SERDES Status: |||||||||||| |||||||
Status OK(+)-----------+ |||||||||||| |||||||
Rx power off(*)-------+| |||||||||||| |||||||
Tx power off(*)------+|| |||||||||||| |||||||
Sig not locked(S)---+||| |||||||||||| |||||||
Rx signal loss(*)--+|||| |||||||||||| |||||||
Modified Parms(m)-+||||| |||||||||||| |||||||
Admin down(D)----+|||||| |||||||||||| |||||||
||||||| |||||||||||| |||||||
Fabric lane-----+ ||||||| |||||||||||| |||||||
SERDES lane--+ | ||||||| |||||||||||| ||||||| Config
Source Dev SL FL vvvvvvv vvvvvvvvvvvv vvvvvvv Rate Topology CRC Errs Remote Dev SL FL Last Change
------- --- -- -- ------- ------------ ------- ------------ -------- -------- ------- --- -- -- ----------------------
43=14/1 FE 82 82 + L T 6250.00 Mbps 3= 2/1 - 3= 2/1 FAP 47 15 FAULT_DETECTED ***
Another type of issue with the SERDES link is the offline state of the link. In the example, the link between DPC card in slot 6 and FSC card in 17 is offline:
23= 6/3 FAP 38 6 D 6250.00 Mbps 50=17/2 1557643 50=17/2 FE 65 65 OFFLINE ***
The total number of active SERDES links and the number of active links is shown in the output of the show fabric status command. In the shown example, two links were counted down, one for each side of the link. One lane down is not a problem. There is a lot of excess fabric capacity, and a single lane does not impact the throughput. The only issue is if the links continually go up and down because of errors, in which case it is possible that user and control traffic gets dropped, and that is why it is better if the link is taken down.
[local]ASR5500> show fabric status
Total number of FAPs: 24
Total number of FEs : 8
Total number of SERDES links: 1600
Total number of active SERDES links: 1598
Note: There is a lot of excess fabric capacity, and a single lane does not impact the throughput of the chassis.
- Normally, the link issue clears on its own. The system goes through the automatic steps called the Eyescan to resolve the issue. The results of the automatic recovery is reported in the
show serdes all-serdes historysections of theshow fabric support details
Note: The FE (Fabric Element) is the FSC card side. The FAP (Fabric Array Processor) is the DPC and/or MIO card side.
The DPC cards have 2 FAPs, the DPC2 cards have only 1 FAP; the MIO cards have 4 FAPs, and FSCs have 2 FEs.
The format in the output of commands is <card #>/<FAP/FE #>, for example, MIO 5 would have 5/1, 5/2, 5/3, 5/4.
A fully loaded DPC2 chassis would have 28 endpoints: 8 (8 DPC) + 8 (2 MIO * 4) + 12 ( 6 FCS * 2)
The example of the FE side that was restored after automatic recovery is shown:
card=5, cpu=0, pid=7808, peer_mode=AFIO_IPC_PEER_MODE_DAEMON, sys_dev_id=47=16/1
Fabric Status:
Topology fault(T)----------------+
Far side not expected(*)--------+|
Logically not connected(L)-----+||
Physically not connected(P)---+|||
Rx Down(*)-------------------+||||
Tx Down(*)------------------+||||| NIF Status:
Code Group(G)--------------+|||||| +--------NIF powered off(*)
Misalignment(M)-----------+||||||| |+-------SERDES powered off(*)
Cell Size(C)-------------+|||||||| ||+------Local side down(l)
Internally fixed(I)-----+||||||||| |||+-----Remote side down(r)
Not Accept Cells(A)----+|||||||||| ||||
||||||||||| ||||
SERDES Status: ||||||||||| ||||
Rx power off(*)------+ ||||||||||| ||||
Tx power off(*)-----+| ||||||||||| ||||
Sig not locked(S)--+|| ||||||||||| ||||
Rx signal loss(*)-+||| ||||||||||| ||||
Admin Down(D)----+|||| ||||||||||| ||||
||||| ||||||||||| ||||
Fabric lane-----+ ||||| ||||||||||| ||||
SERDES lane--+ | ||||| ||||||||||| ||||
Record time Source Dev SL FL vvvvv vvvvvvvvvvv vvvv Remote Dev SL FL CRC Errs Last Change
------------------- ------- --- -- -- ----- ----------- ---- ------- --- -- -- -------- ----------------------
2014-05-18+12:38:17 47=16/1 FE 40 40 I 31= 8/1 FAP 43 11 1 CRC_ERROR
2014-05-18+12:39:27 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_DOWN
2014-05-18+12:39:28 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_START
2014-05-18+13:14:41 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_COMPLETE
2014-05-18+13:14:50 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_UP
Recovery of another side of the line is shown in the example:
card=5, cpu=0, pid=7808, peer_mode=AFIO_IPC_PEER_MODE_DAEMON, sys_dev_id=47=16/1
Fabric Status:
Topology fault(T)----------------+
Far side not expected(*)--------+|
Logically not connected(L)-----+||
Physically not connected(P)---+|||
Rx Down(*)-------------------+||||
Tx Down(*)------------------+||||| NIF Status:
Code Group(G)--------------+|||||| +--------NIF powered off(*)
Misalignment(M)-----------+||||||| |+-------SERDES powered off(*)
Cell Size(C)-------------+|||||||| ||+------Local side down(l)
Internally fixed(I)-----+||||||||| |||+-----Remote side down(r)
Not Accept Cells(A)----+|||||||||| ||||
||||||||||| ||||
SERDES Status: ||||||||||| ||||
Rx power off(*)------+ ||||||||||| ||||
Tx power off(*)-----+| ||||||||||| ||||
Sig not locked(S)--+|| ||||||||||| ||||
Rx signal loss(*)-+||| ||||||||||| ||||
Admin Down(D)----+|||| ||||||||||| ||||
||||| ||||||||||| ||||
Fabric lane-----+ ||||| ||||||||||| ||||
SERDES lane--+ | ||||| ||||||||||| ||||
Record time Source Dev SL FL vvvvv vvvvvvvvvvv vvvv Remote Dev SL FL CRC Errs Last Change
------------------- ------- --- -- -- ----- ----------- ---- ------- --- -- -- -------- ----------------------
2014-05-18+12:38:17 47=16/1 FE 40 40 I 31= 8/1 FAP 43 11 1 CRC_ERROR
2014-05-18+12:39:27 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_DOWN
2014-05-18+12:39:28 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_START
2014-05-18+13:14:41 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 EYESCAN_COMPLETE
2014-05-18+13:14:50 47=16/1 FE 40 40 31= 8/1 FAP 43 11 1 ADMIN_UP
- In the StarOS release 16.1 and higher, the system has the ability to generate SNMP traps when a configured Egress Queue Discard (EGQ) threshold is observed by the chassis. The example of the commands used to set the threshold to 50 EGQ Discards per 30-second period is shown.
[local]asr5500# config
[local]asr5500(config)# fabric egress drop-threshold enable count 50 interval-secs 30
Manual Recovery
When a SERDES link has not been restored after the Eyescan testing and reprogramming, then manual recovery is necessary. Unfortunately with software, we are not able to determine which side of the SERDES link is at fault. We have to take a methodical approach to fix the issue.
Caution: Steps 1 and 2 are mandatory before RMA
- Reseat one card first. Slide out the card and inspect the card backplane for damaged and bent pins on the card and in the chassis’s backplane.
- If damaged and bent pins are observed then take pictures and raise Service Request (SR) with Cisco TAC.
- Monitor for 72 hours.
- If the issue returns, go to Step 2.
- If it clears, the issue is resolved.
- Reseat the other card. Slide out the card and inspect the card backplane for damaged and bent pins on the card and in the chassis’s backplane.
- If damaged and bent pins are observed then take pictures and raise Service Request (SR) with Cisco TAC.
- Monitor for 72 hours.
- If the issue returns, go to step 3.
- If it clears, the issue is resolved.
- Open SR with Cisco TAC and attach collected the show support details before and after cards reseat, and pictures of damaged or bent pins.
When the issue is resolved, the show fabric status looks like this:
[local]ASR5500> show fabric status
Total number of FAPs: 24
Total number of FEs : 8
Total number of SERDES links: 1600
Total number of active SERDES links: 1600
An SNMP trap SERDESLanePermenentlyDown has now been implemented to indicate when a SERDES lane has gone down permanently due to Eyescan failures:
Sun Apr 17 00:05:00 2016 Internal trap notification 1303 (SERDESLanePermanentlyDown) SERDES lane is Down on local: slot 17 device 2 serdes lane index 14, Remote: slot 1 device 1 serdes lane index 40
[local]ASR5500> show fabric status
Total number of FAPs: 16
Total number of FEs : 12
Total number of SERDES links: 1456
Total number of active SERDES links: 1454
Total number of Fabric SERDES with errors: 0
Total number of NIF SERDES with errors : 0
[local]ASR5500> show fabric history
Command: arad system-device-id 1
Command: show serdes all-serdes history
Fabric Status:
+------Not Accept Cells(A)
SERDES Status: |+-----Cell Size(C)
Power off(*)-------------+ ||+----Misalignment(M)
Sig not locked(S)-------+| |||+---Code Group(G)
Admin down(D)----------+|| ||||+--Topology fault(T)
||| |||||
Logical Port---------+ ||| |||||
Fabric lane-----+ | ||| |||||
SERDES lane--+ | | ||| |||||
Record time Source Dev SL FL | vvv vvvvv Remote Dev SL FL CRC Errs Last Change
------------------- ------- --- --- --- --- --- ----- ------- --- --- --- -------- ----------------------
2016-04-16+23:53:05 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED
2016-04-16+23:53:14 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_DOWN
2016-04-16+23:57:02 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_UP
2016-04-16+23:57:02 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED
2016-04-16+23:57:11 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_DOWN
2016-04-17+00:00:59 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_UP
2016-04-17+00:00:59 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED
2016-04-17+00:01:08 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_DOWN
2016-04-17+00:05:00 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - ADMIN_UP
2016-04-17+00:05:00 1= 1/1 FAP 40 8 264 T 42=17/2 FE 14 14 - FAULT_DETECTED
...
Command: fe600 system-device-id 42
Command: show serdes all-serdes history
NIF Status: Fabric Status:
Remote side down(r)-------+ +------------Not Accept Cells(A)
Local side down(l)-------+| |+-----------Internally fixed(*)
SERDES powered off(*)---+|| ||+----------Cell Size(C)
NIF powered off(*)-----+||| |||+---------Misalignment(M)
|||| ||||+--------Code Group(G)
SERDES Status: |||| |||||+-------Tx Down(*)
Rx power off(*)------+ |||| ||||||+------Rx Down(*)
Tx power off(*)-----+| |||| |||||||+-----Physically not connected(P)
Sig not locked(S)--+|| |||| ||||||||+----Logically not connected(L)
Rx signal loss(*)-+||| |||| |||||||||+---Far side not expected(*)
Admin down(D)----+|||| |||| ||||||||||+--Topology fault(T)
||||| |||| |||||||||||
Fabric lane-----+ ||||| |||| |||||||||||
SERDES lane--+ | ||||| |||| |||||||||||
Record time Source Dev SL FL vvvvv vvvv vvvvvvvvvvv Remote Dev SL FL CRC Errs Last Change
------------------- ------- --- -- -- ----- ---- ----------- ------- --- --- --- -------- ----------------------
2016-04-16+23:57:01 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - FAULT_DETECTED
2016-04-16+23:57:11 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - ADMIN_DOWN
2016-04-16+23:57:11 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_START
2016-04-17+00:00:52 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_FAILURE
2016-04-17+00:00:55 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - ADMIN_UP
2016-04-17+00:00:58 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - FAULT_DETECTED
2016-04-17+00:01:08 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - ADMIN_DOWN
2016-04-17+00:01:08 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_START
2016-04-17+00:04:56 42=17/2 FE 14 14 *S A M PL T 1= 1/1 FAP 40 8 - EYESCAN_FAILURE
2016-Apr-17+00:05:00.023 [snmp 22002 info] [5/0/7150 <afctrl:0> trap_api.c:17297] [software internal system syslog] Internal trap notification 1303 (SERDESLanePermanentlyDown) SERDES lane is Down on local: slot 17 device 2 serdes lane index 14, Remote: slot 1 device 1 serdes lane index 40
2016-Apr-17+00:05:00.023 [afctrl 186019 critical] [5/0/7150 <afctrl:0> l_msg_handler.c:1541] [hardware internal system syslog] Fabric device 17/2, serdes lane index 14, (remote fabric device 1/1, serdes lane index 40) is Administratively offline due to excessive calibration failures
2016-Apr-16+23:41:09.247 [system 1009 warning] [6/0/10430 <evlogd:1> evlgd_syslogd.c:162] [software internal system critical-info syslog] CPU[5/0]: afio: afio [5/0/9285] [ 426721.037] afio/afio_fe600_serdes.c:2827: #1: fe600=42=17/2, Fabric SERDES lane transitioned from up to down, serdes=14, devid=1=1/1, serdes=40
2016-Apr-16+23:41:09.247 [system 1009 warning] [5/0/7073 <evlogd:0> evlgd_syslogd.c:162] [software internal system critical-info syslog] CPU[5/0]: afio: afio [5/0/9285] [ 426721.037] afio/afio_fe600_serdes.c:2827: #1: fe600=42=17/2, Fabric SERDES lane transitioned from up to down, serdes=14, devid=1=1/1, serdes=40
Related Information
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
19-Aug-2022 |
Initial Release |
Contributed by Cisco Engineers
- Mike LugoCisco TAC Technical Leader
- Dave DamerjianCisco TAC Technical Leader
- Dennis LanovCisco TAC Technical Leader
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)