Troubleshooting ASR 5X00 Series Hardware Failures
Available Languages
Download Options
Updated:September 11, 2015
Document ID:119146
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
Introduction
This document describes the procedure to troubleshoot issues on Cisco Aggregation Services Router (ASR) 5000 Series and 5500 Series line cards. This procedure highlights some of the commands and the keywords, obtained from the SSD (Show Support Detail) file generated on the ASR 5000 or ASR 5500 chassis. The output of each command provides symptoms of a failed card and helps to pinpoint the problem. The objective here is not to address specific issues on a particular line card, as there are numerous different hardware problems. Moreover, individual situations might vary just enough which makes any generalized troubleshooting guide problematic.
Procedure
- Enter the show alarm outstanding command.
The output of this command displays the failed card. This information can be found in the SSD.
******** show alarm outstanding *******
Sev Object Event
--- ------- -------------------------------------------------------------------------------------
CR Card 7 Card Failure Detected on card 7 Device CPU_0 failure reason CARD_BOOT_TIMEOUT_EXPIRED - Enter the show snmp trap history verbose command.
Dependent upon the issue, traps are generated on the hardware failure that occurred. This information is part of the SSD.
******** show snmp trap history verbose *******
Timestamp Trap Information
------------------------ ----------------------------------------------------
Tue Jan 28 08:24:18 2014 Internal trap notification 9 (CardBootFailed) card 7 type Packet Services Card - Enter the show logs level error command.
Dependent upon the issue, logs are generated on the hardware failure that occurred. This information can be found in the SSD.
******** show logs level error *******
2014-Jan-28+08:39:35.669 [hat 3018 critical] [8/0/4429 <hatsystem:0> atsystem_fail.c:1192]
[hardware internal system syslog] Card Failure Detected on card 7 Device CPU_0
failure reason CARD_BOOT_TIMEOUT_EXPIRED - Enter the show card hardware command.
The output of the command has version information on the card. This information is included in the SSD.
******** show card hardware *******
Card 7:
Card Type : Packet Services Card (R01)
Description : PSC
Starent Part Number : 530-02-0030 09
Starent Serial Num : PLB43074818
Switch Fabric Modes : control plane, switch fabric
(Version Information Unavailable) <--- Missing information due to card not accessible - Enter the show card diag command.
The output of the command includes the current and last failure of the card as well as whether the card is usable. This information is included in the SSD.
******** show card diag *******
Card 7:
Counters:
Successful Warm Boots : 2
(last at Friday March 02 15:02:37 EST 2012)
Successful Cold Boots : 21
(last at Monday December 16 14:09:24 EST 2013)
Total Boot Attempts : 0
In Service Date : Fri Jan 20 15:26:25 2012 (Estimated)
Status:
IDEEPROM Magic Number : Good
Boot Mode : Normal
Card Diagnostics : Pass
Current Failure : Failure: Device=CPU_0, Reason=CARD_BOOT_TIMEOUT_EXPIRED,
(0x03001000)
(last at Tuesday January 28 08:39:35 EST 2014)
Last Failure : Failure: Device=CPU_0, Reason=CARD_BOOT_TIMEOUT_EXPIRED,
(0x03001000)
(last at Tuesday January 28 08:39:35 EST 2014)
Card Usable : No (Current Failure)
Current Environment:
Temperature: Card : 23 C (limit 101 C)
Temperature: CPU0 : 23 C (limit 101 C)
Temperature: CPU1 : 25 C (limit 101 C)
Temperature: LM93 : 25 C (limit 101 C)
Voltage: 3.3V STANDBY : 3.285 V (min 3.130 V, max 3.458 V) - Enter the show task resources command.
The output of the command does not display any information on the failed card. This information can be found in the SSD.
******** show task resources *******
task cputime memory files sessions
cpu facility inst used allc used alloc used allc used allc S status
----------------------- --------- ------------- --------- ------------- ------
6/1 rmmgr 61 3.6% 10% 4.5M 23.0M 15 500 -- -- - good
6/1 npumgr 6 1.5% 100% 81.4M 278.0M 26 1000 -- -- - good
6/1 sitreap 6100841 0.0% 5.0% 0.1M 15.0M 4 100 -- -- - good
6/1 msgd 6100839 0.8% 7.0% 0.8M 15.0M 6 500 -- -- - good
6/1 nscontrol 6123706 0.2% 5.0% 1.4M 15.0M 9 500 -- -- - good
<<<<<<<< 7/1 missing due to card 7 not running >>>>>>>>>
8/0 sitmain 80 0.0% 15% 10.9M 16.0M 14 1000 -- -- - good
8/0 sitparent 80 0.1% 20% 10.5M 14.0M 11 500 -- -- - good
8/0 evlogd 0 0.1% 95% 12.4M 70.0M 16 4000 -- -- - good
8/0 drvctrl 0 2.7% 15% 14.7M 20.0M 19 500 -- -- - good
8/0 hatsystem 0 0.4% 10% 10.2M 15.0M 13 500 -- -- - good
8/0 hatcpu 80 0.1% 10% 10.1M 15.0M 12 500 -- -- - good - Enter the show cpu table command.
The output of the command does not display any information on the failed card. This information is part of the SSD.
******** show cpu table *******
--------Load-------- ------CPU-Usage----- ---------Memory--------
cpu state now 5min 15min now 5min 15min now 5min 15min total
---- ----- ------ ------ ------ ------ ------ ------ ----- ----- ----- -----
1/0 Actve 0.02 0.02 0.05 0.6% 0.5% 0.5% 842M 842M 842M 16.0G
1/1 Actve 2.39 2.48 2.59 11.7% 7.9% 7.9% 138M 137M 137M 512M
2/0 Actve 0.02 0.07 0.11 0.5% 0.5% 0.5% 887M 885M 885M 16.0G
2/1 Actve 2.83 2.78 2.66 10.4% 7.9% 8.0% 138M 137M 137M 512M
3/0 Actve 0.02 0.06 0.11 0.4% 0.4% 0.4% 824M 823M 823M 16.0G
3/1 Actve 3.71 2.96 2.80 13.4% 8.0% 7.8% 139M 137M 137M 512M
4/0 Actve 0.03 0.10 0.13 0.5% 0.5% 0.5% 845M 845M 845M 16.0G
4/1 Actve 2.34 2.55 2.61 11.8% 7.7% 7.7% 138M 137M 137M 512M
5/0 Actve 0.14 0.11 0.13 0.6% 0.5% 0.5% 887M 886M 885M 16.0G
5/1 Actve 2.49 2.84 2.75 12.7% 8.4% 7.9% 138M 137M 137M 512M
6/0 Actve 0.18 0.16 0.14 0.5% 0.4% 0.4% 824M 823M 822M 16.0G
6/1 Actve 2.85 2.78 2.74 12.5% 8.0% 8.0% 138M 137M 137M 512M
<<<<<<<< 7/1 missing due to card 7 not running >>>>>>>>>>
8/0 Actve 0.20 0.09 0.12 18.9% 6.4% 5.2% 1045M 1011M 1009M 4096M
9/0 Sndby 0.02 0.03 0.08 2.0% 0.7% 0.7% 605M 604M 604M 4096M - Enter the show card table all command.
The output of the command indicates the card is offline. This information is part of the SSD.
******** show card table all *******
Slot Card Type Oper State SPOF Attach
----------- -------------------------------- ------------- ---- ------
1: PSC Packet Services Card Active Yes 17 -
2: PSC Packet Services Card A Active Yes 18 -
3: PSC Packet Services Card A Active Yes 19 -
4: PSC Packet Services Card A Active Yes 20 -
5: PSC Packet Services Card A Active Yes 21 -
6: PSC Packet Services Card A Active Yes 22 -
7: PSC Packet Services Card Offline - - -
8: SMC System Management Card Active No 24 25
9: SMC System Management Card Standby Yes - -
10: PSC None - - - -
11: PSC None - - - -
12: PSC None - - - -
13: PSC None - - - -
14: PSC None - - - -
15: PSC None - - - -
16: PSC None - - - -
17: LC 10 Gig Ethernet Line Card Active Yes 1
18: LC 10 Gig Ethernet Line Card Active Yes 2
19: LC 10 Gig Ethernet Line Card Active Yes 3
20: LC 10 Gig Ethernet Line Card Active Yes 4
21: LC 1000 Ethernet Line Card Standby - 5
22: LC 1000 Ethernet Line Card Standby - 6
23: LC None - - -
24: SPIO Switch Processor I/O Card Active No 8
25: SPIO Switch Processor I/O Card Standby - 8
26: LC None - - -
27: LC None - - -
28: LC None - - -
29: LC None - - -
30: LC None - - -
31: LC None - - -
32: LC None - - -
33: LC None - - -
34: LC None - - -
35: LC None - - -
36: LC None - - -
37: LC None - - -
38: LC None - - -
39: LC None - - -
40: RCC Redundancy Crossbar Card Standby -
41: RCC Redundancy Crossbar Card Standby -
42: LC None - - -
43: LC None - - -
44: LC None - - -
45: LC None - - -
46: LC None - - -
47: LC None - - -
48: LC None - Enter the show hardware version command.
The output of the command displays "no data available" as the card is offline. This information is part of the SSD.
******** show hardware version *******
Slot Type SSCB PSR PSR2 BIOS A BIOS B DT DT2
---- ---- ------- --- ---- -------- -------- ------- --------
1: PSC 1.6 2 - 9.2.13 7.8.14 3.16 --
2: PSCA 1.6 - 0 1.1.10 1.0.14 -- 3.20
3: PSCA 1.6 - 0 1.1.10 1.0.14 -- 3.20
4: PSCA 1.6 - 0 1.1.10 1.0.20 -- 3.20
5: PSCA 1.6 - 0 1.1.10 1.0.14 -- 3.20
6: PSCA 1.6 - 0 1.1.10 1.0.14 -- 3.20
7: PSC (no data available)
10: None -- - - -- -- -- --
11: None -- - - -- -- -- --
12: None -- - - -- -- -- --
13: None -- - - -- -- -- --
14: None -- - - -- -- -- --
15: None -- - - -- -- -- --
16: None -- - - -- -- -- -- - Enter the show npuctrl events command.
The output of the command displays the status of the failed cards as offline, booting (if in loop), and so on. This information is part of the SSD.
******** show npuctrl events *******
[41]@01-28-2014 08:26:15.892 e/i=0/0 evt=exporter-done transition=await-exporter->normal spc=no
inst| npumgr-sta| card-state| card-inser inst|npumgr-sta| card-state| card-inser
1| normal| standby| yes 2| normal| standby| yes
3| normal| standby| yes 4| normal| standby| yes
5| export-db| standby| yes 6| import-db| initializi| yes
7| absent| booting| no 8| normal| active| no
9| normal| standby| yes 10| absent| empty| no
11| absent| empty| no 12| absent| empty| no
13| absent| empty| no 14| absent| empty| no
15| absent| empty| no 16| absent| empty| no
[42]@01-28-2014 08:54:08.041 e/i=6/1 evt=start-recovery transition=normal->await-importer-ack spc=no
inst| npumgr-sta| card-state| card-inser inst|npumgr-sta| card-state| card-inser
1| power-up| initializi| no 2| normal| active| yes
3| normal| active| yes 4| normal| active| yes
5| normal| active| yes 6| normal| active| yes
7| absent| offline| no 8| normal| active| no
9| normal| standby| yes 10| absent| empty| no
11| absent| empty| no 12| absent| empty| no
13| absent| empty| no 14| absent| empty| no
15| absent| empty| no 16| absent| empty| no - Enter the show temperature command.
The output of the command displays the temperature of only a few components of the failed card. This information is part of the SSD.
******** show temperature *******
Note:
Cards 6 and 7 are identical PSC cards but more components of good card are
shown than that of failed card.
Card 6: 29/101 C (LM94 )
58/115 C (NPU )
43/101 C (NPU PCB )
44/101 C (DT )
30/101 C (Midplane)
41/101 C (CPU-N1 )
43/110 C (IOH )
30/100 C (DDR-N1C0)
31/100 C (DDR-N1C1)
Card 7: 23/101 C (CPU0 )
25/101 C (CPU1 )
25/101 C (LM93 ) - Enter the show crash list command.
The output of the command displays a crash that occurred on the failed card. Look for the keyword "Hardware Error" in the crash details. This information is part of the SSD.
******** show crash list *******
== ==== ======= ========== =========== ================
# Time Process Card/CPU/ SW HW_SER_NUM
PID VERSION SMC / Crash Card
== ==== ======= ========== =========== ================
1 2014-Feb-20+03:26:14 kernel 02/0/NA 14.0(NA) SAD1726000V/SAD17010035
2 2014-Feb-20+09:24:38 kernel 02/0/NA 14.0(NA) SAD1726000V/SAD17010035
********************* CRASH #01 ***********************
2.6.38-staros-v3-hw-64 #1 SMP PREEMPT Fri May 24 16:25:55 EDT 2013 1 0
SAD17010035 50083
.0 MMIO=90800000 MSI-X=32..63
<4>[ 4.542816] > Module image device Init
<4>[ 4.546668] --> Found components:
<6>[ 4.821670] Refined TSC clocksource calibration: 2000.071 MHz.
<6>[ 4.827687] Switching to clocksource tsc
<4>[ 4.833859] 0,28,29,30,31,37,38,39,40,63,67,69
<6>[ 4.838984] Starfile Build Number: 50083
------- truncated --------
<7>[ 38.972206] dt_fpga 0000:01:00.0: restoring config space at offset 0x3
(was 0x0, writing 0x8)
<7>[ 38.972214] dt_fpga 0000:01:00.0: restoring config space at offset 0x1
(was 0x100000, writing 0x100547)
<6>[ 38.972269] DT reset complete after 0 retries
<6>[ 42.565842] DT is now online
<4>[ 157.055594] svc: failed to register lockdv1 RPC service (errno 97).
<6>[2128512.375560] [Hardware Error]: Machine check events logged
<6>[2193364.461636] [Hardware Error]: Machine check events logged
<0>[2196785.508632] [Hardware Error]: CPU 6: Machine Check Exception: 4
Bank 8: be0000000001009f
<0>[2196785.516865] [Hardware Error]: TSC fa3eb2f263a8b ADDR 106fefd5c0
MISC 1da0600000046343
<0>[2196785.524998] [Hardware Error]: PROCESSOR 0:206c2 TIME 1392846844
SOCKET 1 APIC 20
<0>[2196785.532533] [Hardware Error]: CPU 19: Machine Check Exception: 4
Bank 8: be0000000001009f
<0>[2196785.540840] [Hardware Error]: TSC fa3eb2f2640b7 ADDR 106fefd5c0
MISC 1da0600000046343
<0>[2196785.548976] [Hardware Error]: PROCESSOR 0:206c2 TIME 1392846844
SOCKET 1 APIC 23
----- truncated -------
<0>[2196785.756359] [Hardware Error]: TSC fa3eb2f271a7d ADDR 106fefd5c0
MISC 1da0600000046343
<0>[2196785.764488] [Hardware Error]: PROCESSOR 0:206c2 TIME 1392846844
SOCKET 1 APIC 32
<0>[2196785.772020] [Hardware Error]: CPU 18: Machine Check Exception: 4
Bank 8: be0000000001009f
<0>[2196785.780335] [Hardware Error]: TSC fa3eb2f263a95 ADDR 106fefd5c0
MISC 1da0600000046343
<0>[2196785.788464] [Hardware Error]: PROCESSOR 0:206c2 TIME 1392846844
SOCKET 1 APIC 21
<0>[2196785.795991] [Hardware Error]: Machine check: Processor context
corrupt
<0>[2196785.802660] Kernel panic - not syncing: Fatal machine check on
current CPU
<4>[2196785.809681] Pid: 0, comm: kworker/0:1 Tainted: P M
2.6.38-staros-v3-hw-64 #1
<4>[2196785.817648] Call Trace:
<4>[2196785.820265] <#MC> [<ffffffff8104d444>] ? panic+0xd4/0x210
<4>[2196785.826025] [<ffffffff8104e6ac>] ? printk+0x6c/0x70
<4>[2196785.831148] [<ffffffff810726c4>] ? __atomic_notifier_call_chain+0x54/0x70
<4>[2196785.838162] [<ffffffff810726f6>] ? atomic_notifier_call_chain+0x16/0x20
<4>[2196785.845008] [<ffffffff81016da8>] ? mce_panic+0x208/0x210
<4>[2196785.850564] [<ffffffff810176d4>] ? do_machine_check+0x604/0x890
<4>[2196785.856723] [<ffffffff815a8dbb>] ? machine_check+0x1b/0x20
<4>[2196785.862449] [<ffffffff8129fbaf>] ? intel_idle+0x9f/0x130
<4>[2196785.867993] <<EOE>> [<ffffffff81072e83>] ? pm_qos_request+0x43/0x80
<4>[2196785.874605] [<ffffffff8145e64f>] ? cpuidle_idle_call+0xbf/0x2f0
<4>[2196785.880763] [<ffffffff81001506>] ? cpu_idle+0x56/0xa0
<4>[2196785.886059] [<ffffffff818edcd4>] ? start_secondary+0x1b4/0x230
<4>[2196785.892442] Logging crash to boot flash... - Enter the debug console card command.
This part of SSD displays the details of the hardware error on the failed card. This information can be found in the SSD.
******** debug console card 2 cpu 0 tail 4000 only *******
1392847108.339 card 2-cpu0: <4>[ 162.266071] svc: failed to register lockdv1 RPC
service (errno 97).
1392847187.138 card 2-cpu0: <4>[ 241.063929] mce_notify_irq: 1 callbacks suppressed
1392847187.138 card 2-cpu0: <6>[ 241.068719] [Hardware Error]: Machine check events
logged
1392852796.521 card 2-cpu0: <6>[ 5850.625815] [Hardware Error]: Machine check events
logged
1392860403.004 card 2-cpu0: <6>[13457.344535] [Hardware Error]: Machine check events
logged
1392867612.569 card 2-cpu0: <0>[20667.120164] [Hardware Error]: CPU 9: Machine Check
Exception: 4 Bank 8: be0000000001009f
1392867612.569 card 2-cpu0: <0>[20667.128234] [Hardware Error]: TSC 25c13cfe4d7c ADDR
1045a9d5c0 MISC 1da0600000041281
1392867612.569 card 2-cpu0: <0>[20667.136123] [Hardware Error]: PROCESSOR 0:206c2
TIME 1392867612 SOCKET 1 APIC 30
1392867612.669 card 2-cpu0: <0>[20667.143504] [Hardware Error]: CPU 18: Machine Check
Exception: 4 Bank 8: be0000000001009f
---- truncated -----
1392867612.770 card 2-cpu0: <0>[20667.315270] [Hardware Error]: TSC 25c13d0097e4 ADDR
1045a9d5c0 MISC 1da0600000041281
1392867612.770 card 2-cpu0: <0>[20667.323160] [Hardware Error]: PROCESSOR 0:206c2
TIME 1392867612 SOCKET 1 APIC 24
1392867612.770 card 2-cpu0: <0>[20667.330533] [Hardware Error]: CPU 11: Machine
Check Exception: 4 Bank 8: be0000000001009f
1392867612.870 card 2-cpu0: <0>[20667.338688] [Hardware Error]: TSC 25c13d0099f6 ADDR
1045a9d5c0 MISC 1da0600000041281
1392867612.870 card 2-cpu0: <0>[20667.346574] [Hardware Error]: PROCESSOR 0:206c2 TIME
1392867612 SOCKET 1 APIC 34
1392867612.870 card 2-cpu0: <0>[20667.353947] [Hardware Error]: CPU 23: Machine Check
Exception: 4 Bank 8: be0000000001009f
1392867612.870 card 2-cpu0: <0>[20667.362100] [Hardware Error]: TSC 25c13d0099ec ADDR
1045a9d5c0 MISC 1da0600000041281
1392867612.870 card 2-cpu0: <0>[20667.369992] [Hardware Error]: PROCESSOR 0:206c2 TIME
1392867612 SOCKET 1 APIC 35
1392867612.870 card 2-cpu0: <0>[20667.377359] [Hardware Error]: CPU 21: Machine Check
Exception: 4 Bank 8: be0000000001009f
1392867612.870 card 2-cpu0: <0>[20667.385506] [Hardware Error]: TSC 25c13cfe4d68 ADDR
1045a9d5c0 MISC 1da0600000041281
1392867612.870 card 2-cpu0: <0>[20667.393395] [Hardware Error]: PROCESSOR 0:206c2 TIME
1392867612 SOCKET 1 APIC 31
1392867612.870 card 2-cpu0: <0>[20667.400765] [Hardware Error]: Machine check:
Processor context corrupt
1392867612.870 card 2-cpu0: <0>[20667.407275] Kernel panic - not syncing: Fatal
machine check on current CPU
1392867612.870 card 2-cpu0: <4>[20667.414136] Pid: 0, comm: kworker/0:1 Tainted:
P M 2.6.38-staros-v3-hw-64 #1
1392867612.870 card 2-cpu0: <4>[20667.421945] Call Trace:
1392867612.870 card 2-cpu0: <4>[20667.424399] <#MC> [<ffffffff8104d444>]
? panic+0xd4/0x210
1392867612.870 card 2-cpu0: <4>[20667.430005] [<ffffffff8104e6ac>]
? printk+0x6c/0x70
1392867612.870 card 2-cpu0: <4>[20667.434966] [<ffffffff810726c4>]
? __atomic_notifier_call_chain+0x54/0x70
*1392867660.394 card 2-cpu0:
1392867660.394 card 2-cpu0:
1392867660.394 card 2-cpu0: CFE version 2.2.9 (34401) for BOXER_FROM_FLASH
(32bit,SP,LE,X86)
1392867660.394 card 2-cpu0: Build Date: Wed Jun 30 10:38:21 EDT 2010
(builds@releng7)
1392867660.394 card 2-cpu0: Copyright (C) 2001-2010 Starent Networks Corporation.
1392867660.394 card 2-cpu0:
1392867660.394 card 2-cpu0: Initializing Arena.
1392867660.394 card 2-cpu0: Initializing PCI. [normal]
1392867660.394 card 2-cpu0: Initializing Devices.
1392867660.394 card 2-cpu0: Ophir 82571 Ethernet controller 0x10608086
(Serdes) on 2/0/0
1392867660.394 card 2-cpu0: WARNING: Memory size 49152 MB for cpu0 not
matching with
value 65536 MB in IDEEPROM
1392867660.394 card 2-cpu0:
1392867660.394 card 2-cpu0: CPU type 0x0: 2000MHz
1392867660.394 card 2-cpu0: Total memory: 0xC00000000 bytes (49152MB)
1392867660.394 card 2-cpu0: Total memory used by CFE:
1392867660.394 card 2-cpu0: realmode: 0x00080000 -
0x00087140 (28992 bytes)
1392867660.394 card 2-cpu0: protectedmode: 0x0C800000 -
0x0CC86A6C (4745836 bytes)
1392867660.394 card 2-cpu0: Done board_final_init...
1392867660.394 card 2-cpu0: Booting CFE ROM...
1392867660.394 card 2-cpu0: PID: ASR5K-PSC-64G-K9 , VID: V02, SN: SAD17010035
1392867660.394 card 2-cpu0:
1392867660.394 card 2-cpu0: ERROR: Memory size 49152 MB for cpu0 not matching
with value 65536 MB in IDEEPROM
1392867660.394 card 2-cpu0:
1392867660.394 card 2-cpu0: ERROR: Bus 254 CPU 1 Chan 1 DIMM 0 NotPresent
1392867660.394 card 2-cpu0: Calling managementplane_init()
1392867660.394 card 2-cpu0: Calling managementplane_init() complete
1392867660.394 card 2-cpu0: Calling InitDPManagementPlaneEthernet()
1392867660.399 card 2-cpu0: Calling InitDPManagementPlaneEthernet() complete
1392867660.404 card 2-cpu0: CRITICAL: BIOS Failed to properly Size System
Memory aborting boot
1392867660.405 card 2-cpu0:
1392867660.410 card 2-cpu0: 2/0:cli> Boot Failure Detected. Press Ctrl-C for cli - Enter the show leds all command.
The Run/Fail LEDs should be all green and the Standby LEDs should be all green as well.
[local]st40-1# show leds all
Slot 02: Run/Fail: Green | Active: Green | Standby: Off
Slot 04: Run/Fail: Green | Active: Green | Standby: Off
Slot 06: Run/Fail: Green | Active: Green | Standby: Off
Slot 08: Run/Fail: Green | Active: Green | Standby: Off
Status: Green | Service: Amber |
Slot 09: Run/Fail: Green | Active: Off | Standby: Green
Status: Green | Service: Amber |
Slot 11: Run/Fail: Green | Active: Off | Standby: Green
Slot 13: Run/Fail: Green | Active: Green | Standby: Off
Slot 15: Run/Fail: Green | Active: Off | Standby: Green
Slot 18: Run/Fail: Red | Active: Off | Standby: Off <<hardware failure
Slot 22: Run/Fail: Green | Active: Green | Standby: Off
Slot 24: Run/Fail: Green | Active: Green | Standby: Off
Slot 25: Run/Fail: Green | Active: Off | Standby: Green
Slot 27: Run/Fail: Green | Active: Off | Standby: Off
Slot 34: Run/Fail: Green | Active: Off | Standby: Green
Slot 38: Run/Fail: Green | Active: Off | Standby: Green
Slot 40: Run/Fail: Green | Active: Off | Standby: Green
Slot 41: Run/Fail: Green | Active: Off | Standby: Green - Enter the show port table all command.
The output of the command displays the status of the line cards and their active ports. This information is part of the SSD.
ASR 5000 Chassis
******** show port table all *******
Port Type Admin Oper Link State Redundant
----- ------------------------ -------- ---- ---- ---------------- ---------
22/1 10/100 Ethernet Enabled - Up - 38/1
Untagged Enabled Down - Active -
Tagged VLAN 11 Enabled Up - Active -
Tagged VLAN 12 Enabled Up - Active -
Tagged VLAN 13 Enabled Up - Active -
Tagged VLAN 15 Enabled Up - Active -
Tagged VLAN 16 Enabled Up - Active -
Tagged VLAN 14 Enabled Up - Active -
Tagged VLAN 17 Enabled Up - Active -
22/2 10/100 Ethernet Disabled Down Up Standby 38/2
22/3 10/100 Ethernet Disabled Down Down Standby 38/3
22/4 10/100 Ethernet Disabled Down Down Standby 38/4
22/5 10/100 Ethernet Disabled Down Down Standby 38/5
22/6 10/100 Ethernet Disabled Down Down Standby 38/6
22/7 10/100 Ethernet Disabled Down Down Standby 38/7
22/8 10/100 Ethernet Enabled Down Down Active 38/8
24/1 1000 Ethernet Dual Media Enabled Up Up Active 25/1
24/2 1000 Ethernet Dual Media Disabled Down Down Active 25/2
24/3 RS232 Serial Console Enabled Up Up Active 25/3
24/4 BITS T1/E1 Timing Disabled Down Down Active 25/4
25/1 1000 Ethernet Dual Media Enabled Down Up Standby 24/1
25/2 1000 Ethernet Dual Media Disabled Down Down Standby 24/2
25/3 RS232 Serial Console Enabled Down Up Standby 24/3
25/4 BITS T1/E1 Timing Disabled Down Down Standby 24/4
34/1 1000 Ethernet Enabled Down Down Active None
34/2 1000 Ethernet Disabled Down Down Active None
34/3 1000 Ethernet Disabled Down Down Active None
34/4 1000 Ethernet Disabled Down Down Active None
38/1 10/100 Ethernet Enabled - Down - 22/1
Untagged Enabled Down - Standby -
Tagged VLAN 11 Enabled Down - Standby -
Tagged VLAN 12 Enabled Down - Standby -
Tagged VLAN 13 Enabled Down - Standby -
Tagged VLAN 15 Enabled Down - Standby -
Tagged VLAN 16 Enabled Down - Standby -
Tagged VLAN 14 Enabled Down - Standby -
Tagged VLAN 17 Enabled Down - Standby -
38/2 10/100 Ethernet Disabled Down Down Active 22/2
38/3 10/100 Ethernet Disabled Down Down Active 22/3
38/4 10/100 Ethernet Disabled Down Down Active 22/4
38/5 10/100 Ethernet Disabled Down Down Active 22/5
38/6 10/100 Ethernet Disabled Down Down Active 22/6
38/7 10/100 Ethernet Disabled Down Down Active 22/7
38/8 10/100 Ethernet Enabled Down Down Standby 22/8ASR 5500 Chassis
******** show port table all *******
Port Role Type Admin Oper Link State Pair Redundant
----- ---- ------------------------ -------- ---- ---- ------- ----- ---------
5/1 Mgmt 1000 Ethernet Enabled Up Up Active 6/1 L2 Link
5/2 Mgmt 1000 Ethernet Disabled Down Down Active 6/2 L2 Link
5/3 Mgmt RS232 Serial Console Enabled Down Unkn Standby 6/3 L2 Link
5/10 Srvc 10G Ethernet Enabled - Up - 6/10 LA+ 5/10
Untagged Enabled Up - Active - -
Tagged VLAN 101 Enabled Up - Active - -
Tagged VLAN 102 Enabled Up - Active - -
Tagged VLAN 104 Enabled Up - Active - -
Tagged VLAN 105 Enabled Up - Active - -
Tagged VLAN 103 Enabled Up - Active - -
5/11 Srvc 10G Ethernet Enabled Up Up Active 6/11 LA+ 5/10
5/12 Srvc 10G Ethernet Disabled Down Down Active 6/12 L2 Link
5/13 Srvc 10G Ethernet Disabled Down Down Active 6/13 L2 Link
5/14 Srvc 10G Ethernet Disabled Down Down Active 6/14 L2 Link
5/15 Srvc 10G Ethernet Enabled Up Up Active 6/15 LA+ 5/10
5/16 Srvc 10G Ethernet Enabled Up Up Active 6/16 LA+ 5/10
5/17 Srvc 10G Ethernet Disabled Down Down Active 6/17 L2 Link
5/18 Srvc 10G Ethernet Disabled Down Down Active 6/18 L2 Link
5/19 Srvc 10G Ethernet Disabled Down Down Active 6/19 L2 Link
5/20 Srvc 10G Ethernet Enabled Up Up Active 6/20 LA+ 5/10
5/21 Srvc 10G Ethernet Enabled Up Up Active 6/21 LA+ 5/10
5/22 Srvc 10G Ethernet Disabled Down Down Active 6/22 L2 Link
5/23 Srvc 10G Ethernet Disabled Down Down Active 6/23 L2 Link
5/24 Srvc 10G Ethernet Disabled Down Down Active 6/24 L2 Link
5/25 Srvc 10G Ethernet Enabled Up Up Active 6/25 LA+ 5/10
5/26 Srvc 10G Ethernet Enabled Up Up Active 6/26 LA+ 5/10
5/27 Srvc 10G Ethernet Disabled Down Down Active 6/27 L2 Link
5/28 Srvc 10G Ethernet Disabled Down Down Active 6/28 L2 Link
5/29 Srvc 10G Ethernet Disabled Down Down Active 6/29 L2 Link
6/1 Mgmt 1000 Ethernet Enabled Down Up Standby 5/1 L2 Link
6/2 Mgmt 1000 Ethernet Disabled Down Down Standby 5/2 L2 Link
6/3 Mgmt RS232 Serial Console Enabled Down Unkn Standby 5/3 L2 Link
6/10 Srvc 10G Ethernet Enabled - Up - 5/10 LA! 5/10
Untagged Enabled Up - Active - -
Tagged VLAN 101 Enabled Up - Active - -
Tagged VLAN 102 Enabled Up - Active - -
Tagged VLAN 104 Enabled Up - Active - -
Tagged VLAN 105 Enabled Up - Active - -
Tagged VLAN 103 Enabled Up - Active - -
6/11 Srvc 10G Ethernet Enabled Up Up Active 5/11 LA! 5/10
6/12 Srvc 10G Ethernet Disabled Down Down Standby 5/12 L2 Link
6/13 Srvc 10G Ethernet Disabled Down Down Standby 5/13 L2 Link
6/14 Srvc 10G Ethernet Disabled Down Down Standby 5/14 L2 Link
6/15 Srvc 10G Ethernet Enabled Up Up Active 5/15 LA! 5/10
6/16 Srvc 10G Ethernet Enabled Up Up Active 5/16 LA! 5/10
6/17 Srvc 10G Ethernet Disabled Down Down Standby 5/17 L2 Link
6/18 Srvc 10G Ethernet Disabled Down Down Standby 5/18 L2 Link
6/19 Srvc 10G Ethernet Disabled Down Down Standby 5/19 L2 Link
6/20 Srvc 10G Ethernet Enabled Up Up Active 5/20 LA! 5/10
6/21 Srvc 10G Ethernet Enabled Up Up Active 5/21 LA! 5/10
6/22 Srvc 10G Ethernet Disabled Down Down Standby 5/22 L2 Link
6/23 Srvc 10G Ethernet Disabled Down Down Standby 5/23 L2 Link
6/24 Srvc 10G Ethernet Disabled Down Down Standby 5/24 L2 Link
6/25 Srvc 10G Ethernet Enabled Up Up Active 5/25 LA! 5/10
6/26 Srvc 10G Ethernet Enabled Up Up Active 5/26 LA! 5/10
6/27 Srvc 10G Ethernet Disabled Down Down Standby 5/27 L2 Link
6/28 Srvc 10G Ethernet Disabled Down Down Standby 5/28 L2 Link
6/29 Srvc 10G Ethernet Disabled Down Down Standby 5/29 L2 Link - Enter the show port datalink counters [<slot#>/<port#>] command.
The output of the command displays the statistics of a port. The information might vary dependent upon the port type. Note that the statistics shown here might only give partial information. You need to do further investigation before you make a conclusion on the suspect port. This information is not part of the SSD.
Counters for port 5/15:
Line Card 10 Gigabit Ethernet Port
Rx Counter Data | Tx Counter Data
----------------------- -------------- + ----------------------- -------------
RX Bytes 929593433281710 | TX Bytes 980010533492940
RX Unicast frames 2622395473 | TX Unicast frames 1156468032
RX Multicast frames 6819022 | TX Multicast frames 6883788
RX Broadcast frames 0 | TX Broadcast frames 0
RX Size 64 frames 1072853571 | TX Size 64 frames 2180897214
RX Size 65 .. 127 fr 4229387075 | TX Size 65 .. 127 fr 3889237141
RX Size 128 .. 255 fr 2650809484 | TX Size 128 .. 255 fr 711064618
RX Size 256 .. 511 fr 1361009593 | TX Size 256 .. 511 fr 2349224327
RX Size 512 .. 1023 fr 3559787799 | TX Size 512 .. 1023 fr 1444113684
RX Size 1024 .. 1518 fr 448089039 | TX Size 1024 .. 1518 fr 474195217
RX Size 1519 .. 1522 fr 2185019476 | TX Size 1519 .. 1522 fr 2999521504
RX OverSize frames 0 | TX OverSize frames 0
RX UnderSize frames 0 | TX UnderSize frames 0
RX ExceedMaxSize frames 0
RX Fragment frames 69829 | TX Fragment frames 0
RX Jabber frames 7090511 | TX Jabber frames 0
RX Control frames 0 | TX Control frames 882
RX Pause frames 0 | TX Pause frames 882
RX FCS Error frames 828050684 | TX FCS Error frames 0
RX Length Error frames 0 | TX Length Error frames 0
RX Code Error frames 38410621
RX ExMaxSize Err frames 1362770
----------------------- -------------- + ----------------------- -------------
[local]ASR5K-1# show port datalink counters 17/1
Counters for port 17/1:
Line Card 10 Gigabit Ethernet Port
Rx Counter Data | Tx Counter Data
----------------------- -------------- + ----------------------- -------------
RX Unicast frames 19873 | TX Unicast frames 0
RX Multicast frames 19873 | TX Multicast frames 0
RX Broadcast frames 0 | TX Broadcast frames 0
RX Size 64 frames 0 | TX Size 64 frames 0
RX Size 65 .. 127 fr 0 | TX Size 65 .. 127 fr 0
RX Size 128 .. 255 fr 0 | TX Size 128 .. 255 fr 0
RX Size 256 .. 511 fr 19873 | TX Size 256 .. 511 fr 0
RX Size 512 .. 1023 fr 0 | TX Size 512 .. 1023 fr 0
RX Size 1024 .. 1518 fr 0 | TX Size 1024 .. 1518 fr 0
RX Size > 1518 frames 0 | TX Size > 1518 frames 0
RX Bytes OK 8326787 | TX Bytes OK 0
RX Bytes BAD 0 | TX Bytes BAD 0
RX SHORT OK 0 | TX PAUSE 0
RX SHORT CRC 0 | TX ERR 0
RX OVF 0 |
RX NORM CRC 0 |
RX LONG OK 0 |
RX LONG CRC 0 |
RX PAUSE 0 |
RX FALS CRS 0 |
RX SYM ERR 0 |
RX SPI FRAME COUNT 19873 | TX SPI FRAME COUNT 0
RX SPI LEN ERR 0 | TX SPI LEN ERR 0
RX SPI DIP 2 ERR 0 | TX SPI DIP 4 ERR 0
RX SPI STATUS OOF ERR 0 | TX SPI DATA OOF ERR 0
RX FIFO OVERFLOW 0 | TX FIFO FULL DROP 0
RX PAUSE COUNT 0 | TX DIP 4 PACKET DROP 0
SPI EOP/ABORT 0 |
RX FRAGMENTS COUNT 0 |
RX MAC ERR 0 |
RX JABBER COUNT 0 |
----------------------- -------------- + ----------------------- -------------
[local]st40-1# show port datalink counters 22/2
Counters for port 22/2:
Line Card Fast Ethernet Port
Rx Counter Data | Tx Counter Data
----------------------- -------------- + ----------------------- -------------
RX Unicast frames 13773 | TX Unicast frames 0
RX Multicast frames 126510 | TX Multicast frames 0
RX Broadcast frames 54954 | TX Broadcast frames 0
RX Size 64 frames 109874 | TX Size 64 frames 0
RX Size 65 .. 127 fr 81230 | TX Size 65 .. 127 fr 0
RX Size 128 .. 255 fr 2384 | TX Size 128 .. 255 fr 0
RX Size 256 .. 511 fr 1670 | TX Size 256 .. 511 fr 0
RX Size 512 .. 1023 fr 79 | TX Size 512 .. 1023 fr 0
RX Size 1024 .. 1518 fr 0 | TX Size 1024 .. 1518 fr 0
RX Size > 1518 frames 0 | TX Size > 1518 frames 0
RX Bytes OK 13824764 | TX Bytes OK 0
RX Bytes BAD 0 | TX Bytes BAD 0
RX OVF 0 | TX DEFER 0
RX SHORT OK 0 | TX COL 0
RX SHORT CRC 0 | TX SCOL 0
RX NO SFD 0 | TX MCOL 0
RX NORM CRC 0 | TX XCOL 0
RX NORM ALI 0 | TX LCOL 0
RX LONG OK 0 | TX PAUSE 0
RX LONG CRC 0 | TX ERR 0
RX PAUSE 0 |
RX FALS CRS 0 |
RX SYM ERR 0 |
----------------------- -------------- + ----------------------- -------------
[local]st40-1# show port datalink counters 24/1
Counters for port 24/1:
SPIO 10/100/1000 Ethernet port
Rx Counter Data | Tx Counter Data
----------------------- -------------- + ----------------------- -------------
RX Bytes 148501169 | TX Bytes 138116037
RX BAD frames 0 | TX BAD frames 0
RX Runt frames 0 | TX Runt frames 0
RX Oversize frames 0 | TX Oversize frames 0
RX Good frames 1590640 | TX Good frames 1258465
RX Multicast frames 4393 | TX Collisions 0
RX Broadcast frames 365063 | TX Excessive collis 0
RX Code ERROR 0 | TX Late Collisions 0
RX CRC ERROR 0 | TX CRC ERROR 0
RX length ERROR 0 | TX ABORT 0
RX Align ERROR 0 |
----------------------- -------------- + ----------------------- -------------
# show port datalink counters 5/15
Counters for port 5/15:
Line Card 10 Gigabit Ethernet Port
Rx Counter Data | Tx Counter Data
----------------------- -------------- + ----------------------- -------------
RX OverSize frames 0 |
RX Bytes OK 1558209913 | TX Bytes OK 840628
RX Bytes BAD 0 | TX Bytes BAD 0
RX SHORT OK 0 | TX PAUSE 0
RX SHORT CRC 0 | TX ERR 0
RX OVF 0 |
RX NORM CRC 0 |
RX LONG OK 0 |
RX LONG CRC 82233 |
RX PAUSE 0 |
RX FALS CRS 0 |
RX SYM ERR 0 |
----------------------- -------------- + ----------------------- ------------- - Enter the show port npu counters command.
The output of the command displays the statistics of a port. The statistics show that port 31/1 does not send as much data as it receives. The SSD provides two statistics taken at different times 15 minutes apart. You can use the two statistics in order to determine the delta.
******** show port npu counters *******
Thursday April 30 15:07:06 UTC 2015
Counters for port 31/1
Counter Rx Frames Rx Bytes Tx Frames Tx Bytes
-------------------- ------------- --------------- ------------- ---------------
Unicast 1088589 494910442 86 4368 <<<<<
Multicast 2730 246420 330 35496
Broadcast 10393 665152 6365 292790
IPv4 unicast 1088506 494905130 4 596
IPv4 non-unicast 656 82768 326 35112
IPv6 unicast 0 0 0 0
IPv6 non-unicast 0 0 4 384
Fragments received 258066 198717886 n/a n/a
Packets reassembled 129021 193781884 n/a n/a
Fragments to kernel 24 33204 n/a n/a
HW error 0 0 n/a n/a
Port non-operational 0 0 0 0
SRC MAC is multicast 0 0 n/a n/a
Unknown VLAN tag 0 0 n/a n/a
Other protocols 2074 163652 n/a n/a
Not IPv4 0 0 n/a n/a
Bad IPv4 header 0 0 n/a n/a
IPv4 MRU exceeded 0 0 n/a n/a
TCP tiny fragment 0 0 0 0
No ACL match 0 0 0 0
Filtered by ACL 0 0 0 0
TTL expired 0 0 n/a n/a
Flow lookup twice 0 0 n/a n/a
Unknown IPv4 class 0 0 n/a n/a
Too short: IP 0 0 n/a n/a
Too short: ICMP 0 0 0 0
Too short: IGMP 0 0 0 0
Too short: TCP 0 0 0 0
Too short: UDP 0 0 0 0
Too short: IPIP 0 0 n/a n/a
Too short: GRE 0 0 n/a n/a
Too short: GRE key 0 0 n/a n/a
Don't frag discards n/a n/a 0 0
Fragment packets n/a n/a 0 0
Fragment fragments n/a n/a 0 0
IPv4VlanMap dropped 0 0 n/a n/a
IPSec NATT keep alive 0 0 n/a n/a
MPLS Flow not found 0 0 n/a n/a
MPLS unicast 0 0 0 0
Size < 17 0 0 0 0
Size 17 .. 64 28449 1820528 6450 296712
Size 65 .. 127 549857 46030525 320 33820
Size 128 .. 255 191483 32170025 10 1796
Size 256 .. 511 38611 14091567 1 326
Size 512 .. 1023 30308 20922553 0 0
Size 1024 .. 2047 263006 380787195 0 0
Size 2048 .. 4095 0 0 0 0
Size 4096 .. 4500 0 0 0 0
Size > 4500 0 0 0 0
******** show port npu counters *******
Thursday April 30 15:22:53 UTC 2015
Counters for port 31/1
Counter Rx Frames Rx Bytes Tx Frames Tx Bytes
-------------------- ------------- --------------- ------------- ---------------
Unicast 1458664 690686035 104 5196 <<<<<
Multicast 3566 321706 429 46150
Broadcast 14078 900992 8295 381570
IPv4 unicast 1458563 690679571 4 596
IPv4 non-unicast 859 108182 425 45766
IPv6 unicast 0 0 0 0
IPv6 non-unicast 0 0 4 384
Fragments received 363014 279522687 n/a n/a
Packets reassembled 181494 272589787 n/a n/a
Fragments to kernel 26 36128 n/a n/a
HW error 0 0 n/a n/a
Port non-operational 0 0 0 0
SRC MAC is multicast 0 0 n/a n/a
Unknown VLAN tag 0 0 n/a n/a
Other protocols 2707 213524 n/a n/a
Not IPv4 0 0 n/a n/a
Bad IPv4 header 0 0 n/a n/a
IPv4 MRU exceeded 0 0 n/a n/a
TCP tiny fragment 0 0 0 0
No ACL match 0 0 0 0
Filtered by ACL 0 0 0 0
TTL expired 0 0 n/a n/a
Flow lookup twice 0 0 n/a n/a
Unknown IPv4 class 0 0 n/a n/a
Too short: IP 0 0 n/a n/a
Too short: ICMP 0 0 0 0
Too short: IGMP 0 0 0 0
Too short: TCP 0 0 0 0
Too short: UDP 0 0 0 0
Too short: IPIP 0 0 n/a n/a
Too short: GRE 0 0 n/a n/a
Too short: GRE key 0 0 n/a n/a
Don't frag discards n/a n/a 0 0
Fragment packets n/a n/a 0 0
Fragment fragments n/a n/a 0 0
IPv4VlanMap dropped 0 0 n/a n/a
IPSec NATT keep alive 0 0 n/a n/a
MPLS Flow not found 0 0 n/a n/a
MPLS unicast 0 0 0 0
Size < 17 0 0 0 0
Size 17 .. 64 36271 2321072 8398 386320
Size 65 .. 127 726857 60769944 416 44008
Size 128 .. 255 247902 41688950 13 2262
Size 256 .. 511 52061 19029750 1 326
Size 512 .. 1023 39817 27526320 0 0
Size 1024 .. 2047 373400 540572697 0 0
Size 2048 .. 4095 0 0 0 0
Size 4096 .. 4500 0 0 0 0
Size > 4500 0 0 0 0
Contributed by Cisco Engineers
- Marc BautistaCisco TAC Engineer
- Joe OpioCisco TAC Engineer
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)