Troubleshoot Hardware Failure on Catalyst 9600 Supervisors and Line Cards
Available Languages
Introduction
This document describes how to identify, isolate, and troubleshoot symptoms of hardware failures on Catalyst 9600 Supervisors and Line Cards.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
The information in this document is based on Catalyst 9600 Series Switches.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Terminology
|
Term |
Definition |
Context |
|
Supervisor |
The Cisco Catalyst 9600 Series Supervisor Engine 1 is powered by three UADP 3.0 ASICs and one x86 CPU processor. The three ASICs are interconnected with a 3.2-Tbps ASIC interconnect on each ASIC. Sup-1 provides 9.6 Tbps (4.8 Tbps full duplex). With the Cisco Catalyst 9606R chassis, each slot has 2.4 Tbps with Sup-1. |
This document contains commands to isolate hardware failure on Supervisor engines. |
|
Power On Self Test (P.O.S.T.) |
After the Supervisor and Line Cards are initialized after power on, a Power On Self Test (POST) executes to ensure the hardware integrity of the modules. POST is also executed during Online Insertion & Removal (OIR) of a line card. |
Show post command displays the results of these tests. |
|
Generic Online Diagnostic Subsytems (GOLD.) |
GOLD tests are run after the system is online. Some tests are run periodically as Health Monitoring tests. All the tests can be run on demand by the user. |
It is used to detect module hardware failures. |
|
Online Insertion & Removal (O.I.R.) |
This feature can power-cycle a module in a given slot via the command line interface (CLI). This is useful when there is not an on-site resource that can psychically reseat the module. |
O.I.R. is a great feature to perform isolate hardware failure on modules. |
Chassis Overview
|
Catalyst C9606-R |
|
|
Troubleshoot
Symptoms of Module Hardware Issues
|
Symptom |
Possible Root Cause |
|
Module is not properly installed, not initialized, or encountered hardware failure due to failing the P.O.S.T. |
Verify Module Status
- It is important to first check that the chassis recognizes the module.
- The ideal status for a module is 'ok'.
|
Module Status |
Possible Root Cause |
Action |
|
ok |
N/A. |
None |
|
post-fail |
Potential hardware failure. |
Refer to Troubleshoot Module Status. |
|
other |
Module failed to initialize. |
Refer to Troubleshoot Module Status. |
|
Not listed in output |
Not properly installed or not receiving sufficient power. |
Check for available power budget. Visually inspect L.E.D. status of module. |
Cat9600#show module Chassis Type: C9606R Mod Ports Card Type Model Serial No. ---+-----+--------------------------------------+--------------+-------------- 1 24 24-Port 40GE/12-Port 100GE C9600-LC-24C 2 48 48-Port 10GE / 25GE C9600-LC-48YL 3 0 Supervisor 1 Module C9600-SUP-1 4 0 Supervisor 1 Module C9600-SUP-1 Mod MAC addresses Hw Fw Sw Status ---+--------------------------------+----+------------+------------------+-------- 1 DC8C.37A0.C880 to DC8C.37A0.C8FF 1.0 17.1.1[FC2] 16.12.03 post-fail 2 DC8C.3773.1E00 to DC8C.3773.1E7F 1.0 17.1.1[FC2] 16.12.03 ok 3 DC8C.379F.ED80 to DC8C.379F.EDFF 1.0 17.1.1[FC2] 16.12.03 ok 4 DC8C.379F.EDF1 to DC8C.379F.EDF9 1.0 17.1.1[FC2] 16.12.03 ok //other example Mod MAC addresses Hw Fw Sw Status ---+--------------------------------+----+------------+------------------+-------- 1 DC8C.37A0.C880 to DC8C.37A0.C8FF 1.3 17.3.1r[FC2] 17.03.05 power-deny 2 DC8C.37A0.C880 to DC8C.37A0.C8FF 1.3 17.3.1r[FC2] 17.03.05 power-deny 3 DC8C.37A0.C880 to DC8C.37A0.C8FF 1.4 17.3.1r[FC2] 17.03.05 ok
Troubleshoot Module Status
|
Symptom |
Action |
|
Module not listed or status is not 'ok'. |
Proceed with the next steps. |
Verify Syslog
Line Card Offline
%IOSXE_OIR-6-OFFLINECARD: Card (rp) offline in slot R1 %IOSXE_OIR-6-REMSPA: SPA removed from subslot 6/0, interfaces disabled %SPA_OIR-6-OFFLINECARD: SPA (C9600-LC-48TX) offline in subslot 6/0
Power-Budget Issue
%CMRP_PFU-3-PWR_MGMT_LC_SHUTDOWN: R0/0: cmand: WARNING: Linecard in slot 6 with priority 0 and a power requirement of 315 watts has shutdown. %CMRP_PFU-3-PWR_MGMT_ALARM: Chassis 1 R0/0: cmand: WARNING: System does not have sufficient input power for minimum reliable operation requiring 1161 watts. The system needs 1101 watts of additional power.
Diagnostic Test Failed
*May 29 02:10:19.523: %PM-4-ERR_DISABLE: diagnostics error detected on Fo2/0/1, putting Fo2/0/1 in err-disable state
Check Module Uptime
- Onboard Failure Logging (OBFL) gives us insight to reload reasons such as reload command, software exception, power off, and so on.
- Check time stamps that could correlate to sudden power loss to the module or external factor contributing to the module going down.
Cat9600#show logging onboard rp (active|standby) uptime detail
-------------------------------------------------------------------------------- UPTIME SUMMARY INFORMATION -------------------------------------------------------------------------------- First customer power on : 08/28/2021 13:08:41 Total uptime : 0 years 41 weeks 2 days 15 hours 10 minutes <-- Total Uptime Total downtime : 0 years 7 weeks 6 days 7 hours 11 minutes Number of resets : 14 <-- Total number of resets Number of slot changes : 0 Current reset reason : PowerOn <-- Last reload reason Current reset timestamp : 11/06/2021 17:25:29 Current slot : 3 Chassis type : 55 Current uptime : 0 years 39 weeks 1 days 18 hours 5 minutes <-- Current Uptime -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- UPTIME CONTINUOUS INFORMATION -------------------------------------------------------------------------------- Time Stamp | Reset | Uptime MM/DD/YYYY HH:MM:SS | Reason | years weeks days hours minutes -------------------------------------------------------------------------------- 08/28/2021 13:08:41 PowerOn 0 0 0 0 0 08/28/2021 13:16:29 Reload 0 0 0 0 0 08/28/2021 13:29:28 Reload 0 0 0 0 5 08/28/2021 13:42:30 Reload 0 0 0 0 5 08/28/2021 13:55:14 Image Install 0 0 0 0 5 <-- Reset history by reason
Cat9600#show logging onboard slot (1|2|4|5|6|7) uptime detail -------------------------------------------------------------------------------- UPTIME SUMMARY INFORMATION -------------------------------------------------------------------------------- First customer power on : 08/08/2018 09:30:33 Total uptime : 0 years 11 weeks 1 days 10 hours 40 minutes Total downtime : 4 years 32 weeks 1 days 23 hours 25 minutes Number of resets : 81 Number of slot changes : 5 Current reset reason : CP_RESET_CPU_GOT_RESET Current reset timestamp : 05/24/2023 19:35:58 Current slot : 1 Chassis type : 30 Current uptime : 0 years 2 weeks 0 days 0 hours 0 minutes -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- UPTIME CONTINUOUS INFORMATION -------------------------------------------------------------------------------- Time Stamp | Reset | Uptime MM/DD/YYYY HH:MM:SS | Reason | years weeks days hours minutes -------------------------------------------------------------------------------- 04/26/2021 21:39:45 CP_RESET_CPU_GOT_RESET 0 0 0 0 0 04/27/2021 14:12:04 CP_RESET_POWER_ON 0 0 0 6 0 04/28/2021 14:39:30 CP_RESET_POWER_ON 0 0 0 14 0 04/30/2021 12:46:59 CP_RESET_POWER_ON 0 0 0 13 0
Verify POST
Failed POST example
Cat9600#show post
Stored system POST messages:
Switch C9606R
--------------
Fri May 29 02:10:18 2020 POST: Module: 1 Mac Loopback Begin
Fri May 29 02:10:18 2020 POST: Module: 1 Mac Loopback: loopback Test: End, Status Passed
Fri May 29 02:10:18 2020 POST: Module: 2 Mac Loopback Begin
Fri May 29 02:10:18 2020 POST: Mac Loopback: Failed For Interface : TwentyFiveGigE2/2/0/38
Fri May 29 02:10:18 2020 POST: Mac Loopback: Failed For Interface : TwentyFiveGigE2/2/0/39
Fri May 29 02:10:18 2020 POST: Mac Loopback: Failed For Interface : TwentyFiveGigE2/2/0/40
Fri May 29 02:10:18 2020 POST: Module: 2 Mac Loopback: loopback Test: End, Status Failed
POST passed example
Cat9600#show post Stored system POST messages: Switch C9606R -------------- POST: MBIST Tests : Begin POST: MBIST Tests : End, Status Passed POST: Module: 3 PHY Loopback: loopback Test: Begin POST: Module: 3 PHY Loopback: loopback Test: End, Status Passed POST: Module: 2 PHY Loopback: loopback Test: Begin POST: Module: 2 PHY Loopback: loopback Test: End, Status Passed POST: Module: 1 PHY Loopback: loopback Test: Begin POST: Module: 1 PHY Loopback: loopback Test: End, Status Passed POST: Module: 5 PHY Loopback: loopback Test: Begin POST: Module: 5 PHY Loopback: loopback Test: End, Status Passed
Cat9600#show platform Chassis type: C9606R Slot Type State Insert time (ago) --------- ------------------- --------------------- ----------------- 1 C9600-LC-24C ok 02:45:09 1/0 C9600-LC-24C ok 02:43:15 2 C9600-LC-48YL faulty 02:45:09 2/0 C9600-LC-48YL ok 02:43:14 3 C9600-SUP-1 ok 02:45:09 3/0 C9600-SUP-1 ok 02:43:14 4 C9600-SUP-1 ok 02:45:09 4/0 C9600-SUP-1 ok 02:43:14 R0 C9600-SUP-1 ok, active 02:45:09 R1 C9600-SUP-1 init, standby 02:45:09 P1 C9600-PWR-2KWAC ok 02:44:21 P2 C9600-PWR-2KWAC ok 02:44:21 P3 C9600-PWR-2KWAC ps, f0, f1, fail 02:44:21 P4 C9600-PWR-2KWAC ps, f0, f1, fail 02:44:21 P5 C9606-FAN ok 02:44:21 Slot CPLD Version Firmware Version --------- ------------------- --------------------------------------- 1 19070619 17.1.1[FC2] 2 19070619 17.1.1[FC2] 3 19041620 17.1.1[FC2] 4 19041620 17.1.1[FC2]
GOLD Tests
- If module has failed P.O.S.T or listed as 'faulty' then diagnostics to get the failure reason must be collected.
- Run hardware diagnostics to quickly identify potential hardware failure within the line cards or Supervisors.
Types of diagnostic tests:
- Health monitoring tests - Enabled by default and it runs in background for every few minutes. These tests are non disruptive.
- On demand tests - Run by the user with a CLI. Some of these tests are disruptive.
|
Diagnostic Test |
What is it testing? Context? Impact? |
|
TestGoldPktLoopback |
This test is very similar to the Macloopback test. |
|
TestOBFL |
Verifies the on-board failure logging capabilities. This test prints a diagnostic message and is logged onto the OBFL. |
|
TestThermal |
Verifies the temperature value from the sensor does not exceed the threshold. It is a non-disruptive test and can be run as a health monitor test. |
|
TestPortTxMonitoring |
This test monitors the TX counters of a connected interface. This test verifies if the connected port is able to send the packets or not. It is a non-disruptive test and can be run as a health monitoring test. |
|
TestFanTray |
This test verifies if the fan tray has been inserted and works properly on the board. It is a non-disruptive test and can be run as a health monitoring test. |
|
TestScratchRegister |
The Scratch Register test monitors the health of application-specific integrated circuits (ASICs) it writes values into registers and reads back the values from these registers. It is a non-disruptive test and can be run as a health monitoring test. |
|
TestConsistencyCheckMcast |
This test monitors to check if Mcast programming is correct. This test checks with Forwarding object manger to identify incomplete entries or long pending configurations to hardware. |
Run an on-demand diagnostic test on a module.
Cat9600#diagnostic start switch module 3 test ?
Diagnostics test suite attributes:
M/C/* - Minimal bootup level test / Complete bootup level test / NA
B/* - Basic ondemand test / NA
P/V/* - Per port test / Per device test / NA
D/N/* - Disruptive test / Non-disruptive test / NA
S/* - Only applicable to standby unit / NA
X/* - Not a health monitoring test / NA
F/* - Fixed monitoring interval test / NA
E/* - Always enabled monitoring test / NA
A/I - Monitoring is active / Monitoring is inactive
Test Interval Thre-
ID Test Name Attributes day hh:mm:ss.ms shold
==== ================================== ============ =============== =====
1) TestOBFL ------------------------> *B*N*X**I not configured n/a
2) TestFantray ---------------------> *B*N****A 000 00:01:40.00 1
3) TestThermal ---------------------> *B*N****A 000 00:01:30.00 1
4) TestScratchRegister -------------> *B*N****A 000 00:01:30.00 5
5) TestConsistencyCheck ------------> *B*N****A 000 00:01:30.00 1
WORD Test ID list (e.g. 1,3-6) or Test Name
Cat9600#diagnostic start switch 1 module 3 test 2
//syslog prints upon start and finish of diagnostic test.
*Mar 24 21:48:47.968: %DIAG-6-TEST_RUNNING: switch 1 module 3: Running TestFantray{ID=2} ...
*Mar 24 21:48:47.969: %DIAG-6-TEST_OK: switch 1 module 3: TestFantray{ID=2} has completed successfully <-- Passed
|
Symptom |
Action |
|
Module failed diagnostic test. |
Perform an O.I.R. of the module. |
Run on-demand OIR on a module
Cat9600#hw-module subslot 1/0 oir power-cycle
Proceed with power cycle of module? [confirm]
*Jun 9 20:39:14.686: %IOSXE_OIR-6-REMSPA: SPA removed from subslot 1/0, interfaces disabled
*Jun 9 20:39:14.721: %SPA_OIR-6-OFFLINECARD: SPA (C9600-LC-24C) offline in subslot 1/0
*Jun 9 20:39:16.686: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/1, changed state to down
*Jun 9 20:39:16.686: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/2, changed state to down
*Jun 9 20:39:16.687: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/3, changed state to down
*Jun 9 20:39:16.687: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/4, changed state to down
*Jun 9 20:39:16.689: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/5, changed state to down
*Jun 9 20:39:16.690: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/6, changed state to down
*Jun 9 20:39:16.690: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/7, changed state to down
*Jun 9 20:39:16.690: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/8, changed state to down
*Jun 9 20:39:16.690: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/9, changed state to down
*Jun 9 20:39:16.691: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/10, changed state to down
*Jun 9 20:39:16.691: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/11, changed state to down
*Jun 9 20:39:16.693: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/12, changed state to down
*Jun 9 20:39:16.693: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/13, changed state to down
*Jun 9 20:39:16.694: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/14, changed state to down
*Jun 9 20:39:16.694: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/15, changed state to down
*Jun 9 20:39:16.694: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/16, changed state to down
*Jun 9 20:39:16.694: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/17, changed state to down
*Jun 9 20:39:16.695: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/18, changed state to down
*Jun 9 20:39:16.695: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/19, changed state to down
*Jun 9 20:39:16.697: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/20, changed state to down
*Jun 9 20:39:16.697: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/21, changed state to down
*Jun 9 20:39:16.698: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/22, changed state to down
*Jun 9 20:39:16.698: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/23, changed state to down
*Jun 9 20:39:16.698: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/24, changed state to down
*Jun 9 20:39:16.698: %LINK-3-UPDOWN: Interface HundredGigE1/0/25, changed state to down
*Jun 9 20:39:16.699: %LINK-3-UPDOWN: Interface HundredGigE1/0/26, changed state to down
*Jun 9 20:39:16.699: %LINK-3-UPDOWN: Interface HundredGigE1/0/27, changed state to down
*Jun 9 20:39:16.701: %LINK-3-UPDOWN: Interface HundredGigE1/0/28, changed state to down
*Jun 9 20:39:16.701: %LINK-3-UPDOWN: Interface HundredGigE1/0/29, changed state to down
*Jun 9 20:39:16.702: %LINK-3-UPDOWN: Interface HundredGigE1/0/30, changed state to down
*Jun 9 20:39:16.702: %LINK-3-UPDOWN: Interface HundredGigE1/0/31, changed state to down
*Jun 9 20:39:16.702: %LINK-3-UPDOWN: Interface HundredGigE1/0/32, changed state to down
*Jun 9 20:39:16.702: %LINK-3-UPDOWN: Interface HundredGigE1/0/33, changed state to down
*Jun 9 20:39:16.703: %LINK-3-UPDOWN: Interface HundredGigE1/0/34, changed state to down
*Jun 9 20:39:16.703: %LINK-3-UPDOWN: Interface HundredGigE1/0/35, changed state to down
*Jun 9 20:39:16.704: %LINK-3-UPDOWN: Interface HundredGigE1/0/36, changed state to down
*Jun 9 20:39:16.704: %LINK-3-UPDOWN: Interface HundredGigE1/0/37, changed state to down
*Jun 9 20:39:16.705: %LINK-3-UPDOWN: Interface HundredGigE1/0/38, changed state to down
*Jun 9 20:39:16.705: %LINK-3-UPDOWN: Interface HundredGigE1/0/39, changed state to down
*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/40, changed state to down
*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/41, changed state to down
*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/42, changed state to down
*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/43, changed state to down
*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/44, changed state to down
*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/45, changed state to down
*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/46, changed state to down
*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/47, changed state to down
*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/48, changed state to down
*Jun 9 20:39:44.717: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 1/0
*Jun 9 20:40:15.052: %SPA_OIR-6-ONLINECARD: SPA (C9600-LC-24C) online in subslot 1/0
*interfaces come up again*
|
Symptom |
Action |
|
Module still fails P.O.S.T or module not listed in show module or show inventory command output. |
|
Reseat module (Similar syslogs are printed when the chassis recognizes the module and the module initializes).
*May 24 19:32:53.084: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F0 *May 24 19:32:53.084: %IOSXE_OIR-6-INSCARD: Card (cc) inserted in slot 1 *May 24 19:32:53.084: %IOSXE_OIR-6-ONLINECARD: Card (cc) online in slot 1 *May 24 19:32:53.151: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 1/0
Verify the module Power State is accepted.
Cat9600#show power module
Power Budget Mode : Dual Sup
Power Out of In
Mod Model No State Budget Instantaneous Peak Reset Reset
--- -------------------- -------- ------ ------------- ---- ------ -----
1 C9600-LC-48YL accepted 230 0 0 230 10
2 C9600-LC-24C accepted 200 0 0 200 10
3 C9600-SUP-1 accepted 775 0 0 775 202
4 C9600-SUP-1 accepted 775 0 0 775 202
FM1 C9606-FAN accepted 450 -- -- 450 --
--- -------------------- -------- ------ ------------- ---- ------ -----
Total allocated power: 2430
Total required power: 2430
Verify if there are any failed power supplies.
Cat9600#show power detail
Power Fan States
Supply Model No Type Capacity Status 1 2
------ -------------------- ---- -------- ------------ -----------
PS1 C9600-PWR-2KWAC ac 1050 W active good good
PS2 C9600-PWR-2KWAC ac 2000 W active good good
PS3 C9600-PWR-2KWAC ac n.a. fail bad bad
PS4 C9600-PWR-2KWAC ac n.a. fail bad bad
PS Current Configuration Mode : none
PS Current Operating State : none
Power supplies currently active : 2
Power supplies currently available : 4
Power Summary Maximum
(in Watts) Used Available
------------- ------ ---------
System Power 2905 2990
------------- ------ ---------
Total 2905 2990 <-- total power budget consumed vs maximum available
Power Budget Mode : Dual Sup
Power Out of In
Mod Model No State Budget Instantaneous Peak Reset Reset
--- -------------------- -------- ------ ------------- ---- ------ -----
1 C9600-LC-48TX accepted 315 0 0 315 10
2 C9600-LC-48S accepted 160 0 0 160 10
3 C9600-SUP-1 accepted 775 0 0 775 202
4 C9600-SUP-1 --- 775 -- -- 775 0
5 C9600-LC-48YL accepted 230 0 0 230 10
6 C9600-LC-24C accepted 200 0 0 200 10
FM1 C9606-FAN accepted 450 -- -- 450 --
--- -------------------- -------- ------ ------------- ---- ------ -----
Total allocated power: 2905
Total required power: 2905
Software and Firmware Mismatch
- Supervisors on different software or firmware versions fail to synchronize in High Availability Active/Standby roles.
- Line cards on a newer firmware version than compared to a Supervisor firmware version might fail to initialize.
- Instructions included in 'Upgrading the switch software' section in Release Notes of every major software train such as 16.12.X, 17.3.X, and 17.6.X
- Manually upgrade the firmware/bootldr of the Supervisor.
Note: Please review the Release Notes for Catalyst 9600 Series Switches Cisco IOS XE Amsterdan 17.3.X for more detailed information about how to manually upgrade the firmware/boot loader of the Supervisor.
Cat9600#show firmware version all
Current Bundled
Slot Device Name / Model Firmware Version Firmware Version Mismatch
----- ----------------------------- --------------------- ------------------- ---------
3 Supervisor Rommon (Active) 17.3.1r[FC2] N/A N/A <-- Firmware versions are the same
4 Supervisor Rommon (Standby) 17.3.1r[FC2] N/A N/A <-- Firmware versions are the same
- Eth Lan Spi 0x800005d3 N/A N/A
PS5 Fantray 18101008 N/A N/A
3 Supervisor CPLD
IO FPGA 19041620 19041620 No
Flash FPGA 190308b9 190308b9 No
4 Supervisor CPLD
IO FPGA 19041620 19041620 No
Flash FPGA 190308b9 190308b9 No
1 Line Card / C9600-LC-24C
IO FPGA 19070619 19070619 No
PMFPGA 122515 N/A N/A
Phy Firmware 1.56.1772 1.56.2270 Yes
2 Line Card / C9600-LC-48YL
IO FPGA 19070619 19070619 No
PMFPGA 122515 N/A N/A
Phy Firmware 1.56.1772 1.56.2270 Yes
5 Line Card / C9600-LC-48YL
IO FPGA 19070619 19070619 No
PMFPGA 122515 N/A N/A
Phy Firmware 1.56.1772 1.56.2270 Yes
6 Line Card / C9600-LC-24C
IO FPGA 19070619 19070619 No
PMFPGA 122515 N/A N/A
Phy Firmware 1.56.1772 1.56.2270 Yes
Slot Device Name / Model Firmware Version
--------- --------------------------------------- -----------------------------
PS1 Power Supply (PS PRI, PS_SEC, PS_I2C) (N/A, N/A, N/A)
PS2 Power Supply (PS PRI, PS_SEC, PS_I2C) (N/A, N/A, N/A)
PS3 Power Supply (PS PRI, PS_SEC, PS_I2C) (N/A, N/A, N/A)
PS4 Power Supply (PS PRI, PS_SEC, PS_I2C) (61.08.03, 61.02.0, N/A) Additional Troubleshooting
Note: hardware authentication status runs only at reload of switch. It does not run after physical reseat of line card at same or different slot, and this does not necessarily indicate an issue.
If authentication status NOT pass for a line card that is present, then execute these suggested steps:
- Collect the trace archive, review IOSRP, CMCC, platform_mgr and IOMD files for relevant information about this issue type.
- Physical OIR of affected component OR reload of switch can be tried to see if that recovers.
- If issue persists, open a TAC case.
Cat9600#show platform hardware authentication status
Fan Tray Authentication: pass
Line Card:1 Authentication: pass
Line Card:2 Authentication: pass
SUP0 Authentication: pass
SUP1 Authentication: pass
Line Card:5 Authentication: Not Available
Line Card:6 Authentication: Not AvailableCat9600#show hw-module subslot 1/0 oir internal
IOMD for supervisor slot does not appear
WARNING: This command is not intended for production use
and should only be used under the supervision of
Cisco Systems technical support personnel.
sm(spa_oir_tsm subslot 1/0 TSM), running yes, state ready
Admin Status: admin enabled, Operational Status: ok(1)
Last reset Reason: power up
TSM Context:
configured_spa_type 0xD9A <-- IDPROM is accessible
soft remove fail code 0x0(none)
last_fail_code 0x0(none)
timed_fail_count 0, failed_spa_type 0x0
flags 0x10)
Subslot:
spa type 0xD9A, active spa type 0xD9A
subslot flags 0x0, plugin flags 0x0
TSM Parameters:
wait_psm_ready_timeout 540000 ms, init_timeout 300000 ms
remove_timeout 120000 ms, recovery_delay 5000 ms
fail_time_period 1200000 ms, max_fail_count 5
does not support pre-configurationCat9600# show platform software iomd redundancy
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Local RF state = ACTIVE
Peer RF state = STANDBY HOT
slot PSM STATE SPA INTF HA_STATE HA_ACTIVE
1 ready started ready 00:05:54
2 ready idle <-- potential issue
3 ready started ready 00:05:37 ***active RP
4 ready started ready 00:05:37Cat9600#show idprom module 1 eeprom detail
Slot 1 EEPROM data:
EEPROM version : 4
Compatible Type : 0xFF
Controller Type : 3482
Hardware Revision : 1.0
PCB Part Number : 73-19545-02
Board Revision : A0
Deviation Number : 0
Fab Version : 02
PCB Serial Number : CAT2313L2VP
RMA Test History : 00
RMA Number : 0-0-0-0
RMA History : 00
Top Assy. Part Number : 068-102161-01
Top Assy. Revision : A0
CLEI Code : COUIBGRCAA
Product Identifier (PID) : C9600-LC-24C
Version Identifier (VID) : V01
Base MAC Address : DC 8C 37 A0 C8 80
MAC Address block size : 128
Environment Monitor Data : 06 00 00 00 0C 80 C8 00
A6
Environment Monitor Data : 00 06 00 FA
Manufacturing Test Data : 00 00 00 00 00 00 00 00
Field Diagnostics Data : 00 00 00 00 00 00 00 00
Platform features : 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
Environment Monitor Data :
Description : InltFrnt
Shutdown threshold : 060
Critical threshold : 055
Major threshold : 050
Minor threshold : 045
Environment Monitor Data :
Description : InltRear
Shutdown threshold : 060
Critical threshold : 055
Major threshold : 050
Minor threshold : 045
Environment Monitor Data :
Description : OtltFrnt
Shutdown threshold : 090
Critical threshold : 085
Major threshold : 080
Minor threshold : 075
Environment Monitor Data :
Description : OtltRear
Shutdown threshold : 090
Critical threshold : 085
Major threshold : 080
Minor threshold : 075Use the lc-portmap command to obtain information on the linecard interfaces mapping to the ASIC/Core in the Supervisor.
Cat9600#show platform hardware iomd 1/0 lc-portmap detail
IOMD SUMMARY
==============
spa_type: 0xd9a
my_slot: 1
active-slot: 0
is_active: 1
bulk_sync_done: 1
bulk_sync_in_progress: 0
bulk_sync_received: 0
is_plugin_start_done: 1
Num_ports: 48 num_UDAP 3s: 3
PSM SUMMARY
===============
PSM state: ready
ha_state:
is_spa_ok: 0
is_power_ok: 0
psm_flags: 0x0
bay_flags: 0x0
plugin_flags: 0x0
Port fp asic core mac cntx asic speed xcvr- xcvr- run- phy- i2c- xcvr- if- SerTune- False- Rcvr- Serdes-
port pres type speed num ctrl sm state Status LinkUp idle set
================================================================================================================================
0 1 0 0 0 0 0 40G no 0 unk 0 0 0 0 NOTS no 0 0 1
1 2 0 0 2 1 8 40G no 0 unk 0 0 0 0 NOTS no 0 2 3
2 3 0 0 16 0 16 40G no 0 unk 1 0 0 0 NOTS no 0 16 17
3 4 0 0 18 1 24 40G no 0 unk 1 0 0 0 NOTS no 0 18 19
4 5 0 1 14 1 8 40G no 0 unk 2 0 0 0 NOTS no 0 14 15
5 6 0 1 12 0 0 40G no 0 unk 2 0 0 0 NOTS no 0 12 13
6 7 0 1 30 1 24 40G no 0 unk 3 0 0 0 NOTS no 0 30 31
7 8 0 1 28 0 16 40G no 0 unk 3 0 0 0 NOTS no 0 28 29
8 9 1 0 0 0 0 40G no 0 unk 4 0 0 0 NOTS no 0 0 1
9 10 1 0 2 1 8 40G no 0 unk 4 0 0 0 NOTS no 0 2 3
10 11 1 0 16 0 16 40G no 0 unk 5 0 0 0 NOTS no 0 16 17
11 12 1 0 18 1 24 40G no 0 unk 5 0 0 0 NOTS no 0 18 19
12 13 1 1 14 1 8 40G no 0 unk 6 0 0 0 NOTS no 0 14 15
13 14 1 1 12 0 0 40G no 0 unk 6 0 0 0 NOTS no 0 12 13
14 15 1 1 30 1 24 40G no 0 unk 7 0 0 0 NOTS no 0 30 31
15 16 1 1 28 0 16 40G no 0 unk 7 0 0 0 NOTS no 0 28 29
16 17 2 0 0 0 0 40G no 0 unk 8 1 0 0 NOTS no 0 0 1
17 18 2 0 2 1 8 40G no 0 unk 8 1 0 0 NOTS no 0 2 3
18 19 2 0 16 0 16 40G no 0 unk 9 1 0 0 NOTS no 0 16 17
19 20 2 0 18 1 24 40G no 0 unk 9 1 0 0 NOTS no 0 18 19
20 21 2 1 14 1 8 40G no 0 unk 10 1 0 0 NOTS no 0 14 15
21 22 2 1 12 0 0 40G no 0 unk 10 1 0 0 NOTS no 0 12 13
22 23 2 1 30 1 24 40G no 0 unk 11 1 0 0 NOTS no 0 30 31
23 24 2 1 28 0 16 40G no 0 unk 11 1 0 0 NOTS no 0 28 29
24 25 0 0 0 0 0 100G no 0 unk 0 0 0 0 NOTS no 0 0 1 2 3
25 26 0 0 0 0 0 100G no 0 unk 0 0 0 0 NOTS no 0 0 1 2 3
26 27 0 0 16 0 16 100G no 0 unk 1 0 0 0 NOTS no 0 16 17 18 19
27 28 0 0 16 0 16 100G no 0 unk 1 0 0 0 NOTS no 0 16 17 18 19
28 29 0 1 12 0 0 100G no 0 unk 2 0 0 0 NOTS no 0 12 13 14 15
29 30 0 1 12 0 0 100G no 0 unk 2 0 0 0 NOTS no 0 12 13 14 15
30 31 0 1 28 0 16 100G no 0 unk 3 0 0 0 NOTS no 0 28 29 30 31
31 32 0 1 28 0 16 100G no 0 unk 3 0 0 0 NOTS no 0 28 29 30 31
32 33 1 0 0 0 0 100G no 0 unk 4 0 0 0 NOTS no 0 0 1 2 3
33 34 1 0 0 0 0 100G no 0 unk 4 0 0 0 NOTS no 0 0 1 2 3
34 35 1 0 16 0 16 100G no 0 unk 5 0 0 0 NOTS no 0 16 17 18 19
35 36 1 0 16 0 16 100G no 0 unk 5 0 0 0 NOTS no 0 16 17 18 19
36 37 1 1 12 0 0 100G no 0 unk 6 0 0 0 NOTS no 0 12 13 14 15
37 38 1 1 12 0 0 100G no 0 unk 6 0 0 0 NOTS no 0 12 13 14 15
38 39 1 1 28 0 16 100G no 0 unk 7 0 0 0 NOTS no 0 28 29 30 31
39 40 1 1 28 0 16 100G no 0 unk 7 0 0 0 NOTS no 0 28 29 30 31
40 41 2 0 0 0 0 100G no 0 unk 8 1 0 0 NOTS no 0 0 1 2 3
41 42 2 0 0 0 0 100G no 0 unk 8 1 0 0 NOTS no 0 0 1 2 3
42 43 2 0 16 0 16 100G no 0 unk 9 1 0 0 NOTS no 0 16 17 18 19
43 44 2 0 16 0 16 100G no 0 unk 9 1 0 0 NOTS no 0 16 17 18 19
44 45 2 1 12 0 0 100G no 0 unk 10 1 0 0 NOTS no 0 12 13 14 15
45 46 2 1 12 0 0 100G no 0 unk 10 1 0 0 NOTS no 0 12 13 14 15
46 47 2 1 28 0 16 100G no 0 unk 11 1 0 0 NOTS no 0 28 29 30 31
47 48 2 1 28 0 16 100G no 0 unk 11 1 0 0 NOTS no 0 28 29 30 31
Cat9600#show platform software fed [active | standby] ifm mappings Interface IF_ID Inst Asic Core Port SubPort Mac Cntx LPN GPN Type Active TwentyFiveGigE1/0/1 0x9 0 0 0 0 0 0 0 1 101 NIF Y TwentyFiveGigE1/0/2 0xa 0 0 0 4 0 1 1 2 102 NIF Y TwentyFiveGigE1/0/3 0xb 0 0 0 8 0 2 2 3 103 NIF Y TwentyFiveGigE1/0/4 0xc 0 0 0 12 0 3 3 4 104 NIF Y TwentyFiveGigE1/0/5 0xd 0 0 0 16 0 16 0 5 105 NIF Y TwentyFiveGigE1/0/6 0xe 0 0 0 20 0 17 1 6 106 NIF Y TwentyFiveGigE1/0/7 0xf 0 0 0 24 0 18 2 7 107 NIF Y TwentyFiveGigE1/0/8 0x10 0 0 0 28 0 19 3 8 108 NIF Y TwentyFiveGigE1/0/9 0x11 1 0 1 12 0 15 3 9 109 NIF Y TwentyFiveGigE1/0/10 0x12 1 0 1 8 0 14 2 10 110 NIF Y TwentyFiveGigE1/0/11 0x13 1 0 1 4 0 13 1 11 111 NIF Y TwentyFiveGigE1/0/12 0x14 1 0 1 0 0 12 0 12 112 NIF Y TwentyFiveGigE1/0/13 0x15 1 0 1 28 0 31 3 13 113 NIF Y TwentyFiveGigE1/0/14 0x16 1 0 1 24 0 30 2 14 114 NIF Y TwentyFiveGigE1/0/15 0x17 1 0 1 20 0 29 1 15 115 NIF Y TwentyFiveGigE1/0/16 0x18 1 0 1 16 0 28 0 16 116 NIF Y TwentyFiveGigE1/0/17 0x19 2 1 0 0 0 0 0 17 117 NIF Y TwentyFiveGigE1/0/18 0x1a 2 1 0 4 0 1 1 18 118 NIF Y TwentyFiveGigE1/0/19 0x1b 2 1 0 8 0 2 2 19 119 NIF Y TwentyFiveGigE1/0/20 0x1c 2 1 0 12 0 3 3 20 120 NIF Y TwentyFiveGigE1/0/21 0x1d 2 1 0 16 0 16 0 21 121 NIF Y TwentyFiveGigE1/0/22 0x1e 2 1 0 20 0 17 1 22 122 NIF Y TwentyFiveGigE1/0/23 0x1f 2 1 0 24 0 18 2 23 123 NIF Y TwentyFiveGigE1/0/24 0x20 2 1 0 28 0 19 3 24 124 NIF Y TwentyFiveGigE1/0/25 0x21 3 1 1 12 0 15 3 25 125 NIF Y TwentyFiveGigE1/0/26 0x22 3 1 1 8 0 14 2 26 126 NIF Y TwentyFiveGigE1/0/27 0x23 3 1 1 4 0 13 1 27 127 NIF Y TwentyFiveGigE1/0/28 0x24 3 1 1 0 0 12 0 28 128 NIF Y TwentyFiveGigE1/0/29 0x25 3 1 1 28 0 31 3 29 129 NIF Y TwentyFiveGigE1/0/30 0x26 3 1 1 24 0 30 2 30 130 NIF Y TwentyFiveGigE1/0/31 0x27 3 1 1 20 0 29 1 31 131 NIF Y TwentyFiveGigE1/0/32 0x28 3 1 1 16 0 28 0 32 132 NIF Y TwentyFiveGigE1/0/33 0x29 4 2 0 0 0 0 0 33 133 NIF Y TwentyFiveGigE1/0/34 0x2a 4 2 0 4 0 1 1 34 134 NIF Y TwentyFiveGigE1/0/35 0x2b 4 2 0 8 0 2 2 35 135 NIF Y TwentyFiveGigE1/0/36 0x2c 4 2 0 12 0 3 3 36 136 NIF Y TwentyFiveGigE1/0/37 0x2d 4 2 0 16 0 16 0 37 137 NIF Y TwentyFiveGigE1/0/38 0x2e 4 2 0 20 0 17 1 38 138 NIF Y TwentyFiveGigE1/0/39 0x2f 4 2 0 24 0 18 2 39 139 NIF Y TwentyFiveGigE1/0/40 0x30 4 2 0 28 0 19 3 40 140 NIF Y TwentyFiveGigE1/0/41 0x31 5 2 1 12 0 15 3 41 141 NIF Y TwentyFiveGigE1/0/42 0x32 5 2 1 8 0 14 2 42 142 NIF Y TwentyFiveGigE1/0/43 0x33 5 2 1 4 0 13 1 43 143 NIF Y TwentyFiveGigE1/0/44 0x34 5 2 1 0 0 12 0 44 144 NIF Y TwentyFiveGigE1/0/45 0x35 5 2 1 28 0 31 3 45 145 NIF Y TwentyFiveGigE1/0/46 0x36 5 2 1 24 0 30 2 46 146 NIF Y TwentyFiveGigE1/0/47 0x37 5 2 1 20 0 29 1 47 147 NIF Y TwentyFiveGigE1/0/48 0x38 5 2 1 16 0 28 0 48 148 NIF Y TwentyFiveGigE2/0/2 0x3a 0 0 0 9 0 9 9 2 202 NIF Y TwentyFiveGigE2/0/3 0x3b 0 0 0 10 0 10 10 3 203 NIF Y TwentyFiveGigE2/0/4 0x3c 0 0 0 11 0 11 11 4 204 NIF Y TwentyFiveGigE2/0/5 0x3d 0 0 0 18 0 24 8 5 205 NIF Y TwentyFiveGigE2/0/6 0x3e 0 0 0 25 0 25 9 6 206 NIF Y TwentyFiveGigE2/0/7 0x3f 0 0 0 26 0 26 10 7 207 NIF Y TwentyFiveGigE2/0/8 0x40 0 0 0 27 0 27 11 8 208 NIF Y TwentyFiveGigE2/0/9 0x41 1 0 1 7 0 7 7 9 209 NIF Y TwentyFiveGigE2/0/10 0x42 1 0 1 6 0 6 6 10 210 NIF Y TwentyFiveGigE2/0/11 0x43 1 0 1 5 0 5 5 11 211 NIF Y TwentyFiveGigE2/0/12 0x44 1 0 1 1 0 4 4 12 212 NIF Y TwentyFiveGigE2/0/13 0x45 1 0 1 23 0 23 7 13 213 NIF Y TwentyFiveGigE2/0/14 0x46 1 0 1 22 0 22 6 14 214 NIF Y TwentyFiveGigE2/0/15 0x47 1 0 1 21 0 21 5 15 215 NIF Y TwentyFiveGigE2/0/16 0x48 1 0 1 17 0 20 4 16 216 NIF Y TwentyFiveGigE2/0/17 0x49 2 1 0 2 0 8 8 17 217 NIF Y TwentyFiveGigE2/0/18 0x4a 2 1 0 9 0 9 9 18 218 NIF Y TwentyFiveGigE2/0/19 0x4b 2 1 0 10 0 10 10 19 219 NIF Y TwentyFiveGigE2/0/20 0x4c 2 1 0 11 0 11 11 20 220 NIF Y TwentyFiveGigE2/0/21 0x4d 2 1 0 18 0 24 8 21 221 NIF Y TwentyFiveGigE2/0/22 0x4e 2 1 0 25 0 25 9 22 222 NIF Y TwentyFiveGigE2/0/23 0x4f 2 1 0 26 0 26 10 23 223 NIF Y TwentyFiveGigE2/0/24 0x50 2 1 0 27 0 27 11 24 224 NIF Y TwentyFiveGigE2/0/25 0x51 3 1 1 7 0 7 7 25 225 NIF Y TwentyFiveGigE2/0/26 0x52 3 1 1 6 0 6 6 26 226 NIF Y TwentyFiveGigE2/0/27 0x53 3 1 1 5 0 5 5 27 227 NIF Y TwentyFiveGigE2/0/28 0x54 3 1 1 1 0 4 4 28 228 NIF Y TwentyFiveGigE2/0/29 0x55 3 1 1 23 0 23 7 29 229 NIF Y TwentyFiveGigE2/0/30 0x56 3 1 1 22 0 22 6 30 230 NIF Y TwentyFiveGigE2/0/31 0x57 3 1 1 21 0 21 5 31 231 NIF Y TwentyFiveGigE2/0/32 0x58 3 1 1 17 0 20 4 32 232 NIF Y TwentyFiveGigE2/0/33 0x59 4 2 0 2 0 8 8 33 233 NIF Y TwentyFiveGigE2/0/34 0x5a 4 2 0 9 0 9 9 34 234 NIF Y TwentyFiveGigE2/0/35 0x5b 4 2 0 10 0 10 10 35 235 NIF Y TwentyFiveGigE2/0/36 0x5c 4 2 0 11 0 11 11 36 236 NIF Y TwentyFiveGigE2/0/37 0x5d 4 2 0 18 0 24 8 37 237 NIF Y TwentyFiveGigE2/0/38 0x5e 4 2 0 25 0 25 9 38 238 NIF Y TwentyFiveGigE2/0/39 0x5f 4 2 0 26 0 26 10 39 239 NIF Y TwentyFiveGigE2/0/40 0x60 4 2 0 27 0 27 11 40 240 NIF Y TwentyFiveGigE2/0/41 0x61 5 2 1 7 0 7 7 41 241 NIF Y TwentyFiveGigE2/0/42 0x62 5 2 1 6 0 6 6 42 242 NIF Y TwentyFiveGigE2/0/43 0x63 5 2 1 5 0 5 5 43 243 NIF Y TwentyFiveGigE2/0/44 0x64 5 2 1 1 0 4 4 44 244 NIF Y TwentyFiveGigE2/0/45 0x65 5 2 1 23 0 23 7 45 245 NIF Y TwentyFiveGigE2/0/46 0x66 5 2 1 22 0 22 6 46 246 NIF Y TwentyFiveGigE2/0/47 0x67 5 2 1 21 0 21 5 47 247 NIF Y TwentyFiveGigE2/0/48 0x68 5 2 1 17 0 20 4 48 248 NIF Y
Commands to Collect for TAC
The most common hardware issues and validation commands are covered in this guide, with appropriate remediation steps. However, in the event that this guide did not resolve your issue please collect the command list shown and attach them to your TAC service request.
show tech-support show post show platform hardware authentication status show platform hardware chassis fantray detail show platform software iomd redundancy
request platform software trace archive
service internal (for the standby command 'service internal' needs to be enabled first)
request platform software trace slot rp standby archive target stby-bootflash
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
05-Jul-2023 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)