Contents

Introduction

Prerequisites

Requirements

Components Used

Terminology

Chassis Overview

Troubleshoot

Symptoms of Module Hardware Issues

Verify Module Status

Troubleshoot Module Status

GOLD Tests

Software and Firmware Mismatch

Additional Troubleshooting

Commands to Collect for TAC

Introduction

This document describes how to identify, isolate, and troubleshoot symptoms of hardware failures on Catalyst 9600 Supervisors and Line Cards.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on Catalyst 9600 Series Switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Terminology

Term

Definition

Context

Supervisor

The Cisco Catalyst 9600 Series Supervisor Engine 1 is powered by three UADP 3.0 ASICs and one x86 CPU processor. The three ASICs are interconnected with a 3.2-Tbps ASIC interconnect on each ASIC. Sup-1 provides 9.6 Tbps (4.8 Tbps full duplex). With the Cisco Catalyst 9606R chassis, each slot has 2.4 Tbps with Sup-1.

This document contains commands to isolate hardware failure on Supervisor engines.

Power On Self Test (P.O.S.T.)

After the Supervisor and Line Cards are initialized after power on, a Power On Self Test (POST) executes to ensure the hardware integrity of the modules. POST is also executed during Online Insertion & Removal (OIR) of a line card.

Show post command displays the results of these tests.

Generic Online Diagnostic Subsytems (GOLD.)

GOLD tests are run after the system is online. Some tests are run periodically as Health Monitoring tests. All the tests can be run on demand by the user.

It is used to detect module hardware failures.

Online Insertion & Removal (O.I.R.)

This feature can power-cycle a module in a given slot via the command line interface (CLI). This is useful when there is not an on-site resource that can psychically reseat the module.

O.I.R. is a great feature to perform isolate hardware failure on modules.

Chassis Overview

Catalyst C9606-R

- 6 Slot - 8RU Modular Chassis

- 2 Supervisor Slots + 4 Line Card Slots

Troubleshoot

Symptoms of Module Hardware Issues

Symptom

Possible Root Cause

- One or multiple interfaces are in down/down(notconnect) status on a given line card.

- Supervisor or Linecard is not listed in C.L.I. outputs like show module or show inventory.

- No console output on Supervisor.

- Red or Amber L.E.D. (solid or blinking) on Supervisor or Line Card. No L.E.D. on Supervisor or Line Card.

Module is not properly installed, not initialized, or encountered hardware failure due to failing the P.O.S.T.

Verify Module Status

- It is important to first check that the chassis recognizes the module.

- The ideal status for a module is 'ok'.

Module Status

Possible Root Cause

Action

ok

N/A.

None

post-fail

Potential hardware failure.

Refer to Troubleshoot Module Status.

other

Module failed to initialize.

Refer to Troubleshoot Module Status.

Not listed in output

Not properly installed or not receiving sufficient power.

Check for available power budget.

Visually inspect L.E.D. status of module.

Cat9600#show module

Chassis Type: C9606R

Mod Ports Card Type                                   Model          Serial No.

---+-----+--------------------------------------+--------------+--------------

1   24   24-Port 40GE/12-Port 100GE                  C9600-LC-24C

2   48   48-Port 10GE / 25GE                         C9600-LC-48YL

3   0    Supervisor 1 Module                         C9600-SUP-1

4   0    Supervisor 1 Module                         C9600-SUP-1

Mod MAC addresses                    Hw   Fw           Sw                 Status

---+--------------------------------+----+------------+------------------+--------

1   DC8C.37A0.C880 to DC8C.37A0.C8FF 1.0  17.1.1[FC2]   16.12.03           post-fail

2   DC8C.3773.1E00 to DC8C.3773.1E7F 1.0  17.1.1[FC2]   16.12.03           ok

3   DC8C.379F.ED80 to DC8C.379F.EDFF 1.0  17.1.1[FC2]   16.12.03           ok

4   DC8C.379F.EDF1 to DC8C.379F.EDF9 1.0  17.1.1[FC2]   16.12.03           ok

//other example

Mod MAC addresses                    Hw   Fw           Sw                 Status

---+--------------------------------+----+------------+------------------+--------

1   DC8C.37A0.C880 to DC8C.37A0.C8FF 1.3  17.3.1r[FC2]  17.03.05           power-deny

2   DC8C.37A0.C880 to DC8C.37A0.C8FF 1.3  17.3.1r[FC2]  17.03.05           power-deny

3   DC8C.37A0.C880 to DC8C.37A0.C8FF 1.4  17.3.1r[FC2]  17.03.05           ok

Troubleshoot Module Status

Symptom

Action

Module not listed or status is not 'ok'.

Proceed with the next steps.

Verify Syslog

Line Card Offline

%IOSXE_OIR-6-OFFLINECARD: Card (rp) offline in slot R1

%IOSXE_OIR-6-REMSPA: SPA removed from subslot 6/0, interfaces disabled

%SPA_OIR-6-OFFLINECARD: SPA (C9600-LC-48TX) offline in subslot 6/0

Power-Budget Issue

%CMRP_PFU-3-PWR_MGMT_LC_SHUTDOWN: R0/0: cmand: WARNING: Linecard in slot 6 with priority 0 and a power requirement of 315 watts has shutdown.

%CMRP_PFU-3-PWR_MGMT_ALARM: Chassis 1 R0/0: cmand: WARNING: System does not have sufficient input power for minimum reliable operation requiring 1161 watts. The system needs 1101 watts of additional power.

Diagnostic Test Failed

*May 29 02:10:19.523: %PM-4-ERR_DISABLE: diagnostics error detected on Fo2/0/1, putting Fo2/0/1 in err-disable state

Check Module Uptime

- Onboard Failure Logging (OBFL) gives us insight to reload reasons such as reload command, software exception, power off, and so on.

- Check time stamps that could correlate to sudden power loss to the module or external factor contributing to the module going down.

Cat9600#show logging onboard rp (active|standby) uptime detail

--------------------------------------------------------------------------------

UPTIME SUMMARY INFORMATION

--------------------------------------------------------------------------------

First customer power on : 08/28/2021 13:08:41

Total uptime            :  0  years  41 weeks  2  days  15 hours  10 minutes  <-- Total Uptime

Total downtime          :  0  years  7  weeks  6  days  7  hours  11 minutes

Number of resets        : 14                                                  <-- Total number of resets

Number of slot changes  : 0

Current reset reason    : PowerOn                                             <-- Last reload reason

Current reset timestamp : 11/06/2021 17:25:29

Current slot            : 3

Chassis type            : 55

Current uptime          :  0  years  39 weeks  1  days  18 hours  5  minutes  <-- Current Uptime

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------

UPTIME CONTINUOUS INFORMATION

--------------------------------------------------------------------------------

Time Stamp          | Reset                       | Uptime

MM/DD/YYYY HH:MM:SS | Reason                      | years weeks days hours minutes

--------------------------------------------------------------------------------

08/28/2021 13:08:41   PowerOn                       0     0     0     0     0

08/28/2021 13:16:29   Reload                        0     0     0     0     0

08/28/2021 13:29:28   Reload                        0     0     0     0     5

08/28/2021 13:42:30   Reload                        0     0     0     0     5

08/28/2021 13:55:14   Image Install                 0     0     0     0     5  <-- Reset history by reason

Cat9600#show logging onboard slot (1|2|4|5|6|7) uptime detail

--------------------------------------------------------------------------------

UPTIME SUMMARY INFORMATION

--------------------------------------------------------------------------------

First customer power on : 08/08/2018 09:30:33

Total uptime            :  0  years  11 weeks  1  days  10 hours  40 minutes

Total downtime          :  4  years  32 weeks  1  days  23 hours  25 minutes

Number of resets        : 81

Number of slot changes  : 5

Current reset reason    : CP_RESET_CPU_GOT_RESET

Current reset timestamp : 05/24/2023 19:35:58

Current slot            : 1

Chassis type            : 30

Current uptime          :  0  years  2  weeks  0  days  0  hours  0  minutes

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------

UPTIME CONTINUOUS INFORMATION

--------------------------------------------------------------------------------

Time Stamp          | Reset                       | Uptime

MM/DD/YYYY HH:MM:SS | Reason                      | years weeks days hours minutes

--------------------------------------------------------------------------------

04/26/2021 21:39:45   CP_RESET_CPU_GOT_RESET        0     0     0     0     0

04/27/2021 14:12:04   CP_RESET_POWER_ON             0     0     0     6     0

04/28/2021 14:39:30   CP_RESET_POWER_ON             0     0     0     14    0

04/30/2021 12:46:59   CP_RESET_POWER_ON             0     0     0     13    0

Verify POST

Failed POST example

Cat9600#show post

Stored system POST messages:

Switch C9606R

--------------

Fri May 29 02:10:18 2020 POST: Module: 1 Mac Loopback Begin

Fri May 29 02:10:18 2020 POST: Module: 1 Mac Loopback: loopback Test: End, Status Passed

Fri May 29 02:10:18 2020 POST: Module: 2 Mac Loopback Begin

Fri May 29 02:10:18 2020 POST: Mac Loopback: Failed For Interface :  TwentyFiveGigE2/2/0/38

Fri May 29 02:10:18 2020 POST: Mac Loopback: Failed For Interface :  TwentyFiveGigE2/2/0/39

Fri May 29 02:10:18 2020 POST: Mac Loopback: Failed For Interface :  TwentyFiveGigE2/2/0/40

Fri May 29 02:10:18 2020 POST: Module: 2 Mac Loopback: loopback Test: End, Status Failed

POST passed example

Cat9600#show post

Stored system POST messages:

Switch C9606R

--------------

POST: MBIST Tests : Begin

POST: MBIST Tests : End, Status Passed

POST: Module: 3 PHY Loopback: loopback Test: Begin

POST: Module: 3 PHY Loopback: loopback Test: End, Status Passed

POST: Module: 2 PHY Loopback: loopback Test: Begin

POST: Module: 2 PHY Loopback: loopback Test: End, Status Passed

POST: Module: 1 PHY Loopback: loopback Test: Begin

POST: Module: 1 PHY Loopback: loopback Test: End, Status Passed

POST: Module: 5 PHY Loopback: loopback Test: Begin

POST: Module: 5 PHY Loopback: loopback Test: End, Status Passed

Cat9600#show platform

Chassis type: C9606R

Slot      Type                State                 Insert time (ago)

--------- ------------------- --------------------- -----------------

1         C9600-LC-24C        ok                    02:45:09

1/0      C9600-LC-24C        ok                    02:43:15

2         C9600-LC-48YL       faulty                02:45:09

2/0      C9600-LC-48YL       ok                    02:43:14

3         C9600-SUP-1         ok                    02:45:09

3/0      C9600-SUP-1         ok                    02:43:14

4         C9600-SUP-1         ok                    02:45:09

4/0      C9600-SUP-1         ok                    02:43:14

R0        C9600-SUP-1         ok, active            02:45:09

R1        C9600-SUP-1         init, standby         02:45:09

P1        C9600-PWR-2KWAC     ok                    02:44:21

P2        C9600-PWR-2KWAC     ok                    02:44:21

P3        C9600-PWR-2KWAC     ps, f0, f1, fail      02:44:21

P4        C9600-PWR-2KWAC     ps, f0, f1, fail      02:44:21

P5        C9606-FAN           ok                    02:44:21

Slot      CPLD Version        Firmware Version

--------- ------------------- ---------------------------------------

1         19070619            17.1.1[FC2]

2         19070619            17.1.1[FC2]

3         19041620            17.1.1[FC2]

4         19041620            17.1.1[FC2]

GOLD Tests

- If module has failed P.O.S.T or listed as 'faulty' then diagnostics to get the failure reason must be collected.

- Run hardware diagnostics to quickly identify potential hardware failure within the line cards or Supervisors.

Types of diagnostic tests:

- Health monitoring tests - Enabled by default and it runs in background for every few minutes. These tests are non disruptive.

- On demand tests - Run by the user with a CLI. Some of these tests are disruptive.

Diagnostic Test

What is it testing? Context? Impact?

TestGoldPktLoopback

This test is very similar to the Macloopback test.

TestOBFL

Verifies the on-board failure logging capabilities. This test prints a diagnostic message and is logged onto the OBFL.

TestThermal

Verifies the temperature value from the sensor does not exceed the threshold. It is a non-disruptive test and can be run as a health monitor test.

TestPortTxMonitoring

This test monitors the TX counters of a connected interface. This test verifies if the connected port is able to send the packets or not. It is a non-disruptive test and can be run as a health monitoring test.

TestFanTray

This test verifies if the fan tray has been inserted and works properly on the board. It is a non-disruptive test and can be run as a health monitoring test.

TestScratchRegister

The Scratch Register test monitors the health of application-specific integrated circuits (ASICs) it writes values into registers and reads back the values from these registers. It is a non-disruptive test and can be run as a health monitoring test.

TestConsistencyCheckMcast

This test monitors to check if Mcast programming is correct. This test checks with Forwarding object manger to identify incomplete entries or long pending configurations to hardware.

Run an on-demand diagnostic test on a module.

Cat9600#diagnostic start switch module 3 test ?

Diagnostics test suite attributes:

M/C/* - Minimal bootup level test / Complete bootup level test / NA

B/* - Basic ondemand test / NA

P/V/* - Per port test / Per device test / NA

D/N/* - Disruptive test / Non-disruptive test / NA

S/* - Only applicable to standby unit / NA

X/* - Not a health monitoring test / NA

F/* - Fixed monitoring interval test / NA

E/* - Always enabled monitoring test / NA

A/I - Monitoring is active / Monitoring is inactive

Test Interval Thre-

ID Test Name Attributes day hh:mm:ss.ms shold

==== ================================== ============ =============== =====

1) TestOBFL ------------------------> *B*N*X**I not configured n/a

2) TestFantray ---------------------> *B*N****A 000 00:01:40.00 1

3) TestThermal ---------------------> *B*N****A 000 00:01:30.00 1

4) TestScratchRegister -------------> *B*N****A 000 00:01:30.00 5

5) TestConsistencyCheck ------------> *B*N****A 000 00:01:30.00 1

WORD Test ID list (e.g. 1,3-6) or Test Name

Cat9600#diagnostic start switch 1 module 3 test 2

//syslog prints upon start and finish of diagnostic test.

*Mar 24 21:48:47.968: %DIAG-6-TEST_RUNNING: switch 1 module 3: Running TestFantray{ID=2} ...

*Mar 24 21:48:47.969: %DIAG-6-TEST_OK: switch 1 module 3: TestFantray{ID=2} has completed successfully <-- Passed

Symptom

Action

Module failed diagnostic test.

Perform an O.I.R. of the module.

Run on-demand OIR on a module

Cat9600#hw-module subslot 1/0 oir power-cycle

Proceed with power cycle of module? [confirm]

*Jun 9 20:39:14.686: %IOSXE_OIR-6-REMSPA: SPA removed from subslot 1/0, interfaces disabled

*Jun 9 20:39:14.721: %SPA_OIR-6-OFFLINECARD: SPA (C9600-LC-24C) offline in subslot 1/0

*Jun 9 20:39:16.686: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/1, changed state to down

*Jun 9 20:39:16.686: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/2, changed state to down

*Jun 9 20:39:16.687: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/3, changed state to down

*Jun 9 20:39:16.687: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/4, changed state to down

*Jun 9 20:39:16.689: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/5, changed state to down

*Jun 9 20:39:16.690: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/6, changed state to down

*Jun 9 20:39:16.690: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/7, changed state to down

*Jun 9 20:39:16.690: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/8, changed state to down

*Jun 9 20:39:16.690: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/9, changed state to down

*Jun 9 20:39:16.691: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/10, changed state to down

*Jun 9 20:39:16.691: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/11, changed state to down

*Jun 9 20:39:16.693: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/12, changed state to down

*Jun 9 20:39:16.693: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/13, changed state to down

*Jun 9 20:39:16.694: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/14, changed state to down

*Jun 9 20:39:16.694: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/15, changed state to down

*Jun 9 20:39:16.694: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/16, changed state to down

*Jun 9 20:39:16.694: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/17, changed state to down

*Jun 9 20:39:16.695: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/18, changed state to down

*Jun 9 20:39:16.695: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/19, changed state to down

*Jun 9 20:39:16.697: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/20, changed state to down

*Jun 9 20:39:16.697: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/21, changed state to down

*Jun 9 20:39:16.698: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/22, changed state to down

*Jun 9 20:39:16.698: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/23, changed state to down

*Jun 9 20:39:16.698: %LINK-3-UPDOWN: Interface FortyGigabitEthernet1/0/24, changed state to down

*Jun 9 20:39:16.698: %LINK-3-UPDOWN: Interface HundredGigE1/0/25, changed state to down

*Jun 9 20:39:16.699: %LINK-3-UPDOWN: Interface HundredGigE1/0/26, changed state to down

*Jun 9 20:39:16.699: %LINK-3-UPDOWN: Interface HundredGigE1/0/27, changed state to down

*Jun 9 20:39:16.701: %LINK-3-UPDOWN: Interface HundredGigE1/0/28, changed state to down

*Jun 9 20:39:16.701: %LINK-3-UPDOWN: Interface HundredGigE1/0/29, changed state to down

*Jun 9 20:39:16.702: %LINK-3-UPDOWN: Interface HundredGigE1/0/30, changed state to down

*Jun 9 20:39:16.702: %LINK-3-UPDOWN: Interface HundredGigE1/0/31, changed state to down

*Jun 9 20:39:16.702: %LINK-3-UPDOWN: Interface HundredGigE1/0/32, changed state to down

*Jun 9 20:39:16.702: %LINK-3-UPDOWN: Interface HundredGigE1/0/33, changed state to down

*Jun 9 20:39:16.703: %LINK-3-UPDOWN: Interface HundredGigE1/0/34, changed state to down

*Jun 9 20:39:16.703: %LINK-3-UPDOWN: Interface HundredGigE1/0/35, changed state to down

*Jun 9 20:39:16.704: %LINK-3-UPDOWN: Interface HundredGigE1/0/36, changed state to down

*Jun 9 20:39:16.704: %LINK-3-UPDOWN: Interface HundredGigE1/0/37, changed state to down

*Jun 9 20:39:16.705: %LINK-3-UPDOWN: Interface HundredGigE1/0/38, changed state to down

*Jun 9 20:39:16.705: %LINK-3-UPDOWN: Interface HundredGigE1/0/39, changed state to down

*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/40, changed state to down

*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/41, changed state to down

*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/42, changed state to down

*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/43, changed state to down

*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/44, changed state to down

*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/45, changed state to down

*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/46, changed state to down

*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/47, changed state to down

*Jun 9 20:39:16.713: %LINK-3-UPDOWN: Interface HundredGigE1/0/48, changed state to down

*Jun 9 20:39:44.717: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 1/0

*Jun 9 20:40:15.052: %SPA_OIR-6-ONLINECARD: SPA (C9600-LC-24C) online in subslot 1/0

*interfaces come up again*

Symptom

Action

Module still fails P.O.S.T or module not listed in show module or show inventory command output.

- Visually confirm and document the L.E.D. status of the module; blinking or solid amber or red lights could further indicate hardware issue.

- Physically reseat the module.

- Install a spare module or module from another slot in the chassis to isolate failure to the module or chassis slot.

Reseat module (Similar syslogs are printed when the chassis recognizes the module and the module initializes).

*May 24 19:32:53.084: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F0

*May 24 19:32:53.084: %IOSXE_OIR-6-INSCARD: Card (cc) inserted in slot 1

*May 24 19:32:53.084: %IOSXE_OIR-6-ONLINECARD: Card (cc) online in slot 1

*May 24 19:32:53.151: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 1/0

Verify the module Power State is accepted.

Cat9600#show power module

Power Budget Mode           : Dual Sup

Power                                  Out of  In

Mod  Model No              State     Budget  Instantaneous  Peak  Reset   Reset

---  --------------------  --------  ------  -------------  ----  ------  -----

1    C9600-LC-48YL         accepted  230     0              0     230     10

2    C9600-LC-24C          accepted  200     0              0     200     10

3    C9600-SUP-1           accepted  775     0              0     775     202

4    C9600-SUP-1           accepted  775     0              0     775     202

FM1  C9606-FAN             accepted  450     --             --    450     --

---  --------------------  --------  ------  -------------  ----  ------  -----

Total allocated power:  2430

Total required power:   2430

Verify if there are any failed power supplies.

Cat9600#show power detail

Power                                                       Fan States

Supply  Model No              Type  Capacity  Status        1     2

------  --------------------  ----  --------  ------------  -----------

PS1     C9600-PWR-2KWAC       ac    1050 W    active        good  good

PS2     C9600-PWR-2KWAC       ac    2000 W    active        good  good

PS3     C9600-PWR-2KWAC       ac    n.a.      fail          bad   bad

PS4     C9600-PWR-2KWAC       ac    n.a.      fail          bad   bad

PS Current Configuration Mode : none

PS Current Operating State    : none

Power supplies currently active    : 2

Power supplies currently available : 4

Power Summary          Maximum

(in Watts)    Used    Available

-------------  ------  ---------

System Power   2905    2990

-------------  ------  ---------

Total          2905    2990              <-- total power budget consumed vs maximum available

Power Budget Mode           : Dual Sup

Power                                  Out of  In

Mod  Model No              State     Budget  Instantaneous  Peak  Reset   Reset

---  --------------------  --------  ------  -------------  ----  ------  -----

1    C9600-LC-48TX         accepted  315     0              0     315     10

2    C9600-LC-48S          accepted  160     0              0     160     10

3    C9600-SUP-1           accepted  775     0              0     775     202

4    C9600-SUP-1           ---       775     --             --    775     0

5    C9600-LC-48YL         accepted  230     0              0     230     10

6    C9600-LC-24C          accepted  200     0              0     200     10

FM1  C9606-FAN             accepted  450     --             --    450     --

---  --------------------  --------  ------  -------------  ----  ------  -----

Total allocated power:  2905

Total required power:   2905

Software and Firmware Mismatch

- Supervisors on different software or firmware versions fail to synchronize in High Availability Active/Standby roles.

- Line cards on a newer firmware version than compared to a Supervisor firmware version might fail to initialize.

- Instructions included in 'Upgrading the switch software' section in Release Notes of every major software train such as 16.12.X, 17.3.X, and 17.6.X

- Manually upgrade the firmware/bootldr of the Supervisor.

Note: Please review the Release Notes for Catalyst 9600 Series Switches Cisco IOS XE Amsterdan 17.3.X for more detailed information about how to manually upgrade the firmware/boot loader of the Supervisor.

Cat9600#show firmware version all

Current              Bundled

Slot  Device Name / Model           Firmware Version      Firmware Version    Mismatch

----- ----------------------------- --------------------- ------------------- ---------

3    Supervisor Rommon (Active)    17.3.1r[FC2]          N/A                 N/A        <-- Firmware versions are the same

4    Supervisor Rommon (Standby)   17.3.1r[FC2]          N/A                 N/A        <-- Firmware versions are the same

-    Eth Lan Spi                   0x800005d3            N/A                 N/A

PS5  Fantray                       18101008              N/A                 N/A

3    Supervisor CPLD

IO FPGA                     19041620              19041620            No

Flash FPGA                  190308b9              190308b9            No

4    Supervisor CPLD

IO FPGA                     19041620              19041620            No

Flash FPGA                  190308b9              190308b9            No

1    Line Card / C9600-LC-24C

IO FPGA                     19070619              19070619            No

PMFPGA                      122515                N/A                 N/A

Phy Firmware                1.56.1772             1.56.2270           Yes

2    Line Card / C9600-LC-48YL

IO FPGA                     19070619              19070619            No

PMFPGA                      122515                N/A                 N/A

Phy Firmware                1.56.1772             1.56.2270           Yes

5    Line Card / C9600-LC-48YL

IO FPGA                     19070619              19070619            No

PMFPGA                      122515                N/A                 N/A

Phy Firmware                1.56.1772             1.56.2270           Yes

6    Line Card / C9600-LC-24C

IO FPGA                     19070619              19070619            No

PMFPGA                      122515                N/A                 N/A

Phy Firmware                1.56.1772             1.56.2270           Yes

Slot      Device Name / Model                     Firmware Version

--------- --------------------------------------- -----------------------------

PS1       Power Supply (PS PRI, PS_SEC, PS_I2C)   (N/A, N/A, N/A)

PS2       Power Supply (PS PRI, PS_SEC, PS_I2C)   (N/A, N/A, N/A)

PS3       Power Supply (PS PRI, PS_SEC, PS_I2C)   (N/A, N/A, N/A)

PS4       Power Supply (PS PRI, PS_SEC, PS_I2C)   (61.08.03, 61.02.0, N/A)

Additional Troubleshooting

Note: hardware authentication status runs only at reload of switch. It does not run after physical reseat of line card at same or different slot, and this does not necessarily indicate an issue.

If authentication status NOT pass for a line card that is present, then execute these suggested steps:

1. Collect the trace archive, review IOSRP, CMCC, platform_mgr and IOMD files for relevant information about this issue type.

2. Physical OIR of affected component OR reload of switch can be tried to see if that recovers.

3. If issue persists, open a TAC case.

Cat9600#show platform hardware authentication status

Fan Tray Authentication:   pass

Line Card:1 Authentication:   pass

Line Card:2 Authentication:   pass

SUP0 Authentication:   pass

SUP1 Authentication:   pass

Line Card:5 Authentication:  Not Available

Line Card:6 Authentication:  Not Available

Cat9600#show hw-module subslot 1/0 oir internal

IOMD for supervisor slot does not appear

WARNING: This command is not intended for production use

and should only be used under the supervision of

Cisco Systems technical support personnel.

sm(spa_oir_tsm subslot 1/0 TSM), running yes, state ready

Admin Status: admin enabled,  Operational Status: ok(1)

Last reset Reason: power up

TSM Context:

configured_spa_type 0xD9A                                <-- IDPROM is accessible

soft remove fail code 0x0(none)

last_fail_code 0x0(none)

timed_fail_count 0, failed_spa_type 0x0

flags 0x10)

Subslot:

spa type 0xD9A, active spa type 0xD9A

subslot flags 0x0, plugin flags 0x0

TSM Parameters:

wait_psm_ready_timeout 540000 ms, init_timeout 300000 ms

remove_timeout 120000 ms, recovery_delay 5000 ms

fail_time_period 1200000 ms, max_fail_count 5

does not support pre-configuration

Cat9600# show platform software iomd redundancy

Configured Redundancy Mode = sso

Operating Redundancy Mode = sso

Local RF state = ACTIVE

Peer RF state = STANDBY HOT

slot  PSM STATE   SPA INTF   HA_STATE HA_ACTIVE

1     ready   started     ready    00:05:54

2     ready      idle                                   <-- potential issue

3     ready   started     ready    00:05:37 ***active RP

4     ready   started     ready    00:05:37

Cat9600#show idprom module 1 eeprom detail

Slot 1 EEPROM data:

EEPROM version           : 4

Compatible Type          : 0xFF

Controller Type          : 3482

Hardware Revision        : 1.0

PCB Part Number          : 73-19545-02

Board Revision           : A0

Deviation Number         : 0

Fab Version              : 02

PCB Serial Number        : CAT2313L2VP

RMA Test History         : 00

RMA Number               : 0-0-0-0

RMA History              : 00

Top Assy. Part Number    : 068-102161-01

Top Assy. Revision       : A0

CLEI Code                : COUIBGRCAA

Product Identifier (PID) : C9600-LC-24C

Version Identifier (VID) : V01

Base MAC Address         : DC 8C 37 A0 C8 80

MAC Address block size   : 128

Environment Monitor Data : 06 00 00 00 0C 80 C8 00

A6

Environment Monitor Data : 00 06 00 FA

Manufacturing Test Data  : 00 00 00 00 00 00 00 00

Field Diagnostics Data   : 00 00 00 00 00 00 00 00

Platform features        : 00 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00

Environment Monitor Data :

Description              : InltFrnt

Shutdown threshold       : 060

Critical threshold       : 055

Major threshold          : 050

Minor threshold          : 045

Environment Monitor Data :

Description              : InltRear

Shutdown threshold       : 060

Critical threshold       : 055

Major threshold          : 050

Minor threshold          : 045

Environment Monitor Data :

Description              : OtltFrnt

Shutdown threshold       : 090

Critical threshold       : 085

Major threshold          : 080

Minor threshold          : 075

Environment Monitor Data :

Description              : OtltRear

Shutdown threshold       : 090

Critical threshold       : 085

Major threshold          : 080

Minor threshold          : 075

Use the lc-portmap command to obtain information on the linecard interfaces mapping to the ASIC/Core in the Supervisor.

Cat9600#show platform hardware iomd 1/0 lc-portmap detail

IOMD SUMMARY

==============

spa_type: 0xd9a

my_slot: 1

active-slot: 0

is_active: 1

bulk_sync_done: 1

bulk_sync_in_progress: 0

bulk_sync_received: 0

is_plugin_start_done: 1

Num_ports: 48 num_UDAP 3s: 3

PSM SUMMARY

===============

PSM state: ready

ha_state:

is_spa_ok: 0

is_power_ok: 0

psm_flags: 0x0

bay_flags: 0x0

plugin_flags: 0x0

Port  fp  asic  core  mac  cntx  asic  speed  xcvr-  xcvr-  run-  phy-  i2c-  xcvr-  if-   SerTune-  False-  Rcvr-  Serdes-

port         pres   type   speed num   ctrl  sm     state Status    LinkUp  idle   set

================================================================================================================================

0     1   0     0     0    0     0     40G    no     0      unk   0     0     0     0      NOTS      no      0      0 1

1     2   0     0     2    1     8     40G    no     0      unk   0     0     0     0      NOTS      no      0      2 3

2     3   0     0     16   0     16    40G    no     0      unk   1     0     0     0      NOTS      no      0      16 17

3     4   0     0     18   1     24    40G    no     0      unk   1     0     0     0      NOTS      no      0      18 19

4     5   0     1     14   1     8     40G    no     0      unk   2     0     0     0      NOTS      no      0      14 15

5     6   0     1     12   0     0     40G    no     0      unk   2     0     0     0      NOTS      no      0      12 13

6     7   0     1     30   1     24    40G    no     0      unk   3     0     0     0      NOTS      no      0      30 31

7     8   0     1     28   0     16    40G    no     0      unk   3     0     0     0      NOTS      no      0      28 29

8     9   1     0     0    0     0     40G    no     0      unk   4     0     0     0      NOTS      no      0      0 1

9     10  1     0     2    1     8     40G    no     0      unk   4     0     0     0      NOTS      no      0      2 3

10    11  1     0     16   0     16    40G    no     0      unk   5     0     0     0      NOTS      no      0      16 17

11    12  1     0     18   1     24    40G    no     0      unk   5     0     0     0      NOTS      no      0      18 19

12    13  1     1     14   1     8     40G    no     0      unk   6     0     0     0      NOTS      no      0      14 15

13    14  1     1     12   0     0     40G    no     0      unk   6     0     0     0      NOTS      no      0      12 13

14    15  1     1     30   1     24    40G    no     0      unk   7     0     0     0      NOTS      no      0      30 31

15    16  1     1     28   0     16    40G    no     0      unk   7     0     0     0      NOTS      no      0      28 29

16    17  2     0     0    0     0     40G    no     0      unk   8     1     0     0      NOTS      no      0      0 1

17    18  2     0     2    1     8     40G    no     0      unk   8     1     0     0      NOTS      no      0      2 3

18    19  2     0     16   0     16    40G    no     0      unk   9     1     0     0      NOTS      no      0      16 17

19    20  2     0     18   1     24    40G    no     0      unk   9     1     0     0      NOTS      no      0      18 19

20    21  2     1     14   1     8     40G    no     0      unk   10    1     0     0      NOTS      no      0      14 15

21    22  2     1     12   0     0     40G    no     0      unk   10    1     0     0      NOTS      no      0      12 13

22    23  2     1     30   1     24    40G    no     0      unk   11    1     0     0      NOTS      no      0      30 31

23    24  2     1     28   0     16    40G    no     0      unk   11    1     0     0      NOTS      no      0      28 29

24    25  0     0     0    0     0     100G   no     0      unk   0     0     0     0      NOTS      no      0      0 1 2 3

25    26  0     0     0    0     0     100G   no     0      unk   0     0     0     0      NOTS      no      0      0 1 2 3

26    27  0     0     16   0     16    100G   no     0      unk   1     0     0     0      NOTS      no      0      16 17 18 19

27    28  0     0     16   0     16    100G   no     0      unk   1     0     0     0      NOTS      no      0      16 17 18 19

28    29  0     1     12   0     0     100G   no     0      unk   2     0     0     0      NOTS      no      0      12 13 14 15

29    30  0     1     12   0     0     100G   no     0      unk   2     0     0     0      NOTS      no      0      12 13 14 15

30    31  0     1     28   0     16    100G   no     0      unk   3     0     0     0      NOTS      no      0      28 29 30 31

31    32  0     1     28   0     16    100G   no     0      unk   3     0     0     0      NOTS      no      0      28 29 30 31

32    33  1     0     0    0     0     100G   no     0      unk   4     0     0     0      NOTS      no      0      0 1 2 3

33    34  1     0     0    0     0     100G   no     0      unk   4     0     0     0      NOTS      no      0      0 1 2 3

34    35  1     0     16   0     16    100G   no     0      unk   5     0     0     0      NOTS      no      0      16 17 18 19

35    36  1     0     16   0     16    100G   no     0      unk   5     0     0     0      NOTS      no      0      16 17 18 19

36    37  1     1     12   0     0     100G   no     0      unk   6     0     0     0      NOTS      no      0      12 13 14 15

37    38  1     1     12   0     0     100G   no     0      unk   6     0     0     0      NOTS      no      0      12 13 14 15

38    39  1     1     28   0     16    100G   no     0      unk   7     0     0     0      NOTS      no      0      28 29 30 31

39    40  1     1     28   0     16    100G   no     0      unk   7     0     0     0      NOTS      no      0      28 29 30 31

40    41  2     0     0    0     0     100G   no     0      unk   8     1     0     0      NOTS      no      0      0 1 2 3

41    42  2     0     0    0     0     100G   no     0      unk   8     1     0     0      NOTS      no      0      0 1 2 3

42    43  2     0     16   0     16    100G   no     0      unk   9     1     0     0      NOTS      no      0      16 17 18 19

43    44  2     0     16   0     16    100G   no     0      unk   9     1     0     0      NOTS      no      0      16 17 18 19

44    45  2     1     12   0     0     100G   no     0      unk   10    1     0     0      NOTS      no      0      12 13 14 15

45    46  2     1     12   0     0     100G   no     0      unk   10    1     0     0      NOTS      no      0      12 13 14 15

46    47  2     1     28   0     16    100G   no     0      unk   11    1     0     0      NOTS      no      0      28 29 30 31

47    48  2     1     28   0     16    100G   no     0      unk   11    1     0     0      NOTS      no      0      28 29 30 31

Cat9600#show platform software fed [active | standby] ifm mappings

Interface                 IF_ID    Inst Asic Core Port SubPort Mac  Cntx LPN  GPN  Type Active

TwentyFiveGigE1/0/1       0x9        0   0   0    0      0      0    0    1    101  NIF  Y

TwentyFiveGigE1/0/2       0xa        0   0   0    4      0      1    1    2    102  NIF  Y

TwentyFiveGigE1/0/3       0xb        0   0   0    8      0      2    2    3    103  NIF  Y

TwentyFiveGigE1/0/4       0xc        0   0   0    12     0      3    3    4    104  NIF  Y

TwentyFiveGigE1/0/5       0xd        0   0   0    16     0      16   0    5    105  NIF  Y

TwentyFiveGigE1/0/6       0xe        0   0   0    20     0      17   1    6    106  NIF  Y

TwentyFiveGigE1/0/7       0xf        0   0   0    24     0      18   2    7    107  NIF  Y

TwentyFiveGigE1/0/8       0x10       0   0   0    28     0      19   3    8    108  NIF  Y

TwentyFiveGigE1/0/9       0x11       1   0   1    12     0      15   3    9    109  NIF  Y

TwentyFiveGigE1/0/10      0x12       1   0   1    8      0      14   2    10   110  NIF  Y

TwentyFiveGigE1/0/11      0x13       1   0   1    4      0      13   1    11   111  NIF  Y

TwentyFiveGigE1/0/12      0x14       1   0   1    0      0      12   0    12   112  NIF  Y

TwentyFiveGigE1/0/13      0x15       1   0   1    28     0      31   3    13   113  NIF  Y

TwentyFiveGigE1/0/14      0x16       1   0   1    24     0      30   2    14   114  NIF  Y

TwentyFiveGigE1/0/15      0x17       1   0   1    20     0      29   1    15   115  NIF  Y

TwentyFiveGigE1/0/16      0x18       1   0   1    16     0      28   0    16   116  NIF  Y

TwentyFiveGigE1/0/17      0x19       2   1   0    0      0      0    0    17   117  NIF  Y

TwentyFiveGigE1/0/18      0x1a       2   1   0    4      0      1    1    18   118  NIF  Y

TwentyFiveGigE1/0/19      0x1b       2   1   0    8      0      2    2    19   119  NIF  Y

TwentyFiveGigE1/0/20      0x1c       2   1   0    12     0      3    3    20   120  NIF  Y

TwentyFiveGigE1/0/21      0x1d       2   1   0    16     0      16   0    21   121  NIF  Y

TwentyFiveGigE1/0/22      0x1e       2   1   0    20     0      17   1    22   122  NIF  Y

TwentyFiveGigE1/0/23      0x1f       2   1   0    24     0      18   2    23   123  NIF  Y

TwentyFiveGigE1/0/24      0x20       2   1   0    28     0      19   3    24   124  NIF  Y

TwentyFiveGigE1/0/25      0x21       3   1   1    12     0      15   3    25   125  NIF  Y

TwentyFiveGigE1/0/26      0x22       3   1   1    8      0      14   2    26   126  NIF  Y

TwentyFiveGigE1/0/27      0x23       3   1   1    4      0      13   1    27   127  NIF  Y

TwentyFiveGigE1/0/28      0x24       3   1   1    0      0      12   0    28   128  NIF  Y

TwentyFiveGigE1/0/29      0x25       3   1   1    28     0      31   3    29   129  NIF  Y

TwentyFiveGigE1/0/30      0x26       3   1   1    24     0      30   2    30   130  NIF  Y

TwentyFiveGigE1/0/31      0x27       3   1   1    20     0      29   1    31   131  NIF  Y

TwentyFiveGigE1/0/32      0x28       3   1   1    16     0      28   0    32   132  NIF  Y

TwentyFiveGigE1/0/33      0x29       4   2   0    0      0      0    0    33   133  NIF  Y

TwentyFiveGigE1/0/34      0x2a       4   2   0    4      0      1    1    34   134  NIF  Y

TwentyFiveGigE1/0/35      0x2b       4   2   0    8      0      2    2    35   135  NIF  Y

TwentyFiveGigE1/0/36      0x2c       4   2   0    12     0      3    3    36   136  NIF  Y

TwentyFiveGigE1/0/37      0x2d       4   2   0    16     0      16   0    37   137  NIF  Y

TwentyFiveGigE1/0/38      0x2e       4   2   0    20     0      17   1    38   138  NIF  Y

TwentyFiveGigE1/0/39      0x2f       4   2   0    24     0      18   2    39   139  NIF  Y

TwentyFiveGigE1/0/40      0x30       4   2   0    28     0      19   3    40   140  NIF  Y

TwentyFiveGigE1/0/41      0x31       5   2   1    12     0      15   3    41   141  NIF  Y

TwentyFiveGigE1/0/42      0x32       5   2   1    8      0      14   2    42   142  NIF  Y

TwentyFiveGigE1/0/43      0x33       5   2   1    4      0      13   1    43   143  NIF  Y

TwentyFiveGigE1/0/44      0x34       5   2   1    0      0      12   0    44   144  NIF  Y

TwentyFiveGigE1/0/45      0x35       5   2   1    28     0      31   3    45   145  NIF  Y

TwentyFiveGigE1/0/46      0x36       5   2   1    24     0      30   2    46   146  NIF  Y

TwentyFiveGigE1/0/47      0x37       5   2   1    20     0      29   1    47   147  NIF  Y

TwentyFiveGigE1/0/48      0x38       5   2   1    16     0      28   0    48   148  NIF  Y

TwentyFiveGigE2/0/2       0x3a       0   0   0    9      0      9    9    2    202  NIF  Y

TwentyFiveGigE2/0/3       0x3b       0   0   0    10     0      10   10   3    203  NIF  Y

TwentyFiveGigE2/0/4       0x3c       0   0   0    11     0      11   11   4    204  NIF  Y

TwentyFiveGigE2/0/5       0x3d       0   0   0    18     0      24   8    5    205  NIF  Y

TwentyFiveGigE2/0/6       0x3e       0   0   0    25     0      25   9    6    206  NIF  Y

TwentyFiveGigE2/0/7       0x3f       0   0   0    26     0      26   10   7    207  NIF  Y

TwentyFiveGigE2/0/8       0x40       0   0   0    27     0      27   11   8    208  NIF  Y

TwentyFiveGigE2/0/9       0x41       1   0   1    7      0      7    7    9    209  NIF  Y

TwentyFiveGigE2/0/10      0x42       1   0   1    6      0      6    6    10   210  NIF  Y

TwentyFiveGigE2/0/11      0x43       1   0   1    5      0      5    5    11   211  NIF  Y

TwentyFiveGigE2/0/12      0x44       1   0   1    1      0      4    4    12   212  NIF  Y

TwentyFiveGigE2/0/13      0x45       1   0   1    23     0      23   7    13   213  NIF  Y

TwentyFiveGigE2/0/14      0x46       1   0   1    22     0      22   6    14   214  NIF  Y

TwentyFiveGigE2/0/15      0x47       1   0   1    21     0      21   5    15   215  NIF  Y

TwentyFiveGigE2/0/16      0x48       1   0   1    17     0      20   4    16   216  NIF  Y

TwentyFiveGigE2/0/17      0x49       2   1   0    2      0      8    8    17   217  NIF  Y

TwentyFiveGigE2/0/18      0x4a       2   1   0    9      0      9    9    18   218  NIF  Y

TwentyFiveGigE2/0/19      0x4b       2   1   0    10     0      10   10   19   219  NIF  Y

TwentyFiveGigE2/0/20      0x4c       2   1   0    11     0      11   11   20   220  NIF  Y

TwentyFiveGigE2/0/21      0x4d       2   1   0    18     0      24   8    21   221  NIF  Y

TwentyFiveGigE2/0/22      0x4e       2   1   0    25     0      25   9    22   222  NIF  Y

TwentyFiveGigE2/0/23      0x4f       2   1   0    26     0      26   10   23   223  NIF  Y

TwentyFiveGigE2/0/24      0x50       2   1   0    27     0      27   11   24   224  NIF  Y

TwentyFiveGigE2/0/25      0x51       3   1   1    7      0      7    7    25   225  NIF  Y

TwentyFiveGigE2/0/26      0x52       3   1   1    6      0      6    6    26   226  NIF  Y

TwentyFiveGigE2/0/27      0x53       3   1   1    5      0      5    5    27   227  NIF  Y

TwentyFiveGigE2/0/28      0x54       3   1   1    1      0      4    4    28   228  NIF  Y

TwentyFiveGigE2/0/29      0x55       3   1   1    23     0      23   7    29   229  NIF  Y

TwentyFiveGigE2/0/30      0x56       3   1   1    22     0      22   6    30   230  NIF  Y

TwentyFiveGigE2/0/31      0x57       3   1   1    21     0      21   5    31   231  NIF  Y

TwentyFiveGigE2/0/32      0x58       3   1   1    17     0      20   4    32   232  NIF  Y

TwentyFiveGigE2/0/33      0x59       4   2   0    2      0      8    8    33   233  NIF  Y

TwentyFiveGigE2/0/34      0x5a       4   2   0    9      0      9    9    34   234  NIF  Y

TwentyFiveGigE2/0/35      0x5b       4   2   0    10     0      10   10   35   235  NIF  Y

TwentyFiveGigE2/0/36      0x5c       4   2   0    11     0      11   11   36   236  NIF  Y

TwentyFiveGigE2/0/37      0x5d       4   2   0    18     0      24   8    37   237  NIF  Y

TwentyFiveGigE2/0/38      0x5e       4   2   0    25     0      25   9    38   238  NIF  Y

TwentyFiveGigE2/0/39      0x5f       4   2   0    26     0      26   10   39   239  NIF  Y

TwentyFiveGigE2/0/40      0x60       4   2   0    27     0      27   11   40   240  NIF  Y

TwentyFiveGigE2/0/41      0x61       5   2   1    7      0      7    7    41   241  NIF  Y

TwentyFiveGigE2/0/42      0x62       5   2   1    6      0      6    6    42   242  NIF  Y

TwentyFiveGigE2/0/43      0x63       5   2   1    5      0      5    5    43   243  NIF  Y

TwentyFiveGigE2/0/44      0x64       5   2   1    1      0      4    4    44   244  NIF  Y

TwentyFiveGigE2/0/45      0x65       5   2   1    23     0      23   7    45   245  NIF  Y

TwentyFiveGigE2/0/46      0x66       5   2   1    22     0      22   6    46   246  NIF  Y

TwentyFiveGigE2/0/47      0x67       5   2   1    21     0      21   5    47   247  NIF  Y

TwentyFiveGigE2/0/48      0x68       5   2   1    17     0      20   4    48   248  NIF  Y

Commands to Collect for TAC

The most common hardware issues and validation commands are covered in this guide, with appropriate remediation steps. However, in the event that this guide did not resolve your issue please collect the command list shown and attach them to your TAC service request.

show tech-support

show post

show platform hardware authentication status

show platform hardware chassis fantray detail

show platform software iomd redundancy

request platform software trace archive

service internal (for the standby command 'service internal' needs to be enabled first)

request platform software trace slot rp standby archive target stby-bootflash