Router Hardware Management Guide for Cisco 8000 Series Routers, Cisco IOS XR Release

PDF

Router Hardware Management Guide for Cisco 8000 Series Routers, Cisco IOS XR Release

Fault recovery handling

Updated: July 1, 2026

Want to summarize with AI?

Log in

Describes how the hw-module fault-recovery command sets the number of recovery attempts the router makes before permanently shutting down a faulty card.

The hw-module fault-recovery command is a configuration feature that

  • manages fault recovery attempts for line cards, fabric cards, and route processors, enables administrators to define a specific threshold for recovery attempts before permanent shutdown, and

  • replaces the previous time-based reset mechanism to ensure faulty hardware enters a shutdown state after the configured limit.

Table 1. Feature history table

Feature Name

Release Information

Feature Description

Fault recovery handling

 Release 26.1.1

Introduced in this release on: Centralized Systems (8400 [ASIC: K100])(select variants only*)

* This feature is now supported on Cisco 8404-SYS-D routers.

Fault recovery handling

Release 25.4.1

Introduced in this release on: Fixed Systems (8010 [ASIC: A100])(select variants only*)

*This feature is supported on:

  • 8011-32Y8L2H2FH

  • 8011-12G12X4Y-A/D

Fault recovery handling

 Release 25.1.1

Introduced in this release on: Fixed Systems (8700 [ASIC: K100], 8010 [ASIC: A100])(select variants only*)

*This feature is supported on:

  • 8011-4G24Y4H-I

  • 8712-MOD-M

Fault recovery handling

Release 24.4.1

Introduced in this release on: Fixed Systems (8200 [ASIC: P100], 8700 [ASIC: P100])(select variants only*); Modular Systems (8800 [LC ASIC: P100])(select variants only*)

*This feature is supported on:

  • 8212-48FH-M

  • 8711-32FH-M

  • 88-LC1-36EH

  • 88-LC1-12TH24FH-E

  • 88-LC1-52Y8H-EM

Fault recovery handling

Release 24.2.11

You can now configure the number of fault recovery attempts by a line card, fabric card or a route processor before it permanently shuts down, thus preventing a faulty card from entering into a cycle of automatic recovery.

This feature introduces this change:

CLI:

YANG DATA Model:

In the previous releases, if a line card, fabric card or a route processor experienced a fault, they used to trigger fault recovery and reboot themselves to be operational. Fault recovery mechanism was time based as the fault recovery count used to reset to zero if the card remained operational for more than hour. After the fault recovery count exceeded five, then the faulty card was shut down. As power related faults triggered were not frequent, and fault recovery count used to reset to zero, the card never entered the shut down mode. As a result the card always attempted for fault recovery.

Note

The hw-module fault-recovery configuration is not applicable for BMC instance

Configure fault recovery attempts

This task shows how to configure the fault recovery attempts on the fabric card FC0 and how to verify the configuration on the router.

Procedure

1.

Enter the global configuration mode.

Example:

Router#configure
2.

Configure the fault recovery attempts on the fabric card.

Example:

Router (config)#hw-module fault-recovery location 0/FC0 count 1
3.

Save or commit the changes.

Example:

Router(config)#commit
4.

Verify the configured fault recovery count. Use the show running-config formal | include hw-module command to display the number of times a card can initiate recovery attempts before shutting down.

Example:

Router#show running-config formal | include hw-module
Building configuration...
hw-module fault-recovery location 0/FC0 count 1
5.

Review the system logs generated when the number of fault recovery attempts on the card exceeds the configured count.

Example:

Router:Dec 4 15:44:22.950 PST: shelfmgr[121]: %PLATFORM-SHELFMGR-2-FAULT_ACTION_CARD_SHUTDOWN : Forced shutdown requested for card 0/FC0. Reason Fault retry attempts exceeded configured count(1)

Example:

Router:Dec 4 15:44:25.247 PST: shelfmgr[121]: %PLATFORM-SHELFMGR-4-CARD_SHUTDOWN : Shutting down 0/FC0: Fault retry attempts exceeded configured count(1)
6.

Confirm the reason for the card shutdown. Use the show reboot history command to get the reason of card shutting down. In this example, it shows that the card was shut down due to Fault retry attempts exceeded configured count(1) .

Example:

Router#show reboot history location 0/FC0 detail
Mon Dec  4 15:44:55.827 PST
--------------------------------------------------------------------------------
No   Attribute       Value
--------------------------------------------------------------------------------
1    Time (PST)      Dec 04 2023 15:44:22
     Cause Code      0x0800000d
     Cause String    REBOOT_CAUSE_FM
     Graceful Reload No
     Kdump Requested No
     Reason          Fault retry attempts exceeded configured count(1)
7.

Verify the current state of the card that was shut down due to the fault recovery handling feature. Use the show platform command to see the current state of the card.

Example:


Router#show platform 
Mon Oct 2 21:08:03.383 UTC

Node              Type                     State                    Config state
--------------------------------------------------------------------------------
0/RP0/CPU0        8800-RP(Active)          IOS XR RUN               NSHUT
0/RP0/BMC0        8800-RP                  OPERATIONAL               NSHUT
0/RP1/CPU0        8800-RP(Standby)         IOS XR RUN               NSHUT
0/RP1/BMC0        8800-RP                  OPERATIONAL                NSHUT
0/3/CPU0          8800-LC-48H              IOS XR RUN               NSHUT
0/FC0             8812-FC                  SHUT DOWN                NSHUT
0/FC3             8812-FC                  OPERATIONAL              NSHUT
0/FT0             SF-D-12-FAN              OPERATIONAL              NSHUT
0/FT1             SF-D-12-FAN              OPERATIONAL              NSHUT
0/FT2             SF-D-12-FAN              OPERATIONAL              NSHUT
0/FT3             SF-D-12-FAN              OPERATIONAL              NSHUT
0/PT0             FAM7000-ACHV-TRAY        OPERATIONAL              NSHUT
0/PT1             FAM7000-ACHV-TRAY        OPERATIONAL              NSHUT
0/PT2             FAM7000-ACHV-TRAY        OPERATIONAL              NSHUT
Router#