Field Notice: FN74306 - ASR9K Software Upgrade Issue: Sunstreaker and TAMFW FPD Upgrade Timeout or FPGA Corruption - Workaround Provided

Cisco has identified two critical issues that are related to Sunstreaker and TAMFW field programmable devices (FPDs) that may occur during a Cisco IOS XR Software upgrade on the Cisco ASR 9000 Series Aggregation Services Routers route processors (RPs) and line cards that are mentioned in the Products Affected section of this field notice. These issues affect multiple Cisco IOS XR Software releases and occur in both auto and manual FPD upgrade scenarios if FPDs are not upgraded in the correct sequence.

Cisco ASR 9000 Series Aggregation Services Routers use FPDs that require careful sequencing during Cisco IOS XR Software upgrades. The auto FPD upgrade feature was designed to simplify this sequencing process. However, this feature has been shown to cause timeouts during Sunstreaker and TAMFW FPD upgrades. This leads to field-programmable gate array (FPGA) corruption, which compromises the upgrade integrity. The two issues that have been observed are as follows:

• When the auto FPD upgrade feature is enabled, upgrade operations may time out, causing FPGA corruption and upgrade failures.
• Additionally, if the primary BIOS is upgraded before the Sunstreaker FPD version in manual upgrades using the upgrade hw-module location all fpd all CLI command, FPGA corruption can occur.

Upgrading Cisco IOS XR Software on an RP or line card with the auto FPD upgrade feature enabled will impact the RP or line card and lead to an unrecoverable state.

The following symptoms and error outputs can be seen on impacted Cisco ASR 9000 Series Aggregation Services Routers:

• Upgrade operations for Sunstreaker and TAMFW FPDs may time out. The following error messages may appear during the upgrade process:

  0/3/ADMIN0:Feb 25 01:52:30.697 CST: fpd_agent[3401]: %INFRA-FPD_Driver-2-UPGRADE_ERROR : Abort fpd Sunstreaker upgrade @location 0/3 due to timeout 
  0/3/ADMIN0:Feb 25 01:57:31.198 CST: fpd_agent[3401]: %INFRA-FPD_Driver-2-UPGRADE_ERROR : Abort fpd TAMFW-Sunstreaker upgrade @location 0/3 due to timeout

• FPGA corruption occurs, causing upgrade failures. The following output will indicate that the RP or line card is already affected:

  sysadmin-vm:0_RSP0# sh platform Location  Card Type               HW State      SW State      Config State
  ----------------------------------------------------------------------------
  0/2       A9K-8X100GE-L-SE        OPERATIONAL   OPERATIONAL   NSHUT         
  0/4       A9K-8HG-FLEX-SE         OPERATIONAL   SW_INACTIVE   NSHUT         <<<<<< Indicates that the line card is affected/bricked
  0/RSP0    A9K-RSP880-SE           OPERATIONAL   OPERATIONAL   NSHUT            
  0/FT0     ASR-9010-FAN-V2         OPERATIONAL   N/A           NSHUT         
  0/FT1     ASR-9010-FAN-V2         OPERATIONAL   N/A           NSHUT         
  0/PT0     A9K-AC-PEM-V1           OPERATIONAL   N/A           NSHUT         
  0/PT1     A9K-AC-PEM-V1           OPERATIONAL   N/A           NSHUT         
  
  sysadmin-vm:0_RSP0# show hw-module fpd
  
                                                                       FPD Versions
                                                                      ==============
  Location  Card type         HWver FPD device        ATR Status      Run     Programd
  ------------------------------------------------------------------------------------   
  0/4       A9K-8HG-FLEX-SE   0.0   Aldrin-FPGA           NOT READY   0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   CBC                   NOT READY   0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   IPU-DDR4              NOT READY   0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   Primary-BIOS          NOT READY   0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   SSDa                  N/A         0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   Sunstreaker           NOT READY   0.00      0.00     <<<<<< Indicates the FPGA is corrupted
  0/4       A9K-8HG-FLEX-SE   0.0   TAMFW-Sunstreaker     NOT READY   0.00      0.00     <<<<<< Indicates the FPGA is corrupted
  0/4       A9K-8HG-FLEX-SE   0.0   Trailbreaker-0        NOT READY   0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   Windcharger-0         NOT READY   0.00      0.00    

On Cisco IOS XR Software releases, Cisco recommends disabling the auto FPD upgrade feature before software upgrade and then upgrading the FPDs manually using the following guidelines:

Step 1: Using the following commands, disable the auto FPD upgrade and auto reload features before starting the Cisco IOS XR Software upgrade from Sysadmin and XR-Plane:

XR-Plane:
(config)#fpd auto-upgrade disable
(config)#fpd auto-reload disable
(config)# commit

Sysadmin:
RP/0/RSP0/CPU0:cst-pe1#admin  
sysadmin-vm:0_RSP# config  
sysadmin-vm:0_RSP(config)# fpd auto-upgrade disable  
sysadmin-vm:0_RSP(config)# commit

Step 2: Complete the Cisco IOS XR Software upgrade. For upgrade instructions, see the Software Installation Guide.

Step 3: Execute the show fpd package CLI command on the upgraded Cisco IOS XR Software to check the FPD package details. The following example shows sample output from the command:

RP/0/RSP1/CPU0:ios#show fpd package

=============================== ================================================

                                        Field Programmable Device Package

                                               Req     SW      Min Req   Min Req

Card Type            FPD Description           Reload  Ver     SW Ver  Board Ver

================================================================================

A99-4HG-FLEX-SE      
                    Aldrin-FPGA(A)             YES     1.05      1.05      0.0                       
                    CBC(A)                     NO     63.03     63.03      0.0                       
                    IPU-DDR4(A)                YES     1.05      1.05      0.0                       
                    Moonracer(A)               YES     0.14      0.14      0.0                       
                    Primary-BIOS(A)            YES    25.31     25.31      0.0                       
                    Skywarp-0(A)               YES     0.11      0.11      0.0                       
                    Skywarp-1(A)               YES     0.11      0.11      0.0                       
                    Sunstreaker(A)             YES     0.19      0.19      0.0                       
                    TAMFW-Sunstreaker(A)       YES     2.72      2.72      0.0                       
                    TimingIC-A                 YES     7.216     7.216     0.0                       
                    TimingIC-B

The show hw-module fpd command will show NEED UPGD for the FPDs that require upgrades, as shown in the following example:

Show hw-module fpd<Snip>
0/1 A99-4HG-FLEX-TR 1.0 IPU-DDR4 CURRENT 1.05 1.05 
0/1 A99-4HG-FLEX-TR 1.0 Moonracer NEED UPGD 0.13 0.13 
0/1 A99-4HG-FLEX-TR 1.0 Primary-BIOS NEED UPGD 25.28 25.28 
0/1 A99-4HG-FLEX-TR 1.0 Skywarp-0 CURRENT 0.11 0.11 
0/1 A99-4HG-FLEX-TR 1.0 Skywarp-1 CURRENT 0.11 0.11 
0/1 A99-4HG-FLEX-TR 1.0 SSDa-MICRON N/A 1.00 1.00 
0/1 A99-4HG-FLEX-TR 1.0 Sunstreaker NEED UPGD 0.15 0.15 
0/1 A99-4HG-FLEX-TR 1.0 TAMFW-Sunstreaker NEED UPGD 2.65 2.65

If Sunstreaker FPDs, TAMFW-Sunstreaker FPDs, or both are shown with NEED UPGD, as shown in the preceding example, use the following steps to complete the FPD upgrade process:

Step 4: Perform manual FPD upgrades in the following sequence:

1. Upgrade the Sunstreaker FPD first and the TAMFW FPD second:

   admin upgrade hw-module location <0/1> fpd Sunstreaker force                          <<<<< Wait until "FPD Sunstreaker @location 0/1 FPD upgraded and activated!" syslog appears.
   admin upgrade hw-module location <0/1> fpd TAMFW-Sunstreaker force            <<<<< Wait until "FPD TAMFW-Sunstreaker @location 0/1 FPD upgraded and activated!" syslog appears.

2. Upgrade the BIOS version:

   admin upgrade hw-module location <0/1> fpd Primary-BIOS force             <<<<< Wait until "FPD Primary-BIOS @location 0/1 FPD upgraded and activated!" syslog appears   

3. Update other FPDs that require an upgrade sequentially, one after another.

Step 5: Using the following command, reload the line card for the FPD version change to take effect:

admin hw-module location  <0/1> reload

This approach prevents FPGA corruption and ensures successful upgrade completion. To verify FPD and BIOS versions, run the show hw-module fpd CLI command on both Cisco IOS XR Software and the admin plane. When a successful upgrade has been completed, the output will appear as follows:

0/1        A99-4HG-FLEX-TR       1.0   TimingIC-A            CURRENT     7.216     7.216
0/1        A99-4HG-FLEX-TR       1.0   TimingIC-B            CURRENT     7.216     7.216
0/1        A99-4HG-FLEX-TR       1.0   Aldrin-FPGA           CURRENT     1.05      1.05
0/1        A99-4HG-FLEX-TR       1.0   CBC                   CURRENT    63.03     63.03
0/1        A99-4HG-FLEX-TR       1.0   IPU-DDR4              CURRENT     1.05      1.05
0/1        A99-4HG-FLEX-TR       1.0   Moonracer             CURRENT     0.14      0.14
0/1        A99-4HG-FLEX-TR       1.0   Primary-BIOS          CURRENT    25.30     25.30
0/1        A99-4HG-FLEX-TR       1.0   Skywarp-0             CURRENT     0.11      0.11
0/1        A99-4HG-FLEX-TR       1.0   Skywarp-1             CURRENT     0.11      0.11
0/1        A99-4HG-FLEX-TR       1.0   SSDa-MICRON           N/A         1.00      1.00
0/1        A99-4HG-FLEX-TR       1.0   Sunstreaker           CURRENT     0.19      0.19
0/1        A99-4HG-FLEX-TR       1.0   TAMFW-Sunstreaker     CURRENT     2.72      2.72

Note: It is recommended to upgrade FPDs manually in a sequential order within each affected card. However, sequential upgrade can be done for other impacted cards in parallel without waiting for each card to complete the FPD upgrade. For example, on line cards 0/0 and 0/1, the Sunstreaker FPD upgrades can be performed concurrently. For non-impacted cards, upgrade of the FPDs can be initiated using upgrade hw-module fpd all location 0/x.