Field Notice: FN74306 - Cisco ASR 9K Routers Software Upgrade Issue: Sunstreaker/Sigma/LongShot and TAMFW FPD Upgrade Timeout or FPGA Corruption - Workaround Provided

Cisco has identified critical issues that are related to Sunstreaker/Sigma/LongShot and TAMFW field programmable devices (FPDs). These issues may occur during an upgrade of Cisco IOS XR Software on the Cisco ASR 9000 Series Aggregation Services Routers route processors (RP/RSPs) and line cards that are mentioned in the Products Affected section of this field notice. These issues affect multiple Cisco IOS XR Software releases and occur in both auto and manual FPD upgrade scenarios if FPDs are not upgraded in the correct sequence.

Cisco ASR 9000 Series Aggregation Services Routers use FPDs that require careful sequencing during Cisco IOS XR Software upgrades. The auto FPD upgrade feature was designed to streamline the process. However, this feature has been shown to cause timeouts during Sunstreaker/Sigma/LongShot and TAMFW FPD upgrades. The timeouts may lead to field-programmable gate array (FPGA) corruption, which compromises the upgrade integrity. The two issues that have been observed are as follows:

• When the auto FPD upgrade feature is enabled, upgrade operations may time out, causing FPGA corruption and upgrade failures.
• When conducting a manual upgrade using the upgrade hw-module location all fpd all CLI command, FPGA corruption can occur if the primary BIOS is upgraded before the Sunstreaker/Sigma/LongShot FPD is upgraded.

The following table lists the affected line cards and their respective Akido FPD names:

The following sections contain examples relevant to the specified line cards.

Upgrading Cisco IOS XR Software on an RP or line card with the auto FPD upgrade feature enabled will lead to an unrecoverable state. If the affected line card is already in an unrecoverable state (bricked), then replacement is required.

The following symptoms and error outputs may be seen on affected Cisco ASR 9000 Series Aggregation Services Routers:

• Upgrade operations for Sunstreaker/Sigma/LongShot and TAMFW FPDs may time out. The following error messages may appear during the upgrade process:

  0/4/ADMIN0:Feb 25 01:52:30.697 CST: fpd_agent[3401]: %INFRA-FPD_Driver-2-UPGRADE_ERROR : Abort fpd Sunstreaker upgrade @location 0/3 due to timeout 
  0/4/ADMIN0:Feb 25 01:57:31.198 CST: fpd_agent[3401]: %INFRA-FPD_Driver-2-UPGRADE_ERROR : Abort fpd TAMFW-Sunstreaker upgrade @location 0/3 due to timeout

• FPGA corruption may occur, causing upgrade failures. The following output will indicate that the RP or line card is already affected:

  sysadmin-vm:0_RSP0# sh platform Location  Card Type               HW State      SW State      Config State
  ----------------------------------------------------------------------------
  0/2       A9K-8X100GE-L-SE        OPERATIONAL   OPERATIONAL   NSHUT         
  0/4       A9K-8HG-FLEX-SE         OPERATIONAL   SW_INACTIVE   NSHUT         <<<<<< Indicates that the line card is affected/bricked
  0/RSP0    A9K-RSP880-SE           OPERATIONAL   OPERATIONAL   NSHUT            
  0/FT0     ASR-9010-FAN-V2         OPERATIONAL   N/A           NSHUT         
  0/FT1     ASR-9010-FAN-V2         OPERATIONAL   N/A           NSHUT         
  0/PT0     A9K-AC-PEM-V1           OPERATIONAL   N/A           NSHUT         
  0/PT1     A9K-AC-PEM-V1           OPERATIONAL   N/A           NSHUT         
  
  sysadmin-vm:0_RSP0# show hw-module fpd
  
                                                                       FPD Versions
                                                                      ==============
  Location  Card type         HWver FPD device        ATR Status      Run     Programd
  ------------------------------------------------------------------------------------   
  0/4       A9K-8HG-FLEX-SE   0.0   Aldrin-FPGA           NOT READY   0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   CBC                   NOT READY   0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   IPU-DDR4              NOT READY   0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   Primary-BIOS          NOT READY   0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   SSDa                  N/A         0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   Sunstreaker           NOT READY   0.00      0.00     <<<<<< Indicates the FPGA is corrupted
  0/4       A9K-8HG-FLEX-SE   0.0   TAMFW-Sunstreaker     NOT READY   0.00      0.00     <<<<<< Indicates the FPGA is corrupted
  0/4       A9K-8HG-FLEX-SE   0.0   Trailbreaker-0        NOT READY   0.00      0.00    
  0/4       A9K-8HG-FLEX-SE   0.0   Windcharger-0         NOT READY   0.00      0.00    

To prevent this issue, Cisco recommends using one of the following procedures when upgrading Cisco IOS XR Software on an affected Cisco ASR 9000 Series Aggregation Services Router:

• Use Option 1 to manually upgrade the FPDs.
• Use Option 2 to run upgrade hw-module location all fpd all after installing the fix. 
• Use Option 3 if currently running a Cisco IOS XR release that has an SMU for CSCwo49713. 

The Cisco IOS XR releases in the following table have SMUs for CSCwo49713. To download the SMU, follow these steps:

1. Go to the Cisco Software Download page.
2. Choose the appropriate Cisco ASR9000 product. 
3. Click IOS XR Software Maintenance Upgrade (SMU).
4. Choose the appropriate release for which you want to install the SMU.
5. In the SMU list, look for SMU name containing CSCwo49713. For example, for Cisco products in the ASR 9000 series, find and install the SMU asr9k-sysadmin-7.11.2.CSCwo49713.tar.

Option 1: Software Upgrade and Manual FPD Upgrade

To upgrade Cisco IOS XR Software and manually upgrade the FPDs on a router, complete the following steps:

Step 1: Use the following commands to disable the auto FPD upgrade and auto reload features from Sysadmin and XR-Plane before starting the Cisco IOS XR Software upgrade:

XR-Plane:

(config)#fpd auto-upgrade disable
(config)#fpd auto-reload disable
(config)# commit

Sysadmin:

RP/0/RSP0/CPU0:ios#admin  
sysadmin-vm:0_RSP# config  
sysadmin-vm:0_RSP(config)# fpd auto-upgrade disable  
sysadmin-vm:0_RSP(config)# commit

Step 2: Complete the Cisco IOS XR Software upgrade. For upgrade instructions, see the Software Installation Guide.

Step 3: Use the show fpd package CLI command on the upgraded Cisco IOS XR Software to check the FPD package details. The following example shows sample output from the command:

RP/0/RSP1/CPU0:ios#show fpd package
=============================== ================================================
                                        Field Programmable Device Package
                                ================================================
                                               Req     SW      Min Req   Min Req
Card Type            FPD Description           Reload  Ver     SW Ver  Board Ver
=================== ========================== ====== ======= ======== ========= 
A99-4HG-FLEX-SE      
                    Aldrin-FPGA(A)             YES     1.05      1.05      0.0                       
                    CBC(A)                     NO     63.03     63.03      0.0                       
                    IPU-DDR4(A)                YES     1.05      1.05      0.0                       
                    Moonracer(A)               YES     0.14      0.14      0.0                       
                    Primary-BIOS(A)            YES    25.31     25.31      0.0                       
                    Skywarp-0(A)               YES     0.11      0.11      0.0                       
                    Skywarp-1(A)               YES     0.11      0.11      0.0                       
                    Sunstreaker(A)             YES     0.19      0.19      0.0                       
                    TAMFW-Sunstreaker(A)       YES     2.72      2.72      0.0                       
                    TimingIC-A                 YES     7.216     7.216     0.0                       
                    TimingIC-B

The show hw-module fpd command will show NEED UPGD for the FPDs that require upgrades, as shown in the following example:

RP/0/RSP1/CPU0:ios#show hw-module fpd | i NEED
0/1        A99-4HG-FLEX-TR       1.0   Moonracer             NEED UPGD    25.28     25.28
0/1        A99-4HG-FLEX-TR       1.0   Sunstreaker           NEED UPGD     0.19      0.19
0/1        A99-4HG-FLEX-TR       1.0   TAMFW-Sunstreaker     NEED UPGD     2.72      2.72

If Sunstreaker/Sigma/LongShot FPDs, TAMFW-Sunstreaker FPDs, or both display NEED UPGD, as shown in the preceding example, continue to Step 4 to complete the FPD upgrade process.

Step 4: Perform manual FPD upgrades in the following sequence.

Upgrade the TAMFW FPD first. Then upgrade the Sunstreaker/Sigma/LongShot FPD.

admin upgrade hw-module location <0/1 or all> fpd TAMFW-Sunstreaker force          <<<<< Wait until "FPD TAMFW-Sunstreaker @location 0/1 FPD upgraded and activated!" syslog appears.
admin upgrade hw-module location <0/1 or all> fpd Sunstreaker force                <<<<< Wait until "FPD Sunstreaker @location 0/1 FPD upgraded and activated!" syslog appears.

Note: If multiple line cards need these FPD upgrades, then using location all will allow parallel upgrades of the same FPD on all locations for which the upgrade is required, resulting in quicker task completion.

Upgrade the BIOS version, if needed:

admin upgrade hw-module location <0/1 or all> fpd Primary-BIOS force               <<<<< Wait until "FPD Primary-BIOS @location 0/1 FPD upgraded and activated!" syslog appears   

Note: Using location all will allow parallel upgrade of the same FPD on all locations for which the upgrade is required, resulting in quicker task completion.

Update other FPDs that require an upgrade sequentially, one after another.

Step 5: Use the following command to reload the line card, which will allow the FPD version change to take effect:

admin hw-module location  <0/1> reload

This approach prevents FPGA corruption and ensures successful upgrade completion. To verify FPD and BIOS versions, use the show hw-module fpd CLI command on both Cisco IOS XR Software and the admin plane. When a successful upgrade has been completed, the output will appear as follows:

0/1        A99-4HG-FLEX-TR       1.0   TimingIC-A            CURRENT     7.216     7.216
0/1        A99-4HG-FLEX-TR       1.0   TimingIC-B            CURRENT     7.216     7.216
0/1        A99-4HG-FLEX-TR       1.0   Aldrin-FPGA           CURRENT     1.05      1.05
0/1        A99-4HG-FLEX-TR       1.0   CBC                   CURRENT    63.03     63.03
0/1        A99-4HG-FLEX-TR       1.0   IPU-DDR4              CURRENT     1.05      1.05
0/1        A99-4HG-FLEX-TR       1.0   Moonracer             CURRENT     0.14      0.14
0/1        A99-4HG-FLEX-TR       1.0   Primary-BIOS          CURRENT    25.31     25.31
0/1        A99-4HG-FLEX-TR       1.0   Skywarp-0             CURRENT     0.11      0.11
0/1        A99-4HG-FLEX-TR       1.0   Skywarp-1             CURRENT     0.11      0.11
0/1        A99-4HG-FLEX-TR       1.0   SSDa-MICRON           N/A         1.00      1.00
0/1        A99-4HG-FLEX-TR       1.0   Sunstreaker           CURRENT     0.19      0.19
0/1        A99-4HG-FLEX-TR       1.0   TAMFW-Sunstreaker     CURRENT     2.72      2.72

Note: Cisco recommends upgrading FPDs manually in sequential order within each affected card. However, sequential upgrade can be done for other impacted cards in parallel without waiting for each card to complete the FPD upgrade. For example, on line cards 0/0 and 0/1, the Sunstreaker/Sigma/LongShot FPD upgrades can be performed concurrently. For non-impacted cards, upgrade of the FPDs can be initiated using upgrade hw-module fpd all location 0/x.

Option 2: "upgrade hw-module location all fpd all" after installing the fix

This option is suited for all situations in which the fix is not available in the currently running Cisco IOS XR Software release, but the fix is included in the target Cisco IOS XR Software release natively or through an SMU. First, disable the FPD auto upgrade feature, then upgrade the software. After the upgrade, use the command upgrade hw-module location all fpd all to complete the FPD upgrade. Note: This process will require extra chassis reload after FPD upgrade.

Step 1: Use the following commands to disable the FPD auto upgrade and auto reload features from Sysadmin and XR-Plane before starting the Cisco IOS XR Software upgrade:

XR-Plane:

(config)#fpd auto-upgrade disable
(config)#fpd auto-reload disable
(config)# commit

Sysadmin:

RP/0/RSP0/CPU0:ios#admin  
sysadmin-vm:0_RSP# config  
sysadmin-vm:0_RSP(config)# fpd auto-upgrade disable  
sysadmin-vm:0_RSP(config)# commit

Step 2: Complete the Cisco IOS XR Software upgrade. For upgrade instructions, see the Software Installation Guide.

Step 3: For all RP and line cards that are listed in the Products Affected section of this field notice, wait for RP/LC Authentication syslog messages, as shown in the following example:

fpd_agent[2584]: %DRIVER-FPD_AGENT-3-CARD_AUTH_STATUS : Card Authentication Status : SUCCESS

Step 4: Use the following commands to upgrade all FPD.

RP/0/RSP0/CPU0:ios#upgrade hw-module location all fpd all

Step 5: Use the following commands to reload the chassis or selected location.

Sysadmin:

RP/0/RSP0/CPU0:ios#admin  
sysadmin-vm:0_RSP# hw-module location <all|0/x> reload  

Step 6: Use the show hw-module fpd CLI command on both Cisco IOS XR Software and the admin plane. When a successful upgrade has been completed, the output will appear as follows:

0/1        A99-4HG-FLEX-TR       1.0   TimingIC-A            CURRENT     7.216     7.216
0/1        A99-4HG-FLEX-TR       1.0   TimingIC-B            CURRENT     7.216     7.216
0/1        A99-4HG-FLEX-TR       1.0   Aldrin-FPGA           CURRENT     1.05      1.05
0/1        A99-4HG-FLEX-TR       1.0   CBC                   CURRENT    63.03     63.03
0/1        A99-4HG-FLEX-TR       1.0   IPU-DDR4              CURRENT     1.05      1.05
0/1        A99-4HG-FLEX-TR       1.0   Moonracer             CURRENT     0.14      0.14
0/1        A99-4HG-FLEX-TR       1.0   Primary-BIOS          CURRENT    25.30     25.30
0/1        A99-4HG-FLEX-TR       1.0   Skywarp-0             CURRENT     0.11      0.11
0/1        A99-4HG-FLEX-TR       1.0   Skywarp-1             CURRENT     0.11      0.11
0/1        A99-4HG-FLEX-TR       1.0   SSDa-MICRON           N/A         1.00      1.00
0/1        A99-4HG-FLEX-TR       1.0   Sunstreaker           CURRENT     0.19      0.19
0/1        A99-4HG-FLEX-TR       1.0   TAMFW-Sunstreaker     CURRENT     2.72      2.72

Step 7: Use the following commands to re-enable the FPD auto upgrade and auto reload features from Sysadmin and XR-Plane:

XR-Plane:

(config)#fpd auto-upgrade
(config)#fpd auto-reload
(config)# commit

Sysadmin:

RP/0/RSP0/CPU0:ios#admin  
sysadmin-vm:0_RSP# config  
sysadmin-vm:0_RSP(config)# fpd auto-upgrade  
sysadmin-vm:0_RSP(config)# commit

Option 3: Software Upgrade and SMU Available in the Current Release

On Cisco IOS XR Software releases for which an SMU for CSCwo49713 is available, Cisco recommends installing the SMU in the current release and then proceeding with the software upgrade while keeping the auto FPD upgrade feature enabled.

Step 1: Download and install the appropriate SMU for CSCwo49713, based on the release of Cisco IOS XR that is currently installed. (SMU installation will imply full chassis reload).

Step 2: Use the following commands to enable the FPD auto upgrade and auto reload features from Sysadmin and XR-Plane before starting the Cisco IOS XR Software upgrade:

XR-Plane:

(config)#fpd auto-upgrade
(config)#fpd auto-reload
(config)# commit

Sysadmin:

RP/0/RSP0/CPU0:cst-pe1#admin  
sysadmin-vm:0_RSP# config  
sysadmin-vm:0_RSP(config)# fpd auto-upgrade  
sysadmin-vm:0_RSP(config)# commit

Step 2: Complete the Cisco IOS XR Software upgrade. For upgrade instructions, see the Software Installation Guide.

Step 3: Use the show hw-module fpd CLI command on both Cisco IOS XR Software and the admin plane. When a successful upgrade has been completed, the output will appear as follows:

0/1        A99-4HG-FLEX-TR       1.0   TimingIC-A            CURRENT     7.216     7.216
0/1        A99-4HG-FLEX-TR       1.0   TimingIC-B            CURRENT     7.216     7.216
0/1        A99-4HG-FLEX-TR       1.0   Aldrin-FPGA           CURRENT     1.05      1.05
0/1        A99-4HG-FLEX-TR       1.0   CBC                   CURRENT    63.03     63.03
0/1        A99-4HG-FLEX-TR       1.0   IPU-DDR4              CURRENT     1.05      1.05
0/1        A99-4HG-FLEX-TR       1.0   Moonracer             CURRENT     0.14      0.14
0/1        A99-4HG-FLEX-TR       1.0   Primary-BIOS          CURRENT    25.30     25.30
0/1        A99-4HG-FLEX-TR       1.0   Skywarp-0             CURRENT     0.11      0.11
0/1        A99-4HG-FLEX-TR       1.0   Skywarp-1             CURRENT     0.11      0.11
0/1        A99-4HG-FLEX-TR       1.0   SSDa-MICRON           N/A         1.00      1.00
0/1        A99-4HG-FLEX-TR       1.0   Sunstreaker           CURRENT     0.19      0.19
0/1        A99-4HG-FLEX-TR       1.0   TAMFW-Sunstreaker     CURRENT     2.72      2.72