Upgrading Field-Programmable Devices

In general terms, field-programmable devices (FPDs) are hardware devices implemented on router cards that support separate upgrades. The term “FPD” has been introduced to collectively and generically describe any type of programmable hardware device on SPAs. FPDs were introduced on the Cisco ASR 1000 Series Aggregation Services Routers to support the SPAs.

This chapter describes the information that you need to know to verify image versions and to perform SPA FPD upgrades.

This chapter includes the following sections:

Release History

The following table provides the release history for all FPD-related features on the Cisco ASR 1000 Series Routers.

Table 1 FPD Release History

Release

Modification

Cisco IOS XE Release 2.1

SIPs and SPAs were released on the Cisco ASR 1000 Series Routers for the first time. FPD images were introduced to support these SPAs.

FPD Upgrade

This section provides information if you want to upgrade FPDs for SPAs. These instructions are not always feasible for operating network environments. If these methods of upgrade are not suitable for your situation, see other sections of this document for other methods of upgrading FPDs.

This section contains the following topics:

Overview of FPD Images and Packages

With the Cisco ASR 1000 Series Routers, the Cisco IOS image comes bundled with the FPD image. An FPD incompatibility on a SPA disables all interfaces on that SPA until the incompatibility is addressed.

Upgrading Your Cisco IOS Release and FPD Image

To upgrade your Cisco IOS release and your FPD image, do the following steps:

SUMMARY STEPS

    1.    Download the SPA package for the Cisco IOS release that you are upgrading to. The package contains both the Cisco IOS image and the FPD image.

    2.    Boot the new version of Cisco IOS. When the new version of Cisco IOS boots, search for the bundled FPD image. The FPD images will be updated automatically as part of the Cisco IOS boot process.

    3.    When the router has booted, verify the upgrade was successful by entering the show hw-module all fpd command.


DETAILED STEPS
    Step 1   Download the SPA package for the Cisco IOS release that you are upgrading to. The package contains both the Cisco IOS image and the FPD image.
    Step 2   Boot the new version of Cisco IOS. When the new version of Cisco IOS boots, search for the bundled FPD image. The FPD images will be updated automatically as part of the Cisco IOS boot process.
    Step 3   When the router has booted, verify the upgrade was successful by entering the show hw-module all fpd command.
    What to Do Next

    The time for updating the FPGA can be read in the log messages. The following is a sample for the log message:

    Jan 21 01:36:36.434: %FPD_MGMT-3-MAJOR_VER_MISMATCH: Major image version mismatch detected with I/O FPGA (FPD ID=1) for SPA-3XOC3-ATM-V2 card in subslot 1/1. Image will need to be upgraded from version 1.3 to at least a minimum version of 2.2. Current HW version = 1.80Jan 21 01:36:36.435: %FPD_MGMT-5-UPGRADE_ATTEMPT: Attempting to automatically upgrade the FPD image(s) for SPA-3XOC3-ATM-V2 card in subslot 1/1. Use 'show upgrade fpd progress' command to view the upgrade progress ...Jan 21 01:36:36.671: %FPD_MGMT-6-BUNDLE_DOWNLOAD: Downloading FPD image bundle for SPA-3XOC3-ATM-V2 card in subslot 1/1 ...Jan 21 01:36:36.738: %FPD_MGMT-6-UPGRADE_TIME: Estimated total FPD image upgrade time for SPA-3XOC3-ATM-V2 card in subslot 1/1 = 00:07:00.Jan 21 01:36:36.999: %FPD_MGMT-6-UPGRADE_START: I/O FPGA (FPD ID=1) image upgrade in progress for SPA-3XOC3-ATM-V2 card in subslot 1/1. Updating to version 2.2. PLEASE DO NOT INTERRUPT DURING THE UPGRADE PROCESS (estimated upgrade completion time = 00:07:00)FPD upgrade in progress on hardware, reload/configuration change on those hardware is not recommended as it might cause HW programming failure and result in RMA of the hardware.!-> UPGRADE PROCESS (estimated upgrade completion time = 00:07:00)

    Upgrading Only Your Cisco IOS Release and Retaining Your Current FPD Image

    You may choose to upgrade your Cisco IOS release with or without retaining your current FPD image. You may choose to upgrade your Cisco IOS image and retain your current FPD image, although this is not recommended.

    SUMMARY STEPS

      1.    Use the no upgrade fpd auto command to disable the bundled FPD image.

      2.    Save the configuration before loading the new image.

      3.    Load the new image.


    DETAILED STEPS
      Step 1   Use the no upgrade fpd auto command to disable the bundled FPD image.
      Note    The automatic upgrade feature is disabled once you enter the no upgrade fpd auto command.
      Step 2   Save the configuration before loading the new image.
      Step 3   Load the new image.
      Note    If your current FPD image is not compatible with the new image, the SPAs do not come online.

      Upgrading FPD Images in a Production System

      Adding a SPA to a production system presents the possibility that the SPA may contain versions of FPD images that are incompatible with the Cisco IOS release currently running the router. In addition, the FPD upgrade operation can be a very CPU-intensive operation and therefore the upgrade operation may take more time when it is performed on a production system. The performance impact will vary depending on various factors, including network traffic load, the type of processing engine used, type of SPA, and the type of service configured.

      For these reasons, we recommend that one of the following alternatives be used to perform the FPD upgrade on a production system if possible:

      Using a Nonproduction System to Upgrade the Cisco ASR 1000 Series Aggregation Services Routers SPA FPD Image

      Before beginning the upgrade, ensure:

      • The spare system is running the same version of the Cisco IOS software release that the target production system is running.
      • The automatic upgrade feature is enabled on the spare system. (The automatic upgrade feature is enabled by default. It can also be enabled using the upgrade fpd auto command).

      Use the following procedure to perform an upgrade on a spare system:

      SUMMARY STEPS

        1.    Insert the SPA into the spare system.

        2.    Verify the upgrade was successful by entering the show hw-module all fpd command.

        3.    Remove the SPA from the spare system after the upgrade.

        4.    Insert the SPA into the target production system.


      DETAILED STEPS
        Step 1   Insert the SPA into the spare system.

        If an upgrade is required, the system will perform the necessary FPD image updates so that when this SPA is inserted to the target production system it will not trigger an FPD upgrade operation there.

        Step 2   Verify the upgrade was successful by entering the show hw-module all fpd command.
        Step 3   Remove the SPA from the spare system after the upgrade.
        Step 4   Insert the SPA into the target production system.

        Verifying System Compatibility First

        If a spare system is not available to perform an upgrade, you can check for system compatibility by disabling the automatic upgrade feature before inserting the SPA. (The automatic upgrade feature is enabled by default. It can be disabled using the no upgrade fpd auto command.)

        • If the FPD images on the SPA are compatible with the system, you will only need to re-enable the automatic upgrade feature (the automatic upgrade feature can be re-enabled using the upgrade fpd auto command).
        • If the FPD images on the SPA are not compatible with the system, the SPA is disabled but will not impact system performance by attempting to perform an automatic upgrade.

        Use the following procedure to check the FPD images on the SPA for system compatibility:

        SUMMARY STEPS

          1.    Disable the automatic upgrade feature using the no upgrade fpd auto global configuration command.

          2.    Insert the SPA into the system.

          3.    Re-enable the automatic upgrade feature using the upgrade fpd auto global configuration command.


        DETAILED STEPS
          Step 1   Disable the automatic upgrade feature using the no upgrade fpd auto global configuration command.
          Step 2   Insert the SPA into the system.

          If the FPD images are compatible, the SPA will operate successfully after bootup.

          If the FPD images are not compatible, the SPA is disabled. At this point we recommend that you wait for a scheduled maintenance when the system is offline to manually perform the FPD upgrade as described in the Manually Upgrading SPA FPD Images.

          Step 3   Re-enable the automatic upgrade feature using the upgrade fpd auto global configuration command.

          Optional FPD Procedures

          This section provides information for optional FPD-related functions. None of the topics discussed in this section are necessary for completing FPD upgrades, but may be useful in some FPD-related scenarios. It covers the following topics:

          Manually Upgrading SPA FPD Images

          To manually upgrade the current FPD version on a SPA, use the following command:

          Router# upgrade hw-module subslot 
[slot-number/subslot-number] fpd bundle [reload]

          In this example, slot-number is the slot where the SIP is installed, subslot-number is the subslot number where the SPA is located, fpd indicates the type of upgrade required, bundle selects the bundled FPD package, and reload specifies that the SPA is automatically reloaded after the upgrade. Note that subslot slot-number/subslot-number is used to specify a SPA FPD upgrade. The SPA will automatically be reloaded to complete the FPD upgrade.


          Caution

          An image upgrade can require a long period of time to complete depending on the SPA.

          Upgrading Multiple FPD Images

          A single piece of hardware can contain multiple FPD images. The Cisco ASR 1000 Series Routers can upgrade up to three FPD images simultaneously. However, only one FPD upgrade per router slot can occur at a time, so all FPD images on all SPAs in a single slot will have to wait for another FPD upgrade to finish.

          Users should note that some FPD images require the SPA to reload to complete. The FPD upgrade process will perform this step automatically, so users do not have to intervene. However, the other FPDs in the hardware of the specified slot will have to wait for this reload to complete before their upgrade process begins.


          Note
          With a manual upgrade, you must include the reload option to cause the SPA to reload automatically. See the Manually Upgrading SPA FPD Images.

          During an automatic upgrade, the Cisco ASR 1000 Series Routers will upgrade as many FPDs as possible at a time. No user intervention is possible or necessary. The upgrade process will not stop until all FPD images have been updated.

          During manual upgrades, it is important to note that users can only specify upgrades for a single piece of hardware each time the upgrade hw-module subslot [slot-number/subslot-number] command is entered. The maximum of three simultaneous upgrades applies to manual upgrades as well. If you individually specify multiple manual FPD upgrades, only three FPDs can be upgraded simultaneously and that can only occur when the hardware is in different router slots. The FPD upgrade process will stop when all FPDs for the specified hardware have been upgraded.

          Displaying Current and Minimum Required FPD Image Versions

          To display the current version of FPD images on the SPAs installed on your router, use the show hw-module slot-number/subslot-number | all] fpd command, where slot-number is the slot number where the SIP is installed, and subslot-number is the number of the SIP subslot where a target SPA is located. Entering the all keyword shows information for hardware in all router slots.

          The following examples show the output when using this show command.

          The output display in this example shows that FPD versions on the SPAs in the system meet the minimum requirements:

          Router# show hw-module all fpd
==== ====================== ====== =============================================
                             H/W   Field Programmable   Current   Min. Required
Slot Card Type               Ver.  Device: "ID-Name"    Version      Version
==== ====================== ====== ================== =========== ==============
 0/0 SPA-5X1GE-V2            1.1   1-GE I/O FPGA          1.10        1.10     
---- ---------------------- ------ ------------------ ----------- --------------
 0/1 SPA-8X1FE-TX-V2         1.1   1-FE I/O FPGA          1.1         1.1      
---- ---------------------- ------ ------------------ ----------- --------------
 0/2 SPA-2XCT3/DS0           1.1   1-ROMMON               2.12        2.12     
                                   2-I/O FPGA             2.7         2.7      
                                   3-T3 SUBRATE FPGA      1.4         1.4      
---- ---------------------- ------ ------------------ ----------- --------------
 1/0 SPA-2XOC3-POS           1.0   1-I/O FPGA             3.4         3.4      
---- ---------------------- ------ ------------------ ----------- --------------
 1/1 SPA-8XCHT1/E1           1.1   1-ROMMON               2.12        2.12     
                                   2-I/O FPGA             2.7         2.7      
---- ---------------------- ------ ------------------ ----------- --------------
 1/2 SPA-2XT3/E3             1.0   1-ROMMON               2.12        2.12     
                                   2-I/O FPGA             1.0         1.0      
                                   3-E3 FPGA              1.4         1.4      
                                   4-T3 FPGA              1.4         1.4      
==== ====================== ====== =============================================

          This example shows the output when using the slot-number/subslot-number argument to identify a particular SPA:

          Router# show hw-module subslot 1/0 fpd
==== ====================== ====== =============================================
                             H/W   Field Programmable   Current   Min. Required
Slot Card Type               Ver.  Device: "ID-Name"    Version      Version
==== ====================== ====== ================== =========== ==============
 1/0 SPA-2XOC3-POS           1.0   1-I/O FPGA             3.4         3.4      
==== ====================== ====== =============================================

          The output display in this example shows that the SPA in subslot 0/1 is disabled because one of the programmable devices does not meet the minimum version requirements. 

          Router# show hw-module all fpd
==== ====================== ====== =============================================
                             H/W   Field Programmable   Current   Min. Required
Slot Card Type               Ver.  Device: "ID-Name"    Version      Version
==== ====================== ====== ================== =========== ==============
 0/0 SPA-4XOC3-POS           1.0   1-I/O FPGA             3.4         3.4     
---- ---------------------- ------ ------------------ ----------- --------------
 0/1 SPA-2XT3/E3 <DISABLED>  1.0   1-ROMMON               2.12        2.12     
                                   2-I/O FPGA             0.6         1.0     *
                                   3-E3 FPGA              1.0         1.4     *
                                   4-T3 FPGA              1.0         1.4     *
---- ---------------------- ------ ------------------ ----------- --------------
 0/2 SPA-1X10GE-L-V2         1.0   1-10GE I/O FPGA        1.9         1.9      
---- ---------------------- ------ ------------------ ----------- --------------
 0/3 SPA-1X10GE-L-V2         1.0   1-10GE I/O FPGA        1.9         1.9      
==== ====================== ====== =============================================
 NOTES:
       - FPD images that are required to be upgraded are indicated with a '*'
         character in the "Minimal Required Version" field. 
       - The following FPD image package file is required for the upgrade: 
         asr1000-fpd-bundle.pkg
==== ====================== ====== =============================================

          Displaying Information About the Default FPD Image Package

          You can use the show upgrade fpd package default command to find out which SPAs are supported with your current Cisco IOS release and which FPD image package you need for an upgrade.

          Router# 
show upgrade fpd package default
****************************************************************************
This Cisco IOS software image requires the following default FPD Image 
Package for the automatic upgrade of FPD images (the package is available 
from Cisco.com and is accessible from the Cisco Software Center page where 
this IOS software image can be downloaded):
****************************************************************************
Version: 12.2(XN)
Package Filename: asr1000-fpd-bundle.pkg
        List of card type supported in this package:
                                                              Minimal
                 No. Card Type                                HW Ver.
                ---- ---------------------------------------- -------
                  1) 4xT SERIAL SPA                             0.0  
                  2) 1xCHSTM1 SPA                               0.0  
                  3) 2xCT3 SPA                                  0.100
                  4) 2xCT3 SPA                                  0.200
                  5) 4xCT3 SPA                                  0.100
<additional output removed for readability>

          Verifying the FPD Image Upgrade Progress

          You can use the show upgrade fpd progress command to view a “snapshot” of the upgrade progress while an FPD image upgrade is taking place. The following example shows the type of information this command displays:

          Router# show upgrade fpd progress
 
FPD Image Upgrade Progress Table:
 
 ==== =================== ====================================================
                                               Approx.
                          Field Programmable    Time     Elapsed
 Slot Card Type           Device : "ID-Name"   Needed      Time    State
 ==== =================== ================== ========== ========== ===========
  0/1 SPA-8XCHT1/E1       2-I/O FPGA          00:01:00   00:00:23  Updating...
 ---- ------------------- ------------------ ----------- --------- -----------
  0/2 SPA-2XT3/E3         4-T3 FPGA           --:--:--   --:--:--  Waiting...
 ==== =================== ====================================================

          FPD Image Upgrade Examples

          This section provides examples of automatic and manual FPD image upgrades. It includes the following examples:

          Automatic FPD Image Upgrade Example

          The following example shows the output displayed when a SPA in slot 0 subslot 1 requires an FPD image upgrade and the upgrade fpd auto command is enabled . The required FPD image is automatically upgraded.

          *Nov 28 13:14:22.782: %FPD_MGMT-3-INCOMP_IMG_VER: Incompatible I/O FPGA (FPD ID=2) image version detected for SPA-2XT3/E3 card in subslot 0/1. Detected version = 0.6, minimum required version = 1.0. Current HW version = 1.0.
*Nov 28 13:14:22.783: %FPD_MGMT-5-UPGRADE_ATTEMPT: Attempting to automatically upgrade the FPD image(s) for SPA-2XT3/E3 card in subslot 0/1. Use 'show upgrade fpd progress' command to view the upgrade progress ...
*Nov 28 13:14:22.841: %FPD_MGMT-6-BUNDLE_DOWNLOAD: Downloading FPD image bundle for SPA-2XT3/E3 card in subslot 0/1 ...
*Nov 28 13:14:22.981: %FPD_MGMT-6-UPGRADE_TIME: Estimated total FPD image upgrade time for SPA-2XT3/E3 card in subslot 0/1 = 00:01:00.
*Nov 28 13:14:22.983: %FPD_MGMT-6-UPGRADE_START: I/O FPGA (FPD ID=2) image upgrade in progress for SPA-2XT3/E3 card in subslot 0/1. Updating to version 1.1. PLEASE DO NOT INTERRUPT DURING THE UPGRADE PROCESS (estimated upgrade completion time = 00:01:00) ...
*Nov 28 13:14:43.267: %FPD_MGMT-6-UPGRADE_PASSED: I/O FPGA (FPD ID=2) image in the SPA-2XT3/E3 card in subslot 0/1 has been successfully updated from version 0.6 to version 1.0. Upgrading time = 00:00:20.285
*Nov 28 13:14:43.267: %FPD_MGMT-6-OVERALL_UPGRADE: All the attempts to upgrade the required FPD images have been completed for SPA-2XT3/E3 card in subslot 0/1. Number of successful/failure upgrade(s): 1/0.
*Nov 28 13:14:43.268: %FPD_MGMT-5-CARD_POWER_CYCLE: SPA-2XT3/E3 card in subslot 0/1 is being power cycled for the FPD image upgrade to take effect.

          Manual FPD Image Upgrade Example

          In the following example, FPD for the 2-port T3/E3 SPA in subslot 0/3 is upgraded manually from the bundled FPD image package file. The command excludes the reload option so the SPA does not automatically reload.

          Router#upgrade hw-module subslot 0/3 fpd bundled 
% The following FPD will be upgraded for SPA-2XT3/E3 (H/W ver = 1.1) in subslot 0/3:
         ================== =========== =========== ============ 
         Field Programmable   Current     Upgrade   Estimated 
         Device: "ID-Name"    Version     Version   Upgrade Time 
         ================== =========== =========== ============ 
         2-I/O FPGA             1.0         1.1       00:01:00
         ================== =========== =========== ============ 
% NOTES:
  - Use 'show upgrade fpd progress' command to view the progress of the FPD upgrade.
  - You must reload the card after the upgrade. The FPD upgrade is not     finalized until the card is reloaded. 
  - The output of 'show hw-module <target> fpd' will not reflect the upgraded FPD     version until the card is reloaded to finalize the upgrade.
% Are you sure that you want to perform this operation? [no]: yes
% Initiating the upgrade operation on the target card ...
  (Use "show upgrade fpd progress" command to see upgrade progress)
Router#
*Mar  4 16:13:17.230: %FPD_MGMT-6-UPGRADE_TIME: Estimated total FPD image upgrade time for SPA-2XT3/E3 card in subslot 0/3 = 00:01:00.
*Mar  4 16:13:17.232: %FPD_MGMT-6-UPGRADE_START: I/O FPGA (FPD ID=2) image upgrade in progress for SPA-2XT3/E3 card in subslot 0/3. Updating to version 1.1. PLEASE DO NOT INTERRUPT DURING THE UPGRADE PROCESS (estimated upgrade completion time = 00:01:00) ...
*Mar  4 16:13:42.909: %FPD_MGMT-6-UPGRADE_PASSED: I/O FPGA (FPD ID=2) image in the SPA-2XT3/E3 card in subslot 0/3 has been successfully updated from version 1.0 to version 1.1. Upgrading time = 00:00:25.676
*Mar  4 16:13:42.909: %FPD_MGMT-6-OVERALL_UPGRADE: All the attempts to upgrade the required FPD images have been completed for SPA-2XT3/E3 card in subslot 0/3. Number of successful/failure upgrade(s): 1/0.

          In the following example, FPD for the 2-port T3/E3 SPA in subslot 0/3 is upgraded manually from the bundled FPD image package file. The command includes the reload option so the SPA automatically reloads so the new FPD image takes effect immediately.

          Router#upgrade hw-module subslot 0/3 fpd bundled reload 
% The following FPD will be upgraded for SPA-2XT3/E3 (H/W ver = 1.1) in subslot 0/3:
         ================== =========== =========== ============ 
         Field Programmable   Current     Upgrade   Estimated 
         Device: "ID-Name"    Version     Version   Upgrade Time 
         ================== =========== =========== ============ 
         2-I/O FPGA             1.0         1.1       00:01:00
         ================== =========== =========== ============ 
% NOTES:
  - Use 'show upgrade fpd progress' command to view the progress of the FPD
    upgrade.
  - The target card will be automatically reloaded after the upgrade 
    operation. This reload will interrupt normal operation of the card. If 
    necessary, ensure that appropriate actions have been taken to redirect 
    card traffic before starting the FPD upgrade.
% Are you sure that you want to perform this operation? [no]: yes
% Initiating the upgrade operation on the target card ...
  (Use "show upgrade fpd progress" command to see upgrade progress)
Router#
*Mar  4 16:15:22.862: %FPD_MGMT-6-UPGRADE_TIME: Estimated total FPD image upgrade time for SPA-2XT3/E3 card in subslot 0/3 = 00:01:00.
*Mar  4 16:15:22.864: %FPD_MGMT-6-UPGRADE_START: I/O FPGA (FPD ID=2) image upgrade in progress for SPA-2XT3/E3 card in subslot 0/3. Updating to version 1.1. PLEASE DO NOT INTERRUPT DURING THE UPGRADE PROCESS (estimated upgrade completion time = 00:01:00) ...
*Mar  4 16:15:47.795: %FPD_MGMT-6-UPGRADE_PASSED: I/O FPGA (FPD ID=2) image in the SPA-2XT3/E3 card in subslot 0/3 has been successfully updated from version 1.0 to version 1.1. Upgrading time = 00:00:24.931
*Mar  4 16:15:47.795: %FPD_MGMT-6-OVERALL_UPGRADE: All the attempts to upgrade the required FPD images have been completed for SPA-2XT3/E3 card in subslot 0/3. Number of successful/failure upgrade(s): 1/0.
*Mar  4 16:15:47.796: %FPD_MGMT-5-CARD_POWER_CYCLE: SPA-2XT3/E3 card in subslot 0/3 is being power cycled for the FPD image upgrade to take effect.

          Troubleshooting Problems with FPD Image Upgrades

          This section contains information to help troubleshoot problems that can occur during the upgrade process.

          It contains the following topics:

          Power Failure or Removal of a SPA During an FPD Image Upgrade

          These instructions should only be used if a previous upgrade attempt has failed due to an external factor such as a power failure or a SPA removal.

          If the FPD upgrade operation is interrupted by a power failure or the removal of the SPA, it could corrupt the FPD image. This corruption of the FPD image file makes the SPA unusable by the router and the system will display the following messages when it tries to power up the SPA:


          Note
          To find more information about FPD-related messages, check the system error messages guide for your Cisco IOS XE software release.
          Nov.28 11:30:36:%SPA_OIR-3-RECOVERY_RELOAD:subslot 0/1:Attempting
recovery by reloading SPA
Nov.28 11:30:51:%SPA_OIR-3-HW_INIT_TIMEOUT:subslot 0/1
Nov.28 11:30:56:%SPA_OIR-3-RECOVERY_RELOAD:subslot 0/1:Attempting
recovery by reloading SPA
Nov.28 11:31:11:%SPA_OIR-3-HW_INIT_TIMEOUT:subslot 0/1
Nov.28 11:31:16:%SPA_OIR-3-RECOVERY_RELOAD:subslot 0/1:Attempting
recovery by reloading SPA
Nov.28 11:31:31:%SPA_OIR-3-HW_INIT_TIMEOUT:subslot 0/1
Nov.28 11:31:31:%SPA_OIR-3-SPA_POWERED_OFF:subslot 0/1:SPA 4xOC3 POS
SPA powered off after 5 failures within 600 seconds

          The show hw-module all fpd command can be used to verify that the SPA is using a corrupted FPD image. In this example, the SPA in slot 0/1 is corrupted.

          Router# show hw-module all fpd
==== ====================== ====== =============================================
                             H/W   Field Programmable   Current   Min. Required
Slot Card Type               Ver.  Device: "ID-Name"    Version      Version
==== ====================== ====== ================== =========== ==============
 0/0 SPA-4XOC3-POS           1.0   1-I/O FPGA             3.4         3.4     
---- ---------------------- ------ ------------------ ----------- --------------
 0/1 SPA-4XOC3-PO<DISABLED>  1.0   ????????????           ?.?         ?.?
---- ---------------------- ------ ------------------ ----------- --------------
 0/2 SPA-1X10GE-L-V2         1.0   1-10GE I/O FPGA        1.9         1.9      
---- ---------------------- ------ ------------------ ----------- --------------
 0/3 SPA-1X10GE-L-V2         1.0   1-10GE I/O FPGA        1.9         1.9      
==== ====================== ====== =================================================

          Performing an FPD Recovery Upgrade

          The recovery upgrade procedure can only be performed on a SPA that has been powered off by the system after it has failed all of the retries attempted to initialize the SPA.

          The following example displays the output of an attempt to perform a recovery upgrade before all the initialization retries have been attempted for the SPA in subslot 0/1.


          Note
          Because a recovery upgrade is done at a more conservative speed, it may take more than the estimated upgrade time.

          Note
          Other factors can cause the system to ask “Do you want to perform the recovery upgrade operation?” Only answer y to this question if you have attempted an FPD upgrade that has failed due to a power failure or a SPA removal.If you are prompted for this question without having previously had a failed upgrade attempt for one of the aforementioned reasons, contact Cisco Technical Support.
          Nov.28 11:29:55:%SPA_OIR-3-RECOVERY_RELOAD:subslot 0/1:Attempting
recovery by reloading SPA
Nov.28 11:30:10:%SPA_OIR-3-HW_INIT_TIMEOUT:subslot 0/1
Nov.28 11:30:15:%SPA_OIR-3-RECOVERY_RELOAD:subslot 0/1:Attempting
recovery by reloading SPA
Nov.28 11:30:31:%SPA_OIR-3-HW_INIT_TIMEOUT:subslot 0/1
Router# upgrade hw-module subslot 0/1 fpg bundled
% Cannot get FPD version information for version checking. If a previous
upgrade attempt has failed for the target card, then a recovery upgrade
would be required to fix the failure.
% The following FPD(s) will be upgraded for SPA-4XOC3-POS (H/W ver =
1.0) in subslot 0/1:
    ================== =========== =========== ============ 
    Field Programmable   Current     Upgrade   Estimated 
    Device:"ID-Name"    Version     Version   Upgrade Time 
    ================== =========== =========== ============ 
    1-I/O FPGA             ?.?         3.4       00:00:50
    ================== =========== =========== ============ 
% Do you want to perform the recovery upgrade operation? [no]:y
% Cannot perform recovery upgrade operation because the target card is
not in a failed state. Please try again later.

          Once the following error message is displayed, you can perform the recovery upgrade:

          Nov.28 11:31:31:%SPA_OIR-3-SPA_POWERED_OFF:subslot 0/1:SPA 4xOC3 POS SPA powered off after 5 failures within 600 seconds

          Note
          You must wait to see this error message before you attempt the upgrade.

          Perform the manual FPD image upgrade method using the upgrade hw-module subslot command to recover from a corrupted image after the SPA has been powered off by the system. In this command, slot-number is the slot where the SIP is installed, subslot-number is the subslot of the SIP where the SPA is located.

          The following example displays the console output of a recovery upgrade operation:

          Router# upgrade hw-module subslot 0/1 fpd bundled
% Cannot get FPD version information from SPA-4XOC3-POS in subslot 0/1.
  If a previous upgrade attempt on the target card was interrupted, then the
  corruption of FPD image might have prevented the card from coming online. 
  If this is the case, then a recovery upgrade would be required to fix the 
  failure.
(Hit ENTER to proceed with recovery upgrade operation) [confirm]
% The following FPD will be upgraded for SPA-4XOC3-POS (H/W ver = 1.0) in subslot 0/1:
         ================== =========== =========== ============ 
         Field Programmable   Current     Upgrade   Estimated 
         Device: "ID-Name"    Version     Version   Upgrade Time 
         ================== =========== =========== ============ 
         1-I/O FPGA             ?.?         3.4       00:00:50
         ================== =========== =========== ============ 
% NOTES:
  - Use 'show upgrade fpd progress' command to view the progress of the FPD
    upgrade.
  - Since the target card is currently in disabled state, it will be 
    automatically reloaded after the upgrade operation for the changes to 
    take effect.
% Do you want to perform the recovery upgrade operation? [no]: y
% Starting recovery upgrade operation in the background ... 
  (Use "show upgrade fpd progress" command to see upgrade progress)
Router#
*Nov 28 17:46:10.191:%FPD_MGMT-6-UPGRADE_TIME: Estimated total FPD image upgrade time for SPA-4XOC3-POS card in subslot 0/1 = 00:00:50.
*Nov 28 17:46:10.233:%FPD_MGMT-6-UPGRADE_START: I/O FPGA (FPD ID=1) image upgrade in progress for SPA-4XOC3-POS card in subslot 0/1. Updating to version 3.4. PLEASE DO NOT INTERRUPT DURING THE UPGRADE PROCESS (estimated upgrade completion time = 00:00:50) ...
*Nov 28 17:46:55.948:%FPD_MGMT-6-UPGRADE_PASSED: I/O FPGA (FPD ID=1) image in the SPA-4XOC3-POS card in subslot 0/1 has been successfully updated from version ?.? to version 3.4. Upgrading time = 00:00:45.715
*Nov 28 17:46:55.948:%FPD_MGMT-6-OVERALL_UPGRADE: All the attempts to upgrade the required FPD images have been completed for SPA-4XOC3-POS card in subslot 0/1. Number of successful/failure upgrade(s): 1/0.
*Nov 28 17:46:55.949:%FPD_MGMT-5-CARD_POWER_CYCLE: SPA-4XOC3-POS card in subslot 0/1 is being power cycled for the FPD image upgrade to take effect.

          Verifying a Successful Upgrade

          After the upgrade process is complete, you can use the show hw-module all fpd command to verify that the FPD image has been successfully upgraded:

          Router# show hw-module all fpd
==== ====================== ====== =============================================
                             H/W   Field Programmable   Current   Min. Required
Slot Card Type               Ver.  Device: "ID-Name"    Version      Version
==== ====================== ====== ================== =========== ==============
 0/0 SPA-4XOC3-POS           1.0   1-I/O FPGA             3.4         3.4     
---- ---------------------- ------ ------------------ ----------- --------------
 0/1 SPA-4XOC3-POS           1.0   1-I/O FPGA             3.4         3.4
---- ---------------------- ------ ------------------ ----------- --------------
 0/2 SPA-1X10GE-L-V2         1.0   1-10GE I/O FPGA        1.9         1.9      
---- ---------------------- ------ ------------------ ----------- --------------
 0/3 SPA-1X10GE-L-V2         1.0   1-10GE I/O FPGA        1.9         1.9 
==== ====================== ====== =============================================