Field Notice: FN - 70586 - ASR920 Router Upgrade to IOS XE Release 17.3.1 Fails When Router is in Consolidated Binary Mode - Workaround Provided

Available Languages

Updated:August 7, 2020
Document ID:FN70586

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision Publish Date Comments
1.0
31-Jul-20
Initial Release

Products Affected

Affected Product ID Comments
ASR-920-12SZ-A
ASR-920-12SZ-A=
ASR-920-12SZ-D
ASR-920-12SZ-D=

Defect Information

Defect ID Headline
CSCvt67719 HS1: Device stuck with rommon upgrade from 29r to 38r
CSCvu34503 ASR920/RSP2: Bundle 43r ROMMON Changes to 17.3.1

Problem Description

The routers listed in the Products Affected section will fail to boot during the upgrade to IOS XE Release 17.3.1 because of the image size increase.

The releases bundled with FPGA (0x00010039 or 0x10000007) will suspend the router during the ROMMON upgrade process.

The upgrade process will succeed if the upgrade is performed with the router in sub package mode. In order to perform a clean upgrade, follow the upgrade instructions that are provided in the Workaround/Solution section of this Field Notice.

The upgrade process will fail if the upgrade is performed with the router in consolidated binary mode. Recovery instructions are also provided in the Workaround/Solution section.

Background

Because of image size increase, the router fails to boot during upgrade and it will reboot continuously with this console message: mount package service fail.

In order to boot the IOS XE image 17.3.1 successfully, the ROMMON version 43r is needed. The latest FPGA version (0x00010040 or 0x10000008) is also needed in order to ensure that the router does not become suspended during the upgrade.

Problem Symptom

Sample error logs are provided below.

Mount packages service fail log error

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

File reception completed.

Boot image size = 511841050 (0x1e82131a) bytes

…

[103.397489] localhost /usr/sbin/boot_verify_package[3444]: /tmp/sw/isos/asr920igp-sipspa.BLD_V173_THROTTLE_LATEST_20200502_210845_V17_3_0_149.SSA.pkg: Digital Signature Verification Failed, error=No keys for the specific key type found

…

Apr 10 03:04:28.858: %BOOT-3-SYSD_STARTFAIL: R0/0: Failed to launch boot task mount_packages.service ( exit-code )
Apr 10 03:04:37.454: %BOOT-0-BOOT_COMPLETE_FAIL: R0/0: Critical boot tasks failed: droputil.service iptables.service mount_packages.service oom.service xinetd@-etc-xinetd.conf.service xinetd@-etc-xinetd_telnetd.conf.service
>>> Rebooting

Failed to set wall message, ignoring: The name org.freedesktop.login1 was not provided by any .service files
Failed to reboot system via logind: The name org.freedesktop.login1 was not provided by any .service files
in reset_local_rp.sh!

…

Router suspend during ROMMON upgrade error 

System Bootstrap, Version 15.6(29r)S, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2018 by cisco Systems, Inc.
Compiled Mon 16-Jul-18 23:28 by 
Starting Initialization of FMAN0
Loading ucode for FMAN0, size: 31424, ver: 106.04.14
Silicon Rev Major:Minor [1:1]
Initializing the pci.. 

RC-1 Vendor-Id:0x8241957

FPGA Vendor-Id:0x861137

IOFPGA version[19052139] … 
... Checking 1G Network PLL LOCK Status ... 
PASS: 1G HSS0 PLL LOCKED HSS0_STATUS_REG Address 0x14300040 Data 0xf080002 ... 
PASS: 1G HSS1 PLL LOCKED HSS1_STATUS_REG Address 0x14300044 Data 0xf010002 ... 
... Checking 10G Network PLL LOCK Status ... 
PASS: 10G HSS0 PLL LOCKED HSS0_STATUS_REG Address 0x14400040 Data 0x5010002 ... 
PASS: 10G HSS0 PLL LOCKED HSS1_STATUS_REG Address 0x14400044 Data 0x1080002 
... UEA platform with 1048576 Kbytes of main memory 

Rommon upgrade requested 
Flash upgrade reset 1 in progress 
......

Workaround/Solution

In order to perform the upgrade to IOS-XE Release 17.3.1, you must boot the software in sub package mode. In sub package mode, the router boots without any issues, and auto-ROMMON upgrade to version 43r happens on boot.

Follow these steps in order to boot the router in sub package mode:

  1. Copy the 17.3.1 release binary and expand it to configure packages.conf.
  2. Set the boot variable to packages.conf and reload the router.
  3. The auto-ROMMON upgrade to version 43r and the FPGA upgrade to latest version are completed in parallel and result in a successful upgrade to IOS XE 17.3.1

NOTE: Detailed instructions to boot the device in sub package mode are available here:
https://www.cisco.com/c/en/us/td/docs/routers/asr920/upgrade/guide/b-upgrade-3x-16x-asr920/b-upgrade-3x-16x-asr920_chapter_01.html#concept_7E5AFB3D86B64137870F69CE244BF04E

Router Recovery

If the upgrade failed because the router was not in sub package mode, the mount boot package error might cause the router to enter a contnuous system reboot state, or the ROMMON upgrade issue might cause the router to enter a suspend state. This section provides instructions to recover the router from each of these states.

Follow these steps in order to recover the router from a continuous system reboot state because of a mount boot package error:

  1. Ensure that you are connected to the router in console mode.
  2. After you log in, press CTRL+C in order to prevent autoboot and enter ROMMON mode.
  3. Run the command “unset BOOT” in order to remove the existing boot variable.
  4. Run the command “BOOT= bootflash:<Your earlier release>” [Example: asr920igp-universalk9_npe.16.09.05.SPA.bin]
  5. Run the command “sync” in order to sync the configuration.
  6. Run the command “reset” in order to load the earlier image.
  7. When the system is back to the console prompt, follow the upgrade instructions above.

Follow these steps in order to recover the router from suspend state because of the ROMMON upgrade issue:

  1. Power cycle the router.
  2. Run the command #show platform on the router in order to see if the ROMMON upgrade was successful. 
    #sh platform
Chassis type: ASR-920-12SZ-D
Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0/0 12xGE-10GE-FIXED ok 00:04:59
R0 ASR-920-12SZ-D ok, active 00:09:35
F0 ok, active 00:09:35
P0 ASR920-PSU0 ok never
P1 ASR920-PSU1 ps, fail never
P2 ASR920-FAN ok never

Slot CPLD Version Firmware Version
--------- ------------------- ---------------------------------------
R0 20050708 15.6(43r)S
F0 20050708 15.6(43r)S
  3. Follow the upgrade instructions above.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.

Was this Document Helpful?

FeedbackFeedback

Contact Cisco