Troubleshoot IR1101 Software Upgrade Issue

Introduction

This document describes the steps to identify and mitigate the software upgrade issue on Cisco IR1101 router  via Cisco IOS® XE SD-WAN Manager.

Problem Description

This article complements the Field Notice FN74237 pubished for the IR1101 upgrade issues.

Cisco IR1101 routers can fail to upgrade successfully from Cisco IOS® 17.10/17.11 to later software releases, due to insufficient flash memory space. The issue is seen on the devices while performing the upgrade via Catalyst SD-WAN Manager in install mode.

Scenario

Routers running Cisco IOS XE 17.10.x/17.11.x versions cannot upgrade to higher versions using Catalyst SD-WAN manager if the available space is less than 1400MB.

In the IR1101, the software upgrade fails when updating from versions 17.10.x or 17.11.x to higher releases. This occurs, as the upgrade process requires that the free storage space must be double the size of the image in the boot flash. Therefore, if the available space in the boot flash is less than double the image size, the upgrade can fail. Refer to the defect below.

Cisco bug ID CSCwf84896

Error message from SD-WAN Manager logs

These messages can be seen in the upgrade job logs. 

[25-Sep-2024 13:11:03 AEST] Device: All installed versions: 17.10.01.0.1479;
[25-Sep-2024 13:11:06 AEST] Device: Received installation request for software version(5-tuple): 17.12.04.0.4826
[25-Sep-2024 13:11:19 AEST] Device: /flash requires 703602 KB of free space, but only 536416 KB is available
[25-Sep-2024 13:11:19 AEST] Software Install failed

Workaround

Upgrade the image from Cisco IOS XE version 17.10.x/ 17.11.x to 17.12.05 first as an intermediate step and then proceed to the desired release afterwards. For this upgrade to be successful, the Catalyst SD-WAN Manager must be running version 20.12 or higher.

Preparatory steps for the Upgrade

Backup Configuration

Ensure you back up the current running configuration of the IR1101 device to avoid data loss.

Verify Compatibility

Confirm that the IR1101 hardware and the current SD-WAN setup are compatible with the target software version.

If the upgrade is from 17.10.1a /17.11.1a to 17.12.05 then use SD-WAN Manager version 20.12 or higher release.

If the upgrade is from 17.12.05 to 17.15.1a then use version 20.15

Obtain Image

Download the intermediate image version 17.12.05 and the final target software version from software.cisco.com

Device Space check and clean up

1.Check the available space in device by executing command

Router#dir bootflash: | i bytes
                                                                                      
   2648363008 bytes total (1304428544 bytes free)

2. Make sure there is 1,225 to 1,244 MBof free space in the bootflash before we proceed with the upgrade process.

3.If the device has less than 1225 MB of space in flash, please perform a cleanup by executing the delete command via SSH to the device through the vManage GUI. Execute these commands one by one.

delete /force /recursive bootflash:/.EXPAND*

delete /force /recursive bootflash:/admintech_trace

delete /force /recursive bootflash:/admintech

delete /force /recursive bootflash:/collated_log*

delete /force /recursive bootflash:/license_evlog/*.log

delete /force /recursive bootflash:/sdavc/*

delete /force /recursive bootflash:/vmanage-admin/*.tar.gz

delete /force /recursive bootflash:/core/*.tar.gz

delete /force /recursive bootflash:/core/*.core.gz

delete /force /recursive bootflash:/core/*core.flat.gz

delete /force /recursive bootflash:/syslog/*

delete /force /recursive bootflash:/crashinfo_*

delete /force /recursive bootflash:/tracelogs/*

 

Alternatively,  this cleaning practice can be integrated as part of the SD-WAN configuration. It can be automated by leveraging an EEM script that triggers the process when an upgrade is about to be performed by SD-WAN Manager. Add the EEM applet to a CLI Add-On Profile in your SD-WAN Configuration Group:

event manager applet eem_clear_bootflash authorization bypass

  event syslog pattern "Started Pre-Upgrade Check"

  action 0005 syslog msg "Cleaning bootflash: started ..."

  action 0010 cli command "enable"

  action 0020 cli command "delete /force /recursive bootflash:/.EXPAND*"

  action 0030 cli command "delete /force /recursive bootflash:/admintech_trace"

  action 0040 cli command "delete /force /recursive bootflash:/admintech"

  action 0050 cli command "delete /force /recursive bootflash:/collated_log*"

  action 0060 cli command "delete /force /recursive bootflash:/license_evlog/*.log"

  action 0070 cli command "delete /force /recursive bootflash:/sdavc/*"

  action 0080 cli command "delete /force /recursive bootflash:/vmanage-admin/*.tar.gz"

  action 0090 cli command "delete /force /recursive bootflash:/core/*.tar.gz"

  action 0100 cli command "delete /force /recursive bootflash:/core/*.core.gz"

  action 0110 cli command "delete /force /recursive bootflash:/core/*core.flat.gz"

  action 0120 cli command "delete /force /recursive bootflash:/syslog/*"

  action 0130 cli command "delete /force /recursive bootflash:/crashinfo_*"

  action 0140 cli command "delete /force /recursive bootflash:/tracelogs/*"

  action 0900 syslog msg "Cleaning bootflash: completed."

  exit

Before an upgrade is performed, the script can be triggered to remove all unwanted files from bootflash.

4. Also make sure no IOX app is installed in the device so that the required space can be available in boot flash.

Upgrade steps using Cisco Catalyst SD-WAN  Manager.

1. Access SD-WAN Manager:

• Log into Cisco SD-WAN Manager with the administrator account & confirm it is running versions 20.12 or higher.

2. Upload Software (17.12.05):

• Navigate to the Maintenance – software repository section.
• Select the option to upload new image.
• Upload the image version 17.12.05 .

3. Install Software:

• Initiate the upgrade process to install the 17.12.05 image.
• Monitor the upgrade process and ensure the device successfully reboots into the intermediate software version.

[30-Oct-2024 16:31:08 UTC] Software Install action submitted for execution

[30-Oct-2024 16:31:10 UTC] Executing device action Software Install

[30-Oct-2024 16:31:10 UTC] Installing and activating software image

[30-Oct-2024 16:31:11 UTC]Current active partition: 17.10.01.0.1479

[30-Oct-2024 16:31:11 UTC] Upgrade Requested for SW version : 17.12.05

[30-Oct-2024 16:31:14 UTC] Configuring upgrade confirm timer to 15 minutes

[30-Oct-2024 16:31:14 UTC] Software Image ir1101-universalk9.17.12.05.bin

[30-Oct-2024 16:31:14 UTC] Sending requested upgrade action to the device

[30-Oct-2024 16:31:14 UTC] Software image download once started can take upto 60 minutes

[30-Oct-2024 16:31:18 UTC] [in_progress] Started Pre-Upgrade Check: Pre-Upgrade Checks for 'Download and Upgrade' workflow

[30-Oct-2024 16:32:35 UTC] [success] Image Validity Check: Image is compatible with the platform

[30-Oct-2024 16:32:35 UTC] [success] Image compatibility with controller: Image controller-version 20.12 is compatible withvManage version 20.12

[30-Oct-2024 16:32:35 UTC] [success] Disk Space Check: Disk has enough space to install the new image

[30-Oct-2024 16:32:35 UTC] [success] SD-WAN Installer Space Check: SD-WAN installer has sufficient space

[30-Oct-2024 16:32:35 UTC] [success] System Load Check: System has healthy CPU levels

[30-Oct-2024 16:32:35 UTC] [success] Memory Usage Check: System has healthy Memory levels

[30-Oct-2024 16:32:35 UTC] [success] Config-register Check: Autoboot is enabled

[30-Oct-2024 16:32:35 UTC] [success] Control-connection status Check: Control-connection to vManage is UP

[30-Oct-2024 16:32:35 UTC] [success] Finished Pre-Upgrade Check: Proceeding with the requested workflow

[30-Oct-2024 16:32:38 UTC] Device: Downloading <</software/package/ir1101-universalk9.17.12.05.SPA.bin>>started

[30-Oct-2024 16:39:57 UTC] Device: All installed versions: 17.10.01.0.1479;

[30-Oct-2024 16:40:00 UTC] Device: Received installation request for software version(5-tuple): 17.12.05

[30-Oct-2024 16:44:59 UTC] Device: Installed 17.12.05

[30-Oct-2024 16:45:01 UTC] Device: Reloading the device to activate 17.12.05

[30-Oct-2024 16:57:07 UTC] Device: Post upgrade check sucessfull. Current active partition is 17.12.05

[30-Oct-2024 16:57:32 UTC] Device: Successfully applied upgrade-confirm

[30-Oct-2024 16:57:48 UTC] Software Install complete

Note: vManage term in the message refers to SD-WAN Manager.

4. Set default to 17.12.05 and remove 17.10.04 from device in SD-WAN Manager.

5. Verify Upgrade:

• After reboot, login back into SD-WAN Manager.
• Verify that the device is running the 17.12.05 image.

Note: If the target upgrade is 17.12.05 then upgrade process ends here, if target upgrade to 17.15.01a, then proceed to the next step.

6. 17.12.05 TO 17.15.01a upgrade from require SD-WAN Manager version 20.15:

  Upload 17.15.01a  Image:

• Again, navigate to the software upgrade section.
• Select the option to upload new image.
• Upload the software version 17.15.01

   Install 17.15.01a Image:

• Initiate the upgrade process to perform the software installation
• Monitor the process and ensure the device successfully reboots into the final software version.

[3-Nov-2024 6:57:40 UTC] Software Install action submitted for execution

[3-Nov-2024 6:57:42 UTC] Executing device action Software Install

[3-Nov-2024 6:57:42 UTC] Installing and activating software image

[3-Nov-2024 6:57:46 UTC]Current active partition: 17.12.05

[3-Nov-2024 6:57:46 UTC] Upgrade Requested for SW version : 17.15.01a.0.193

[3-Nov-2024 6:57:53 UTC] Configured upgrade confirm timer is 15 minutes

[3-Nov-2024 6:57:53 UTC] Software Image ir1101-universalk9.17.15.01a.SPA.bin

[3-Nov-2024 6:57:53 UTC] Sending requested upgrade action to the device

[3-Nov-2024 6:57:53 UTC] Software image download once started can take upto 60 minutes

[3-Nov-2024 6:57:56 UTC] [in_progress] Started Pre-Upgrade Check: Pre-Upgrade Checks for 'Download and Upgrade' workflow

[3-Nov-2024 6:58:14 UTC] [success] Image Validity Check: Image is compatible with the platform

[3-Nov-2024 6:58:14 UTC] [success]Image compatibility with controller: Image controller-version 20.15 is compatible with vManage version 20.15

[3-Nov-2024 6:58:14 UTC] [success] Disk Space Check: Disk has enough space to install the new image

[3-Nov-2024 6:58:14 UTC] [success] SD-WAN Installer Space Check: SD-WAN installer has sufficient space

[3-Nov-2024 6:58:14 UTC] [success] System Load Check: System has healthy CPU levels

[3-Nov-2024 6:58:14 UTC] [success] Memory Usage Check: System has healthy Memory levels

[3-Nov-2024 6:58:14 UTC] [success] Config-register Check: Autoboot is enabled

[3-Nov-2024 6:58:14 UTC] [success] Control-connection status Check: Control-connection to vManage is UP

[3-Nov-2024 6:58:14 UTC] [success] Finished Pre-Upgrade Check: Proceeding with the requested workflow

[3-Nov-2024 6:58:16 UTC] Device: Downloading<</software/package/ir1101-universalk9.17.12.05.SPA.bin>>started

[3-Nov-2024 7:05:33 UTC] Device: All installed versions: 17.12.04.0.4826;

[3-Nov-2024 7:05:37 UTC] Device: Received installation request for software version(5-tuple): 17.15.01a.0.193

[3-Nov-2024 7:11:22 UTC] Device: Installed 17.15.01a.0.193

[3-Nov-2024 7:11:24 UTC] Device: Reloading the device to activate 17.15.01a.0.193

[3-Nov-2024 7:22:56 UTC] Device: Post upgrade check sucessfull. Current active partition is 17.15.01a.0.193

[3-Nov-2024 7:23:42 UTC] Device: Successfully applied upgrade-confirm

[3-Nov-2024 7:24:00 UTC] Software Install complete

Steps after the upgrade

Remove the 17.12.05 intermediate image that is no longer needed to free up the space for other purposes.

1. Set default version to 17.15.01a from Maintenance -> select device -> Software Image actions -> Set default versions

2. Remove intermediate 17.12.05 build from vmanage GUI by navigating to Maintenance -> select device -> Software Image actions -> Delete available software

From drop down select 17.12.05

Key points

• Software Versions: Ensure that you use the exact image versions mentioned (17.12.05 as intermediate and 17.15.01a as final version) for upgrade fron 17.10.x to 17.15.x.
• This is applicable only to IR1101’s that are in Controller mode that uses SD-WAN Manager to perform the upgrade process.
• Since this is a two step upgrade process, maintenance window can be planned accordingly.
• Make sure to remove 17.12.05 image from SD-WAN Manager after the upgrade is successfully completed.