Troubleshoot IOS XE SD-WAN Upgrade Failure: Insufficient Space

Introduction

This document describes the process to diagnose and resolve the issue when a Cisco IOS® XE SD-WAN upgrade fails due to insufficient storage capacity.

Prerequisites

WAN Edge Router

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco Software-Defined Wide Area Network (SD-WAN)
• Cisco IOS® XE Command Line Interface (CLI)

For Cisco IOS XE SD-WAN upgrade information Upgrade Cisco IOS XE SD-WAN Router with the Use of CLI or vManage

Components Used

This document is based on these software and hardware versions:

• Router c1111-8P
• Software Version 17.3.2
• Software Version 17.3.4
• Router in controller mode

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Conditions

1. Unable to upgrade the router model c1111-8p to 17.3 and other versions before 17.5.

2. Consider these errors:

Not enough space under bootflash for image download and package expansion on image installation. Required space: 1207202432 Available space: 659275776. Image installation failed.
Not enough space in SD-WAN installer filesystem. Please remove one of the installed images and try again. SD-WAN installer filesystem size - 396672 Kilobytes, Available space - 196986 Kilobytes, Required space - 243602 Kilobytes.

3. Upgrade fails as shown in these images:

Image1

Image2

4. There is only one version installed. Use the command show sdwan software for verification.

Example

edge#show sdwan software 
VERSION           ACTIVE  DEFAULT  PREVIOUS  CONFIRMED  TIMESTAMP                  
-----------------------------------------------------------------------------------
17.03.04a.0.5574  true    true     false     user       2021-10-22T23:00:30-00:00  

Total Space:387M Used Space:191M Available Space:192M

Troubleshoot

1. Delete unnecessary files with this command:

delete /force /recursive bootflash:/<directory>/<file ext>

You can delete the next list of files. They are internal logs created for troubleshooting purposes.

Caution: Delete them if you are sure there is no other investigation in progress.

bootflash:/vmanage-admin/*.tar.gz
bootflash:/core/*.core.gz
bootflash:/core/*.tar.gz
bootflash:/syslog/*
bootflash:/tracelogs/*
bootflash:/crashinfo_*

Example

delete /force /recursive bootflash:/tracelog/*
delete /force /recursive bootflash:/ vmanage-admin/*.tar.gz

Warning: Do not delete these files. They are essential files for router functionality.

bootflash:/.sdwaninstaller
bootflash:/.sdwaninstallerfs
bootflash:/.cdb_backup folder
bootflash:/sdwan folder
bootflash:/*pkg
bootflash:/*.conf.

2. Reboot helps to release bootflash memory as shown in the example.

Caution: Control plane and Data plane flaps during reboot. 

*** Before Reboot ***

edge#show sdwan software 
VERSION           ACTIVE  DEFAULT  PREVIOUS  CONFIRMED  TIMESTAMP                  
-----------------------------------------------------------------------------------
17.03.04a.0.5574  true    true     false     user       2021-10-22T23:00:30-00:00  
Total Space:387M Used Space:191M Available Space:192M  >>> Here
edge#

*** Rebooting edge
edge#reload
Proceed with reload? [confirm]Connection to 192.168.1.6 closed by remote host.

*** After Reboot ***
edge#show sdwan software
VERSION           ACTIVE  DEFAULT  PREVIOUS  CONFIRMED  TIMESTAMP                  
-----------------------------------------------------------------------------------
17.03.04a.0.5574  true    true     false     user       2021-10-22T23:00:30-00:00  

Total Space:387M Used Space:190M Available Space:193M >>>> Here

Work Around

Apply the next steps if previous actions did not work to upgrade the router c1111-8p. 

Issue 1

If the upgrade issue is related to this error Not enough space under bootflash, reference to the next instructions.

Step 1. Enable IOx.

Config-transaction
iox
commit
end

Step 2. Run the next command to move the IOx log from bootflash to flash.

app-hosting move system techsupport to flash:

Step 3. Disable IOx.

Config-transaction
no iox
commit
end

Example

Before IOx enabling
edge#dir
Directory of bootflash:/
<snipped>
2908606464 bytes total (646643712 bytes free)
edge#                

After IOx disabling
edge#dir
Directory of bootflash:/
<snipped>
2908606464 bytes total (1671614464 bytes free)
edge#

Step 4. Proceed to upgrade.

request platform software sdwan software install bootflash:<version.bin>
request platform software sdwan software activate <IOS version>
request platform software sdwan software upgrade-confirm
request platform software sdwan software set-default <IOS version>

Tip: For Upgrade via vManage Upgrade SD-WAN cEdge Router with the Use of CLI or vManage

Issue 2

If the upgrade issue is related to this error 'Not enough space in SD-WAN installer filesystem', then reference these instructions.

Step 1. Log in to the shell. 

For more information, reference to .

Step 2. Go to the .sdwaninstaller/images directory. Use Linux commands to find and remove old files.

cd /bootflash/.sdwaninstaller/images
ls -l
rm -rf <old file>

Step 3. Exit from the shell with Linux command.

exit

step 4. Proceed to upgrade.

request platform software sdwan software install bootflash:<version.bin>
request platform software sdwan software activate <IOS version>
request platform software sdwan software upgrade-confirm
request platform software sdwan software set-default <IOS version>

Tip: For Upgrade via vManage Upgrade SD-WAN cEdge Router with the Use of CLI or vManage

Example

**** Before to Clean Up ****

[23-Sep-2022 18:08:05 UTC] Installing software image
[23-Sep-2022 18:08:09 UTC] Current active partition: 17.03.04a.0.5574
[23-Sep-2022 18:08:10 UTC] Configuring upgrade confirm timer to 15 minutes
[23-Sep-2022 18:08:11 UTC] Software image: c1100-universalk9.17.03.05.SPA.bin
[23-Sep-2022 18:08:11 UTC] Software image download may take upto 180 minutes
[23-Sep-2022 18:08:17 UTC] Not enough space in SDWAN installer filesystem. Please remove one of the installed images and try again. SDWAN installer filesystem size - 396672 Kilobytes, Available space - 196986 Kilobytes, Required space - 243602 Kilobytes.
[23-Sep-2022 18:08:17 UTC] Failed to process device request -  
Error type : application 
Error tag : operation-failed 
Error info : <error-info>
  <bad-element>install</bad-element>
</error-info>

edge#show sdwan software
VERSION           ACTIVE  DEFAULT  PREVIOUS  CONFIRMED  TIMESTAMP                  
-----------------------------------------------------------------------------------
17.03.04a.0.5574  true    true     false     user       2021-10-22T23:00:30-00:00  

Total Space:387M Used Space:190M Available Space:193M  >>> Here
edge#

**** Clean old files from sdwaninstaller ****
[edge:/]$ cd /bootflash/.sdwaninstaller/images
[edge:/bootflash/.sdwaninstaller/images]$ ls -l
total 2
drwxr-xr-x. 3 root root 1024 Oct  3  2019 SR_BOOT=usb0:c1100-ucmk9.16.9.3.SPA.bin
drwxr-xr-x. 3 root root 1024 Sep 23 13:33 c1100-universalk9.17.03.04a.SPA.bin
[edge:/bootflash/.sdwaninstaller/images]$ rm -rf SR_BOOT=usb0:c1100-ucmk9.16.9.3.SPA.bin
[edge:/bootflash/.sdwaninstaller/images]$ ls -l
total 1
drwxr-xr-x. 3 root root 1024 Sep 23 13:33 c1100-universalk9.17.03.04a.SPA.bin
[edge:/bootflash/.sdwaninstaller/images]$ exit
exit
Session log harddisk:tracelogs/system_shell_R0-0.18547_0.20220923134926.bin closed.
edge#
edge#show sdwan software 
VERSION           ACTIVE  DEFAULT  PREVIOUS  CONFIRMED  TIMESTAMP                  
-----------------------------------------------------------------------------------
17.03.04a.0.5574  true    true     false     user       2021-10-22T23:00:30-00:00  

Total Space:387M Used Space:120M Available Space:262M				>>>> Space increased


**** Upgrade Successful:
[23-Sep-2022 18:53:38 UTC] Installing software image
[23-Sep-2022 18:53:43 UTC] Current active partition: 17.03.04a.0.5574
[23-Sep-2022 18:53:44 UTC] Configuring upgrade confirm timer to 15 minutes
[23-Sep-2022 18:53:44 UTC] Software image: c1100-universalk9.17.03.05.SPA.bin
[23-Sep-2022 18:53:44 UTC] Software image download may take upto 180 minutes
[23-Sep-2022 18:53:54 UTC] Connection Instance: 2, Color: private1
[23-Sep-2022 18:53:54 UTC] Downloading http://192.168.16.17:8080/software/package/c1100-universalk9.17.03.05.SPA.bin?deviceId=10.10.10.10
[23-Sep-2022 19:06:04 UTC] Downloaded http://192.168.16.17:8080/software/package/c1100-universalk9.17.03.05.SPA.bin?deviceId=10.10.10.10
[23-Sep-2022 19:09:31 UTC] install_add: START Fri Sep 23 14:06:10 COT 2022
Sep 23 14:06:12.207 %INSTALL-5-INSTALL_START_INFO: R0/0: install_engine: Started install add bootflash:c1100-universalk9.17.03.05.SPA.bin
[23-Sep-2022 19:09:31 UTC] Installed 17.03.05.0.6600

Caveat

Possible Problems during the upgrade via vManage GUI:

• Not enough space

Fix: verify the bootflash and delete all the images you are not using. You can delete all the packages via the CLI file with the command # delete bootflash:*17.03.01*.

This command deletes all the files that have the name 17.03.01.

Edge#delete bootflash:*17.03.01*
Delete filename [*17.03.01*]?
Delete bootflash:/isr4300-firmware_dreamliner.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_dsp_analogbri.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_dsp_sp2700.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_dsp_tilegx.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_ngwic_t1e1.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_async.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_bri_st_fw.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_cwan.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_ge.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_shdsl.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_ssd.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_nim_xdsl.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_prince.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-rpboot.17.03.01a.SPA.pkg? [confirm]
Delete bootflash:/isr4300-firmware_sm_10g.17.03.01a.SPA.pkg? [confirm]

You can also boot from a USB drive, delete all images from bootflash, copy the bin file into the bootflash, and reboot.

Note: If the router is in controller mode all the commands from IOS-XE related to packages do not work.

Note: Change the boot variable via CLI is also disabled in controller mode. You can move the device to autonomous mode, upgrade, and then move it back to controller mode.

• The image is already present in the router

You can verify if the version is already present via the vManage GUI or CLI with the command show sdwan software.

Fix: Remove the unused images via the vManage GUI or with the command request platform software sdwan software remove <Name>.

• We have more than three images in the router

You can verify if the version is already present via the vManage GUI or CLI with the command show sdwan software .

Fix: Remove the unused images via the vManage GUI or with the command request platform software sdwan software remove <Name>  .

Related Information

• Upgrade SD-WAN cEdge Router with the Use of CLI or vManage
• Cisco Technical Support & Downloads