Upgrade Failure on Multi Instance HA FTD

Introduction

This document describes how to upgrade failure on multi-instance High Availability (HA) FTD.

Problem

The troubleshoot file generates many logs at the time of the FTD multi-instance HA pair upgrade from 6.4 to 6.6 due to the disk issue. The file generates a large number of logs due to the log rotation process and does not rotate the files correctly.

Solution

1. Check for the files that consume high disk and do not rotate the logs properly:

root@:/ngfw/var/log# du -sh * | grep G

1.1G    process_stderr.log

2.3G    process_stdout.log

4.4G    top.log

2. Try to rotate the files (process_stderr.log, process_stdout.log, top.log). For example:

root@:/ngfw/var/log# gzip process_stderr.log

root@:/ngfw/var/log# ls -l process_stderr.log*

-rw-r--r-- 1 root root     1506 Jan 17 01:35 process_stderr.log

-rw-r--r-- 1 root root   771675 Mar 16  2020 process_stderr.log.1.gz

-rw-r--r-- 1 root root   570153 Mar  8  2020 process_stderr.log.2.gz

-rw-r--r-- 1 root root   744427 Mar  2  2020 process_stderr.log.3.gz

-rw-r--r-- 1 root root   570641 Feb 23  2020 process_stderr.log.4.gz

-rw-r--r-- 1 root root 61548926 Jan 17 01:34 process_stderr.log.gz

root@:/ngfw/var/log# mv process_stderr.log.gz process_stderr.log.5.gz

root@:/ngfw/var/log# ls -l process_stderr.log*

-rw-r--r-- 1 root root     2436 Jan 17 01:36 process_stderr.log

-rw-r--r-- 1 root root   771675 Mar 16  2020 process_stderr.log.1.gz

-rw-r--r-- 1 root root   570153 Mar  8  2020 process_stderr.log.2.gz

-rw-r--r-- 1 root root   744427 Mar  2  2020 process_stderr.log.3.gz

-rw-r--r-- 1 root root   570641 Feb 23  2020 process_stderr.log.4.gz

-rw-r--r-- 1 root root 61548926 Jan 17 01:34 process_stderr.log.5.gz

root@:/ngfw/var/log# gzip top.log

root@:/ngfw/var/log# mv top.log.gz top.log.1.gz

root@:/ngfw/var/log# ls -l top.log*

3. After rotation is done if any logs consume more disk, move to the common folder and download from FMC GUI for reference and delete the file:

root@:/ngfw/var/log# mv top.log.5.gz /ngfw/var/common

root@:/ngfw/var/common# rm  top.log.5.gz 

4. Create the workaround to disable generation of the troubleshoot file script while you upgrade:

root@FW:/ngfw/var/sf# mkdir upgrade-scripts

root@FW:/ngfw/var/sf# cd upgrade-scripts/

root@FW:/ngfw/var/sf/upgrade-scripts# mkdir 6.6.1

root@FW:/ngfw/var/sf/upgrade-scripts# cd 6.6.1

root@FW:/ngfw/var/sf/upgrade-scripts/6.6.1# mkdir 000_start

root@FW:/ngfw/var/sf/upgrade-scripts/6.6.1# cd 000_start

root@FW:/ngfw/var/sf/upgrade-scripts/6.6.1/000_start# touch /ngfw/var/sf/upgrade-scripts/6.6.1/000_start/400_run_troubleshoot.sh

Note: The troubleshoot script does not run from the upgrade package, it runs from the blank file you have created and, the upgrade does not face a less disk space available issue.

5. Start the upgrade.