Reduce Secure Firewall 7.6 FTD HA Upgrade Failure

Introduction

This document describes troubleshooting to address FTD upgrade failures from versions 7.0 to 7.2, particularly in High Availability (HA) deployments.

Background Information

Over half of these failures stem from issues during the 200_enable_maintenance_mode phase, with existing HA validations mainly performing basic active/standby state checks, which are insufficient for comprehensive HA transitions.

With the Secure Firewall 7.6 update, improved HA validations have been introduced to tackle these problems. These enhancements include thorough checks for HA state transitions, extended timeouts for synchronization processes, and enhanced error reporting. This update aims to significantly reduce post-upgrade HA issues and overall upgrade failures, ensuring a smoother and more reliable upgrade process for HA deployments.

Migrated from: https://confluence-eng-rtp2.cisco.com/conf/display/IFT/FTD+HA+Upgrade+Failure+Reduction

Problem

• There are a significant number of FTD upgrade failures reported by customers across 7.0, 7.1, and 7.2 releases for HA deployments.
• More than 50% of failures come from FTD HA deployments. Failures in 200_enable_maintenance_mode contribute to HA failures.
• Existing HA state validations are basic validations like active/standby state checks and do not validate the HA transitions completely.

What’s New (Solution)

Improved HA validations for FTD upgrade:

• Validation for HA state transition
• Improved FTD HA upgrade timeouts for HA transition state like config sync(7200 seconds), app sync(1200 seconds), and bulk sync(7200 seconds)
• Giving more control to FMC on when to start or fail the FTD upgrade
• Improved error reporting and recovery message for FTD HA upgrades

As compared to previous releases, it has:

• Improved HA validations helps reduce the post-upgrade HA creation issues in HA deployments
• Improved validations helps to reduce FTD upgrade failures

Prerequisites

Supported Platforms

• Manager(s) and Version (s) : FMC 7.6.0
• Application (ASA/FTD) and Minimum Version of Application: FTD 7.6.0; FMC managing 7.6.0 FTD HA
• Supported Platforms: All platforms running FTD HA
  
  

Note: This feature applies to only FMC-managed FTD HA deployments. This feature does not apply to FDM-managed FTD HA or clustered devices.

Feature Overview

• This feature helps in reducing FTD upgrade failures in HA deployment by checking the HA states of upgraded units by FMC after the reboot part of the upgrade process.
• After the upgrade reboot, FMC checks for active/standby state and any failures in HA synchronization.
• FTD notifies FMC on when to start or fail the upgrade on the second node in the form of a new HA advance troubleshoot.
• If there is any failure in joining the HA post-upgrade reboot, an appropriate message is displayed on the FMC UI.

New Upgrade Workflow for FTD HA

Standby Unit is the First to Upgrade

First unit upgrade (Standby Unit)

• During the first unit upgrade, the upgrade script initiates action_queue task to collect HA advance troubleshoot data at 999_finish stage.
• Inserted task execution starts only after post-upgrade reboot and collects troubleshoot information in the form of JSON file.
• The same JSON file is synced to FMC.
• Once the first node exits from maintenance mode, FMC triggers a remote action_queue task on the active unit in order to collect the HA advance troubleshoot (Active unit needs to be 7.6 or above). If the active unit is found lower than 7.6, no troubleshoot is collected from the active unit and FMC takes decision based only on troubleshoot collected from the standby unit.

Once HA advance troubleshoot is collected from both units, FMC decides to start the upgrade or block the upgrade on the second node (active unit).

Second Unit Upgrade (Active Unit)

• Similar to the standby unit, the upgrade script initiates the action_queue task to collect HA advance troubleshoot at 999_finish stage.
• Inserted task execution start only post-upgrade reboot and generates troubleshoot information in the form of a JSON file.
• The same file is synced to FMC.
• If either of the units reports HA failure, HA failure data is shown on the FMC UI on the upgrade tab.
• In case of any failure in joining HA post-upgrade reboot, the upgrade is marked completed and on the same upgrade tab HA validation failures are reported.

HA Advance Troubleshoot

• HA advance troubleshoot is a new single JSON file introduced as part of this feature which contains HA information. It is generated after the reboot after an upgrade and sent from the FTD to the FMC.
• File name and path: /ngfw/var/sf/sync/ha/upgrade_troubleshoot
• As soon as FMC collects the HA advance troubleshoot from the first (standby) unit, FMC triggers a remote task to collect the same information from the active unit.
  • This remote data collection is only supported when the devices are running 7.6 or higher.
  • If devices are found running a version less than 7.6, then remote data collection is skipped. So, in this case, FMC would only collect data from the standby unit and decide for further action.

• HA advance troubleshoot generation is quick. If Lina is down and fails to generate the report, it immediately exits.
  • Device reboot time depends on platform to platform and reboot time is the same as we have documented for each platform.

HA Advance Troubleshoot Report

Each HA unit generates an HA advance troubleshoot data in form of JSON file post-upgrade reboot and shares it with FMC. Here are validation examples when there is a failure and success.

Example of HA Validation Failure

File: /ngfw/var/sf/sync/ha/upgrade_troubleshoot

{
"failover_lan" : "NA",
"error_code" : "1046 -
STARTUP_FAILOVER_CONFIG_NOT_PRESENT",
"current_time" : 1701369637,
"peer_HA_state" : "Not Detected",
"FMC_AQ_ID" : "0",
"state_link" : "NA",
"json_time" : "18:40:37 UTC Nov 30 2023",
"my_HA_state" : "Disabled",
"my_HA_role" : "Secondary",
"return_status" : "STATUS_ERROR",
"message" : "Failover config is not present on the startup
config. Device is in standalone state. Please configure failover.",
"peer_HA_role" : "Primary"
}

Example of Successful HA Validation

File: /ngfw/var/sf/sync/ha/upgrade_troubleshoot

{
"return_status" : "STATUS_OK",
"message" : "No Action required.",
"current_time" : 1699526448,
"my_HA_state" : "Standby Ready",
"FMC_AQ_ID" : "0",
"retry_count" : "3",
"error_code" : "0000 - HA_OK",
"peer_HA_role" : "Secondary",
"failover_lan" : "up",
"peer_HA_state" : "Active",
"my_HA_role" : "Primary",
"state_link" : "up",
"json_time" : "10:40:48 UTC Nov 09 2
}

HA Advance Troubleshoot Contents

Location of HA Advance Troubleshoot File

HA advance troubleshoot JSON file location:

On FTD: /ngfw/var/sf/sync/ha/upgrade_troubleshoot
On FMC: /var/sf/peers/<peer-uuid>/sync/ha/upgrade_troubleshoot

• The HA troubleshoot relies on the lina command.
  • If troubleshoot fails to generate at /ngfw/var/sf/sync/ha/upgrade_troubleshoot, user can refer to logs at: /ngfw/var/log/ha_upgrade_troubleshoot.log
• /ngfw/var/sf/sync/ha/upgrade_troubleshoot and /ngfw/var/log/ha_upgrade_troubleshoot.log files are part of FTD Troubleshoot file.

Tips for HA Advance Troubleshoot Generation Issues

Sometimes HA advance troubleshoot is not be generated due to system state and reason for this could be lina down or action queue process is down post upgrade reboot. If lina or action queue is down, this is a problem.

In such cases, check to see if lina and ActionQueue processes are running by using this command in expert mode:

pmtool status | grep lina
lina (system) - Running 5503  Indicates Lina is up and running

pmtool status | grep ActionQueueScrape
ActionQueueScrape (system) - Running 5268  Indicates action queue is up and running

Return Status and Action in HA Advance Troubleshoot

• STATUS_INIT: This indicates the HA troubleshoot has been triggered.
• STATUS_OK: The device is in a stable state. No action is required.
• STATUS ERROR: This determines an error has occurred due to which HA is not formed. The user needs to take an action based on the message displayed or the user needs to contact TAC.
• STATUS_RETRY: The device can be in one of the intermediate states. The HA troubleshoot keeps retrying after a fixed interval based on the state until STATUS_ERROR or STATUS_OK is encountered.
  • Based on failures encountered STATUS ERROR, the HA failures are categorized into 2 cases:
    • User Intervention – These HA failures can be fixed by the user, and the user can resume the upgrade, where the TAC intervention is not required.
    • TAC intervention – For these HA failures, the user cannot fix it by themselves; TAC intervention is required.

Error Code and Classification

Based on error codes, Errors are classified as shown here:

User Intervention Messages

TAC Intervention Messages

Firewall Management Center UI Changes

Software Architecture

This feature is highly dependent on existing action queue framework. The feature uses underlying lina CLI to generate the HA advance troubleshoot data.

FAQs

Q: Is the feature applicable for FTD upgrade revert functionality?
A: No. This feature is not applicable for revert functionality as FTD revert works in parallel, not 1-by-1.

Q: If upgrade fails at 200_enable_maintenance_mode.pl, does it generate the advance troubleshoot data?
A: No. HA advance troubleshoot is generated only after post-upgrade reboot and not during upgrade failure

Q: If upgrade is blocked due to HA validations on second unit, can a user trigger upgrade on second unit alone?
A: Yes. User has to select the HA pair again for upgrade and FMC triggers the upgrade only on non-upgraded unit.