Secure Firewall FTD Deployment Failure

Issue

Network disruptions and outages have been observed on Cisco Firewall Firepower Threat Defense (FTD). Repeated incidents have led to denied traffic, including SNMP communications, and have required device reboots and ongoing monitoring to identify the root cause and mitigate further impact.

Environment

• Cisco Secure Firewall Firepower 1140 appliances (impacts any FTD model)

• FTD software versions: 7.4.2.4 (other versions also impacted)

• Dynamic object-based Access Control Policies (ACPs)

• Frequent policy deployments

Resolution

To address the recurring failover and policy deployment issues on Cisco Secure Firewall FTD devices, a comprehensive set of troubleshooting and remediation steps must be followed. The workflow listed is structured to provide clear separation and explanation of each step, including monitoring, data collection, diagnostics, and upgrade guidance.

1: Use packet-tracers to check routing and access for the intended traffic.

firepower# packet-tracer input INPUTNAMEIF tcp SRCIP 54321 DSTIP 443
firepower# packet-tracer input INPUTNAMEIF icmp SRCIP 8 0 DSTIP

2: Use captures at the FTD to determine if packets are being dropped upon entry 'by configured rule' even though a valid rule and route exists for the traffic.

firepower# capture 1 interface INPUTIFNAME trace detail trace-count 1000 match ip host SRCIP host DSTIP
firepower# capture x type asp-drop all match ip host SRCIP host DSTIP
firepower# show capture
capture 1 type raw-data trace detail trace-count 1000 interface inside [Capturing - 31565 bytes] 
  match ip 10.1.1.0 255.255.255.0 any 
capture x type asp-drop all [Capturing - 31565 bytes] 
  match ip 10.1.1.0 255.255.255.0 any 

3: Check the FTD messages logs for evidence of defect CSCwo78475.

> expert
admin@FTD-1:~$ sudo su
Password:
root@FTD-1:/Volume/home/admin# cat /ngfw/var/log/messages | grep -E "New inspector|did not finish|swapped"
Feb 10 18:35:03 FTD-device SF-IMS[28366]: New inspector is not initializing Identity API because it's already inited. 
Feb 10 18:35:03 FTD-device SF-IMS[28366]: New inspector has different policy groups or ABP name to ID mappings from existing Identity API. Need to rebuild user group hash, group bit hash and ABP name to ID mapping 
Feb 10 18:35:10 FTD-device SF-IMS[28366]: Reading the muster data snapshot did not finish in time: 4 sec. 
Feb 10 18:36:22 FTD-device SF-IMS[28366]: Identity API state swapped

4: Match the timestamps for these logs with those for deployment logs in the FTD.

Feb 10 18:34:45 FTD-device policy_apply.pl[18923]: INFO  Deployment type is NORMAL_DEPLOYMENT and device_version is 7.4.2.4  (Framework::FTDHA 59 <- Framework 845 <- Transaction 1142)
Feb 10 18:37:03 FTD-device policy_apply.pl[30894]: INFO  finalizeDeviceDeployment - sandbox = /var/cisco/deploy/sandbox took 1 (memory = 189.78 MB, change = 65.10 MB)  (Framework 4878<1504 <- Transaction 1833 <- main 231)

5: If the FTDs are in HA, failover to the standby FTD and check the same afterward to ensure traffic recovery.

6: If matching logs and conditions are found in the FTD, the device is impacted by the defect and can be upgraded to 7.4.3. In the meantime, deployments can be limited to after-hours to reduce traffic impact.

Cause

The underlying cause of the observed traffic impacts and policy deployment issues is attributed to known defect affecting FTD software, notably:

• Cisco Bug ID CSCwo78475: Traffic hits incorrect Access Control Policy (ACP) rules during policy deployment on FTD devices with dynamic objects. This can result in legitimate traffic being denied, even when proper rules exist in the running configuration. Fixed in version 7.4.3.

Related Content

• Cisco Bug ID CSCwo78475: Traffic hits incorrect ACP rules during policy deployment on FTD with dynamic objects

• Cisco Technical Support & Downloads: Cisco Technical Support & Downloads