Issue
High CPU utilization was observed on FTD devices, leading to connectivity problems and preventing users from accessing critical business applications. The firewall exhibited elevated datapath and Snort CPU usage, with users experiencing latency and intermittent access issues. Investigation revealed a large number of embryonic TCP connections, with a significant portion originating from internal security scanners, resulting in resource exhaustion and degraded performance.
Environment
- Cisco Secure Firewall Firepower Threat Defense (FTD)
- Hardware: Cisco Firepower 1150
- Software Version: 7.4.2.3
- Managed by: Firepower Management Center (FMC)
- High Availability (HA) configuration
- Datapath and Snort CPU consistently at or near 100%
- High number of embryonic TCP connections due to internal scanners
- Recent changes: Log collector configurations applied and reverted; access rule deployment; observed failover event
- Systems generating high connections identified as internal Qualys scanners
Resolution
Identified High CPU Usage on DATAPATH used for traffic processing.
device# show processes cpu-usage sorted non-zero
Hardware: FPR-1150
Cisco Adaptive Security Appliance Software Version 9.20(2)43
ASLR enabled, text region 562a19048000-562a1e49126d
PC Thread 5Sec 1Min 5Min Process
- - 99.7% 99.7% 99.7% DATAPATH-4-22658
- - 99.7% 99.7% 99.6% DATAPATH-3-22657
- - 99.7% 99.6% 99.6% DATAPATH-2-22656
- - 99.6% 99.7% 99.7% DATAPATH-5-22659
- - 97.5% 97.1% 97.1% DATAPATH-1-22655
- - 97.4% 97.1% 97.1% DATAPATH-0-22654
0x0000562a1b8c55e3 0x0000151e97f523e0 1.1% 1.6% 1.6% CP Processing
0x0000562a1d408771 0x0000151e97f434a0 0.4% 0.2% 0.0% Unicorn Proxy Thread
0x0000562a1b6ba40a 0x0000151e97f3cb80 0.3% 0.3% 0.3% appagent_async_client_receive_thread
0x0000562a1cfebc65 0x0000151e97f43f80 0.1% 0.1% 0.1% IP SLA Mon Event Processor
0x0000562a1d328a89 0x0000151e97f64240 0.1% 0.1% 0.1% lina logclient Rx data thread
0x0000562a1d72eb46 0x0000151e97f417a0 0.0% 0.1% 0.0% cli_xml_request_process
0x0000562a1df983a5 0x0000151e97f69940 0.0% 0.1% 0.0% Checkheaps
From the FTD CLI, an output of show conn detail was exported for review of Connection Statistics by internal automation tools.
CAUTION: The output of show conn detail from the CLI can be extremely long if the connection count is over 100,000. Ensure sufficient time is allotted for this collection.
The disk0 corresponds with /mnt/disk0/ directory in the FTD backend. Export the file accordingly.
device# show conn detail | redirect disk0:/shconndetMMDDYY.txt
Review the connection statistics from the results of the tool for embryonic connections in high amounts:
Total Emryonic Conns: 121611. This is 87.984% of the total conns (138219)
--
Top-5 Embryonic IPs (SYN, but not SYN/ACK - 'aA' flags) going through the device
IP Count Percent
--------------------------------------------------------------
10.5.30.77 81519 33.517%
10.1.30.102 40042 16.463%
10.1.212.14 907 0.373%
10.1.204.4 837 0.344%
10.1.21.122 804 0.331%
After identifying the source IPs (in this case, internal security scanners), prevent the source from generating the traffic and clear its connections from the FTD.
device# clear conn add 10.5.30.77
4563 connection(s) deleted.
device# show conn count
5936 in use, 465189 most used
Inspect Snort:
preserve-connection: 4451 enabled, 0 in effect, 432406 most enabled, 0 most in effect
Monitor CPU Utilization After Mitigation to confirm that the cause was traffic-induced.
device# show cpu
CPU utilization for 5 seconds = 9%; 1 minute: 28%; 5 minutes: 70%
Traffic connectivity should return to normal, and latency should no longer be observed.
Cause
The root cause of high CPU and connectivity issues was excessive embryonic connections generated by internal security scanners. These connections, primarily SYN packets without corresponding SYN/ACK responses, overwhelmed the FTD datapath and Snort processes. The high volume of incomplete connections led to resource exhaustion, resulting in sustained high CPU utilization, intermittent connectivity, and impact on business-critical application access.
Related Content
Feedback