Troubleshoot FTD Asymmetric Routing Causing TCP Drops and Dual-Zone Connection Logs

Issue

Inbound connections to a DMZ-hosted application behind a Cisco Secure Firewall Threat Defense (FTD) HA pair are intermittently showing two security zones as the traffic source for the same connection attempt. FMC connection events display the expected allowed connection from the outside zone, followed by denied traffic associated with the inside zone, resulting in failed TCP connectivity.

• The FTD route and packet-tracers confirm proper routing, NAT, and rules between the source and destination.

• However, the FTD system support trace only show incoming packets from the source, but not returning traffic from destination.

  > system support trace
  .....
  10.x.x.200 40812 -> 172.x.x.36 9997 6 AS=0 ID=6 GR=1-1 New firewall session
  10.x.x.200 40812 -> 172.x.x.36 9997 6 AS=0 ID=6 GR=1-1 using HW or preset rule order 23, 'intended_allow_rule', action Allow and prefilter rule 0
  10.x.x.200 40812 -> 172.x.x.36 9997 6 AS=0 ID=6 GR=1-1 allow action
  10.x.x.200 40812 -> 172.x.x.36 9997 6 AS=0 ID=6 GR=1-1 app event changes: created
  10.x.x.200 40812 -> 172.x.x.36 9997 6 AS=0 ID=9 GR=1-1 Got start of flow event from hardware with flags 00004001 and rule id 268434433
  10.x.x.200 40812 -> 172.x.x.36 9997 6 AS=0 ID=9 GR=1-1 Generating an SOF event with rule_id = 268434433 ruleAction = 4 ruleReason = None
  10.x.x.200 39194 -> 172.x.x.36 9997 6 AS=0 ID=7 GR=1-1 Got end of flow event from hardware with flags 00002001 and rule id 0
  10.x.x.200 39194 -> 172.x.x.36 9997 6 AS=0 ID=7 GR=1-1 Rule Match Data: rule_id 0, rule_action 0 rev_id 0, rule_flags 2
  10.x.x.200 39194 -> 172.x.x.36 9997 6 AS=0 ID=7 GR=1-1 Generating an EOF event with rule_id = 268434459 ruleAction = 2 ruleReason = None
  10.x.x.200 39194 -> 172.x.x.36 9997 6 AS=0 ID=7 GR=1-1 Received EOF, deleting the snort session
  10.x.x.200 39194 -> 172.x.x.36 9997 6 AS=0 ID=7 GR=1-1 Deleting Firewall session flags=0x20190040, logFlags=0x2001

• From the host, the connection eventually times out.

• Captures on the FTD 'outside' interface confirm that 10.x.x.200 sends SYN and Echo Request to the external destination IP 10.x.x.220.

  firepower# show capture outside_intf
     1: 13:59:03.051388       10.x.x.200.39194 > 10.x.x.220.9997: S 320277106:320277106(0) win 65535 <mss 1460,nop,wscale 8,nop,nop,sackOK>
     2: 13:59:04.063458       10.x.x.200.39194 > 10.x.x.220.9997: S 320277106:320277106(0) win 65535 <mss 1460,nop,wscale 8,nop,nop,sackOK>
     3: 13:59:06.073009       10.x.x.200.39194 > 10.x.x.220.9997: S 320277106:320277106(0) win 65535 <mss 1460,nop,wscale 8,nop,nop,sackOK>
     4: 13:59:10.081477       10.x.x.200.39194 > 10.x.x.220.9997: S 320277106:320277106(0) win 65535 <mss 1460,nop,wscale 8,nop,nop,sackOK>
     5: 13:59:18.090876       10.x.x.200.40812 > 10.x.x.220.9997: S 320277106:320277106(0) win 65535 <mss 1460,nop,wscale 8,nop,nop,sackOK>
     6: 13:59:24.127984       10.x.x.200 > 10.x.x.220 icmp: echo request

• The FTD UNNATs this to the internal destination IP address 172.x.x.36.

  firepower# show capture 1 packet-number 1 trace detail
  .....
  Phase: 3
  Type: UN-NAT
  Subtype: static
  Result: ALLOW
  Elapsed time: 3186 ns
  Config:
  nat (dmz_zone_1,outside_primary) source static dmz1-172.x.x.36 outside-10.x.x.220
  Additional Information:
  NAT divert to egress interface dmz_zone_1(vrfid:0)
  Untranslate 10.x.x.220/0 to 172.x.x.36/0

• The FTD permits the traffic with the appropriate Allow rule as intended.

  Phase: 5
  Type: ACCESS-LIST
  Subtype:
  Result: ALLOW
  Elapsed time: 0 ns
  Config:
  access-group CSM_FW_ACL_ global
  access-list CSM_FW_ACL_ advanced permit object-group ping-echo ifc outside_primary any ifc dmz_zone_1 object dmz1-172.x.x.36 rule-id 268434459
  access-list CSM_FW_ACL_ remark rule-id 268434459: ACCESS POLICY: nf-policy - Mandatory
  access-list CSM_FW_ACL_ remark rule-id 268434459: L7 RULE: intended_allow_rule
  object-group service ping-echo
   service-object icmp echo
  Additional Information:
   This packet will be sent to snort for additional processing where a verdict will be reached
   Forward Flow based lookup yields rule:
   in  id=0x146e745dbd00, priority=12, domain=permit, deny=false
          hits=3532, user_data=0x0000000000000000, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
          src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=8, tag=any, ifc=outside_primary(vrfid:0)
          dst ip/id=172.x.x.36, mask=255.255.255.255, icmp-code=0, tag=any, ifc object-group id 4590,    dscp=0x0, input_ifc=any, output_ifc=any

• The same traffic then correctly proceeds to exit on the "dmz_zone_1" interface toward the DMZ destination 172.x.x.36.

  Result:
  input-interface: outside_primary(vrfid:0)
  input-status: up
  input-line-status: up
  output-interface: dmz_zone_1(vrfid:0)
  output-status: up
  output-line-status: up
  Action: allow
  Time Taken: 106476 ns

• Captures on the FTD dmz_zone_1 interface confirm that 10.x.x.200 sends SYN and Echo Request to the internal destination IP 10.x.x.220.

  firepower# show capture dmz_intf
     1: 13:59:03.051709       802.1Q vlan#600 P0 10.x.x.200.39194 > 172.x.x.36.9997: S 1245309079:1245309079(0) win 65535 <mss 1380,nop,wscale 8,nop,nop,sackOK>
     2: 13:59:04.063610       802.1Q vlan#600 P0 10.x.x.200.39194 > 172.x.x.36.9997: S 1245309079:1245309079(0) win 65535 <mss 1380,nop,wscale 8,nop,nop,sackOK>
     3: 13:59:06.073146       802.1Q vlan#600 P0 10.x.x.200.39194 > 172.x.x.36.9997: S 1245309079:1245309079(0) win 65535 <mss 1380,nop,wscale 8,nop,nop,sackOK>
     4: 13:59:10.081630       802.1Q vlan#600 P0 10.x.x.200.39194 > 172.x.x.36.9997: S 1245309079:1245309079(0) win 65535 <mss 1380,nop,wscale 8,nop,nop,sackOK>
     5: 13:59:18.091166       802.1Q vlan#600 P0 10.x.x.200.40812 > 172.x.x.36.9997: S 1210202278:1210202278(0) win 65535 <mss 1380,nop,wscale 8,nop,nop,sackOK>
     6: 13:59:24.128197       802.1Q vlan#600 P0 10.x.x.200 > 172.x.x.36 icmp: echo request

• However, the FTD dmz_zone_1 interface never show return traffic from internal destination IP 172.x.x.36.

• Captures confirm that the destination 172.x.x.36 SYN,ACK and Echo Reply come in on FTD inside interface instead.

  firepower# show capture inside_intf
     1: 13:59:03.051861       802.1Q vlan#667 P0 172.x.x.36.9997 > 10.x.x.200.39194: S 3938871380:3938871380(0) ack 1245309080 win 29200 <mss 1460,nop,wscale 7>
     2: 13:59:03.051938       802.1Q vlan#667 P0 10.x.x.200.39194 > 172.x.x.36.9997: R 1245309080:1245309080(0) ack 3938871381 win 29200
     3: 13:59:03.051968       802.1Q vlan#667 P0 10.x.x.200.39194 > 172.x.x.36.9997: R 1245309080:1245309080(0) ack 3938871381 win 29200
     4: 13:59:04.063839       802.1Q vlan#667 P0 172.x.x.36.9997 > 10.x.x.200.39194: S 3938871380:3938871380(0) ack 1245309080 win 29200 <mss 1460,nop,wscale 7>
     5: 13:59:04.063915       802.1Q vlan#667 P0 10.x.x.200.39194 > 172.x.x.36.9997: R 1245309080:1245309080(0) ack 3938871381 win 29200
     6: 13:59:04.064068       802.1Q vlan#667 P0 10.x.x.200.39194 > 172.x.x.36.9997: R 1245309080:1245309080(0) ack 3938871381 win 29200
    .....
    23: 13:59:24.128411       802.1Q vlan#667 P0 172.x.x.36 > 10.x.x.200 icmp: echo reply

• As the destination SYN,ACK returns on the incorrect interface, the FTD drops the traffic as it is identified as an initial packet that is not a SYN.

  

  Note: Other TCP drops can occur due to asymmetric routing and can be seen in the FTD show asp drop output.

  Result:
  input-interface: inside(vrfid:0)
  input-status: up
  input-line-status: up
  output-interface: outside_primary(vrfid:0)
  output-status: up
  output-line-status: up
  Action: drop
  Time Taken: 15572 ns
  Drop-reason: (tcp-not-syn) First TCP packet not SYN, Drop-location: frame snp_sp:7337 flow (NA)/NA
  .....
  firepower# show asp drop
  Frame drop:
    SVC Module does not have a channel for reinjection (mp-svc-no-channel)                                        3
    SVC Module unable to fragment packet (mp-svc-no-fragment)                   25
    Flow is being freed (flow-being-freed)                                      78
    No valid adjacency (no-adjacency)                                          707
    Flow is denied by configured rule (acl-drop)                              1738
    First TCP packet not SYN (tcp-not-syn)                                     760
    TCP failed 3 way handshake (tcp-3whs-failed)                                34
    TCP RST/FIN out of order (tcp-rstfin-ooo)                                 1785
    TCP SEQ in SYN/SYNACK invalid (tcp-seq-syn-diff)                             5
    TCP SYNACK on established conn (tcp-synack-ooo)                              1
    TCP packet SEQ past window (tcp-seq-past-win)                               87
    TCP invalid ACK (tcp-invalid-ack)                                            1
    TCP RST/SYN in window (tcp-rst-syn-in-win)                                   1
    DNS Inspect id not matched (inspect-dns-id-not-matched)                     12
    FP L2 rule drop (l2_acl)                                                    34
    Dropped pending packets in a closed socket (np-socket-closed)              145
    NAT failed (nat-xlate-failed)                                                1
    Connection to PAT address without pre-existing xlate (nat-no-xlate-to-pat-pool)                            6940
    TCP Proxy retransmited packet drop (tcp-proxy-retransmit-drop)              90
    TCP Proxy probe reset injected (tcp-proxy-probe-rst-injected)              223
    TCP Proxy probe receive drop (tcp-proxy-probe-tcp-probe-drop)              129
    Blocked or blacklisted by the firewall preprocessor (firewall)               2
    Blocked or blacklisted by the stream preprocessor (stream)                 301
    Packet is blacklisted by snort (snort-blacklist)                            81
    Packet is blocked as requested by snort (snort-block)                       26

Environment

• Cisco Secure Firewall Threat Defense all models and versions

• Cisco Secure Firewall Management Center all models and versions

• External routing device or proxy configured adjacent to the FTD impacting the same traffic.

Resolution

1. Resolve the asymmetric routing behavior at the device which is sending the return traffic to the wrong FTD interface.

2. Otherwise, configure TCP state bypass to attempt to resolve asymmetric routing.

• This allows he firewall to handle TCP sessions that have inbound and outbound packets arriving on different interfaces.

• The bypass disables certain stateful checks that cause drops in asymmetric routing scenarios but must be used cautiously and selectively due to security trade-offs.

• Be aware the this is an attempt to work around asymmetric routing caused outside of the FTD, and it is best to resolve that factor as a primary step.

Bypass TCP state checks for asymmetrical routing (TCP state bypass): https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/device-config/760/management-center-device-config-76/advanced-access-service-policies.html#ID-2068-00000337

[Video] Bypass TCP state checks for asymmetrical routing (TCP state bypass): https://www.youtube.com/watch?v=v6vJUpPqdYk

Cause

The issue was caused by asymmetric routing external to the FTD firewall. Inbound packets entered the firewall through the outside_primary interface and were correctly forwarded to the destination on dmz_zone_1, but the returning packets from 172.x.x.36 arrived back on the FTD inside interface instead of the expected dmz_zone_1 interface.

Because the return traffic arrived on a different interface than the original flow, the FTD stateful inspection engine treated the packets as invalid and dropped them with the drop reason "tcp-not-syn First TCP packet not SYN". This behavior is expected for stateful TCP inspection when asymmetric routing occurs across multiple interfaces.

Related Content

• Bypass TCP state checks for asymetrical routing (TCP state bypass)

• FTD: How to enable TCP State Bypass Configuration using FlexConfig Policy

• Cisco Technical Support & Downloads