Document ID: 115010
Updated: Nov 02, 2012
Contributed by Michael Robertson and Jay Johnston, Cisco TAC Engineers.
Due to the design of the Firewall Services Module's (FWSM) packet processing, xlates built by incorrectly routed packets can cause traffic failures for connections through the firewall. In order to select an egress interface for an inbound packet, the FWSM first checks to see if the destination IP of the inbound packet matches any existing global IP/Network in a NAT translation (xlate) for that interface in its xlate table. If a match is found, the egress interface is simply chosen based on the local interface in the xlate entry and the firewall does not consult the routing table to make the egress interface decision.
The default behavior of the FWSM is to build an xlate entry for the source IP of any permitted packet that is received on one of its interfaces. If a packet is routed through the network incorrectly (for any number of reasons) and arrives inbound on the wrong interface of the FWSM, an xlate is built to reflect this. When this occurs, entries in the xlate table can override entries in the routing table and cause traffic failures for the affected destinations.
This document describes the symptoms and triggers for this issue, how to diagnose it, and provides solutions for preventing it from occurring.
Cisco recommends that you have knowledge of FWSMs.
This document is not restricted to specific software and hardware versions.
Refer to Cisco Technical Tips Conventions for more information on document conventions.
interface Vlan1 nameif outside security-level 0 ip address 192.168.100.50 255.255.255.0 ! interface Vlan10 nameif inside security-level 100 ip address 10.10.1.50 255.255.255.0 ! interface Vlan20 nameif dmz security-level 50 ip address 10.20.1.50 255.255.255.0 ! same-security-traffic permit intra-interface access-list outside_in extended permit tcp any host 10.30.1.1 eq www access-list inside_in extended permit ip any any access-group inside_in in interface inside access-group outside_in in interface outside route outside 0.0.0.0 0.0.0.0 192.168.100.254 route dmz 10.30.1.0 255.255.255.0 10.20.1.254
Connections from the client PC at 172.16.1.10 to the web server at 10.30.1.1 fail.
A packet capture on the outside interface shows a TCP SYN from the client PC arriving at the FWSM's interface.
FWSM# show capture outside 3 packets seen, 3 packets captured 1: 13:58:09.280752960 802.1Q vlan#1 P0 172.16.1.10.57389 > 10.30.1.1.80: S 918518428:918518428(0) win 8192 <mss 1380,nop,nop,sackOK> 2: 13:58:12.280755950 802.1Q vlan#1 P0 172.16.1.10.57389 > 10.30.1.1.80: S 918518428:918518428(0) win 8192 <mss 1380,nop,nop,sackOK> 3: 13:58:18.280761960 802.1Q vlan#1 P0 172.16.1.10.57389 > 10.30.1.1.80: S 918518428:918518428(0) win 8192 <mss 1380,nop,nop,sackOK> 3 packets shown
A packet capture on the dmz interface does not show that packet leaving the firewall.
FWSM# show capture dmz 0 packet seen, 0 packet captured 0 packet shown
No entry is built in the FWSM's connection table and syslogs do not show any information related to the client or server IP addresses.
At a fundamental level, this issue is caused by an entry in the FWSM's xlate table that was built by an incorrectly routed packet. Because of the way the FWSM's packet processing is designed, the firewall checks the xlate table before it checks the routing table to determine the egress interface. As a result, if a packet matches an existing xlate the egress interface will be selected based on that entry, even if the entry conflicts with what is listed in the routing table. In other words, the xlate table takes precedence over the routing table.
In order to diagnose this issue, check the output of the show xlate debug command:
FWSM# show xlate debug Flags: D - DNS, d - dump, I - identity, i - inside, n - no random, o - outside, r - portmap, s - static 3 in use, 3 most used NAT from inside:10.30.1.1 to outside:10.30.1.1 flags Ii idle 0:00:00 timeout 3:00:00 connections 0 NAT from inside:10.30.1.1 to inside:10.30.1.1 flags Ii idle 0:00:07 timeout 3:00:00 connections 0 NAT from dmz:10.30.1.1 to outside:10.30.1.1 flags Ii idle 0:00:10 timeout 3:00:00 connections 0
Note: The debug keyword in the show xlate is crucial. Without it, the xlate entries will not include the interface names that the entry is associated with.
The xlate table shows that there are 3 xlates built for the web server. The first xlate is built between the inside interface and the outside interface. The second xlate is built as a hairpinned or u-turned xlate on the inside interface. The third xlate is built between the dmz and the outside interface. The I flag indicates that this is an identity xlate and the IP is not actually being translated.
The first interface listed in the entry is the "real" or "local" interface where the IP is supposed to actually exist. The second interface listed is the "mapped" or "global" interface where the IP is being translated. Neither of these xlates shown are correct. This is because the web server (10.30.1.1) actually exists behind the dmz interface. The third xlate is correct for this network design.
The connection failure occurs because of the first xlate listed in the table. When the client's TCP SYN packet arrives on the outside interface destined to 10.30.1.1, the FWSM checks the xlate table and matches the first entry. This entry indicates that the packet should egress on the inside interface, which is incorrect, and the packet is blackholed.
By default, the FWSM will automatically build an identity xlate for any traffic that does not match an explicitly configured NAT rule. Because of this, even if a packet erroneously arrives on an incorrect interface, an xlate will be built. Specifically for this case, packets sourced from 10.30.1.1 arrived inbound on the inside interface instead of arriving on the dmz interface as is expected.
The first xlate (inside > outside) was built when the web server tried to ping a non-existent IP address (10.199.199.1). The echo request left the web server destined to its default gateway (the DMZ router). The DMZ router forwarded the packet toward the inside router, per its static route:
S 10.0.0.0/8 [1/0] via 10.50.1.254
Because the 10.199.199.0/24 network does not actually exist anywhere, the inside router simply follows its default route and sends the packet to the FWSM's inside interface:
S* 0.0.0.0/0 [1/0] via 10.20.1.50
Likewise, the FWSM also does not have a route for the destination network. Therefore, it selects the outside interface as the egress interface and builds an identity xlate from inside > outside:
S 0.0.0.0 0.0.0.0 [1/0] via 192.168.100.254, outside
The second xlate (inside > inside) was built when the web server tried to access the DNS server while the inside router's 10.40.1.254 interface was temporarily down due to a link flap. The DNS request left the web server destined to its default gateway (the DMZ router). The DMZ router forwarded the packet toward the inside router, per its static route:
S 10.0.0.0/8 [1/0] via 10.50.1.254
However, the inside router's interface connected to the 10.40.1.0/24 network was temporarily down and its directly connected route for this network was missing. Therefore, the only matching route in the routing table was the default route back toward the FWSM:
S* 0.0.0.0/0 [1/0] via 10.20.1.50
The packet was routed to the FWSM's inside interface. The FWSM's routing table indicated that the destination network of 10.40.1.0/24 existed behind the same inside interface:
S 10.40.1.0 255.255.255.0 [1/0] via 10.10.1.254, inside
Because the same-security-traffic permit intra-interface command is enabled, the FWSM will allow the u-turned xlate to be built.
To summarize, the first xlate was triggered by:
A broad 10.0.0.0/8 route configured on the DMZ router
A permit ip any any ACL configured on the FWSM's inside interface
The second xlate was triggered by:
A flapping interface on the Inside router
same-security-traffic permit intra-interface configured on the FWSM
There are many different possible solutions to this problem. First and foremost, deleting the xlate from the table should allow traffic to start working again until the xlate is rebuilt. This can be done with the clear xlate command. For example:
FWSM# clear xlate interface inside local 10.30.1.1 global 10.30.1.1
Note: Any connections that are using the deleted xlate(s) will also be torn down.
Once that is complete, the focus should be on preventing the xlates from returning. Often times, the most preferred way to do this is to fix the routing configuration in the environment to prevent traffic from arriving on the wrong FWSM interface. The FWSM also offers a handful of configuration options to address these issues.
This solution takes careful planning and a deep understanding of the network environment. In the first example above, the 10.0.0.0/8 route on the DMZ router is technically incorrect since the entire /8 network does not exist beyond its 10.50.1.253 interface. Instead, some options that exist are:
Eliminate the 10.50.1.0/24 network all together and simply route all traffic through the FWSM. This also provides better segmentation and security between the Inside and DMZ networks.
Configure a static route on the DMZ for only 10.40.1.0/24 and remove the 10.0.0.0/8 route.
Use a dynamic routing protocol between the Inside and DMZ routers to correctly advertise only the networks that actually exist.
There are often many possibilities for adjusting the routing configuration, but the end goal is to ensure that traffic from a given host is able to arrive only on a single FWSM interface.
The same-security-traffic permit intra-interface command allows the FWSM to u-turn or hairpin traffic on an interface. This means that a packet can enter the firewall on the same interface it leaves on. This functionality is disabled by default and has very little use in most FWSM designs. Because the FWSM uses VLAN interfaces, traffic that stays within the same VLAN should never be processed by the FWSM.
In the second example above, the same-security-traffic permit intra-interface command allowed a packet to both enter and leave the inside interface. Disabling same-security-traffic permit intra-interface would prevent this behavior and drop the packet before an xlate was ever built:
FWSM(config)# no same-security-traffic permit intra-interface
In both examples above, the xlates were built when a packet from the web server incorrectly arrived on the inside interface. In order to prevent the problem all together, the FWSM can be configured to drop packets that arrive on the wrong interface.
The FWSM requires that all traffic be permitted by an ACL before it can pass. Therefore, this functionality can be achieved by only permitting traffic from appropriate source networks on each interface. In the examples above, the inside interface permits all IP traffic:
access-list inside_in extended permit ip any any
Instead, this should be changed to only permit traffic from the 10.10.1.0/24 and 10.40.1.0/24 subnets:
access-list inside_in extended permit ip 10.10.1.0 255.255.255.0 any access-list inside_in extended permit ip 10.40.1.0 255.255.255.0 any
In some environments, this is not a feasible option due to the size and/or scale of the different networks passing through the FWSM. However, this functionality can be achieved more simply using a feature called Unicast Reverse Path Forwarding (uRPF).
When the uRPF feature is enabled, the FWSM will compare the source IP address of the first packet of every connection against its routing table. If the route that is found does not match up with the interface that the packet arrived on, that packet will be dropped due to a RPF failure.
In the example above, the FWSM has a static route that uses the dmz interface to reach the 10.30.1.0/24 network. Therefore, if uRPF is enabled on the inside interface, packets sourced from the web server (10.30.1.1) that arrive incorrectly on the inside interface will be dropped.
In order to enable uRPF, apply the ip verify reverse-path command to each interface in question. For example:
FWSM(config)# ip verify reverse-path interface inside
In both of the examples above, the xlates are created with the Ii flags. These flags indicate that the xlate is an identity translation (I) that originated on a high security (i) interface. By default, the FWSM will build these xlates for any traffic that does not match an explicit NAT/PAT rule. In order to disable this behavior, the xlate-bypass command can be enabled in FWSM 3.2(1) and later:
This feature will prevent the FWSM from building identity xlates in the first place. Thus, the traffic in the examples above would not be redirected to an incorrect interface due to an xlate table entry. However, the traffic will still pass through the FWSM untranslated.
In order to determine the egress interface for a packet, the FWSM will always consult its xlate table before looking at its routing table. If that packet matches an existing xlate, the egress interface is selected based on the xlate's associated interface. This happens regardless of any contradictions that might be found in the routing table. In this way, the xlate table takes precedence over the routing table.
Because the FWSM will always build an xlate entry for all new connections by default, this can cause traffic failures in cases where incorrectly routed packets cause the FWSM to build an xlate. As outlined above, there are many possible scenarios where this can occur but all relate back to a packet being received on an incorrect interface. This document covered these possible issues:
A broad routing config sends packets in an incorrect direction
The FWSM is configured to permit traffic from incorrect source networks
The FWSM is configured to hairpin/u-turn traffic
In order to quickly restore connectivity for connections that fail due to a wrong xlate, delete the entry with the clear xlate command. This document also covered multiple solutions for preventing these xlates from returning in the future, including:
Resolve incorrect routing configurations using more specific routes
Disable same-security-traffic permit intra-interface
Drop packets that arrive on an incorrect interface using ACLs or uRPF
The Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers.
Refer to Cisco Technical Tips Conventions for information on conventions used in this document.