This document describes a problem where a mobility path connection (using User Datagram Protocol (UDP) and IP protocol 93) that traverses an Adaptive Security Appliance (ASA) might go down and continue to fail until the mobility devices are reloaded, or the mobility path traffic is stopped and left inactive for a short time and then restarted.
Cisco recommends that you have knowledge of these topics:
Cisco Adaptive Security Appliance (ASA)
Wireless LAN Controller (WLC)
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
The problem is triggered when the WLC at 10.10.1.2 sends traffic destined to the WLC at 10.10.9.3. These packets cause the ASA to build a connection in its connection table that sends the mobility traffic out the wrong ASA interface (inside).
This issue is caused by the destination interface "dmz" of the ASA being in the down/down state at the time the connection was built, which results in the connection being built out a different, non-optimal interface. The dmz interface might be down due to a cable problem, an ethernet or port-channel negotiation issue, or it might be administratively shut down.
At the time of the problem, the mobility path connections can be seen as being created as "intra-interface" of the ASA, which is routing the packets back out the same inside interface they arrived on:
The mobility endpoint at 10.10.1.2 continues to send traffic destined to 10.10.9.3, which matches these existing connections. Even if the dmz interface were to progress to the up/up state, mobility traffic sourced from 10.10.1.2 would match the existing connections in the table (instead of building a new connection to the dmz interface) which resets the timeout of the connections on the ASA, which prolongs the problem.
In summary, these events can trigger the issue:
The device at 10.10.1.2 sends a protocol 97 or UDP packet to 10.10.9.3.
The ASA receives the packet on the inside interface, but the dmz interface is down, which results in the more specific route to the destination network missing from the routing table. Since the same-security permit intra-interface command is enabled on the ASA, it follows a static route configured for the 10.0.0.0/8 network back through the inside interface, builds a connection in the connection table, and then sends the packet back out the inside interface towards the internal network.
At some point the dmz interface might come back up and the route is added back to the table; however, since the connection for the protocol 97 traffic was already built in step #2, subsequent packets will match the connection and the routing table is overwritten, and the traffic does not reach the server on the dmz.
One possible solution for this issue is to remove the same-security permit intra-interface command from the ASA. This solution prevents the u-turn connection from being built back out the same interface on which the original packet was received, which allows the correct connection to be built when the interface comes up. However, depending on the routing table of the ASA, this solution might not work (the traffic might be routed to another interface other than the intended destination based on the routing table), and the same-security permit intra-interface command might be necessary for other connections on the ASA.
For this specific instance, the problem was successfully mitigated by enabling the timeout floating-conn feature. This feature, which is not enabled by default, caused the ASA to tear down these connections one minute after a more preferred route to one of the endpoints is added to the routing table out a new interface of the ASA, which occurs when the dmz interface comes up. The connections are then immediately rebuilt when the next packet arrives at the ASA, using the more preferred interface (dmz, instead of inside for the 10.10.9.3 host).
ASA(config)# timeout floating-conn 0:01:00
When the problem is mitigated, the correct connections are built in the ASA connection table and connectivity is automatically restored:
ASA# show conn address 10.10.1.2 15329 in use, 133142 most used 97 dmz 10.10.9.3 inside10.10.1.2, idle 0:00:00, bytes 3175742510 UDP dmz 10.10.9.3:16666 inside 10.10.1.2:16666, idle 0:00:00, bytes 40651338, flags - 97 dmz 10.10.9.3 inside10.10.1.2, idle 0:00:00, bytes 1593457240 ASA#