502 / 504 GATEWAY_TIMEOUT errors when browsing to certain sites

If the web server does not respond to the WSA's SYN packets, after a certain amount of attempts, the client will be sent a 502 Gateway Timeout error.

Typical causes for this are:
 1. The web server or web server network is having issues.
 2. A network issue on the WSA network is preventing the SYN packets from getting to the Internet.
 3. A firewall or similar device is dropping either the WSA SYN packets or the web server's SYN/ACK
 4. IP spoofing is enabled on the WSA, but is not properly configured (no return path redirection)

Troubleshooting steps:

The first step is to verify if the WSA can ICMP ping the web server. This can be done by using the following CLI command:

WSA> ping www.example.com

If the ping fails, it does not mean that the server is down. It may mean that ICMP packets are getting blocked somewhere in the path. If the ping succeeds, then we can know for sure that the WSA has a basic layer3 level of connectivity to the web server.

A telnet test will verify if the WSA has the ability to establish a TCP connection on port 80 to the web server. See the instructions further in this article for performing a telnet test.

Network issues or Firewall block

If the ping is successful, but the telnet fails, there is a good possibility that a filtering device, such as a firewall, is preventing this traffic from getting through the network. It is recommended that the firewall logs and/or packet captures from the firewall are analyzed for further details.

IP Spoofing enable, but not properly configured

If explicitly proxying through the WSA or the telnet test is successful, this shows that the WSA can communicate directly to the web server, but when a client proxies through the WSA with IP spoofing, there is a problem.

Without client IP spoofing:

• The WSA sends a SYN to the web server using its own IP address as the source. When the packet comes back, it goes directly to the WSA.

With client IP spoofing:

• The WSA sends the SYN, but instead, uses the client's IP as the source. Without a special network setup, the return packet will be sent to the client instead of the WSA.
• In order to use client IP spoofing, the network must be configured in a very specific way in order to facilitate that the packets are redirected properly. If the web server return path packets are being sent to the client instead of the WSA, the WSA will never see the servers SYN/ACK and will send a 502 Gateway Timeout error back to the client.

 If the WSA receives a TCP reset packet on its upstream connection to the web server, the WSA will send a 504 Gateway Timeout error to the client.
 

Typical causes for this are:
 1. The Cisco Layer 4 Traffic Monitor (L4TM) is blocking the WSA proxy from connecting the web server.
 2. A firewall, IDS, IPS, or other packet inspection device is blocking the WSA. 
 
Troubleshooting steps:

First determine if the TCP RST is coming from the L4TM or from another device.

If the L4TM is blocking this traffic, the traffic will show up in the GUI reports under "Monitor -> L4 Traffic Monitor". Otherwise, the RST is coming from a different device.

L4TM Blocking:

It is recommended that if the L4TM is blocking, do not block on ports that the WSA proxy is also running on. There are multiple reasons for this:

1. The WSA proxy provides a friendly error message in the case of problem, instead of just TCP resetting the connection. This will help limit confusion from the end users when they are blocked.
2. The WSA proxy has the ability to scan and block specific content, whereas the L4TM blocks all traffic matching an blacklisted IP address.

In order to configure the L4TM to not block on proxy ports, go to "GUI -> Security Services -> L4 Traffic Monitor".

If the site is a known bad web site, but there are reasons why the traffic should be allowed, the site can be white listed in:
"GUI -> Web Security Manager -> L4 Traffic Monitor -> Allow List"

Firewall / IDS / IPS Blocking:

If another device on the networking is blocking the WSA from connecting to the web server, it is recommended to analyze the following:

1. Firewall block logs
2. Ingress / Egress packet captures during the problem

The block logs may quickly confirm if the device is blocking the WSA. Sometimes a firewall, IPS, or IDS will block traffic and NOT log it appropriately. If this is the case, the only way to prove where the TCP RST is coming from, is to obtain ingress and egress captures from the device. If a RST is being sent out the ingress interface and no packets traveled through the egress side, the security device is definitely the cause.

If the WSA sends an HTTP GET, but never receives a response, it will send a 504 Gateway Timeout error to the client.

Typical causes for this are:

• A firewall, IDS, IPS, or other packet inspection device is allowing the TCP connection, but blocking the HTTP content from reaching the web server. In this case, the telnet test may help isolate which kind of HTTP data is being blocked.

The firewall block logs may quickly confirm if / why the device is blocking the WSA. Sometimes a firewall, IPS, or IDS will block traffic and NOT log it appropriately. If this is the case, the only way to prove where the TCP RST is coming from, is to obtain ingress and egress captures from the device. If a RST is being sent out the ingress interface and no packets traveled through the egress side, the security device is definitely the cause.

 From the WSA CLI, run the telnet command:

WSA> telnet

Please select which interface you want to telnet from.
1. Auto
2. Management (192.168.15.200/24: wsa.hostname.com)
3. P1 (192.168.113.199/24: data.com)
[1]> 3

Enter the remote hostname or IP address.
[]> www.example.com

Enter the remote port.
[25]> 80

Trying 10.3.2.99...
Connected to www.example.com.
Escape character is '^]'.

Note: The "Connected" message in red, indicates that TCP successfully established between the WSA and web server.

An HTTP request can manually be sent through this telnet session as well. The following is a sample request that can be typed after the "Connected" message:

-------------------------------------------------------------------------------------

GET http://www.example.com HTTP/1.1
HOST: www.example.com
{Enter}
-------------------------------------------------------------------------------------

Note: Make sure to add the extra carriage return at the end, otherwise the server will not respond to the request.