Preventing LocalDirector from Failing Servers [Cisco LocalDirector 400 Series]

Document ID: 15061

Cisco has announced the end-of-sale for the Cisco LocalDirector. For more information, refer to the LocalDirector 400 Series End-of-Life and End-of-Sale Notices and Product Bulletins.

Introduction
Causes of Failed Servers

Server Fails to Respond to New Connections
Server Responds to Connections with TCP Resets
Using the data Command
No Answer Messages

Preventing LocalDirector from Failing Servers
Related Information

Introduction

LocalDirector can fail servers for the following reasons:

Server does not respond to new connections.
Server responds to connections with Transmission Control Protocol (TCP) resets.
Use of the data command in the LocalDirector configuration.
No answer message is received from the real server.
LocalDirector does not see a reply from the server to a TCP SYN (NO ANSWER).
LocalDirector receives a TCP RST from the server (RST REASSIGN).

To determine what causes LocalDirector real servers to fail, you must diagnose the problem. Use the show real command and you will see output similar to the following:

LocalDirector (config)# show real

  Real Machines:                                                  
     Machine             Connect  State  Thresh   No Answ Reassigns    TCP Reset Reassigns   DataIn Conns 
     server2:0:0:tcp       0      Failed    8           8                      0                  0 
     server1:0:0:tcp       0      Failed    8           0                      8                  0 
     Machine - IP address or name of the server, port(if a port-bound server), bind_id, and protocol.
     Connect - The current number of connections to the server. This does not include direct connections 
	 to the server that are bridged by the LocalDirector.
     State - IS(in service), OOS(out-of-service), failed or testing.
     Thresh - Threshold value for reassignments before server is marked as failed. Default 8.
     No Answer Reassigns - Number of connections that are not answered by a real server.
     TCP Reset Reassigns - Number of connections that are reassigned because a
     real server responded with an RST packet on a new connection.
     DataIn Conns - Number of clients requesting but not receiving data.

Causes of Failed Servers

In a basic environment, LocalDirector can declare a server failed because of a variety of reasons, all of which are server problems.

Server Fails to Respond to New Connections

If a server does not respond to new connections, No Answer Reassign counters start increasing.

Server Responds to Connections with TCP Resets

If the server responds with a TCP reset, the TCP Reset Reassign value starts going up. Once either of the previous values reaches the threshold—the default is 8—LocalDirector declares that a server has failed. LocalDirector then tries to bring the server back into service state by directing live connections after a retry. The default is one minute. To specify the number of minutes before a failed server receives a live connection to check its state, use the retry command.

Using the `data` Command

Using the data command in the LocalDirector configuration also can cause server failure. To limit the number of connections to a server that has an open connection to a client, but which is not sending data in response to a request, use the data command. The data command waits until after the TCP handshake has taken place and a request from the client comes through, for example a HTTP GET. The data command makes sure the server responds with data that fulfills the request instead of timing out.

If the server fails to respond to a client's request for data after the TCP handshake, the data counter will increment by 1. Once the data counter has reached the number that is specified in the LocalDirector config, the server will be marked as failed because of an application failure (Real machine 'x' Failed Application). Investigate why the server is sending TCP resets or not answering to connections.

If you want LocalDirector to take longer before reporting that a server has failed, use the threshold command to increase the threshold value.

To set the maximum number of connections that LocalDirector sends to a real server, use the maxconns command. To remove a maxconns value, use the no maxconns command. The default value for the maxconns command is 0, or unlimited connections.

You must first bind this virtual server to a real server. The maxconns command action applies to all real servers bound to this virtual server. The default is no limitation, so the maximum is limited only by available resources. You can, however, set a limit to avoid overloading your servers. You can disable the server state testing by setting the reassign threshold to 0, which effectively means unlimited. The server is never taken out of service automatically, only the timeouts. TCP resets increment the counters.

No Answer Message

A server also can fail if a no answer message is seen from the real server after a TCP SYN packet from the client has been assigned. This can occur two ways:

The real server is down or unable to respond.
The real server's response is circumventing the LocalDirector on the path back to the client. LocalDirector does not see the reply and cannot use Network Address Translation (NAT) to return the packet back to the virtual address.

To increase the rate at which servers are considered failed, adjust the failure counter by reducing the threshold and reassign values.