Field Notice: FN - 64111 - Cisco Email Security Appliance (ESA) - Change in URL Reputation Feature Servers Requires Configuration Change in Order to Avoid Work Queue Backups - Configuration Change Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	14-Mar-16	Initial Release
10.0	18-Oct-17	Migration to new field notice system

Products Affected

Affected OS Type	Affected Release	Affected Release Number	Comments
NON-IOS	9	9.1.0,9.6.0,9.7.0,9.7.1
NON-IOS	8	8.5.5,8.5.6,8.5.7

Defect Information

Defect ID	Headline
CSCuy44285	Too many connection resets to SDS servers

Problem Description

A change in the server pool used by the URL Reputation Feature might cause the Email Security Appliance (ESA) to start to back up the work queue. This change in the server pool takes place on April 4th, 2016.

Background

A change in the server pool used by the URL Reputation Feature might cause the ESA to start to back up the work queue. This change in the server pool takes place on April 4th, 2016.

Problem Symptom

If URL filtering is enabled and these behaviors are observed, the ESA experiences the issue and the workaround provided has to be applied.

The Work Queue on the ESA Backs Up

The ESA stores the Web Client logs under the web_client log subscription. If this message is present in this log it is a good indication that the appliance faces the described issue.

[ Unexpected workqueue growth ]

 

A Large Number of 'Request already expired' Messages are in the Web Client Logs

Thu Feb 11 23:00:49 2016 Warning: cache: THR: cache_manager-cloud_connector: SRC: antispam: MID: 99: REQ_ID: 64878.1455231634: Error requesting ?http://www.someURL.com': (Request already expired)

Thu Feb 11 23:00:49 2016 Warning: cache: THR: cache_manager-cloud_connector: SRC: work_queue: MID: 100: REQ_ID: 61581051: Error requesting ?http://www.someWebSitecom': (Request already expired)

Thu Feb 11 23:00:49 2016 Warning: cache: THR: cache_manager-cloud_connector: SRC: work_queue: MID: 100: REQ_ID: 61581051: Error requesting 'http:/wwww.anotherURL.com: (Request already expired)

Thu Feb 11 23:00:49 2016 Warning: cache: THR: cache_manager-cloud_connector: SRC: work_queue: MID: 100: REQ_ID: 61581051: Error requesting ?http://www.funysites.com/funny/URL': (Request already expired)

Thu Feb 11 23:00:49 2016 Info: rpc_server: THR: request_handler_5: SRC: antispam: MID: 100: REQ_ID: 64881.1455231634: URL lookup success = False; whitelisted = 0, cache hits = 0, cache misses = 2, ignored or errored= 2, time taken = 7.029145 sec

 

Alerts Similar to These Examples are Sent by the Appliance

The Warning message is:

Unable to connect to Cisco Web Security Service.

URL Filtering will not work correctly.

Please verify all network, proxy and firewall settings.

Connection to "v2.sds.cisco.com" failed.

The last error seen on this connection: "Server busy or service unavailable"

Version: 9.7.0-125

Serial Number: 8xxxxxxx.9yyyy

Timestamp: 11 Feb 2016 23:00:03 +0000

 

or

 

The Warning message is:

Unable to connect to Cisco Web Security Service.

URL Filtering will not work correctly.

Please verify all network, proxy and firewall settings.

Connection to "v2.sds.cisco.com" failed.

The last error seen on this connection: "Request failed with code: 28 (SSL_write() error: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry)"

Unable to connect to Cisco Web Security Service.

URL Filtering will not work correctly.

Please verify all network, proxy and firewall settings.

Connection to "v2.sds.cisco.com" failed.

The last error seen on this connection: "Unknown error"

Last message occurred 46 times between Fri Feb 12 22:13:54 2016 and Fri Feb 12 22:16:27 2016.

Version: 9.7.0-125

Serial Number: 8xxxxxxx.9yyyy

Timestamp: 12 Feb 2016 23:07:14 +0000

Workaround/Solution

As part of this migration, the OpenSSL software will be patched in order to improve the pool's security posture. A bug has been uncovered between the software used by the ESA to talk to these servers and the OpenSSL patch.

In order to avoid the workqueue backups, you can reduce the amount of URLs sent for verification at the same time. Complete these steps in order to make this change:

1. Secure Shell (SSH) into the appliance.

2. Enter the command websecurityadvancedconfig.

3. Change the value for "Enter the threshold value for outstanding requests" from the default to 5.

4. Do not change any other option.

5. Commit the change.

 

ESA>websecurityadvancedconfig

Enter URL lookup timeout (includes any DNS lookup time) in seconds:

[5]>

Enter the URL cache size (no. of URLs):

[810000]>

Do you want to disable DNS lookups? [N]>

Enter the maximum number of URLs that should be scanned:

[100]>

Enter the Web security service hostname:

[v2.sds.cisco.com]>

 

Enter the threshold value for outstanding requests:

[50]>5

Do you want to verify server certificate? [Y]>

Enter the default time-to-live value (seconds):

[30]>

Do you want to include additional headers? [N]>

Enter the default debug log level for RPC server:

[Info]>

Enter the default debug log level for URL cache:

[Info]>

Enter the default debug log level for HTTP client:

[Info]>

ESA> commit

Please enter some comments describing your changes:

[]> updated the threshold value for outstanding requests to 5

Do you want to save the current configuration for rollback? [Y]>

Changes committed: Thu Mar 03 18:40:57 2016 GMT

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email
• By telephone

Receive Email Notification For New Field Notices

Cisco Notification Service—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.