FWSM New Connections Drop Intermittently after an Upgrade to Release 4.1.11 or Later

Introduction

This document describes a specific problem with intermittent traffic drops on the Firewall Services Module (FWSM) after a software upgrade to Release 4.1.11 or later.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on the FWSM with Software Release 4.1(11) or later.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information

As per FWSM behavior, if you use TCP as the logging transport protocol in order to send messages to a syslog server, it denies new connections as a security measure if the FWSM is unable to reach the syslog server. You can use the logging permit-hostdown command in order to remove this restriction.

Problem

There is an intermittent traffic drop issue on the FWSM after an upgrade to Release 4.1.11 or later. The FWSM starts to deny all new connections.

The most noticeable traffic drop is for Internet Control Message Protocol (ICMP), since each ICMP Echo request is treated as a new connection. Connectivity is restored once the TCP connection to the syslog server is successful.

For FWSM Release 4.1.11 or later, if the TCP-based syslog server is not reachable even with "permit-hostdown" policy, the FWSM denies all new connections. The "logging permit-hostdown" feature no longer works after a FWSM upgrade to Release 4.1.11 or later.

The FWSM continues to reconnect to the TCP syslog server every minute till the time server is up. Thus, a single TCP handshake failure results in a minimum one minute outage for all new connections, because the FWSM tries to contact the TCP syslog server again, only after one minute.

Conditions

• The FWSM runs Release 4.1.11 or later.
• The FWSM should be in single mode.
• The TCP syslog server must be unreachable from the FWSM.

Verify

In order to identify this behavior, check the slow path (NP3) statistics. The Deny Conns (Conn State) counter increases if the TCP-based syslog server is not reachable, even with "permit-hostdown" policy.

pri/act# show clock
09:31:55.070 GMT Thu May 15 2014

pri/act# show np3 stats | ex : 0
<<NP 3 stats>>
  Discard Statistics
  ------------------

  Egress Discards          : 34412
    ACL Denied Packets     : 157
    Rev Route Lkup Fail    : 202
    Self Route Packets     : 40
    Deny Conns (Conn State): 34013 <------Counter to monitor
    
pri/act# show clock
09:32:06.020 GMT Thu May 15 2014

pri/act# show np3 stats | ex : 0
<<NP 3 stats>>
  Discard Statistics
  ------------------

  Egress Discards          : 46634
    ACL Denied Packets     : 157
    Rev Route Lkup Fail    : 202
    Self Route Packets     : 40
    Deny Conns (Conn State): 46235 <------Counter seen increasing

Solution

A defect was filed to track this issue, but it will not be fixed since the FWSM has reached the end of the Software Maintenance release date.

End-of-Sale and End-of-Life Announcement for Firewall Services Modules

In order to fix this issue, change the logging server configuration to UDP transport.

logging host inside 192.x.x.x 17/5514

Related Information

• Command Reference : logging permit-hostdown
• Command Reference : logging host
• Technical Support &  Documentation - Cisco Systems