Guest

Cisco Services Modules

Field Notice: FN - 61935 - Catalyst 6500 Series and 7600 Series Service Module Incompatibility With Distributed EtherChannel and Packet Re-Circulation


Revised January 29, 2007

March 14, 2005


Products Affected

Product

Comments

HW Rev

WS-SVC-ADM-1-K9

Anomaly Detection Module

all

WS-SVC-AGM-1-K9

Anomaly Guard Module

all

WS-SVC-FWM-1-K9

Firewall Services Module

All

WS-SVC-IDS2-BUN-K9

Intrusion Detection Service Module 2

All

WS-SVC-MWAM-1

Multi-Processor WAN Application Module

All

WS-SVC-NAM-1

Network Analysis Module 1

All

WS-SVC-NAM-2

Network Analysis Module 2

All

WS-SVC-PSD-1

Persistent Storage Device Module

All

WS-SVC-SSL-1-K9

SSL Module

All

WS-SVC-WLAN-1-K9

Wireless LAN Service Module

All

Problem Description

When the listed service modules transmit packets to VLANs also used with Distributed EtherChannel (DEC), those packets may be dropped and lost.

Background

The listed service modules do not support packet re-circulation. Packet re-circulation is a specific means to forward packets internal to the chassis between modules. When the service module attempts to forward a packet onto such a VLAN with packet-recirculation enabled, it may fail and the packet might be lost. When packet re-circulation is not enabled on the destination VLAN, even if DEC is present, this problem will not be encountered.

This failure only occurs when the packet being forwarded is about to be routed/L3-switched to another VLAN within the chassis immediately after leaving the service module. Packets being L2 forwarded from the service module directly to internal or external destinations on the same VLAN/network as the service module's egress virtual interface will not be dropped even if packet re-circulation is enabled on that VLAN.

Note that packets are not dropped as they are transmitted to the service module. If a service module only receives packets from a given VLAN, for example - an IDS sensing interface, this traffic will not be at risk even if the VLAN has packet re-circulation enabled.

There are two scenarios in which packet re-circulation will be enabled on a VLAN. In both scenarios the VLAN must be associated with interface ports in the chassis which are part of a DEC logical grouping. For the purpose of this field notice, distribution of EtherChannel ports across any kind of interface line module, fabric-enabled or not, is considered DEC.

Scenario 1: DEC with DFC3 line module

The first scenario requires that a Distributed Forwarding Card (DFC) 3 or later be installed in the system. Any VLANs associated with DEC ports on an interface module with a DFC3 or later will have packet re-circulation enabled. Note that DFCs are only supported in native IOS systems.

The following table notes which DFC cards do and do not enable packet re-circulation on VLANs with DEC.

DFC Model

Product ID

Enables Packet Re-Circulation with DEC

DFC

WS-F6K-DFC

No

DFC 3A

WS-F6K-DFC3A

WS-F6700-DFC3A

Yes

DFC 3B

WS-F6K-DFC3B

WS-F6700-DFC3B

Yes

DFC 3BXL

WS-F6K-DFC3BXL

WS-F6700-DFC3BXL

Yes

Scenario 2: DEC with non fabric-enabled and non fabric-enabled line modules

Non fabric-enabled line modules communicate with fabric-enabled modules via the supervisor in flow through mode. Any VLANs associated with DEC ports distributed across both fabric-enabled and non-fabric enabled line modules operating in flow through mode will have packet re-circulation enabled. This condition can be met even if there is no DFC in the system and can occur in both native IOS and CatOS systems.

The following table notes which series of interface line cards are and are not fabric enabled.

Interface Module Series

Example Product IDs

Fabric Enabled

Enables Packet Re-Circulation with DEC

6000

WS-X6024-10FL-MT

No

Yes

6100

WS-X6148-GE-TX

No

Yes

6200

WS-X6224-100FX-MT

No

Yes

6300

WS-X6348-RJ21V

No

Yes

6400

WS-X6408A-GBIC

No

Yes

6500

WS-X6548-GE-TX

Yes

No

6700

WS-X6748-GE-TX

Yes

No

6800

WS-X6816-GBIC

Yes

No

Note: A redundant supervisor is considered a fabric-enabled module for the purpose of the second failure scenario. If DEC is configured spanning non fabric-enabled line modules and a redundant supervisor module, then packet re-circulation will occur.

Note: The CSM service module (WS-X6066-SLB-APC) is a non fabric-enabled line module which associates with every VLAN in the system. If it is installed in conjunction with fabric-enabled line modules, then every VLAN will have packet re-circulation enabled.

Note: The line modules involved in setting up the conditions for this problem do not experience the failure themselves. Only the service modules listed in the affected products section experience the failure once the necessary conditions are met.

Problem Symptoms

Packets will flow normally through the system and into the service module. The service module will act normally upon the packets according to its application, such as firewall, NAM, IDS, and others. However, if a packet is forwarded out of the service module onto a VLAN with packet re-circulation, it may be dropped without reaching its next destination module.

The same holds true for packets generated by the service module for management or monitoring traffic, such as SNMP traps, IDS alarms, and others. Any such traffic which is generated by the service module and transmitted out onto a VLAN with packet re-circulation may be dropped.

There is no error message or other warning when packets are dropped in this manner. The service module forwarding the packet will give all indications that it is acting out its functions properly, but the packet will not reach the destination module.

Here are six sample scenarios to illustrate how this failure may or may not be encountered. Note that while specific service modules are listed, the failure scenarios will be encountered with any listed service module when the same use scenarios are in effect.

Scenario 1: Traffic passing through Firewall Service Module to be routed directly out of the chassis.

Network traffic enters the chassis on VLAN 100 and is forwarded to the FWSM. The FWSM permits the traffic to route through it and it is forward onto VLAN 101 which has DEC and packet re-circulation enabled. The traffic then is L2 forwarded directly out of the chassis to an external network device via a port associated to VLAN 101. Under this scenario, this problem is not encountered and the packets are forwarded properly because no L3 routing/switching follows the packets forwarding to the VLAN with DEC and packet re-circulation.

Scenario 2: Traffic passing through Firewall Service Module to be routed to another VLAN/network before leaving the chassis.

Network traffic enters the chassis on VLAN 100 and is forwarded to the FWSM. The FWSM permits the traffic to route through it and it is forward onto VLAN 101 which has DEC and packet re-circulation enabled. The traffic is then supposed to be L3 switched to VLAN 102 before being forwarded out of the chassis to an external network device via a port associated to VLAN 102. Under this scenario, this problem is encountered and the packets are dropped by the service module because an L3 route/switch event directly follows the packet forwarding to the VLAN with DEC and packet re-circulation.

Scenario 3: Intrusion Detection Service Module alarm generation to be forwarded directly to external monitoring host.

An IDSM is monitoring network traffic on VLAN 100 for malicious behavior. It detects a known network attack and generates an alarm packet on its management VLAN 101 which has DEC and packet re-circulation enabled. The destination monitoring host is directly connected to a port associated to VLAN 101 and the packet is L2 forwarded out of the chassis. Under this scenario, this problem is not encountered and the alarm packets are forwarded properly because no L3 routing/switching follows the packets forwarding to the VLAN with DEC and packet re-circulation.

Scenario 4: Intrusion Detection Service Module alarm generation to be routed to another VLAN/network before reaching external monitoring host

An IDSM is monitoring network traffic on VLAN 100 for malicious behavior. It detects a known network attack and generates an alarm packet on its management VLAN 101 which has DEC and packet re-circulation enabled. The destination monitoring host is connected to a port associated to VLAN 102 and the packet is supposed to be L3 switched to VLAN 102 before being forwarded out of the chassis. Under this scenario, this problem is encountered and the alarm packets are dropped by the service module because an L3 route/switch event directly follows the packet forwarding to the VLAN with DEC and packet re-circulation.

Scenario 5: Network Analysis Module management traffic to be routed directly out of the chassis to an external management host

An administrator attempts to logon to the NAM management interface via a management host directly connected to a port on VLAN 100 which has DEC and packet re-circulation enabled. Management traffic (telnet or NAM Web GUI) enters the chassis on VLAN 100 and is forwarded directly to the NAM management interface. The return management traffic is then L2 forwarded directly back out of the chassis to the external host on VLAN 100. Under this scenario, this problem is not encountered and the packets are forwarded properly because no L3 routing/switching follows the packets forwarding to the VLAN with DEC and packet re-circulation.

Scenario 6: Network Analysis Module management traffic to be routed to a second VLAN/network before reaching and external management host

An administrator attempts to logon to the NAM management interface via a management host directly connected to a port on VLAN 100. Management traffic (telnet or NAM Web GUI) enters the chassis on VLAN 100 and is L3 switched to VLAN 101 which has DEC and packet re-circulation enabled before being forwarded to the NAM management interface. The return management traffic is then supposed to be L3 switched back to VLAN 100 before being forwarded out of the chassis to the external host. Under this scenario, this problem is encountered and the management traffic is dropped by the service module because an L3 route/switch event directly follows the packet forwarding to the VLAN with DEC and packet re-circulation.

Workaround/Solution

The recommended solution for native IOS systems is to upgrade to release 12.2(18)SXF7 or later which will remove the requirement to run the traffic in bus mode.

Customers who have not upgraded to release 12.2(18)SXF7 but who are running at least 12.2(17d)SXB7 or 12.2(18)SXE1 must implement the fabric switching-mode force busmode command as described below.

A new IOS command, fabric switching-mode force busmode , is implemented in releases 12.2(17d)SXB7 and 12.2(18)SXE1. This command forces all affected service modules to communicate via the chassis shared bus instead of the switched fabric. Communicating over the shared bus forces the supervisor to handle the packet re-circulation allowing the service modules to communicate properly on VLANs meeting the conditions listed above. Fabric enabled modules which are not affected by this problem will continue to communicate via the switch fabric even if this command is enabled.

After issuing the fabric switching-mode force busmode command or the no variant, all affected fabric-enabled service modules power cycle immediately. The mode change occurs as the modules come up after the power cycle.

Note: The shared bus is capable of handling several times the maximum throughput of any single service module. If more than two or three service modules are configured to communicate over the shared bus and they are being pushed to their performance limit, then testing should be performed to insure that they can meet their performance requirements. If the aggregate sustained service module performance is being limited by the shared bus, then one of the following workarounds should be considered instead.

The following workarounds are alternatives for CatOS systems and for native IOS systems which do not support the fabric switching-mode force busmode command.

Workaround 1: Use single-module EtherChannel instead of DEC.

The first workaround is to replace all DEC port groups used in VLANs directly connected to the service module with standard single-module EtherChannel groups. Doing so will eliminate the packet re-circulation on the selected VLANs allowing the service module to properly forward packets. This option has minimal impact to the network architecture but eliminates the high availability gains achieved by using DEC to span the EtherChannel over multiple interface modules. Note that DEC port groups which do not carry VLANs directly connected to the service module do not need to be converted to single EtherChannel.

Workaround 2: Remove DFC3 cards from the system.

The second workaround is to remove the DFC3 cards from the system if they are the cause of the packet re-circulation. Doing so will eliminate the packet re-circulation on all VLANs allowing the service module to properly forward packets. This has minimal impact to the network architecture and retains the high availability gains achieved by using DEC. However, it will lead to performance degradation for the whole system, which must be carefully considered.

Workaround 3: Replace non fabric-enabled line modules with fabric-enabled line modules.

The third workaround is to replace any non-fabric enabled line modules if they are the cause of the packet re-circulation with fabric-enabled line modules (without DFC3s). This has minimal impact to the network architecture but requires a replacement of one or more line modules and re-configuration of all their ports.

Note: If there are both DFC cards and non-fabric enabled line modules utilized in conjunction with DEC, then both workarounds 3 and 4 must be followed in order to eliminate the packet re-circulation.

Workaround 4: Move the service module default gateway and any other destination gateways external to the 6500/7600 chassis.

By moving all the service module destination gateways external to the chassis, it is insured that all traffic will be L2 forwarded out of the service modules directly to an external port. Doing so will insure that no L3 route/switch events follow the service module's transmission of packets. This may or may not have significant network architecture implications, but will retain the high availability gains of DEC as well as the full performance of the DFC cards.

Workaround 5: Place the service module in a separate chassis.

The fifth workaround is to remove the service module(s) from the systems and place them into another separate chassis/system. This may be connected to the original system via single EtherChannel or DEC if the conditions listed above do not exist in the new system. This may have significant impact to the network architecture and requires additional hardware, however it preserves the DEC and performance of the original system.

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered user and you must be logged in.

DDTS

Description

CSCee10005 (registered customers only)

Service module connectivity issues with DEC

How To Identify Hardware

Issue the show module command in IOS or CatOS to identify what modules and sub-modules are installed in the system.

The example below shows a system with a Firewall Service Module (WS-SVC-FWM-1) installed in slot 7, as well as line cards with Distributed Forwarding Card 3BXL sub-modules in slots 4 and 9. Slot 1 contains a non fabric-enabled 6100 series line card.

s720-l2reg1#show module
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1   48  48-port 10/100 mb RJ45                 WS-X6148-RJ45V     SAL0651A8L8
  2   48  SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX     SAD084001L3
  3   16  SFM-capable 16 port 1000mb GBIC        WS-X6516A-GBIC     SAL0705CD6Z
  4   48  CEF720 48 port 1000mb SFP              WS-X6748-SFP       SAD08320HPC
  5    2  Supervisor Engine 720 (Active)         WS-SUP720-BASE     SAD0833034R
  6    2  Supervisor Engine 720 (Hot)            WS-SUP720-BASE     SAD0839060F
  7    6  Firewall Module                        WS-SVC-FWM-1       SAD070900BF
  8   16  SFM-capable 16 port 1000mb GBIC        WS-X6516-GBIC      SAL0716C5G9
  9   48  CEF720 48 port 1000mb SFP              WS-X6748-SFP       SAD074804F2

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  000b.bea9.a6f0 to 000b.bea9.a71f   1.2   5.4(2)       8.5(0.15)JAC Ok
  2  0003.3234.0fda to 0003.3234.1009  10.1   7.2(1)       8.5(0.15)JAC Ok
  3  0009.11f7.d488 to 0009.11f7.d497   1.0   7.2(1)       8.5(0.15)JAC Ok
  4  0011.938a.64ac to 0011.938a.64db   1.3   12.2(14r)S5  12.2(PP_SPL_ Ok
  5  0011.21b9.92dc to 0011.21b9.92df   3.1   8.1(3)       12.2(PP_SPL_ Ok
  6  0011.21b5.3bec to 0011.21b5.3bef   3.1   8.1(3)       12.2(PP_SPL_ Ok
  7  0003.feab.1940 to 0003.feab.1947   1.1   7.2(1)       1.1(3)       Ok
  8  000b.5ff9.b800 to 000b.5ff9.b80f   5.3   6.3(1)       8.5(0.15)JAC Ok
  9  0005.9a38.96ee to 0005.9a38.96ef   0.504 12.2(14r)S5  12.2(PP_SPL_ Ok

Mod Sub-Module                  Model              Serial        Hw     Status 
--- --------------------------- ------------------ ------------ ------- -------
  1 Inline Power Module         WS-F6K-PWR                       2.0    Ok
  4 Distributed Forwarding Card WS-F6700-DFC3BXL   SAD074606RU   0.207  Ok
  5 Policy Feature Card 3       WS-F6K-PFC3A       SAD0831087C   2.4    Ok
  5 MSFC3 Daughterboard         WS-SUP720          SAD083203NR   2.3    Ok
  6 Policy Feature Card 3       WS-F6K-PFC3A       SAD0838048U   2.4    Ok
  6 MSFC3 Daughterboard         WS-SUP720          SAD083801P6   2.3    Ok
  9 Distributed Forwarding Card WS-F6700-DFC3BXL   SAD07490237   0.202  Ok

Mod Online Diag Status
--- -------------------
  1 Pass
  2 Pass
  3 Pass
  4 Pass
  5 Pass
  6 Pass
  7 Pass
  8 Pass
  9 Pass

How To Identify DEC

Issue the show etherchannel command in IOS to determine if EtherChannel groups are distributed among multiple modules or not. Use the summary option to display a one-line summary per channel group.

The example below shows that EtherChannel group 10 contains ports on multiple modules (4 and 8) and is therefore Distributed EtherChannel.

s720-l2reg2#show etherchannel summary
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        u - unsuitable for bundling
Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+---------------------------------------
------+-------------+-----------+--------
10     Po10(SU)         -        Gi4/3(P)   Gi8/1(P)

Issue the show port channel command in CatOS to determine whether or not EtherChannel groups are distributed among multiple modules.

The example below shows that EtherChannel ID (Ch Id) 769 contains ports on multiple modules (1 and 2), and is therefore Distributed EtherChannel.

Console> show port channel
Port  Status     Channel   Admin Ch
                  Mode      Group Id
----- ---------- --------- ----- -----
  1/1  connected  on        195   769
  2/1  connected  on        195   769

Revision History

Revision

Date

Comment

1.1

29-JAN-2007

Changed Workaround section to include software upgrade as the favored solution.

1.0

08-FEB-2005

Initial Public Release

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.