Guest

Cisco Security Modules for Routers and Switches

Field Notice: FN - 62175 - Catalyst 6500 Series and 7600 Series IPSec VPN (WS-SVC-IPSEC-1) Issues With Fragmentation - Recommended Workaround Provided


October 03, 2005


NOTICE:

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Products Affected

Product

Comments

6503VPN - WS-C6503-E-VPN-K9

VPN Service Module Bundle

6503VPN - WS-C6506-E-VPN-K9

VPN Service Module Bundle

6503VPN - WS-SVC-IPSEC-1

VPN Service Module

Problem Description

When configuring GRE/IPSec or DMVPN on a Catalyst 6500 or C7600 series and a VPN Service Module, a number of bugs actually prevent the platform from either performing its task properly or to interact with other platforms in a CPU efficient manner. By properly setting all the parameters and applying a simple workaround, the feature can be made to work well but is not forgiving toward errors.

The perception of the problem is made worse by slightly different behaviors across IOS versions and Supervisor version or revisions.

Depending on the DDTS hit or on the configuration, the symptoms can vary from sheer packet losses to high CPU issue. Most of the time, the high CPU will be seen on the platform that receives packets from a misconfigured device.

The purpose if this Field Notice is to specify how the feature is expected to be configured on all platforms and to provide the workaround needed for the Catalyst 6500 and C7600 platforms equipped with a VPNSM to work and interact well with other devices, whether they are Catalyst 6500 or classic IOS platforms.

Background

Scope of the explanation:

C6500 and C7600 + VPNSM GRE handling

The C6500 and C7600 equipped with a VPNSM can handle GRE/IPSec in different ways, depending on the configuration, packet features and Supervisor revision.

  1. If the packet has IP options, the packet is punted to the Route Processor (software) no matter what normally handles GRE (VPNSM or Sup720). IPSec is still performed on the VPNSM.

  2. If the tunnel interface is setup with special features such as tunnel key, all GRE handling is performed on the Route Processor (software).IPSec is still performed on the VPNSM.

  3. Sup2/MSFC2 GRE encapsulation / decapsulation is performed by the VPNSM.

  4. Sup720 (PFC 3a, 3b and 3bxl)

    1. If the tunnel source is unique, used on a single tunnel, GRE encapsulation / decapsulation is performed by the Sup720. IPSec is still performed on the VPNSM.

    2. If the tunnel source is used on more than one tunnel, GRE handling is performed by the VPNSM.

If GRE is handled by the Sup720, the following remarks apply:

  1. Tunnel path-mtu-discovery is ignored.

  2. The DF bit on the GRE packet is ALWAYS set.

  3. Packets to be fragmented before GRE encapsulation are punted to the Route Processor (software).

  4. Fragmented GRE packets to be reassembled are punted to the Route Processor (software).

If GRE is handled by the VPNSM:

  1. Tunnel path-mtu-discovery is ignored.

  2. All fragmentation and reassembly operations are performed in the VPNSM (hardware).

  3. IPSec and GRE MTU related operations are linked together as explained in the next section.

If GRE is handled by the Route Processor, due to exception punting or configuration, the packet flow is normal but a performance impact is to be expected if too many packets are to be forwarded by the Route Processor.

Fragmentation order in GRE/IPSec

This explanation pertains to GRE/IPSec in general and, as a corollary, applies to DMVPN. The packet flow below is valid for classic IOS and C6500/C7600 architectures.

When a clear packet enters a router on its ingress interface and should be routed out via a GRE tunnel interface to be encrypted, the following high level operations occur:

  1. If GRE is handled by the VPNSM, skip this step. Otherwise, the clear packet size is compared to the path MTU (PMTU) of the tunnel. If tunnel-path-mtu discovery is not activated, the PMTU is always equal to the IP MTU of the tunnel. If GRE encapsulation is performed by the PFC (Supervisor hardware), tunnel path-mtu-discovery is not supported and the PMTU is the tunnel interface IP MTU.

    If the packet size is greater than the tunnel PMTU, the packet DF bit is checked.

    1. If the DF bit is set, the clear packet is dropped and an ICMP message, Packet-Too-Big, is sent back to the source.

    2. If the DF bit is clear, the clear packet is fragmented and each fragment is sent back to step 1.

  2. If GRE is handled by the VPNSM, skip this step. The GRE packet DF bit is fixed:

    On a generic platform:

    1. If GRE tunnel path-mtu-discovery is activated and recognized on the platform, the DF bit of the clear packet is copied over onto the GRE packet header.

    2. Otherwise, the DF bit of the GRE packet is cleared (set to zero).

  3. The Security Association (SA) path MTU value is computed. In reality, this value is not computed for each packet but computed once and for all at each step in Path-MTU-Discovery or at configuration time.

    1. If GRE is handled by the VPNSM, the overhead size is currently fixed at 84 bytes, no matter what the transform-set is; these 84 bytes take in account ESP and GRE overhead. Two cases arise due to CSCed93707.

      1. Before Cisco IOS® Release 12.2(17d)SXB01, the SA PMTU is equal to the lesser of the GRE tunnel IP MTU and physical interface IP MTU minus 84 bytes. For example, min (Physical MTU, GRE MTU) - 84.

      2. From Cisco IOS Release 12.2(17d)SXB01 onward, including all SXD and SXE releases, the SA PMTU is set to GRE IP MTU - 84.

    2. In all the other cases, the SA PMTU is originally set to the SA MTU which in turn is equal to the media MTU (the egress interface MTU) minus IPSec overhead. On the VPNSM, the IPSec overhead is always 84 bytes.

  4. IPSec checks whether pre-fragmentation, also know as Look Ahead Fragmentation (LAF), is activated

    1. If GRE is handled by the VPNSM, operations 1&2 are actually collapsed here

      1. If LAF is activated, the IP packet (the packet that entered the router and skipped step 1 and above) length is compared to the SA PMTU.

        1. If the IP packet is longer than the SA MTU, the packet DF bit is checked.

        a. If the DF bit is set, the packet is punted to the RP and an ICMP packet-too-big is to the originator.

        b. If the DF bit is cleared, the IP packet is fragmented and re-circulated at step 4.a.

        2. If the IP packet is smaller than the SA MTU, the packet is GRE/IPSec encapsulated in one shot and forwarded

      2. If LAF is not activated, the packet is GRE/IPSec encapsulated, encrypted and forwarded

    2. If GRE is handled by anything else than the VPNSM:

      1. The GRE packet (created by step 1 and 2 above) length is compared to the Security Association path MTU.

        1. If the GRE packet is longer than the SA PMTU, the GRE packet DF bit is checked.

        a. If the DF bit is set, the tunnel PMTU is updated. This step typically is shortcut but can happen in some cases.

        b. If the DF bit is cleared, the GRE packet is fragmented and each GRE fragment is re-circulated at step 4.b.

        2. If the GRE packet is short enough, it is encapsulated and encrypted into an IPSec packet (ESP or AH) and forwarded.

      2. If pre-fragmentation is not activated, the packet is encapsulated into an IPSec packet (ESP or AH), encrypted and forwarded.

  5. The IPSec packet DF bit is fixed, set, cleared or copied over from the GRE packet, according to the IPSec configuration (df-bit clear, set or copy). The default is to copy.

  6. The IPSec packet length is checked against the outgoing interface IP MTU.

    1. If the IPSec packet is bigger than the interface IP MTU, the DF bit is checked.

      1. If the DF-bit is set, the packet is dropped and the SA Path MTU is updated. This step typically does not happen.

      2. If the DF-bit is cleared, the packet is fragmented and each IPSec fragment is re-circulated from point 6.

    2. If the IPSec packet is small enough, which it typically is, the packet is forwarded out.

  7. The packet is transported by the network and can be fragmented at any point, similarly to point 6.

Reassembly

Before an IPSec or GRE packet can be decrypted or decapsulated, the packet must be complete. If needed, the packet must be reassembled from its underlying fragments. Fragmentation is a cheap operation as it can be performed in hardware (on the VPNSM) or in the CEF switching path (on classic IOS platforms). On the other hand, reassembly is only performed in hardware on the VPNSM and any other platform would require punting the packet to the process switching flow. This operation is typically slow, costly and can be spotted through packet losses and high CPU. Even when done in hardware, reassembly buffers must be readily available under the penalty of packet losses. If fragments are lost, reassembly buffers get consumed and both software and hardware reassembly are impacted.

Good practices

The best strategy to prevent reassembly issues is to prevent fragmentation of packets that should be reassembled by network devices. In short, fragmentation of GRE and IPSec must be avoided at all cost. In the packet flow above, it means preventing fragmentation at point 1, 4, 6 or 7.

The best solution to this problem is to have a properly set GRE IP MTU or GRE Path MTU in order to let network hosts (end stations) TCP stacks perform path-mtu-discovery. If this can't be reliably configured, a possible workaround is to force a low TCP segment size by either configuring a low MTU or a low MSS on the end stations or forcing the MSS negotiation to a low value with the command ip tcp adjust-mss along the path. This command is not supported by the Catalyst 6000 or c7600 family and must be deployed on a regular architecture.

As of August 1, 2005 the following must be taken in account:

  • Neither the VPNSM nor the Sup720 support tunnel path-mtu-discovery.

  • All Sup720 revisions punt packets to the RP in order to perform fragmentation before GRE encapsulation.

The best option is to:

  • set the IP MTU of the GRE tunnels to a value that will prevent fragmentation after point 1

  • Delegate the GRE tunnels to the VPNSM (this is less important).

Problem Symptoms

Problem 1 - packet losses

The problem symptoms are twofold, depending on the exact configuration and caveat hit. It is enough to see one of the symptoms below to be affected.

Problem 2 - high CPU

If one of the devices exhibits a high CPU, issue the following commands:

Router# show ip traffic | include assembled 

Frags: 10324586 reassembled, 459274 timeouts, 543208 couldn't reassemble

If the counters are high and/or increasing rapidly, this is very likely to be the culprit. You can confirm this by issuing

Router# show proc cpu | include IP Input

66       54624   1319916         41  50.00%  45.03%  40.02%   0 IP Input

If the CPU time consumed by IP Input is high, this is likely to confirm the symptoms. A final verification can be made by dumping the input queue:

show buffers input-interface GigabitEthernet 0/1 dump

If most dumped packets are fragments, visible in the IP flags field, this is definitively a match.

Note: The high CPU and reassembly symptoms are visible on the side of the VPN opposite to the problem.

Note: Misconfigurations can cause a high CPU due to punting but IP Input would not engage and reassembly counters would not increase. Contact the TAC to confirm the exact source of the problem if you think you are affected.

Identifying GRE support by the VPNSM

The command show crypto vlan shows how a given GRE tunnel is supported. In the following example, Tunnel1 and Tunnel2 GRE encapsulation is handled by the VPNSM. Tunnel0 (not appearing) is not handled by the VPNSM.

Router#show crypto vlan

Interface VLAN 100 on IPSec Service Module port 5/1 connected to FastEthernet3/1 with crypto map set CMAP

     Tunnel1 is accelerated via IPSec SM in slot 5

     Tunnel2 is accelerated via IPSec SM in slot 5

Workaround/Solution

Basic solution (classic IOS platforms only)

In order to address the problem, it is important to lower the GRE IP MTUs of both sides of the VPNSM to a value low enough to prevent post-GRE or post-IPSec fragmentation.

Note: This is a solution to the problem and not a workaround.

This is done by changing the following value on the GRE tunnel interface:

interface Tunnel0

  ip mtu 1400

  [?]

crypto ipsec fragmentation before-encryption

crypto ipsec df-bit set

The example above assumes a path MTU between the IPSec devices of about 1500 bytes. If the path MTU can not be determined exactly, lower the IP MTU value on both sides until the symptoms disappear.

Dynamic solution (classic IOS platforms only)

The GRE tunnel can discover the GRE Path MTU by itself. This feature may not work if firewalls on the path of the IPSec packets filter out ICMP packet-too-big (type 3, code 4). This feature is not supported on Catalyst 6500 or C7600.

interface Tunnel0

  ip mtu 1400

  tunnel path-mtu-discovery

  [?]

crypto ipsec fragmentation before-encryption

crypto ipsec df-bit set

Workaround and solution for VPNSM on c6500 and c7600 (not for DMVPN)

The solution is almost the same as for classic IOS (we want to fragment before encapsulating) but we need an additional workaround for CSCsb12076 that prevents transport mode from working.

interface Tunnel0

  ip mtu 1400

  [?]

crypto ipsec fragmentation before-encryption

crypto ipsec df-bit set

crypto ipsec transform-set MyTS esp-3des esp-sha

  mode tunnel ! THIS IS VERY IMPORTANT

Extra step for Sup720/VPNSM (not for DMVPN)

On a Catalyst 6500 or C7600 equipped with Sup720's, GRE handling is performed by the Sup720 instead of the VPNSM. Since fragmentation is better and faster performed on the VPNSM, a workaround must be applied. The Sup720 can not support two GRE tunnels with the same tunnel source (source sharing). If such a configuration is entered, tunnel support will fall back to the VPNSM. For each configured GRE tunnel, a dummy, non functional tunnel must be created that will have no other purpose than moving GRE support to the VPNSM. If an existing GRE tunnel is configured as

interface Tunnel0
 
 ip mtu 1400

  tunnel source FastEthernet 3/1

  [?]

Then another tunnel must be configured with just a tunnel source, no destination and no ip address:

interface Tunnel1

  tunnel source FastEthernet 3/1

  [do not add anything else; this is a ?dummy? tunnel to force VPNSM takeover]

Workaround for DMVPN on C6500 or C7600

Due to CSCeh37078, it is not possible to apply the solutions above. There is only one workaround for this:

interface Tunnel0

  ip mtu 1500 ! or whatever the ip mtu of the input interface is

  [?]       

crypto ipsec fragmentation after-encryption

crypto ipsec df-bit clear

It has to be noted that packets will be fragmented after encryption, which is contrary to good practices.

Note: This workaround will have a negative performance impact on the receiver end as it has to reassemble IPSec packets before decrypting them.

There is currently no other solution.

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered user and you must be logged in.

DDTS

Description

CSCsb12076 (registered customers only)

VPNSM drops large GRE packets in Transport mode; use Tunnel mode instead

CSCsa81259 (registered customers only)

SUP720-3B & 3BXL handle frag. & reassem. of GRE pkt differently (fragmented GRE packets not seen by EARL for GRE reassembly and decap)

CSCeh37078 (registered customers only)

First fragment cannot get to Macedon if pkts from CE to dmvpn spoke (When remote peer sends large packet, first GRE fragment not seen by VPNSM when using a combination of (DMVPN, SXE train, Sup720). No probs with regular GRE/IPSec)

CSCec45417 (registered customers only)

VPNSM does not honor DF bit for GRE over IPSEC (fixed in 12.2(14)SY03)

CSCea38318 (registered customers only)

DDTS needs to be fixed to support path-mtu-discovery when the GRE tunnel is handled by the VPNSM

CSCee64248 (registered customers only)

DDTS needs to be fixed to support path-mtu-discovery when the GRE tunnel is handled by the VPNSM

How To Identify Hardware Levels

Use the show module command to determine the modules inserted in the chassis:

Router#show module 


Mod   Ports   Card Type                            Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
 6    2  Supervisor Engine 720 (Active)         WS-SUP720-3B       SAD090902XR
 7    2  IPSec VPN Accelerator                  WS-SVC-IPSEC-1     SAD061402PC

The affected modules are a combination of WS-SVC-IPSEC-1 with WS-SUP720-3A, WS-SUP720-3B and WS-SUP720-3BXL.

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.