Cisco IOS Software Releases 12.3 T

Field Notice: FN - 62394 - Incorrect IP Fragmentation for GRE Over IPSec Configurations

May 22, 2006



Products Affected

12.3(11)T and later

Problem Description

IOS may unnecessarily fragment GRE packets if GRE is configured over IPSec using a cryto map and the map is applied to both the tunnel and the physical outbound interface. The result is that on the other end of the tunnel, the receiving router has to perform IP reassembly on the fragmented GRE packet, thus causing high CPU on that router.


In IOS versions prior to 12.2(13)T, including all 12.2 mainline releases, in order to configure GRE over IPSec, that is, to encrypt GRE packets using IPSec as the L3 transport protocol, the crypto map needs to be applied to both the tunnel and the outbound physical interface. This requirement was historical when the initial IPSec implementation inherited it from Cisco Encryption Technology (CET), and has since been removed in 12.2(13)T and later. In this case the router could only do IPsec encryption after GRE encapsulation. The IPsec crypto ACL would be configured to match the GRE/IP encapsulated Data/IP packet, as shown here:

access-list <number> permit gre host <GRE-tunnel-src> host <GRE-tunnel-dst>

With the newer crypto implementation in 12.2(13)T and later, when a crypto map is applied to an interface, it always means crypto processing of the packet occurs before encapsulation on that interface, regardless of whether that is a physical interface or a GRE tunnel interface. This implies that for the GRE over IPSec configuration, the crypto map would only be applied to the outbound physical interface. It is no longer necessary to configure it on the tunnel. It also means now IPSec over GRE can be configured, that is, to transport IPSec packets inside of a GRE tunnel, by only applying the crypto map to the tunnel interface and configuring the IPSec crypto ACL to match data IP (clear-text) packets.

Additional changes were made in 12.3(11)T to make the various tunnel MTU calculations more consistent with the above described behavior. Specifically, the IPSec SA path MTU is the minimum of the ip mtu value from all the interfaces where the crypto map is applied. This, however, introduces an issue with the legacy configuration where crypto map is applied to both the tunnel and physical interfaces, which is what this field notice is intended to address. We will use the following example to illustrate this particular problem.


R1 has a GRE over IPSec tunnel to R2 configured over the internet. It has a crypto map configured on both the tunnel and the physical outbound interfaces. Also it has ip mtu 1400 configured on the tunnel interface to account for both the GRE and IPSec overhead.

The GRE and IPSec packet processing for large packets that require fragmentation works as follows:

  1. Since the crypto map is applied to both interfaces, and the ip mtu on the tunnel interface is 1400 while it is 1500 on the outbound physical interface, the IPSec SA path MTU is initialized to be the minimum of the two values, that is, 1400 bytes. This is because the router will assume IPSec over GRE if the crypto map is applied to the tunnel interface in the newer implementation as discussed previously. This can be seen in the show crypto ipsec sa output as shown below:

    R1#show crypto ipsec sa | in mtu|interface|peer|spi 
        interface: Tunnel0 
            current_peer port 500 
            path mtu 1400, ip mtu 1400 
            current outbound spi: 0xD5BB07F6(3585804278) 
            spi: 0xA6FECFD3(2801717203) 
            spi: 0xD5BB07F6(3585804278) 
        interface: Ethernet0/0 
            current_peer port 500 
            path mtu 1400, ip mtu 1400 
            current outbound spi: 0xD5BB07F6(3585804278) 
            spi: 0xA6FECFD3(2801717203) 
            spi: 0xD5BB07F6(3585804278) 
  2. A large packet enters the ingress interface on R1, and a route lookup determines that it needs to be routed to the tunnel interface. If the packet size is larger than the configured IP MTU value on the tunnel interface (1400), the Don't Fragment (DF) bit is checked.

    1. If the DF bit is set, then R1 will perform PMTUD by dropping the packet and sending an ICMP Packet too big but DF bit set (type 3 code 4) message back to the source host.

    2. If the DF bit is not set, then the IP data packet will be fragmented into two fragments before it is GRE encapsulated. By default, the resulting GRE packets will not have the DF bit set in the encapsulating GRE/IP header.

  3. After GRE encapsulation, the GRE packets are forwarded to the outbound physical interface after route lookup.

  4. When the router determines that the GRE packet needs to be IPSec encrypted and encapsulated, it checks to see if IPSec pre-fragmentation, also known as Look Ahead Fragmentation (LAF) is enabled.

    1. If LAF is not configured, then IPSec goes ahead encrypts, encapsulates, and forwards the packet to the interface driver.

    2. If LAF is enabled, which is the default behavior for GRE over IPSec, then the length of the GRE packet is compared to the IPSec SA path MTU less the IPSec overhead. If the GRE packet length is bigger, then IPSec will fragment the GRE packet into twofragments before it performs IPSec encryption and encapsulation.

    This step can cause unnecessary fragmentation for certain packet sizes that are efficiently small and do not need to be fragmented. For example, a GRE packet of length 1424 bytes (1400 bytes of IP payload and 24 bytes of GRE overhead, assuming no tunnel key) will fit into a 1500 byte IP datagram even after IPSec encapsulation, but it still gets fragmented by IPSec based on the logic above because it is bigger than the IPSec SA path MTU (1400) less the IPSec overhead.

  5. After the receiver decrypts the IPSec packets that contain the GRE fragments, it needs to process switch these packets before they can be reassembled. This process is typically slow and costly in terms of the CPU usage.

Problem Symptoms

The problem can be observed on a GRE over IPSec tunnel end point router with two common symptoms:

  1. High CPU usage in the IP Input process, as shown below:

    R2#show process cpu | include CPU|IP Input 
      CPU utilization for five seconds: 63%/0%; one minute: 45%; five minutes: 38% 
        31 8140 6934 1173 52.69% 42.05% 32.41% 0 IP Input 
  2. Large amount of IP reassembly, as reported by show ip traffic shown below:

    R2#show ip traffic | in reassemble|fragment 
        Frags: 7499 reassembled, 12 timeouts, 0 couldn't reassemble 
            0 fragmented, 0 fragments, 0 couldn't fragment 

Note: Fragmentation counts are likely okay, but the reassemble counts should be checked on the other end of the tunnel.


The unnecessary fragmentation can be avoided by applying one of the following configuration alternatives:

  1. If crypto map is used for GRE over IPSec configuration, make sure crypto map is only applied to the physical outbound interface, and not the tunnel interface.

    Note: For GRE over IPSec configuration, it is always recommended to configure a crypto map local-address for the IPSec local termination point.

  2. Disable IPSec pre-fragmentation (LAF) either globally or on the physical interface that the crypto map is applied to. Here is an example of disabling LAF globally:

    R1#config t 
    Enter configuration commands, one per line. End with CNTL/Z. 
    R1(config)# crypto ipsec fragmentation after-encryption 
  3. Use tunnel protection instead of crypto maps. For details, see the Dynamic Multipoint VPN (DMVPN) Feature Guide

Please note that with the above three workarounds, reassembly may still occur on the receiving tunnel end point, depending on the specific configuration used. However, a more detailed discussion on general MTU tuning and best practices in dealing with fragmentation issues is outside of the scope of this field notice. For additional background information and further reading, see the IP Fragmentation and PMTUD white paper.

Revision History






Initial Public Release

NetPro Discussion Forums - Featured Conversations

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.