Document ID: 115801
Updated: Mar 20, 2013
Contributed by Atri Basu, Anu M Chacko, and Wen Zhang, Cisco TAC Engineers.
This document describes the IPsec issue when security associations (SAs) become out of sync between the peer devices.
There are no specific requirements for this document.
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Refer to Cisco Technical Tips Conventions for more information on document conventions.
One of the most common IPsec issues is that SAs can become out of sync between the peer devices. As a result, an encrypting device will encrypt traffic with SAs that its peer does not know about. These packets are dropped on the peer with this message logged to the syslog:
Sep 2 13:27:57.707: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=188.8.131.52, prot=50, spi=0xB761863E(3076621886), srcaddr=10.1.1.1
Note: With NAT-T, RECVD_PKT_INV_SPI messages were not correctly reported until Cisco bug ID CSCsq59183 (registered customers only) was fixed. (IPsec does not report RECVD_PKT_INV_SPI messages with NAT-T)
Note: On the Cisco Aggregation Services Routers (ASR) platform, the %CRYPTO-4-RECVD_PKT_INV_SPI messages were not implemented until Cisco IOS® XE Release 2.3.2 (12.2(33)XNC2). Also note with the ASR platform, this particular drop is registered both under the global qfp drop counter well as in the IPsec feature drop counter as shown in these examples:
Router# show platform hardware qfp active statistics drop | inc Ipsec IpsecDenyDrop 0 0 IpsecIkeIndicate 0 0 IpsecInput 0 0 <====== IpsecInvalidSa 0 0 IpsecOutput 0 0 IpsecTailDrop 0 0 IpsecTedIndicate 0 0
Router# show platform hardware qfp active feature ipsec datapath drops all | in SPI 4 IN_US_V4_PKT_SA_NOT_FOUND_SPI 64574 <====== 7 IN_TRANS_V4_IPSEC_PKT_NOT_FOUND_SPI 0 12 IN_US_V6_PKT_SA_NOT_FOUND_SPI 0
First, it is important to note that this particular message is rate-limited in Cisco IOS at a rate of 1 per minute for the obvious security reasons. If this message for a particular flow (src/dst/spi) only shows up once in the log, then it could just be a transient condition at the same time as the IPsec rekey where one peer may start to use the new SA while the peer device is not quite ready to use the same SA. This is normally not a problem as it is only temporary and would only affect a few packets. However, there have been bugs where this could be a problem. For examples, please see Cisco bug IDs CSCsl68327 (registered customers only) (Packet loss during rekey) or CSCtr14840 (registered customers only) (ASR: packet drops during phase 2 rekey under certain conditions).
On the other hand, there is a problem if more than one instance of the same messages are observed to report the same SPI for the same flow, such as these messages:
Sep 2 13:36:47.287: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=184.108.40.206, prot=50, spi=0x1DB73BBB(498547643), srcaddr=10.1.1.1 Sep 2 13:37:48.039: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=220.127.116.11, prot=50, spi=0x1DB73BBB(498547643), srcaddr=10.1.1.1
This is an indication that traffic is blackholed, and may not recover until the SAs expire on the sending device or until Dead Peer Detection (DPD) kicks in.
At this point, Cisco recommends that you enable the invalid spi recovery feature. For example, enter the crypto isakmp invalid-spi-recovery command. There are a few things you need to know about what this command does and does not do:
First, invalid spi recovery only serves as a recovery mechanism when the SAs are out of sync. It helps recover from this condition, but does not address the root cause of why SAs are out of sync in the first place. In order to better understand the root cause, you need to enable isakmp and ipsec debug on both tunnel end points. If the problem occurs often, then obtain the debugs and try to address the root cause and not just mask the problem.
There is a common misconception of what the crypto isakmp invalid-spi-recovery command actually does. Even without this command, Cisco IOS already performs a kind of invalid SPI recovery functionality when it sends a DELETE notify for the SA received to the sending peer if it already has an IKE SA with that peer. Again, this occurs regardless of whether the crypto isakmp invalid-spi-recovery command is turned on or not.
With the crypto isakmp invalid-spi-recovery command, it tries to address the condition where a router receives IPsec traffic with invalid SPI and it does not have an IKE SA with that peer. In that case, it will try to establish a new IKE session with the peer and then send a DELETE notify over the newly created IKE SA. However, this command does not work under all crypto configurations. The only configurations that this command will work for are static crypto maps where the peer is explicitly defined and static peers derived from instantiated crypto maps, such as VTI. Here is a summary of commonly used crypto configurations and whether invalid spi recovery works with that configuration or not:
|Static crypto map||Yes|
|Dynamic crypto map||No|
|P2P GRE with TP||Yes|
|mGRE TP that uses w/ static NHRP mapping||Yes|
|mGRE TP that uses w/ dynamic NHRP mapping||No|
Many times the invalid SPI error message occurs intermittently. This makes it difficult to troubleshoot as it becomes very hard to collect the relevant debugs. EEM scripts can be very useful in this case.
For more details, refer to EEM Scripts used to Troubleshoot Tunnel Flaps Caused by Invalid Security Parameter Indexes.
The Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers.
Refer to Cisco Technical Tips Conventions for information on conventions used in this document.