IPSec Negotiation IKE Protocols

IPsec FAQ: Why are Avaya phones no longer able to connect via IPsec VPN after code upgrade on the ASA?

Document ID: 116294

Updated: Jul 17, 2013

Contributed by Atri Basu and Gustavo Medina, Cisco TAC Engineers.



This document describes a problem encountered when Avaya is deployed on a system in which the phones use the built-in Internet Protocol Security (IPsec) client.

Why are Avaya phones no longer able to connect via IPSEC VPN after code upgrade on the Cisco Adaptive Security Appliance (ASA)?

In order to understand this problem, you need to understand how Network Address Translation traversal (NAT-T) and NAT discovery (NAT-D) works. The NAT-D process is comprised of these steps:

  1. Detects one or more NAT devices between IPsec hosts.
  2. Identifies if the peer supports NAT-T.
  3. Negotiates the use of User Datagram Protocol (UDP) encapsulation of IPsec packets through NAT devices in the Internet Key Exchange (IKE).

NAT-D sends the hashes of the IP addresses and ports of both IKE peers from each end to the other. If both ends calculate those hashes and produce the same results, they know there is no NAT between. The hashes are sent as a series of NAT-D payloads. Each payload contains one hash. In the case of multiple hashes, multiple NAT-D payloads are sent. Normally, there are only two NAT-D payloads. The NAT-D payloads are included in the third and fourth packets of the Main Mode, and in the second and third packets of the Aggressive Mode. Since this example uses a remote access tunnel, it is the Aggressive Mode.

One of the details included in the NAT-D payloads is the Vendor ID (VID). The exchange of VIDs between peers helps determine the NAT-T capability of the remote host, as described in Request for Comments (RFC) 3947:

The format of the NAT-D packet is:

1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
| Next Payload | RESERVED | Payload length |
~ HASH of the address and port

The payload type for the NAT discovery payload is 20.

The current accepted payload type of the NAT-D payload is 20. If you look at the debugs on the ASA, you see:

[IKEv1]IP =, IKE_DECODE RECEIVED Message (msgid=0) with payloads:
HDR + KE (4) + NONCE (10) + UNKNOWN (15), *** ERROR *** + UNKNOWN (15),
*** ERROR *** + NONE (0) total length : 232

Here are snapshots from the packet captures:

ASA to phone:

Phone to ASA:

Avaya does not recognize payload 20, and the ASA does not understand payload type 15. The explanation for this behavior is because, in 2004, the same RFC defined the payload type as 15. Therefore, since 2004, the Avaya phones that use this payload type are no longer RFC compliant. So, why did it work with older code? Because, like Avaya, some of the older code (Version 8.0.x) still supports the old ID. However, the newer code (Versions 8.2.1+) is supposed to be compliant with the new RFC value and should not support payload type15. Nonetheless, you can find various versions around that still support payload type15, which is what causes the problem.

Avaya needs to fix the firmware on the phone so that the built-in VPN client uses the right paylod ID. Unfortunately, some other Avaya phones, like the 46xx Series, are no longer in production and will not get a fix. In this case, you either need to obtain new equipment or need to downgrade the ASA to a version on which it was working. Obviously this latter option is not available if you upgraded in order to get a bug fix in the first place. Any of Cisco's software versions that work with the older payload ID need to be reported and the issue fixed on those versions.

Updated: Jul 17, 2013
Document ID: 116294