This document describes what debugs to collect for most of the common Group Encrypted Transport VPN (GETVPN) issues.
Cisco recommends that you have knowledge of these topics:
Syslog Server Use
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Background Information - GETVPN Troubleshooting Tools
GETVPN provides an extensive set of troubleshooting tools in order to ease the troubleshoot process. It is important to understand which of these tools are available, and when they are appropriate for each troubleshooting task. When troubleshooting, it is always a good idea to start with the least intrusive methods, so that the production environment is not negatively impacted. In order to assist that process, this section describes some of the commonly used tools available:
Control Plane Debugging Tools
Show commands are commonly used in order to show runtime operations in a GETVPN environment.
GETVPN has an enhanced set of syslog messages for significant protocol events and error conditions. This should always be the first place to look before you run any debugs.
Group Domain of Interpretation (GDOI) Event Trace
This feature was added in Version 15.1(3)T. Event tracing offers lightweight, always-on tracing for significant GDOI events and errors. There is also exit-path tracing with traceback enabled for exception conditions.
GDOI Conditional Debugs
This feature was added in Version 15.1(3)T. It allows filtered debugs for a given device based on the peer address, and should always be used when possible, especially on the Key Server.
Global Crypto and GDOI Debugs
These are the all of the various GETVPM debugs. Admins must use caution when debugging in large-scale environments. With GDOI debugs, five debug levels are provided for further debugging granularity:
Important messages to the user and protocol issues
State transitions and events such as send and receive rekeys
Most detailed debug message information
Includes dump of detailed packet information
All of the above
Data Plane Debugging Tools
Here are some data plane debugging tools:
IP Precedence Accounting
IP Cisco Express Forwarding (CEF) Global and Per-feature Drop Counters
Embedded Packet Capture (EPC)
Data Plane Debugs (IP packet and CEF debugs)
Logging Facility Preparation and Other Best Practices
Before you begin to troubleshoot, ensure that you have prepared the logging facility as described here. Some best practices are also listed here:
Check the router amount of free memory, and configure logging buffered debugging to a large value (10 MB or more if possible).
Disable logging to the console, monitor, and syslog servers.
Retrieve the logging buffer content with the show log command at regular intervals, every 20 mins to an hour, in order to prevent log loss due to buffer reuse.
Whatever happens, enter the show tech command from affected Group Members (GMs) and Key Servers (KSs), and examine the output of the show ip route command in global and each Virtual Routing and Forwarding (VRF) involved, if any are required.
Use Network Time Protocol (NTP) in order to sync the clock between all devices that are debugged. Enable millisecond (msec) timestamps for both debug and log messages:
service timestamps debug datetime msec service timestamps log datetime msec
Make sure show command outputs are timestamped.
Router#terminal exec prompt timestamp
When you collect show command outputs for control plane events or data plane counters, always collect multiple iterations of the same output.
Troubleshoot IKE Establishment
When the registration process first begins, GMs and KSs negotiate Internet Key Exchange (IKE) Sessions in order to protect GDOI traffic.
On the GM, check that IKE is successfully established:
gm1#show crypto isakmp sa IPv4 Crypto ISAKMP SA dst src state conn-id status 172.16.1.9 172.16.1.1 GDOI_REKEY 1068 ACTIVE 172.16.1.1 172.16.1.9 GDOI_IDLE 1067 ACTIVE
Note: The GDOI_IDLE state, which is the base of the registration, times out quickly and disappears, because it is not needed anymore after the initial registration.
On the KS, you should see:
ks1#show crypto isakmp sa IPv4 Crypto ISAKMP SA dst src state conn-id status 172.16.1.1 172.16.1.9 GDOI_IDLE 1001 ACTIVE
Note: The rekey session only appears when needed on the KS.
Complete these steps if you do not reach that state:
For insight about the cause of the failure, check the output from this command:
router# show crypto isakmp statistics
If the previous step is not helpful, you can get protocol-level insights if you enable the usual IKE debugs:
router# debug crypto isakmp
Notes: * Even though IKE is used, it is not used on the usual UDP/500 port, but rather on UDP/848. * If you encounter an issue at this level, provide the debugs for both KS and the affected GM.
Due to the dependence on Rivest-Shamir-Adleman (RSA) sigs for the group rekeys, the KS must have an RSA key configured, and it must have the same name as the one specified in the group configuration.
In order to check this, enter this command:
ks1# show crypto key mypubkey rsa
Troubleshoot the Initial Registration
On the GM, in order to check the registration status, examine the output of this command:
gm1# show crypto gdoi | i Registration status Registration status : Registered gm1#
If the output indicates anything other than Registered, enter these commands:
On the GMs:
Shut down crypto-enabled interfaces.
Caution: It is expected that out-of-band management is enabled.
Force the GM to register with this command on the GM:
clear crypto gdoi
Policy Issue Occurs POST Registration, and Pertains to the Merge of Global Policy and Local Overrides
This issue usually manifests itself in the form of messages that indicate that an encrypted packet was recieved for which the local policies indicate that it is not supposed to be encrypted and vice versa. All of the data requested in the previous section and the show tech command output is required in this case.
The TBAR feature requires time-keeping across groups, and therefore requires the GMs pseudo-time clocks to be constantly resynced. This is performed during rekey or every two hours, whichever comes first.
Note: All output and debugs must be collected at the same time from both GMs and KS so that they can be correlated appropriately.
In order to investigate issues that occur at this level, collect this output.
On the GMs:
gm1# show crypto gdoi gm1# show crypto gdoi replay
On the KS:
ks1# show crypto gdoi ks members ks1# show crypto gdoi ks replay
In order to investigate TBAR time-keeping in a more dynamic way, enable these debugs:
On the GM:
gm1# debug crypto gdoi gm rekey packet gm1# debug crypto gdoi replay packet (verbosity might need to be lowered)
On the KS:
ks1# debug crypto gdoi ks rekey packet ks1# debug crypto gdoi replay packet (verbosity might need to be lowered)
As of Cisoc IOS Version 15.2(3)T, the ability to record TBAR errors has been added, which makes it easier to spot these errors. On the GM, use this command in order to check if there are any TBAR errors:
R103-GM#show crypto gdoi gm replay Anti-replay Information For Group GETVPN: Timebased Replay: Replay Value : 512.11 secs Input Packets : 0 Output Packets : 0 Input Error Packets : 0 Output Error Packets : 0 Time Sync Error : 0 Max time delta : 0.00secs
TBAR Error History (sampled at 10pak/min): No TBAR errors detected
Cooperative (COOP) establishes an IKE session in order to protect interKSs communication, so the troubleshooting technique previously described for IKE establishment is applicable here as well.
COOP-specific troubleshooting comprises output checks of this command on all KSs involved:
ks# show crypto gdoi ks coop
Note: The most common mistake made with deployment of COOP KSs is to forget to import the same RSA key (both private and public) for the group on all KSs. This causes problems during rekeys. In order to check and compare public keys amongst KSs, compare the output of the show crypto key mypubkey rsa command from each KS.
If protocol-level troubleshooting is required, enable this debug on all KSs involved:
ks# debug crypto gdoi ks coop packet
Why do you see this error message "% Setting rekey authentication rejected"?
You see this error message when you configure the KS after this line is added:
The reason for this error message is usually because the key labelled GETVPN_KEYS does not exist. In order to fix this, create a key with the correct label using the command:
crypto key generate rsa mod <modulus> label <label_name>
Note: Add the exportable keyword at the end if this is a COOP deployment and then import the same key in the other KS
Can a router configured as KS for one GETVPN group also function as a GM for the same group?
No. All GETVPN deployments require a dedicated KS which cannot participate as a GM for the same groups. This feature is not supported, because adding GM functionality to KS with all possible interactions like encryption, routing, QoS, etc., is not optimal for the health of this crucial network device. It must be available at all times for the entire GETVPN deployment to work.