Introduction
This document describes what debugs to collect for most of the common Group Encrypted Transport VPN (GETVPN) issues.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
Components Used
This document is not restricted to specific software and hardware versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Background Information - GETVPN Troubleshooting Tools
GETVPN provides an extensive set of troubleshooting tools in order to ease the troubleshoot process. It is important to understand which of these tools are available, and when they are appropriate for each troubleshooting task. When troubleshooting, it is always a good idea to start with the least intrusive methods, so that the production environment is not negatively impacted. In order to assist that process, this section describes some of the commonly used tools available:
Control Plane Debugging Tools
Show Commands
Show commands are commonly used in order to show runtime operations in a GETVPN environment.
Syslogs
GETVPN has an enhanced set of syslog messages for significant protocol events and error conditions. This should always be the first place to look before you run any debugs.
Group Domain of Interpretation (GDOI) Event Trace
This feature was added in Version 15.1(3)T. Event tracing offers lightweight, always-on tracing for significant GDOI events and errors. There is also exit-path tracing with traceback enabled for exception conditions.
GDOI Conditional Debugs
This feature was added in Version 15.1(3)T. It allows filtered debugs for a given device based on the peer address, and should always be used when possible, especially on the Key Server.
Global Crypto and GDOI Debugs
These are the all of the various GETVPM debugs. Admins must use caution when debugging in large-scale environments. With GDOI debugs, five debug levels are provided for further debugging granularity:
GM1#debug crypto gdoi gm rekey ?
all-levels All levels
detail Detail level
error Error level
event Event level
packet Packet level
terse Terse level
| Debug Level |
What You Will Get |
| Error |
Error conditions |
| Terse |
Important messages to the user and protocol issues |
| Event |
State transitions and events such as send and receive rekeys |
| Detail |
Most detailed debug message information |
| Packet |
Includes dump of detailed packet information |
| All |
All of the above |
Data Plane Debugging Tools
Here are some data plane debugging tools:
- Access Lists
- IP Precedence Accounting
- Netflow
- Interface Counters
- Crypto Counters
- IP Cisco Express Forwarding (CEF) Global and Per-feature Drop Counters
- Embedded Packet Capture (EPC)
- Data Plane Debugs (IP packet and CEF debugs)
Troubleshoot
Logging Facility Preparation and Other Best Practices
Before you begin to troubleshoot, ensure that you have prepared the logging facility as described here. Some best practices are also listed here:
Troubleshoot IKE Establishment
When the registration process first begins, GMs and KSs negotiate Internet Key Exchange (IKE) Sessions in order to protect GDOI traffic.
- On the GM, check that IKE is successfully established:
gm1#show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
172.16.1.9 172.16.1.1 GDOI_REKEY 1068 ACTIVE
172.16.1.1 172.16.1.9 GDOI_IDLE 1067 ACTIVE
Note: The GDOI_IDLE state, which is the base of the registration, times out quickly and disappears, because it is not needed anymore after the initial registration.
- On the KS, you should see:
ks1#show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
172.16.1.1 172.16.1.9 GDOI_IDLE 1001 ACTIVE
Note: The rekey session only appears when needed on the KS.
Complete these steps if you do not reach that state:
- For insight about the cause of the failure, check the output from this command:
router# show crypto isakmp statistics
- If the previous step is not helpful, you can get protocol-level insights if you enable the usual IKE debugs:
router# debug crypto isakmp
Notes:
* Even though IKE is used, it is not used on the usual UDP/500 port, but rather on UDP/848.
* If you encounter an issue at this level, provide the debugs for both KS and the affected GM.
- Due to the dependence on Rivest-Shamir-Adleman (RSA) sigs for the group rekeys, the KS must have an RSA key configured, and it must have the same name as the one specified in the group configuration.
In order to check this, enter this command:
ks1# show crypto key mypubkey rsa
Troubleshoot the Initial Registration
On the GM, in order to check the registration status, examine the output of this command:
gm1# show crypto gdoi | i Registration status
Registration status : Registered
gm1#
If the output indicates anything other than Registered, enter these commands:
On the GMs:
On the KSs:
Troubleshoot Policy-Related Issues
Policy Issue Occurs PRIOR to Registration (Related to Fail-close Policy)
This issue only affects GMs, so collect this output from the GM:
gm1# show crypto ruleset
Note: In Cisco IOS-XE?, this output is always empty since packet classification in not done in the software.
The show tech command output from the affected device provides the rest of the required information.
Policy Issue Occurs POST Registration, and Pertains to the Global Policy that is Pushed
There are usually two ways that this problem manifests:
- The KS cannot push the policies to the GM.
- There is a partial application of the policy amongst the GMs.
In order to help troubleshoot either issue, complete these steps:
- On the affected GM, collect this output:
gm1# show crypto gdoi acl
gm1# show crypto ruleset
- Enable these debugs on GM:
gm1# debug crypto gdoi infra packet
gm1# debug crypto gdoi gm acls packet
- On the KS to which the affected GM registers, collect this output:
ks1# show crypto gdoi ks members
ks1# show crypto gdoi ks policy
Note: In order to identify which KS the GM connects to, enter the show crypto gdoi group command.
- On the same KS, enable these debugs:
ks1# debug crypto gdoi infra packet
ks1# debug crypto gdoi ks acls packet
- Force the GM to register with this command on the GM:
clear crypto gdoi
Policy Issue Occurs POST Registration, and Pertains to the Merge of Global Policy and Local Overrides
This issue usually manifests itself in the form of messages that indicate that an encrypted packet was recieved for which the local policies indicate that it is not supposed to be encrypted and vice versa. All of the data requested in the previous section and the show tech command output is required in this case.
Troubleshoot Rekey Issues
On the GMs:
On the KSs:
Troubleshoot Time-based Anti-replay (TBAR)
The TBAR feature requires time-keeping across groups, and therefore requires the GMs pseudo-time clocks to be constantly resynced. This is performed during rekey or every two hours, whichever comes first.
Note: All output and debugs must be collected at the same time from both GMs and KS so that they can be correlated appropriately.
In order to investigate issues that occur at this level, collect this output.
In order to investigate TBAR time-keeping in a more dynamic way, enable these debugs:
- On the GM:
gm1# debug crypto gdoi gm rekey packet
gm1# debug crypto gdoi replay packet (verbosity might need to be lowered)
- On the KS:
ks1# debug crypto gdoi ks rekey packet
ks1# debug crypto gdoi replay packet (verbosity might need to be lowered)
As of Cisoc IOS Version 15.2(3)T, the ability to record TBAR errors has been added, which makes it easier to spot these errors. On the GM, use this command in order to check if there are any TBAR errors:
R103-GM#show crypto gdoi gm replay
Anti-replay Information For Group GETVPN:
Timebased Replay:
Replay Value : 512.11 secs
Input Packets : 0 Output Packets : 0
Input Error Packets : 0 Output Error Packets : 0
Time Sync Error : 0 Max time delta : 0.00secs
TBAR Error History (sampled at 10pak/min):
No TBAR errors detected
For more information about how to troubleshoot TBAR issues, refer to Time Based Anti-Replay Failure.
Troubleshoot KS Redundancy
Cooperative (COOP) establishes an IKE session in order to protect interKSs communication, so the troubleshooting technique previously described for IKE establishment is applicable here as well.
COOP-specific troubleshooting comprises output checks of this command on all KSs involved:
ks# show crypto gdoi ks coop
Note: The most common mistake made with deployment of COOP KSs is to forget to import the same RSA key (both private and public) for the group on all KSs. This causes problems during rekeys. In order to check and compare public keys amongst KSs, compare the output of the show crypto key mypubkey rsa command from each KS.
If protocol-level troubleshooting is required, enable this debug on all KSs involved:
ks# debug crypto gdoi ks coop packet
FAQ
Why do you see this error message "% Setting rekey authentication rejected"?
You see this error message when you configure the KS after this line is added:
KS(gdoi-local-server)#rekey authentication mypubkey rsa GETVPN_KEYS
% Setting rekey authentication rejected.
The reason for this error message is usually because the key labelled GETVPN_KEYS does not exist. In order to fix this, create a key with the correct label using the command:
crypto key generate rsa mod <modulus> label <label_name>
Note: Add the exportable keyword at the end if this is a COOP deployment and then import the same key in the other KS
Can a router configured as KS for one GETVPN group also function as a GM for the same group?
No. All GETVPN deployments require a dedicated KS which cannot participate as a GM for the same groups. This feature is not supported, because adding GM functionality to KS with all possible interactions like encryption, routing, QoS, etc., is not optimal for the health of this crucial network device. It must be available at all times for the entire GETVPN deployment to work.
Related Information
Feedback