Table Of Contents
Troubleshooting IPsec
Overview
Troubleshooting IPsec Issues
Verifying IKE Configuration Compatibility
IPsec Compatibility for iSCSI
Verifying IPsec Configuration Compatibility
Verifying Security Policy Databases Compatibility
Verifying Interface Status
Verifying Security Associations
Security Associations Do Not Re-Key
Clearing Security Associations
Debugging the IPsec Process
Debugging the IKE Process
Obtaining Statistics from the IPsec Process
Troubleshooting IPsec
This chapter describes procedures used to troubleshoot IP security (IPsec) and Internet Key Exchange (IKE) encryption in the Cisco MDS 9000 Family Switch products. It includes the following sections:
•
Overview
•
Troubleshooting IPsec Issues
Overview
The IP security (IPsec) protocol is a framework of open standards that provides data confidentiality, data integrity, and data authentication between participating peers. It is developed by the Internet Engineering Task Force (IETF). IPsec provides security services at the IP layer, including protecting one or more data flows between a pair of hosts, between a pair of security gateways, or between a security gateway and a host. The overall IPsec implementation is per the latest version of RFC 2401. Cisco MDS SAN-OS IPsec implements RFC 2402 through RFC 2410.
Troubleshooting IPsec Issues
This section provides the procedures required to troubleshoot IKE and IPsec issues in an FCIP configuration. Figure 11-1 shows a simple FCIP configuration where FCIP Tunnel 2 carries encrypted data between switches MDS A and MDS C.
Figure 11-1 Simple FCIP Configuration
This section includes the following topics:
•
Verifying IKE Configuration Compatibility
•
IPsec Compatibility for iSCSI
•
Verifying IPsec Configuration Compatibility
•
Verifying Security Associations
•
Security Associations Do Not Re-Key
•
Clearing Security Associations
•
Debugging the IPsec Process
•
Debugging the IKE Process
•
Obtaining Statistics from the IPsec Process
Verifying IKE Configuration Compatibility
To verify the compatibility of the IKE configurations of MDS A and MDS C shown in Figure 11-1, follow these steps:
Step 1
Ensure that the preshared keys are identical on each switch. Issue the show crypto ike domain ipsec key command on both switches. Example command outputs for configuration shown in Figure 11-1 follow:
MDSA# show crypto ike domain ipsec key
key ctct address 10.10.100.232
MDSC# show crypto ike domain ipsec key
key ctct address 10.10.100.231
Step 2
Ensure that at least one matching policy that has the same encryption algorithm, hash algorithm, and Diffie-Hellman (DH) group, is configured on each switch. Issue the show crypto ike domain ipsec policy command on both switches. Example command outputs for the configuration shown in Figure 11-1 follow:
MDSA# show crypto ike domain ipsec policy
Priority 1, auth pre-shared, lifetime 86300 secs, encryption 3des, hash md5, DH group 1
MDSC# show crypto ike domain ipsec policy
Priority 1, auth pre-shared, lifetime 86300 secs, encryption 3des, hash md5, DH group 1
IPsec Compatibility for iSCSI
Table 11-1 lists the supported settings for encryption and authentication algorithms to be used in iSCSI configurations on the Windows 2000 and Linux platforms.
Table 11-1 Encryption and Authentication Algorithms for iSCSI Configurations
Platform
|
IPsec
|
Microsoft iSCSI initiator, Microsoft IPsec implementation on Windows 2000 platform
|
3DES, SHA-1
|
Cisco iSCSI initiator, FreeSwan IPsec implementation on Linux platform
|
3DES, MD5
|
Verifying IPsec Configuration Compatibility
To verify the compatibility of the IPsec configurations of MDS A and MDS C shown in Figure 11-1 follow these steps:
Step 1
Issue the show crypto map domain ipsec command and the show crypto transform-set domain ipsec command. The following example command outputs display the fields discussed in Step 2 through Step 7.
MDSA# show crypto map domain ipsec
Crypto Map "cmap-01" 1 ipsec
Peer = 10.10.100.232
IP ACL = acl1
permit ip 10.10.100.231 255.255.255.255 10.10.100.232 255.255.255.255
Security Association Lifetime: 3000 gigabytes/120 seconds
PFS (Y/N): Y
PFS Group: group5
Interface using crypto map set cmap-01:
MDSC# show crypto map domain ipsec
Crypto Map "cmap-01" 1 ipsec
Peer = 10.10.100.231
IP ACL = acl1
permit ip 10.10.100.232 255.255.255.255 10.10.100.231 255.255.255.255
Security Association Lifetime: 3000 gigabytes/120 seconds
PFS (Y/N): Y
PFS Group: group5
Interface using crypto map set cmap-01:
MDSA# show crypto transform-set domain ipsec
Transform set:tfs-01 {esp-3des null}
Transform set:tfs-02 {esp-3des esp-md5-hmac}
Transform set:ipsec_default_transform_set {esp-aes 128 esp-sha1-hmac}
MDSC# show crypto transform-set domain ipsec
Transform set:tfs-01 {esp-3des null}
Transform set:tfs-02 {esp-3des esp-md5-hmac}
Transform set:ipsec_default_transform_set {esp-aes 128 esp-sha1-hmac}
Step 2
Ensure the ACLs are compatible on the show crypto map domain ipsec command outputs of both switches.
Step 3
Ensure the peer configuration is correct on the show crypto map domain ipsec command outputs of both switches.
Step 4
Ensure the transform sets are compatible on the show crypto transform-set domain ipsec command outputs of both switches.
Step 5
Ensure that the PFS settings on the show crypto map domain ipsec command outputs are configured the same on both switches.
Step 6
Ensure the security association (SA) lifetime settings on the show crypto map domain ipsec command outputs are large enough to avoid excessive re-keys (the default settings ensure this).
Step 7
Ensure that the crypto map set is applied to the correct interface on the show crypto map domain ipsec command outputs of both switches.
Verifying Security Policy Databases Compatibility
To verify the security policy databases (SPDs) are compatible on both switches, follow these steps:
Step 1
Issue the show crypto spd domain ipsec command on both switches to display the SPD. The example command outputs follow:
MDSA# show crypto spd domain ipsec
Policy Database for interface:GigabitEthernet7/1, direction:Both
# 0: deny udp any port eq 500 any <-----------Clear test policies for IKE
# 1: deny udp any any port eq 500 <-----------Clear test policies for IKE
# 2: permit ip 10.10.100.231 255.255.255.255 10.10.100.232 255.255.255.255
# 127: deny ip any any <------------Clear test policy for all other traffic
MDSC# show crypto spd domain ipsec
Policy Database for interface:GigabitEthernet1/2, direction:Both
# 0: deny udp any port eq 500 any
# 1: deny udp any any port eq 500
# 2: permit ip 10.10.100.232 255.255.255.255 10.10.100.231 255.255.255.255
Step 2
Issue the show crypto-accelerator interface gigabitethernet slot/port spd inbound command on both switches to display SPD information from the crypto-accelerator.
Note
To issue commands with the internal keyword, you must have an account that is a member of the network-admin group.
The example command outputs follow:
MDSA# show ipsec internal crypto-accelerator interface gigabitethernet 7/1 spd inbound
Destination IP Address :*
Source port :500, Destination port :* Protocol UDP
Physical port:0/0, Vlan_id:0/0
Destination IP Address :*
Source port :*, Destination port :500 Protocol UDP
Physical port:0/0, Vlan_id:0/0
Source IP Address :10.10.100.232/255.255.255.255
Destination IP Address :10.10.100.231/255.255.255.255
Source port :*, Destination port :* Protocol *
Physical port:0/1, Vlan_id:0/4095
Destination IP Address :*
Source port :*, Destination port :* Protocol *
Physical port:0/0, Vlan_id:0/0
MDSC# show ipsec internal crypto-accelerator interface gigabitethernet 1/2 spd inbound
Destination IP Address :*
Source port :500, Destination port :* Protocol UDP
Physical port:0/0, Vlan_id:0/0
Destination IP Address :*
Source port :*, Destination port :500 Protocol UDP
Physical port:0/0, Vlan_id:0/0
Source IP Address :10.10.100.231/255.255.255.255
Destination IP Address :10.10.100.232/255.255.255.255
Source port :*, Destination port :* Protocol *
Physical port:1/1, Vlan_id:0/4095
Destination IP Address :*
Source port :*, Destination port :* Protocol *
Physical port:0/0, Vlan_id:0/0
Verifying Interface Status
To verify the status of the interfaces, follow these steps:
Step 1
Issue the show interface gigabitethernet command on both switches. Verify that the interfaces are up and their internet addresses are correct. Issue the no shutdown command if necessary. The example command outputs follow:
MDSA# show interface gigabitethernet 7/1
GigabitEthernet7/1 is up
Hardware is GigabitEthernet, address is 0005.3001.804e
Internet address is 10.10.100.231/24
Auto-Negotiation is turned on
5 minutes input rate 7728 bits/sec, 966 bytes/sec, 8 frames/sec
5 minutes output rate 7968 bits/sec, 996 bytes/sec, 8 frames/sec
7175 packets input, 816924 bytes
0 multicast frames, 0 compressed
0 input errors, 0 frame, 0 overrun 0 fifo
7285 packets output, 840018 bytes, 0 underruns
0 output errors, 0 collisions, 0 fifo
MDSC# show interface gigabitethernet 1/2
GigabitEthernet1/2 is up
Hardware is GigabitEthernet, address is 0005.3001.7f0f
Internet address is 10.10.100.232/24
Auto-Negotiation is turned on
5 minutes input rate 7528 bits/sec, 941 bytes/sec, 8 frames/sec
5 minutes output rate 7288 bits/sec, 911 bytes/sec, 8 frames/sec
7209 packets input, 835518 bytes
0 multicast frames, 0 compressed
0 input errors, 0 frame, 0 overrun 0 fifo
7301 packets output, 827630 bytes, 0 underruns
0 output errors, 0 collisions, 0 fifo
Step 2
Issue the show interface fcip command on both switches. Verify that each interface is using the correct profile, the peer internet addresses are configured correctly, and the FCIP tunnels are compatible. Issue the no shutdown command if necessary. The example command outputs follow:
MDSA# show interface fcip 1
Hardware is GigabitEthernet
Port WWN is 21:90:00:0d:ec:02:64:80
Peer port WWN is 20:14:00:0d:ec:08:5f:c0
Admin port mode is auto, trunk mode is on
Trunk vsans (admin allowed and active) (1,100,200,302-303,999,3001-3060)
Trunk vsans (isolated) (100,200,302-303,999,3001-3060)
Trunk vsans (initializing) ()
Using Profile id 1 (interface GigabitEthernet7/1)
Peer Internet address is 10.10.100.232 and port is 3225
FCIP tunnel is protected by IPSec
Write acceleration mode is off
Tape acceleration mode is off
Tape Accelerator flow control buffer size is automatic
IP Compression is disabled
Special Frame is disabled
Maximum number of TCP connections is 2
QOS control code point is 0
TCP Connection Information
Control connection:Local 10.10.100.231:3225, Remote 10.10.100.232:65492
Data connection:Local 10.10.100.231:3225, Remote 10.10.100.232:65494
20 Attempts for active connections, 0 close of connections
Current retransmission timeout is 200 ms
Round trip time:Smoothed 2 ms, Variance:3
Advertized window:Current:118 KB, Maximum:14 KB, Scale:6
Peer receive window:Current:128 KB, Maximum:128 KB, Scale:6
Congestion window:Current:14 KB, Slow start threshold:204 KB
Current Send Buffer Size:14 KB, Requested Send Buffer Size:0 KB
5 minutes input rate 2960 bits/sec, 370 bytes/sec, 4 frames/sec
5 minutes output rate 3184 bits/sec, 398 bytes/sec, 4 frames/sec
3628 frames input, 340644 bytes
3610 Class F frames input, 338396 bytes
18 Class 2/3 frames input, 2248 bytes
0 Error frames timestamp error 0
3624 frames output, 359140 bytes
3608 Class F frames output, 357332 bytes
16 Class 2/3 frames output, 1808 bytes
MDSC# show interface fcip 1
Hardware is GigabitEthernet
Port WWN is 20:14:00:0d:ec:08:5f:c0
Peer port WWN is 21:90:00:0d:ec:02:64:80
Admin port mode is auto, trunk mode is on
Trunk vsans (admin allowed and active) (1)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
Using Profile id 1 (interface GigabitEthernet1/2)
Peer Internet address is 10.10.100.231 and port is 3225
FCIP tunnel is protected by IPSec
Write acceleration mode is off
Tape acceleration mode is off
Tape Accelerator flow control buffer size is automatic
IP Compression is disabled
Special Frame is disabled
Maximum number of TCP connections is 2
QOS control code point is 0
TCP Connection Information
Control connection:Local 10.10.100.232:65492, Remote 10.10.100.231:3225
Data connection:Local 10.10.100.232:65494, Remote 10.10.100.231:3225
22 Attempts for active connections, 1 close of connections
Current retransmission timeout is 200 ms
Round trip time:Smoothed 2 ms, Variance:3
Advertized window:Current:128 KB, Maximum:14 KB, Scale:6
Peer receive window:Current:118 KB, Maximum:118 KB, Scale:6
Congestion window:Current:15 KB, Slow start threshold:204 KB
Current Send Buffer Size:14 KB, Requested Send Buffer Size:0 KB
5 minutes input rate 3192 bits/sec, 399 bytes/sec, 4 frames/sec
5 minutes output rate 2960 bits/sec, 370 bytes/sec, 4 frames/sec
3626 frames input, 359324 bytes
3610 Class F frames input, 357516 bytes
16 Class 2/3 frames input, 1808 bytes
0 Error frames timestamp error 0
3630 frames output, 340828 bytes
3612 Class F frames output, 338580 bytes
18 Class 2/3 frames output, 2248 bytes
Verifying Security Associations
To verify security associations (SAs), follow these steps:
Step 1
Issue the show crypto sad domain ipsec command to verify the current peer, mode, and the inbound and outbound index of each switch. The example command outputs follow:
MDSA# show crypto sad domain ipsec
interface:GigabitEthernet7/1
Crypto map tag:cmap-01, local addr. 10.10.100.231
local ident (addr/mask):(10.10.100.231/255.255.255.255)
remote ident (addr/mask):(10.10.100.232/255.255.255.255)
current_peer:10.10.100.232
local crypto endpt.:10.10.100.231, remote crypto endpt.:10.10.100.232
mode:tunnel, crypto algo:esp-3des, auth algo:esp-md5-hmac
current outbound spi:0x822a202 (136487426), index:1
lifetimes in seconds::3600
lifetimes in bytes::483183820800
current inbound spi:0x38147002 (940863490), index:1
lifetimes in seconds::3600
lifetimes in bytes::483183820800
MDSC# show crypto sad domain ipsec
interface:GigabitEthernet1/2
Crypto map tag:cmap-01, local addr. 10.10.100.232
local ident (addr/mask):(10.10.100.232/255.255.255.255)
remote ident (addr/mask):(10.10.100.231/255.255.255.255)
current_peer:10.10.100.231
local crypto endpt.:10.10.100.232, remote crypto endpt.:10.10.100.231
mode:tunnel, crypto algo:esp-3des, auth algo:esp-md5-hmac
current outbound spi:0x38147002 (940863490), index:513
lifetimes in seconds::3600
lifetimes in bytes::483183820800
current inbound spi:0x822a202 (136487426), index:513
lifetimes in seconds::3600
lifetimes in bytes::483183820800
Step 2
The SA index can be used to look at the SA in the crypto-accelerator. Issue the show ipsec internal crypto-accelerator interface gigabitethernet slot/port sad [inbound | outbound] sa-index command to display the inbound or outbound SA information. The hard limit bytes and soft limit bytes fields display the lifetime in bytes. The hard limit expiry secs and the soft limit expiry secs fields display the lifetime in seconds.
Note
To issue commands with the internal keyword, you must have an account that is a member of the network-admin group.
The example command outputs follow:
MDSA# show ipsec internal crypto-accelerator interface gigabitethernet 7/1 sad inbound 1
sw172.22.48.91# show ipsec internal crypto-accelerator interface gigabitethernet 7/1 sad
inbound 1
Mode :Tunnel, flags:0x492300000000000
Encrypt algorithm is DES/3DES
Source ip address 10.10.100.232/255.255.255.255
Destination ip address 10.10.100.231/255.255.255.255
Physical port 0, mask:0x1
Source port no 0 mask:0x0
Hard limit 483183820800 bytes
Soft limit 401042571264 bytes
SA byte count 845208 bytes <----Elapsed traffic
SA user byte count 845208 bytes <----Elapsed traffic
Error count:auth:0, pad:0, replay:0
Hard limit expiry 1100652419 secs (since January 1, 1970), remaining 219 7 secs
Soft limit expiry 1100652386 secs (since January 1, 1970), remaining 216 4 secs
Antireplay window:0xffffffff.0xffffffff.0xffffffff.0xffffffff
MDSC# show ipsec internal crypto-accelerator interface gigabitethernet 1/2 sad inbound 513
Mode :Tunnel, flags:0x492300000000000
Encrypt algorithm is DES/3DES
Source ip address 10.10.100.231/255.255.255.255
Destination ip address 10.10.100.232/255.255.255.255
Physical port 1, mask:0x1
Source port no 0 mask:0x0
Hard limit 483183820800 bytes
Soft limit 420369924096 bytes
SA byte count 873056 bytes <----Elapsed traffic
SA user byte count 873056 bytes <----Elapsed traffic
Error count:auth:0, pad:0, replay:0
Packet count 7137
Hard limit expiry 1100652419 secs (since January 1, 1970), remaining 214 1 secs
Soft limit expiry 1100652394 secs (since January 1, 1970), remaining 211 6 secs
Antireplay window:0xffffffff.0xffffffff.0xffffffff.0xffffffff
MDSA# show ipsec internal crypto-accelerator interface gigabitethernet 7/1 sad outbound 1
SPI 136487426 (0x822a202), MTU 1400, MTU_delta 4
Mode :Tunnel, flags:0x92100000000000
Tunnel options index:0, ttl:0x40, flags:0x1
Encrypt algorithm is DES/3DES
Tunnel source ip address 10.10.100.231
Tunnel destination ip address 10.10.100.232
Hard limit 483183820800 bytes
Soft limit 376883380224 bytes
SA byte count 874544 bytes <----Elapsed traffic
SA user byte count 874544 bytes <----Elapsed traffic
Hard limit expiry 1100652419 secs (since January 1, 1970), remaining 208 9 secs
Soft limit expiry 1100652384 secs (since January 1, 1970), remaining 205 4 secs
Outbound MAC table index:1
MDSC# show ipsec internal crypto-accelerator interface gigabitethernet 1/2 sad outbound
513
SPI 940863490 (0x38147002), MTU 1400, MTU_delta 4
Mode :Tunnel, flags:0x92100000000000
Tunnel options index:0, ttl:0x40, flags:0x1
Encrypt algorithm is DES/3DES
Tunnel source ip address 10.10.100.232
Tunnel destination ip address 10.10.100.231
Hard limit 483183820800 bytes
Soft limit 449360953344 bytes
SA byte count 855648 bytes <----Elapsed traffic
SA user byte count 855648 bytes <----Elapsed traffic
Hard limit expiry 1100652419 secs (since January 1, 1970), remaining 206 4 secs
Soft limit expiry 1100652397 secs (since January 1, 1970), remaining 204 2 secs
Outbound MAC table index:125
Security Associations Do Not Re-Key
A lifetime counter (in seconds and bytes) is maintained as soon as an SA is created. When the time limit expires, the SA is no longer operational and is automatically renegotiated (re-keyed) if traffic is present. If there is no traffic, the SA will not be re-keyed and the tunnel will go down.
The re-key operation starts when the soft lifetime expires. That happens approximately 20 to 30 seconds before the time-based lifetime expires, or when approximately 10 to 20 percent of the bytes are remaining in the bytes-based lifetime.
To troubleshoot this problem, follow these steps:
Step 1
Verify that traffic was flowing when the soft SA lifetime expired.
Step 2
Verify that the configurations are still compatible.
Clearing Security Associations
To clear a specific SA, obtain the SA index value and issue the clear crypto sa domain ipsec interface gigabitethernet slot/port outbound sa-index command.
To obtain the SA index value, issue the show crypto sad domain ipsec command.
Debugging the IPsec Process
To get debug messages printed on the console, the following debugs can be enabled for IPsec:
•
debug ipsec error for error messages.
•
debug ipsec warning for warning messages.
•
debug ipsec config for configuration messages.
•
debug ipsec flow for SA related messages.
Debugging the IKE Process
The following commands show the internal state of the IKE process:
•
show crypto ike domain ipsec initiator
•
show crypto ike domain ipsec sa
Obtaining Statistics from the IPsec Process
To obtain statistics from the IPsec process, issue the show crypto global domain ipsec command and the show crypto global domain ipsec interface gigabitethernet slot/port command. The show crypto global domain ipsec command output displays statistics for all SAs. Example command output follows:
MDSA# show crypto global domain ipsec
Number of crypto map sets:1
IKE transaction stats:0 num, 64 max
The show crypto global domain ipsec interface gigabitethernet slot/port command output displays interface level statistics. Example command output follows:
MDSA# show crypto global domain ipsec interface gigabitethernet 7/1
IPSec interface statistics:
IKE transaction stats:0 num
Inbound SA stats:1 num, 512 max
Outbound SA stats:1 num, 512 max