Troubleshooting
This chapter describes techniques that you can use to troubleshoot the operation of the VSPA in a Catalyst 6500 Series switch.
It includes the following sections:
•General Troubleshooting Information
•Monitoring the VSPA
•Troubleshooting Specific Problems on the VSPA
•Using Crypto Conditional Debug
•Preparing for Online Insertion and Removal of a VSPA or SSC-600
Note For detailed information on Cisco IOS IPsec cryptographic operations and policies, the Cisco IOS Security Configuration Guide, Release 12.2 at this URL:
http://www.cisco.com/en/US/docs/ios/12_2/security/configuration/guide/fsecur_c.html
For more information about the commands used in this chapter, see the Cisco IOS Security Command Reference at this URL:
http://www.cisco.com/en/US/docs/ios/security/command/reference/sec_book.html
Also refer to the related Cisco IOS Release 12.2 software configuration guide, command reference, and master index publications. For more information about accessing these publications, see the "Related Documentation" section on page xvi.
General Troubleshooting Information
This section describes general information for troubleshooting the VSPA and the SSC-600. It includes the following sections:
•Interpreting Console Error Messages
•Using debug Commands
•Using show Commands
•Using Online Diagnostics
Interpreting Console Error Messages
The Catalyst 6500 Series switch can generate error messages and other system messages to inform the operator of events that might require attention. These messages can be displayed on the console, or sent to a logging host using the System Logging (Syslog) protocol or Simple Network Management Protocol (SNMP).
System error messages are organized in the documentation according to the particular system facility that produces the messages. The VSPA and SSC-600 use the following facility names in error messages:
•VSPA—WS_IPSEC_3 (also VSPA, VPNSPA)
•SSC-600—CAT6000_SSC (also C7600_SSC600)
To view the explanations and recommended actions for Catalyst 6500 Series switch error messages, including messages related to service modules, refer to the following documents:
•Cisco IOS Release 12.2SX System Message Guide at this URL:
http://www.cisco.com/en/US/docs/ios/12_2sx/system/messages/122sxsms.html
•System Messages for 12.2S (for error messages in Release 12.2S) at this URL:
http://www.cisco.com/en/US/docs/ios/system/messages/guide/consol_smg.html
Using debug Commands
For information about debug commands specific to the Cisco IOS software release 12.2SX, see the Cisco IOS Master Command List, Release 12.2SX at this URL:
http://www.cisco.com/en/US/docs/ios/mcl/122sxmcl/12_2sx_mcl_book.html
Caution
Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use
debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support personnel. Moreover, it is best to use
debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased
debug command processing overhead will affect system use.
For information about available crypto conditional debugging commands, see the "Using Crypto Conditional Debug" section.
For more information about other debug commands that can be used on a Catalyst 6500 Series switch, see the Cisco IOS Debug Command Reference, Release 12.2 at this URL:
http://www.cisco.com/en/US/docs/ios/12_2/debug/command/reference/122debug.html
Using show Commands
There are several show commands that you can use to monitor and troubleshoot modules on the Catalyst 6500 Series switch. This chapter describes using the show hw-module slot command to perform troubleshooting of your VSPA.
For more information about using show commands to verify and monitor your VSPA, see the "Displaying the Module Hardware Type" section on page 1-10.
For more information about security-related show commands, see the Cisco IOS Security Command Reference at this URL:
http://www.cisco.com/en/US/docs/ios/12_2/security/command/reference/fsecur_r.html
Using Online Diagnostics
Using the online diagnostics features provided by the Catalyst 6500 Series switch, you can test and verify the hardware functionality of the VSPA at bootup, as part of background health monitoring, or on demand.
Note In Cisco IOS Release 12.2(33)SXI, a bootup diagnostics failure in the VSPA results in a restart of the VSPA and the SSC-600. Health monitoring failures exceeding a certain threshold result in a syslog message and a restart of the VSPA.
In Cisco IOS Release 12.2(33)SXI1 and later releases, a bootup diagnostics failure in the VSPA results in a restart of only the VSPA. Health monitoring failures exceeding a certain threshold result in a syslog message.
For more information about online diagnostics, see the Catalyst 6500 Release 12.2SXH and Later Software Configuration Guide at this URL:
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/diags.html
Monitoring the VSPA
This section describes commands that can be used to display information about the VSPA hardware and configuration. It consists of the following subsections:
•Displaying VSPA Hardware and System Information
•Displaying VSPA Configuration Information
Displaying VSPA Hardware and System Information
To display hardware and system information, use the following commands:
•show diagbus, show module, show crypto eli—See the "Displaying Information About VSPA Ports" section.
•show crypto engine accelerator statistic slot—See the "Displaying Platform and Network Interface Controller Statistics for the VSPA" section.
•show hw-module slot fpd—See the "Displaying Information About Hardware Revision Levels" section.
Displaying Information About VSPA Ports
To display information about the type of modules that are installed in the switch, use the show diagbus command.
The following example shows output from the show diagbus command on a Catalyst 6500 Series switch with a VSPA installed in subslot 1 of a SSC-600 that is installed in slot 5:
2-subslot Services SPA Carrier-600 controller
Board is analyzed ipc ready
HW rev 0.3, board revision A01
Serial Number: abc Part number: 73-6348-01
Slot database information:
Flags: 0x2004 Insertion time: 0x3DB5F4BC (4d20h ago)
256 MBytes Total on Board SDRAM
IOS (tm) cwlc Software (smsc-DWDBG-M), Experimental Version 12.2(20050623:231413)
subslot 5/1: WS-IPSEC-3 (0x3D7), status: ok
For information about the show module and show crypto eli commands, see the "Displaying the Module Hardware Type" section on page 1-10.
Displaying Platform and Network Interface Controller Statistics for the VSPA
To display platform statistics, and, optionally, network interface controller statistics, use the show crypto engine accelerator statistic slot command.
The following example shows output from the show crypto engine accelerator statistic command on a Catalyst 6500 Series switch with a VSPA in subslot 0 of a SSC-600 that is installed in slot 1. The output displays platform statistics for the VSPA and also displays the network interface controller statistics.
Router# show crypto engine accelerator statistic slot 1/0 detail
Decryption Side Data Path Statistics
====================================
Packets RX...............: 454260
Packets TX...............: 452480
IPSec Transport Mode.....: 0
IPSec Tunnel Mode........: 452470
AH Packets...............: 0
ESP Packets..............: 452470
GRE Decapsulations.......: 0
NAT-T Decapsulations.....: 0
Clear....................: 8
ICMP.....................: 0
Packets Drop.............: 193
Authentication Errors....: 0
Decryption Errors........: 0
Replay Check Failed......: 0
Policy Check Failed......: 0
Illegal CLear Packet.....: 0
GRE Errors...............: 0
SPD Errors...............: 0
HA Standby Drop..........: 0
Hard Life Drop...........: 0
Invalid SA...............: 191
SPI No Match.............: 0
Destination No Match.....: 0
Protocol No Match........: 0
Reassembly Frag RX.......: 0
IPSec Fragments..........: 0
IPSec Reasm Done.........: 0
Clear Fragments..........: 0
Clear Reasm Done.........: 0
Datagrams Drop...........: 0
Fragments Drop...........: 0
Decryption Side Controller Statistics
=====================================
Frames RX................: 756088
Bytes RX.................: 63535848
Mcast/Bcast Frames RX....: 2341
RX Less 128Bytes.........: 756025
RX Less 512Bytes.........: 58
RX Less 1KBytes..........: 2
RX Less 9KBytes..........: 3
RX Frames Drop...........: 0
Frames TX................: 452365
Bytes TX.................: 38001544
Mcast/Bcast Frames TX....: 9
TX Less 128Bytes.........: 452343
TX Less 512Bytes.........: 22
TX Less 1KBytes..........: 0
TX Less 9KBytes..........: 0
Encryption Side Data Path Statistics
====================================
Packets RX...............: 756344
Packets TX...............: 753880
IPSec Transport Mode.....: 0
IPSec Tunnel Mode........: 753869
GRE Encapsulations.......: 0
NAT-T Encapsulations.....: 0
LAF prefragmented........: 0
Fragmented...............: 0
Clear....................: 753904
ICMP.....................: 0
Packets Drop.............: 123
IKE/TED Drop.............: 27
Authentication Errors....: 0
Encryption Errors........: 0
HA Standby Drop..........: 0
Hard Life Drop...........: 0
Invalid SA...............: 191
Reassembly Frag RX.......: 0
Clear Fragments..........: 0
Clear Reasm Done.........: 0
Datagrams Drop...........: 0
Fragments Drop...........: 0
Encryption Side Controller Statistics
=====================================
Frames RX................: 454065
Bytes RX.................: 6168274/
Mcast/Bcast Frames RX....: 1586
RX Less 128Bytes.........: 1562
RX Less 512Bytes.........: 452503
RX Less 1KBytes..........: 0
RX Less 9KBytes..........: 0
RX Frames Drop...........: 0
Frames TX................: 753558
Bytes TX.................: 100977246
Mcast/Bcast Frames TX....: 2
TX Less 128Bytes.........: 3
TX Less 512Bytes.........: 753555
TX Less 1KBytes..........: 0
TX Less 9KBytes..........: 0
Displaying Information About Hardware Revision Levels
To display information about the hardware revision of the SSC-600 and the VSPA as well as the version of the field-programmable devices (FPDs) that are on the carrier card and the module, use the show hw-module slot fpd command. Cisco technical engineers might need this information to debug or troubleshoot problems with a module installation.
The following example shows output from the show hw-module slot command on a Catalyst 6500 Series switch with a VSPA installed in subslot 0 of an SSC-600 that is installed in slot 6:
Router# show hw-module slot 2 fpd
==== ====================== ====== =============================================
H/W Field Programmable Current Min. Required
Slot Card Type Ver. Device: "ID-Name" Version Version
==== ====================== ====== ================== =========== ==============
2 WS-SSC-600 0.302 1-SSC-600 I/O FPGA 1.0 1.0
2-SSC-600 DPRX FPG 0.9 0.9
3-SSC-600 DPTX FPG 0.13 0.13
---- ---------------------- ------ ------------------ ----------- --------------
2/0 WS-IPSEC-3 0.29 1-BOOTLOADER 1.1 1.1
==== ====================== ====== =============================================
Displaying VSPA Configuration Information
Except where noted, examples of command output in this section use the following example topology:
loopback -------- Router1 ------------------------ Router2 -------- loopback
12.0.0.0 10.10.40.0 13.0.0.0
To display information about the VSPA configuration, use the following commands:
•show crypto vlan—See the "Displaying Information About Access and Routed Ports That Are Connected" section, "Displaying the VPN Running State" section, and "Displaying Information About IP Multicast Over a GRE Tunnel" section.
•show interfaces trunk—See the "Displaying Information About the VLANs Allowed by a Trunk Port" section.
•show crypto isakmp policy—See the "Displaying Information About IKE Policies" section.
•show crypto ipsec transform-set—See the "Displaying Information About IPsec Transform Sets" section.
•show crypto map—See the "Displaying Information About Crypto Maps" section.
•show crypto isakmp sa—See the "Displaying Information About SAs at a Peer" section.
•show crypto ipsec sa—See the "Displaying Information About IPsec Security Associations" section.
•show crypto key mypubkey rsa—See the "Displaying Information About RSA Public Keys" section.
•show crypto key pubkey-chain rsa—See the "Displaying Information About RSA Public Keys" section.
•show crypto pki certificates—See the "Displaying Information About Certificates" section.
•show crypto pki trustpoints—See the "Displaying Information About Trustpoints" section.
•show crypto session—See the "Displaying Information About Crypto Sessions" section.
•show interfaces tunnel—See the "Displaying Tunnel Interface Information" section.
•show redundancy linecard-group—See the "Displaying Information About a BFG Configuration" section.
•show crypto ace redundancy—See the "Displaying Information About a BFG Configuration" section.
•show ip nhrp—See the "Displaying Information About the NHRP Cache" section.
For a detailed description of the information displayed by the show commands, refer to the "IP Security and Encryption" chapter of the Cisco IOS Security Command Reference.
Displaying Information About Access and Routed Ports That Are Connected
To verify that an access or routed port is connected, use the show crypto vlan command. The following is sample output from the command when a port is crypto-connected to a port VLAN:
Router1# show crypto vlan
Interface VLAN 2 on IPSec Service Module port GigabitEthernet2/0/1 connected to VLAN 502
with crypto map set mymap1
The following is sample output from the command when a port is crypto-connected to a physical interface:
Router1# show crypto vlan
Interface VLAN 2 on IPSec Service Module port GigabitEthernet2/0/1 connected to Gi3/2 with
crypto map set mymap2
Displaying Information About the VLANs Allowed by a Trunk Port
To display information about the VLANs allowed by a trunk port, use the show interfaces trunk command. The following is sample output from the command:
Router1# show interfaces trunk
Port Mode Encapsulation Status Native vlan
Gi2/0/1 on 802.1q trunking 1
Gi2/0/2 on 802.1q trunking 1
Port Vlans allowed on trunk
Port Vlans allowed and active in management domain
Port Vlans in spanning tree forwarding state and not pruned
Displaying the VPN Running State
To display the VPN running state, use the show crypto vlan command.
In the following example, the interface VLAN belongs to the VSPA inside port:
Router1# show crypto vlan
Interface VLAN 2 on IPSec Service Module port GigabitEthernet2/0/1 connected to Gi3/2
In the following example, either the interface VLAN is missing on the VSPA inside port, the VSPA is removed from the chassis, or the VSPA was moved to a different subslot:
Router1# show crypto vlan
Interface VLAN 2 connected to VLAN 3 (no IPSec Service Module attached)
Displaying Information About IKE Policies
To display information about IKE policies, use the show crypto isakmp policy command. The following is sample output from the command:
Router1# show crypto isakmp policy
Protection suite of priority 1
encryption algorithm: Three key triple DES
hash algorithm: Secure Hash Standard
authentication method: Rivest-Shamir-Adleman Signature
Diffie-Hellman group: #1 (768 bit)
lifetime: 86400 seconds, no volume limit
encryption algorithm: DES - Data Encryption Standard (56 bit keys).
hash algorithm: Secure Hash Standard
authentication method: Rivest-Shamir-Adleman Signature
Diffie-Hellman group: #1 (768 bit)
lifetime: 86400 seconds, no volume limit
Note If a user enters an IKE encryption method that the hardware does not support, a warning message will be displayed in the show crypto isakmp policy command output:
WARNING:encryption hardware does not support the configured encryption method for ISAKMP policy
value
Displaying Information About IPsec Transform Sets
To display information about transform set configurations, use the show crypto ipsec transform-set command. The following is sample output from the command:
Router1# show crypto ipsec transform-set
Transform set tset: { esp-3des esp-sha-hmac }
will negotiate = { Tunnel, },
Note If a user enters an IPsec transform that the hardware (the IPsec peer) does not support, a warning message will be displayed in the show crypto ipsec transform-set command output:
WARNING:encryption hardware does not support transform.
Displaying Information About Crypto Maps
To display information about crypto map configurations, use the show crypto map command. The following is sample output from the command:
Crypto Map "mymap1" 10 ipsec-isakmp
Extended IP access list 101
access-list 101 permit ip host 12.0.0.1 host 13.0.0.1
Security association lifetime: 4608000 kilobytes/3600 seconds
Interfaces using crypto map test:
using crypto engine WS-IPSEC-3[2/0]
Displaying Information About IPsec Security Associations
To display information about IPsec security associations, use the show crypto ipsec sa command.
Note When you first enter the show crypto ipsec sa command, the packet counters will not show the correct values. Subsequent instances of the command will display the correct values.
The following is sample output from the command:
Router1# show crypto ipsec sa
Crypto map tag: mymap1, local addr 10.10.40.1
local ident (addr/mask/prot/port): (12.0.0.1/255.255.255.255/0/0)
remote ident (addr/mask/prot/port): (13.0.0.1/255.255.255.255/0/0)
current_peer 10.10.40.2 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
#pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0
local crypto endpt.: 10.10.40.1, remote crypto endpt.: 10.10.40.2
path mtu 1500, ip mtu 1500
current outbound spi: 0xDAADD709(3668825865)
PFS (Y/N): N, DH group: none
spi: 0x7D99ACA4(2107223204)
transform: esp-3des esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2001, flow_id: :1, sibling flags 80000040, crypto map: test
sa timing: remaining key lifetime (k/sec): (4391325/3403)
replay detection support: Y
spi: 0xDAADD709(3668825865)
transform: esp-3des esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 2002, flow_id: :2, sibling flags 80000040, crypto map: test
sa timing: remaining key lifetime (k/sec): (4391325/3403)
replay detection support: Y
Displaying Information About SAs at a Peer
To display information about all current IKE SAs at a peer, use the show crypto isakmp sa command. The following is sample output from the command:
Router1# show crypto isakmp sa
dst src state conn-id slot status
10.10.40.2 10.10.40.1 QM_IDLE 68001 ACTIVE
Displaying Information About RSA Public Keys
To display information the RSA public keys configured for your switch, use the show crypto key mypubkey rsa command. The following is sample output from the command:
Router1# show crypto key mypubkey rsa
% Key pair was generated at: 13:43:52 PST Jan 15 2009
Key name: router1.example.com
Storage Device: not specified
Usage: General Purpose Key
30820122 300D0609 2A864886 F70D0101 01050003 82010F00 3082010A 02820101
00D6D5A6 2D42D542 805FAAFB 65B72A10 21AA7ACC F86D6C5A 7E290221 0EEC370B
FF790FAC AE654451 E038BEAE 7AD85C45 75A2E5D3 95866F6C 6C888C1F 197C3BC3
3EB02714 A5F43640 D1951900 053F080E 99B547A7 A317C852 BF9B2D91 20D3BABC
6EA39317 FF22EF39 308F9838 1F395A56 F04473F7 52D41353 3C294E44 B7BDC8A0
7BB82EBF 405D7853 1E26F8DA 3251FD9F 072E8697 D131F3E2 042D8F4F 0D421361
846FCB1F B266E0AB 3B2AB6E6 1FA5927A B6E5C29E CFFA77C0 EF07831E D060A950
C6769919 E2C54267 226FC037 DF996B5D 04D0F0DD 5C0340D4 F8F17CB7 D74531E8
BF378EFA EADB73A8 7F02370A EE4AA992 F2B7A9D0 DA0CFB9C 2B3C3F2C 79B854EA
% Key pair was generated at: 13:43:53 PST Jan 15 2009
Key name: router1.example.com.server
307C300D 06092A86 4886F70D 01010105 00036B00 30680261 00C18A19 1B4BE742
31692B90 FD290B6A CC085CF8 2E9591F0 9B70951C 07E7942B 55AB8540 2BCDEF8D
A17629D9 8C9369CD AB693089 E4DA150F 6ADF7CFB 78197CF3 84FBC959 90F37EC5
83B0136D 1A93ACF4 040CD25F 22D7428E 7950E0ED AB45B011 DD020301 0001
To display a list of all the RSA public keys stored on your switch (including the public keys of peers that have sent your switch their certificates during peer authentication for IPsec), or to display details of a particular RSA public key stored on your switch, use the show crypto key pubkey-chain rsa command. The following is sample output from the command:
Router1# show crypto key pubkey-chain rsa
Codes: M - Manually configured, C - Extracted from certificate
Code Usage IP-Address/VRF Keyring Name
C Signing default X.500 DN name:
C Signing default router1.example.com
Displaying Information About Certificates
To display information about your certificate, the certificate of the CA, and any RA certificates, use the show crypto pki certificates command. The following is sample output from the command:
Router1# show crypto pki certificates
Certificate Serial Number: 18FBAA1C000000000062
Certificate Usage: General Purpose
Name: router1.example.com
hostname=router1.example.com
http://caserver/CertEnroll/mscavpn1.crl
start date: 13:41:42 PST Jan 15 2009
end date: 13:51:42 PST Jan 15 2010
Associated Trustpoints: MSCA
Certificate Serial Number: 30EF1D0A2F5B98A44B0AE1BF562636EC
Certificate Usage: Signature
http://caserver/CertEnroll/mscavpn1.crl
start date: 16:12:15 PST Dec 22 2008
end date: 16:19:24 PST Dec 22 2013
Associated Trustpoints: MSCA
Displaying Information About Trustpoints
To display the trustpoints that are configured in the switch, use the show crypto pki trustpoints command. The following is sample output from the command:
Router1# show crypto pki trustpoints
Serial Number: 30EF1D0A2F5B98A44B0AE1BF562636EC
SCEP URL: http://43.0.111.111:80/certsrv/mscep/mscep.dll
Displaying Information About Crypto Sessions
To display status information for active crypto sessions, use the show crypto session command. The output will include the following:
•Interface
•IKE peer description, if available
•IKE SAs that are associated with the peer by which the IPsec SAs are created
•IPsec SAs serving the flows of a session
The following is sample output from the command:
Router1# show crypto session
Crypto session current status
Session status: UP-ACTIVE
Peer: 10.10.40.2 port 500
IKE SA: local 10.10.40.1/500 remote 10.10.40.2/500 Active
IPSEC FLOW: permit ip host 12.0.0.1 host 13.0.0.1
Active SAs: 2, origin: crypto map
The following is sample output from the command using the
detail keyword:
Router1# show crypto session detail
Crypto session current status
Code: C - IKE Configuration mode, D - Dead Peer Detection
K - Keepalives, N - NAT-traversal, T - cTCP encapsulation
X - IKE Extended Authentication
Session status: UP-ACTIVE
Peer: 10.10.40.2 port 500 fvrf: (none) ivrf: (none)
Phase1_id: router2.example.com
IKE SA: local 10.10.40.1/500 remote 10.10.40.2/500 Active
Capabilities:(none) connid:68001 lifetime:23:59:27
IPSEC FLOW: permit ip host 12.0.0.1 host 13.0.0.1
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 9 drop 0 life (KB/Sec) 4398172/3567
Outbound: #pkts enc'ed 9 drop 0 life (KB/Sec) 4398172/3567
Displaying Tunnel Interface Information
To display tunnel interface information, use the show interfaces tunnel command. The following is sample output from the command:
Router# show interfaces tunnel 1
Tunnel4 is up, line protocol is down
Hardware is Routing Tunnel
Internet address is 10.1.1.1/24
MTU 1514 bytes, BW 9 Kbit, DLY 500000 usec, rely 255/255, load 1/255
Encapsulation TUNNEL, loopback not set
Tunnel source 9.2.2.1, destination 6.6.6.2
Tunnel protocol/transport GRE/IP, key disabled, sequencing disabled
Tunnel TOS 0xF, Tunnel TTL 128
Checksumming of packets disabled, fast tunneling enabled
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Output queue 0/0, 1 drops; input queue 0/75, 0 drops
30 second input rate 0 bits/sec, 0 packets/sec
30 second output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets, 0 restarts
Note This sample output does not use the example topology shown at the start of the "Displaying VSPA Configuration Information" section.
Table 13-1 describes significant fields shown in the display.
Table 13-1 show interfaces tunnel Field Descriptions
|
|
Tunnel is {up | down} |
Interface is currently active and inserted into ring (up) or inactive and not inserted (down). |
line protocol is {up | down | administratively down} |
Shows line protocol up if a valid route is available to the tunnel destination. Shows line protocol down if no route is available, or if the route would be recursive. |
Hardware |
Specifies the hardware type. |
MTU |
Maximum transmission unit of the interface. |
BW |
Bandwidth of the interface in kilobits per second. |
DLY |
Delay of the interface in microseconds. |
rely |
Reliability of the interface as a fraction of 255 (255/255 is 100 percent reliability), calculated as an exponential average over 5 minutes. |
load |
Load on the interface as a fraction of 255 (255/255 is completely saturated), calculated as an exponential average over 5 minutes. |
Encapsulation |
Encapsulation method is always TUNNEL for tunnels. |
loopback |
Indicates whether loopback is set or not. |
Keepalive |
Indicates whether keepalives are set or not. |
Tunnel source |
IP address used as the source address for the tunnel packets. |
destination |
IP address of the tunnel destination. |
Tunnel protocol |
Tunnel transport protocol (the protocol the tunnel is using). This is based on the tunnel mode command, which defaults to GRE. |
key |
(Optional) ID key for the tunnel interface. |
sequencing |
(Optional) Indicates whether the tunnel interface drops datagrams that arrive out of order. |
Last input |
Number of hours, minutes, and seconds (or never) since the last packet was successfully received by an interface and processed locally on the switch. Useful for knowing when a dead interface failed. This field is not updated by fast-switched traffic. |
output |
Number of hours, minutes, and seconds (or never) since the last packet was successfully transmitted by an interface. |
output hang |
Number of hours, minutes, and seconds (or never) since the interface was last reset because of a transmission that took too long. When the number of hours in any of the "last" fields exceeds 24 hours, the number of days and hours is displayed. If that field overflows, asterisks are displayed. |
Last clearing |
Time at which the counters that measure cumulative statistics (such as number of bytes transmitted and received) shown in this report were last reset to zero. Note that variables that might affect routing (for example, load and reliability) are not cleared when the counters are cleared. Three asterisks (***) indicate the elapsed time is too large to be displayed. 0:00:00 indicates the counters were cleared more than 231 ms (and less than 232 ms) ago. |
Output queue, drops Input queue, drops |
Number of packets in output and input queues. Each number is followed by a slash, the maximum size of the queue, and the number of packets dropped because of a full queue. |
30 second input rate, 30 second output rate |
Average number of bits and packets transmitted per second in the last 30 seconds. The 30-second input and output rates should be used only as an approximation of traffic per second during a given 30-second period. These rates are exponentially weighted averages with a time constant of 30 seconds. A period of four time constants must pass before the average will be within two percent of the instantaneous rate of a uniform stream of traffic over that period. |
packets input |
Total number of error-free packets received by the system. |
bytes |
Total number of bytes, including data and MAC encapsulation, in the error-free packets received by the system. |
no buffer |
Number of received packets discarded because there was no buffer space in the main system. Compare with ignored count. Broadcast storms on Ethernet networks and bursts of noise on serial lines are often responsible for no input buffer events. |
broadcasts |
Total number of broadcast or multicast packets received by the interface. |
runts |
Number of packets that are discarded because they are smaller than the minimum packet size of the medium. |
giants |
Number of packets that are discarded because they exceed the maximum packet size of the medium. |
CRC |
Cyclic redundancy checksum generated by the originating LAN station or far-end device does not match the checksum calculated from the data received. On a LAN, this usually indicates noise or transmission problems on the LAN interface or the LAN bus itself. A high number of CRCs is usually the result of a station transmitting bad data. |
frame |
Number of packets received incorrectly having a CRC error and a noninteger number of octets. |
overrun |
Number of times the serial receiver hardware was unable to hand received data to a hardware buffer because the input rate exceeded the receiver's ability to handle the data. |
ignored |
Number of received packets ignored by the interface because the interface hardware ran low on internal buffers. These buffers are different than the system buffers mentioned previously in the buffer description. Broadcast storms and bursts of noise can cause the ignored count to be increased. |
abort |
Illegal sequence of one bits on a serial interface. This usually indicates a clocking problem between the serial interface and the data link equipment. |
packets output |
Total number of messages transmitted by the system. |
bytes |
Total number of bytes, including data and MAC encapsulation, transmitted by the system. |
underruns |
Number of times that the far-end transmitter has been running faster than the near-end switch's receiver can handle. This may never be reported on some interfaces. |
output errors |
Sum of all errors that prevented the final transmission of datagrams out of the interface being examined. Note that this may not balance with the sum of the enumerated output errors, as some datagrams may have more than one error, and others may have errors that do not fall into any of the specifically tabulated categories. |
collisions |
Number of messages retransmitted because of an Ethernet collision. This usually is the result of an overextended LAN (Ethernet or transceiver cable too long, more than two repeaters between stations, or too many cascaded multiport transceivers). Some collisions are normal. However, if your collision rate climbs to around 4 or 5 percent, you should consider verifying that there is no faulty equipment on the segment and moving some existing stations to a new segment. A packet that collides is counted only once in output packets. |
interface resets |
Number of times an interface has been reset. The interface may be reset by the administrator or automatically when an internal error occurs. |
restarts |
Number of times that the controller was restarted because of errors. |
Displaying Information About a BFG Configuration
To display information about a BFG configuration, use the show redundancy linecard-group and show crypto ace redundancy commands. The following is sample output from the commands:
Router# show redundancy linecard-group 1
Line Card Redundancy Group:1 Mode:feature-card
Router1# show crypto ace redundancy
--------------------------------------
LC Redundancy Group ID : 1
Pending Configuration Transactions: 0
Current State : OPERATIONAL
Number of blades in the group : 2
--------------------------------------
Completed Bulk Synchronization
Initialization Timer not running
Completed Bulk Synchronization
Initialization Timer not running
Note This sample output does not use the example topology shown at the start of the "Displaying VSPA Configuration Information" section.
Displaying Information About the NHRP Cache
To display information about the Next Hop Resolution Protocol (NHRP) cache, use the show ip nhrp and the show crypto sockets commands. The following is sample output from the commands:
10.10.1.75/32 via 10.10.1.75, Tunnel5 created 00:32:11, expire 00:01:46
Type: dynamic, Flags: authoritative unique registered
NBMA address: 172.16.175.75
10.10.1.76/32 via 10.10.1.76, Tunnel5 created 00:26:41, expire 00:01:37
Type: dynamic, Flags: authoritative unique registered
NBMA address: 172.16.175.76
10.10.1.77/32 via 10.10.1.77, Tunnel5 created 00:31:26, expire 00:01:33
Type: dynamic, Flags: authoritative unique registered
NBMA address: 172.17.63.20
Router# show crypto sockets
Number of Crypto Socket connections 1
Tu0 Peers (local/remote): 9.1.1.1/11.1.1.1
Local Ident (addr/mask/port/prot): (9.1.1.1/255.255.255.255/0/47)
Remote Ident (addr/mask/port/prot): (11.1.1.1/255.255.255.255/0/47)
IPSec Profile: "MyIpsecProf"
Client: "TUNNEL SEC" (Client State: Active)
Crypto Sockets in Listen state:
Client: "TUNNEL SEC" Profile: "MyIpsecProf" Map-name: "Tunnel0-head-0"
Note This sample output does not use the example topology shown at the start of the "Displaying VSPA Configuration Information" section.
Displaying Information About IP Multicast Over a GRE Tunnel
To display information about an IP multicast over a GRE tunnel configuration, enter the show crypto vlan and show ip mroute commands.
Enter the show crypto vlan command to check that the tunnel has been taken over by the VSPA. The following is sample output from the command:
Interface VLAN 100 on IPSec Service Module port Gi7/0/1 connected to Po1 with crypto map
set map_t3
Tunnel15 is accelerated via IPSec SM in subslot 7/0
Enter the show ip mroute command and look for the H flag to check that the IP multicast traffic is hardware-switched. The following is sample output from the command:
Router# show ip mroute 230.1.1.5
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report, Z - Multicast Tunnel
Y - Joined MDT-data group, y - Sending to MDT-data group
Outgoing interface flags: H - Hardware switched, A - Assert winner
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 230.1.1.5), 01:23:45/00:03:16, RP 15.15.1.1, flags: SJC
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Tunnel15, Forward/Sparse-Dense, 00:25:47/00:03:16
(120.1.0.3, 230.1.1.5), 01:23:46/00:03:25, flags: T
Incoming interface: GigabitEthernet8/1, RPF nbr 0.0.0.0, RPF-MFD
Outgoing interface list:
Tunnel15, Forward/Sparse-Dense, 00:25:47/00:03:16, H
Note This sample output does not use the example topology shown at the start of the "Displaying VSPA Configuration Information" section.
Troubleshooting Specific Problems on the VSPA
This section provides additional information about troubleshooting specific problems related to the VSPA. It includes the following subsections:
•Clearing IPsec Security Associations
•Troubleshooting Trunk Port Configurations
•Troubleshooting a Blade Failure Group
•Troubleshooting IKE Policy and Transform Sets
Clearing IPsec Security Associations
You can clear (and reinitialize) IPsec security associations by using the clear crypto sa command.
Using the clear crypto sa command without parameters will clear out the full SA database, which will clear out active security sessions. You may also specify the peer, map, or entry keywords to clear out only a subset of the SA database. For more information, refer to the clear crypto sa command in the Cisco IOS Security Command Reference, Release 12.2.
If you want to also remove the IKE (phase 1) SAs, follow the clear crypto sa command with the clear crypto isa command. Alternatively, you can use the clear crypto session command to achieve the same result as the clear crypto sa and the clear crypto isa commands. The clear crypto session command supports many of the same parameters as the clear crypto sa command.
Troubleshooting Trunk Port Configurations
Caution
When you configure an Ethernet port as a trunk port, all the VLANs are allowed on the trunk port by default. This default configuration does not work well with the VSPA and causes network loops. To avoid this problem, you must explicitly specify only the desirable VLANs.
For more information on trunk configuration guidelines, review the "Configuring a Trunk Port" section on page 3-14.
To verify which ports are assigned to the VLAN, enter the show vlan id number command, using the interface VLAN identifier. Following is an example of a trunk port configuration and the output of the show vlan id command:
Router# show run interface gi 1/3
Building configuration...
Current configuration : 175 bytes
interface GigabitEthernet1/3
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,502-504,1002-1005
Interface VLAN 2 on IPSec Service Module port Gi7/0/1 connected to VLAN 502 with crypto
map set testtag_1
Interface VLAN 3 on IPSec Service Module port Gi7/0/1 connected to VLAN 503 with crypto
map set testtag_2
Interface VLAN 4 on IPSec Service Module port Gi7/0/1 connected to VLAN 504 with crypto
map set testtag_3
---- -------------------------------- --------- -------------------------------
2 VLAN0002 active Gi7/0/1
VLAN Type SAID MTU Parent RingNo BridgeNo Stp BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
2 enet 100002 1500 - - - - - 0 0
Primary Secondary Type Ports
------- --------- ----------------- ------------------------------------------
---- -------------------------------- --------- -------------------------------
502 VLAN0502 active Gi1/3, Gi7/0/2
VLAN Type SAID MTU Parent RingNo BridgeNo Stp BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
502 enet 100502 1500 - - - - - 0 0
Primary Secondary Type Ports
------- --------- ----------------- ------------------------------------------
Troubleshooting a Blade Failure Group
To enable VSPA debugging for a blade failure group, enter the debug crypto ace b2b command:
Router# debug crypto ace b2b
ACE B2B Failover debugging is on
Troubleshooting IKE Policy and Transform Sets
Any IPsec transforms or IKE encryption methods that the current hardware does not support should be disabled; they are ignored whenever an attempt to negotiate with the peer is made.
If a user enters an IPsec transform or an IKE encryption method that the hardware does not support, a warning message will be generated. These warning messages are also generated at boot time. When an encrypted card is inserted, the current configuration is scanned. If any IPsec transforms or IKE encryption methods are found that are not supported by the hardware, a warning message will be generated.
Using Crypto Conditional Debug
The crypto conditional debug feature provides three command-line interface (CLI) commands that allow you to debug an IP Security (IPsec) tunnel on the basis of predefined crypto conditions such as the peer IP address, connection-ID of a crypto engine, and security parameter index (SPI). By limiting debug messages to specific IPsec operations and reducing the amount of debug output, you can better troubleshoot a switch with a large number of tunnels.
The crypto conditional debug commands—debug crypto condition, debug crypto condition unmatched, and show crypto debug-condition—allow you to specify conditions (filter values) in which to generate and display debug messages related only to the specified conditions.
Table 13-2 lists the supported condition types.
Table 13-2 Supported Condition Types for Crypto Conditional Debug Commands
|
|
connid |
An integer between 1 and 32766. Relevant debug messages will be shown if the current IPsec operation uses this value as the connection-ID to interface with the crypto engine. |
flowid |
An integer between 1 and 32766. Relevant debug messages will be shown if the current IPsec operation uses this value as the flow-ID to interface with the crypto engine. |
fvrf |
The name string of a virtual private network (VPN) routing and forwarding (VRF) instance. Relevant debug messages will be shown if the current IPsec operation uses this VRF instance as its front-door VRF (FVRF). |
ivrf |
The name string of a VRF instance. Relevant debug messages will be shown if the current IPsec operation uses this VRF instance as its inside VRF (IVRF). |
peer group |
A Unity group name string. Relevant debug messages will be shown if the peer is using this group name as its identity. |
peer hostname |
A fully qualified domain name (FQDN) string. Relevant debug messages will be shown if the peer is using this string as its identity. |
peer ipv4 |
A single IP address. Relevant debug messages will be shown if the current IPsec operation is related to the IP address of this peer. |
peer subnet |
A subnet and a subnet mask that specify a range of peer IP addresses. Relevant debug messages will be shown if the IP address of the current IPsec peer falls into the specified subnet range. |
peer username |
A username string. Relevant debug messages will be shown if the peer is using this username as its identity. |
spi |
A 32-bit unsigned integer. Relevant debug messages will be shown if the current IPsec operation uses this value as the SPI. |
Note If connid, flowid, or spi is used as a debug condition, the debug messages for a related IPsec flow are generated. An IPsec flow has two connection-IDs, flow-IDs, and SPI values—one inbound and one outbound. Either one of the two connection-IDs, flow-IDs, and SPI values can be used as the debug condition that triggers debug messages for the IPsec flow.
Crypto Conditional Debug Configuration Guidelines and Restrictions
When configuring crypto conditional debug, follow these guidelines and restrictions:
•This feature does not support debug message filtering for hardware crypto engines.
•Although conditional debugging is useful for troubleshooting peer-specific or functionality-related Internet Key Exchange (IKE) and IPsec problems, conditional debugging may not be able to define and check large numbers of debug conditions.
•Because extra space is needed to store the debug condition values, additional processing overhead is added to the CPU and memory usage is increased. Thus, enabling crypto conditional debugging on a switch with heavy traffic should be used with caution.
•Your switch will perform conditional debugging only after at least one of the global crypto debug commands—debug crypto isakmp, debug crypto ipsec, or debug crypto engine—has been enabled. This requirement helps to ensure that the performance of the switch will not be impacted when conditional debugging is not being used.
Enabling Crypto Conditional Debug Filtering
To enable crypto conditional debug filtering, perform the following tasks:
.
|
|
|
Step 1 |
Router# enable |
Enables privileged EXEC mode. |
Step 2 |
Router# debug crypto condition [connid integer engine-id integer] [flowid integer engine-id integer] [fvrf string] [ivrf string] [peer [group string] [hostname string] [ipv4 ipaddress] [subnet subnet mask] [username string]] [spi integer] [reset] |
Defines conditional debug filters. See Table 13-2 for descriptions of values. |
Step 3 |
Router# show crypto debug-condition {[peer] [connid] [spi] [fvrf] [ivrf] [unmatched]} |
Displays crypto debug conditions that have already been enabled in the switch. |
Step 4 |
Router# debug crypto isakmp |
Enables global IKE debugging. |
Step 5 |
Router# debug crypto ipsec |
Enables global IPsec debugging. |
Step 6 |
Router# debug crypto engine |
Enables global crypto engine debugging. |
Step 7 |
Router# debug crypto condition unmatched [isakmp | ipsec | engine] |
(Optional) Displays debug conditional crypto messages when no context information is available to check against debug conditions. If none of the optional keywords are specified, all crypto-related information will be shown. |
Disabling Crypto Conditional Debugging
Before you disable crypto conditional debugging, you must first disable any crypto global debug CLIs that you have issued. You can then disable crypto conditional debugging. To disable crypto conditional debugging, enter the following command:
Router# debug crypto condition reset
Enabling Crypto Error Debug Messages
Enabling the debug crypto error command displays only error-related debug messages, which allows you to easily determine why a crypto operation, such as an IKE negotiation, has failed within your system. To enable crypto error debug messages, enter the following command from privileged EXEC mode:
Router# debug crypto {isakmp | ipsec | engine} error
Note When enabling this command, ensure that global crypto debug commands are not enabled; otherwise, the global commands will override any possible error-related debug messages.
For complete configuration information for crypto conditional debug support, refer to this URL:
http://www.cisco.com/en/US/docs/ios/12_3t/12_3t2/feature/guide/gt_dbcry.html
Preparing for Online Insertion and Removal of a VSPA or SSC-600
The Catalyst 6500 Series switch supports online insertion and removal (OIR) of the SSC-600, in addition to each of the VSPAs. You can remove an SSC-600 with its VSPAs still intact, or you can remove a VSPA independently from the SSC-600, leaving the SSC-600 installed in the switch.
An SSC-600 can remain installed in the switch with one VSPA remaining active while you remove another VSPA from one of the SSC-600 subslots. If you are not planning to immediately replace a VSPA into the SSC-600, then be sure to install a blank filler plate in the subslot. The SSC-600 should always be fully installed with either functional VSPAs or blank filler plates.
Note When you remove a VSPA that has some ports participating in crypto connection, the crypto connections remain intact. When you reinsert the same type of VSPA into the same slot, the traffic starts to run again on all the crypto connections. To move the VSPA to a different slot, you must first manually remove the crypto connections before removing the VSPA. You can enter the no crypto connect vlan command from any interface when the associated physical port is removed.
For more information about activating and deactivating VSPAs in preparation for OIR, see the Cisco VPN Services Port Adapter Hardware Installation Guide.