Cisco VPN Services Port Adapter Configuration Guide
Troubleshooting
Downloads: This chapterpdf (PDF - 340.0KB) The complete bookPDF (PDF - 8.37MB) | Feedback

Troubleshooting

Table Of Contents

Troubleshooting

General Troubleshooting Information

Interpreting Console Error Messages

Using debug Commands

Using show Commands

Using Online Diagnostics

Monitoring the VSPA

Displaying VSPA Hardware and System Information

Displaying Information About VSPA Ports

Displaying Platform and Network Interface Controller Statistics for the VSPA

Displaying Information About Hardware Revision Levels

Displaying VSPA Configuration Information

Displaying Information About Access and Routed Ports That Are Connected

Displaying Information About the VLANs Allowed by a Trunk Port

Displaying the VPN Running State

Displaying Information About IKE Policies

Displaying Information About IPsec Transform Sets

Displaying Information About Crypto Maps

Displaying Information About IPsec Security Associations

Displaying Information About SAs at a Peer

Displaying Information About RSA Public Keys

Displaying Information About Certificates

Displaying Information About Trustpoints

Displaying Information About Crypto Sessions

Displaying Tunnel Interface Information

Displaying Information About a BFG Configuration

Displaying Information About the NHRP Cache

Displaying Information About IP Multicast Over a GRE Tunnel

Troubleshooting Specific Problems on the VSPA

Clearing IPsec Security Associations

Troubleshooting Trunk Port Configurations

Troubleshooting a Blade Failure Group

Troubleshooting IKE Policy and Transform Sets

Using Crypto Conditional Debug

Crypto Conditional Debug Configuration Guidelines and Restrictions

Enabling Crypto Conditional Debug Filtering

Disabling Crypto Conditional Debugging

Enabling Crypto Error Debug Messages

Preparing for Online Insertion and Removal of a VSPA or SSC-600


Troubleshooting


This chapter describes techniques that you can use to troubleshoot the operation of the VSPA in a Catalyst 6500 Series switch.

It includes the following sections:

General Troubleshooting Information

Monitoring the VSPA

Troubleshooting Specific Problems on the VSPA

Using Crypto Conditional Debug

Preparing for Online Insertion and Removal of a VSPA or SSC-600


Note For detailed information on Cisco IOS IPsec cryptographic operations and policies, the Cisco IOS Security Configuration Guide, Release 12.2 at this URL:

http://www.cisco.com/en/US/docs/ios/12_2/security/configuration/guide/fsecur_c.html


For more information about the commands used in this chapter, see the Cisco IOS Security Command Reference at this URL:

http://www.cisco.com/en/US/docs/ios/security/command/reference/sec_book.html

Also refer to the related Cisco IOS Release 12.2 software configuration guide, command reference, and master index publications. For more information about accessing these publications, see the "Related Documentation" section on page xvi.

General Troubleshooting Information

This section describes general information for troubleshooting the VSPA and the SSC-600. It includes the following sections:

Interpreting Console Error Messages

Using debug Commands

Using show Commands

Using Online Diagnostics

Interpreting Console Error Messages

The Catalyst 6500 Series switch can generate error messages and other system messages to inform the operator of events that might require attention. These messages can be displayed on the console, or sent to a logging host using the System Logging (Syslog) protocol or Simple Network Management Protocol (SNMP).

System error messages are organized in the documentation according to the particular system facility that produces the messages. The VSPA and SSC-600 use the following facility names in error messages:

VSPA—WS_IPSEC_3 (also VSPA, VPNSPA)

SSC-600—CAT6000_SSC (also C7600_SSC600)

To view the explanations and recommended actions for Catalyst 6500 Series switch error messages, including messages related to service modules, refer to the following documents:

Cisco IOS Release 12.2SX System Message Guide at this URL:

http://www.cisco.com/en/US/docs/ios/12_2sx/system/messages/122sxsms.html

System Messages for 12.2S (for error messages in Release 12.2S) at this URL:

http://www.cisco.com/en/US/docs/ios/system/messages/guide/consol_smg.html

Using debug Commands

For information about debug commands specific to the Cisco IOS software release 12.2SX, see the Cisco IOS Master Command List, Release 12.2SX at this URL:

http://www.cisco.com/en/US/docs/ios/mcl/122sxmcl/12_2sx_mcl_book.html


Caution Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support personnel. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.

For information about available crypto conditional debugging commands, see the "Using Crypto Conditional Debug" section.

For more information about other debug commands that can be used on a Catalyst 6500 Series switch, see the Cisco IOS Debug Command Reference, Release 12.2 at this URL:

http://www.cisco.com/en/US/docs/ios/12_2/debug/command/reference/122debug.html

Using show Commands

There are several show commands that you can use to monitor and troubleshoot modules on the Catalyst 6500 Series switch. This chapter describes using the show hw-module slot command to perform troubleshooting of your VSPA.

For more information about using show commands to verify and monitor your VSPA, see the "Displaying the Module Hardware Type" section on page 1-10.

For more information about security-related show commands, see the Cisco IOS Security Command Reference at this URL:

http://www.cisco.com/en/US/docs/ios/12_2/security/command/reference/fsecur_r.html

Using Online Diagnostics

Using the online diagnostics features provided by the Catalyst 6500 Series switch, you can test and verify the hardware functionality of the VSPA at bootup, as part of background health monitoring, or on demand.


Note In Cisco IOS Release 12.2(33)SXI, a bootup diagnostics failure in the VSPA results in a restart of the VSPA and the SSC-600. Health monitoring failures exceeding a certain threshold result in a syslog message and a restart of the VSPA.

In Cisco IOS Release 12.2(33)SXI1 and later releases, a bootup diagnostics failure in the VSPA results in a restart of only the VSPA. Health monitoring failures exceeding a certain threshold result in a syslog message.


For more information about online diagnostics, see the Catalyst 6500 Release 12.2SXH and Later Software Configuration Guide at this URL:

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/diags.html

Monitoring the VSPA

This section describes commands that can be used to display information about the VSPA hardware and configuration. It consists of the following subsections:

Displaying VSPA Hardware and System Information

Displaying VSPA Configuration Information

Displaying VSPA Hardware and System Information

To display hardware and system information, use the following commands:

show diagbus, show module, show crypto eli—See the "Displaying Information About VSPA Ports" section.

show crypto engine accelerator statistic slot—See the "Displaying Platform and Network Interface Controller Statistics for the VSPA" section.

show hw-module slot fpd—See the "Displaying Information About Hardware Revision Levels" section.

Displaying Information About VSPA Ports

To display information about the type of modules that are installed in the switch, use the show diagbus command.

The following example shows output from the show diagbus command on a Catalyst 6500 Series switch with a VSPA installed in subslot 1 of a SSC-600 that is installed in slot 5:

Router# show diagbus

Slot 5: Logical_index 10
        2-subslot Services SPA Carrier-600 controller
        Board is analyzed ipc ready 
        HW rev 0.3, board revision A01
        Serial Number: abc Part number: 73-6348-01

        Slot database information:
        Flags: 0x2004   Insertion time: 0x3DB5F4BC (4d20h ago)

        Controller Memory Size:
                248 MBytes CPU Memory
                8 MBytes Packet Memory
                256 MBytes Total on Board SDRAM
        IOS (tm) cwlc Software (smsc-DWDBG-M), Experimental Version 12.2(20050623:231413) 

        SPA Information:
        subslot 5/1: WS-IPSEC-3 (0x3D7), status: ok

For information about the show module and show crypto eli commands, see the "Displaying the Module Hardware Type" section on page 1-10.

Displaying Platform and Network Interface Controller Statistics for the VSPA

To display platform statistics, and, optionally, network interface controller statistics, use the show crypto engine accelerator statistic slot command.

The following example shows output from the show crypto engine accelerator statistic command on a Catalyst 6500 Series switch with a VSPA in subslot 0 of a SSC-600 that is installed in slot 1. The output displays platform statistics for the VSPA and also displays the network interface controller statistics.

Router# show crypto engine accelerator statistic slot 1/0 detail

VPN module in slot 1/0


Decryption Side Data Path Statistics
====================================
Packets RX...............: 454260
Packets TX...............: 452480

IPSec Transport Mode.....: 0
IPSec Tunnel Mode........: 452470
AH Packets...............: 0
ESP Packets..............: 452470
GRE Decapsulations.......: 0
NAT-T Decapsulations.....: 0
Clear....................: 8
ICMP.....................: 0

Packets Drop.............: 193
Authentication Errors....: 0
Decryption Errors........: 0
Replay Check Failed......: 0
Policy Check Failed......: 0
Illegal CLear Packet.....: 0
GRE Errors...............: 0
SPD Errors...............: 0
HA Standby Drop..........: 0

Hard Life Drop...........: 0
Invalid SA...............: 191
SPI No Match.............: 0
Destination No Match.....: 0
Protocol No Match........: 0

Reassembly Frag RX.......: 0
IPSec Fragments..........: 0
IPSec Reasm Done.........: 0
Clear Fragments..........: 0
Clear Reasm Done.........: 0
Datagrams Drop...........: 0
Fragments Drop...........: 0


Decryption Side Controller Statistics 

=====================================
Frames RX................: 756088
Bytes RX.................: 63535848
Mcast/Bcast Frames RX....: 2341
RX Less 128Bytes.........: 756025
RX Less 512Bytes.........: 58
RX Less 1KBytes..........: 2
RX Less 9KBytes..........: 3
RX Frames Drop...........: 0

Frames TX................: 452365
Bytes TX.................: 38001544
Mcast/Bcast Frames TX....: 9
TX Less 128Bytes.........: 452343
TX Less 512Bytes.........: 22
TX Less 1KBytes..........: 0
TX Less 9KBytes..........: 0


Encryption Side Data Path Statistics
====================================
Packets RX...............: 756344
Packets TX...............: 753880
IPSec Transport Mode.....: 0
IPSec Tunnel Mode........: 753869
GRE Encapsulations.......: 0
NAT-T Encapsulations.....: 0
LAF prefragmented........: 0

Fragmented...............: 0
Clear....................: 753904
ICMP.....................: 0

Packets Drop.............: 123
IKE/TED Drop.............: 27
Authentication Errors....: 0
Encryption Errors........: 0
HA Standby Drop..........: 0

Hard Life Drop...........: 0
Invalid SA...............: 191

Reassembly Frag RX.......: 0
Clear Fragments..........: 0
Clear Reasm Done.........: 0
Datagrams Drop...........: 0
Fragments Drop...........: 0


Encryption Side Controller Statistics
=====================================
Frames RX................: 454065
Bytes RX.................: 6168274/
Mcast/Bcast Frames RX....: 1586
RX Less 128Bytes.........: 1562
RX Less 512Bytes.........: 452503
RX Less 1KBytes..........: 0
RX Less 9KBytes..........: 0
RX Frames Drop...........: 0

Frames TX................: 753558
Bytes TX.................: 100977246
Mcast/Bcast Frames TX....: 2
TX Less 128Bytes.........: 3
TX Less 512Bytes.........: 753555
TX Less 1KBytes..........: 0
TX Less 9KBytes..........: 0

Router# 

Displaying Information About Hardware Revision Levels

To display information about the hardware revision of the SSC-600 and the VSPA as well as the version of the field-programmable devices (FPDs) that are on the carrier card and the module, use the show hw-module slot fpd command. Cisco technical engineers might need this information to debug or troubleshoot problems with a module installation.

The following example shows output from the show hw-module slot command on a Catalyst 6500 Series switch with a VSPA installed in subslot 0 of an SSC-600 that is installed in slot 6:

Router# show hw-module slot 2 fpd 

==== ====================== ====== =============================================
                             H/W   Field Programmable   Current   Min. Required
Slot Card Type               Ver.  Device: "ID-Name"    Version      Version
==== ====================== ====== ================== =========== ==============
   2 WS-SSC-600              0.302 1-SSC-600 I/O FPGA     1.0         1.0
                                   2-SSC-600 DPRX FPG     0.9         0.9
                                   3-SSC-600 DPTX FPG     0.13        0.13
                                   4-ROMMON               1.7         1.7
---- ---------------------- ------ ------------------ ----------- --------------
 2/0 WS-IPSEC-3              0.29  1-BOOTLOADER           1.1         1.1
                                   2-IO FPGA              1.0         1.0
                                   3-CAM FPGA             1.0         1.0
==== ====================== ====== =============================================

Displaying VSPA Configuration Information

Except where noted, examples of command output in this section use the following example topology:

                          gi3/2              gi3/2 
loopback -------- Router1 ------------------------ Router2 -------- loopback
         12.0.0.0                10.10.40.0                13.0.0.0
          network                  network                  network 

To display information about the VSPA configuration, use the following commands:

show crypto vlan—See the "Displaying Information About Access and Routed Ports That Are Connected" section, "Displaying the VPN Running State" section, and "Displaying Information About IP Multicast Over a GRE Tunnel" section.

show interfaces trunk—See the "Displaying Information About the VLANs Allowed by a Trunk Port" section.

show crypto isakmp policy—See the "Displaying Information About IKE Policies" section.

show crypto ipsec transform-set—See the "Displaying Information About IPsec Transform Sets" section.

show crypto map—See the "Displaying Information About Crypto Maps" section.

show crypto isakmp sa—See the "Displaying Information About SAs at a Peer" section.

show crypto ipsec sa—See the "Displaying Information About IPsec Security Associations" section.

show crypto key mypubkey rsa—See the "Displaying Information About RSA Public Keys" section.

show crypto key pubkey-chain rsa—See the "Displaying Information About RSA Public Keys" section.

show crypto pki certificates—See the "Displaying Information About Certificates" section.

show crypto pki trustpoints—See the "Displaying Information About Trustpoints" section.

show crypto session—See the "Displaying Information About Crypto Sessions" section.

show interfaces tunnel—See the "Displaying Tunnel Interface Information" section.

show redundancy linecard-group—See the "Displaying Information About a BFG Configuration" section.

show crypto ace redundancy—See the "Displaying Information About a BFG Configuration" section.

show ip nhrp—See the "Displaying Information About the NHRP Cache" section.

For a detailed description of the information displayed by the show commands, refer to the "IP Security and Encryption" chapter of the Cisco IOS Security Command Reference.

Displaying Information About Access and Routed Ports That Are Connected

To verify that an access or routed port is connected, use the show crypto vlan command. The following is sample output from the command when a port is crypto-connected to a port VLAN:

Router1# show crypto vlan

Interface VLAN 2 on IPSec Service Module port GigabitEthernet2/0/1 connected to VLAN 502 
with crypto map set mymap1

The following is sample output from the command when a port is crypto-connected to a physical interface:

Router1# show crypto vlan

Interface VLAN 2 on IPSec Service Module port GigabitEthernet2/0/1 connected to Gi3/2 with 
crypto map set mymap2

Displaying Information About the VLANs Allowed by a Trunk Port

To display information about the VLANs allowed by a trunk port, use the show interfaces trunk command. The following is sample output from the command:

Router1# show interfaces trunk 

Port                Mode         Encapsulation  Status        Native vlan
Gi2/0/1             on           802.1q         trunking      1
Gi2/0/2             on           802.1q         trunking      1

Port                Vlans allowed on trunk
Gi2/0/1             2
Gi2/0/2             502

Port                Vlans allowed and active in management domain
Gi2/0/1             2
Gi2/0/2             502

Port                Vlans in spanning tree forwarding state and not pruned
Gi2/0/1             2
Gi2/0/2             502

Displaying the VPN Running State

To display the VPN running state, use the show crypto vlan command.

In the following example, the interface VLAN belongs to the VSPA inside port:

Router1# show crypto vlan 

Interface VLAN 2 on IPSec Service Module port GigabitEthernet2/0/1 connected to Gi3/2

In the following example, either the interface VLAN is missing on the VSPA inside port, the VSPA is removed from the chassis, or the VSPA was moved to a different subslot:

Router1# show crypto vlan 

  Interface VLAN 2 connected to VLAN 3 (no IPSec Service Module attached)

Displaying Information About IKE Policies

To display information about IKE policies, use the show crypto isakmp policy command. The following is sample output from the command:

Router1# show crypto isakmp policy 

Global IKE policy
Protection suite of priority 1
        encryption algorithm:   Three key triple DES
        hash algorithm:         Secure Hash Standard
        authentication method:  Rivest-Shamir-Adleman Signature
        Diffie-Hellman group:   #1 (768 bit)
        lifetime:               86400 seconds, no volume limit
Default protection suite
        encryption algorithm:   DES - Data Encryption Standard (56 bit keys).
        hash algorithm:         Secure Hash Standard
        authentication method:  Rivest-Shamir-Adleman Signature
        Diffie-Hellman group:   #1 (768 bit)
        lifetime:               86400 seconds, no volume limit


Note If a user enters an IKE encryption method that the hardware does not support, a warning message will be displayed in the show crypto isakmp policy command output:

WARNING:encryption hardware does not support the configured encryption method for ISAKMP policy
value


Displaying Information About IPsec Transform Sets

To display information about transform set configurations, use the show crypto ipsec transform-set command. The following is sample output from the command:

Router1# show crypto ipsec transform-set 

Transform set tset: { esp-3des esp-sha-hmac  } 
   will negotiate = { Tunnel,  }, 


Note If a user enters an IPsec transform that the hardware (the IPsec peer) does not support, a warning message will be displayed in the show crypto ipsec transform-set command output:

WARNING:encryption hardware does not support transform.


Displaying Information About Crypto Maps

To display information about crypto map configurations, use the show crypto map command. The following is sample output from the command:

Router1# show crypto map 

Crypto Map "mymap1" 10 ipsec-isakmp
        Peer = 10.10.40.2
        Extended IP access list 101
            access-list 101 permit ip host 12.0.0.1 host 13.0.0.1
        Current peer: 10.10.40.2
        Security association lifetime: 4608000 kilobytes/3600 seconds
        PFS (Y/N): N
        Transform sets={ 
                tset, 
        }
        Interfaces using crypto map test:
                Vlan2
 using crypto engine WS-IPSEC-3[2/0]

Displaying Information About IPsec Security Associations

To display information about IPsec security associations, use the show crypto ipsec sa command.


Note When you first enter the show crypto ipsec sa command, the packet counters will not show the correct values. Subsequent instances of the command will display the correct values.


The following is sample output from the command:

Router1# show crypto ipsec sa 
 
   
interface: Vlan2
    Crypto map tag: mymap1, local addr 10.10.40.1

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (12.0.0.1/255.255.255.255/0/0)
   remote ident (addr/mask/prot/port): (13.0.0.1/255.255.255.255/0/0)
   current_peer 10.10.40.2 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
    #pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 10.10.40.1, remote crypto endpt.: 10.10.40.2
     path mtu 1500, ip mtu 1500
     current outbound spi: 0xDAADD709(3668825865)
     PFS (Y/N): N, DH group: none

     inbound esp sas:
      spi: 0x7D99ACA4(2107223204)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2001, flow_id: :1, sibling flags 80000040,  crypto map: test
        sa timing: remaining key lifetime (k/sec): (4391325/3403)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0xDAADD709(3668825865)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 2002, flow_id: :2, sibling flags 80000040,  crypto map: test
        sa timing: remaining key lifetime (k/sec): (4391325/3403)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE

     outbound ah sas:
          
     outbound pcp sas:

Displaying Information About SAs at a Peer

To display information about all current IKE SAs at a peer, use the show crypto isakmp sa command. The following is sample output from the command:

Router1# show crypto isakmp sa 

IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
10.10.40.2      10.10.40.1      QM_IDLE          68001 ACTIVE

Displaying Information About RSA Public Keys

To display information the RSA public keys configured for your switch, use the show crypto key mypubkey rsa command. The following is sample output from the command:

Router1# show crypto key mypubkey rsa 

% Key pair was generated at: 13:43:52 PST Jan 15 2009
Key name: router1.example.com
 Storage Device: not specified
 Usage: General Purpose Key
 Key is not exportable.
 Key Data:
  30820122 300D0609 2A864886 F70D0101 01050003 82010F00 3082010A 02820101 
  00D6D5A6 2D42D542 805FAAFB 65B72A10 21AA7ACC F86D6C5A 7E290221 0EEC370B 
  FF790FAC AE654451 E038BEAE 7AD85C45 75A2E5D3 95866F6C 6C888C1F 197C3BC3 
  3EB02714 A5F43640 D1951900 053F080E 99B547A7 A317C852 BF9B2D91 20D3BABC 
  6EA39317 FF22EF39 308F9838 1F395A56 F04473F7 52D41353 3C294E44 B7BDC8A0 
  7BB82EBF 405D7853 1E26F8DA 3251FD9F 072E8697 D131F3E2 042D8F4F 0D421361 
  846FCB1F B266E0AB 3B2AB6E6 1FA5927A B6E5C29E CFFA77C0 EF07831E D060A950 
  C6769919 E2C54267 226FC037 DF996B5D 04D0F0DD 5C0340D4 F8F17CB7 D74531E8 
  BF378EFA EADB73A8 7F02370A EE4AA992 F2B7A9D0 DA0CFB9C 2B3C3F2C 79B854EA 
  A1020301 0001
% Key pair was generated at: 13:43:53 PST Jan 15 2009
Key name: router1.example.com.server
Temporary key
 Usage: Encryption Key
 Key is not exportable.
 Key Data:
  307C300D 06092A86 4886F70D 01010105 00036B00 30680261 00C18A19 1B4BE742 
  31692B90 FD290B6A CC085CF8 2E9591F0 9B70951C 07E7942B 55AB8540 2BCDEF8D 
  A17629D9 8C9369CD AB693089 E4DA150F 6ADF7CFB 78197CF3 84FBC959 90F37EC5 
  83B0136D 1A93ACF4 040CD25F 22D7428E 7950E0ED AB45B011 DD020301 0001

To display a list of all the RSA public keys stored on your switch (including the public keys of peers that have sent your switch their certificates during peer authentication for IPsec), or to display details of a particular RSA public key stored on your switch, use the show crypto key pubkey-chain rsa command. The following is sample output from the command:

Router1# show crypto key pubkey-chain rsa 
 
   
Codes: M - Manually configured, C - Extracted from certificate

Code Usage         IP-Address/VRF         Keyring          Name
C    Signing                              default          X.500 DN name: 
                              cn=mscavpn1
                              ou=isbu
                              o=example

C    Signing                              default          router1.example.com

Displaying Information About Certificates

To display information about your certificate, the certificate of the CA, and any RA certificates, use the show crypto pki certificates command. The following is sample output from the command:

Router1# show crypto pki certificates 

Certificate
  Status: Available
  Certificate Serial Number: 18FBAA1C000000000062
  Certificate Usage: General Purpose
  Issuer: 
    cn=mscavpn1
    ou=isbu
    o=example
  Subject:
    Name: router1.example.com
    Serial Number: 0001925D
    cn=router1
    ou=isbu
    o=example
    hostname=router1.example.com
    serialNumber=1925D
  CRL Distribution Points: 
    http://caserver/CertEnroll/mscavpn1.crl
  Validity Date: 
    start date: 13:41:42 PST Jan 15 2009
    end   date: 13:51:42 PST Jan 15 2010
  Associated Trustpoints: MSCA 

CA Certificate
  Status: Available
  Certificate Serial Number: 30EF1D0A2F5B98A44B0AE1BF562636EC
  Certificate Usage: Signature
  Issuer: 
    cn=mscavpn1
    ou=isbu
    o=example
  Subject: 
    cn=mscavpn1
    ou=isbu
    o=example
  CRL Distribution Points: 
    http://caserver/CertEnroll/mscavpn1.crl
  Validity Date: 
    start date: 16:12:15 PST Dec 22 2008
    end   date: 16:19:24 PST Dec 22 2013
  Associated Trustpoints: MSCA

Displaying Information About Trustpoints

To display the trustpoints that are configured in the switch, use the show crypto pki trustpoints command. The following is sample output from the command:

Router1# show crypto pki trustpoints 

Trustpoint MSCA:
    Subject Name: 
    cn=mscavpn1
    ou=isbu
    o=example
          Serial Number: 30EF1D0A2F5B98A44B0AE1BF562636EC
    Certificate configured.
    SCEP URL: http://43.0.111.111:80/certsrv/mscep/mscep.dll

Displaying Information About Crypto Sessions

To display status information for active crypto sessions, use the show crypto session command. The output will include the following:

Interface

IKE peer description, if available

IKE SAs that are associated with the peer by which the IPsec SAs are created

IPsec SAs serving the flows of a session

The following is sample output from the command:
Router1# show crypto session 

Crypto session current status

Interface: Vlan2
Session status: UP-ACTIVE     
Peer: 10.10.40.2 port 500
  IKE SA: local 10.10.40.1/500 remote 10.10.40.2/500 Active
  IPSEC FLOW: permit ip host 12.0.0.1 host 13.0.0.1 
        Active SAs: 2, origin: crypto map

The following is sample output from the command using the detail keyword:

Router1# show crypto session detail 

Crypto session current status
 
Code: C - IKE Configuration mode, D - Dead Peer Detection     
K - Keepalives, N - NAT-traversal, T - cTCP encapsulation     
X - IKE Extended Authentication
 
Interface: Vlan2
Uptime: 00:00:32
Session status: UP-ACTIVE     
Peer: 10.10.40.2 port 500 fvrf: (none) ivrf: (none)
      Phase1_id: router2.example.com
      Desc: (none)
  IKE SA: local 10.10.40.1/500 remote 10.10.40.2/500 Active 
          Capabilities:(none) connid:68001 lifetime:23:59:27
  IPSEC FLOW: permit ip host 12.0.0.1 host 13.0.0.1 
        Active SAs: 2, origin: crypto map
        Inbound:  #pkts dec'ed 9 drop 0 life (KB/Sec) 4398172/3567
        Outbound: #pkts enc'ed 9 drop 0 life (KB/Sec) 4398172/3567

Displaying Tunnel Interface Information

To display tunnel interface information, use the show interfaces tunnel command. The following is sample output from the command:

Router# show interfaces tunnel 1 

Tunnel4 is up, line protocol is down 
Hardware is Routing Tunnel 
Internet address is 10.1.1.1/24 
MTU 1514 bytes, BW 9 Kbit, DLY 500000 usec, rely 255/255, load 1/255 
Encapsulation TUNNEL, loopback not set 
Keepalive set (10 sec) 
Tunnel source 9.2.2.1, destination 6.6.6.2 
Tunnel protocol/transport GRE/IP, key disabled, sequencing disabled 
Tunnel TOS 0xF, Tunnel TTL 128 
Checksumming of packets disabled, fast tunneling enabled 
Last input never, output never, output hang never 
Last clearing of "show interface" counters never 
Queueing strategy, fifo 
Output queue 0/0, 1 drops; input queue 0/75, 0 drops 
30 second input rate 0 bits/sec, 0 packets/sec 
30 second output rate 0 bits/sec, 0 packets/sec 
0 packets input, 0 bytes, 0 no buffer 
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 
0 packets output, 0 bytes, 0 underruns 
0 output errors, 0 collisions, 0 interface resets, 0 restarts 


Note This sample output does not use the example topology shown at the start of the "Displaying VSPA Configuration Information" section.


Table 13-1 describes significant fields shown in the display.

Table 13-1 show interfaces tunnel Field Descriptions  

Field
Description

Tunnel is {up | down}

Interface is currently active and inserted into ring (up) or inactive and not inserted (down).

line protocol is {up | down | administratively down}

Shows line protocol up if a valid route is available to the tunnel destination. Shows line protocol down if no route is available, or if the route would be recursive.

Hardware

Specifies the hardware type.

MTU

Maximum transmission unit of the interface.

BW

Bandwidth of the interface in kilobits per second.

DLY

Delay of the interface in microseconds.

rely

Reliability of the interface as a fraction of 255 (255/255 is 100 percent reliability), calculated as an exponential average over 5 minutes.

load

Load on the interface as a fraction of 255 (255/255 is completely saturated), calculated as an exponential average over 5 minutes.

Encapsulation

Encapsulation method is always TUNNEL for tunnels.

loopback

Indicates whether loopback is set or not.

Keepalive

Indicates whether keepalives are set or not.

Tunnel source

IP address used as the source address for the tunnel packets.

destination

IP address of the tunnel destination.

Tunnel protocol

Tunnel transport protocol (the protocol the tunnel is using). This is based on the tunnel mode command, which defaults to GRE.

key

(Optional) ID key for the tunnel interface.

sequencing

(Optional) Indicates whether the tunnel interface drops datagrams that arrive out of order.

Last input

Number of hours, minutes, and seconds (or never) since the last packet was successfully received by an interface and processed locally on the switch. Useful for knowing when a dead interface failed.

This field is not updated by fast-switched traffic.

output

Number of hours, minutes, and seconds (or never) since the last packet was successfully transmitted by an interface.

output hang

Number of hours, minutes, and seconds (or never) since the interface was last reset because of a transmission that took too long. When the number of hours in any of the "last" fields exceeds 24 hours, the number of days and hours is displayed. If that field overflows, asterisks are displayed.

Last clearing

Time at which the counters that measure cumulative statistics (such as number of bytes transmitted and received) shown in this report were last reset to zero. Note that variables that might affect routing (for example, load and reliability) are not cleared when the counters are cleared.

Three asterisks (***) indicate the elapsed time is too large to be displayed.

0:00:00 indicates the counters were cleared more than 231 ms (and less than 232 ms) ago.

Output queue, drops
Input queue, drops

Number of packets in output and input queues. Each number is followed by a slash, the maximum size of the queue, and the number of packets dropped because of a full queue.

30 second input rate,
30 second output rate

Average number of bits and packets transmitted per second in the last 30 seconds.

The 30-second input and output rates should be used only as an approximation of traffic per second during a given 30-second period. These rates are exponentially weighted averages with a time constant of 30 seconds. A period of four time constants must pass before the average will be within two percent of the instantaneous rate of a uniform stream of traffic over that period.

packets input

Total number of error-free packets received by the system.

bytes

Total number of bytes, including data and MAC encapsulation, in the error-free packets received by the system.

no buffer

Number of received packets discarded because there was no buffer space in the main system. Compare with ignored count. Broadcast storms on Ethernet networks and bursts of noise on serial lines are often responsible for no input buffer events.

broadcasts

Total number of broadcast or multicast packets received by the interface.

runts

Number of packets that are discarded because they are smaller than the minimum packet size of the medium.

giants

Number of packets that are discarded because they exceed the maximum packet size of the medium.

CRC

Cyclic redundancy checksum generated by the originating LAN station or far-end device does not match the checksum calculated from the data received. On a LAN, this usually indicates noise or transmission problems on the LAN interface or the LAN bus itself. A high number of CRCs is usually the result of a station transmitting bad data.

frame

Number of packets received incorrectly having a CRC error and a noninteger number of octets.

overrun

Number of times the serial receiver hardware was unable to hand received data to a hardware buffer because the input rate exceeded the receiver's ability to handle the data.

ignored

Number of received packets ignored by the interface because the interface hardware ran low on internal buffers. These buffers are different than the system buffers mentioned previously in the buffer description. Broadcast storms and bursts of noise can cause the ignored count to be increased.

abort

Illegal sequence of one bits on a serial interface. This usually indicates a clocking problem between the serial interface and the data link equipment.

packets output

Total number of messages transmitted by the system.

bytes

Total number of bytes, including data and MAC encapsulation, transmitted by the system.

underruns

Number of times that the far-end transmitter has been running faster than the near-end switch's receiver can handle. This may never be reported on some interfaces.

output errors

Sum of all errors that prevented the final transmission of datagrams out of the interface being examined. Note that this may not balance with the sum of the enumerated output errors, as some datagrams may have more than one error, and others may have errors that do not fall into any of the specifically tabulated categories.

collisions

Number of messages retransmitted because of an Ethernet collision. This usually is the result of an overextended LAN (Ethernet or transceiver cable too long, more than two repeaters between stations, or too many cascaded multiport transceivers). Some collisions are normal. However, if your collision rate climbs to around 4 or 5 percent, you should consider verifying that there is no faulty equipment on the segment and moving some existing stations to a new segment. A packet that collides is counted only once in output packets.

interface resets

Number of times an interface has been reset. The interface may be reset by the administrator or automatically when an internal error occurs.

restarts

Number of times that the controller was restarted because of errors.


Displaying Information About a BFG Configuration

To display information about a BFG configuration, use the show redundancy linecard-group and show crypto ace redundancy commands. The following is sample output from the commands:

Router# show redundancy linecard-group 1

Line Card Redundancy Group:1 Mode:feature-card
Class:load-sharing
Cards:
Subslot: 1/0
Subslot: 2/0

Router1# show crypto ace redundancy
--------------------------------------
LC Redundancy Group ID            : 1
Pending Configuration Transactions: 0
Current State                     : OPERATIONAL
Number of blades in the group     : 2
Slots
--------------------------------------
 
Slot: 1 Subslot: 0
Slot state: 0x36
Booted
Received partner config
Completed Bulk Synchronization
Crypto Engine in Service
Rebooted 0 times
Initialization Timer not running
 
Slot: 2 Subslot: 0
Slot state: 0x36
Booted
Received partner config
Completed Bulk Synchronization
Crypto Engine in Service
Rebooted 0 times
Initialization Timer not running


Note This sample output does not use the example topology shown at the start of the "Displaying VSPA Configuration Information" section.


Displaying Information About the NHRP Cache

To display information about the Next Hop Resolution Protocol (NHRP) cache, use the show ip nhrp and the show crypto sockets commands. The following is sample output from the commands:

Router# show ip nhrp 
 
   
10.10.1.75/32 via 10.10.1.75, Tunnel5 created 00:32:11, expire 00:01:46
 
   
  Type: dynamic, Flags: authoritative unique registered 
 
   
  NBMA address: 172.16.175.75 
 
   
10.10.1.76/32 via 10.10.1.76, Tunnel5 created 00:26:41, expire 00:01:37
 
   
  Type: dynamic, Flags: authoritative unique registered 
 
   
  NBMA address: 172.16.175.76 
 
   
10.10.1.77/32 via 10.10.1.77, Tunnel5 created 00:31:26, expire 00:01:33
 
   
  Type: dynamic, Flags: authoritative unique registered 
 
   
  NBMA address: 172.17.63.20

Router# show crypto sockets 
 
Number of Crypto Socket connections 1
 
   Tu0 Peers (local/remote): 9.1.1.1/11.1.1.1 
       Local Ident  (addr/mask/port/prot): (9.1.1.1/255.255.255.255/0/47)
       Remote Ident (addr/mask/port/prot): (11.1.1.1/255.255.255.255/0/47)
       IPSec Profile: "MyIpsecProf"
       Socket State: Open
       Client: "TUNNEL SEC" (Client State: Active)
 
Crypto Sockets in Listen state:
Client: "TUNNEL SEC" Profile: "MyIpsecProf" Map-name: "Tunnel0-head-0"


Note This sample output does not use the example topology shown at the start of the "Displaying VSPA Configuration Information" section.


Displaying Information About IP Multicast Over a GRE Tunnel

To display information about an IP multicast over a GRE tunnel configuration, enter the show crypto vlan and show ip mroute commands.

Enter the show crypto vlan command to check that the tunnel has been taken over by the VSPA. The following is sample output from the command:

Router# show crypto vlan
 
Interface VLAN 100 on IPSec Service Module port Gi7/0/1 connected to Po1 with crypto map 
set map_t3 
Tunnel15 is accelerated via IPSec SM in subslot 7/0

Enter the show ip mroute command and look for the H flag to check that the IP multicast traffic is hardware-switched. The following is sample output from the command:

Router# show ip mroute 230.1.1.5
 
IP Multicast Routing Table 
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected, 
L - Local, P - Pruned, R - RP-bit set, F - Register flag, 
T - SPT-bit set, J - Join SPT, M - MSDP created entry, 
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement, 
U - URD, I - Received Source Specific Host Report, Z - Multicast Tunnel 
Y - Joined MDT-data group, y - Sending to MDT-data group 
Outgoing interface flags: H - Hardware switched, A - Assert winner 
Timers: Uptime/Expires 
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 230.1.1.5), 01:23:45/00:03:16, RP 15.15.1.1, flags: SJC 
Incoming interface: Null, RPF nbr 0.0.0.0 
Outgoing interface list: 
Tunnel15, Forward/Sparse-Dense, 00:25:47/00:03:16

(120.1.0.3, 230.1.1.5), 01:23:46/00:03:25, flags: T
Incoming interface: GigabitEthernet8/1, RPF nbr 0.0.0.0, RPF-MFD
Outgoing interface list:
Tunnel15, Forward/Sparse-Dense, 00:25:47/00:03:16, H


Note This sample output does not use the example topology shown at the start of the "Displaying VSPA Configuration Information" section.


Troubleshooting Specific Problems on the VSPA

This section provides additional information about troubleshooting specific problems related to the VSPA. It includes the following subsections:

Clearing IPsec Security Associations

Troubleshooting Trunk Port Configurations

Troubleshooting a Blade Failure Group

Troubleshooting IKE Policy and Transform Sets

Clearing IPsec Security Associations

You can clear (and reinitialize) IPsec security associations by using the clear crypto sa command.

Using the clear crypto sa command without parameters will clear out the full SA database, which will clear out active security sessions. You may also specify the peer, map, or entry keywords to clear out only a subset of the SA database. For more information, refer to the clear crypto sa command in the Cisco IOS Security Command Reference, Release 12.2.

If you want to also remove the IKE (phase 1) SAs, follow the clear crypto sa command with the clear crypto isa command. Alternatively, you can use the clear crypto session command to achieve the same result as the clear crypto sa and the clear crypto isa commands. The clear crypto session command supports many of the same parameters as the clear crypto sa command.

Troubleshooting Trunk Port Configurations


Caution When you configure an Ethernet port as a trunk port, all the VLANs are allowed on the trunk port by default. This default configuration does not work well with the VSPA and causes network loops. To avoid this problem, you must explicitly specify only the desirable VLANs.

For more information on trunk configuration guidelines, review the "Configuring a Trunk Port" section on page 3-14.

To verify which ports are assigned to the VLAN, enter the show vlan id number command, using the interface VLAN identifier. Following is an example of a trunk port configuration and the output of the show vlan id command:


Router# show run interface gi 1/3
Building configuration...
 
Current configuration : 175 bytes
!
interface GigabitEthernet1/3
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,502-504,1002-1005
 switchport mode trunk
 no ip address
end
 

Router# show crypto vlan

Interface VLAN 2 on IPSec Service Module port Gi7/0/1 connected to VLAN 502 with crypto 
map set testtag_1
Interface VLAN 3 on IPSec Service Module port Gi7/0/1 connected to VLAN 503 with crypto 
map set testtag_2
Interface VLAN 4 on IPSec Service Module port Gi7/0/1 connected to VLAN 504 with crypto 
map set testtag_3
 
Router# show vlan id 2
 
VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
2    VLAN0002                         active    Gi7/0/1
 
VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
2    enet  100002     1500  -      -      -        -    -        0      0   
 
Remote SPAN VLAN
----------------
Disabled
 
Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------
 
Router# show vlan id 502
 
VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
502  VLAN0502                         active    Gi1/3, Gi7/0/2
 
VLAN Type  SAID       MTU   Parent RingNo BridgeNo Stp  BrdgMode Trans1 Trans2
---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------
502  enet  100502     1500  -      -      -        -    -        0      0   
 
Remote SPAN VLAN
----------------
Disabled
 
Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------

Router#

Troubleshooting a Blade Failure Group

To enable VSPA debugging for a blade failure group, enter the debug crypto ace b2b command:

Router# debug crypto ace b2b

ACE B2B Failover debugging is on

Troubleshooting IKE Policy and Transform Sets

Any IPsec transforms or IKE encryption methods that the current hardware does not support should be disabled; they are ignored whenever an attempt to negotiate with the peer is made.

If a user enters an IPsec transform or an IKE encryption method that the hardware does not support, a warning message will be generated. These warning messages are also generated at boot time. When an encrypted card is inserted, the current configuration is scanned. If any IPsec transforms or IKE encryption methods are found that are not supported by the hardware, a warning message will be generated.

Using Crypto Conditional Debug

The crypto conditional debug feature provides three command-line interface (CLI) commands that allow you to debug an IP Security (IPsec) tunnel on the basis of predefined crypto conditions such as the peer IP address, connection-ID of a crypto engine, and security parameter index (SPI). By limiting debug messages to specific IPsec operations and reducing the amount of debug output, you can better troubleshoot a switch with a large number of tunnels.

The crypto conditional debug commands—debug crypto condition, debug crypto condition unmatched, and show crypto debug-condition—allow you to specify conditions (filter values) in which to generate and display debug messages related only to the specified conditions.

Table 13-2 lists the supported condition types.

Table 13-2 Supported Condition Types for Crypto Conditional Debug Commands 

Condition Type (Keyword)
Description

connid

An integer between 1 and 32766. Relevant debug messages will be shown if the current IPsec operation uses this value as the connection-ID to interface with the crypto engine.

flowid

An integer between 1 and 32766. Relevant debug messages will be shown if the current IPsec operation uses this value as the flow-ID to interface with the crypto engine.

fvrf

The name string of a virtual private network (VPN) routing and forwarding (VRF) instance. Relevant debug messages will be shown if the current IPsec operation uses this VRF instance as its front-door VRF (FVRF).

ivrf

The name string of a VRF instance. Relevant debug messages will be shown if the current IPsec operation uses this VRF instance as its inside VRF (IVRF).

peer group

A Unity group name string. Relevant debug messages will be shown if the peer is using this group name as its identity.

peer hostname

A fully qualified domain name (FQDN) string. Relevant debug messages will be shown if the peer is using this string as its identity.

peer ipv4

A single IP address. Relevant debug messages will be shown if the current IPsec operation is related to the IP address of this peer.

peer subnet

A subnet and a subnet mask that specify a range of peer IP addresses. Relevant debug messages will be shown if the IP address of the current IPsec peer falls into the specified subnet range.

peer username

A username string. Relevant debug messages will be shown if the peer is using this username as its identity.

spi

A 32-bit unsigned integer. Relevant debug messages will be shown if the current IPsec operation uses this value as the SPI.



Note If connid, flowid, or spi is used as a debug condition, the debug messages for a related IPsec flow are generated. An IPsec flow has two connection-IDs, flow-IDs, and SPI values—one inbound and one outbound. Either one of the two connection-IDs, flow-IDs, and SPI values can be used as the debug condition that triggers debug messages for the IPsec flow.


Crypto Conditional Debug Configuration Guidelines and Restrictions

When configuring crypto conditional debug, follow these guidelines and restrictions:

This feature does not support debug message filtering for hardware crypto engines.

Although conditional debugging is useful for troubleshooting peer-specific or functionality-related Internet Key Exchange (IKE) and IPsec problems, conditional debugging may not be able to define and check large numbers of debug conditions.

Because extra space is needed to store the debug condition values, additional processing overhead is added to the CPU and memory usage is increased. Thus, enabling crypto conditional debugging on a switch with heavy traffic should be used with caution.

Your switch will perform conditional debugging only after at least one of the global crypto debug commands—debug crypto isakmp, debug crypto ipsec, or debug crypto engine—has been enabled. This requirement helps to ensure that the performance of the switch will not be impacted when conditional debugging is not being used.

Enabling Crypto Conditional Debug Filtering

To enable crypto conditional debug filtering, perform the following tasks:

.

 
Command
Purpose

Step 1 

Router# enable

Enables privileged EXEC mode.

Step 2 

Router# debug crypto condition [connid integer engine-id integer] [flowid integer engine-id integer] [fvrf string] [ivrf string] [peer [group string] [hostname string] [ipv4 ipaddress] [subnet subnet mask] [username string]] [spi integer] [reset]

Defines conditional debug filters. See Table 13-2 for descriptions of values.

Step 3 

Router# show crypto debug-condition {[peer] [connid] [spi] [fvrf] [ivrf] [unmatched]}

Displays crypto debug conditions that have already been enabled in the switch.

Step 4 

Router# debug crypto isakmp

Enables global IKE debugging.

Step 5 

Router# debug crypto ipsec

Enables global IPsec debugging.

Step 6 

Router# debug crypto engine

Enables global crypto engine debugging.

Step 7 

Router# debug crypto condition unmatched [isakmp | ipsec | engine]

(Optional) Displays debug conditional crypto messages when no context information is available to check against debug conditions. If none of the optional keywords are specified, all crypto-related information will be shown.

Disabling Crypto Conditional Debugging

Before you disable crypto conditional debugging, you must first disable any crypto global debug CLIs that you have issued. You can then disable crypto conditional debugging. To disable crypto conditional debugging, enter the following command:

Router# debug crypto condition reset

Enabling Crypto Error Debug Messages

Enabling the debug crypto error command displays only error-related debug messages, which allows you to easily determine why a crypto operation, such as an IKE negotiation, has failed within your system. To enable crypto error debug messages, enter the following command from privileged EXEC mode:

Router# debug crypto {isakmp | ipsec | engine} error

Note When enabling this command, ensure that global crypto debug commands are not enabled; otherwise, the global commands will override any possible error-related debug messages.


For complete configuration information for crypto conditional debug support, refer to this URL:

http://www.cisco.com/en/US/docs/ios/12_3t/12_3t2/feature/guide/gt_dbcry.html

Preparing for Online Insertion and Removal of a VSPA or SSC-600

The Catalyst 6500 Series switch supports online insertion and removal (OIR) of the SSC-600, in addition to each of the VSPAs. You can remove an SSC-600 with its VSPAs still intact, or you can remove a VSPA independently from the SSC-600, leaving the SSC-600 installed in the switch.

An SSC-600 can remain installed in the switch with one VSPA remaining active while you remove another VSPA from one of the SSC-600 subslots. If you are not planning to immediately replace a VSPA into the SSC-600, then be sure to install a blank filler plate in the subslot. The SSC-600 should always be fully installed with either functional VSPAs or blank filler plates.


Note When you remove a VSPA that has some ports participating in crypto connection, the crypto connections remain intact. When you reinsert the same type of VSPA into the same slot, the traffic starts to run again on all the crypto connections. To move the VSPA to a different slot, you must first manually remove the crypto connections before removing the VSPA. You can enter the no crypto connect vlan command from any interface when the associated physical port is removed.


For more information about activating and deactivating VSPAs in preparation for OIR, see the Cisco VPN Services Port Adapter Hardware Installation Guide.