Cisco 10000 Series Router SIP and SPA Software Configuration Guide - Troubleshooting the SIP [Cisco 10000 Series Routers]

Table Of Contents

Troubleshooting the SIP

General Troubleshooting Information

Interpreting Console Error Messages

Using debug Commands

Using show Commands

Using the Cisco IOS Event Tracer to Troubleshoot Problems

Online Insertion and Removal of a SIP

Deactivating a SIP

Reactivating a SIP

Verifying the Status of a SIP

Online Insertion and Removal of a SPA

Deactivating a SPA

Reactivating a SPA

Verifying the Status of a SPA

Troubleshooting the SIP

This chapter describes techniques that you can use to troubleshoot the operation of your SIPs:

•General Troubleshooting Information

•Using the Cisco IOS Event Tracer to Troubleshoot Problems

•Online Insertion and Removal of a SIP

The first section provides information about basic interface troubleshooting. If you are having a problem with your SPA, use the steps in the "Using the Cisco IOS Event Tracer to Troubleshoot Problems" section to begin to investigate a possible interface configuration problem.

To perform advanced troubleshooting, see the other sections in this chapter.

General Troubleshooting Information

This section describes general information for troubleshooting SIPs and SPAs. It includes the following sections:

•Interpreting Console Error Messages

•Using debug Commands

•Using show Commands

Interpreting Console Error Messages

To view the explanations and recommended actions for Cisco 10000 series router error messages, use the Error Message Decoder tool at:

http://cisco.com/en/US/products/hw/routers/ps133/products_system_message_guides_list.html

System error messages are organized in the documentation according to the particular system facility that produces the messages. The SIP error messages use the following facility names:

•Cisco 10000 SIP-600—SIP600

Using debug Commands

Along with the other debug commands supported on the Cisco 10000 series router, you can obtain specific debug information for the SIP on the Cisco 10000 series router using the debug hw-module privileged EXEC command. The debug hw-module command is intended for use by Cisco Systems technical support personnel. For more information about the debug hw-module command, see Chapter 12, "SIP and SPA Commands."

Caution Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.

For more information about other debug commands supported on the Cisco 10000 series router, refer to the Cisco IOS Command Reference publication for your Cisco IOS software release.

Using show Commands

There are several show commands that you can use to monitor and troubleshoot the SIP on the Cisco 10000 series router. This chapter describes using the show hw-module subslot command to perform troubleshooting of your SPA.

For more information about show commands to verify and monitor SIPs, see the following chapters of this guide:

•Chapter 3, "Cisco 10000 SIP-600 Overview"

•Chapter 12, "SIP and SPA Commands"

Using the Cisco IOS Event Tracer to Troubleshoot Problems

Note This feature is intended for use as a software diagnostic tool and should be configured only under the direction of a Cisco Technical Assistance Center (TAC) representative.

The Event Tracer feature provides a binary trace facility for troubleshooting Cisco IOS software. This feature gives Cisco service representatives additional insight into the operation of the Cisco IOS software and can be useful in helping to diagnose problems in the unlikely event of an operating system malfunction or, in the case of redundant systems, Route Processor (RP) switchover.

Event tracing works by reading informational messages from specific Cisco IOS software subsystem components that have been preprogrammed to work with event tracing, and by logging messages from those components into system memory. Trace messages stored in memory can be displayed on the screen or saved to a file for later analysis.

The SPAs currently support the spa component to trace SPA OIR-related events.

For more information about using the Event Tracer feature, refer to:

http://www.cisco.com/en/US/products/sw/iosswrel/ps1829/products_feature_guide09186a0080087164.html

Online Insertion and Removal of a SIP

The Cisco 10000 series router supports online insertion and removal (OIR) of the SPA interface processor (SIP) and of the shared port adapters (SPAs). Therefore, you can remove a SIP with its SPAs still intact, or you can remove a SPA independently from the SIP, leaving the SIP installed in the router.

Although graceful deactivation of a SIP is preferred using the hw-module slot reset hold command, the Cisco 10000 series router does support removal of the SIP without deactivating it first. If you plan to remove a SIP, you can deactivate the SIP first, using the hw-module slot reset hold command in privilege EXEC mode. When you deactivate a SIP using this command, it automatically deactivates each of the SPAs that are installed in that SIP. Therefore, it is not necessary to deactivate each of the SPAs prior to deactivating the SIP.

The hw-module slot reset command reloads the SIP. Beyond the reset command, the two options hold and release, either holds back the reset function or releases the reset function, implying deactivation and reactivation respectively, of the SIP and its installed interfaces.

Note Prior to using to hold and release commands, you must first enable service internal on the router by using the service internal command in global configuration mode.

Deactivating a SIP

To deactivate a SIP and its installed SPAs prior to removal of the SIP, use the following command in privilege EXEC mode:

Command

Purpose

Router# hw-module slot slot reset hold

Deactivates the SIP and any installed interfaces.

•slot—Specifies the chassis slot number where the SIP is installed.

•hold—Deactivates the SIP in the specified slot.

Reactivating a SIP

Once you deactivate a SIP, whether or not you have performed an OIR, you must use the hw-module slot reset release command in privilege EXEC mode to reactivate the SIP.

If you did not issue a command to deactivate the SPAs installed in a SIP, but you did deactivate the SIP using the hw-module slot reset release command, then you do not need to reactivate the SPAs after an OIR of the SIP. The installed SPAs automatically reactivate upon reactivation of the SIP in the router.

For example, consider the case where you remove a SIP from the router to replace it with another SIP. You reinstall the same SPAs into the new SIP. When you enter the hw-module slot reset release command on the router, the SPAs automatically reactivate with the new SIP.

To activate a SIP and its installed SPAs after the SIP has been deactivated, use the following command in privilege EXEC mode:

Command

Purpose

Router# hw-module slot slot reset release

Activates the SIP in the specified slot and its installed SPAs.

•slot—Specifies the chassis slot number where the SIP is installed.

•release—Reactivates the SIP and its installed SPAs.

The following is an example of the commands used to deactivate and activate a SIP:
Router# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Router(config)# service internal
Router(config)# end
Router# hw-module slot 1 reset ?
  hold     Hold in reset
  release  Release from reset
<cr>

Verifying the Status of a SIP

To view the status of a SIP, use the show diag command in privilege EXEC mode.
Router# show diag 1/0
Slot/Subslot 1/0:
        4jacket-1 card, 0 ports
        Card is full slot size
        Card is analyzed
        Card detected 07:42:30 ago
        Card uptime 0 days, 7 hours, 42 minutes, 31 seconds
        Card idle time 0 days, 3 hours, 3 minutes, 17 seconds
        Voltage status:
EEPROM contents, slot 1/0:
        Controller Type          : 1380
        Hardware Revision        : 1.0
        PCB Part Number          : 73-10771-01
        Board Revision           :
        Deviation Number         : 0-0
        Fab Version              : 01
        PCB Serial Number        : CSJ10497186
        RMA Test History         : 00
        RMA Number               : 0-0-0-0
        RMA History              : 00
        Top Assy. Part Number    : 800-27953-01
        CLEI Code                : UNASSIGNED
        Product Identifier (PID) : 10000-SIP-600
        Version Identifier (VID) : V00
LCMON version, slot 1/0
        LCDOS (C10000 PowerQUICC-III Line Card MONitor Image  : DEVELOPMENT BUILD 
jkotelly-spumoni_lc /vob/lcdos/obj-c10k-pq3-lcmon 102
)
        Built by jkotelly at Thu Nov  9 10:54:20 2006.
        Reset reason 0x00000008 (PRE hard reset).
Operational Image version, slot 1/0
        LCDOS (C10000 4 SPA Jacket Card Image (Spumoni) : DEVELOPMENT BUILD 
BLD-mayflower_lc.lcdos-nightly-mayflower-c10k /vob/lcdos/ob
j-c10k-spumoni 102) major version 1189663592.
        Built by ashah at Thu Sep 13 02:08:59 2007.
        SW Version 4.0
        Code MD5 068FBA5273DC8F598CC66E34D867A197
        FPGA MD5 89C8DD3DCD939BDD43DB19D514416137
        Expected Switchover Action: NO ACTION
        ECC 1 bit errors since last cleared (dd hh mm ss) = 0 (0 days, 0 hours, 0 minutes, 
0 seconds)
        ECC 1 bit errors while up (total) = 0
        Number of crashdumps recorded = 8
        SPA Information:
subslot 1/0   SPA-5X1GE-V2         ok
        SW Version 1.0
        Expected Switchover Action: NO ACTION
        Product Identifier (PID) : SPA-5X1GE-V2
        Version Identifier (VID) : V01
        PCB Serial Number        : JAB105201VV
        Top Assy. Part Number    : 68-2616-01
        Top Assy. Revision       : D0
        Hardware Revision        : 1.1
        CLEI Code                : CNUIANAAAA
        The Transceiver in slot 1 subslot 0 port 0 is ENABLED.
        The Transceiver in slot 1 subslot 0 port 1 is ENABLED.
        The Transceiver in slot 1 subslot 0 port 2 is ENABLED.
        The Transceiver in slot 1 subslot 0 port 3 is ENABLED.
        The Transceiver in slot 1 subslot 0 port 4 is ENABLED.
        SPA Information:
subslot 1/1   SPA-5X1GE-V2         ok
        SW Version 1.0
        Expected Switchover Action: NO ACTION
        Product Identifier (PID) : SPA-5X1GE-V2
        Version Identifier (VID) : V01
        PCB Serial Number        : JAB110202HP
        Top Assy. Part Number    : 68-2616-01
        Top Assy. Revision       : E0
        Hardware Revision        : 1.1
        CLEI Code                : CNUIANAAAA
        The Transceiver in slot 1 subslot 1 port 0 is ENABLED.
        The Transceiver in slot 1 subslot 1 port 1 is ENABLED.
        The Transceiver in slot 1 subslot 1 port 2 is ENABLED.
        The Transceiver in slot 1 subslot 1 port 3 is ENABLED.
        The Transceiver in slot 1 subslot 1 port 4 is ENABLED.
        SPA Information:
subslot 1/2   SPA-5X1GE-V2         ok
        SW Version 1.0
        Expected Switchover Action: NO ACTION
        Product Identifier (PID) : SPA-5X1GE-V2
        Version Identifier (VID) : V01
        PCB Serial Number        : JAB1042060D
        Top Assy. Part Number    : 68-2616-01
        Top Assy. Revision       : D0
        Hardware Revision        : 1.1
        CLEI Code                : CNUIANAAAA
        The Transceiver in slot 1 subslot 2 port 0 is ENABLED.
        The Transceiver in slot 1 subslot 2 port 1 is ENABLED.
        The Transceiver in slot 1 subslot 2 port 2 is ENABLED.
        The Transceiver in slot 1 subslot 2 port 3 is ENABLED.
        The Transceiver in slot 1 subslot 2 port 4 is ENABLED.
        SPA Information:
subslot 1/3   SPA-5X1GE-V2         ok
        SW Version 1.0
        Expected Switchover Action: NO ACTION
        Product Identifier (PID) : SPA-5X1GE-V2
        Version Identifier (VID) : V01
        PCB Serial Number        : JAB110202GA
        Top Assy. Part Number    : 68-2616-01
        Top Assy. Revision       : E0
        Hardware Revision        : 1.1
        CLEI Code                : CNUIANAAAA
        The Transceiver in slot 1 subslot 3 port 0 is ENABLED.
        The Transceiver in slot 1 subslot 3 port 1 is ENABLED.
        The Transceiver in slot 1 subslot 3 port 2 is ENABLED.
        The Transceiver in slot 1 subslot 3 port 3 is ENABLED.
        The Transceiver in slot 1 subslot 3 port 4 is ENABLED.
Router# 
Online Insertion and Removal of a SPA

The Cisco 10000 series router supports OIR of a SPA independently of removing the SIP. This means that a SIP can remain installed in the router with one SPA remaining active, while you remove another SPA from one of the SIP subslots. If you are not planning to immediately replace a SPA into the SIP, then be sure to install a blank filler plate in the subslot. The SIP should always be fully installed with either functional SPAs or blank filler plates.

The interface configuration is retained (recalled) if a SIP or SPA is removed and then replaced with one of the same type.

If you are planning to remove a SIP along with its SPAs, then you do not need to follow the instructions in this section. To remove a SIP, see the Installing and Removing a SIP chapter in the Cisco10000 Series SIP and SPA Hardware Configuration Guide.

Deactivating a SPA

Although graceful deactivation of a SPA is preferred using the hw-module subslot shutdown command, the Cisco 10000 series router does support removal of the SPA without deactivating it first. Before deactivating a SPA, ensure that the SIP is seated securely into the slot before pulling out the SPA itself.

If you are preparing for an OIR of a SPA, there are no standard interfaces to be shut down prior to deactivation of the SPA. The hw-module subslot shutdown command automatically stops traffic on the Gigabit Ethernet interfaces and deactivates them along with the SPA in preparation for OIR. Also, you do not need to independently restart any Gigabit Ethernet interfaces on a SPA after OIR of a SPA or SIP.

Note You do not need to shut down a SPA if you are removing it and replacing it with the same exact model of SPA in an online insertion and removal (OIR) operation. However, you must shut down a SPA and preprovision the line card slot whenever you are replacing a SPA with a different model of SPA.

To deactivate a SPA and all of its interfaces prior to removal of the SPA, use the following command in global configuration mode:

Command

Purpose

Router(config)# hw-module subslot slot/subslot shutdown [powered | unpowered]

Deactivates the SPA in the specified slot and subslot of the SIP.

•slot—Specifies the chassis slot number where the SIP is installed.

•subslot—Specifies the subslot number on a SIP where a SPA is installed.

•powered—Shuts down the SPA and all of its interfaces, and leaves them in an administratively down state with power enabled. This is the default state.

•unpowered—Shuts down the SPA and all of its interfaces, and leaves them in an administratively down state without power.

Router(config)# no card slot/subslot spa-type

(Optional) Preprovisions a line card slot in the Cisco 10000 series router to accept the specified SPA.

•slot—Specifies the number of the line card slot to preprovision.

•subslot—Specifies the number of the line card subslot to preprovision.

•spa-type—Specifies the type of SPA. The SPA type options depend on your Cisco IOS software version.

Reactivating a SPA

If you deactivate a SPA using the hw-module subslot shutdown global configuration command and need to reactivate it without performing an OIR, you need to use the no hw-module subslot shutdown global configuration command to reactivate the SPA and its interfaces.

To activate a SPA and its interfaces after the SPA has been deactivated, use the following command in global configuration mode:

Router(config)# no hw-module subslot slot/subslot shutdown

Activates the SPA and its interfaces in the specified slot and subslot of the SIP.

•slot—Specifies the chassis slot number where the SIP is installed.

•subslot—Specifies the subslot number on a SIP where a SPA is installed.

Verifying the Status of a SPA

When you deactivate a SPA, the corresponding interfaces are also deactivated. These interfaces no longer appear in the output of the show interface command.

To verify the deactivation of a SPA, enter the show hw-module subslot oir command in privileged EXEC configuration mode. Observe the Operational Status field associated with the SPA that you want to verify.

In the following example, the SPA located in subslot 1 of the SIP in chassis slot 7 is administratively down after entering the hw-module subslot shutdown powered command:
Router(config)# hw-module subslot 7/1 shutdown powered
Performing request on SPA 7/1 in jacket card in slot 7
Router(config)# exit
Router# show hw-module subslot 7/1 oir
Module        Model                Operational Status
------------- -------------------- ------------------------
subslot 7/1   SPA-5X1GE-V2         admin down
Router# 