Troubleshooting the SIPs and SSC
This chapter describes techniques that you can use to troubleshoot the operation of your SIPs.
It includes the following sections:
•General Troubleshooting Information
•Using the Cisco IOS Event Tracer to Troubleshoot Problems
•Troubleshooting Oversubscription on the Cisco 7600 SIP-400
•Preparing for Online Insertion and Removal of SIPs, SSCs, and SPAs
The first section provides information about basic interface troubleshooting. If you are having a problem with your SPA, use the steps in the "Using the Cisco IOS Event Tracer to Troubleshoot Problems" section to begin your investigation of a possible interface configuration problem.
To perform more advanced troubleshooting, see the other sections in this chapter.
General Troubleshooting Information
This section describes general information for troubleshooting SIPs, SSCs, and SPAs. It includes the following sections:
•Interpreting Console Error Messages
•Using debug Commands
•Using show Commands
Interpreting Console Error Messages
To view the explanations and recommended actions for Catalyst 6500 Series switch error messages, including messages related to Catalyst 6500 Series switch SIPs and SSCs, see the Catalyst 6500 Series Cisco IOS System Message Guide, 12.2SX.
System error messages are organized in the documentation according to the particular system facility that produces the messages. The SIP and SSC error messages use the following facility names:
•Cisco 7600 SIP-200—C7600_SIP200
•Cisco 7600 SIP-400—SIP400
•Cisco 7600 SIP-600—SIP600
•Cisco 7600 SSC-400—C7600_SSC400
Using debug Commands
Along with the other debug commands supported on the Catalyst 6500 Series switch, you can obtain specific debug information for SIPs and SSCs on the Catalyst 6500 Series switch using the debug hw-module privileged exec command.
The debug hw-module command is intended for use by Cisco Systems technical support personnel. For more information about the debug hw-module subslot command and other debug commands, see the Cisco IOS Debug Command Reference, Release 12.2.
Caution
Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use
debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use
debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased
debug command processing overhead will affect system use.
For information about other debug commands supported on the Catalyst 6500 Series switch, the Cisco IOS Debug Command Reference, Release 12.2. For more information about other commands that can be used on a Catalyst 6500 Series switch, see the Catalyst 6500 Series Cisco IOS Command Reference, 12.2SX.
Using show Commands
There are several show commands that you can use to monitor and troubleshoot the SIPs and SSCs on the Catalyst 6500 Series switch. This chapter describes using the show hw-module slot commands to perform troubleshooting of your SPA.
For more information about show commands to verify and monitor SIPs and SSCs, see Chapter 4, "Configuring the SIPs and SSC."
Using the Cisco IOS Event Tracer to Troubleshoot Problems
Note This feature is intended for use as a software diagnostic tool and should be configured only under the direction of a Cisco Technical Assistance Center (TAC) representative.
The Event Tracer feature provides a binary trace facility for troubleshooting Cisco IOS software. This feature gives Cisco service representatives additional insight into the operation of the Cisco IOS software and can be useful in helping to diagnose problems in the unlikely event of an operating system malfunction or, in the case of redundant systems, Route Processor switchover.
Event tracing works by reading informational messages from specific Cisco IOS software subsystem components that have been preprogrammed to work with event tracing, and by logging messages from those components into system memory. Trace messages stored in memory can be displayed on the screen or saved to a file for later analysis.
The SPAs currently support the "spa" component to trace SPA OIR-related events.
For more information about using the Event Tracer feature, refer to the following URL:
http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/evnttrcr.html
Troubleshooting Oversubscription on the Cisco 7600 SIP-400
As of Cisco IOS Release 12.2(18)SXF, when using the Cisco 7600 SIP-400 with the 2-Port Gigabit Ethernet SPA or the 1-Port OC-48c/STM-16 ATM SPA, consider the following oversubscription guidelines:
•The Cisco 7600 SIP-400 only supports installation of one 1-Port OC-48c/STM-16 ATM SPA without any other SPAs installed in the SIP.
•The Cisco 7600 SIP-400 supports installation of up to two 2-Port Gigabit Ethernet SPAs without any other SPAs installed in the SIP.
•The Cisco 7600 SIP-400 supports installation of any combination of OC-3 or OC-12 POS or ATM SPAs, up to a combined ingress bandwidth of OC-48 rates.
•The Cisco 7600 SIP-400 supports installation of any combination of OC-3 or OC-12 POS or ATM SPAs up to a combined ingress bandwidth of OC-24 rates, when installed with a single 2-Port Gigabit Ethernet SPA.
Configurations on the Cisco 7600 SIP-400 with an unsupported aggregate SPA bandwidth greater than OC-48 generates the following error message:
Error Message SLOT 3: 00:00:05: %SIPSPA-4-MAX_BANDWIDTH: Total SPA bandwidth exceeds
line card capacity of 2488 Mbps
Preparing for Online Insertion and Removal of SIPs, SSCs, and SPAs
The Catalyst 6500 Series switch supports online insertion and removal (OIR) of the SPA interface processor (SIP) or SPA services card (SSC), in addition to each of the shared port adapters (SPAs). Therefore, you can remove a SIP or SSC with its SPAs still intact, or you can remove a SPA independently from the SIP or SSC, leaving the SIP or SSC installed in the switch.
This section includes the following topics on OIR support:
•Preparing for Online Removal of a SIP or SSC
•Verifying Deactivation and Activation of a SIP or SSC
•Preparing for Online Removal of a SPA
•Verifying Deactivation and Activation of a SPA
•Deactivation and Activation Configuration Examples
Note For simplicity, any reference to "SIP" in this section also applies to the SSC.
Preparing for Online Removal of a SIP or SSC
To perform OIR of a SIP or SSC, power down a SIP (which automatically deactivates any installed SPAs) and remove the SIP with the SPAs still intact.
Although graceful deactivation of a SIP by using the no power enable module command is preferred, the Catalyst 6500 Series switch does support removal of the SIP without deactivating it first. If you plan to remove a SIP, you can deactivate the SIP first, using the no power enable module global configuration command. When you deactivate a SIP using this command, it automatically deactivates each of the SPAs that are installed in that SIP. Therefore, it is not necessary to deactivate each of the SPAs prior to deactivating the SIP.
Either a blank filler plate or a functional SPA should reside in every subslot of a SIP during normal operation.
For more information about the recommended procedures for physical removal of the SIP, refer to the Cisco 7600 Series Router SIP, SSC, and SPA Hardware Installation Guide.
Deactivating a SIP or SSC
To deactivate a SIP or SSC and its installed SPAs prior to removal of the SIP, use the following command in global configuration mode:
|
|
Router(config)# no power enable module slot |
Shuts down any installed interfaces, and deactivates the SIP in the specified slot, where: •slot—Specifies the chassis slot number where the SIP is installed. |
For more information about chassis slot numbering, refer to the "Identifying Slots and Subslots for SIPs, SSCs, and SPAs" section in this guide.
Reactivating a SIP or SSC
Once you deactivate a SIP or SSC, whether or not you have performed an OIR, you must use the power enable module global configuration command to reactivate the SIP.
If you did not issue a command to deactivate the SPAs installed in a SIP, but you did deactivate the SIP using the no power enable module command, then you do not need to reactivate the SPAs after an OIR of the SIP. The installed SPAs automatically reactivate upon reactivation of the SIP in the switch.
For example, if you remove a SIP from the switch to replace it with another SIP, you will reinstall the same SPAs into the new SIP. When you enter the power enable module command on the switch, the SPAs will automatically reactivate with the new SIP.
To activate a SIP and its installed SPAs after the SIP has been deactivated, use the following command in global configuration mode:
|
|
Router(config)# power enable module slot |
Activates the SIP in the specified slot and its installed SPAs, where: •slot—Specifies the chassis slot number where the SIP is installed. |
For more information about chassis slot numbering, refer to the "Identifying Slots and Subslots for SIPs, SSCs, and SPAs" section in this guide.
Verifying Deactivation and Activation of a SIP or SSC
To verify the deactivation of a SIP or SSC, enter the show module command in privileged EXEC configuration mode. Observe the Status field associated with the SIP that you want to verify.
The following example shows that the Cisco 7600 SIP-400 located in slot 13 is deactivated. This is indicated by its "PwrDown" status.
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
13 0 4-subslot SPA Interface Processor-400 7600-SIP-400 JAB0851042X
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
13 00e0.aabb.cc00 to 00e0.aabb.cc3f 0.525 12.2(PP_SPL_ 12.2(PP_SPL_ Ok
To verify activation and proper operation of a SIP, enter the show module command and observe "Ok" in the Status field as shown in the following example:
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
2 0 4-subslot SPA Interface Processor-200 7600-SIP-200 JAB074905S1
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
2 0000.0000.0000 to 0000.0000.003f 0.232 12.2(2004082 12.2(2004082 Ok
Preparing for Online Removal of a SPA
The Catalyst 6500 Series switch supports OIR of a SPA independently of removing the SIP or SSC. This means that a SIP can remain installed in the switch with one SPA remaining active, while you remove another SPA from one of the SIP subslots. If you are not planning to immediately replace a SPA into the SIP, then be sure to install a blank filler plate in the subslot. The SIP should always be fully installed with either functional SPAs or blank filler plates.
The interface configuration is retained (recalled) if a SIP or SPA is removed and then replaced with one of the same type. This is not the case if you replace a Cisco 7600 SIP-200 with a Cisco 7600 SIP-400 or vice versa.
If you are planning to remove a SIP along with its SPAs, then you do not need to follow the instructions in this section. To remove a SIP, see the "Preparing for Online Removal of a SIP or SSC" section.
Deactivating a SPA
Although graceful deactivation of a SPA is preferred using the hw-module subslot shutdown command, the Catalyst 6500 Series switch does support removal of the SPA without deactivating it first. Before deactivating a SPA, ensure that the SIP is seated securely into the slot before removing the SPA itself.
Note If you are preparing for an OIR of a SPA, it is not necessary to independently shut down each of the interfaces prior to deactivation of the SPA. The hw-module subslot shutdown command automatically stops traffic on the interfaces and deactivates them along with the SPA in preparation for OIR. You also do not need to independently restart any interfaces on a SPA after OIR of a SPA or SIP.
To deactivate a SPA and all of its interfaces prior to removal of the SPA, use the following command in global configuration mode:
|
|
Router(config)# hw-module subslot slot/subslot shutdown [powered | unpowered]
|
Deactivates the SPA in the specified slot and subslot of the SIP, where: •slot—Specifies the chassis slot number where the SIP is installed. •subslot—Specifies subslot number on a SIP where a SPA is installed. •powered—(Optional) Shuts down the SPA and all of its interfaces, and leaves them in an administratively down state with power enabled. This is the default state. •unpowered—(Optional) Shuts down the SPA and all of its interfaces, and leaves them in an administratively down state without power. |
For more information about chassis slot and SIP subslot numbering, refer to the "Identifying Slots and Subslots for SIPs, SSCs, and SPAs" section in this guide.
Reactivating a SPA
Note You do not need to reactivate a SPA after an OIR of either the SIP or a SPA if you did not deactivate the SPA prior to removal. If the switch is running, then the SPAs automatically start upon insertion into the SIP or with insertion of a SIP into the switch.
If you deactivate a SPA using the hw-module subslot shutdown global configuration command and need to reactivate it without performing an OIR, you need to use the no hw-module subslot shutdown global configuration command to reactivate the SPA and its interfaces.
To activate a SPA and its interfaces after the SPA has been deactivated, use the following command in global configuration mode:
|
|
Router(config)# no hw-module subslot slot/subslot shutdown
|
Activates the SPA and its interfaces in the specified slot and subslot of the SIP, where: •slot—Specifies the chassis slot number where the SIP is installed. •subslot—Specifies subslot number on a SIP where a SPA is installed. |
Verifying Deactivation and Activation of a SPA
When you deactivate a SPA, the corresponding interfaces are also deactivated. This means that these interfaces will no longer appear in the output of the show interface command.
To verify the deactivation of a SPA, enter the show hw-module subslot all oir command in privileged EXEC configuration mode. Observe the Operational Status field associated with the SPA that you want to verify.
In the following example, the SPA located in subslot 1 of the SIP in slot 2 of the switch is administratively down from the hw-module subslot shutdown command:
Router# show hw-module subslot all oir
Module Model Operational Status
-------------- ------------------ -------------------------
subslot 2/0 SPA-4XOC3-POS ok
subslot 2/1 SPA-4XOC3-ATM admin down
To verify activation and proper operation of a SPA, enter the show hw-module subslot all oir command and observe "ok" in the Operational Status field as shown in the following example:
Router# show hw-module subslot all oir
Module Model Operational Status
-------------- ------------------ -------------------------
subslot 2/0 SPA-4XOC3-POS ok
subslot 2/1 SPA-4XOC3-ATM ok
Deactivation and Activation Configuration Examples
This section provides the following examples of deactivating and activating SIPs and SPAs:
•Deactivation of a SIP Configuration Example
•Activation of a SIP Configuration Example
•Deactivation of a SPA Configuration Example
•Activation of a SPA Configuration Example
Deactivation of a SIP Configuration Example
Deactivate a SIP when you want to perform OIR of the SIP. The following example deactivates the SIP that is installed in slot 5 of the switch, its SPAs, and all of the interfaces. The corresponding console messages are shown:
Router# configure terminal
Router(config)# no power enable module 5
1w4d: %OIR-6-REMCARD: Card removed from slot 5, interfaces disabled
1w4d: %C6KPWR-SP-4-DISABLED: power to module in slot 5 set off (admin request)
Activation of a SIP Configuration Example
Activate a SIP if you have previously deactivated it. If you did not deactivate the SPAs, the SPAs automatically reactivate with reactivation of the SIP.
The following example activates the SIP that is installed in slot 5 of the switch, its SPA, and all of the
interfaces (as long as the hw-module subslot shutdown command was not issued to also deactivate
the SPA):
Router# configure terminal
Router(config)# power enable module 5
Notice that there are no corresponding console messages shown with the activation. If you reenter the
power enable module command, a message is displayed indicating that the module is already
enabled:
Router(config)# power enable module 5
% module is already enabled
Deactivation of a SPA Configuration Example
Deactivate a SPA when you want to perform OIR of that SPA. The following example deactivates the SPA (and its interfaces) that is installed in subslot 0 of the SIP located in slot 2 of the switch and removes power to the SPA. Notice that no corresponding console messages are shown.
Router# configure terminal
Router(config)# hw-module subslot 2/0 shutdown unpowered
Activation of a SPA Configuration Example
Activate a SPA if you have previously deactivated it. If you have not deactivated a SPA and its interfaces during OIR of a SIP, then the SPA is automatically reactivated upon reactivation of the SIP.
The following example activates the SPA that is installed in slot 2 of the switch and all of its
interfaces:
Router# configure terminal
Router(config)# no hw-module subslot 2/0 shutdown
Router#