Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Software Configuration Guide
Troubleshooting the SIP
Downloads: This chapterpdf (PDF - 252.0KB) The complete bookPDF (PDF - 4.44MB) | Feedback

Table of Contents

Troubleshooting the SIP

General Troubleshooting Information

Interpreting Console Error Messages

Using debug Commands

Using show Commands

Using the Cisco IOS Event Tracer to Troubleshoot Problems

Preparing for Online Insertion and Removal of a SIP and SPAs

Preparing for Online Removal of a SIP

Deactivating a SIP

Reactivating a SIP

Verifying the Deactivation and Activation of a SIP

Preparing for Online Removal of a SPA

Deactivating a SPA

Reactivating a SPA

Verifying the Deactivation and Activation of a SPA

Deactivation and Activation Configuration Examples

Deactivation of a SIP Configuration

Activation of a SIP Configuration

Deactivation of a SPA Configuration

Activation of a SPA Configuration Example

Troubleshooting the SIP

This chapter describes techniques that you can use to troubleshoot the operation of your SIPs.

It includes the following sections:

The first section provides information about basic interface troubleshooting.

To perform more advanced troubleshooting, see the other sections in this chapter.

General Troubleshooting Information

This section describes general information for troubleshooting SIPs and SPAs. It includes the following sections:

Interpreting Console Error Messages

System error messages are organized according to the particular system facility that produces the messages. The SIP error messages use the following facility names:

  • ASR1000_SIP
  • ASR1000_SIP_SPA

Using debug Commands

Along with the other debug commands supported on the Cisco Aggregation Services Router 1000 Series, you can obtain specific debug information for the SIP on the Cisco ASR 1000 Series Routers using the debug hw-module command in privileged EXEC mode.

The debug hw-module command is intended for use by Cisco technical support personnel.


Caution Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.

For information about other debug commands supported on the Cisco Aggregation Services Router 1000 Series, refer to the Cisco IOS Debug Command Reference publications for relevant Cisco IOS releases.

Using show Commands

There are several show commands that you can use to monitor and troubleshoot the SIP and SPA on the Cisco ASR 1000 Series Routers. This chapter describes using the show hw-module subslot command to perform troubleshooting of your SPA.

Using the Cisco IOS Event Tracer to Troubleshoot Problems

This feature is intended for use as a software diagnostic tool and should be configured only under the direction of a Cisco Technical Assistance Center (TAC) representative.

The Event Tracer feature provides a binary trace facility for troubleshooting Cisco IOS software. This feature gives Cisco service representatives additional insight into the operation of the Cisco IOS software and can be useful in helping to diagnose problems in the unlikely event of an operating system malfunction or, in the case of redundant systems, Route Processor switchover.

Event tracing works by reading informational messages from specific Cisco IOS software subsystem components that have been preprogrammed to work with event tracing, and by logging messages from those components into system memory. Trace messages stored in memory can be displayed on the screen or saved to a file for later analysis.

The SPAs currently support the “spa” component to trace SPA OIR-related events.

For more information about using the Event Tracer feature, refer to the following URL:

http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/evnttrcr.html.

Preparing for Online Insertion and Removal of a SIP and SPAs

The Cisco Aggregation Services Router 1000 Series support online insertion and removal (OIR) of the SPA interface processor (SIP), in addition to each of the shared port adapters (SPAs). Therefore, you can remove a SIP with its SPAs still intact, or you can remove a SPA independently from the SIP, leaving the SIP installed in the router.

This section includes the following topics on OIR support:

Preparing for Online Removal of a SIP

The Cisco Aggregation Services Router 1000 Series support OIR of a SIP. To do this, you can power down a SIP (which automatically deactivates any installed SPAs) and remove the SIP with the SPAs still intact.


Note If SIP-40 is installed with the bandwidth value greater than 11.2G and if SIP-40 is replaced with SIP-10 then the bandwidth of SIP-10 is reset to its default maximum bandwidth of 11.2G. An informational message indicating “Setting input bandwidth to ESI max bandwidth: 11199896” is displayed on the console. This message is displayed when SIP-40 is downgraded to SIP-10 from Cisco IOS XE Release 3.1.0S onwards.


Although graceful deactivation of a SIP is preferred using the hw-module slot stop command, the Cisco Aggregation Services Router 1000 Series do support removal of the SIP without deactivating it first. If you plan to remove a SIP, you can deactivate the SIP first, using the hw-module slot stop command in global configuration mode. When you deactivate a SIP using this command, it automatically deactivates each of the SPAs that are installed in that SIP. Therefore, it is not necessary to deactivate each of the SPAs prior to deactivating the SIP.


Note Either a blank filler plate or a functional SPA should reside in every subslot of a SIP during normal operation.


For more information about the recommended procedures for physical removal of the SIP, refer to the Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Hardware Installation Guide.

Deactivating a SIP

To deactivate a SIP and its installed SPAs prior to removal of the SIP, use the following command in global configuration mode:

 

Command
Purpose

Router(config)# hw-module slot slot-number stop

Shuts down any installed interfaces, and deactivates the SIP in the specified slot, where:

  • slot-number— Specifies the chassis slot number where the SIP is installed.
  • stop Deactivates the SIP and places it in reset mode.

For more information about chassis slot numbering, see the “Identifying Slots and Subslots for the SIPs and SPAs” section.

Reactivating a SIP

If you did deactivate the SIP using the hw-module slot stop command, then you do not need to reactivate the SPAs after an OIR of the SIP. The installed SPAs automatically reactivate upon reactivation of the SIP in the Cisco Aggregation Services Router 1000 Series.

For example, consider the case where you remove a SIP from the Cisco Aggregation Services Router 1000 Series to replace it with another SIP. You reinstall the same SPAs into the new SIP. When you enter the hw-module slot start command on the Cisco Aggregation Services Router 1000 Series, the SPAs automatically reactivate with the new SIP.

To activate a SIP and its installed SPAs after the SIP has been deactivated, use the following command in global configuration mode:

 

Command
Purpose

Router(config)# hw-module slot slot-number start

Activates the SIP in the specified slot and its installed SPAs, where:

  • slot-number— Specifies the chassis slot number where the SIP is installed.
  • start —Activates the SIP and removes it from reset mode.

For more information about chassis slot numbering, see the “Identifying Slots and Subslots for the SIPs and SPAs” section.

Verifying the Deactivation and Activation of a SIP

To verify the deactivation of a SIP, enter the show platform command in privileged EXEC configuration mode. Observe the State field associated with the SIP that you want to verify.

The following example shows that the Cisco ASR1000-SIP10 located in slot 1 is deactivated.

Router# show platform
Chassis type: ASR1006
 
Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0 ASR1000-SIP10 ok 2d20h
0/0 SPA-1XOC12-POS ok 2d20h
0/1 SPA-1XOC12-POS ok 1d21h
0/2 SPA-5X1GE-V2 ok 2d20h
0/3 SPA-4XOC3-POS-V2 ok 2d20h
1 ASR1000-SIP10 disabled 1d23h
R0 ASR1000-RP1 ok, active 1w3d
F0 ASR1000-ESP10 ok, active 1w3d
P0 Unknown ps, fail never
P1 Unknown ps, fail never
 
Slot CPLD Version Firmware Version
--------- ------------------- ---------------------------------------
0 06120701 12.2(20071009:051408) [gschnorr-mcp_...
1 07091401 12.2(20070807:170946) [gschnorr-mcp_...
R0 07010515 12.2(33r)XN2
F0 07021400 12.2(20071030:180953) [gschnorr-mcp_...
 
 

To verify activation and proper operation of a SIP, enter the show platform command and observe “ok” in the State field as shown in the following example:

 
Router# show platform
Chassis type: ASR1006
 
Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0 ASR1000-SIP10 ok 03:19:46
0/0 SPA-5X1GE-V2 ok 03:18:28
0/1 SPA-8X1FE-TX-V2 ok 03:18:28
0/2 SPA-2XCT3/DS0 ok 03:18:19
1 ASR1000-SIP10 ok 03:19:46
1/0 SPA-2XOC3-POS ok 03:18:28
1/1 SPA-8XCHT1/E1 ok 03:18:28
1/2 SPA-2XT3/E3 ok 03:18:18
R0 ASR1000-RP1 ok, active 03:19:46
F0 ASR1000-ESP10 ok, active 03:19:46
P0 ASR1006-PWR-AC ok 03:19:14
P1 ASR1006-FAN ok 03:19:14
 

The following example shows the Cisco ASR 1000-SIP10 installed in slot 0 and Cisco ASR1000-SIP40 installed in slots 1,2,3, and 4 for the Cisco ASR 1013 Router:

Router# show platform
Chassis type: ASR1013
 
Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0 ASR1000-SIP10 ok 1w0d
1 ASR1000-SIP40 ok 1w0d
1/1 SPA-5X1GE-V2 ok 1w0d
2 ASR1000-SIP40 ok 1w0d
2/1 SPA-1X10GE-L-V2 ok 1w0d
2/3 SPA-1X10GE-L-V2 ok 1w0d
3 ASR1000-SIP40 ok 1w0d
3/3 SPA-4XT3/E3 ok 1w0d
4 ASR1000-SIP40 ok 1w0d
4/2 SPA-5X1GE-V2 ok 1w0d
4/3 SPA-4XCT3/DS0 ok 1w0d
5 ASR1000-SIP40 ok 1w0d
R0 ASR1000-RP2 ok, active 1w0d
R1 ASR1000-RP2 ok, standby 1w0d
F0 ASR1000-ESP40 ok, active 1w0d
P0 ASR1013-PWR-AC ok 1w0d
P1 ASR1013-PWR-AC ps, fail 1w0d
P2 ASR1013-PWR-AC ok 1w0d
P3 ASR1013-PWR-AC ps, fail 1w0d
 
Slot CPLD Version Firmware Version
--------- ------------------- ---------------------------------------
0 00200800 15.0(1r)S
1 00200800 15.0(1r)S
2 00200800 15.0(1r)S
3 00200800 15.0(1r)S
4 00200800 15.0(1r)S
5 00200800 15.0(1r)S
R0 10021901 15.0(1r)S
R1 10021901 15.0(1r)S
F0 1001270D 15.0(1r)S

Preparing for Online Removal of a SPA

The Cisco Aggregation Services Router 1000 Series support OIR of a SPA independently of removing the SIP. This means that a SIP can remain installed in the Cisco ASR 1000 Series Aggregation Services Routers with one SPA remaining active, while you remove another SPA from one of the SIP subslots. If you are not planning to immediately replace a SPA into the SIP, then be sure to install a blank filler plate in the subslot. The SIP should always be fully installed with either functional SPAs or blank filler plates.

If you are planning to remove a SIP along with its SPAs, then you do not need to follow the instructions in this section. To remove a SIP, see the “Preparing for Online Removal of a SIP” section.

Deactivating a SPA

Although graceful deactivation of a SPA is preferred using the hw-module subslot stop command in EXEC mode, the Cisco Aggregation Services Router 1000 Series support removal of the SPA without deactivating it first. Before deactivating a SPA, ensure that the SIP is seated securely into the slot before pulling out the SPA itself.


Note If you are preparing for an OIR of a SPA, it is not necessary to independently shut down each of the interfaces prior to deactivation of the SPA. The hw-module subslot stop command in global configuration mode automatically stops traffic on the interfaces and deactivates them along with the SPA in preparation for OIR. In similar fashion, you do not need to independently restart any interfaces on a SPA after OIR of a SPA or SIP.


Choosing the Mode of the hw-module subslot Command

You can deactivate a SPA and all of its interfaces by using the hw-module subslot command in one of two modes:

  • If you choose to use the hw-module subslot shutdown command in global configuration mode, you change the configuration so that no matter how many times the router is rebooted, the SPA will not boot. This command is useful when you need to shut down a SPA in a remote location and ensure that it does not come back up if the router is rebooted.
  • If you choose to use the hw-module subslot stop command in global configuration mode, you cause the SPA to gracefully shut down. However, the SPA will come up again when executing the hw-module subslot start command.

To deactivate a SPA and all of its interfaces prior to removal of the SPA, use the following commands in global configuration mode:

 

Command
Purpose

Router(config)# hw-module subslot slot-number/subslot-number shutdown [ powered | unpowered ]

Deactivates the SPA in the specified slot and subslot of the SIP, where:

  • slot-number— Specifies the chassis slot number where the SIP is installed.
  • subslot-number —Specifies the subslot number on a SIP where a SPA is installed.
  • powered —(Optional) Shuts down the SPA and all of its interfaces, and leaves them in an administratively down state with power enabled. This is the default state.
  • unpowered —(Optional) Shuts down the SPA and all of its interfaces, and leaves them in an administratively down state without power.

Router(config)# hw-module subslot slot- number /subslot- number {reload | stop | start}

Deactivates the SPA in the specified slot and subslot of the SIP, where:

  • slot-number— Specifies the chassis slot number where the SIP is installed.
  • subslot-number —Specifies the subslot number on a SIP where a SPA is installed.
  • reload —Stops and restarts the specified SPA.
  • stop —Stops the specified SPA .
  • start—Starts the specified SPA.

For more information about chassis slot and SIP subslot numbering, refer to the “Identifying Slots and Subslots for the SIPs and SPAs” section.

Reactivating a SPA


Note You do not need to reactivate a SPA after an OIR of either the SIP or a SPA if you did not deactivate the SPA prior to removal when using the hw-module subslot command in global configuration mode. If the Cisco ASR 1000 Series Aggregation Services Routers is running, then the SPAs automatically start upon insertion into the SIP or with insertion of a SIP into the Cisco ASR 1000 Series Aggregation Services Routers.


If you deactivate a SPA using the hw-module subslot stop command, and need to reactivate it without performing an OIR, you must use the hw-module subslot start command to reactivate the SPA and its interfaces.

To activate a SPA and its interfaces after the SPA has been deactivated, use the following command in global configuration mode:

 

Command
Purpose

Router(config)# no hw-module subslot slot-number/subslot-number shutdown

 

Activates the SPA and its interfaces in the specified slot and subslot of the SIP, where:

  • slot- number Specifies the chassis slot number where the SIP is installed.
  • subslot- number —Specifies subslot number on a SIP where a SPA is installed.

Verifying the Deactivation and Activation of a SPA

When you deactivate a SPA, the corresponding interfaces are also deactivated. This means that these interfaces will no longer appear in the output of the show interface command.

To verify the deactivation of a SPA, enter the show hw-module subslot all oir command in privileged EXEC mode. Observe the Operational Status field associated with the SPA that you want to verify.

In the following example, the SPA located in subslot 1 of the SIP in slot 2 of the Cisco ASR 1000 Series Aggregation Services Routers is administratively down from the hw-module subslot shutdown command:

Router# show hw-module subslot all oir
Module Model Operational Status
-------------- ------------------ -------------------------
subslot 2/0 SPA-1X10GE-L-V2 ok
subslot 2/1 SPA-8X1GE-V2 admin down
 

To verify activation and proper operation of a SPA, enter the show hw-module subslot all oir command and observe “ok” in the Operational Status field as shown in the following example:

Router# show hw-module subslot all oir
Module Model Operational Status
-------------- ------------------ -------------------------
subslot 2/0 SPA-1X10GE-L-V2 ok
subslot 2/1 SPA-8X1GE-V2 ok

Deactivation and Activation Configuration Examples

This section provides the following examples of deactivating and activating SIPs and SPAs:

Deactivation of a SIP Configuration

Deactivate a SIP when you want to perform OIR of the SIP. The following example deactivates the SIP that is installed in slot 1 of the Cisco ASR 1000 Series Aggregation Services Routers, its SPAs, and all of the interfaces:

Router(config)# hw-module slot 1 stop

Note Deactivation of SIP is not supported on Cisco ASR 1001, Cisco ASR 1002 and Cisco ASR 1002-X routers.


Activation of a SIP Configuration

Activate a SIP if you have previously deactivated it. If you did not deactivate the SPAs, the SPAs automatically reactivate with reactivation of the SIP.

The following example activates the SIP that is installed in slot 1 of the Cisco ASR 1000 Series Aggregation Services Routers, its SPAs, and all of the interfaces (as long as the hw-module subslot shutdown command was not issued to also deactivate the SPA):

Router(config)# hw-module slot 1 start
 

There are no corresponding console messages shown with activation using this command. If you re-enter the hw-module slot start command, a message is displayed indicating that the module is already enabled.

Deactivation of a SPA Configuration

Deactivate a SPA when you want to perform OIR of that SPA. The following example deactivates the SPA (and its interfaces) that is installed in subslot 0 of the SIP located in slot 1 of the Cisco ASR 1000 Series Aggregation Services Routers and removes power to the SPA:

Router(config)# hw-module subslot 1/0 shutdown unpowered

Activation of a SPA Configuration Example

Activate a SPA if you have previously deactivated it. If you have not deactivated a SPA and its interfaces during OIR of a SIP, then the SPA is automatically reactivated upon reactivation of the SIP.

The following example activates the SPA that is installed in subslot 0 of the SIP located in slot 1 of the Cisco ASR 1000 Series Aggregation Services Routers:

Router(config)# no hw-module subslot 1/0 shutdown