Troubleshooting the SIP

This chapter describes techniques that you can use to troubleshoot the operation of your SIPs.

It includes the following sections:

The first section provides information about basic interface troubleshooting.

To perform more advanced troubleshooting, see the other sections in this chapter.

General Troubleshooting Information

This section describes general information for troubleshooting SIPs and SPAs.

Interpreting Console Error Messages

System error messages are organized according to the particular system facility that produces the messages. The SIP error messages use the following facility names:

  • ASR1000_SIP
  • ASR1000_SIP_SPA

Using debug Commands

Along with the other debug commands supported on the Cisco Aggregation Services Router 1000 Series, you can obtain specific debug information for the SIP on the Cisco ASR 1000 Series Routers using the debug hw-module command in privileged EXEC mode.

The debug hw-module command is intended for use by Cisco technical support personnel.


Caution

Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.

For information about other debug commands supported on the Cisco Aggregation Services Router 1000 Series, refer to the following:

Using show Commands

There are several show commands that you can use to monitor and troubleshoot the SIP and SPA on the Cisco ASR 1000 Series Routers. This chapter describes using the show hw-module subslot command to perform troubleshooting of your SPA.

Using the Cisco IOS Event Tracer to Troubleshoot Problems

This feature is intended for use as a software diagnostic tool and should be configured only under the direction of a Cisco Technical Assistance Center (TAC) representative.

The Event Tracer feature provides a binary trace facility for troubleshooting Cisco IOS software. This feature gives Cisco service representatives additional insight into the operation of the Cisco IOS software and can be useful in helping to diagnose problems in the unlikely event of an operating system malfunction or, in the case of redundant systems, Route Processor switchover.

Event tracing works by reading informational messages from specific Cisco IOS software subsystem components that have been preprogrammed to work with event tracing, and by logging messages from those components into system memory. Trace messages stored in memory can be displayed on the screen or saved to a file for later analysis.

The SPAs currently support the “spa” component to trace SPA OIR-related events. For more information about using the Event Tracer feature, refer to the following URL: http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/bsm/configuration/15-mt/bsm-15-mt-book/bsm-event-tracer.html.

Preparing for Online Insertion and Removal of a SIP and SPAs

The Cisco Aggregation Services Router 1000 Series support online insertion and removal (OIR) of the SPA interface processor (SIP), in addition to each of the shared port adapters (SPAs). Therefore, you can remove a SIP with its SPAs still intact, or you can remove a SPA independently from the SIP, leaving the SIP installed in the router.

This section includes the following topics on OIR support:

Preparing for Online Removal of a SIP

The Cisco Aggregation Services Router 1000 Series support OIR of a SIP. To do this, you can power down a SIP (which automatically deactivates any installed SPAs) and remove the SIP with the SPAs still intact.


Note
If SIP-40 is installed with the bandwidth value greater than 11.2G and if SIP-40 is replaced with SIP-10 then the bandwidth of SIP-10 is reset to its default maximum bandwidth of 11.2G. An informational message indicating “Setting input bandwidth to ESI max bandwidth: 11199896” is displayed on the console. This message is displayed when SIP-40 is downgraded to SIP-10 from Cisco IOS XE Release 3.1.0S onwards.

Although graceful deactivation of a SIP is preferred using the hw-module slot stop command, the Cisco Aggregation Services Router 1000 Series do support removal of the SIP without deactivating it first. If you plan to remove a SIP, you can deactivate the SIP first, using the hw-module slot stop command in global configuration mode. When you deactivate a SIP using this command, it automatically deactivates each of the SPAs that are installed in that SIP. Therefore, it is not necessary to deactivate each of the SPAs prior to deactivating the SIP.


Note
Either a blank filler plate or a functional SPA should reside in every subslot of a SIP during normal operation.

For more information about the recommended procedures for physical removal of the SIP, refer to the Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Hardware Installation Guide.

Deactivating a SIP

To deactivate a SIP and its installed SPAs prior to removal of the SIP, use the following command in global configuration mode:

Command

Purpose

Router(config)# hw-module slot slot-number stop

Shuts down installed interfaces and deactivates the SIP in the specified slot.

  • slot-number—Specifies the chassis slot number where the SIP is installed

  • stop —Deactivates the SIP and places it in reset mode

For more information about chassis slot numbering, see the “Identifying Slots and Subslots for the SIPs and SPAs” section in the Configuring the SIP chapter.

Reactivating a SIP

If you did deactivate the SIP using the hw-module slot stop command, then you do not need to reactivate the SPAs after an OIR of the SIP. The installed SPAs automatically reactivate upon reactivation of the SIP in the Cisco Aggregation Services Router 1000 Series.

For example, consider the case where you remove a SIP from the Cisco Aggregation Services Router 1000 Series to replace it with another SIP. You reinstall the same SPAs into the new SIP. When you enter the hw-module slot start command on the Cisco Aggregation Services Router 1000 Series, the SPAs automatically reactivate with the new SIP.

To activate a SIP and its installed SPAs after the SIP has been deactivated, use the following command in global configuration mode:

Command

Purpose

Router(config)# hw-module slot slot-number start

Activates the SIP in the specified slot and its installed SPAs.

  • slot-number—Specifies the chassis slot number where the SIP is installed

  • start —Activates the SIP and removes it from reset mode

For more information about chassis slot numbering, see the “Identifying Slots and Subslots for the SIPs and SPAs” section in the Configuring the SIP chapter.

Verifying the Deactivation and Activation of a SIP

To verify the deactivation of a SIP, enter the show platform command in privileged EXEC configuration mode. Observe the State field associated with the SIP that you want to verify.

The following example shows that the Cisco ASR1000-SIP10 located in slot 1 is deactivated. 


Router# show platform
Chassis type: ASR1006             
Slot      Type                State                 Insert time (ago) 
--------- ------------------- --------------------- ----------------- 
0         ASR1000-SIP10       ok                    2d20h         
 0/0      SPA-1XOC12-POS      ok                    2d20h         
 0/1      SPA-1XOC12-POS      ok                    1d21h         
 0/2      SPA-5X1GE-V2        ok                    2d20h         
 0/3      SPA-4XOC3-POS-V2    ok                    2d20h         
1         ASR1000-SIP10       disabled              1d23h         
R0        ASR1000-RP1         ok, active            1w3d          
F0        ASR1000-ESP10       ok, active            1w3d          
P0        Unknown             ps, fail              never         
P1        Unknown             ps, fail              never         
Slot      CPLD Version        Firmware Version                        
--------- ------------------- --------------------------------------- 
0         06120701            12.2(20071009:051408) [gschnorr-mcp_...
1         07091401            12.2(20070807:170946) [gschnorr-mcp_...
R0        07010515            12.2(33r)XN2                        
F0        07021400            12.2(20071030:180953) [gschnorr-mcp_...

To verify activation and proper operation of a SIP, enter the show platform command and observe “ok” in the State field as shown in the following example: 


Router# show platform
Chassis type: ASR1006             
Slot      Type                State                 Insert time (ago) 
--------- ------------------- --------------------- ----------------- 
0         ASR1000-SIP10       ok                    03:19:46      
 0/0      SPA-5X1GE-V2        ok                    03:18:28      
 0/1      SPA-8X1FE-TX-V2     ok                    03:18:28      
 0/2      SPA-2XCT3/DS0       ok                    03:18:19      
1         ASR1000-SIP10       ok                    03:19:46      
 1/0      SPA-2XOC3-POS       ok                    03:18:28      
 1/1      SPA-8XCHT1/E1       ok                    03:18:28      
 1/2      SPA-2XT3/E3         ok                    03:18:18      
R0        ASR1000-RP1         ok, active            03:19:46      
F0        ASR1000-ESP10       ok, active            03:19:46      
P0        ASR1006-PWR-AC      ok                    03:19:14      
P1        ASR1006-FAN         ok                    03:19:14

The following example shows the Cisco ASR 1000-SIP10 installed in slot 0 and Cisco ASR1000-SIP40 installed in slots 1,2,3, and 4 for the Cisco ASR 1013 Router: 


Router# show platform 
Chassis type: ASR1013
Slot      Type                State                 Insert time (ago)
--------- ------------------- --------------------- -----------------
0         ASR1000-SIP10       ok                    1w0d
1         ASR1000-SIP40       ok                    1w0d
 1/1      SPA-5X1GE-V2        ok                    1w0d
2         ASR1000-SIP40       ok                    1w0d
 2/1      SPA-1X10GE-L-V2     ok                    1w0d
 2/3      SPA-1X10GE-L-V2     ok                    1w0d
3         ASR1000-SIP40       ok                    1w0d
 3/3      SPA-4XT3/E3         ok                    1w0d
4         ASR1000-SIP40       ok                    1w0d
 4/2      SPA-5X1GE-V2        ok                    1w0d
 4/3      SPA-4XCT3/DS0       ok                    1w0d
5         ASR1000-SIP40       ok                    1w0d
R0        ASR1000-RP2         ok, active            1w0d
R1        ASR1000-RP2         ok, standby           1w0d
F0        ASR1000-ESP40       ok, active            1w0d
P0        ASR1013-PWR-AC      ok                    1w0d
P1        ASR1013-PWR-AC      ps, fail              1w0d
P2        ASR1013-PWR-AC      ok                    1w0d
P3        ASR1013-PWR-AC      ps, fail              1w0d
Slot      CPLD Version        Firmware Version
--------- ------------------- ---------------------------------------
0         00200800            15.0(1r)S
1         00200800            15.0(1r)S
2         00200800            15.0(1r)S
3         00200800            15.0(1r)S
4         00200800            15.0(1r)S
5         00200800            15.0(1r)S
R0        10021901            15.0(1r)S
R1        10021901            15.0(1r)S
F0        1001270D            15.0(1r)S

Preparing for Online Removal of a SPA

The Cisco Aggregation Services Router 1000 Series support OIR of a SPA independently of removing the SIP. This means that a SIP can remain installed in the Cisco ASR 1000 Series Aggregation Services Routers with one SPA remaining active, while you remove another SPA from one of the SIP subslots. If you are not planning to immediately replace a SPA into the SIP, then be sure to install a blank filler plate in the subslot. The SIP should always be fully installed with either functional SPAs or blank filler plates.

If you are planning to remove a SIP along with its SPAs, then you do not need to follow the instructions in this section. To remove a SIP, see the Preparing for Online Removal of a SIP.

Deactivating a SPA

Although graceful deactivation of a SPA is preferred using the hw-module subslot stop command in EXEC mode, the Cisco Aggregation Services Router 1000 Series support removal of the SPA without deactivating it first. Before deactivating a SPA, ensure that the SIP is seated securely into the slot before pulling out the SPA itself.


Note
If you are preparing for an OIR of a SPA, it is not necessary to independently shut down each of the interfaces prior to deactivation of the SPA. The hw-module subslot stop command in global configuration mode automatically stops traffic on the interfaces and deactivates them along with the SPA in preparation for OIR. In similar fashion, you do not need to independently restart any interfaces on a SPA after OIR of a SPA or SIP.
Choosing the Mode of the hw-module subslot Command

You can deactivate a SPA and all of its interfaces by using the hw-module subslot command in one of two modes:

  • If you choose to use the hw-module subslot shutdown command in global configuration mode, you change the configuration so that no matter how many times the router is rebooted, the SPA will not boot. This command is useful when you need to shut down a SPA in a remote location and ensure that it does not come back up if the router is rebooted.

  • If you choose to use the hw-module subslot stop command in global configuration mode, you cause the SPA to gracefully shut down. However, the SPA will come up again when executing the hw-module subslot start command.

To deactivate a SPA and all of its interfaces prior to removal of the SPA, use the following commands in global configuration mode:

Command

Purpose

Router(config)# hw-module subslot slot-number/ subslot-number shutdown powered | unpowered

Deactivates the SPA in the specified slot and subslot of the SIP, where:

  • slot-number—Specifies the chassis slot number where the SIP is installed.

  • subslot-number —Specifies the subslot number on a SIP where a SPA is installed.

  • powered —(Optional) Shuts down the SPA and all of its interfaces, and leaves them in an administratively down state with power enabled. This is the default state.

  • unpowered —(Optional) Shuts down the SPA and all of its interfaces, and leaves them in an administratively down state without power.

Router(config)# hw-module subslot slot-number/ subslot-number {reload | stop | start }

Deactivates the SPA in the specified slot and subslot of the SIP, where:

  • slot-number—Specifies the chassis slot number where the SIP is installed.

  • subslot-number—Specifies the subslot number on a SIP where a SPA is installed.

  • reload —Stops and restarts the specified SPA.

  • stop —Stops the specified SPA.

  • start —Starts the specified SPA.

For more information about chassis slot and SIP subslot numbering, refer to the “Identifying Slots and Subslots for the SIPs and SPAs” section in the Configuring the SIP chapter.

Reactivating a SPA


Note
You do not need to reactivate a SPA after an OIR of either the SIP or a SPA if you did not deactivate the SPA prior to removal when using the hw-module subslot command in global configuration mode. If the Cisco ASR 1000 Series Aggregation Services Routers is running, then the SPAs automatically start upon insertion into the SIP or with insertion of a SIP into the Cisco ASR 1000 Series Aggregation Services Routers.

If you deactivate a SPA using the hw-module subslot stop command, and need to reactivate it without performing an OIR, you must use the hw-module subslot start command to reactivate the SPA and its interfaces.

To activate a SPA and its interfaces after the SPA has been deactivated, use the following command in global configuration mode:

Command

Purpose

Router(config)# no hw-module subslot slot-number/subslot-number shutdown

Activates the SPA and its interfaces in the specified slot and subslot of the SIP, where:

  • slot-number — Specifies the chassis slot number where the SIP is installed.
  • subslot-number —Specifies subslot number on a SIP where a SPA is installed.

Verifying the Deactivation and Activation of a SPA

When you deactivate a SPA, the corresponding interfaces are also deactivated. This means that these interfaces will no longer appear in the output of the show interface command.

To verify the deactivation of a SPA, enter the show hw-module subslot all oir command in privileged EXEC mode. Observe the Operational Status field associated with the SPA that you want to verify.

In the following example, the SPA located in subslot 1 of the SIP in slot 2 of the Cisco ASR 1000 Series Aggregation Services Routers is administratively down from the hw-module subslot shutdown command: 


Router# show hw-module subslot all oir
Module         Model              Operational Status
-------------- ------------------ -------------------------
subslot 2/0    SPA-1X10GE-L-V2      ok
subslot 2/1    SPA-8X1GE-V2          admin down

To verify activation and proper operation of a SPA, enter the show hw-module subslot all oir command and observe “ok” in the Operational Status field as shown in the following example: 


Router# show hw-module subslot all oir
Module         Model              Operational Status
-------------- ------------------ -------------------------
subslot 2/0    SPA-1X10GE-L-V2      ok
subslot 2/1    SPA-8X1GE-V2         ok

Deactivation and Activation Configuration Examples

This section provides the following examples of deactivating and activating SIPs and SPAs:

Deactivation of a SIP Configuration

Deactivate a SIP when you want to perform OIR of the SIP. The following example deactivates the SIP that is installed in slot 1 of the Cisco ASR 1000 Series Aggregation Services Routers, its SPAs, and all of the interfaces: 


Router(config)# hw-module slot 1 stop

Note
Deactivation of SIP is not supported on Cisco ASR 1001, Cisco ASR 1002 and Cisco ASR 1002-X routers.

Activation of a SIP Configuration

Activate a SIP if you have previously deactivated it. If you did not deactivate the SPAs, the SPAs automatically reactivate with reactivation of the SIP.

The following example activates the SIP that is installed in slot 1 of the Cisco ASR 1000 Series Aggregation Services Routers, its SPAs, and all of the interfaces (as long as the hw-module subslot shutdown command was not issued to also deactivate the SPA): 


Router(config)# hw-module slot 1 start

There are no corresponding console messages shown with activation using this command. If you re-enter the hw-module slot start command, a message is displayed indicating that the module is already enabled.

Deactivation of a SPA Configuration

Deactivate a SPA when you want to perform OIR of that SPA. The following example deactivates the SPA (and its interfaces) that is installed in subslot 0 of the SIP located in slot 1 of the Cisco ASR 1000 Series Aggregation Services Routers and removes power to the SPA: 


Router(config)# hw-module subslot 1/0 shutdown unpowered

Activation of a SPA Configuration Example

Activate a SPA if you have previously deactivated it. If you have not deactivated a SPA and its interfaces during OIR of a SIP, then the SPA is automatically reactivated upon reactivation of the SIP.

The following example activates the SPA that is installed in subslot 0 of the SIP located in slot 1 of the Cisco ASR 1000 Series Aggregation Services Routers: 


Router(config)# no hw-module subslot 1/0 shutdown