Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Hardware Installation Guide
Troubleshooting the Installation
Downloads: This chapterpdf (PDF - 346.0KB) The complete bookPDF (PDF - 4.25MB) | Feedback

Table of Contents

Troubleshooting the Installation

Troubleshooting the Hardware

SIP is Deactivated

SIP Transitions Repeatedly from On to Off

Troubleshooting the Cisco WebEx Node and SPA-DSP for ASR 1000 Series Hardware

Cisco WebEx Node SPA and SPA-DSP is in Out-of-Service State

Cisco WebEx Node SPA Has Issued Hardware Error Messages

Hard Disk Drive Errors on the Cisco WebEx Node SPA

Collecting the show tech-support Output from the Cisco WebEx Node SPA Console

Reformatting the Hard Disk Drive on the Cisco WebEx Node SPA

Prerequisites

Troubleshooting the 1-Port 10-Gigabit Ethernet LAN/WAN-PHY SPA for ASR 1000 Series Hardware

Feature Change to CPU- Usage show Command

Using debug Commands

Packing a SPA for Shipment

Packing a SIP for Shipment

Troubleshooting the Installation

This chapter describes how to troubleshoot the installation of SIPs and SPAs on the Cisco ASR 1000 Series Aggregation Services Routers. This chapter contains the following sections:

For additional troubleshooting information, also refer to the Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Software Configuration Guide .

Troubleshooting the Hardware

This section describes troubleshooting the installation of the SIPs and SPAs. Possible problems, observations and comments, and solutions are indicated for the following troubleshooting symptoms:

SIP is Deactivated

Table 1-1 describes the possible problems and solutions for a deactivated SIP.

Table 1-1 Possible Problems and Solutions for a Deactivated SIP

 

SIP Is Deactivated
Possible Problem
Observations and Comments
Solutions

SPA is not fully seated in the SIP

Output of the show diag slot command

SPA STATUS LED is off

Follow this procedure:

1. Remove the SPA from the SIP.

2. Inspect the SIP and the SPA. Verify there are no bent pins or parts and that there is nothing lodged in the two devices that could prevent a good connection.

3. Insert the SPA in the SIP by sliding the SPA all the way into the SIP until the SPA is firmly seated in the SPA interface connector. When fully seated in the SIP, the SPA might be slightly behind the SIP faceplate.

SPA DSP is out of service

Output of the show hw-module subslot all oir command indicates ‘out of service’ as Operational state for a SPA-DSP.

SPA STATUS LED is off

Refer Table 1-3 for solutions and details regarding the hardware-based out-of-service states.

SPA is not supported on the SIP

Error message indicating the SPA is not supported

Output of the show diag slot command

SIP STATUS LED is off

Install a SPA supported on the SIP.

SPA is not at the minimum hardware revision level

Error message indicating the SPA is not at the minimum FPGA revision level

Output of the show hw-module subslot fpd command

Output of the show diag slot command

SPA STATUS LED is off

Follow the FPD upgrade process to update the FPGA.

For more information about performing FPD upgrades, refer to the “Upgrading Field-Programmable Devices” chapter in the Cisco 7600 Series Router SIP, SSC, and SPA Software Configuration Guide .

SPA is misconfigured

 

Refer to the configuration section of the SPA installation and configuration guide. Also, refer to the Cisco IOS software configuration documentation listed in the “$paratext>” section.

SIP Transitions Repeatedly from On to Off

Table 1-2 describes the possible problems and solutions when a SIP is transitioning from on to off state.

Table 1-2 Possible Problems and Solutions for Symptom of SIP Transitions

 

Possible Problem
Observations and Comments
Solutions

SIP is booting up; this is normal operation

SIP STATUS LED alternates green, amber, or off

Wait 30 seconds until the boot process completes and the STATUS LED stays on.

SIP does not go beyond the bootup stage

SIP STATUS LED transitions continue and alternates green, amber, or off

Follow the recommended action for the displayed error message.

Troubleshooting the Cisco WebEx Node and SPA-DSP for ASR 1000 Series Hardware

This section describes some of the possible hardware problems that can occur with the Cisco WebEx Node for ASR 1000 Series (also called the Cisco WebEx Node SPA) hardware and DSP SPA for Cisco ASR 1000 Series. It includes the following problems:

Cisco WebEx Node SPA and SPA-DSP is in Out-of-Service State

Both hardware and software problems can cause “out of service” states on the Cisco WebEx Node SPA and SPA-DSP.

To verify the out-of-service reason for the Cisco WebEx Node SPA or a SPA-DSP, use the show hw-module subslot all oir command as shown in the following example:

Router# show hw-module subslot all oir
Module Model Operational Status
------------- -------------------- ------------------------
subslot 0/0 SPA-2X1GE-V2 ok
subslot 0/1 SPA-DSP out of service (failed too many time)
subslot 1/1 SPA-WMA-K9 out of service (failed too many time)

Table 1-3 describes the possible problems and solutions for hardware-based out-of-service states. For more information about other software-based out-of-service states, refer to the troubleshooting chapter for the Cisco WebEx Node SPA or SPA-DSP in the Cisco ASR 1000 Series Aggregation Services Routers SIP and SPA Software Configuration Guide.

Table 1-3 Possible Problems and Solutions for Hardware-Based Out-of-Service States on the Cisco WebEx Node SPA and SPA-DSP

Possible Problem
Observations and Comments
Solutions

SPA is not inserted correctly.

The show hw-module subslot all oir command displays a “failed too many time” reason code for the out of service operational status.

Other than a SPA installation problem, other possible reasons for a HW-INIT-TIMEOUT include corruption of the power controller (ADM1066) on the SPA or corruption of the Complex Programmable Logic Device (CPLD) image during an FPD upgrade.

1. Verify the installation of the SPA and attempt to reseat it in the SIP.

2. If reinstallation of the SPA does not correct the problem, run the show logging command to look for other error messages that might indicate the reason for the failure (a HW_INIT_TIMEOUT is most common).

3. Check for physical connector damage, such as a bent pin on the SPA.

The FPD image was corrupted for some of the following possible reasons:

  • SPA was removed during an FPD upgrade
  • A reload of the router occurred during an FPD upgrade.
  • A power failure occurred on the router during an FPD upgrade.

The show hw-module subslot all oir command displays a “failed too many time” reason code for the out of service operational status and the show logging command shows a HW-INIT-TIMEOUT failure.

The following is an example of a HW-INIT-TIMEOUT message:

*Mar 3 23:27:05.903: %SPA_OIR-6-ONLINECARD: SPA (SPA-WMA-K9) online in subslot 1/1

*Mar 3 23:27:16.488: %SPA_OIR-3-HW_INIT_TIMEOUT: subslot 1/0

*Mar 3 23:27:21.488: %SPA_OIR-3-RECOVERY_RELOAD: subslot 1/0: Attempting recovery by reloading SPA

*Mar 3 23:27:21.489: %SPA_OIR-6-OFFLINECARD: SPA (SPA-WMA-K9) offline in subslot 1/0

1. Enter the upgrade hw-module subslot fpd bundled command to start recovery of the FPD upgrade.

If the problem was due to an FPD image corruption problem, then the SPA should boot normally after the upgrade is complete.

2. If the FPD upgrade completes successfully but you still have an error, then the SPA probably has a hardware problem.

Temperature or voltage sensors have values that are not in the range of safe operation for the Cisco WebEx Node SPA and SPA-DSP.

  • The show hw-module subslot all oir command displays an “Environmental limits exceeded” reason code.
  • The show hw-module subslot sensors command displays the actual values for the temperature and voltage on the Cisco WebEx Node SPA, as shown in the following example:

Router# show hw-module subslot 0/0 sensors

SPA-WMA-K9[0/0] temperature sensor 0, reading: 31C

SPA-WMA-K9[0/0] temperature sensor 1, reading: 36C

SPA-WMA-K9[0/0] temperature sensor 2, reading: 48C

SPA-WMA-K9[0/0] temperature sensor 3, reading: 51C

SPA-WMA-K9[0/0] temperature sensor 4, reading: 57C

SPA-WMA-K9[0/0] temperature sensor 5, reading: 32C

SPA-WMA-K9[0/0] nominal: 3.300V, reading: 3.257V

SPA-WMA-K9[0/0] nominal: 2.500V, reading: 2.454V

SPA-WMA-K9[0/0] nominal: 5.000V, reading: 5.038V

SPA-WMA-K9[0/0] nominal: 1.200V, reading: 1.195V

SPA-WMA-K9[0/0] nominal: 1.100V, reading: 1.097V

SPA-WMA-K9[0/0] nominal: 1.000V, reading: 0.998V

SPA-WMA-K9[0/0] nominal: 0.900V, reading: 0.892V

SPA-WMA-K9[0/0] nominal: 1.800V, reading: 1.785V

 

1. Enter the show logging | include TEMP_SHUTDOWN command and look for a temperature shutdown error message.

The following is an example of an error message due to excessive temperature:

*Jan 27 17:46:22.285: %SPA-0-TEMP_SHUTDOWN: SIP1/1: SPA-WMA-K9[1/1] temperature too high on sensor 2. (90C) SPA has been shut down.

 

Tip If you do not find a temperature shutdown error message, it’s possible that the logging buffer has overflowed from too many warning messages. Enter the show logging | include TEMP_WARNING command to confirm.

2. If you find a temperature error message, then verify that the fan within the power supply is working correctly.

Refer to the “Troubleshooting the Cooling Subsystem” section of the “Troubleshooting Initial Startup Problems” chapter in the Cisco ASR 1000 Series Aggregation Services Router Hardware Installation Guide .

3. If no temperature error message is found, enter the show logging | include VOLT_SHUTDOWN command to find a voltage shutdown error message.

If an error messages indicates a voltage problem, then the SIP could be providing an unstable voltage to the SPA, or there might be a hardware problem on the Cisco WebEx Node SPA itself.

Try installing the SPA into another SIP and see if the problem continues. If the problem still occurs, then there is a voltage problem on the Cisco WebEx Node SPA.

Temperature or voltage sensors have values that are not in the range of safe operation for the Cisco WebEx Node SPA and SPA-DSP.

Execute the show hw-module subslot sensors command for a SPA-DSP to display the actual values for the temperature and voltage on the SPA-DSP, as shown in the following example:

Router# show hw-module subslot 0/2 sensors

SPA-DSP[0/2] temperature sensor 0, reading: 43C SPA-DSP[0/2] temperature sensor 1, reading: 38C SPA-DSP[0/2] temperature sensor 2, reading: 44C SPA-DSP[0/2] nominal: 3.300V, reading: 3.275V SPA-DSP[0/2] nominal: 2.500V, reading: 2.517V SPA-DSP[0/2] nominal: 1.800V, reading: 1.786V SPA-DSP[0/2] nominal: 3.300V, reading: 3.293V SPA-DSP[0/2] nominal: 1.200V, reading: 1.194V SPA-DSP[0/2] nominal: 1.000V, reading: 0.993V

 

Execute the show hw-module subslot sensors limit command for a SPA-DSP to compare the warning, critical, and shutdown temperatures and voltage values with actual values. The show hw-module subslot sensors limit command not only displays the actual voltage and temperature values, but also provides the temperature and voltage limit values for the warning, critical, and shutdown states for a sensor.

Router# show hw-module subslot 0/2 sensors limits

Temperature sensors for SPA-DSP[0/2]:

Sensor Reading Low Warning Critical Shutdown

0 43C -5C 70C 85C 91C

1 38C -5C 62C 77C 83C

2 44C -5C 67C 82C 88C

Voltage sensors for SPA-DSP[0/2]:

Nominal Reading LowShut LowWarn HighWarn HighShut

3.300V 3.278V 2.970V 3.069V 3.531V 3.630V

2.500V 2.517V 2.250V 2.325V 2.675V 2.750V

1.800V 1.785V 1.620V 1.674V 1.926V 1.980V

3.300V 3.293V 2.970V 3.069V 3.531V 3.630V

1.200V 1.194V 1.080V 1.116V 1.284V 1.320V

1.000V 0.993V 0.900V 0.930V 1.070V 1.100V

Tip Use the WebEx Node solutions mentioned above for SPA-DSP.

Cisco WebEx Node SPA Has Issued Hardware Error Messages

There are several different error messages for the Cisco WebEx Node SPA that might appear in the output of the show logging command. Table 1-4 provides an example of error messages of the SPA_CPU_ERR type.

For more information about other system error messages for the Cisco WebEx Node SPA, refer to the System Messages for Cisco IOS XE document.

Table 1-4 SPA_CPU_ERR Messages on the Cisco WebEx Node SPA

 

Possible Problem
Observations and Comments
Solutions

Hardware problem on the Cisco WebEx Node SPA.

The show logging command shows a SPA_CPU_ERR message for the SPA_SRVCS_ENGINE facility, as shown in the following example:

*Mar 24 23:52:06.527: %SPA_SRVCS_ENGINE-3-SPA_CPU_ERR: SIP0/1: SPA-WMA-K9[0/1]: SPA CPU HW errors: Octeon L2D ECC double bit (core 4): fadr: 0x23e86, syn0: 0x0, syn1: 0xe7c00

*Mar 24 23:52:06.527: %SPA_SRVCS_ENGINE-3-HW_ERR: SIP0/1: SPA-WMA-K9[0/1]: A Hardware device error was detected by SPA CPU, trying to recover the SPA by reload.

1. Look up the error message in the Error Message Decoder tool or in the System Messages for Cisco IOS XE document.

2. Record the message exactly as it appears on the console.

3. Collect the output from the show tech-support command on the Cisco WebEx Node SPA console and contact Cisco technical support to create a Return Materials Authorization (RMA) case.

For information about running the show tech-support command on the Cisco WebEx Node SPA console, see the “Collecting the show tech-support Output from the Cisco WebEx Node SPA Console” section.

DSP SPA: A hardware device error was detected

DSP firmware is a software that performs all DSP services. DSP firmware is downloaded from SIP to the SPA-DSP. If DSP firmware is not downloaded to SPA-DSP, the following error message is displayed on console: “DSP SPA: A hardware device error was detected”.

Reload the SPA-DSP using the hw-module subslot slot/subslot reload command from privilege exec command mode.

Hard Disk Drive Errors on the Cisco WebEx Node SPA

The Cisco WebEx Node SPA has a hard disk drive (HDD) used by the Cisco WebEx Node application. It stores logs and provides a cache for meeting data.

Table 1-5 describes the possible problems and solutions associated with the HDD that the Cisco ASR Series 1000 Routers can detect during a boot of the Cisco WebEx Node SPA.

 

Table 1-5 Possible HDD Problems and Solutions on the Cisco WebEx Node SPA

Possible Problem
Observations and Comments
Solutions

File System Check Error

The system routinely checks for file system integrity during a SPA boot, or when the system detects a previous unclean shutdown operation of SPA.

Most of the time errors in the file system can be corrected automatically during SPA bootup, but there are cases that manual intervention is required to fix file system errors.

When this type of error occurs, the following syslog message is logged on IOS:

*Apr 14 22:42:17.415: %SPA_SE1-3-DISK_CHECK_ERROR: SIP0/1: SPA-WMA-K9[0/1]: Disk file system check error detected, manual execution of disk checking operation will be required to correct this problem. The SPA will continue operating with reduced disk space.

1. Access the Cisco WebEx Node SPA system console using the hw-module subslot service-engine session command, as shown in the following example:

Router# hw-module subslot 0/1 service-engine session
MontaVista(R) Linux(R) Carrier Grade Edition 5.0 (custom)
Linux/mips64 2.6.21_mvlcge500-octeon-mips64_octeon_v2_be
Vegas Shell -- CGE 5.0 Version
Copyright (c) 1985-2008 by Cisco Systems, Inc.
All rights reserved.

 

2. From the SPA console, run the fsck disk VEGAS shell command, and respond to the prompts as shown in the following example:

Timesaver The fsck disk command supports an option to respond “yes” to all prompts to allow you to run the fsck disk automatically without any manual response.

service-spa# fsck disk

Running file system check on /dev/sda2 partition ...

e2fsck 1.40 (29-Jun-2007)

APPLICATION contains a file system with errors, check forced.

Pass 1: Checking inodes, blocks, and sizes

Inode 7 has illegal block(s). Clear<y>? yes

 

 

Illegal block #24716 (1515870810) in inode 7. CLEARED.

Illegal block #24717 (1515870810) in inode 7. CLEARED.

Illegal block #24718 (1515870810) in inode 7. CLEARED.

Illegal block #24719 (1515870810) in inode 7. CLEARED.

Illegal block #24720 (1515870810) in inode 7. CLEARED.

Illegal block #24721 (1515870810) in inode 7. CLEARED.

Illegal block #24722 (1515870810) in inode 7. CLEARED.

Illegal block #24723 (1515870810) in inode 7. CLEARED.

Illegal block #24724 (1515870810) in inode 7. CLEARED.

Illegal block #24725 (1515870810) in inode 7. CLEARED.

Illegal block #24726 (1515870810) in inode 7. CLEARED.

Too many illegal blocks in inode 7.

Clear inode<y>? yes

 

Restarting e2fsck from the beginning...

Resize inode not valid. Recreate<y>? yes

 

APPLICATION contains a file system with errors, check forced.

Pass 1: Checking inodes, blocks, and sizes

Pass 2: Checking directory structure

Pass 3: Checking directory connectivity

Pass 4: Checking reference counts

Pass 5: Checking group summary information

Free blocks count wrong for group #0 (28103, counted=28104).

Fix<y>? yes

Free blocks count wrong (242639, counted=242640).

Fix<y>? yes

 

 

APPLICATION: ***** FILE SYSTEM WAS MODIFIED *****

APPLICATION: 12/125696 files (0.0% non-contiguous), 8367/251007 blocks

3. Exit the SPA console using the exit command, as shown in the following example:

service-spa# exit

Router#

 

4. Reload the Cisco WebEx Node SPA using the hw-module subslot reload command.

File System Mounting Error

The following message appears on the console:

*Apr 15 23:14:31.721: %SPA_SE1-3-DISK_MOUNT_ERROR: SIP0/0: SPA-WMA-K9[0/0]: Disk mount error detected, manual execution of disk checking operation might be required to correct this problem. The SPA will continue operating with reduced disk space.

Follow the recommended steps for the File System Check Error .

HDD loosens during handling of the SPA

The system cannot detect the disk and displays the following message on the console:

*Apr 15 23:00:06.664: %SPA_SE1-3-DISK_NOT_PRESENT: SIP0/0: SPA-WMA-K9[0/0]: Cannot detect the presence of disk drive on the SPA. Please check for loose cable connection to the drive. The SPA will continue operating without the disk.

1. Check for a loose cable connection to the HDD on the SPA, and reconnect the cable.

2. Collect the output from the show tech-support command on the Cisco WebEx Node SPA console and contact Cisco technical support.

For information about running the show tech-support command on the Cisco WebEx Node SPA console, see the “Collecting the show tech-support Output from the Cisco WebEx Node SPA Console” section.

Other HDD hardware errors

For other hardware errors on the HDD, you might see the following message on the console:

*Apr 15 23:04:06.678: %SPA_SE1-3-DISK_UNKNOWN_STATE: SIP0/0: SPA-WMA-K9[0/0]: The disk drive on the SPA doesn't seems to be on known operational state. Please check for loose cable connection to the drive. The SPA will continue operating without the disk.

1. Check for a loose cable connection to the HDD on the SPA, and reconnect the cable.

2. Collect the output from the show tech-support command on the Cisco WebEx Node SPA console and contact Cisco technical support.

For information about running the show tech-support command on the Cisco WebEx Node SPA console, see the “Collecting the show tech-support Output from the Cisco WebEx Node SPA Console” section.

Collecting the show tech-support Output from the Cisco WebEx Node SPA Console

When reporting a problem to Cisco technical support, you should first obtain the output of the show tech-support command from the WebEx Node SPA console. The following example show how to run the command and redirect the output to a file on a TFTP server:

service-spa# show tech-support > tftp://tftpboot_server/show-tech-output-2009-06-71.txt
Trying to connect to tftp server......
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!
TFTP put operation was successful
 
153782 bytes copied
service-spa#

Reformatting the Hard Disk Drive on the Cisco WebEx Node SPA


NoteThis procedure is only recommended if a manual file system repair does not fix file system errors. This procedure is only recommended if a manual file system repair does not fix file system errors.


Prerequisites

Before you can format the hard disk on a Cisco WebEx Node SPA, the disk must be unmounted not in use. If you attempt to format the disk while the disk is mounted, you will receive a warning as shown in the following example:

service-spa# format disk
Do you want to format [y/n]: y
%WARNING: cannot perform disk formatting operation as there are partitions mounted and being used by the system.
 

To prepare the disk for formatting, complete the following steps:


Step 1 Access the SIP console using the request platform software console attach command as shown in the following example:

Router# request platform software console attach 0/0
#
# Connecting to the SPA console on 0/0.
# Enter Control-C to exit the console connection.
#
Router>
 

Step 2 Enter the enable command and run the test hw-module subslot bay disk-repair set command as shown in the following example:

Router> enable
Router# test hw-module subslot 0 disk-repair set
WARNING: This command is not intended for production use
and should only be used under the supervision of
Cisco Systems technical support personnel.
 
SPA is set to not mount disk filesystem on next reboot. Please reload to take effect!
 
Router#
 

Step 3 Return to the RP console by entering Control-C and then reload the SPA using the hw-module subslot reload command as shown in the following example:

<Control-C>
Router# hw-module subslot 0/0 reload
 

Step 4 Confirm if the following message appears on the router console:

*Apr 7 08:23:49.974: %SPA_SE1-3-SKIPPED_DISK_MOUNT: SIP0/0: SPA-WMA-K9[0/0]: Disk mounting operation has been skipped for manual disk repair operation. Please follow the procedures on hardware configuration guide on how to enter into service engine console to continue with the disk repair operation.
 

Step 5 Access the Cisco WebEx Node SPA’s VEGAS shell console using the hw-module subslot service-engine session command as shown in the following example:

Router# hw-module subslot 0/0 service-engine session
 
 
MontaVista(R) Linux(R) Carrier Grade Edition 5.0 (custom)
Linux/mips64 2.6.21_mvlcge500-octeon-mips64_octeon_v2_be
 
 
Vegas Shell -- CGE 5.0 Version
Copyright (c) 1985-2008 by Cisco Systems, Inc.
All rights reserved.
 

Step 6 Run the format disk command and confirm the operation as shown in the following example:

service-spa# format disk
Do you want to format [y/n]: y
Drive /dev/sda already has 4 partitions. Need to delete them to re-create new ones!
 
Delete partition 4 ...
 
Delete partition 3 ...
 
Delete partition 2 ...
 
Delete partition 1 ...
 
 
Creating partitions in /dev/sda ....
 
 
Creating APPLICATION partition on /dev/sda1 ...
mke2fs 1.40 (29-Jun-2007)
Filesystem label=APPLICATION
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
125696 inodes, 251007 blocks
12550 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=260046848
8 block groups
32768 blocks per group, 32768 fragments per group
15712 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
 
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
 
Creating CONFIG partition on /dev/sda2 ...
mke2fs 1.40 (29-Jun-2007)
Filesystem label=CONFIG
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
26208 inodes, 104420 blocks
5221 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
13 block groups
8192 blocks per group, 8192 fragments per group
2016 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729
 
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 20 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
 
 
Creating DATA partition on /dev/sda4 ...
mke2fs 1.40 (29-Jun-2007)
Filesystem label=DATA
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
30146560 inodes, 60269855 blocks
3013492 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
1840 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872
 
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Creating SWAP space in /dev/sda3 ...
Setting up swapspace version 1, size = 2056315904 bytes
 
***********************************
Format operation result: SUCCESSFUL
***********************************
service-spa#


 


Tip The SPA-DSP has neither a CPU nor a hard disk as compared to the WebEx Node SPA. Hence, there are no CPU error messages or hard disk error messages for a SPA-DSP.


Troubleshooting the 1-Port 10-Gigabit Ethernet LAN/WAN-PHY SPA for ASR 1000 Series Hardware

This section describes some of the hardware problems that may occur with the 1-Port 10-Gigabit Ethernet LAN/WAN-PHY SPA for also called the SPA-1X-10GE-WL-V2 SPA.

Table 1-6 describes the possible problems and solutions for hardware-based out-of-service states.

Table 1-6 Possible Reasons and Solutions for Hardware-Based Errors on the 1-Port 10-Gigabit Ethernet LAN/WAN-PHY SPA

Possible Problem
Observations and Comments
Solutions

SPA Hardware initialization timeout

The SPA initialization is performed by SIP. If the initialization of hardware has timed out, a HW-INIT-TIMEOUT message is displayed on the console.

The show hw-module subslot all oir command displays a “failed too many times” reason code for the out-of-service operational status and the show logging command shows a HW-INIT-TIMEOUT failure.

The following is an example of a HW-INIT-TIMEOUT message:

*Mar 3 23:27:05.903: %SPA_OIR-6-ONLINECARD: SPA (SPA-1X10GE-WL-V2) online in subslot 1/1

*Mar 3 23:27:16.488: %SPA_OIR-3-HW_INIT_TIMEOUT: subslot 1/0

*Mar 3 23:27:21.488: %SPA_OIR-3-RECOVERY_RELOAD: subslot 1/0: Attempting recovery by reloading SPA

*Mar 3 23:27:21.489: %SPA_OIR-6-OFFLINECARD: SPA (SPA-1X10GE-WL-V2) offline in subslot 1/0

1. Reload or reseat (reinsert) the SPA.

2. Enter the upgrade hw-module subslot fpd bundled command to start the recovery of the FPD upgrade.

If the problem was due to an FPD image corruption, the SPA should boot normally after the upgrade is complete.

3. If the FPD upgrade completes successfully, but you still have an error, the SPA probably has a hardware problem.

Feature Change to CPU-Usage show Command

To display the current CPU-Usage for a FRU, the show platform software process slot slot monitor cycles cycles interval interval lines lines command is used from the console. The way this command is run on a FRU, is changed from Cisco IOS XE Release 3.4.0s.

Old Behavior

If lines is specified in the show platform software process slot slot monitor cycles cycles interval interval lines lines command, the results generated and displayed did not reflect accurate CPU utilization at the time. This was because the command was executed as many cycles as specified one cycle at a time.

The following example shows the output of the show platform software process slot slot monitor cycles cycles interval interval lines lines command.

This command produced inaccurate CPU usage details as illustrated in the following example:

Router# show platform software process slot r0 monitor cycles 4 interval 3 lines 5
top - 18:26:00 up 4 min, 0 users, load average: 1.89, 1.97, 0.86
Tasks: 133 total, 2 running, 131 sleeping, 0 stopped, 0 zombie
Cpu(s): 34.4%us, 32.3%sy, 0.0%ni, 27.4%id, 3.2%wa, 0.5%hi, 2.2%si, 0.0%st
Mem: 3874984k total, 1553444k used, 2321540k free, 102904k buffers
Swap: 0k total, 0k used, 0k free, 963120k cached
 
top - 18:26:03 up 4 min, 0 users, load average: 1.89, 1.97, 0.86
Tasks: 133 total, 3 running, 130 sleeping, 0 stopped, 0 zombie
Cpu(s): 34.0%us, 31.9%sy, 0.0%ni, 28.3%id, 3.2%wa, 0.5%hi, 2.2%si, 0.0%st
Mem: 3874984k total, 1553444k used, 2321540k free, 102904k buffers
Swap: 0k total, 0k used, 0k free, 963120k cached
 
top - 18:26:07 up 4 min, 0 users, load average: 1.74, 1.94, 0.86
Tasks: 133 total, 3 running, 130 sleeping, 0 stopped, 0 zombie
Cpu(s): 33.6%us, 31.5%sy, 0.0%ni, 29.2%id, 3.1%wa, 0.5%hi, 2.2%si, 0.0%st
Mem: 3874984k total, 1553452k used, 2321532k free, 102904k buffers
Swap: 0k total, 0k used, 0k free, 963120k cached
 
top - 18:26:10 up 4 min, 0 users, load average: 1.60, 1.91, 0.85
Tasks: 133 total, 2 running, 131 sleeping, 0 stopped, 0 zombie
Cpu(s): 33.2%us, 31.1%sy, 0.0%ni, 30.0%id, 3.1%wa, 0.5%hi, 2.2%si, 0.0%st
Mem: 3874984k total, 1553452k used, 2321532k free, 102904k buffers
Swap: 0k total, 0k used, 0k free, 963124k cached

 

New Behavior

The show platform software process slot slot monitor cycles cycles interval interval command produced accurate CPU usage details, if executed without the lines number-of-lines parameters. The show command, if executed without the lines number-of-lines keywords is executed in batch-mode.


NoteTo display accurate CPU usage details, execute the To display accurate CPU usage details, execute the show platform software process slot slot monitor cycles cycles interval interval command without the lines number-of-lines keyword or use the top command top -s -b -n iterations - d interval from Linux shell.


The following example provides accurate CPU usage details (note the value fluctuation in the first column) when ‘lines’ is not specified. When the command is executed with the lines specified, it still has the old behavior where the first column will not be consistent with the CPU usage on the FRU.

The following example provides accurate CPU usage details with no line count specified:

Router# show platform software process slot r0 monitor cycles 4 interval 3
top - 18:26:20 up 4 min, 0 users, load average: 1.35, 1.84, 0.84
Tasks: 132 total, 3 running, 129 sleeping, 0 stopped, 0 zombie
Cpu(s): 32.2%us, 30.1%sy, 0.0%ni, 32.2%id, 3.0%wa, 0.4%hi, 2.1%si, 0.0%st
Mem: 3874984k total, 1553072k used, 2321912k free, 102904k buffers
Swap: 0k total, 0k used, 0k free, 963140k cached
 
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2776 root 20 0 2648 1152 884 R 2.0 0.0 0:00.01 top
14538 root 20 0 4468 2828 1308 S 2.0 0.1 0:00.79 droputil.sh
1 root 20 0 2156 644 556 S 0.0 0.0 0:01.34 init
2 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 15 -5 0 0 0 S 0.0 0.0 0:00.08 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root 15 -5 0 0 0 S 0.0 0.0 0:00.03 events/0
6 root 15 -5 0 0 0 S 0.0 0.0 0:00.02 khelper
9 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 netns
62 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/0
70 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 ata/0
71 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 ata_aux
77 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 khubd
80 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod
125 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pdflush


The following example provides CPU usage details with line count specified:

Router# show platform software process slot r0 monitor cycles 10 interval 5 lines 10 | inc %id
Cpu(s): 20.6%us, 21.5%sy, 0.0%ni, 53.3%id, 1.6%wa, 0.5%hi, 2.4%si, 0.0%st
Cpu(s): 20.4%us, 21.3%sy, 0.0%ni, 53.7%id, 1.6%wa, 0.5%hi, 2.4%si, 0.0%st
Cpu(s): 20.2%us, 21.1%sy, 0.0%ni, 54.2%id, 1.6%wa, 0.5%hi, 2.4%si, 0.0%st
Cpu(s): 20.0%us, 20.9%sy, 0.0%ni, 54.7%id, 1.6%wa, 0.5%hi, 2.3%si, 0.0%st
Cpu(s): 19.8%us, 20.7%sy, 0.0%ni, 55.1%id, 1.6%wa, 0.5%hi, 2.3%si, 0.0%st
Cpu(s): 19.8%us, 20.9%sy, 0.0%ni, 54.8%id, 1.6%wa, 0.5%hi, 2.4%si, 0.0%st
Cpu(s): 19.6%us, 21.0%sy, 0.0%ni, 54.7%id, 1.5%wa, 0.5%hi, 2.6%si, 0.0%st
Cpu(s): 19.5%us, 20.8%sy, 0.0%ni, 55.2%id, 1.5%wa, 0.5%hi, 2.5%si, 0.0%st
Cpu(s): 19.3%us, 20.6%sy, 0.0%ni, 55.6%id, 1.5%wa, 0.5%hi, 2.5%si, 0.0%st
Cpu(s): 19.1%us, 20.4%sy, 0.0%ni, 56.0%id, 1.5%wa, 0.5%hi, 2.5%si, 0.0%st
 
 

Using debug Commands

The debug hw-module subslot command is intended for use by Cisco technical support personnel. For more information about the debug hw-module subslot command, see the Cisco 7600 Series Router SIP, SSC, and SPA Software Configuration Guide .


Caution Because debugging output is assigned high priority in the CPU process, it can render the system unusable. For this reason, use debug commands only to troubleshoot specific problems or during troubleshooting sessions with Cisco technical support staff. Moreover, it is best to use debug commands during periods of lower network traffic and fewer users. Debugging during these periods decreases the likelihood that increased debug command processing overhead will affect system use.

Packing a SPA for Shipment

This section provides step-by-step instructions for packing a SPA for shipment. Before beginning this procedure, you should have the following original Cisco packaging materials:

  • Clamshell inner container
  • Foam cushion (2)
  • Exterior carton

NoteFor information on removing a SPA, see the For information on removing a SPA, see the “SPA Installation and Removal” section.



Caution The Cisco original packaging is to be used for the shipment of all SPAs and cable management brackets. Failure to properly use Cisco packaging can result in damage or loss of a product.


Caution During this procedure, wear grounding wrist straps to avoid ESD damage to the card.

See Preventing Electrostatic Discharge Damage for information about preventing ESD.


NoteThese instructions assume that the SPA and cable management brackets have been removed from the router according to the recommended procedures specified in this guide. These instructions assume that the SPA and cable management brackets have been removed from the router according to the recommended procedures specified in this guide.


To pack a SPA for shipment, see Figure 1-1 and perform the following steps:


Step 1 Open the clamshell container and place the SPA into the appropriate cavity.


Caution Always handle the SPA by the carrier edges and handle; never touch the SPA components or connector pins.

Step 2 Close the clamshell container.

Step 3 Check that the clamshell container is fully closed. Apply tape or a label closure over the opening to ensure that the container stays closed during shipping.

Step 4 Place one foam cushion in the bottom of the exterior carton.

Step 5 Place the clamshell container into the carton on top of the foam.

Step 6 Place the second foam cushion on top of the clamshell.

Step 7 Close the exterior carton.

Step 8 Apply tape over the exterior carton flap to ensure that the carton stays closed during shipping.


 

Figure 1-1 Packing a SPA for Shipment

 

1

Clamshell

2

Cushion foam

3

Exterior carton

 

 

Packing a SIP for Shipment

This section provides step-by-step instructions for packing a SIP for shipment. Before beginning this procedure, you should have the following original Cisco packaging materials:

  • Static shielding bag
  • Corrugated spacer
  • Bottom support
  • Smaller inner carton
  • Two foam packing cushions
  • Larger exterior carton

NoteFor information on removing a SIP, see the For information on removing a SIP, see the “SIP Installation and Removal” section.



Caution The Cisco original packaging is to be used for the shipment of all SIPs. Failure to properly use Cisco packaging can result in damage or loss of a product.


Caution During this procedure, wear grounding wrist straps to avoid ESD damage to the card.

See Preventing Electrostatic Discharge Damage for information about preventing ESD.


NoteThese instructions assume that the SIP has been removed from the router according to the recommended procedures specified in this guide. These instructions assume that the SIP has been removed from the router according to the recommended procedures specified in this guide.


To pack a SIP for shipment, see Figure 1-2, and perform the following steps:


Step 1 Insert the SIP into the static shielding bag. (Not shown in diagram.)

Step 2 Place the bottom support into the smaller inner carton.

Step 3 Place the bagged SIP onto the bottom support.

Step 4 Place the corrugated spacer on top of the bagged SIP.

Step 5 Close the smaller inner carton and tape the sides closed.

Step 6 Place the sealed smaller inner carton containing the SIP into the two foam packing cushions (the carton only fits one way).

Step 7 Place the sealed smaller inner carton and packing cushions into the larger exterior carton, and seal the larger exterior carton with tape for shipment.


 

Figure 1-2 Packing a SIP for Shipment

 

1

Corrugated spacer

2

Bottom support for inner carton

3

Inner carton

4

Foam packing cushions

5

Exterior carton