Table Of Contents
Troubleshooting the Installation
Troubleshooting Overview
Troubleshooting Using a Subsystem Approach
Normal Router Startup Sequence
Identifying Startup Issues
Troubleshooting the Power Subsystem
Troubleshooting an AC Power Supply
Troubleshooting an AC Power Shelf Installation
Troubleshooting a DC Power Supply
Troubleshooting a DC Power Shelf Installation
Additional Power Subsystem Troubleshooting Information
Troubleshooting the Power Distribution System
Troubleshooting the Processor Subsystem
Troubleshooting the RP
Troubleshooting Using the RP Alphanumeric Display
Troubleshooting the Line Cards
Troubleshooting Using the Alarm Cards
Monitoring Critical, Major, and Minor Alarm Status
Troubleshooting the Switch Fabric
Analyzing the Data
crc16 Output
Grant Parity and Request Errors
Properly Seating Switch Fabric Cards
Troubleshooting the Cooling Subsystem
Blower Module Operation
Power Supply Operation
Overtemperature Conditions
Isolating Cooling Subsystem Problems
Troubleshooting the Installation
This chapter contains general troubleshooting information to help isolate the cause of difficulties you might encounter during the installation and initial startup of the system.
The procedures in this chapter assume that the system is running the original configuration. If you altered the original hardware configuration or changed any default configuration settings, the recommendations in this chapter may not apply.
Although an overtemperature condition is unlikely at initial startup, environmental monitoring functions are included in this chapter because they also monitor internal voltages.
Troubleshooting the installation is presented in the following sections:
•
Troubleshooting Overview
•Troubleshooting the Power Subsystem
•Troubleshooting the Processor Subsystem
•Troubleshooting the Switch Fabric
•Troubleshooting the Cooling Subsystem
Troubleshooting Overview
This section describes the methods used in troubleshooting the router. The troubleshooting methods are organized according to the major subsystems in the router.
If you are unable to solve a problem on your own, you can contact a Cisco customer service representative for assistance. When you call, have the following information ready:
•Date you received the router and the chassis serial number (located on a label on the back of the chassis).
•Installed line cards.
–Use the show hardware command to determine which line cards are installed if possible.
•Cisco IOS software release number.
–Use the show version command to determine this information if possible.
•Brief description of the symptoms and steps you have taken to isolate and solve the issue.
•Maintenance agreement or warranty information.
Troubleshooting Using a Subsystem Approach
To solve a system problem, try to isolate the problem to a specific subsystem. Compare current router behavior with expected router behavior. Because a startup issue is usually attributable to one component, it is most efficient to examine each subsystem, rather than trying to troubleshoot each router component.
For troubleshooting purposes in this chapter, the router consists of the following subsystems:
•Power subsystem—Includes the following components:
–AC-input or DC-input power supplies, also called power entry modules (PEMs). The router chassis is shipped with fully-redundant PEMs installed in the chassis.
•Chassis backplane power distribution. -48 VDC power from the power supplies is transferred to the chassis backplane. The -48 VDC is distributed to all of the cards through the backplane connectors. The blower module receives power from the chassis backplane through a wiring harness and passes MBus data back to the chassis backplane.
•Processor subsystem—Includes redundant RPs, line cards, switch fabrics, and two alarm cards. The RP and line cards are equipped with onboard processors. The RP downloads a copy of the Cisco XR IOS image to each line card processor. The system uses an alphanumeric display (on each line card and RP) to display status and error messages, which can help in troubleshooting.
•Cooling subsystem—Consists of 2 blower modules, which circulate air through the card cages to cool the cards, and fan in each of the power modules, which circulates cooling air through the power module.
Normal Router Startup Sequence
You can generally determine when and where the router failed during the startup sequence by checking the status LEDs on the power modules, and the alphanumeric displays on the RP and line cards.
In a normal router startup sequence, the following sequence of events and conditions occur:
1. The fans in the blower module receive power and begin drawing air through the chassis.
The blower module OK indicator is on.
2. The fan in each PEM receives power and begins drawing air through the power supply.
The power supply Pwr OK indicator is on.
3. As the power on and boot process progresses for the RP and each installed line card, the status of each card appears on the alphanumeric display on the front panel of the card:
–The upper row of the display is powered by the DC-to-DC converter on the card.
–The lower row of the display is powered by the +5 VDC provided through the backplane.
Identifying Startup Issues
Table 4-1 shows the contents of the alphanumeric displays on the RP and the line cards, as well as the normal LED states on the alarm card, the power modules (AC or DC), and the blower modules after a successful system startup.
Table 4-1 Alphanumeric Displays and LEDs at System Startup
Component
|
Type of Indicator
|
Display Contents/LED Status and Meaning
|
RP
|
Alphanumeric display
|
Upper row: MSTR
Lower row: PRP
The RP is enabled and recognized by the system; a valid Cisco IOS software image is running.
|
Line Cards
|
Alphanumeric display
|
Upper row: IOS
Lower row: RUN
The line card is enabled and ready for use.
|
Alarm Cards
|
Detected alarm severity
Alarm card
CSC 0 and 1
SFC 0, 1, 2, and 3,
|
Critical: Off
Major: Off
Minor: Off
Enabled: On
Fail: Off
Enabled: On
Fail: Off
Enabled: On
Fail: Off
|
AC Power Supplies
|
Power status
|
PWR OK: On
FAULT: Off
TEMP: Off
OC: Off
The correct power module voltages are present and no faults have been detected.
|
DC Power Supplies
|
Power status
|
PWR OK: On
FAULT: Off
TEMP: Off
OC: Off
The correct power module voltages are present and no faults have been detected.
|
Blower Modules
|
Blower status
|
OK: On
FAIL: Off
|
Troubleshooting the Power Subsystem
This section contains information to troubleshoot the power subsystems:
•Troubleshooting an AC Power Supply
•Troubleshooting an AC Power Shelf Installation
•Troubleshooting a DC Power Supply
•Troubleshooting the Power Distribution System
Troubleshooting an AC Power Supply
AC-input power supplies are monitored for internal temperature, voltage, and current load by the MBus module on the alarm cards, and by the master MBus module on the RP. If the router detects an extreme condition, it generates an alarm on the alarm card and logs the appropriate warning messages on the console.
Figure 4-1 identifies the components of the AC power supply.
Figure 4-1 AC Power Supply Components
1
|
Ejector handle
|
2
|
Captive screw
|
Use the following procedure to troubleshoot the AC power supply if it is not operating properly after installation.
Step 1 Make sure the power supply is seated properly:
•Eject and reseat the PEM. Check that:
–The ejector lever is locked into place by its captive screw on the ejector lever is tightened securely.
Step 2 Make sure the router is powered on and that all power cords are connected properly:
•Power cords on the back panel of the power shelf are secured in place with their retention clips.
•Power cords at the power source end are securely plugged into their own AC power outlet.
•The source AC circuit breaker is switched on.
Step 3 Check the power supply status LED indicators:
•PWR OK (green)—Indicates the power supply is operating normally, and the source AC voltage is within the nominal operating range of 200 VAC to 240 VAC. This indicator lights when the power supply is properly seated in position.
•FAULT (yellow)—Indicates the system detected a fault within the power supply or the incoming voltage is too low. This indicator remains off during normal operation.
If the indicator is on:
–Check that the source voltage is within the correct range: 170 to 262 VAC
–Remove and then apply power to the power supply by disconnecting its power cord. If the indicator remains on, replace the existing power supply with a spare.
–If the spare power supply also fails, the problem could be a faulty power shelf backplane connector. Power off the router and contact a Cisco service representative for assistance.
•TEMP (yellow)—Indicates that the power supply is in an overtemperature condition, causing a shut-down to occur.
Note If the temp indicator is on, the fault indicator is also on.
–Verify that the power supply fan is operating properly.
–Verify that the blower modules are operating properly.
–If the power supply fan and blower modules are operating properly, replace the existing power supply with a spare.
•TEMP (flashing yellow)—Indicates that a power supply fan is locked or malfunctioning.
Note If the temp indicator is flashing, the fault indicator also goes on.
–Check to see if the fan is operating. Remove any obstructions to the fan.
–If the fan is not operating, replace the power supply.
•OC (steady, or flashing yellow after 10 seconds)—Indicates the output current of the power supply has exceeded its limit and that an overload or short has occurred.
Note If the OC indicator is on or flashing, the fault indicator also goes on.
–Remove and then apply power to the power supply by disconnecting its power cord.
–If the indicator remains on, try reseating the power supply.
–If the indicator remains on, replace the power supply.
Because both the standard and optional AC-input power subsystems use redundant power supplies, a problem with the DC output voltage to the backplane from only one power supply should not affect router operation. Because the router is equipped with multiple AC power supplies, it powers on and operates even if one power supply fails.
Troubleshooting an AC Power Shelf Installation
Use the following procedure to troubleshoot an AC power shelf if it does not operate properly after installation.
Step 1 Make sure that the power shelf is seated properly:
•The jackscrew is tightened securely.
•The captive screws on the flanges are tightened securely.
Step 2 Make sure each power supply is seated properly:
•The ejector lever is locked into place by its spring clip.
If the power shelf is still not operating properly, go to "Troubleshooting an AC Power Supply" section.
Troubleshooting a DC Power Supply
DC-input power supplies are monitored for internal temperature, voltage, and current load by the MBus module on the alarm cards, and by the master MBus module on the RP. If the router detects an extreme condition, it generates an alarm on the alarm card and logs the appropriate warning messages on the console.
Figure 4-2 identifies the components of a DC power supply.
Figure 4-2 DC Power Supply Components
1
|
Handle
|
3
|
Ejector lever
|
2
|
Fan
|
4
|
Power switch
|
Use the following procedure to troubleshoot the DC PEM if it is not operating properly after installation.
Step 1 Make sure the PEM is seated properly:
•Eject and reseat the PEM. Make sure:
–The captive screw on the ejector lever are tightened securely.
–The power switch is in the on (1) position.
Step 2 Make sure the router is powered on and that all power cords are connected properly. Check that:
•Power cables are securely connected to their terminal studs on the back panel.
•Power cables are connected to a dedicated 60 A DC service.
•The source DC circuit breaker is switched on.
Step 3 Check the PEM status indicators:
•PWR OK (green) — Indicates that the PEM is operating normally, and the source DC voltage is within the nominal operating range of -48 to -60 VDC. This indicator should light when the PEM circuit breaker is switched on.
•FAULT (yellow) — Indicates that the system has detected a fault within the PEM or the incoming voltage is too low. This indicator remains off during normal operation.
–Check that the source voltage is within the correct range: -40.5 to -75 VDC.
–Toggle the PEM circuit breaker off and then on. If the indicator remains on after several attempts to power it on, replace the existing PEM with a spare.
–If the spare PEM also fails, the problem could be a faulty power shelf backplane connector. Power off the router and contact a Cisco service representative for assistance.
•TEMP (yellow)—Indicates that the PEM is in an overtemperature condition causing a shut-down to occur.
Note If the temp indicator is on, the fault indicator also goes on.
–Verify that the power supply fan is operating properly.
–Verify that the blower modules are operating properly.
–If the power supply fan and the blower modules are operating properly, replace the existing PEM with a spare.
•TEMP (flashing yellow)—Indicates that a power supply fan is locked or malfunctioning.
Note If the temp indicator is flashing, the fault indicator is also on.
–Check to see if the fan is operating. Remove any obstructions to the fan.
–If the fan is not operational, replace the power supply.
•OC (steady, or flashing yellow after 10 seconds)—Indicates the output current of the power supply has exceeded its limit and that an overload or short circuit has occurred.
Note If the OC indicator is on or flashing, the fault indicator also goes on.
–Remove and then apply power to the power supply by disconnecting its power cord.
–If the indicator remains on, try reseating the power supply.
–If the indicator remains on, replace the power supply.
Because there are redundant power supplies, a problem with the DC output voltage to the backplane from only one PEM should not affect router operation. Because the router is equipped with multiple DC power supplies, it powers on and operates even if one power supply fails.
Troubleshooting a DC Power Shelf Installation
Use the following procedure to troubleshoot the DC power shelf if it is not operating properly after installation.
Step 1 Make sure that the power shelf is seated properly:
•The jackscrew is tightened securely.
•The captive screws on the flanges are tightened securely.
Step 2 Make sure each power supply is seated properly:
•Eject and reseat the PEM.
•The captive screw on the ejector lever is tightened securely.
•The power switch is in the on (1) position.
•If the power shelf is still not operating properly, go to "Troubleshooting a DC Power Supply" section
Additional Power Subsystem Troubleshooting Information
This section contains additional troubleshooting information to help you isolate the cause of a power problem.
The MBus modules powering the alphanumeric displays on the RP and line cards are powered by +5 VDC from the backplane. The blower modules use -48 VDC from the backplane. If both the RP and the blower modules are operating, all internal correct DC voltages are present.
Enter the show environment command at the user EXEC mode prompt to display temperature and voltage information for each installed card, blower module, and PEM as shown in this example:
Slot # Hot Sensor Inlet Sensor
Slot # PEM Over Temperature Sensors
Slot # Hot Sensor Inlet Sensor
Troubleshooting the Power Distribution System
The power distribution system consists of:
•AC or DC PEMs which supply -48 VDC to the backplane,
•The chassis backplane which carries voltage to chassis components.
•DC-to-DC converters which convert -48 VDC from the backplane to the correct voltages required by the line cards.
Use the following procedure to troubleshoot the power distribution system.
Step 1 Check each power supply to make sure that:
•The ejector lever is fully closed and properly secured by the its captive screw.
•The PWR OK indicator is on.
•The FAULT and TEMP indicators are both off.
•The OC indicator is off.
If the power supplies meet the above criteria, then the correct source power is present and within tolerance. The power supplies are functioning properly.
Step 2 Make sure the blower modules are operating.
•If the blower modules are functioning, then the -48 VDC from the chassis backplane and the cables from the backplane to the blower modules are functioning properly.
•If a blower module is not functioning, there may be a problem with either the blower module itself, or the -48 VDC power supplied to the blower module. Eject and reseat the blower module.
•If a blower module is still not operating there could be a problem with the blower module controller card or cable. Replace the blower module.
Contact your Cisco representative if replacing the blower module does not fix the problem.
Troubleshooting the Processor Subsystem
The router processor subsystem consists of the RPs, line cards, and alarm cards. The RPs and line cards have two onboard processors; one serves as the main (or master processor), and the other serves as the MBus module processor. The MBus module processor monitors the environment and controls the onboard DC-to-DC converters.
Note A minimally configured router must have an RP installed in slot 7 of the upper card cage. If the router is equipped with an optional, redundant RP, that RP must be installed in the far left slot in the lower card cage (slot 8).
This section contains information to troubleshooting the processor subsystem, including:
•Troubleshooting the RP
•Troubleshooting the Line Cards
•Troubleshooting Using the Alarm Cards
Troubleshooting the RP
When the router is powered on, the alphanumeric display on the RP indicate the following (Figure 4-3):
•Upper row—Indicates which RP software component is running. At the end of a successful boot process, this display reads MSTR.
•Lower row—Indicates the current phase of the boot process. At the end of a successful boot process, this display reads PRP.
Figure 4-3 RP Alphanumeric Display
Troubleshooting Using the RP Alphanumeric Display
You can use the alphanumeric display to isolate a problem with the RP. The two rows on the alphanumeric display are powered separately:
•The upper row receives power from the DC-to-DC converters on the RP.
•The lower row is powered directly from the MBus on the RP through the chassis backplane.
–If the lower row is not operating, the MBus module may be malfunctioning.
–If the MBus module is operating, the lower row could be on even if the RP failed to powered on.
•If neither the upper nor the lower row is on, but the power modules and the blower modules are operational, the RP may not be installed properly, or the +5 VDC output from the chassis backplane is faulty.
–Make sure that the system is powered on.
–Initialize the RP by ejecting it from the chassis backplane and then reseating it.
Caution The soft reset (NMI) switch is
not a mechanism for resetting the RP and reloading the Cisco IOS image. It is intended for software development use. To prevent system problems or loss of data, use the soft reset switch
only when instructed by a Cisco certified service representative.
•If both the upper and the lower displays are operating, check the meaning of the messages (see Table 4-2).
When the DC-to-DC converters are powered-on by the MBus module, the RP processor begins the boot process and displays various status messages. Some messages appear briefly; while others appear for several seconds. If the messages appear to stop at a particular point, the boot process may be halted.
–Make a note of the message.
–Turn off power to the router, then turn on the power again to reset the router and start the boot process. If the router halts again, replace the RP (see "Removing and Replacing RP and Line Cards from the Upper and Lower Card Cages" section on page 5-46).
Table 4-2 Troubleshooting Using the RP Alphanumeric Display Messages
Message
|
Description
|
LMEM
TEST
|
Running low memory test
|
LCAH
TEST
|
Initializing lower 15K cache
|
BSS
INIT
|
Initializing main memory for ROM
|
NVRAM
INIT
|
Initializing NVRAM
|
EXPT
INIT
|
Initializing interrupt handlers
|
TLB
INIT
|
Initializing TLB
|
CACH
INIT
|
Initializing CPU data and instruction cache
|
CACH
PARY
|
Enabling CPU cache parity
|
MEM
INIT
|
Initializing main memory
|
NVRAM
SIZE
|
Detecting the NVRAM size
|
PCMC
INIT
|
Initializing the PCMCIA
|
EXIT
INIT
|
Exiting the initialization sequence
|
IOS
UP
|
Running Cisco XR IOS software
|
The RP has 8 device or port LED activity indicators that show the status of the Ethernet connections (Figure 4-4).
Figure 4-4 RP LEDs
LINK
|
Indicates link activity
|
EN
|
Indicates the port is enabled
|
TX
|
Indicates data transmission
|
RX
|
Indicates data reception
|
The RJ-45 port LEDs on the RP indicate:
•Which Flash memory card slot is active.
–Each LED lights when its corresponding PCMCIA slot is accessed (SLOT 0 and SLOT 1).
•Which Ethernet connection is in use.
–4 RJ-45 Ethernet port activity LEDs indicate link activity (LINK), port enabled (EN), data transmission (TX), and data reception (RX).
•What is occurring on the Ethernet interface.
–2 Ethernet port-selection LEDs (labeled PRIMARY) identify which of the Ethernet connections are selected. Because both ports are supported on the PRP, the LED on port ETH0 is always on. The LED on port ETH1 lights when it is selected.
PRP-3 Alphanumeric LEDs
The following section discusses the alphanumeric LED messages and the console output displayed in sequence for a single PRP-3 and for dual PRP-3. The alphanumeric LED messages help in identifying the state of the route processor and accordingly troubleshooting the problems faced.
Single PRP-3 Scenario
Table 4-3 displays the alphanumeric LED messages and the console output when the chassis is powered on or when the PRP-3 board is inserted into the slot.
Table 4-3 Single
LED Display
|
Description or Console Message
|
02A8/HW
OK/RIO
|
Immediately when the board is powered on.
|
OK/CPU
SENT/RPT
|
Displays "1.330GHz dual-core MPC8641D Rev 2.1, 532MHz MPXclk".
|
INIT/MEM
|
Displays "Discovering memory in slot DIMM1 ......................... Found 2GB DIMM
Discovering memory in slot DIMM2 ......................... Found 2GB DIMM"
|
TEST/MEM
|
Displays "Testing low memory ....................................... OK
Loading main ROMMON image ................................ OK
Verifying loaded image ................................... OK
Load succeeded; launching target ......................... OK"
|
LNCH/RMON
|
Displays "Cisco ROMMON System Bootstrap, Version 0.16.0 (bld1) DEVELOPMENT SOFTWARE
Compiled on 08/27/08 at 15:04:49 PDT [BLD-rommon]
Copyright (c) 1994-2008 by Cisco Systems, Inc.
MPC8641D platform with 4 GB of main memory"
|
RDY/RP
|
DISPLAYS "Loading disk0:c12k-os-mbi-3.8.0.15I/mbiprp-rp.vm (14809672 bytes)... !!!"
|
RUN/IOX
|
Displays "RP/0/2/CPU0:Sep 10 15:56:29.018: syslogd_helper: [84]: dsc_event_handler: Got SysMgr dSC
event : 1
RP/0/2/CPU0:Jan 1 00:00:04.809 : mbus-prp3[58]: mbus-prp3: mbus_platform_init() failed (0x6).
RP/0/2/CPU0:Sep 10 15:56:07.015 : dumper[53]: No HDD Controller found by process dumper
RP/0/2/CPU0:Sep 10 15:56:21.538 : sysmgr[85]: %OS-SYSMGR-5-NOTICE : Card is COLD started
RP/0/2/CPU0:Sep 10 15:56:22.622 : dsc[169]: Memory Sanity Check Enabled
RP/0/2/CPU0:Sep 10 15:56:29.007 : dsc[169]: %PLATFORM-DSC-3-ERR_I_AM_DSC : Setting myself as DSC
RP/0/2/CPU0:Sep 10 15:57:20.071 : sysldr[370]: %PLATFORM-POWERMGR-3-ROM_ERROR_STATUS :
Unable to get Mbus ROM status from SYSDB. Error ='sysdb' detected the 'warning' condition 'A SysDB client tried to access a nonexistent item or list an empty directory'
RP/0/2/CPU0:Sep 10 15:57:25.078 : sysldr[370]: % PLATFORM-SYSLDR-6-INFO : Waiting for startup config to be applied before booting LCs
Primary Clock is CSC_1 Fabric Clock is Redundant
Bandwidth Mode : Full Bandwidth"
|
RP/ACTV
|
When RP is up and running Cisco IOS XR software.
|
PRP-3 Alphanumeric Display LED—Chassis Is Powered ON/Inserting PRP-3
Table 4-4 displays the alphanumeric LED messages and the console output when the chassis is loaded from ROMMON.
Table 4-4 Single PRP-3 Scenario — Chassis loaded from ROMMON
LED Display
|
Description or Console Message
|
INIT/NV
|
Displays the LED message during the initialization of the NVRAM infra code.
|
OK/RIO
|
Displays the LED message immediately when the board is powered ON.
|
OK/CPU
|
Displays "1.330GHz dual-core MPC8641D Rev 2.1, 532MHz MPXclk".
|
CONT...
|
|
Dual PRP-3 Scenario
If dual PRP-3 is installed and the chassis is powered on or a board is inserted, all the LED messages are the same as for a single PRP-3, except for an LED message `1404/MRAM' that is displayed on the front panel when the console display is as follows:
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
San Jose, California 95134-1706
Cisco IOS XR Software for the Cisco XR PRP, Version 3.8.0.15I
Copyright (c) 2008 by Cisco Systems, Inc.
RP/0/2/CPU0:Sep 10 16:34:19.351: syslogd_helper: [84]:
dsc_event_handler: Got SysMgr dSC event : 1
Tip The LED message "I404/MRAM" is displayed between RDY/RP and RUN/IOX alphanumeric messages.
Table 4-5 displays the LED alphanumeric messages on a standby PRP-3, if the current active PRP-3 is reloaded.
Table 4-5 Dual PRP-3 Scenario
LED Display
|
Description or Console Message
|
INIT/NV
|
Displays the LED message during the initialization of the NVRAM infra code.
|
02A8/HW
OK/RIO
|
LED message displayed immediately after the board is powered on.
|
OK/CPU
SENT/RPT
|
Displays "1.330GHz dual-core MPC8641D Rev 2.1, 532MHz MPXclk".
|
INIT/MEM
|
Displays "Discovering memory in slot DIMM1 ......................... Found 2GB DIMM
Discovering memory in slot DIMM2 ......................... Found 2GB DIMM"
|
TEST/MEM
|
Displays "Testing low memory ....................................... OK
Loading main ROMMON image ................................ OK
Verifying loaded image ................................... OK
Load succeeded; launching target ......................... OK"
|
LNCH/RMON
|
Displays "Cisco ROMMON System Bootstrap, Version 0.16.0 (bld1) DEVELOPMENT SOFTWARE
Compiled on 08/27/08 at 15:04:49 PDT [BLD-rommon]
Copyright (c) 1994-2008 by Cisco Systems, Inc.
MPC8641D platform with 4 GB of main memory"
|
RDY/RP
or
PWRD
|
DISPLAYS "Loading disk0:c12k-os-mbi-3.8.0.15I/mbiprp-rp.vm (14809672 bytes)... !!!"
Tip Instead of RDY/RP, sometimes PWRD is displayed in the normal booting process.
|
RUN/MBI
|
Displays "Copyright (c) 2008 by Cisco Systems, Inc.
Install (Node Preparation): Install device root is /disk0/
Install (Node Preparation): Using boot device sequence compactflash: from rommon
Install (Node Preparation): Trying device disk0:
Install (Node Preparation): Checking size of device disk0:
Install (Node Preparation): OK
Install (Node Preparation): Checking free space on disk0:
Install (Node Preparation): OK
Install (Node Preparation): Starting package and meta-data sync
Install (Node Preparation): Cleaning packages not in sync list
Install (Node Preparation): Complete
Install (Node Preparation): Syncing package/meta-data contents: /disk0/instdb/ldpath
Install (Node Preparation): Please Wait...
Install (Node Preparation): Completed syncing: /disk0/instdb/ldpath
Install (Node Preparation): Syncing package/meta-data
contents: /disk0/instdb/ldpath.committed
Install (Node Preparation): Please Wait...
Install (Node Preparation): Completed syncing: /disk0/instdb/ldpath.committed
Install (Node Preparation): Completed sync of all packages and meta-data."
|
RP/STBY
|
ios con0/2/CPU0 is in standby
|
Troubleshooting the Line Cards
As each line card powers on, a power-on self-test (POST) is performed on the line card memory. A full set of field diagnostics can also be run on a line card from the system console providing a pass/fail message both in the line card alphanumeric LED display and on the system console.
Check the following to help isolate a problem with the line cards:
•Are both banks of alphanumeric LED displays on?
–The two displays are powered separately. The left display receives power from the DC-DC converter on the line card. The right display is powered directly from the power supply. So, even if the line card has not powered up, the right display could be on. If both displays are off, the line card might not be fully plugged into the backplane connector, there might be a problem with the MBus module on the line card, or the system power supply may be off.
–If both displays are on, check the message being displayed. As soon as the DC-DC converter is turned on by the MBus module, the processor on the line card begins the boot process. Status messages are displayed in the alphanumeric displays as the boot process continues on the line card.
The system attempts to boot identical line cards in parallel. Further, the system boots line cards as soon as they are powered on and become available for backup.
During the line card boot process, which occurs immediately after the RP boot process, you can observe the alphanumeric LED displays on each line card. For additional information about the displays and definitions, see the Cisco IOS XR Troubleshooting Guide (see Obtaining Documentation and Submitting a Service Request, page -xiv).
Troubleshooting Using the Alarm Cards
The router is equipped with two alarm cards:
•One card occupies the dedicated far left slot of the upper card cage.
•A second alarm card occupies the dedicated far right slot of the lower card cage.
In both card cages, the alarm card slot differs from the rest of the card cage slots in that it is labeled to identify it as an alarm card slot, it is physically narrower, and has a different backplane connector.
The following components and indicators are on the front panel of the alarm cards (Figure 4-5):
•Critical (red), Major (red), and Minor (yellow) indicators that identify system level alarm conditions detected by the system through the MBus.
–These indicators are normally off.
•Audio alarm cutoff switch.
•25-pin cable connection to an external alarm.
•Alarm card indicators:
–ENABLED (green)—the alarm card is operational and functioning properly.
–FAIL (yellow)—the alarm card in that slot is faulty.
•A pair of status LEDs that correspond to each of the CFC and SFC card slots in the switch fabric and clock scheduler card cage:
–ENABLED (green)
On—the card installed in that slot is operational and functioning properly.
Off—either the slot is empty or the card installed in that slot is faulty.
–FAIL (yellow)—the card in that slot is faulty.
Figure 4-5 Status LEDs on the Alarm Card
Monitoring Critical, Major, and Minor Alarm Status
The alarms can warn of an overtemperature condition:
•On a component in the card cage
•A fan failure in a blower module
•An overcurrent condition in a power supply
•An out-of-tolerance voltage on one of the cards
The alarm LEDs are controlled by MBus software, which sets the threshold levels for triggering the different stages of alarms.
The RP continuously polls the system for temperature, voltage, current, and fan speed values. If a threshold value is exceeded, the RP sets the appropriate alarm severity level on the alarm card which lights the corresponding LED, and energizes the appropriate alarm display relays to activate any external audible or visual alarms wired to the alarm display. The RP also logs a message about the threshold violation on the system console.
Note You can use the audio alarm cutoff switch to visually check that the alarm card indicators are operating properly. If no audible alarm is active, pressing the audio alarm cutoff switch temporarily lights all of the indicators on the alarm card front panel. If an indicator does not light it means that LED is faulty.
Troubleshooting the Switch Fabric
This section describes the procedures needed to troubleshoot problems with the switch fabric. The RP and the line cards connect through the crossbar switch fabric, which provides a high-speed physical path for most inter-card communication. Among the messages passed between the RP and the line cards over the switch fabric are, actual packets being routed and received, forwarding information, traffic statistics, and most management and control information. This information is useful in diagnosing hardware-related failures.
Note This section is recommended only for advanced Cisco IOS software operators and system administration personnel. Refer to the appropriate Cisco IOS software publications for detailed Cisco IOS information.
Use the following procedure to collect the needed data from the RPs and line cards in order to troubleshoot the switch fabric.
Step 1 Enter the show controllers fia command for the primary and secondary RPs and save the output.
Step 2 Enter the attach <slot #> command to access a line card.
Note Use the attach command when you connect to the line card. The execute-on command is dependent upon the inter-process communication (IPC) which operates over the switch fabric. If you are having problems with IPC, the commands that run remotely through the switch fabric can time out. The attach <slot #> command travels over the MBus and not the IPC.
Step 3 Enter the show controllers fia command for all installed line cards and save the output from each.
Step 4 Proceed to the next section, Analyzing the Data.
Analyzing the Data
Switch fabric problems can occur due to failures in any of the following components:
•RPs
•Line card hardware
•The backplane
•CSCs/SFCs
When troubleshooting switch fabric errors, look for patterns with regard to which components are reporting errors. For example, if you combine the show controllers fia output from all RPs and line cards, you can determine if there is an error pattern. The following subsections discuss the values within the output that can help you determine any error patterns.
crc16 Output
The crc16 data line from the show controllers fia command is an important indicator of hardware problems. If one line card or one CSC/SFC has been on line, inserted and removed, you can expect to see some crc16 error data. However, this number should not continue to increase. If the number is increasing, you may need to replace a faulty hardware component. It is very important to correlate the data from both the primary RP and the secondary RP and all installed line cards. The example output below shows the status of the primary RP. The crc16 data line is underlined and is showing errors from sfc1.
Router#show controllers fia
Fabric configuration: Full bandwidth, redundant fabric
Master Scheduler: Slot 17 Backup Scheduler: Slot 16
redund fifo parity 0 redund overflow 0 cell drops 0
crc32 lkup parity 0 cell parity 0 crc32 0
Switch cards present 0x001F Slots 16 17 18 19 20
Switch cards monitored 0x001F Slots 16 17 18 19 20
Name: csc0 csc1 sfc0 sfc1 sfc2
-------- -------- -------- -------- --------
state Off Off Off Off Off
sca not pres 0 req error 0 uni FIFO overflow 0
grant parity 0 multi req 0 uni FIFO undrflow 0
cntrl parity 0 uni req 0 crc32 lkup parity 0
multi FIFO 0 empty dst req 0 handshake error 0
In the example output below, you can see the status of the line card in slot 2. The crc16 data line is underlined and is showing errors from sfc1.
Entering Console for 4 port ATM Over SONET OC-3c/STM-1 in Slot: 2
Type "exit" to end this session
Press RETURN to get started!
LC-Slot2#show controllers fia
redund FIFO parity 0 redund overflow 0 cell drops 0
crc32 lkup parity 0 cell parity 0 crc32 0
Switch cards present 0x001F Slots 16 17 18 19 20
Switch cards monitored 0x001F Slots 16 17 18 19 20
Name: csc0 csc1 sfc0 sfc1 sfc2
-------- -------- -------- -------- --------
state Off Off Off Off Off
sca not pres 0 req error 0 uni fifo overflow 0
grant parity 0 multi req 0 uni fifo undrflow 0
cntrl parity 0 uni req 0 crc32 lkup parity 0
multi fifo 0 empty DST req 0 handshake error 0
Disconnecting from slot 2.
Connection Duration: 00:00:21
After you have gathered the show controllers fia command data from the RPs and line cards, you can create a table similar to Table 4-6.
Table 4-6 Error Data Collection Table
Card Slot
|
CSC 0
|
CSC 1
|
SFC 0
|
SFC 1
|
SFC 2
|
SFC 3
|
SFC 4
|
0
|
|
|
|
ERROR
|
|
|
|
1
|
|
|
|
|
|
|
|
2
|
|
|
|
ERROR
|
|
|
|
3
|
|
|
|
ERROR
|
|
|
|
4
|
|
|
|
|
|
|
|
5
|
|
|
|
ERROR
|
|
|
|
6
|
|
|
|
|
|
|
|
7
|
|
|
|
ERROR
|
|
|
|
8
|
|
|
|
|
|
|
|
Table 4-6 indicates that more than one line card is reporting errors coming from SFC 1. Therefore, the first step to correcting this problem is to check or replace SFC 1. Whenever a replacement is recommended, first verify that the card is correctly seated (see the "Properly Seating Switch Fabric Cards" section).
Note Always reseat the corresponding card as the first line of troubleshooting to be sure it is correctly seated. If, after reseating the card, the crc errors are still increasing, then replace the part.
The common failure patterns and recommended actions for crc16 errors are as follows (one step at a time until the problem goes away):
1. Errors indicated on more than one line card from the same switch fabric card:
a. Replace the switch fabric card in the slot corresponding to the errors.
b. Replace all switch fabric cards.
c. Replace the backplane.
2. Errors indicated on one line card from more than one switch fabric card:
a. Replace the line card.
b. If errors are incrementing, replace the current master CSC.
c. If errors are not incrementing and the current master is CSC0, replace CSC1.
Grant Parity and Request Errors
Another troubleshooting indicator are the console logs or the output of the show log command, in the form of grant parity and request errors. Look for the type of message that indicates a grant parity error.
The common failure patterns and recommended actions for grant parity and request errors are as follows (one step at a time until the problem goes away):
1. Grant errors on more than one line card:
a. Replace the CSC (see the note below to know which one should be swapped).
b. Replace the backplane.
2. Grant errors on one line card:
a. Replace the line card.
b. Replace the CSC (see the note below to know which one should be swapped).
c. Replace the backplane.
Note If multiple line cards are reporting grant parity or request errors and the router is still functioning, then a CSC switchover has occurred. The failed CSC is the one that is currently the backup CSC (not the one listed as Master Scheduler in the show controllers fia output). If Halted is next to the heading From Fabric FIA Errors or To Fabric FIA Errors, or if the router is no longer forwarding traffic, then a CSC switchover has not occurred and the failing CSC is the one listed as Master Scheduler. By default, the CSC in slot 17 is the primary and the CSC in slot 16 is the backup.
Properly Seating Switch Fabric Cards
The switch fabric cards in the router can be challenging to insert, and may require a small amount of force to seat correctly. If either of the CSCs are not seated properly, you may see an error message.
When dealing with switch fabric and line card booting problems, it is important to verify that all CSCs and SFCs are correctly seated and powered on.
Troubleshooting the Cooling Subsystem
The cooling subsystem of the router consists of an upper and lower blower module in the chassis and a fan in each of the power supplies. The blower modules and the power supply fans circulate air to maintain acceptable operating temperatures within the router (Figure 4-6).
This section contains information to troubleshooting the cooling subsystem and includes:
•Blower Module Operation
•Power Supply Operation
•Overtemperature Conditions
•Isolating Cooling Subsystem Problems
Figure 4-6 Cooling Air Flow
Blower Module Operation
The blower modules maintain acceptable operating temperatures for the internal components by drawing cooling air through a replaceable air filter into the card cages. The blowers occupy a bay near the top and at the bottom of the router.
Each blower module contains three fans, a controller card, and two front panel status LEDs. A snap-on plastic front cover fits over the front panel, but the LEDs are visible through the front covers.
–Green—The blower module is functioning properly.
–Red—There is a fault detected in the blower module.
•If the air temperature inside the chassis rises, blower speed increases to provide additional cooling air to the internal components.
•If the internal air temperature continues to rise beyond the specified threshold, the system environmental monitor shuts down all internal power to prevent equipment damage due to excessive heat.
•If the system detects that one or more of the fans in a blower module has failed, it displays a warning message on the system console and displays a blower failure message on the RP alphanumeric display. In addition, the remaining fans go to full speed to compensate for the loss of the failed fan.
Power Supply Operation
Each AC or DC power supply is equipped with a fan that draws cooler air in through the front of the power module and forces warmer out the back of the power shelf.
•If the power source is within the required range, the power supply fan remains on.
•If the fan fails:
–Power supply detects an internal overtemperature condition
–Fault and Temp indicators light
–Power supply sends an overtemperature warning to the system and then shuts down the system.
For additional power supply troubleshooting information, see the "Troubleshooting the Power Subsystem" section.
Overtemperature Conditions
The following console error message indicates that the system has detected an overtemperature condition or out-of-tolerance power value inside the system:
%ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdown
The preceding message could also indicate a faulty component or temperature sensor. Enter the show environment command or the show environment all command at the user EXEC prompt to display information about the internal system environment. The information generated by these commands include:
•Voltage measurements on each card from the DC-to-DC converter
•The +5 VDC for the MBus module
•The operating voltage for the blower module.
•Temperature measurements received by two sensors on each card (one for inlet air temperature and one for the card's hot-spot temperature), as well as temperature measurements from sensors located in each power supply.
If an environmental shutdown results from an overtemperature or out-of-tolerance condition, the Fault indicator on the power supply lights before the system shuts down.
Although an overtemperature condition is unlikely at initial system startup, make sure that:
•Heated exhaust air from other equipment in the immediate environment is not entering the chassis card cage vents.
•You allow sufficient air flow by maintaining a minimum of 6 inches (15.24 cm) of clearance at both the inlet and exhaust openings on the chassis and the power modules to allow cool air to enter freely and hot air to be expelled from the chassis.
Isolating Cooling Subsystem Problems
Use the following procedure to isolate a problem with the chassis cooling system if you have an overtemperature condition.
Step 1 Make sure the blower modules are operating properly when you power on the system.
•To determine if a blower module is operating, check the two LED indicators on the blower module front panel:
–OK (green)—The blower module is functioning properly and receiving -48 VDC power, indicating that the cables from the chassis backplane to the blower module are good.
–Fail (red)—A fault is detected in the blower module. Replace the blower module.
–If neither indicator is on and the blower is not operating, there may be a problem with either the blower module or the -48 VDC power supplied to the blower module. Go to Step 2.
Step 2 Eject and reseat the blower module making sure the captive screws are securely tightened.
If the blower module still does not function, go to Step 3.
Step 3 Check for -48 VDC power by looking at the LED indicators on each power supply:
•If the Pwr OK indicator is on and the Fault indicator is off on each power supply, it indicates that the blower is receiving -48 VDC.
–If the blower module is still not functioning, there could be a problem with the blower module controller card or an undetected problem in the blower module cable. Replace the blower module.
–If the new blower module does not function, contact a Cisco customer service representative for assistance.
•If the Fault indicator is on, the power supply is faulty. Replace the power supply.
•If the Temp and Fault indicators are on, an overtemperature condition exists.
–Verify that the power supply fan is operating properly.
–If the fan is not operating, replace the power supply.
Contact your Cisco representative if replacing the power supply does not fix the problem.
Feedback
