Table Of Contents
Solving Hardware Problems
Performing Diagnostics
POST Sequence of Events
Cisco ICS 7750 Card LEDs
System Card LED and Shutdown Buttons
LED Behavior During System Bootup
LED Behavior After System Bootup
LED Behavior After Shutdown
Diagnostic Commands
Using the ping Command
Using the traceroute Command
Power Subsystem
System Alarm Processor
Troubleshooting the SAP
Power Supply Modules
Troubleshooting Power Supply Modules
Fans
Checking Fan Speed and Operation
Troubleshooting Fans
System Processing Engine
Troubleshooting SPEs
Routing and Switching Subsystem
Troubleshooting the Routing and Switching Subsystem
How to Reset Chassis Slot Cards Through The SSP
How to Remotely Reset Chassis Slot Cards Through The SAP
Troubleshooting ASIs, MRPs, and WICs
Troubleshooting the SSP
Troubleshooting Cisco ICS 7750 Booting Problems
Monitoring the Cisco ICS 7750 Boot Process
MRP300, MRP3-8FXS, and MRP3-16FXS Boot Sequence
Initial Bootup and Discovery
ASI and MRP200 Boot Sequence
General ASI or MRP200 Boot Problems
Troubleshooting the SPE Running System Manager
Best Practices for Using the IOS CLI
Specific System and MRP-Related Bootup Problems
Troubleshooting MRP200, ASI81, and ASI160 Bootup Problems
Troubleshooting MRP300, MRP3-8FXS, and MRP3-16FXS Bootup Problems
Best Practices for Reseating Cards in the Chassis
SSP Card Boot Problems
Useful MRP Troubleshooting Commands
Solving Hardware Problems
This chapter explains how to solve hardware problems with the Cisco Integrated Communications System (ICS). The chapter is organized as follows:
•
Performing Diagnostics
•Power Subsystem
•Fans
•System Processing Engine
•Routing and Switching Subsystem
•Troubleshooting Cisco ICS 7750 Booting Problems
•Useful MRP Troubleshooting Commands
Note For a description of the features, modifications, and caveats for the
Cisco Integrated Communications System 7750 (Cisco ICS 7750)
release 2.4.0, refer to the Release Notes for System Software Release 2.4.0 on the Cisco ICS 7750.
Performing Diagnostics
You can use a card's power-on self-test (POST) or other diagnostics to locate hardware faults. In some cases, you can run diagnostics on an individual card while the rest of the system continues to operate because only the card being tested is out of service. In other cases, you must take the system off line to run diagnostics; for instance, testing the system processing engine (SPE) running System Manager may take the system off line if there is no secondary SPE310 running Cisco CallManager.
You can run diagnostics remotely through a Telnet or modem connection, locally from a console connected to the console port on the system alarm processor (SAP), or directly through a monitor, mouse and keyboard attached to the SPE310 card.
POST diagnostics are available for the following cards:
•SPEs, including disk drives
•Analog station interface (ASI) and multiservice route processor (MRP) cards, including installed WAN interface cards (WICs), voice interface cards (VICs), and voice WAN interface cards (VWICS)
•SAP
•System switch processor (SSP)
POST diagnostics can be initiated either by unseating and reseating a card or by using a hardware reset command from the SSP. For instance, from enable mode on the SSP, enter the following command:
SSP#hw-module chassis slot 5 restart hold 5
Entering this command causes the card in slot 5 to restart after holding its power off for 5 seconds.
The set request command used in enable mode (after invoking SLPENABLEPASSWORD) on the SAP can also initiate POST. Viewing POST information through the console port on the SAP provides significantly more information than any other method, such as Telnet. From the SAP menu, select the card that you want to check immediately after its POST test has been initiated.
Caution There are only one SAP card and one SSP card in a chassis. If you take the SAP off line, the system's ability to detect alarms associated with the operating environment, fans, and power supply modules will be degraded until the SAP is returned to an on line status. If you take the SSP off line, the system loses LAN connectivity, and calls being made to or from Cisco IP Phones that are routed through that SSP will be disconnected until the SSP is brought back on line.
Note Depending on your system configuration, taking SPEs, ASIs, or MRPs off line can adversely affect users connected to the system. For example, if Cisco CallManager is running on only one SPE, taking that SPE off line disconnects calls to or from the PSTN and prevents the system from processing further PSTN traffic until that SPE is on line. Similarly, taking an ASI or MRP off line that is in the process of routing voice or WAN traffic will prevent that traffic from reaching its destination.
POST Sequence of Events
During POST, some or all of the following events can occur:
•A diagnostic image is downloaded to the card to be tested. The downloading process erases the IOS software from the card memory and installs the diagnostic software image.
•Normal card operation is suspended; the card being tested no longer sends or receives network traffic.
•As testing proceeds, the console displays diagnostic messages.
•A card pass or fail message is displayed on the console when testing ends.
•If the card passes the diagnostic tests, the IOS software is reloaded into the card's memory, and the card resumes normal operations.
Note If a card fails diagnostics, see the troubleshooting help later on in this chapter.
Cisco ICS 7750 Card LEDs
The Cisco ICS 7750 is a chassis-based system that consolidates multiple cards with different functions (SPEs, MRPs, SSP, and SAP), and power supply modules into a single system unit. Each system card has LEDs that provide visual indications about the status of the card.
The following sections describe the LEDs and LED behavior on the various system cards:
•System Card LED and Shutdown Buttons
•LED Behavior During System Bootup
•LED Behavior After System Bootup
•LED Behavior After Shutdown
System Card LED and Shutdown Buttons
The LEDs that are common to each of the Cisco ICS 7750 cards are the alarm and status LEDs. Some cards have additional special-purpose LED(s), depending on the card's function. There is also one shutdown button on each card.
Table 3-1 summarizes the LEDs that are available on each system card.
Table 3-1 Cisco ICS 7750 Card LEDs
LED
|
LED Color Options
|
SPE
|
MRP
|
|
|
Alarm
|
Yellow, amber, red
|
X
|
X
|
X
|
X
|
Status
|
Green
|
X
|
X
|
X
|
X
|
Slot
|
Green
|
|
X3
|
|
|
Power4
|
Green
|
|
|
|
X
|
Fan
|
Yellow
|
|
|
|
X
|
Temperature
|
Yellow
|
|
|
|
X
|
LED Behavior During System Bootup
During system bootup, the LEDs on each system card are controlled by the card on which they reside.
Table 3-2 shows the color and status of each LED on the MRP and SSP cards during the cards' various states from system power up to system bootup.
Table 3-2 MRP and SSP Card LED States
LED
|
Rommon Mode
|
During IOS Bootup
|
After IOS Bootup
|
Alarm LED (yellow)
|
Steady
|
Steady
|
Off, controlled by IOS1
|
Status LED (green)
|
Blinking
|
Blinking
|
Steady, controlled by IOS
|
Slot LEDs2 (green)
|
Steady if the slot is populated
|
Steady if the slot is populated
|
Steady if the slot is populated
|
Table 3-3 shows the color and status of each LED on the SPE card during its various states from system power up to system bootup.
Table 3-3 SPE LED States
LED
|
In BIOS
|
During System Bootup
|
After System Bootup
|
Alarm LED (yellow)
|
Steady
|
Steady
|
Off, controlled by FMM
|
Status LED (green)
|
Blinking
|
Blinking
|
Steady, controlled by FMM
|
Table 3-4 shows the color and status of each LED on the SAP card during its various states from system power up to system bootup.
Table 3-4 SAP LED States
LED
|
During Firmware Bootup
|
After Firmware Bootup
|
Alarm LED (yellow)
|
Steady
|
Off, controlled by SAP firmware
|
Status LED (green)
|
Blinking
|
Steady
|
Power LEDs
|
Steady
|
Steady if it is normal
Blinking if it fails, controlled by SAP firmware
|
Fan LEDs
|
Off
|
Off if it is normal
Blinking if it fails, controlled by SAP firmware
|
Temperature LED
|
Off
|
Off if it is normal
Blinking if it fails, controlled by SAP firmware
|
LED Behavior After System Bootup
Once the Cisco ICS 7750 boots up, each card in the chassis transfers the alarm LED control functionality to ICS System Manager and FMM (on the SPE running System Manager). ICS System Manager and FMM determine the appropriate behavior of the alarm LED on each card, based on the traps/events sent by the cards in the chassis. The only color options that are used by ICS System Manager and FMM are yellow and amber, depending on the severity of the failure.
After system bootup, the status LED remains in its bootup state, as described in Table 3-2, Table 3-3, and Table 3-4 unless the shutdown button is pressed.
Slot LEDs on the MRP also remain in their bootup state, as described in Table 3-2.
Power, fan, and temperature LEDs on the SAP are controlled by the SAP firmware. Status changes are communicated to ICS System Manager and FMM.
LED Behavior After Shutdown
When you press the shutdown button on a system card, that action results in a complete shutdown of the card. The shutdown sequence is as follows:
•The status LED begins to blink immediately and keeps blinking during the shutdown preparation process; on the MRP, SSP, and SAP, the status LED blinks for 15 seconds.
•On the SPE, the operating system shuts down first. Then the card powers down.
•On the MRP, SSP, and SAP, card functionality completely stops.
•The software or firmware on a system card turns off the status LED on the card after the shutdown preparation period.
Note Refer to the Cisco Interface Cards Installation Guide for information on LED behavior for VICs, WICs, or VWICs.
Diagnostic Commands
Many of the cards in the Cisco ICS 7750 support commands that can help you better understand what is happening in your internetwork. This section provides information on the following topics:
•Using the ping Command
•Using the traceroute Command
Using the ping Command
The ping command is a test that is available on Cisco internetworking devices and on many host systems. In TCP/IP, this diagnostic tool is also known as an Internet Control Message Protocol (ICMP) echo request.
To check host reachability and network connectivity, use the IOS ping command. This command enables you to confirm basic network connectivity on many kinds of networks.
For IP connections, in EXEC mode, the ping command sends ICMP echo messages. If a station receives an ICMP echo message, it sends an ICMP echo reply message to the source.
In privileged EXEC mode, the ping command permits you to specify the supported IP header options. With these options specified, the target device can perform a more extensive range of test options.
It is a good idea to use the ping command when the network is functioning properly to see how the command works under normal conditions, so that you have something to compare to when troubleshooting.
Using the traceroute Command
In EXEC mode, the IOS traceroute command discovers the routes that packets follow to their destinations. In privileged EXEC mode, traceroute (also referred to as an extended trace) enables you to specify the supported IP header options, allowing the target device to perform a more extensive range of testing.
The traceroute command uses the error message generated by devices (routers or cards) when a datagram exceeds its time-to-live (TTL) value. Probe datagrams are sent initially with a TTL value of 1, which causes the first device to discard the probe datagrams and send back "time exceeded" error messages.
The traceroute command then sends several probes and displays the round-trip time for each. After every third probe, the TTL is increased by 1.
Each outgoing packet can result in one of two error messages. A "time exceeded" error message indicates that an intermediate device has seen and discarded the probe. A "port unreachable" error message indicates that the destination node has received the probe and discarded it because it could not deliver the packet to an application.
If the timer goes off before a response comes in, an asterisk (*) is displayed.
The trace operation terminates when the destination responds, when the maximum TTL is exceeded, or when you interrupt the trace with the escape sequence.
As with ping, it is a good idea to use the traceroute command when the network is functioning properly to see how the command works under normal conditions, so that you have something to compare to when troubleshooting.
For more information on ways to use the traceroute EXEC and privileged EXEC commands, refer to the "Troubleshooting and Fault Management Commands" chapter in the Cisco IOS Configuration Fundamentals Command Reference.
Power Subsystem
This section explains how to determine and resolve problems associated with the power subsystem, which includes the following components:
•System Alarm Processor
•Power Supply Modules
Note The SAP monitors the physical state of the system, including the availability of system power and cooling air.
System Alarm Processor
This section explains how to isolate problems associated with the SAP. Figure 3-1 shows the front panel of the SAP card.
Figure 3-1 SAP Card
Troubleshooting the SAP
Complete the following steps to isolate SAP problems:
Step 1 Check the system LEDs. If no SAP LEDs are on, ensure that the SAP is properly seated in the chassis and that the power supply modules and fans are functioning properly. (See "Power Supply Modules" section or "Fans" section for additional information.)
Step 2 Check all the SAP LEDs:
•If the ALARM LED is on (amber or yellow), the system has most likely found a problem outside the SAP. Check system status for additional indications that can help you isolate the problem.
•Verify that the STATUS LED is on (green), indicating that the system software has initialized successfully and that the system is operational.
•If the STATUS LED is off, the SAP might have pulled away from the backplane. If the SAP is not seated properly, it may cause the system to hang.
•If one or both of the PWR LEDs are off when they should be on (green), see "Power Supply Modules" section.
•If the FAN LED is off or the TEMP LED is on (yellow), see "Fans" section.
Power Supply Modules
This section explains how to isolate power supply module problems. Figure 3-2 shows the front panel of the power supply module.
Figure 3-2 Power Supply Module
Troubleshooting Power Supply Modules
Follow these steps to isolate power supply module problems; refer to Table 3-5 if necessary:
Step 1 Verify that the fans are operating and that the STATUS LED of at least one SPE is on (green). Look at the SAP card: If the FAN and STATUS LEDs are on (green) but the power supply AC OK LED is off, there is probably a faulty power supply module LED.
Step 2 Verify that the system power switch is set to the on position.
Step 3 Verify that the power source, power cable, and at least one power supply module are functioning correctly. Swap parts to determine whether one of the components is faulty. (Refer to the Cisco ICS 7750 Installation and Configuration Guide.)
Table 3-5 lists symptoms of and possible solutions for power problems.
Table 3-5 Power Supply Module Problems and Solutions
Symptom
|
Power Supply Module LED Status 1 Change
|
Possible
Cause
|
Solutions
|
The system shuts down after being on for a short time.
|
AC OK: on
OVERTEMP: off
DC OK: off
|
Power or cooling problem
|
•Verify that the area in which the system is installed meets the environmental requirements in the Cisco ICS 7750 Installation and Configuration Guide and in the "Site Requirements" section in the Regulatory Compliance and Safety document that came with your system.
•Verify that nothing is blocking the air intake or exhaust. (See "Fans" section.)
•The power supply module has failed. Install a new power supply module.
|
The system attempts to boot, but all LEDs remain off.
|
AC OK: on
OVERTEMP: off
DC OK: off
|
Power problem
|
•Ensure that the power on/off switch is in the on position.
•The power supply module has failed. Install a new power supply module.
|
The power supply module is not operating within its normal operating tolerances (in a single power supply module configuration).
|
AC OK: on
OVERTEMP: off
DC OK: off
|
Defective power supply module
|
•If the system is still operating, insert a new power supply module into the unused power supply slot, and turn it on. If the newly installed power supply module is functioning properly, remove the defective power supply module in the other slot.
•If the system is not operating, replace the defective power supply module.
|
The power supply module has exceeded its maximum operating temperature and is about to shut down (in a single power supply module configuration).
|
AC OK: on
OVERTEMP: on
DC OK: off
|
Cooling problem
|
•Correct any problems associated with the fans. (See "Fans" section.) The power supply module will restart if the high-temperature condition is caused by a problem with the fans.
•Replace the power supply module.
|
A power supply module is producing out-of-tolerance power (in a dual power supply module configuration).
|
AC OK: on
OVERTEMP: off
DC OK: off
|
Defective power supply module
|
Replace the defective power supply.
|
A power supply module has exceeded its maximum operating temperature and is about to shut down (in a dual power supply module configuration).
|
AC OK: on
OVERTEMP: on
DC OK: off
|
Cooling or power supply module problem
|
•Correct any problems associated with the fans. (See "Fans" section.) The power supply module will restart if the high-temperature condition is caused by a problem with the fans.
•Replace the defective power supply module.
|
Fans
This section explains how to isolate problems associated with the fans, which are located in the fan tray at the bottom of the system chassis.
Checking Fan Speed and Operation
The SAP on the Cisco ICS 7750 monitors the operational status of each component in the system.
Working with the Fault Management Module (FMM) and ICS System Manager, the SAP can remotely alert you of environmental, functional, or operational problems detected within the system. Access is available through the console port on the front of the card.
One element of the Cisco ICS 7750 that can affect performance is the speed of its fans. The fans run at different speeds, depending on the system's operation:
•At power up, the fans should run at high speed (high noise level) for up to 10 seconds. After that time, they should go into normal operation mode.
•At normal operation, the fans should be spinning at a low speed (low noise level).
•During a software upgrade, the fans should run quietly.
•Following a software upgrade, the fans should run as they do during the power-up process, spinning fast for about 10 seconds and then slowing down to normal operation.
Follow these steps to check the fans for proper operation:
Step 1 On a PC, open a HyperTerminal session with the SAP card.
Step 2 At the prompt, enter enable mode by typing slpenablepassword, followed by the password for the card.
Step 3 Enter the following command:
where fan# is the number of the fan that you wish to check.
The following is an example of output that shows fans running at normal operation:
AlarmCard>get fan-speed :1
AlarmCard>get fan-speed :2
AlarmCard>get fan-speed :3
AlarmCard>get fan-speed :4
Troubleshooting Fans
Follow these steps to isolate fan problems. Refer to Table 3-6 if necessary.
Step 1 To verify that the fans are operating, place your hand near the air exhaust on the back of the chassis, and check for a warm flow of air. If the fans are not operating, there might be a problem with the fan tray or with the power source. If the fans are not operating and if the FAN LED on the SAP is off, ensure that the fan tray is seated properly. If the LEDs on the SAP are off, ensure that the SAP is also seated properly. If there are no LEDs illuminated on the SAP, disengage the ejector levers to unseat the card, and then reseat it to ensure that it is seated properly in the chassis.
Step 2 If the system and the fans start up but shut down after about 2 minutes, one or more fans might have failed.
If one or more fans fail to operate or if one or more fans have tachometer speeds that are not within the acceptable range, you might need to replace the fan tray.
Step 3 If you see the following message at startup, the system has detected an overtemperature condition or out-of-tolerance power inside the chassis:
This message might indicate an environmental problem; ensure that there is adequate ventilation and unrestricted air flow in the location where the Cisco ICS 7750 is installed. This message might also indicate a malfunctioning component or a faulty temperature sensor within the Cisco ICS 7750 chassis.
Step 4 Ensure that heated exhaust air from other equipment is not entering the inlet vents and that there is sufficient clearance around the chassis to allow cooling air to flow.
Step 5 Ensure that the SAP is properly installed. If the SAP is not properly installed, the fans will not operate.
Table 3-6 lists symptoms of and possible solutions for fan problems.
Table 3-6 Fan Problems and Solutions
Syslog Message
|
Symptom
|
SAP LED
Status Change
|
Possible
Cause
|
Solutions
|
|
The system is overheated and will shut down 10 seconds after sending a warning message.
|
ALARM: on (amber)
TEMP: on
|
High ambient air temperature or air intake or exhaust blockage
|
•Verify that the room temperature in which the system is located is less than 104°F (40°C).
•Verify that there are no air intake blockages at the front of the chassis.
•Verify that there are no air exhaust blockages at the back of the chassis.
•Verify that there are no internal system blockages.
|
|
A system card (other than the SAP) is overheated.
|
ALARM: on (amber)
TEMP: on
|
High ambient air temperature, air intake or
exhaust blockage, or card failure
|
•Verify that the room temperature in which the system is located is less than 104°F (40°C).
•Verify that there are no air intake blockages at the front of the chassis.
•Verify that there are no air exhaust blockages at the back of the chassis.
•Verify that there are no internal system blockages.
•Replace the card.
|
The ambient
temp x
exceeded
recommended
value
|
The temperature inside the chassis is over 104°F (40°C).
|
ALARM: on (amber)
TEMP: on
|
High ambient air temperature or air intake or exhaust blockages
|
•Verify that the room temperature in which the system is located is less than 104°F (40°C).
•Verify that there are no air intake blockages at the front of the chassis.
•Verify that there are no air exhaust blockages at the back of the chassis.
•Verify that there are no internal system blockages.
|
|
A fan tachometer reading drops below the designated threshold, or a fan stops completely.
|
ALARM: on (amber)
FAN: on
TEMP: on
|
Improperly seated or defective fan tray
|
•Remove, inspect, and reinsert the fan tray.
•If the fans are still not operating, install a new fan tray.
•Refer to the
Cisco ICS 7750 Installation and Configuration Guide.
|
|
All fan tachometer readings fall to zero.
|
ALARM: on (amber)
FAN: on
TEMP: on
|
Missing or improperly seated fan tray
|
•Remove, inspect, and reinsert the fan tray.
•If the fans are still not operating, install a new fan tray.
•If the fans are still not operating, install a new SAP.
•Refer to the
Cisco ICS 7750 Installation and Configuration Guide.
|
System Processing Engine
This section explains how to isolate problems associated with the SPE. Figure 3-3 shows the front panel of the SPE310.
Figure 3-3 SPE Card
Note Refer to the Installing Memory, PVDM, and VPN Modules in ASI Cards, MRP Cards, and SPE Cards in the Cisco ICS 7750 documentation for a detailed view of the SPE310 card.
Note The Cisco ICS 7750 supports a maximum of 5 SPEs in a single chassis.
Troubleshooting SPEs
Follow these steps to isolate problems with the SPE:
Step 1 Check the system LEDs. If no SPE LEDs are on, ensure that the SAP, power supply modules and fans are functioning properly. (See "Power Supply Modules" section and "Fans" section for additional information.)
Step 2 Check all the SPE LEDs.
a. If any ALARM LED is on (amber or yellow), except during system boot or POST, the system has detected a hardware failure in that SPE. Contact a technical support representative for instructions.
b. Verify that the STATUS LED on all SPEs is on (green), indicating that the system software has initialized successfully and that the system is operational.
c. If the STATUS LED on an SPE is off, that SPE might have become disengaged from the backplane or might be in a shutdown state. If an SPE is not seated properly, it may hang the system.
Table 3-7 lists symptoms of and possible solutions for hardware-related problems on the SPE.
Table 3-7 SPE Problems and Solutions
Symptom
|
SAP LED Status Change
|
SPE LED Status Change
|
Possible
Cause
|
Solutions
|
The system cannot read or write to an SPE hard disk.
|
ALARM: on (amber)
|
None
|
Inadequate security privileges (read/write permission) or hard disk errors
|
•The SPE may be out of disk space. Use Windows Explorer to determine how much space is left on the hard disk and, if necessary, move data to another server.
•You may not have the necessary authorization. If attempting to write to the disk, use ICSConfig to determine whether you have administrator privileges.
•The hard disk may have errors. Close all applications running on the disk and use the Windows 2000 Check Disk utility to scan for and attempt recovery of bad sectors.
Note The disk is not available to run other tasks while Check Disk is running.
•Replace the SPE.
|
SPE memory parity errors have occurred.
|
ALARM: on (amber)
|
ALARM: on (amber)
|
Defective memory or hard disk
|
Replace the SPE.
|
SPE failed to boot.
|
ALARM: on (yellow)
|
ALARM: on (yellow)
STATUS: off
|
Defective memory module and defective hard disk in SPE
|
Replace the SPE.
|
Routing and Switching Subsystem
This section explains how to isolate problems associated with the routing and switching subsystem, which includes the following components:
•One or more ASI cards or MRP cards. Each ASI 81 card and MRP card can include different combinations of WICs, VICs, and VWICs.
•SSP.
Note For problems associated with VICs and trunks or for other voice-related issues, see Chapter 9, "Solving Voice Problems." For instructions on how to solve problems concerning Catalyst 3524-PWR XL switches, refer to the documentation that came with the Catalyst 3524-PWR XL switches.
Figure 3-4 shows the front panels of an MRP card (on the left) and an SSP card (on the right).
Note Refer to the Installing Memory, PVDM, and VPN Modules in ASI Cards, MRP Cards, and SPE Cards in the Cisco ICS 7750 documentation for a detailed view of the MRP200, ASI81, ASI160, MRP300, MRP3-8FXS, and MRP3-16FXS cards.
Figure 3-4 Routing and Switching Subsystem (MRP and SSP)
Troubleshooting the Routing and Switching Subsystem
The problems listed below could require replacement of an ASI, MRP, or SSP:
•An ASI or MRP fails, even when moved to another slot. (If the card fails in one slot but operates properly in another, there might be a problem with the backplane or SSP.)
•Diagnostics that loop data through the SSP fail on two or more Catalyst 3524-PWR XL switches that are connected to that SSP.
•Traffic is not passing through the system, but the ASIs, the MRPs and the SSP all appear to be functioning normally.
•The system has data transmission problems that do not go away when you replace the card that appears to be failing, or the system has problems that occur in several cards simultaneously. (Problems of this type can also indicate a faulty backplane.)
•You cannot fully insert an ASI, MRP, or SSP into its slot. This problem most likely indicates damage to the connectors on either the card or the backplane. Inspect all the connectors. If you find damage, replace the card or the chassis.
How to Reset Chassis Slot Cards Through The SSP
This procedure can be useful when you are remotely troubleshooting a Cisco ICS 7750, or when you mistakenly shut down the SPE instead of restarting it. Generally, when restarts of an SPE fail, you need to unseat the SPE and then reseat it in order to get it to boot. To perform these actions, you need to have hands-on access to the Cisco ICS 7750. If the Cisco ICS 7750 is remotely located, you will need to obtain on-site support in order to proceed.
To restart Cisco ICS 7750 cards using the SSP, start at the command-line interface (CLI) for the SSP. This can be through Telnet or through the console of the Cisco ICS 7750, using a 2511 access server if you are not locally connected to the system.
At the CLI, issue the following command:
Switch#hw-module chassis slot n restart
where n represents the slot number in which the line card to restart is installed.
Switch#hw-module chassis slot n
where n represents the chassis slot number.
Options for this command include a delay before the line card is restarted, as well for a hold in the unpowered state between the bring-down and bring-up phases. Examples follow:
Switch#hw-module chassis slot 1 restart ?
delay Seconds to delay before asserting reset
hold Seconds to assert reset
Switch#hw-module chassis slot 1 restart delay ?
<0-2147483647> Delay in seconds
Switch#hw-module chassis slot 8 restart hold ?
<0-2147483647> Seconds to assert reset
How to Remotely Reset Chassis Slot Cards Through The SAP
You can use the SSP for remote troubleshooting purposes, as described in "How to Reset Chassis Slot Cards Through The SSP" section, you can also access the SAP menu without being directly connected to the Cisco ICS 7750 chassis. Either method may be used, depending on user preference, the state of the system, and the task to be performed.
Follow these steps to set up remote troubleshooting capability, using the SAP console:
Step 1 At the Cisco ICS 7750 chassis, plug the blue RJ-45 console cable into the console port on the SAP.
Step 2 Plug the other end of the console cable (DB-9 connector) into Com port 1 or Com port 2 (if you are using voice mail, plug this end into Com port 2).
Step 3 Access the SPE running System Manager, using a Terminal Services Client connection.
Step 4 Open the Hyperterminal application on the SPE, using Com 1 or Com 2 (make sure to set up your Hyperterminal connection to recognize this port).
You will be presented with the SAP prompt as if you were directly connected to the Cisco ICS 7750.
Step 5 From the SAP menu, enter SLPENABLEPASSWORD to access the administrator functions on the SAP.
Step 6 Enter the administrator password.
Step 7 At the prompt, enter the set reset command, as follows:
AlarmCard>set reset 5,1,1
In this example, the SAP will reset (re-enable) the card in slot 5 after 1 second.
Troubleshooting ASIs, MRPs, and WICs
Table 3-8 lists symptoms of and possible solutions for problems with ASIs, MRPs, and WICs.
Table 3-8 ASI, MRP, and WIC Problems and Solutions
Symptom
|
ASI or MRP LED Status Change
|
Possible Cause
|
Solutions
|
All ASIs and MRPs do not display status as expected.
|
STATUS: off
(all ASIs and MRPs)
|
Power problem
|
Check your system power connections and power supply modules. (See "Power Subsystem" section.)
|
A single ASI or MRP does not display status as expected.
|
STATUS: off (single ASI or MRP)
|
Improperly seated card or chassis slot problem
|
•If the STATUS LED remains off, verify that the card is seated properly.
•Assuming that the power LED of other ASIs or MRPs is on, try inserting the card in a different slot.
|
A WAN interface is down.
|
ALARM: on (amber)
|
Improperly configured or defective WIC
|
•If the state change is unexpected, verify that the WIC that is reporting the problem is properly configured.
•Contact your technical support representative for assistance.
|
There is loss of signal on a WAN serial link.
|
ALARM: on (amber)
|
Cable or WIC problem
|
•Replace the cable attached to the WIC that is reporting the error.
•Have your telephone service provider perform a loopback test to verify the integrity of the WAN link.
•Replace the faulty WIC.
•Contact your technical support representative for assistance.
Note See "Solving Serial Connection Problems," for additional information.
|
The WIC is not supported.
|
ALARM: on (amber)
|
WIC installed in ASI 81 or MRP not supported by currently loaded IOS software
|
•Determine whether the WIC is supported by the IOS software release running on the ASI or MRP.
•Contact your technical support representative for assistance.
|
An ASI or MRP initialization error has occurred.
|
ALARM: on (amber)
|
Checksum error detected while downloading WIC firmware
|
Copy the error message exactly as it appears, and contact your technical support representative.
|
The ASI or MRP has failed to boot.
|
ALARM: on (yellow)
STATUS: off
|
Improperly seated or defective ASI or MRP, IOS image not downloaded from SPE (see Table 3-10 for additional information)
|
•Remove and reinsert the ASI or MRP.
•Replace the ASI or MRP.
•Using System Manager, check the System Manager Software Upgrade table on the SPE running System Manager to see if an appropriate IOS image has been delivered to the ASI or MRP card.
|
A WIC minor alarm occurs.
|
ALARM: on (yellow)
|
WIC link integrity failure or incorrect WIC configuration
|
•Verify that the WIC configuration is correct.
•Have your telephone service provider perform a loopback test.
|
There are WIC transmit or receive errors.
|
ALARM: on (yellow)
|
•Consecutive frame bits on receive line
•Errors on receive or transmit lines
•Remote signal or frame errors on receive and transmit lines
|
•Check the status of the carrier line.
•Contact your technical support representative for assistance.
|
Note WIC, VIC, and VWIC connectivity that is lost for an extended period of time, such as 30 minutes, may not be regained without resetting the far end of the connection.
Troubleshooting the SSP
Table 3-9 lists symptoms of and possible solutions for problems with an SSP.
Table 3-9 SSP Problems and Solutions
Symptom
|
SSP LED Status Change
|
Possible Cause
|
Solutions
|
SSP does not display status as expected.
|
STATUS: off
|
Power problem or improperly seated card.
|
•Check system power connections and power supply modules. (See "Power Subsystem" section.)
•If the STATUS LED remains off, verify that the SSP is seated properly.
|
An SSP interface is down.
|
ALARM: on (yellow)
|
Interface intentionally disabled or errors on network.
|
•If the state change is unexpected, verify that the interface has not been explicitly disabled by an administrator.
•Look for an address violation, such as an address mismatch or duplication. If there is any address duplication, disable address violation detection.
•Verify that there are no network connection errors, such as a loss of link beat or jabber.
•Contact your technical support representative for assistance.
|
SSP fails to boot.
|
STATUS: blinking (green)
|
Switch software has become corrupted or there is no bootable file in Flash.
|
This situation can cause the SSP to drop into ROMMON mode with no bootable file in Flash.
•Reseat the SSP in the chassis.
•If the problem persists, use XMODEM to copy the correct image from the PC to the SSP (ensure the baudrate on the SSP is configured with the default value of 9600). Refer to Recovery From Corrupt or Missing Software Image - Cisco Catalyst 2900XL, 3500XL, and 2950 Series Switches for additional information.
•Contact your technical support representative for assistance.
|
Troubleshooting Cisco ICS 7750 Booting Problems
This section describes various ways to troubleshoot problems that may prevent or interfere with the ability of the Cisco ICS 7750 to boot. This section is organized as follows:
•Monitoring the Cisco ICS 7750 Boot Process
•MRP300, MRP3-8FXS, and MRP3-16FXS Boot Sequence
•ASI and MRP200 Boot Sequence
•General ASI or MRP200 Boot Problems
•Specific System and MRP-Related Bootup Problems
•SSP Card Boot Problems
Monitoring the Cisco ICS 7750 Boot Process
You can monitor the boot process on the Cisco ICS 7750 to verify that it comes up correctly, as follows:
Step 1 Using a HyperTerminal connection, connect a PC to the console port of the SAP card.
The HyperTerminal connection will display information that is flowing across the serial connection.
Step 2 Power on the Cisco ICS 7750 chassis.
The SAP card should display fan, temperature, power supply, and hardware sensor information.
Step 3 Once the display stops, press CTRL-backslash (\), and choose the SSP switch card (option C).
The SSP will boot in the regular Catalyst switch sequence. However, although it appears to be done booting, it is not done.
Several minutes will elapse with little apparent activity. During this time, the SPE running System Manager is launching various essential services.
•If this is a new Cisco ICS 7750 system or if it is the first time the SPE has been booted up, 8 to 10 minutes may elapse. The amount of time will also depend on the software package installed on the SPE.
•If this is just a reboot of a configured system, the time will be less—3 to 5 minutes. There is little point in trying to monitor that directly. It is recommended that you do not attempt to access the SPE during this period.
After a few minutes, the SPE running System Manager will download a configuration to the SSP switch card. You will know that the download is completed when you see (on the SSP) the following:
*Mar 1 00:00:23.938 UTC: %SYS-5-CONFIG_I: Configured from console by
vty0 (10.0.0.1)
Note The IP address, 10.0.0.1, is the default IP address for the SPE running System Manager. This IP address may be different after you have configured the system using ICSConfig.
If you then display the configuration of the SSP, you will find the following:
•It has a password. (If this is the initial boot of the system or of a replacement SPE running System Manager, the password will be changeme.)
•It has an IP address on Interface VLAN 1. (If this is the initial boot of the system or of a replacement SPE running System Manager, the IP address will be in the 10.0.0.0 network, with a subnet mask of 255.255.255.0.).
Note If any of these settings is incorrect, the system will not function, even if you attempt to change settings or otherwise intervene manually. Contact the Cisco TAC for assistance.
If the SSP has the correct configuration (IP address and password), press CTRL-backslash (\), and enter the slot number of an MRP card.
The MRP may still be broadcasting a BOOTP request. If so, let it continue because it may take several minutes for the MRPs to receive their boot information—their IP address, IOS image, and configuration.
The reason why MRP BOOTP requests go unanswered before the MRP boots up is that the MRPs are the last cards to boot in the chassis. Before the SPE running System Manager can answer the MRPs' boot requests, that SPE must first boot itself and start ICS System Manager. It then answers the BOOTP requests from other cards, such as MRP200 and ASI cards, that may be present. Also, before the Cisco Network Registrar (CNR) responds to a BOOTP request, it must ensure that the request has originated from within the same chassis. Typically, the requests from an MRP or ASI, that fall within the first 30 seconds after the SPE running System Manager has fully started, will not receive a response.
When the MRP starts booting from 10.0.0.1 (the SPE running System Manager), you can assume that the boot process is likely to succeed.
When all the MRPs boot from the SPE running System Manager, the Cisco ICS 7750 is ready to operate.
You can use the same monitoring process to verify that the system boots correctly after running the initial configuration program (ICSConfig). Of course, the IP address and password on the SSP must match the ones you assigned in ICSConfig.
If the boot process is not successful, see "Specific System and MRP-Related Bootup Problems" section for additional information.
MRP300, MRP3-8FXS, and MRP3-16FXS Boot Sequence
The MRP200 and MRP300 cards are both voice-and-data-capable routers that support digital and analog voice trunks and WAN routing interfaces to link remote Ethernet LANs to the Public Switched Telephone Network (PSTN), to existing private branch exchanges (PBXs), and to most common analog devices, such as fax machines and teleconferencing stations.
The MRP300, MRP3-8FXS, and MRP3-16FXS cards are Flash-based cards. They have additional functionality provided by 16 MB of onboard Flash memory, with 64 MB of add-on Flash memory available as an option.
Configuration files for the Flash-based cards (MRP300, MRP3-8FXS, and MRP3-16FXS) are loaded and saved on these cards in nonvolatile RAM (NVRAM). The performance impact is an improvement in boot time. When there is an IOS image in Flash memory, the Flash-based MRP card boot time is reduced because it does not have to TFTP the image from the SPE.
Note The MRP Flash-based cards should obtain their IP address and initial configuration from the ICS System Manager that is running in the same chassis, and not from any other external Dynamic Host Control Protocol (DHCP) server.
Initial Bootup and Discovery
The MRP Flash-based cards are shipped with a default IOS image in Flash memory, and with no configuration in their NVRAM. The first time the Flash-based MRP boots up in a Cisco ICS 7750 chassis, the card will send DHCP requests (see number 2, below).
If the boot field of the configuration register, which controls the way the system boots, is set to "Boot to ROMMON" (0x0), then the system will remain at the ROM Monitor (ROMMON) prompt. Setting the configuration register to 0x1 will not initiate BOOTP requests, and the MRP will try to boot from Flash memory.
If the boot field of the configuration register is set to anything other than "Boot to ROMMON," then either of the following will occur:
•The system will stay in ROMMON mode if the Flash memory is corrupt or if there is no image in Flash. No attempt is made to obtain an IOS image.
•ROMMON boots from the image in Flash memory if there is an IOS image in Flash memory.
After IOS has booted, the following should occur:
1. IOS enters the standard IOS prompt if the Fast Ethernet interface and all its subinterfaces (if there are any), are configured with IP addresses by the startup-configuration in NVRAM.
2. If the Fast Ethernet interface, or any of its subinterfaces, are not configured with IP addresses because the startup-configuration in NVRAM did not complete, IOS will attempt to discover the IP addresses by sending DHCP requests. IOS will not perform DHCP auto-installation. DHCP responses are accepted from any DHCP server.
3. If there is no startup-configuration in NVRAM, then IOS will attempt to discover the IP addresses and download the startup-configuration through the standard DHCP auto-installation mechanism, from ICS System Manager in the same chassis only.
•The Vendor class identifier field in DHCP request messages sent during DHCP auto-installation will be filled with any one of the following strings, as appropriate:
–ICS7750-MRP300—for MRP300
–ICS7750-MRP3-8FXS—for MRP3-8FXS
–ICS7750-MRP3-16FXS—for MRP3-16FXS
4. The DHCP response message from ICS System Manager will contain the IP address, the IP address of the TFTP server, and the configuration filename. During DHCP auto-installation, the Flash-based MRP card looks for the pattern ICS7750-SPE in sub-option 66 of Option 43 (Vendor specific information) in the DHCP response. This entry ensures that the Flash-based MRP card obtains its IP address and initial configuration from the ICS System Manager, and not from any external DHCP server.
5. If the DHCP response contains the expected pattern (in sub-option 66 of Option 43) and a configuration filename, the Flash-based MRP card downloads the configuration file from the TFTP server and merges it with the running-configuration file.
6. The Flash-based MRP card is now configured with the passwords and IP address delivered by the ICS System Manager. The configuration is saved locally in the Flash-based card's NVRAM. When a configuration change is made to the Flash-based card by using ICSConfig, the ICS System Manager telnets to the Flash-based card and issues the copy running-config startup-config command, which saves the configuration in that card's NVRAM. When manual changes are made to the configuration by using the CLI from the MRP's console, the running-config will be written to the startup-config in NVRAM when the following command is executed: copy running-config startup-config.
Note Once NVRAM is populated, the configuration from NVRAM will be read to configure the Flash-based MRP card on subsequent boots. From this point on, the MRP does not use ICS System Manager to obtain its bootup information.
Note When a Flash-based MRP card is moved from one chassis to another, the startup-config must be erased so that the Flash-based card can obtain its new IP address from the ICS System Manager in the new chassis. Failure to erase the startup-config will result in the inability of ICS System Manager to discover the card.
7. The Flash-based MRP card then sends Cisco Discovery Protocol (CDP) packets. ICS System Manager discovers the Flash-based MRP card from the CDP cache on the SSP. The configuration is not automatically saved to NVRAM.
8. If ICS System Manager does not respond to the Flash-based card's DHCP requests, then after a short period of time (~17 seconds), the Flash-based card enters the standard initial setup dialog. DHCP requests will continue to be sent in the background, until the setup dialog is quit or aborted or until the Fast Ethernet interface is configured with an IP address.
Note If the Flash-based MRP card is reloaded after changing, and before saving, the configuration to NVRAM, the MRP will boot with the previously saved configuration. If the MRP is reloaded after erasing NVRAM, then it will follow the boot procedure outlined above.
ASI and MRP200 Boot Sequence
The ASI and MRP200 cards do not contain Flash memory. The configuration files for these cards are loaded and saved to the SPE running System Manager.
When ASI and MRP200 cards boot, the following sequence of events should occur:
1. The card runs a bootstrap program called ROM Monitor (ROMMON) that helps to initialize the card processor and to boot the operating system. While in ROM Monitor mode, the card sends a BOOTP request.
2. Cisco Network Registrar software, which operates on the SPE running System Manager, replies to the BOOTP request sent by the ASI or MRP200, provided that the ASI or MRP200 has been discovered by the SPE running System Manager. The BOOTP reply contains the locations of both the IOS image and the IOS configuration file for the MRP200 and ASI cards.
Note There is potential for other BOOTP servers to respond as well; for this reason, it is important that the Cisco ICS 7750 be isolated from the network while initially running ICSConfig and during periods of troubleshooting.
3. The SPE builds configuration data for the booting ASI or MRP200 and stores the data in a configuration file in the TFTPPath folder. (The IOS image is made available for the TFTP server through the system software installation process, or through software upgrade from the Software Upgrade page on the ICS System Manager.)
4. The ASI or MRP200 card obtains the download information it needs from the BOOTP response (the card's IP address, subnet mask, gateway, TFTP server, and the name of the IOS image that should run on it).
5. The ASI or MRP200 card downloads and then starts the IOS image. Then it downloads its configuration file, putting the configuration file into its running configuration.
6. The ASI or MRP200 completes the boot sequence.
General ASI or MRP200 Boot Problems
If the ASI or MRP200 cards do not boot on the Cisco ICS 7750, you should first determine at what point the process is failing by watching the bootup sequence from the console connection on the SAP.
Use a HyperTerminal connection to the SAP to see whether the ASI or MRP200 is receiving the following:
•A valid IP address
•TFTP server
•IOS image name
•Configuration (.cfg file)
Troubleshooting the SPE Running System Manager
If the ASI or MRP200 is not receiving the required items—valid IP address, TFTP server, IOS image name, and configuration file—follow these guidelines to troubleshoot the problem, using a direct connection to the SPE or using a Terminal Services client connection:
•Examine the CNR logs on the SPE running System Manager. These logs are located at C:\Program Files\Network Registrar\logs. This should help you determine whether the ASI or MRP200 is receiving the required data to boot.
•Check to make sure that CNR is started on the SPE running System Manager by navigating to Start > Programs > Administrative Tools > Services > AIC Server Agent. AIC Server Agent must be running to serve IP addresses and reply to BOOTP requests from the ASI and/or MRP200.
•Check for DHCP request/response flow on the SPE running System Manager (view the C:\Program Files\Network Registrar\logs\name_dhcp_1_log). DHCP/BOOTP activity is logged to this file.
•If the ASI or MRP200 receives a response from the SPE, check to see whether its image exists in the TFTPPath folder on the SPE running System Manager (see the C:\Program Files\Cisco\TFTPPath).
If you see any conflicts or problems, you can use the information to work with a Cisco technical support representative to resolve the situation.
Best Practices for Using the IOS CLI
ICS System Manager is designed to communicate with, and to monitor the status of, all the components in the chassis. To enable ICS System Manager to perform these functions, a configuration program (ICSConfig) provides the capability to easily configure key system parameters, such as the IP addresses of system cards, passwords, destination for syslog messages, and Simple Network Management Protocol (SNMP) community strings.
To enable ICS System Manager to perform all of its functions as a system management tool, it is important that you use ICSConfig or ICS System Manager, as appropriate, rather than the IOS CLI, when you enter key system parameters.
With the exception of the procedures listed below, you can enter all IOS CLI commands that are available for use in any IOS software release that is intended for use on the Cisco ICS 7750.
You should always use ICSConfig for the following tasks:
•Passwords
–Changing the login password, which gives ICS System Manager continued Telnet access to system cards
–Changing the Windows 2000 administrator password, which grants those with administrator privileges continued access to SPE310s
–Changing the enable or secret password, which makes it possible for administrators to enter certain IOS commands
•Card configurations
–Assigning or changing the IP addresses or subnet mask of system cards
•SNMP settings
–Changing read-only and read/write SNMP community strings of the SNMP server
–Changing the server destination of SNMP traps
–Managing the SNMP server
•Logging
–Changing the syslog logging host
Note SNMP community strings and system passwords are case-sensitive.
The following list includes tasks that cannot be configured on the Cisco ICS 7750 using the IOS CLI under any circumstances:
•Shutting down an Ethernet interface
•Changing an Ethernet or VLAN interface
•Disabling Cisco Discovery Protocol (CDP) on an Ethernet or VLAN interface
The following list includes tasks that cannot be performed on the SPEs under any circumstances:
•Configuring Domain Name System (DNS) on SPEs
•Invoking the Cisco Network Registrar (CNR) dhcp.exe from c:\program files\network registrar\bin
Specific System and MRP-Related Bootup Problems
The following sections describe a few scenarios that can cause booting problems on non-Flash-bases MRP cards (ASI81, ASI160, or MRP200) and on Flash-based MRP cards (MRP300, MRP3-8FXS, or MRP3-16FXS) cards.
•Troubleshooting MRP200, ASI81, and ASI160 Bootup Problems
•Troubleshooting MRP300, MRP3-8FXS, and MRP3-16FXS Bootup Problems
Troubleshooting MRP200, ASI81, and ASI160 Bootup Problems
Table 3-10 lists symptoms of and possible solutions for problems with MRP200, ASI81, and ASI160 cards.
Table 3-10 ASI81, ASI160, and MRP200 Bootup Problems and Solutions
Problem
|
Symptom
|
Possible Cause
|
Solutions
|
ASI and/or MRP does not appear to be functional.
|
ASI and/or MRP does not display status as expected.
|
•Power problem.
•Card(s) improperly seated or chassis slot problem.
|
•Check your system power connections and power supply modules. (See "Power Subsystem" section.)
•If the STATUS LED remains off, verify that the card is seated properly in the chassis.
•If the power LED of other ASIs or MRPs is on, try inserting the card in a different slot in the chassis.
|
ASI and/or MRP does not send BOOTP requests.
|
ASI and/or MRP is stuck at the rommon 1> prompt.
|
Incorrect configuration register setting on the ASI or MRP.
|
Use confreg in ROMMON mode to change the configuration register setting to 0x2102, and reset the card:
1. rommon>confreg 0x2102
2. rommon>i (or reset)
|
ASI and/or MRP does not get valid TFTP server or IOS image.
|
The ASI and/or MRP receives the BOOTP response from the SPE but times out during the download process.
|
The IOS image has not been imported into the database, or an appropriate IOS image has not been delivered to the ASI and/or MRP card (see Table 3-8.
|
Check to make sure that the IOS image is in the C:\Program Files\Cisco\TFTPPath directory on the SPE running System Manager. Then follow these steps to use tftpdnld to download the ASI/MRP IOS image:
1. Display the current variables.
rommon>set
2. Set the IP address of the ASI/MRP.
rommon 2 >IP_ADDRESS=10.0.0.3
3. Set the subnet mask.
rommon 3 > IP_SUBNET_MASK=255.255.255.0
4. Set the IP address of the TFTP Server.
rommon 4 >TFTP_SERVER=10.0.0.1
5. Enter the filename to get.
rommon 5 > TFTP_FILE=ics7700-sv3y-mz.12.2(4)YH
6. rommon 6>sync
7. rommon 6 >tftpdnld
This process is a temporary workaround that allows you to manually download the IOS image for the MRP/ASI card. An IOS image must be in the TFTPPath directory on the SPE running System Manager for this process to work. If there is no IOS image in the TFTPPath directory on the SPE, try doing a search for the image name on the hard drive to determine if the IOS image is located in a different directory on the SPE. If it is located in a different directory, copy it to TFTPPath. If it is not located on the SPE, download a copy of the IOS image from the IOS Planner Upgrade page on CCO.
|
ICS System Manager fails to respond to ASI and/or MRP BOOTP requests.
|
ASI and/or MRP continually sends BOOTP requests but gets no response from the SPE, so the ASI/MRP card fails to boot.
|
ICSConfig data may be out of sync because of manual CLI changes (passwords, IP addresses, community strings), insufficient IP addresses in CNR scope, duplicate IP address assignment, or subnet addressing issues.
|
•Always configure IP addresses, community strings, and passwords through ICSConfig. See "Best Practices for Using the IOS CLI" section.
•Ensure that there are enough IP addresses defined in ICSConfig to allocate IP addresses to all the cards in the chassis.
•Ensure that there are no duplicate IP addresses in your network.
•Ensure that the AIC Server Agent Service (CNR) is running on the SPE.
•Ensure that the SPE and SSP are in the same subnet and within the address range allocated for the cards.
•Ensure that the date and time on the SPE are set correctly (otherwise, CNR may not start).
Note See "Troubleshooting the SPE Running System Manager" section for additional information.
|
ICS System Manager fails to respond to ASI and/or MRP BOOTP requests.
|
ASI and/or MRP continually sends BOOTP requests but gets no response.
|
Problem with Ethernet connectivity
|
•If the problem is at the physical layer:
a. Reseat the ASI or MRP in the chassis slot to reset the card.
b. Move the ASI or MRP card to another slot in the chassis to see if it will boot up.
•Verify that the IP address of the ASI or MRP is unique and is not assigned to another device in the network.
•Contact your technical support representative for assistance.
|
ICS System Manager fails to respond to ASI and/or MRP BOOTP requests.
|
ASI and/or MRP continually sends BOOTP requests but gets no response.
|
SPE running System Manager has failed to boot.
|
•Wait for the SPE to come up, if the system was just started.
•Shut down and restart the SPE, using ICS System Manager.
•Remove and reinsert the SPE in the slot, or move it to another slot in the chassis. See "Best Practices for Reseating Cards in the Chassis" section.
•Contact your technical support representative.
|
Troubleshooting MRP300, MRP3-8FXS, and MRP3-16FXS Bootup Problems
Table 3-11 lists symptoms of and possible solutions for problems with MRP300, MRP3-8FXS, and MRP3-16FXS cards.
Note When a Flash-based MRP card is moved from one chassis to another, the startup-config must be erased so that the Flash-based card can obtain its new IP address from the ICS System Manager in the new chassis. Failure to erase the startup-config will result in the inability of ICS System Manager to discover the card.
Table 3-11 MRP300, MRP3-8FXS, and MRP3-16FXS Bootup Problems and Solutions
Problem
|
Symptom
|
Possible Cause
|
Solutions
|
MRP does not appear to be functional.
|
MRP does not display status as expected.
|
•Power problem.
•Card(s) improperly seated or chassis slot problem.
|
•Check your system power connections and power supply modules. (See "Power Subsystem" section.)
•If the STATUS LED remains off, verify that the card is seated properly in the chassis.
•If the power LED of another MRP cards is on, try inserting the card in a different slot in the chassis.
|
MRP fails to boot from Flash memory or manual boot.
|
MRP is stuck at the rommon 1> prompt, and a manual boot flash: results in an error. See Example 3-1 for an example of a manual boot flash: with a valid image.
|
Image in Flash memory is not valid or is corrupted.
|
If you see a manual boot flash error, such as the following:
device does not contain a valid
magic number
boot: cannot open "flash:"
boot: cannot determine first file
name on device "flash:"
use tftpdnld in ROMMON mode to copy the valid image to Flash memory, and reset the card. See Table 3-10 for tftpdnld instructions.
|
MRP does not boot from Flash memory.
|
MRP is stuck at the rommon 1> prompt.
|
Incorrect configuration register setting on the MRP.
|
•Use the confreg command in ROMMON mode to change the configuration register setting to 0x2102, and reset the card:
a. rommon>confreg 0x2102
b. rommon>i
•Or, enter boot flash: at the command prompt, and then change the config-reg setting in IOS CLI after the MRP boots:
a. MRP#config t
b. MRP# (config)# config-register 0x2102
|
ICS System Manager fails to recognize the MRP.
|
MRP cannot be discovered by System Manager and cannot boot.
|
•Mismatched SNMP community strings result in authentication failure.
•Mismatched startup-config results in conflict with IP address or with log in and enable passwords.
•MRP configuration is done outside of ICSConfig.
|
•If the MRP had a startup-config in NVRAM, check to make sure that its SNMP community strings match those entered in ICSConfig.
•Check that the IP address of the MRP is in the same subnet as the chassis.
•Check that the login and enable passwords on the MRP are the same as those configured in ICSConfig.
•Use ICSConfig to configure the MRP parameters; do not use the IOS CLI.
•Erase the startup-config before installing an MRP into a new chassis (if there is a startup-config, System Manager cannot deliver a configuration to the MRP).
•Contact your technical support representative for assistance with problems that you cannot solve.
|
Example 3-1 Manual boot flash: with a Valid Image in Flash Memory
A manual boot flash: looks like the following example when there is a valid image in Flash memory:
Enter dir flash: to see the file:
File size Checksum File name
9362188 bytes (0x8edb0c) 0xae67 ics7700-bnr2sv3y-mz.122-4.XL1
program load complete, entry point: 0x80008000, size: 0x8ed9f0
Self decompressing the image :
######################################################################
####
Best Practices for Reseating Cards in the Chassis
You may need to remove and reinsert a card in the Cisco ICS 7750 chassis. Follow these steps to safely reseat chassis cards:
Step 1 If you are reseating an SPE310 card, make sure you shut down the card before you remove it. On the target SPE, navigate to Start > Shut Down. The STATUS LED blinks and then turns off.
Step 2 Other cards may be removed without shutting them down. Disengage the card from the chassis backplane by completing the following steps:
a. Completely loosen the card captive screws.
b. Press the upper and lower ejector levers outward at the same time to disengage the card from the backplane.
Caution Always use the ejector levers to disengage or seat cards. Failure to use the levers can cause erroneous system error messages that indicate a card failure. Do not use the ejector levers to lift or support the weight of the card.
c. Grasp the ejector levers, and gently pull the card about one inch out of the chassis.
Caution When installing cards, be sure to apply equal pressure to the top and bottom of the card. You can damage the card and the chassis backplane if you apply too much pressure to the bottom or the top of the card.
Step 3 Gently slide the card back into the chassis until you feel resistance. Because there are grounding clips near the front and rear of the card guides, you might need to increase the amount of force that you use to get the card past the grounding clips. If you encounter extreme resistance, pull the card out slightly, and push it back in again.
Step 4 Press the upper and lower ejection levers inward at the same time until they lock into their slots. This firmly seats the card into the chassis.
The card will reboot and the system will complete the upload if you restarted an ASI or MRP200 card.
SSP Card Boot Problems
If you replace the SSP card with a different SSP card, you might notice the following problems with the replacement SSP card:
•It might fail to log the SPE running System Manager as its logging host.
•It might fail to log the SPE running System Manager as its SNMP server.
•Its console password, vty password, read-write Community String, and Trap Community String might not be set.
To solve this problem, complete the following steps:
Step 1 Open a HyperTerminal session with the SAP card.
Step 2 Log in as an administrator (user ID administrator), and enter your password (the default is changeme).
Step 3 Press Ctrl-backslash (\), and use the SAP card menu to switch to the SPE card running System Manager.
Step 4 Log in as an administrator (user ID administrator), and enter your password (the default is changeme).
Step 5 Enter the following command:
Step 6 Press Ctrl-backslash (\), and use the SAP card menu to switch to the SSP card.
Step 7 Enter privileged EXEC mode by entering the following command:
Step 8 Enter your enable password.
Step 9 Erase the SSP card startup configuration by entering the following command:
Note Make sure that you answer No when prompted to save the current configuration.
Step 10 Restart the SSP card by entering the following command:
The SSP card reboots. When the SSP card has finished booting, its STATUS LED will turn green, and its ALARM LED should go off.
Step 11 Press Ctrl-backslash (\), and use the SAP card menu to switch to the SPE running System Manager.
Step 12 Enter the following command:
Useful MRP Troubleshooting Commands
The following are some useful troubleshooting commands for the MRP and ASI cards. Refer to Troubleshoot & Debug VoIP Calls - the Basics for additional information and examples of the various command outputs.
•show controller t1—Displays information about the T1 links to verify the status of the digital T1 connection between the MRP and the switch (CO or PBX). Shows whether the link is up or down and whether it is functioning properly.
•show voice-port—Displays configuration information about a specific voice port, such as the port state and the parameters configured on the voice port of the voice interface card (VIC).
•show running-config—Shows running-configuration settings.
•show version—Displays configuration of the system hardware, the software version, the names and sources of configuration files, the boot images, the amount of that time the MRP has been up and running, how the MRP was restarted, the name of the MRP image file, a log of how the system was last booted (both as a result of normal system startup and of system error), and the configuration register contents displayed in hexadecimal notation.
•show stacks—Displays the stack usage of processes and interrupt routines, showing the reason for the last system reboot.
•show voice dsp—Shows the current status of all DSP voice channels to verify available MIPS.
•sh voice call summ—On the MRP, displays the signaling channel state on a per time slots basis.
•show call active voice—Shows the active call table.
•show dial-peer voice—Displays configuration information for dial peers.
•show interface—Displays statistics for all interfaces configured on the MRP.
•show ip interface—Displays the usability status of interfaces configured for IP.
