Cisco ICS 7750 Troubleshooting Guide, 2.6.0
Solving Hardware Problems
Downloads: This chapterpdf (PDF - 646.0KB) The complete bookPDF (PDF - 3.39MB) | Feedback

Solving Hardware Problems

Table Of Contents

Solving Hardware Problems

Performing Diagnostics

POST Sequence of Events

Cisco ICS 7750 Card LEDs

System Card LED and Shutdown Buttons

LED Behavior During System Bootup

LED Behavior After System Bootup

LED Behavior After Shutdown

Diagnostic Commands

Using the ping Command

Using the traceroute Command

Power Subsystem

System Alarm Processor

Troubleshooting the SAP

Power Supply Modules

Troubleshooting Power Supply Modules

Fans

Checking Fan Speed and Operation

Troubleshooting Fans

System Processing Engine

Troubleshooting SPEs

Routing and Switching Subsystem

Troubleshooting the Routing and Switching Subsystem

How to Reset Chassis Slot Cards Through The SSP

How to Remotely Reset Chassis Slot Cards Through The SAP

Troubleshooting ASIs, MRPs, and WICs

Troubleshooting the SSP

Troubleshooting Cisco ICS 7750 Booting Problems

Monitoring the Cisco ICS 7750 Boot Process

MRP300, MRP3-8FXS, MRP3-16FXS, and MRP3-8FXOM1 Boot Sequence

Initial Bootup and Discovery

ASI and MRP200 Boot Sequence

General ASI or MRP200 Boot Problems

Troubleshooting the SPE Running System Manager

Best Practices for Using the Cisco IOS CLI

Specific System and MRP-Related Bootup Problems

Troubleshooting MRP200, ASI81, and ASI160 Bootup Problems

Troubleshooting Flash-Based MRP Bootup Problems

Best Practices for Reseating Cards in the Chassis

Replacing an MRP200 or ASI Card with a Flash-Based MRP Card

SSP Card Boot Problems

Useful MRP Troubleshooting Commands

Support for VLAN Functionality

Enabling VLAN Functionality

Configuring VLAN Support on the Flash-Based MRP Card

Configuring VLAN Support on the SSP

Backing Up and Restoring the VLAN Configuration on the SSP

Troubleshooting VLANs


Solving Hardware Problems


This chapter explains how to solve hardware problems with the Cisco Integrated Communications System (ICS) 7750. The chapter is organized as follows:

Performing Diagnostics

Power Subsystem

Fans

System Processing Engine

Routing and Switching Subsystem

Troubleshooting Cisco ICS 7750 Booting Problems

Useful MRP Troubleshooting Commands

Support for VLAN Functionality


Note For a description of the features, modifications, and caveats for the
Cisco Integrated Communications System 7750 (Cisco ICS 7750) release 2.6.0, refer to the Release Notes for System Software Release 2.6.0 on the
Cisco ICS 7750.



Tip For the latest information about the configuration modifications that affect the number of supported voice channels on the MRP300 cards, and the caveats associated with that support, refer to the Release Notes for Cisco IOS Release 12.2(8)YN on the Cisco ICS 7750.


Performing Diagnostics

You can use a card's power-on self-test (POST) or other diagnostics to locate hardware faults. In some cases, you can run diagnostics on an individual card while the rest of the system continues to operate because only the card being tested is out of service. In other cases, you must take the system off line to run diagnostics; for instance, testing the system processing engine (SPE) running System Manager may take the system off line if there is no secondary SPE310 running Cisco CallManager.

You can run diagnostics remotely through a Telnet or modem connection, locally from a console connected to the console port on the system alarm processor (SAP), or directly through a monitor, mouse, and keyboard attached to the SPE310 card.

POST diagnostics are available for the following cards:

SPEs, including disk drives

Multiservice route processor (MRP) and analog station interface (ASI) cards, including installed WAN interface cards (WICs), voice interface cards (VICs), and voice WAN interface cards (VWICS)

SAP

System switch processor (SSP)

POST diagnostics can be initiated either by unseating and reseating a card or by using a hardware reset command from the SSP. For instance, from enable mode on the SSP, enter the following command:

SSP#hw-module chassis slot 5 restart hold 5

Entering this command causes the card in slot 5 to restart after holding its power off for 5 seconds.

The set request command used in enable mode (after invoking the SLPENABLEPASSWORD command) on the SAP can also initiate POST. Viewing POST information through the console port on the SAP provides significantly more information than any other method, such as Telnet. From the SAP menu, select the card that you want to check immediately after its POST test has been initiated.


Caution There are only one SAP card and one SSP card in a chassis. If you take the SAP off line, the system's ability to detect alarms associated with the operating environment, fans, and power supply modules will be degraded until the SAP is returned to an on line status. If you take the SSP off line, the system loses LAN connectivity, and calls being made to or from Cisco IP Phones that are routed through the Cisco ICS 7750 will be disconnected until the SSP is brought back on line.


Note Depending on your system configuration, taking SPEs, ASIs, or MRPs off line can adversely affect users connected to the system. For example, if Cisco CallManager is running on only one SPE, taking that SPE off line disconnects calls to or from the PSTN and prevents the system from processing further PSTN traffic until that SPE is on line. Similarly, taking an ASI or MRP off line that is in the process of routing voice or WAN traffic will prevent that traffic from reaching its destination.


POST Sequence of Events

During POST, some or all of the following events can occur:

A diagnostic image is downloaded to the card to be tested. The downloading process erases the Cisco IOS software from the card memory and installs the diagnostic software image.

Normal card operation is suspended; the card being tested no longer sends or receives network traffic.

As testing proceeds, the console displays diagnostic messages.

A card pass or fail message is displayed on the console when testing ends.

If the card passes the diagnostic tests, the Cisco IOS software is reloaded into the card's memory, and the card resumes normal operations.


Note If a card fails diagnostics, see the troubleshooting help in this chapter.


Cisco ICS 7750 Card LEDs

The Cisco ICS 7750 is a chassis-based system that consolidates multiple cards with different functions (SPEs, MRPs, SSP, and SAP) and power supply modules into a single system unit. Each system card has LEDs that provide visual indications about the status of the card.

The following sections describe the LEDs and LED behavior on the system cards:

System Card LED and Shutdown Buttons

LED Behavior During System Bootup

LED Behavior After System Bootup

LED Behavior After Shutdown

System Card LED and Shutdown Buttons

The LEDs that are common to each of the Cisco ICS 7750 cards are the alarm and status LEDs. Some cards have additional special-purpose LED(s), depending on the card's function. Each card also has one shutdown button.

Table 3-1 summarizes the LEDs that are available on each system card.

Table 3-1 Cisco ICS 7750 Card LEDs

LED
LED Color
SPE
MRP
SSP 1
SAP 2

Alarm

Yellow, amber, red

X

X

X

X

Status

Green

X

X

X

X

Slot

Green

 

X3

   

Power4

Green

     

X

Fan

Yellow

     

X

Temperature

Yellow

     

X

1 The SSP also contains two port status LEDs. These LEDs are controlled by the Cisco IOS software running on the SSP. Refer to the Catalyst 2900 Series XL Installation and Configuration Guide for a description of the port status LEDs.

2 The SAP has two power LEDs, one for each power supply module.

3 The MRP has two slot LEDs, one for each VIC, WIC, or VWIC slot. Exceptions are the MRP3-16FXS and ASI160 cards, which do not contain any slot LED.

4 The system power supply module has three LEDs—AC OK, DC OK, and Over Temp.


LED Behavior During System Bootup

During system bootup, the LEDs on each system card are controlled by the card on which they reside.

Table 3-2 shows the color and status of each LED on the MRP and SSP cards during the cards' various states from system power up to system bootup.

Table 3-2 LED States on MRP and SSP Cards

LED
Rommon Mode
During Cisco IOS Bootup
After Cisco IOS Bootup

Alarm LED (yellow)

Steady

Steady

Off, controlled by IOS1

Status LED (green)

Blinking

Blinking

Steady, controlled by IOS

Slot LEDs2 (green)

Steady if the slot is populated

Steady if the slot is populated

Steady if the slot is populated

1 This is a new behavior beginning with the Cisco IOS software packaged with ICS System Software release 2.4.0 (MRP Cisco IOS software version 12.2(4)YH and SSP Cisco IOS software version 12.0(5)WC5). With the Cisco IOS software versions packaged before ICS System Software release 2.4.0, the LED is in a steady state after Cisco IOS bootup until ICS System Manager turns it off.

2 The SSP does not have slot LEDs.


Table 3-3 shows the color and status of each LED on the SPE card during its various states from system power up to system bootup.

Table 3-3 LED States on SPE Card

LED
In BIOS
During System Bootup
After System Bootup

Alarm LED (yellow)

Steady

Steady

Off, controlled by FMM

Status LED (green)

Blinking

Blinking

Steady, controlled by FMM


Table 3-4 shows the color and status of each LED on the SAP card during its various states from system power up to system bootup.

Table 3-4 LED States on SAP Card

LED
During Firmware Bootup
After Firmware Bootup

Alarm LED (yellow)

Steady

Off, controlled by SAP firmware

Status LED (green)

Blinking

Steady

Power LEDs

Steady

Steady if it is normal

Blinking if it fails; controlled by SAP firmware

Fan LEDs

Off

Off if it is normal

Blinking if it fails; controlled by SAP firmware

Temperature LED

Off

Off if it is normal

Blinking if it fails; controlled by SAP firmware


LED Behavior After System Bootup

Once the Cisco ICS 7750 boots up, each card in the chassis transfers the alarm LED control functionality to ICS System Manager and FMM (on the SPE running System Manager). ICS System Manager and FMM determine the appropriate behavior of the alarm LED on each card, based on the traps or events sent by the cards in the chassis. The only colors used by ICS System Manager and FMM are yellow and amber, depending on the severity of the failure.

After system bootup, the status LED remains in its bootup state, as described in Table 3-2, Table 3-3, and Table 3-4, unless the shutdown button is pressed.

Slot LEDs on the MRP also remain in their bootup state, as described in Table 3-2.

Power, fan, and temperature LEDs on the SAP are controlled by the SAP firmware. Status changes are communicated to ICS System Manager and FMM.

LED Behavior After Shutdown

When you press the shutdown button on a system card, the card completely shuts down. The shutdown sequence is as follows:

The status LED begins to blink immediately and keeps blinking during the shutdown preparation process; on the MRP, SSP, and SAP, the status LED blinks for 15 seconds.

On the SPE, the operating system shuts down first. Then the card powers down.

On the MRP, SSP, and SAP, card functionality completely stops.

The software or firmware on a system card turns off the status LED on the card after the shutdown preparation period.


Note Refer to the Cisco Interface Cards Installation Guide for information on LED behavior for VICs, WICs, or VWICs.


Diagnostic Commands

Many of the cards in the Cisco ICS 7750 support commands that can help you better understand what is happening in your internetwork. This section provides information on the following topics:

Using the ping Command

Using the traceroute Command

Using the ping Command

The ping command is a test that is available on Cisco internetworking devices and on many host systems. In TCP/IP, this diagnostic tool is also known as an Internet Control Message Protocol (ICMP) echo request.

To check host reachability and network connectivity, use the Cisco IOS ping command. This command enables you to confirm basic network connectivity on many kinds of networks.

For IP connections, in EXEC mode, the ping command sends ICMP echo messages. If a station receives an ICMP echo message, it sends an ICMP echo reply message to the source.

In privileged EXEC mode, the ping command permits you to specify the supported IP header options. With these options specified, the target device can perform a more extensive range of test options.

It is a good idea to use the ping command when the network is functioning properly to see how the command works under normal conditions, so that you have something to compare to when troubleshooting.

Using the traceroute Command

In EXEC mode, the Cisco IOS traceroute command discovers the routes that packets follow to their destinations. In privileged EXEC mode, traceroute (also referred to as an extended trace) enables you to specify the supported IP header options, allowing the target device to perform a more extensive range of testing.

The traceroute command uses the error message generated by devices (routers or cards) when a datagram exceeds its time-to-live (TTL) value. Probe datagrams are sent initially with a TTL value of 1, which causes the first device to discard the probe datagrams and send back "time exceeded" error messages.

The traceroute command then sends several probes and displays the round-trip time for each. After every third probe, the TTL is increased by 1.

Each outgoing packet can result in one of two error messages. A "time exceeded" error message indicates that an intermediate device has seen and discarded the probe. A "port unreachable" error message indicates that the destination node has received the probe and discarded it because it could not deliver the packet to an application.

If the timer goes off before a response comes in, an asterisk (*) is displayed.

The trace operation terminates when the destination responds, when the maximum TTL is exceeded, or when you interrupt the trace with the escape sequence.

As with ping, it is a good idea to use the traceroute command when the network is functioning properly to see how the command works under normal conditions, so that you have something to compare to when troubleshooting.

For more information on ways to use the traceroute EXEC and privileged EXEC commands, refer to the "Troubleshooting and Fault Management Commands" chapter in the Cisco IOS Configuration Fundamentals Command Reference.

Power Subsystem

This section explains how to determine and resolve problems associated with the power subsystem, which includes the following components:

System Alarm Processor

Power Supply Modules


Note The SAP monitors the physical state of the system, including the availability of system power and cooling air.


System Alarm Processor

This section explains how to isolate problems associated with the SAP. Figure 3-1 shows the front panel of the SAP card.

Figure 3-1 SAP Card

Troubleshooting the SAP

Complete the following steps to isolate SAP problems:


Step 1 Check the system LEDs. If no SAP LEDs are on, ensure that the SAP is properly seated in the chassis and that the power supply modules and fans are functioning properly. (See the "Power Supply Modules" section or the "Fans" section for additional information.)

Step 2 Check all the SAP LEDs:

If the ALARM LED is on (amber or yellow), the system has most likely found a problem outside the SAP. Check system status for additional indications that can help you isolate the problem.

Verify that the STATUS LED is on (green), indicating that the system software has initialized successfully and that the system is operational.

If the STATUS LED is off, the SAP might have pulled away from the backplane. If the SAP is not seated properly, it may cause the system to hang.

If one or both of the PWR LEDs are off when they should be on (green), see the "Power Supply Modules" section.

If the FAN LED is off or the TEMP LED is on (yellow), see the "Fans" section.


Power Supply Modules

This section explains how to isolate power supply module problems. Figure 3-2 shows the front panel of the power supply module.

Figure 3-2 Power Supply Module


Note The Cisco ICS 7750 supports a maximum of five SPEs in a single chassis, of which one SPE must be running ICS System Manager. One power supply will fully support this configuration. A second power supply may be added as an option for the purpose of providing redundancy.


Troubleshooting Power Supply Modules

Follow these steps to isolate power supply module problems; refer to Table 3-5 if necessary:


Step 1 Verify that the fans are operating and that the STATUS LED of at least one SPE is on (green). Look at the SAP card: If the FAN and STATUS LEDs are on (green) but the power supply AC OK LED is off, there is probably a faulty power supply module LED.

Step 2 Verify that the system power switch is set to the on position.

Step 3 Verify that the power source, power cable, and at least one power supply module are functioning correctly. Swap parts to determine whether one of the components is faulty. (Refer to the Cisco ICS 7750 Installation and Configuration Guide.)


Table 3-5 lists symptoms of and possible solutions for power problems.

Table 3-5 Power Supply Module Problems and Solutions 

Symptom
Power Supply Module LED Status 1 Change
Possible
Cause
Solutions

The system shuts down after being on for a short time.

AC OK: on

OVERTEMP: off

DC OK: off

Power or cooling problem

Verify that the area in which the system is installed meets the environmental requirements in the Cisco ICS 7750 Installation and Configuration Guide and in the "Site Requirements" section in the Regulatory Compliance and Safety document that came with your system.

Verify that nothing is blocking the air intake or exhaust. (See the "Fans" section.)

The power supply module has failed. Install a new power supply module.

The system attempts to boot, but all LEDs remain off.

AC OK: on

OVERTEMP: off

DC OK: off

Power problem

Ensure that the power on/off switch is in the on position.

The power supply module has failed. Install a new power supply module.

The power supply module is not operating within its normal operating tolerances (in a single power supply module configuration).

AC OK: on

OVERTEMP: off

DC OK: off

Defective power supply module

If the system is still operating, insert a new power supply module into the unused power supply slot, and turn it on. If the newly installed power supply module is functioning properly, remove the defective power supply module in the other slot.

If the system is not operating, replace the defective power supply module.

The power supply module has exceeded its maximum operating temperature and is about to shut down (in a single power supply module configuration).

AC OK: on

OVERTEMP: on

DC OK: off

Cooling problem

Correct any problems associated with the fans. (See the "Fans" section.) The power supply module will restart if the high-temperature condition is caused by a problem with the fans.

Replace the power supply module.

A power supply module is producing out-of-tolerance power (in a dual power supply module configuration).

AC OK: on

OVERTEMP: off

DC OK: off

Defective power supply module

Replace the defective power supply.

A power supply module has exceeded its maximum operating temperature and is about to shut down (in a dual power supply module configuration).

AC OK: on

OVERTEMP: on

DC OK: off

Cooling or power supply module problem

Correct any problems associated with the fans. (See the "Fans" section.) The power supply module will restart if the high-temperature condition is caused by a problem with the fans.

Replace the defective power supply module.

1 In dual power supply module configuration, the LED Status column shows the condition of the power supply module that is reporting an error.


Fans

This section explains how to isolate problems associated with the fans, which are located in the fan tray at the bottom of the system chassis.

Checking Fan Speed and Operation

The SAP on the Cisco ICS 7750 monitors the operational status of each component in the system.

Working with the Fault Management Module (FMM) and ICS System Manager, the SAP can remotely alert you of environmental, functional, or operational problems detected within the system. Access is available through the console port on the front of the card.

One element of the Cisco ICS 7750 that can affect performance is the speed of its fans. The fans run at different speeds, depending on the system's operation:

At power up, the fans should run at high speed (high noise level) for up to 10 seconds. After that time, they should go into normal operation mode.

At normal operation, the fans should be spinning at a low speed (low noise level).

During a software upgrade, the fans should run quietly.

Following a software upgrade, the fans should run as they do during the power-up process, spinning fast for about 10 seconds and then slowing down to normal operation.

Follow these steps to check the fans for proper operation:


Step 1 On a PC, open a HyperTerminal session with the SAP card.

Step 2 At the prompt, enter enable mode by typing slpenablepassword, followed by the password for the card.

Step 3 Enter the following command:

get fan-speed :<fan#> 

where fan# is the number of the fan that you wish to check.

The following is an example of output that shows fans running at normal operation:

AlarmCard>get fan-speed :1
INF FAN-SPEED : 1, 2250
AlarmCard>get fan-speed :2
INF FAN-SPEED : 2, 2250
AlarmCard>get fan-speed :3
INF FAN-SPEED : 3, 2250
AlarmCard>get fan-speed :4
INF FAN-SPEED : 4, 2280


Troubleshooting Fans

Follow these steps to isolate fan problems. Refer to Table 3-6 if necessary.


Step 1 To verify that the fans are operating, place your hand near the air exhaust on the back of the chassis, and check for a warm flow of air.

If the fans are not operating, there might be a problem with the fan tray or with the power source.

If the fans are not operating and if the FAN LED on the SAP is off, ensure that the fan tray is seated properly.

If the LEDs on the SAP are off, ensure that the SAP is also seated properly.

If there are no LEDs illuminated on the SAP, disengage the ejector levers to unseat the card, and then reseat it to ensure that it is seated properly in the chassis.

Step 2 If the system and the fans start up but shut down after about 2 minutes, one or more fans might have failed.

If one or more fans fail to operate or if one or more fans have tachometer speeds that are not within the acceptable range, you might need to replace the fan tray.

Step 3 If you see the following message at startup, the system has detected an overtemperature condition or out-of-tolerance power inside the chassis:

The system is overheated 

This message might indicate an environmental problem; ensure that there is adequate ventilation and unrestricted air flow in the location where the Cisco ICS 7750 is installed.

This message might also indicate a malfunctioning component or a faulty temperature sensor within the Cisco ICS 7750 chassis.

Step 4 Ensure that heated exhaust air from other equipment is not entering the inlet vents and that there is sufficient clearance around the chassis to allow cooling air to flow.

Step 5 Ensure that the SAP is properly installed. If the SAP is not properly installed, the fans will not operate.


Table 3-6 lists symptoms of and possible solutions for fan problems.

Table 3-6 Fan Problems and Solutions 

Syslog Message
Symptom
SAP LED
Status Change
Possible
Cause
Solutions
The  
system is 
overheated

The system is overheated and will shut down 10 seconds after sending a warning message.

ALARM: on (amber)

TEMP: on

High ambient air temperature or air intake or exhaust blockage

Verify that the room temperature in which the system is located is less than 104°F (40°C).

Verify that there are no air intake blockages at the front of the chassis.

Verify that there are no air exhaust blockages at the back of the chassis.

Verify that there are no internal system blockages.

Card x is 
overheated

A system card (other than the SAP) is overheated.

ALARM: on (amber)

TEMP: on

High ambient air temperature, air intake or
exhaust blockage, or card failure

Verify that the room temperature in which the system is located is less than 104°F (40°C).

Verify that there are no air intake blockages at the front of the chassis.

Verify that there are no air exhaust blockages at the back of the chassis.

Verify that there are no internal system blockages.

Replace the card.

The ambient 
temp x 
exceeded 
recommended 
value

The temperature inside the chassis is over 104°F (40°C).

ALARM: on (amber)

TEMP: on

High ambient air temperature or air intake or exhaust blockages

Verify that the room temperature in which the system is located is less than 104°F (40°C).

Verify that there are no air intake blockages at the front of the chassis.

Verify that there are no air exhaust blockages at the back of the chassis.

Verify that there are no internal system blockages.

Fan x 
failure

A fan tachometer reading drops below the designated threshold, or a fan stops completely.

ALARM: on (amber)

FAN: on

TEMP: on

Improperly seated or defective fan tray

Remove, inspect, and reinsert the fan tray.

If the fans are still not operating, install a new fan tray.

Refer to the
Cisco ICS 7750 Installation and Configuration Guide.

Fan tray 
absent

All fan tachometer readings fall to zero.

ALARM: on (amber)

FAN: on

TEMP: on

Missing or improperly seated fan tray

Remove, inspect, and reinsert the fan tray.

If the fans are still not operating, install a new fan tray.

If the fans are still not operating, install a new SAP.

Refer to the
Cisco ICS 7750 Installation and Configuration Guide.


System Processing Engine

This section explains how to isolate problems associated with the SPE. Figure 3-3 shows the front panel of the SPE310.

Figure 3-3 SPE Card


Note Refer to the Installing Memory, PVDM, and VPN Modules in ASI Cards, MRP Cards, and SPE Cards in the Cisco ICS 7750 documentation for a detailed view of the SPE310 card.



Note The Cisco ICS 7750 supports a maximum of five SPEs in a single chassis.


Troubleshooting SPEs

Follow these steps to isolate problems with the SPE:


Step 1 Check the system LEDs. If no SPE LEDs are on, ensure that the SAP, power supply modules and fans are functioning properly. (See the "Power Supply Modules" section and the "Fans" section for additional information.)

Step 2 Check all the SPE LEDs.

If any ALARM LED is on (amber or yellow), except during system boot or POST, the system has detected a hardware failure in that SPE. Contact a technical support representative for instructions.

Verify that the STATUS LED on all SPEs is on (green), indicating that the system software has initialized successfully and that the system is operational.

If the STATUS LED on an SPE is off, that SPE might have become disengaged from the backplane or might be in a shutdown state. If an SPE is not seated properly, it may hang the system.


Table 3-7 lists symptoms of and possible solutions for hardware-related problems on the SPE.

Table 3-7 SPE Problems and Solutions 

Symptom
SAP LED Status Change
SPE LED Status Change
Possible
Cause
Solutions

The system cannot read or write to an SPE hard disk.

ALARM: on (amber)

None

Inadequate security privileges (read/write permission) or hard disk errors

The SPE may be out of disk space. Use Windows Explorer to determine how much space is left on the hard disk and, if necessary, move data to another server.

You may not have the necessary authorization. If you are attempting to write to the disk, use ICSConfig to determine whether you can access the system as an administrator.

The hard disk may have errors. Close all applications running on the disk, and use the Windows 2000 Check Disk utility to scan for and attempt recovery of bad sectors.

Note The disk is not available to run other tasks while Check Disk is running.

Replace the SPE.

SPE memory parity errors have occurred.

ALARM: on (amber)

ALARM: on (amber)

Defective memory or hard disk

Replace the SPE.

SPE failed to boot.

ALARM: on (yellow)

ALARM: on (yellow)

STATUS: off

Defective memory module and defective hard disk in SPE

Replace the SPE.


Routing and Switching Subsystem

This section explains how to isolate problems associated with the routing and switching subsystem, which includes the following components:

One or more ASI cards or MRP cards. Each ASI81, MRP3-8FXS, MRP3-8FXOM1, and MRP card can include different combinations of WICs, VICs, and VWICs.

SSP.


Note For problems associated with VICs and trunks or for other voice-related issues, see Chapter 9, "Solving Voice Problems." For instructions on how to solve problems concerning Catalyst 3524-PWR XL switches, refer to the documentation that came with the Catalyst 3524-PWR XL switches.


Figure 3-4 shows the front panels of an MRP300 card (on the left) and an SSP card (on the right).


Note Refer to the Installing Memory, PVDM, and VPN Modules in ASI Cards, MRP Cards, and SPE Cards in the Cisco ICS 7750 documentation for a detailed view of the MRP200, ASI81, ASI160, MRP300, MRP3-8FXS, MRP3-16FXS, and MRP3-8FXOM1 cards.



Tip For the latest information about the configuration modifications that affect the number of supported voice channels on the MRP300 cards, and the caveats associated with that support, refer to the Release Notes for Cisco IOS Release 12.2(8)YN on the Cisco ICS 7750.


Figure 3-4 Routing and Switching Subsystem (MRP300 and SSP)

Troubleshooting the Routing and Switching Subsystem

The problems listed below could require replacement of an ASI, MRP, or SSP:

An ASI or MRP fails, even when moved to another slot. (If the card fails in one slot but operates properly in another, there might be a problem with the backplane or SSP.)

Diagnostics that loop data through the SSP fail on two or more Catalyst 3524-PWR XL switches that are connected to that SSP.

Traffic is not passing through the system, but the ASIs, the MRPs and the SSP all appear to be functioning normally.

The system has data transmission problems that do not go away when you replace the card that appears to be failing, or the system has problems that occur in several cards simultaneously. (Problems of this type can also indicate a faulty backplane.)

You cannot fully insert an ASI, MRP, or SSP into its slot. This problem most likely indicates damage to the connectors on either the card or the backplane. Inspect all the connectors. If you find damage, replace the card or the chassis.

How to Reset Chassis Slot Cards Through The SSP

This procedure can be useful when you are remotely troubleshooting a Cisco ICS 7750, or when you mistakenly shut down the SPE instead of restarting it. Generally, when restarts of an SPE fail, you need to unseat the SPE and then reseat it in order to get it to boot. To perform these actions, you need to have hands-on access to the Cisco ICS 7750. If the Cisco ICS 7750 is remotely located, you will need to obtain on-site support in order to proceed.

To restart Cisco ICS 7750 cards using the SSP, start at the Cisco IOS command-line interface (CLI) for the SSP. This can be through Telnet or through the console of the Cisco ICS 7750, using a 2511 access server if you are not locally connected to the system.

At the Cisco IOS CLI prompt, issue the following command:

Switch#hw-module chassis slot n restart

where n represents the slot number in which the line card to restart is installed.

Options for this command include a delay before the line card is restarted, as well as a hold in the unpowered state between the bring-down and bring-up phases.

For example:

Switch#hw-module chassis slot 1 restart ?
  delay  Seconds to delay before asserting reset
  hold   Seconds to assert reset
  <cr>

Switch#hw-module chassis slot 1 restart delay ?
  <0-2147483647>  Delay in seconds
  <cr>

Switch#hw-module chassis slot 8 restart hold ?
  <0-2147483647>  Seconds to assert reset

How to Remotely Reset Chassis Slot Cards Through The SAP

You can use the SSP for remote troubleshooting purposes, as described in the "How to Reset Chassis Slot Cards Through The SSP" section. You can also access the SAP menu without being directly connected to the Cisco ICS 7750 chassis. Either method may be used, depending on user preference, the state of the system, and the task to be performed.

Follow these steps to set up remote troubleshooting capability, using the SAP console:


Step 1 At the Cisco ICS 7750 chassis, plug the blue RJ-45 console cable into the console port on the SAP.

Step 2 Plug the other end of the console cable (DB-9 connector) into Com port 1 or Com port 2 (if you are using voice mail, plug this end into Com port 2).

Step 3 Access the SPE running System Manager, using a Terminal Services Client connection.

Step 4 Open the HyperTerminal application on the SPE, using Com 1 or Com 2 (make sure to set up your HyperTerminal connection to recognize this port).

You will be presented with the SAP prompt as if you were directly connected to the Cisco ICS 7750.

Step 5 From the SAP menu, enter the SLPENABLEPASSWORD command to access the administrator functions on the SAP.

Step 6 Enter the administrator password.

Step 7 At the prompt, enter the set reset command, as follows:

AlarmCard>set reset 5,1,1

In this example, the SAP will reset (re-enable) the card in slot 5 after holding for 1 second and waiting for 1 second

where:

5 represents the slot number of the card to be reset.

1 represents the time_to_hold_reset (the time in seconds for the SAP to hold during the reset operation).

1 represents the time_to_wait_reset (the time in seconds for the SAP to wait before performing the reset operation); this is an optional parameter.


Troubleshooting ASIs, MRPs, and WICs

Table 3-8 lists symptoms of and possible solutions for problems with ASIs, MRPs, and WICs.

Table 3-8 ASI, MRP, and WIC Problems and Solutions 

Symptom
ASI or MRP LED Status Change
Possible Cause
Solutions

All ASIs and MRPs do not display status as expected.

STATUS: off
(all ASIs and MRPs)

Power problem

Check your system power connections and power supply modules. (See the "Power Subsystem" section.)

A single ASI or MRP does not display status as expected.

STATUS: off (single ASI or MRP)

Improperly seated card or chassis slot problem

If the STATUS LED remains off, verify that the card is seated properly.

Assuming that the power LED of other ASIs or MRPs is on, try inserting the card in a different slot.

A WAN interface is down.1

ALARM: on (amber)

Improperly configured or defective WIC

Verify that the WIC is properly configured. If you still encounter problems, contact your technical support representative for assistance.

There is loss of signal on a WAN serial link. (See Footnote 1.)

ALARM: on (amber)

Cable or WIC problem

Replace the cable attached to the WIC that is reporting the error.

Have your telephone service provider perform a loopback test to verify the integrity of the WAN link.

Replace the faulty WIC.

Contact your technical support representative for assistance.

Note See "Solving Serial Connection Problems," for additional information.

The WIC is not supported. (See Footnote 1.)

ALARM: on (amber)

WIC installed in ASI81 or MRP is not supported by currently loaded Cisco IOS software

Determine whether the WIC is supported by the Cisco IOS software running on the ASI or MRP.

Contact your technical support representative for assistance.

An ASI or MRP initialization error has occurred.

ALARM: on (amber)

Checksum error detected while downloading WIC firmware

Copy the error message exactly as it appears, and contact your technical support representative.

Very low call completion rate on the MRP or ASI card.

ALARM: off

High CPU utilization or memory allocation failure; might be caused by turning off fast switching on the MRP or ASI

Check to make sure that fast switching is enabled on the MRP. If it is disabled, configure ip route-cache on the MRP's FastEthernet 0/0 interface. See the following configuration example:

MRP(config)#int fastEthernet 0/0
MRP(config-if)#ip address 
10.10.11.6 255.255.255.0
MRP(config-if)#ip route-cache

Refer to "Configuring Fast Switching" in Cisco IOS Switching Paths for additional information.

The ASI or MRP has failed to boot.

ALARM: on (yellow)

STATUS: off

Improperly seated or defective ASI or MRP, Cisco IOS image not downloaded from the SPE (see Table 3-10 for more information)

Remove and reinsert the ASI or MRP.

Replace the ASI or MRP.

Using System Manager, check the System Manager Software Upgrade table on the SPE running System Manager to see if an appropriate Cisco IOS image has been delivered to the ASI or MRP card.

A WIC minor alarm occurs.

ALARM: on (yellow)

WIC link integrity failure or incorrect WIC configuration

Verify that the WIC configuration is correct.

Have your telephone service provider perform a loopback test.

There are WIC transmit or receive errors.

ALARM: on (yellow)

Consecutive frame bits on receive line

Errors on receive or transmit lines

Remote signal or frame errors on receive and transmit lines

Check the status of the carrier line.

Contact your technical support representative for assistance.

1 The Flash-based MRP cards boot up faster than the non-Flash-based MRPs, so these cards might be functional before the SPE is fully operational. This means that the alarm LED might not be able to detect the condition(s) noted in Table 3-8. See the "MRP300, MRP3-8FXS, MRP3-16FXS, and MRP3-8FXOM1 Boot Sequence" section for additional information.



Note WIC, VIC, and VWIC connectivity that is lost for an extended period of time, such as 30 minutes, may not be regained without resetting the far end of the connection.


Troubleshooting the SSP

Table 3-9 lists symptoms of and possible solutions for problems with an SSP.

Table 3-9 SSP Problems and Solutions 

Symptom
SSP LED Status Change
Possible Cause
Solutions

SSP does not display status as expected.

STATUS: off

Power problem or improperly seated card.

Check system power connections and power supply modules. (See the "Power Subsystem" section.)

If the STATUS LED remains off, verify that the SSP is seated properly.

An SSP interface is down.

ALARM: on (yellow)

Interface intentionally disabled or errors on network.

If the state change is unexpected, verify that the interface has not been explicitly disabled by an administrator.

Look for an address violation, such as an address mismatch or duplication. If there is any address duplication, disable address violation detection.

Verify that there are no network connection errors, such as a loss of link beat or jabber.

Contact your technical support representative for assistance.

SSP fails to boot.

STATUS: blinking (green)

Switch software has become corrupted, or there is no bootable file in Flash memory.

This situation can cause the SSP to drop into ROMMON mode with no bootable file in Flash memory.

Reseat the SSP in the chassis.

If the problem persists, use XMODEM to copy the correct image from the PC to the SSP (ensure that the baud rate on the SSP is configured with the default value of 9600). Refer to Recovery From Corrupt or Missing Software
Image - Cisco Catalyst 2900XL, 3500XL, and 2950 Series Switches
for additional information.

Contact your technical support representative for assistance.


Troubleshooting Cisco ICS 7750 Booting Problems

This section describes various ways to troubleshoot problems that may prevent or interfere with the ability of the Cisco ICS 7750 to boot. This section is organized as follows:

Monitoring the Cisco ICS 7750 Boot Process

MRP300, MRP3-8FXS, MRP3-16FXS, and MRP3-8FXOM1 Boot Sequence

ASI and MRP200 Boot Sequence

General ASI or MRP200 Boot Problems

Specific System and MRP-Related Bootup Problems

SSP Card Boot Problems

Monitoring the Cisco ICS 7750 Boot Process

You can monitor the boot process on the Cisco ICS 7750 to verify that it comes up correctly, as follows:


Step 1 Using a HyperTerminal connection, connect a PC to the console port of the SAP card.

The HyperTerminal connection will display information that is flowing across the serial connection.

Step 2 Power on the Cisco ICS 7750 chassis.

The SAP card should display fan, temperature, power supply, and hardware sensor information.

Step 3 Once the display stops, press CTRL-backslash (\), and choose the SSP switch card (option C).


The SSP will boot in the regular Catalyst switch sequence. However, although it appears to be done booting, it is not done.

Several minutes will elapse with little apparent activity. During this time, the SPE running System Manager is launching various essential services.

If this is a new Cisco ICS 7750 system or if it is the first time the SPE has been booted up, 8 to 10 minutes may elapse. The amount of time will also depend on the software package installed on the SPE.

If this is just a reboot of a configured system, the time will be less—3 to 5 minutes. There is little point in trying to monitor that directly. It is recommended that you do not attempt to access the SPE during this period.

After a few minutes, the SPE running System Manager will download a configuration to the SSP switch card. You will know that the download is completed when you see (on the SSP) the following:

*Mar  1 00:00:23.938 UTC: %SYS-5-CONFIG_I: Configured from console by 
vty0 (10.0.0.1)


Note The IP address, 10.0.0.1, is the default IP address for the SPE running System Manager. This IP address may be different after you have configured the system using ICSConfig.


If you then display the configuration of the SSP, you will find the following:

It has a password. (If this is the initial boot of the system or of a replacement SPE running System Manager, the password will be changeme.)

It has an IP address on Interface VLAN 1. (If this is the initial boot of the system or of a replacement SPE running System Manager, the IP address will be in the 10.0.0.0 network, with a subnet mask of 255.255.255.0.)


Note If any of these settings is incorrect, the system will not function, even if you attempt to change settings or otherwise intervene manually. Contact the Cisco TAC for assistance.


If the SSP has the correct configuration (IP address and password), press CTRL-backslash (\), and enter the slot number of an MRP card.


Note The Flash-based MRP cards (MRP300, MRP3-8FXS, MRP3-16FXS, and MRP3-8FXOM1) do not require an operational SPE to boot up. The configuration files for the Flash-based MRPs are saved on each MRP in nonvolatile RAM (NVRAM), and the Cisco IOS image for each MRP is loaded and saved on the Flash memory. See the "MRP300, MRP3-8FXS, MRP3-16FXS, and MRP3-8FXOM1 Boot Sequence" section for more information on the Flash-based MRP cards.


If an MRP200, ASI81, or ASI160 card is installed, it may still be broadcasting a BOOTP request. Let the card continue broadcasting because it may take several minutes for the MRPs to receive their boot information—their IP address, the Cisco IOS image, and configuration.

The reason why non-Flash-based MRP BOOTP requests go unanswered before the MRP boots up is that the MRPs are the last cards to boot in the chassis. Before the SPE running System Manager can answer the MRPs' boot requests, that SPE must first boot itself and start ICS System Manager. It then answers the BOOTP requests from other cards, such as MRP200 and ASI cards, that may be present. Also, before the Cisco Network Registrar (CNR) responds to a BOOTP request, it must ensure that the request has originated from within the same chassis. Typically, the requests from an MRP or ASI, that fall within the first 30 seconds after the SPE running System Manager has fully started, will not receive a response.

When the MRP starts booting from 10.0.0.1 (the SPE running System Manager), you can assume that the boot process is likely to succeed.

When all the MRPs boot from the SPE running System Manager, the Cisco ICS 7750 is ready to operate.

You can use the same monitoring process to verify that the system boots correctly after running the initial configuration program (ICSConfig). Of course, the IP address and password on the SSP must match the ones you assigned in ICSConfig.

If the boot process is not successful, see the "Specific System and MRP-Related Bootup Problems" section for additional information.

MRP300, MRP3-8FXS, MRP3-16FXS, and MRP3-8FXOM1 Boot Sequence

The MRP200 and MRP300 cards are both voice-and-data-capable routers that support digital and analog voice trunks and WAN routing interfaces to link remote Ethernet LANs to the public switched telephone network (PSTN), to existing private branch exchanges (PBXs), and to most common analog devices, such as fax machines and teleconferencing stations.

The MRP300, MRP3-8FXS, MRP3-16FXS, and MRP3-8FXOM1 cards are Flash-based cards with nonvolatile RAM (NVRAM). They have additional functionality provided by 16 MB of onboard Flash memory, with 64 MB of add-on Flash memory available as an option. The configuration files for the Flash-based MRPs are saved on each MRP in NVRAM, and the Cisco IOS image for each MRP is loaded and saved on the Flash memory.

The Flash-based cards have improved performance, that is shorter boot time. With a Cisco IOS image in Flash memory, the Flash-based MRP card boot time is reduced because it does not have to obtain the image from the SPE using TFTP. This allows for a faster boot-up process on the Flash-based MRPs, and it enables the data network to be functional before the SPE is fully operational.

Beginning with ICS System Software release 2.5.0, the Flash-based MRP cards can also be configured as virtual local area network (VLAN) routers. See the "Support for VLAN Functionality" section.

The MRP3-8FXOM1, available with ICS System Software release 2.6.0, includes eight built-in FXO M1 analog interfaces for connecting eight analog trunks between a central office (CO) and an IP telephony system. Designed as a high-density card, the MRP3-8FXOM1 can be used to provide trunking redundancy when Primary Rate Interface (PRI) or T1 channel associated signaling (CAS) trunks become disabled. The MRP3-8FXOM1 can also be used as a low-cost alternative to fractional T1s (used for connecting to the CO). Like the MRP3-8FXS, the MRP3-8FXOM1 also has an open VWIC slot (slot 1) that accepts all Cisco ICS 7750-supported VWICs.


Note The MRP3-8FXOM1 card is supported for use in North American M1-compliant countries (meeting North American M1 requirements for 600-ohm resistive termination).


New features that are available with the Cisco IOS software release supporting the MRP3-8FXOM1 include the following:

The capability for DSP resources in PVDM0 and PVDM1 to be pooled and shared for use by the FXO ports and digital voice ports (such as T1 ports). Resource sharing enables the most efficient use of DSPs and allows for the creation of two DSP groups, even if only one clock source is configured, to maximize DSP resource availability (ten DSPs). See the "Codec Complexity, DSP Groups, and PVDM Guidelines" section on page 9-14 for additional information.

Simultaneous support for 30 voice calls on the T1 or E1 controller(s) and eight FXO calls through the MRP3-8FXOM1 interface.

Calling line identification service (caller ID) data pass-through.

Dual tone multifrequency (DTMF) dialing and detection, and generation of pulse dialing.

Recognition of distinctive ringing that allows two logical telephone numbers to be mapped to a single telephone line.

Software-programmable analog gain to set codec transmit amplification and receiver attenuation, timer delays such as hook-flash delay and other signaling timers, and line termination.

Multichannel support for up to a maximum of eight channel groups, PRI data (PRI-D) channels, or PRI dialer calls on a single T1 or E1 controller, with supported speeds on the individual channels of 48 kbps, 56 kbps, and 64 kbps (if multichannel support is not enabled, speeds of 48 kbps and 56 kbps are not supported). Multichannel support is enabled by the tdm multichannel command.

When you configure a T1 or E1 PRI, the total number of channels displayed in the output of the show run command might not correspond to the actual number of usable PRI voice channels. Use the show voice dsp command to display the number of voice channels that are usable for voice. A syslog message will be generated for each Bearer channel (B-channel) time slot that is not usable for voice when the PRI group is configured.

For example:

%IPM_DSPRM-3-INSUFFICIENT: Insufficient DSP resources for 
timeslot 20 on port 1/1:23

To work around this situation, refer to the "T1/E1 PRI Configuration" section in the Cisco 1- and 2-port T1/E1 Multiflex Voice/WAN Interface Cards for the Cisco 1751 and Cisco 1760 Routers before you configure the T1 or E1 for PRI capability.

Use the isdn incoming-voice command to enable voice or data over voice calls on PRI interfaces (the default is to treat incoming calls as dialer).


Note High-Level Data Link Control (HDLC) is a protocol that provides Cisco serial encapsulation. In non-multichannel mode, only four HDLC resources are available. In multichannel mode, eight HDLC resources are available. For more information about the MRP3-8FXOM1 card, multichannel and serial channel controller (SCC) non-multichannel modes, and the new Cisco IOS CLI commands, see the "Codec Complexity, DSP Groups, and PVDM Guidelines" section on page 9-14, or refer to the Cisco ICS 7750 Installation and Configuration Guide.


Initial Bootup and Discovery

The MRP Flash-based cards are shipped with a default Cisco IOS image in Flash memory, and with no configuration in their NVRAM. The first time the Flash-based MRP boots up in a Cisco ICS 7750 chassis, the card will send DHCP requests (see item number 2 in the sequence on page 3-39).


Note The MRP Flash-based cards should obtain their IP address and initial configuration from the ICS System Manager that is running in the same chassis, and not from any other external Dynamic Host Control Protocol (DHCP) server.


If the boot field of the configuration register, which controls the way the system boots, is set to "Boot to ROMMON" (0x0), then the system will remain at the ROM Monitor (ROMMON) prompt. Setting the configuration register to 0x2102 will not initiate BOOTP requests, and the MRP will try to boot from Flash memory.

If the boot field of the configuration register is set to anything other than "Boot to ROMMON," then one of the following will occur:

If the Flash memory is corrupt or if there is no image in Flash, the system will stay in ROMMON mode. No attempt is made to obtain a Cisco IOS image from the SPE running System Manager.

If there is a Cisco IOS image in Flash memory, ROMMON boots from the image in Flash memory.

Standard Cisco IOS boot behavior is followed with respect to the configuration register setting.


Note For additional information about the boot field in the configuration register, refer to the "Rebooting" section in the Cisco IOS Configuration Fundamentals Configuration Guide, Release 12.2, Part 2: File Management.


After the Cisco IOS software has booted, the following should occur:

1. The Cisco IOS software enters the standard Cisco IOS prompt if the Fast Ethernet interface and all its subinterfaces (if there are any), are configured with IP addresses by the startup-configuration in NVRAM.

2. If new IP interfaces are discovered during bootup, the Cisco IOS software will attempt to discover the IP addresses by sending DHCP requests. The Cisco IOS software will not perform DHCP auto-installation. DHCP responses are accepted from any DHCP server.

3. If there is no startup-configuration in NVRAM, then the Cisco IOS software will attempt to discover the IP addresses and download the startup-configuration through the standard DHCP auto-installation mechanism, from ICS System Manager in the same chassis only.

The Vendor class identifier field in DHCP request messages sent during DHCP auto-installation will be filled with any one of the following strings, as appropriate:

ICS7750-MRP300—for MRP300

ICS7750-MRP3-8FXS—for MRP3-8FXS

ICS7750-MRP3-16FXS—for MRP3-16FXS

ICS7750-MRP3-8FXOM1—for MRP3-8FXOM1

4. The DHCP response message from ICS System Manager will contain the IP address, the IP address of the TFTP server, and the configuration filename. During DHCP auto-installation, the Flash-based MRP card looks for the pattern ICS7750-SPE in sub-option 66 of Option 43 (Vendor-specific information) in the DHCP response. This entry ensures that the Flash-based MRP card obtains its IP address and initial configuration from the ICS System Manager, and not from any external DHCP server.

5. If the DHCP response contains the expected pattern (in sub-option 66 of Option 43) and a configuration filename, the Flash-based MRP card downloads the configuration file from the TFTP server and merges it with the running-configuration file.

6. The Flash-based MRP card is now configured with the passwords and IP address delivered by the ICS System Manager. The configuration is saved locally in the Flash-based card's NVRAM. When a configuration change is made to the Flash-based card by using ICSConfig, the ICS System Manager establishes a Telnet session to the Flash-based card and issues the copy running-config startup-config command, which saves the configuration in that card's NVRAM. When manual changes are made to the configuration by using the Cisco IOS CLI from the MRP's console, the running-config will be written to the startup-config in NVRAM when the following command is executed: copy running-config startup-config.

Once NVRAM is populated, the configuration from NVRAM will be read to configure the Flash-based MRP card on subsequent boots. From this point on, the MRP does not use ICS System Manager to obtain its boot information.


Note When a Flash-based MRP card is moved from one chassis to another, the startup-config must be erased so that the Flash-based card can obtain its new IP address from the ICS System Manager in the new chassis. Failure to erase the startup-config will result in the inability of ICS System Manager to discover the card.


7. The Flash-based MRP card then sends Cisco Discovery Protocol (CDP) packets. ICS System Manager discovers the Flash-based MRP card from the CDP cache on the SSP. The configuration is not automatically saved to NVRAM.

8. If ICS System Manager does not respond to the Flash-based card's DHCP requests, then after a short period of time (approximately 17 seconds), the Flash-based card enters the standard initial setup dialog. DHCP requests will continue to be sent in the background, until the setup dialog is quit or aborted or until the Fast Ethernet interface is configured with an IP address.

If the setup dialog is aborted, the Fast Ethernet interface will be shut down and the MRP will not be discovered by ICS System Manager.

When the DHCP response is received from ICS System Manager (after the MRP has entered the setup dialog), the Fast Ethernet interface is configured with an IP address allocated by ICS System Manager, the setup dialog is then aborted, and the MRP enters into the standard Cisco IOS prompt.


Note If the Flash-based MRP card is reloaded after changing, and before saving, the configuration to NVRAM, the MRP will boot with the previously saved configuration. If the MRP is reloaded after the NVRAM is erased, then it will follow the boot procedure outlined above.


ASI and MRP200 Boot Sequence

The ASI and MRP200 cards do not contain Flash memory. The configuration files for these cards are loaded and saved on the SPE running System Manager.

The ASI81 card can support eight connections to analog telephones, fax machines, and teleconferencing stations. The ASI160 card provides support for 16 connections.

When ASI and MRP200 cards boot, the following sequence of events should occur:

1. The card runs a bootstrap program called ROM Monitor (ROMMON) that helps to initialize the card processor and to boot the operating system. While in ROM Monitor mode, the card sends a BOOTP request.

2. Cisco Network Registrar software, which operates on the SPE running System Manager, replies to the BOOTP request sent by the ASI or MRP200, provided that the ASI or MRP200 has been discovered by the SPE running System Manager. The BOOTP reply contains the locations of both the Cisco IOS image and the Cisco IOS configuration file for the MRP200 and ASI cards.


Note There is potential for other BOOTP servers to respond as well; for this reason, it is important that the Cisco ICS 7750 be isolated from the network while initially running ICSConfig and during periods of troubleshooting.


3. The SPE builds configuration data for the booting ASI or MRP200 and stores the data in a configuration file in the TFTPPath folder. (The Cisco IOS image is made available for the TFTP server through the system software installation process or through software upgrade from the Software Upgrade page on the ICS System Manager.)

4. The ASI or MRP200 card obtains the download information it needs from the BOOTP response (the card's IP address, subnet mask, gateway, TFTP server, and the name of the Cisco IOS image that should run on it).

5. The ASI or MRP200 card downloads and then starts the Cisco IOS image. Then it downloads its configuration file, putting the configuration file into its running configuration.

6. The ASI or MRP200 completes the boot sequence.

General ASI or MRP200 Boot Problems

If the ASI or MRP200 cards do not boot on the Cisco ICS 7750, you should first determine at what point the process is failing by watching the bootup sequence from the console connection on the SAP.

Use a HyperTerminal connection to the SAP to see whether the ASI or MRP200 is receiving the following:

A valid IP address

TFTP server

Cisco IOS image name

Configuration (.cfg file)

Troubleshooting the SPE Running System Manager

If the ASI or MRP200 is not receiving the required items—valid IP address, TFTP server, Cisco IOS image name, and configuration file—follow these guidelines to troubleshoot the problem, using a direct connection to the SPE or using a Terminal Services client connection:

Examine the CNR logs on the SPE running System Manager. These logs are located at C:\Program Files\Network Registrar\logs. This should help you determine whether the ASI or MRP200 is receiving the required data to boot.

Check to make sure that CNR is started on the SPE running System Manager by navigating to Start > Programs > Administrative Tools > Services > AIC Server Agent. AIC Server Agent must be running to serve IP addresses and reply to BOOTP requests from the ASI and/or MRP200.

Check for DHCP request/response flow on the SPE running System Manager (view the C:\Program Files\Network Registrar\logs\name_dhcp_1_log). DHCP/BOOTP activity is logged to this file.

If the ASI or MRP200 receives a response from the SPE, check to see whether its image exists in the TFTPPath folder on the SPE running System Manager (see the C:\Program Files\Cisco\TFTPPath).

If you see any conflicts or problems, you can use the information to work with a Cisco technical support representative to resolve the situation.

Best Practices for Using the Cisco IOS CLI

ICS System Manager is designed to communicate with, and to monitor the status of, all the components in the chassis. To enable ICS System Manager to perform these functions, a configuration program (ICSConfig) provides the capability to easily configure key system parameters, such as the IP addresses of system cards, passwords, destination for syslog messages, and Simple Network Management Protocol (SNMP) community strings.

To enable ICS System Manager to perform all of its functions as a system management tool, it is important that you use ICSConfig or ICS System Manager, as appropriate, rather than the Cisco IOS CLI, when you enter key system parameters.

With the exception of the procedures listed below, you can enter all of the Cisco IOS CLI commands that are available for use in any Cisco IOS software release that is intended for use on the Cisco ICS 7750.

You should always use ICSConfig for the following tasks:

Passwords

Changing the login password, which gives ICS System Manager continued Telnet access to system cards

Changing the Windows 2000 administrator password, which grants those with administrator privileges continued access to SPE310s

Changing the enable or secret password, which makes it possible for administrators to enter certain Cisco IOS commands

Card configurations

Assigning or changing the IP addresses or subnet mask of system cards

SNMP settings

Changing read-only and read/write SNMP community strings of the SNMP server

Changing the server destination of SNMP traps

Managing the SNMP server

Logging

Changing the syslog logging host


Note SNMP community strings and system passwords are case-sensitive, and they should be configured only through ICSConfig.


The following tasks cannot be configured on the Cisco ICS 7750 using the Cisco IOS CLI under any circumstances:

Shutting down an Ethernet interface

Changing the IP address of a Fast Ethernet interface (unless you have configured VLAN support)

Disabling Cisco Discovery Protocol (CDP) on an Ethernet or VLAN interface

The following tasks cannot be performed on the SPEs under any circumstances:

Configuring Domain Name System (DNS) on SPEs

Invoking the Cisco Network Registrar (CNR) dhcp.exe from c:\program files\network registrar\bin

Specific System and MRP-Related Bootup Problems

The following sections describe a few scenarios that can cause booting problems on non-Flash-based MRP cards (ASI81, ASI160, or MRP200) and on Flash-based MRP cards (MRP300, MRP3-8FXS, MRP3-16FXS, or MRP3-8FXOM1) cards.

Troubleshooting MRP200, ASI81, and ASI160 Bootup Problems

Troubleshooting Flash-Based MRP Bootup Problems

Troubleshooting MRP200, ASI81, and ASI160 Bootup Problems

Table 3-10 lists symptoms of and possible solutions for problems with MRP200, ASI81, and ASI160 cards.

Table 3-10 ASI81, ASI160, and MRP200 Bootup Problems and Solutions 

Problem
Symptom
Possible Cause
Solutions

ASI and/or MRP does not appear to be functional.

ASI and/or MRP does not display status as expected.

Power problem

Card(s) improperly seated or chassis slot problem

Check your system power connections and power supply modules. (See the "Power Subsystem" section.)

If the STATUS LED remains off, verify that the card is seated properly in the chassis. If this condition exists in slot 0 on an ASI81 card, the card might need to be replaced. Contact your local support representative for assistance.

If the power LED of other ASIs or MRPs is on, try inserting the card in a different slot in the chassis.

ASI and/or MRP does not send BOOTP requests.

ASI and/or MRP is stuck at the rommon 1> prompt.

Incorrect configuration register setting on the ASI or MRP

Use confreg in ROMMON mode to change the configuration register setting to 0x101. With the configuration register set to 0x101, the MRP200 will send out BOOTP requests. After you change the config-reg setting, reset the card:

1. rommon>confreg 0x101

2. rommon>i (or reset)

ASI and/or MRP does not get valid TFTP server or Cisco IOS image.

The ASI and/or MRP receives the BOOTP response from the SPE but times out during the download process.

The Cisco IOS image has not been imported into the database, or an appropriate Cisco IOS image has not been delivered to the ASI and/or MRP card (see Table 3-8

Check to make sure that the Cisco IOS image is in the C:\Program Files\Cisco\TFTPPath directory on the SPE running System Manager. Then follow these steps to use tftpdnld to download the ASI/MRP Cisco IOS image:

1. Display the current variables.

rommon>set

2. Set the IP address of the ASI/MRP.

rommon 2 >IP_ADDRESS=10.0.0.3

3. Set the subnet mask.

rommon 3 > IP_SUBNET_MASK=255.255.255.0

4. Set the IP address of the TFTP Server.

rommon 4 >TFTP_SERVER=10.0.0.1

5. Enter the filename to get.

rommon 5 > TFTP_FILE=ics7700-sv3y-mz.12.2(4)YH

6. rommon 6>sync

7. rommon 6 >tftpdnld

This process is a temporary workaround that allows you to manually download the Cisco IOS image for the MRP/ASI card. A Cisco IOS image must be in the TFTPPath directory on the SPE running System Manager for this process to work. If there is no Cisco IOS image in the TFTPPath directory, do a search for the image name on the hard drive to determine if the Cisco IOS image is located in a different directory on the SPE. If it is located in a different directory, copy it to TFTPPath. If it is not located on the SPE, download a copy of the Cisco IOS image from the Cisco IOS Planner Upgrade page.

ICS System Manager fails to respond to ASI and/or MRP BOOTP requests.

ASI and/or MRP continually sends BOOTP requests but gets no response from the SPE, so the ASI/MRP card fails to boot.

ICSConfig data may be out of sync because of manual Cisco IOS CLI changes (such as passwords, IP addresses, and community strings), insufficient IP addresses in CNR scope, duplicate IP address assignment, or subnet addressing issues

Always configure IP addresses, community strings, and passwords through ICSConfig. See the "Best Practices for Using the Cisco IOS CLI" section.

Ensure that there are enough IP addresses defined in ICSConfig to allocate IP addresses to all the cards in the chassis.

Ensure that there are no duplicate IP addresses in your network.

Ensure that the AIC Server Agent Service (CNR) is running on the SPE.

Ensure that the SPE and SSP are in the same subnet and within the address range allocated for the cards.

Ensure that the date and time on the SPE are set correctly (otherwise, CNR may not start).

Note See the "Troubleshooting the SPE Running System Manager" section for additional information.

ICS System Manager fails to respond to ASI and/or MRP BOOTP requests.

ASI and/or MRP continually sends BOOTP requests but gets no response.

Problem with Ethernet connectivity

If the problem is at the physical layer:

a. Reseat the ASI or MRP in the chassis slot to reset the card.

b. Move the ASI or MRP card to another slot in the chassis to see if it will boot up.

Verify that the IP address of the ASI or MRP is unique and is not assigned to another device in the network.

Contact your technical support representative for assistance.

ICS System Manager fails to respond to ASI and/or MRP BOOTP requests.

ASI and/or MRP continually sends BOOTP requests but gets no response.

SPE running System Manager has failed to boot

Wait for the SPE to come up, if the system was just started.

Shut down and restart the SPE, using ICS System Manager.

Remove and reinsert the SPE in the slot, or move it to another slot in the chassis. See the "Best Practices for Reseating Cards in the Chassis" section.

Contact your technical support representative.


Troubleshooting Flash-Based MRP Bootup Problems

Table 3-11 lists symptoms of and possible solutions for problems with Flash-based MRPs (MRP300, MRP3-8FXS, MRP3-16FXS, and MRP3-8FXOM1 cards).


Note When a Flash-based MRP card is moved from one chassis to another, the startup-config must be erased so that the Flash-based MRP can obtain its new IP address from the ICS System Manager in the new chassis. Failure to erase the startup-config will result in the inability of ICS System Manager to discover the MRP card.


Table 3-11 Flash-Based MRP Bootup Problems and Solutions 

Problem
Symptom
Possible Cause
Solutions

MRP does not appear to be functional.

MRP does not display status as expected.

Power problem

Card(s) improperly seated or chassis slot problem

Check your system power connections and power supply modules. (See the "Power Subsystem" section.)

If the STATUS LED remains off, verify that the card is seated properly in the chassis. If this condition exists in slot 0 on an MRP3-8FXS, or MRP3-8FXOM1 card, the card might need to be replaced. Contact your local support representative for assistance.

If the power LED of another MRP cards is on, try inserting the card in a different slot in the chassis.

MRP fails to boot from Flash memory or manual boot.

MRP is stuck at the rommon 1> prompt, and a manual boot flash: results in an error. See Example 3-1 for an example of a manual boot flash: with a valid image.

Image in Flash memory is not valid or is corrupted

If you see a manual boot flash error, such as the following:

rommon 3>boot flash:
device does not contain a valid 
magic number
boot: cannot open "flash:"
boot: cannot determine first file 
name on device "flash:"

Use tftpdnld in ROMMON mode to copy the valid image to Flash memory, and reset the card. See Table 3-10 for tftpdnld instructions.

MRP does not boot from Flash memory.

MRP is stuck at the rommon 1> prompt.

Incorrect configuration register setting on the MRP

Use the confreg command in ROMMON mode to change the configuration register setting to 0x2102. With the configuration register set to 0x2102, the MRP will try to boot from Flash Memory.

Follow these steps to reset the card after changing the config-reg setting:

a. rommon>confreg 0x2102

b. rommon>i

Or, enter boot flash: at the command prompt, and then change the configuration register setting using the Cisco IOS CLI after the MRP boots:

a. MRP#config t

b. MRP#(config)# config-register 0x2102

ICS System Manager fails to recognize the MRP.

MRP cannot be discovered by System Manager and cannot boot.

Mismatched SNMP community strings result in authentication failure

Mismatched startup-config results in conflict with IP address or with login and enable passwords

MRP configuration is done outside of ICSConfig

If the MRP had a startup-config in NVRAM, check to make sure that its SNMP community strings match those entered in ICSConfig.

Check that the IP address of the MRP is in the same subnet as the chassis.

Check that the login and enable passwords on the MRP are the same as those configured in ICSConfig.

Use ICSConfig to configure the MRP parameters; do not use the Cisco IOS CLI.

Erase the startup-config before installing an MRP into a new chassis (if there is a startup-config, System Manager cannot deliver a configuration to the MRP).

Contact your technical support representative for assistance with problems that you cannot solve.


Example 3-1 Manual boot flash: with a Valid Image in Flash Memory

A manual boot flash: looks like the following example when there is a valid image in Flash memory:

Enter dir flash: to see the file:
rommon 17 >dir flash:
16MB of Flash found
 File size           Checksum   File name
 9362188 bytes (0x8edb0c)   0xae67    ics7700-bnr2sv3y-mz.122-4.XL1
rommon 18 >boot flash:
program load complete, entry point: 0x80008000, size: 0x8ed9f0
Self decompressing the image : 
######################################################################
####

Best Practices for Reseating Cards in the Chassis

You may need to remove and reinsert a card in the Cisco ICS 7750 chassis. Follow these steps to safely reseat chassis cards:


Step 1 If you are reseating an SPE310 card, make sure that you shut down the card before you remove it. On the target SPE, navigate to Start > Shut Down. The STATUS LED blinks and then turns off.

Step 2 Other cards may be removed without shutting them down. Disengage the card from the chassis backplane by completing the following steps:

a. Completely loosen the card captive screws.

b. Press the upper and lower ejector levers outward at the same time to disengage the card from the backplane.


Caution Always use the ejector levers to disengage or seat cards. Failure to use the levers can cause erroneous system error messages that indicate a card failure. Do not use the ejector levers to lift or support the weight of the card.

c. Grasp the ejector levers, and gently pull the card about one inch out of the chassis.


Caution When installing cards, be sure to apply equal pressure to the top and bottom of the card. You can damage the card and the chassis backplane if you apply too much pressure to the bottom or the top of the card.

Step 3 Gently slide the card back into the chassis until you feel resistance. Because there are grounding clips near the front and rear of the card guides, you might need to increase the amount of force that you use to get the card past the grounding clips. If you encounter extreme resistance, pull the card out slightly, and push it back in again.

Step 4 Press the upper and lower ejection levers inward at the same time until they lock into their slots. This firmly seats the card into the chassis.

The card will reboot and the system will complete the upload if you restarted an ASI or MRP200 card.


Replacing an MRP200 or ASI Card with a Flash-Based MRP Card

Follow these steps to replace an MRP200 or ASI card with a Flash-based MRP card (MRP300, MRP3-8FXS, MRP3-16FXS, or MRP3-8FXOM1):


Step 1 Save the existing MRP200 or ASI card configuration file in the TFTP path directory on the SPE running System Manager (C:\Program Files\Cisco Systems\TFTPPath).

The name of the configuration file will be in the format of the MRP200 or ASI card's MAC address, followed by .cfg. For example:

000196664ab7.cfg

This is a safety precaution to ensure that you have a backup of the configuration file during the MRP swap.

Step 2 Remove the MRP200 or ASI card from the chassis. Refer to the "Removing an ASI, MRP, SSP, or SAP" section in the Cisco ICS 7750 FRU Installation and Replacement.

Step 3 Insert the Flash-based MRP card into the chassis. Refer to the "Installing an ASI or MRP" section in the Cisco ICS 7750 FRU Installation and Replacement.

The Flash-based MRP card will receive a DHCP address and its initial configuration from the SPE running System Manager.

Step 4 Run ICSConfig to configure the Flash-based MRP card with the same IP address as the MRP200 or ASI card that you removed from the chassis.

a. To run ICSConfig, enter the following URL into your web browser:

http://SPE IP address/icsconfig/

where the address entered is the IP address of the SPE running System Manager.

b. Log in as administrator (user ID administrator), and enter the password (the default is changeme).

c. Click ICS 7700 System Setup.

d. Assign the IP address that was configured for the old MRP200 or ASI card to the new Flash-based MRP card. (Refer to the Cisco ICS 7750 Installation and Configuration Guide for instructions.)

Step 5 In enable mode on the Flash-based MRP, execute the following command to download the original MRP200 or ASI card configuration file to the Flash-based MRP card:

MRP#copy tftp:MAC_address.cfg running-config

This command copies the MRP configuration file from TFTP to the running configuration file on the Flash-based MRP card.

Step 6 In enable mode on the Flash-based MRP, execute the following command to save the configuration file to NVRAM on the Flash-based MRP card:

MRP#copy running-config startup-config


SSP Card Boot Problems

If you replace the SSP card with a different SSP card, you might notice the following problems with the replacement SSP card:

It might fail to log the SPE running System Manager as its logging host.

It might fail to log the SPE running System Manager as its SNMP server.

Its console password, vty password, read-write Community String, and Trap Community String might not be set.


Note If you have configured VLAN support on the Cisco ICS 7750, see the "Backing Up and Restoring the VLAN Configuration on the SSP" section for information about preserving the VLAN configuration when resolving SSP problems.


To solve these problems, complete the following steps:


Step 1 Open a HyperTerminal session with the SAP card.

Step 2 Log in as an administrator (user ID administrator), and enter your password (the default is changeme).

Step 3 Press Ctrl-backslash (\), and use the SAP card menu to switch to the SPE card running System Manager.

Step 4 Log in as an administrator (user ID administrator), and enter your password (the default is changeme).

Step 5 Enter the following command:

net stop FMMServer

Step 6 Press Ctrl-backslash (\), and use the SAP card menu to switch to the SSP card.

Step 7 Enter privileged EXEC mode by entering the following command:

switch>enable

Step 8 Enter your enable password.

Step 9 Erase the SSP card startup configuration by entering the following command:

switch#erase start


Note Make sure that you answer No when prompted to save the current configuration.


Step 10 Restart the SSP card by entering the following command:

switch#reload

The SSP card reboots. When the SSP card has finished booting, its STATUS LED will turn green, and its ALARM LED should go off.

Step 11 Press Ctrl-backslash (\), and use the SAP card menu to switch to the SPE running System Manager.

Step 12 Enter the following command:

net start FMMServer
 
   

Useful MRP Troubleshooting Commands

The following are some useful troubleshooting commands for the MRP and ASI cards. Refer to Troubleshoot & Debug VoIP Calls - the Basics for additional information and examples of the various command outputs.

show controller t1Displays information about the T1 links to verify the status of the digital T1 connection between the MRP and the switch (CO or PBX). Shows whether the link is up or down and whether it is functioning properly.

show voice-port—Displays configuration information about a specific voice port, such as the port state and the parameters configured on the voice port of the voice interface card (VIC).

show running-configShows running-configuration settings.

show versionDisplays configuration of the system hardware, the software version, the names and sources of configuration files, the boot images, the amount of time that the MRP has been up and running, how the MRP was restarted, the name of the MRP image file, a log of how the system was last booted (both as a result of normal system startup and of system error), and the configuration register contents displayed in hexadecimal notation.

show stacksDisplays the stack usage of processes and interrupt routines, showing the reason for the last system reboot.

show voice dspShows the current status of all DSP voice channels to verify available MIPS.

sh voice call summOn the MRP, displays the signaling channel state on a per time slots basis.

show call active voiceShows the active call table.

show dial-peer voiceDisplays configuration information for dial peers.

show interfaceDisplays statistics for all interfaces configured on the MRP.

show ip interfaceDisplays the usability status of interfaces configured for IP.

The following troubleshooting commands are specific to the Media Gateway Control Protocol (MGCP). Refer to Using Media Gateway Control Protocol with the Cisco ICS 7750 for additional information about MGCP support on the Cisco ICS 7750.

show mgcpVerifies the status of the MRP's MGCP parameters.

show mgcp endpoint—Shows the voice ports (endpoints) that are under MGCP control in the MRP. This command verifies which voice ports have been bound to the MGCP application.

show mgcp connection—Displays any active MGCP connections.

show mgcp statistics—Shows statistical information related to MGCP activity on the MRP.

show ccm-manager—Shows the status and availability of the primary and backup Cisco CallManagers.

debug mgcp [all | error | events | packets | parser]—Debugs MGCP. See the "Debugging VoIP Problems" section on page 9-25 for additional information about debugging commands.

Support for VLAN Functionality

This section describes virtual LAN (VLAN) functionality on the Cisco ICS 7750, and includes the following sections:

Enabling VLAN Functionality

Configuring VLAN Support on the Flash-Based MRP Card

Configuring VLAN Support on the SSP

Backing Up and Restoring the VLAN Configuration on the SSP

Troubleshooting VLANs

VLAN is a logical grouping of LAN segments, often associated with IP subnets. (See the "Dividing Networks with Subnets" section on page 8-7.) With VLAN functionality, you can group ports on a switch to control unicast, multicast, and broadcast traffic such that traffic originating from a particular VLAN is only flooded out of other ports that belong to the same VLAN.

When voice and data packets travel on the same VLAN, the contention for resources can affect voice quality. When multiple VLANs and subnets are used, voice packets can be separated into a different queue from data packets, making network behavior more predictable.

Beginning with release 2.5.0, ICS System Software has been enhanced to include support for VLANs and quality of service (QoS) on the Cisco ICS 7750. With this support, you can enable VLAN usage by configuring all system cards on the management VLAN (VLAN 1). You can also configure multiple data and voice VLANs outside of the Cisco ICS 7750 chassis, using VLAN identifiers of any value, except for 0 and 4095, which are reserved.


Note Because they are all in the same VLAN (VLAN 1), all system cards in the Cisco ICS 7750 chassis should be reachable by the SPE running System Manager.


When IP phones reside on a voice VLAN other than VLAN 1, all traffic between the IP phones and Cisco CallManager must be routed, including:

Call setup and teardown

Supplementary services (call park, conference, signaling)

Music on hold (MoH) packets

Voice mail between an SPE running Cisco Unity Voice Messaging and IP phones

All voice traffic between IP phones on different VLANs


Note Separating voice and data traffic onto separate VLANs does not necessarily improve QoS for voice traffic. Other QoS mechanisms and tools must be used in conjunction with VLANs to improve QoS for voice. Refer to QoS Technical Tips and Voice Quality of Service (QoS) for additional information.


The implementation of VLANs and QoS includes the ability for the Flash-based MRP cards (MRP300, MRP3-8FXS, MRP3-16FXS, and MRP3-8FXOM1) to be configured as VLAN routers to route between multiple VLANs configured on the LAN. One or more Flash-based MRPs can be configured as VLAN routers, and external VLAN routers can be used instead of, or in addition to, the Flash-based MRPs. Although the Flash-based MRP supports a maximum of 300 subinterfaces (VLANs), the maximum number of VLANs that can be supported on the Cisco ICS 7750 is 256 because of a hardware limitation on the SSP.

As VLAN routers, the Flash-based MRP cards can reside on other VLANs besides VLAN 1. At least one of the Flash-based MRP cards and/or external routers must be configured to reside on all VLANs in order for it to be able to route traffic between different VLANs.

Additional functionality provided by a DHCP server on the Flash-based MRP cards can be configured to provision IP addresses for external devices, such as PCs and phones, on VLANs other than VLAN 1.


Caution The DHCP server on the Flash-based MRP card will respond to DHCP requests from other system cards if it is configured to issue addresses on VLAN 1. To avoid the problem of duplicate IP addresses, make sure that the scopes of the Cisco IOS DHCP server and the System Manager do not overlap, and that the IP address of the DHCP server is reachable by System Manager. If the scopes overlap, duplicate IP addresses might be issued by System Manager. ICSConfig will not issue IP addresses for devices outside VLAN 1.

For information about configuring a DHCP server on the Flash-based MRP card, refer to the "Configuring DHCP" section in the Cisco IOS IP Configuration Guide, Release 12.2.

Enabling VLAN Functionality

To enable VLAN functionality on the Cisco ICS 7750, you need to manually configure both the VLAN-capable, Flash-based MRP card and the SSP through the Cisco IOS command-line interface (CLI). See the "Configuring VLAN Support on the Flash-Based MRP Card" section, and the "Configuring VLAN Support on the SSP" section for additional information.

When you configure VLAN usage on the Cisco ICS 7750, system cards cannot be moved around in the chassis without first reconfiguring their switch port definitions on the SSP. Moving cards between "static access vlan 1" and "dot1q trunk native vlan 1" will have the following effects:

If an SPE or an MRP200 is moved into a slot in the chassis that had been configured as a trunk port, the SSP will forward other VLAN traffic to these cards, and the VLAN traffic will be dropped.

If a Flash-based MRP is moved into a slot in the chassis that had been configured as a "static access vlan 1" port, VLANs other than VLAN 1 will not work.

To avoid reconfiguration when you need to replace a system card, make sure that you install the replacement card into the same slot in the chassis from which the original card was removed.

Configuring VLAN Support on the Flash-Based MRP Card

To configure VLAN functionality on your system, and to define the Flash-based MRP card as the designated VLAN router, you need to manually configure the Flash-based MRP.


Note When you configure VLAN functionality on your system, you must use ICSConfig to configure the management VLAN IP address of the Flash-based MRP card on the primary Fast Ethernet interface 0/0 for that MRP. You cannot add a native VLAN subinterface for the Flash-based MRP, nor can you configure the primary Fast Ethernet interface on the Flash-based MRP by using the command no ip address.


Follow the steps in the example below to configure the Flash-based MRP card. (The following are sample configurations [for example, VLAN 3 and VLAN 4, and some QoS settings]. You can choose whatever numbers you wish to define for your VLAN and QoS configurations.)


Step 1 Select Fast Ethernet 0/0 for the trunk configuration:

MRP#configure terminal
MRP(config)# int fastEthernet 0/0

This address is assigned by ICSConfig and should not be modified or removed by the user:

MRP(config-if)#ip address 10.10.10.6 255.255.255.0
MRP(config-if)#no shut
MRP(config-if)#exit

A subinterface with native VLAN should not be created on the MRP300.

Step 2 Create VLAN 3 on the MRP300:

MRP(config)#int fastEthernet 0/0.3

Step 3 Enter the trunking encapsulation as dot1q:

MRP(config-if)#encapsulation dot1q 3

Step 4 Configure layer 3 information on the subinterface 0/0.3:

MRP(config-if)#ip address 10.10.11.6 255.255.255.0
MRP(config-if)#exit

Step 5 Create VLAN 4 on the MRP300:

MRP(config)#int fastEthernet 0/0.4

Step 6 Enter the trunking encapsulation as dot1q:

MRP(config-if)#encapsulation dot1q 4

Step 7 Configure layer 3 information on the subinterface 0/0.4:

MRP(config-if)#ip address 10.10.12.6 255.255.255.0
MRP(config-if)#exit
MRP(config)#^Z

Step 8 Save the configuration:

MRP#copy run start

Step 9 Enable Cisco Express Forwarding (CEF):

MRP(config)#ip cef


Note To configure layer 3 to layer 2 service-policy mapping in the Fast Ethernet subinterface, perform Step 10 through Step 12 as shown in the configuration examples below. The Cisco ICS 7750 does not support layer 3 to layer 2 service-policy mapping in the Fast Ethernet main interface.


Step 10 Set the class-map:

MRP(config)#class-map l3-to-l2-high
MRP(config-cmap)#match ip dscp cs5
MRP(config)#class-map l3-to-l2-med 
MRP(config-cmap)#match ip dscp cs3
MRP(config)#class-map l3-to-l2-low
MRP(config-cmap)#match ip dscp cs1

Step 11 Set the policy map:

MRP(config)#policy-map output-l3-to-l2
MRP(config-pmap)#class l3-to-l2-high
MRP(config-pmap-c)#set cos 5
MRP(config-pmap-c)#exit
MRP(config-pmap)#class l3-to-l2-med
MRP(config-pmap-c)#set cos 3
MRP(config-pmap-c)#exit
MRP(config-pmap)#class l3-to-l2-low
MRP(config-pmap-c)#set cos 0

Step 12 Apply the policy map:

MRP(config)#interface fastEthernet 0/0.2
MRP(config-if)#service-policy output output-l3-to-l2
MRP(config-if)#^Z


Note To configure layer 2 to layer 3 service-policy mapping in the Fast Ethernet subinterface, perform Step 13 through Step 15 as shown in the configuration examples below.


Step 13 Set the class-map for layer 2 to layer 3 mapping:

MRP(config)#class-map match-all l2-to-l3-high
MRP(config-cmap)#match cos 4 5
MRP(config)#class-map match-all l2-to-l3-med
MRP(config-cmap)#match cos 2 3
MRP(config)#class-map match-all l2-to-l3-low
MRP(config-cmap)#match cos 0 1

Step 14 Set the policy map for layer 2 to layer 3 mapping:

MRP(config)#policy-map input-l2-to-l3
MRP(config-pmap)#class l2-to-l3-high
MRP(config-pmap-c)#set ip dscp cs5
MRP(config-pmap-c)#exit
MRP(config-pmap)#class l2-to-l3-med
MRP(config-pmap-c)#set ip dscp cs3
MRP(config-pmap-c)#exit
MRP(config-pmap)#class l2-to-l3-low
MRP(config-pmap-c)#set ip dscp cs1

Step 15 Apply the policy map for layer 2 to layer 3 mapping:

MRP(config)#interface fastethernet 0/0.2
MRP(config-if)#service-policy input input-l2-to-l3
MRP(config-if)#^Z



Note You must also ensure that the internal port of the SSP, which connects to the Flash-based MRP that is part of multiple VLANs, is configured as "dot1q trunk native vlan 1." See the "Configuring VLAN Support on the SSP" section.



Caution When you use the VLAN functionality on the Cisco ICS 7750, the routing Flash-based MRP card must be hardware and software VLAN-capable. If a Flash-based MRP card is configured as the only VLAN router, all call processing might be lost (along with access to other IP devices) if the MRP goes offline. Call processing services might be disrupted if the MRP is routing calls across VLANs. If fault tolerance is critical, you can use HSRP for more reliable and redundant routing. For additional information about HSRP, refer to Hot Standby Router Protocol (HSRP) Features and Functionality.

Configuring VLAN Support on the SSP

To configure VLAN functionality on your system, you must also manually configure the SSP. You must configure the internal ports on the SSP (Fast Ethernet 0/3 through Fast Ethernet 0/8) as "static access vlan 1" or "trunking native vlan 1," depending on the type of card that is installed in the chassis slot. (Fast Ethernet 0/1 and Fast Ethernet 0/2 are the ports on the SSP that connect to other switches and routers in the network.)

Be aware of the following when you configure VLAN support:

At least one external port of the SSP must be in trunking mode if VLAN support is enabled.

Only dot1q encapsulation is supported on the SSP ports.

A multi-VLAN port cannot be configured when a trunk port is configured on the SSP (this is a restriction on all Cisco Catalyst switches).

External switches must be configured separately from the Cisco ICS 7750 before the VLAN-enabled network will operate properly.

Internal ports can be configured as access ports on VLAN 1 (no VLAN tagging) or trunk ports native to VLAN 1 (tagging using 802.1q encapsulation). Only the port associated with a routing-enabled Flash-based MRP card can be configured as a trunk port.

The SSP is in VLAN Trunking Protocol (VTP) server mode by default. Alternatively, the SSP can be configured for VTP transparent mode or client mode.


Tip It is recommended that you run the SSP in VTP transparent mode if the SSP is not part of any VTP domain.


The SSP can support a maximum of 250 VLANs and 64 Spanning Tree Protocol (STP) instances. If there are more VLANs on the network than the supported number, pruning must be enabled on the trunk ports to reduce the number of VLANs on the SSP.

The most recent version of the SSP image (available with ICS System Software release 2.5.0) sends DHCP requests instead of BOOTP requests. This change appears as a class ID of ICS7750-SSP80 in the DHCP request.

The SPE and the MRP200 cards are designated as endpoints when using VLAN support on the Cisco ICS 7750; no VLAN configuration is required on these cards. However, switch port definitions might need to be configured on the SSP for the SPE or the MRP200. See Step 2 in the procedure below.


Note Because all system cards in the Cisco ICS 7750 chassis must be on the management VLAN (VLAN 1) when using VLAN functionality, the SPE on which Cisco CallManager is running must also be on VLAN 1. Cisco CallManager will not function properly if the SPE on which it is running has been configured for multiple subinterfaces.


Follow these steps to configure VLAN functionality on the SSP:


Step 1 Check to make sure that the management VLAN on the SSP is configured as VLAN 1; this is necessary for the SPE to access the SSP.

For example:

Switch#configure terminal
Switch(config)#interface VLAN 1
Switch(config-subif)#management

Step 2 Check to make sure that the interface for the SPE or the MRP200 (in the slot in which the SPE or the MRP200 is connected) is configured as "static access vlan 1." This is the default setting on a new SSP; if the configuration is missing, configure the interface for the SPE (or MRP200).

For example, if the SPE is installed in slot 5, the configuration on the SSP would look as follows:

Switch#configure terminal
Switch(config)#int fastEthernet 0/7
Switch(config-if)#switchport mode access
Switch(config-if)#switchport access vlan 1


Note The internal port of the SSP which connects to the SPE or to the MRP200 must be defined as "static access vlan 1."


Step 3 If you want to define class of service (CoS), set layer 2 CoS to 5 for all packets leaving from the port configured in Step 2:

Switch(config)#int fastEthernet 0/7
Switch(config-if)#switchport priority default 5

Step 4 Configure the internal port of the SSP that connects to the Flash-based MRP card (which is part of multiple VLANs) as "dot1q trunk native vlan 1."

For example, if the Flash-based MRP is installed in slot 1, the configuration on the SSP would look as follows:

Switch#configure terminal
Switch(config)#interface fastEthernet 0/3
Switch(config-if)#switchport mode trunk
Switch(config-if)#switchport trunk encapsulation dot1q
Switch(config-if)#switchport trunk native vlan 1


Note The internal port of the SSP which connects to the Flash-based MRP, which would be part of multiple VLANs, must be configured as "dot1q trunk native vlan 1." If the Flash-based MRP is not acting as a VLAN router, you can configure the internal port of the SSP as "static access
vlan 1."


Step 5 Configure at least one of the external ports on the SSP that connect to other switches as "dot1q trunk" to enable the exchange of 802.1p/q header information between the switches.

For example:

Switch#configure terminal
Switch(config)#interface fastEthernet 0/1
Switch(config-if)#switchport mode trunk
Switch(config-if)#switchport trunk encapsulation dot1q
Switch(config-if)#switchport trunk native vlan 1


Backing Up and Restoring the VLAN Configuration on the SSP

The VLAN configuration on the SSP is saved in Flash memory under the vlan.dat file. If you have configured VLAN functionality on the Cisco ICS 7750, make sure that you back up and save the configuration file on the SSP.

Backing Up the vlan.dat File

The procedure for backing up the vlan.dat file is the same procedure that is used for copying a Flash file to a network server. Follow these steps to copy and restore data:


Step 1 To copy the vlan.dat file to a TFTP network server, use the following command in EXEC mode from the command prompt on the SSP:

Switch#copy flash:vlan.dat tftp://<ip address of server>/<dir>/<fn>

where <ip address of server> is the IP address of the TFTP server, <dir> is the directory name on the TFTP server, and <fn> is the filename to write on the TFTP server.

Step 2 You might also need to back up and restore the VLAN information, depending on the VTP mode of the SSP.

If the SSP is operating in VTP transparent mode, back up and restore the vlan.dat file, along with the configuration text. Reload the SSP for the vlan.dat file to take effect.

If the SSP is operating in VTP server or client mode, the SSP will relearn the VLAN configuration in the network through VTP advertisements. You will need to reset the correct VTP domain and password on the SSP. Refer to the "Configuring VLANs" section in the Catalyst 2900 XL and Catalyst 3500 XL Software Configuration Guide, Releases 12.0(5)WC4 and 12.0(5)WC5.

Problems with VTP advertisements might occur in the following situations:

If the configuration text contains "switchport access vlan 3" and no vlan.dat file, the SSP will go into VTP transparent mode. The SSP will create a vlan.dat file, using the current information in its configuration, and will choose a domain name of UPGRADE. In this situation, use the Cisco IOS CLI to configure the SSP for server or client mode, and define the correct domain name and password. The SSP should learn the VLAN configuration from the advertisements. For additional information, refer to the "Configuring VLANs" section in the Catalyst 2900 XL and Catalyst 3500 XL Software Configuration Guide, Releases 12.0(5)WC4 and 12.0(5)WC5.

If all switches in the network lose power and the SSP is the only server switch (with no vlan.dat file), the client will not advertise after it reboots. In this situation, all VLANs must be recreated on the server.

If the SSP (in server or client mode) is the only switch in the VTP domain, the SSP will not receive any advertisements; therefore, the vlan.dat file should be backed up and restored as in VTP transparent mode.


Tip It is recommended that you run the SSP in VTP transparent mode if the SSP is not part of any VTP domain.



Restoring the vlan.dat File

To restore the vlan.dat file and copy the system image into the current Flash configuration on the SSP, reload the SSP, and execute the following command:

Switch#copy tftp://<ip address of server>/<dir>/<fn> flash:vlan.dat

where <ip address of server> is the IP address of the TFTP server, <dir> is the directory name on the TFTP server, and <fn> is the filename to copy from on the TFTP server.


Note Refer to Loading and Maintaining System Images for additional information about copying images between Flash memory and a network server.


The following documentation is available to provide guidance on configuring VLANs on the Cisco ICS 7750:

For detailed information about VLAN support on the Cisco ICS 7750, refer to the Cisco ICS 7750 Installation and Configuration Guide.

For information about configuring VLANs, refer to the "Configuring VLANs" section in the Catalyst 2900 XL and Catalyst 3500 XL Software Configuration Guide, Releases 12.0(5)WC4 and 12.0(5)WC5.

For additional information about configuring components other than the components in the Cisco ICS 7750 (such as switches, phones, and PCs), refer to the following:

The "Creating and Maintaining VLANs" section in the Cisco IOS Desktop Switching Software Configuration Guide

The "Configuring the Switch Ports" section in the Catalyst 2900 XL and Catalyst 3500 XL Software Configuration Guide

The "Configuring VLAN Settings" section in Configuring and Verifying Network Settings on the Cisco IP Phone

Troubleshooting VLANs

Table 3-12 lists symptoms of and possible solutions for problems associated with VLANs.

Table 3-12 VLAN Problems and Solutions 

User Action
Effect
System Behavior
Solutions

DHCP Issues:

Turn on the Cisco IOS DHCP server on the Flash-based MRP card for non-overlapping subnets (other VLANs) with System Manager.

All devices on other VLANs will get their IP addresses from the Flash-based MRP card.

Expected.

No action is required.

Turn on the Cisco IOS DHCP server on the Flash-based MRP card for VLAN 1, along with Cisco Network Registrar (CNR), and use the same IP address scope as CNR.

The Flash-based MRP will respond to DHCP requests from other system cards (MRP200, SSP, and core software SPE) and from other devices on VLAN 1.

The MRP200 will not use the IP address. It must be manually reclaimed. The SSP and core software SPE will accept the IP addresses. Duplicate IP addresses could be allocated by CNR.

Make sure that the IP address scopes on the DHCP server and CNR do not overlap.

Run ICSConfig to configure the IP address for the MRP200, and make sure there are no duplicate IP addresses.

Turn on the Cisco IOS DHCP server on the Flash-based MRP card for VLAN 1, along with CNR, but use a different IP address scope than CNR.

The Flash-based MRP card will respond to DHCP requests from the MRP200, SSP, and the core software SPE, and from other devices on VLAN 1.

The MRP200 will not use the IP address. It must be manually reclaimed. The SSP and core software SPE will accept the IP addresses. System Manager will recognize the IP addresses, but you will be prompted to change them by ICSConfig.

Run ICSConfig to configure the IP address for the MRP200.

Change the IP addresses of the SSP and core software SPE cards when prompted by ICSConfig.

Misconfiguration and Bootup Issues:

A new MRP200 is inserted in a switch port that is not configured as "static access vlan 1." A helper address is set to route the MRP to forward BOOTP requests to System Manager.

The MRP200 will not boot.

ICSConfig will not be able to detect the MRP200, and will report error code 301/200.

The ICSConfig diagnostic tool will detect the VLAN misconfiguration. Follow the links provided in the online error message to resolve the problem.

A new MRP300 or core software SPE is inserted in a switch port that is not configured as "static access vlan 1." A helper address is set to route the MRP to forward DHCP requests to System Manager.

The MRP300 will not receive its configuration and IP address. The core software SPE will not receive its IP address.

ICSConfig will not be able to detect the MRP300 or the core software SPE, and will report error code 301/200.

The ICSConfig diagnostic tool will detect the VLAN misconfiguration. Follow the links provided in the online error message to resolve the problem.

Moving Cards Between Slots:

The MRP200, SPE running System Manager, or core software SPE is moved into a switch port that is configured for "dot1q trunk native vlan 1."

The MRP200 or SPE will be flooded with additional traffic, which will be dropped.

ICSConfig will work and recognize the system cards.

Move the MRP200, SPE running System Manager, or core software SPE into a switch port that is configured for "static access vlan 1," or reconfigure the switch port of the slot in which the card is installed.

The MRP300 is moved into a switch port that is configured for "static access vlan 1."

If the MRP300 is acting as the VLAN router, VLANs other than VLAN 1 will not operate correctly.

Call processing will be affected because the Cisco IP phones will not be able to contact Cisco CallManager.

Move the MRP300 into a switch port that is configured for "dot1q trunk native vlan 1," or reconfigure the switch port of the slot in which the MRP300 is installed.

Misconfiguration of the SSP Management VLAN:

The SPE running System Manager is on VLAN 1, but the SSP management VLAN is not configured as VLAN 1.

Pings will be successful between the SPE running System Manager and the SSP.

The SSP will be discovered, but ICSConfig will report an error showing that the SSP is on a different subnet.

Reconfigure the SSP management VLAN as VLAN 1.

Follow the links provided in the ICSConfig diagnostic tool online error message to resolve the problem.

Misconfiguration of VLAN Settings on the SSP Switch Port:

The SPE running System Manager and the SSP management VLAN are not configured on VLAN 1.

Pings will be successful between the SPE running System Manager and the SSP.

The SSP will be discovered by ICSConfig, but the Inventory Discovery (ID) module will report the card type as "unknown." The SPE running System Manager will not be detected.

Reconfigure the SPE and the SSP to reside on VLAN 1.

Follow the links provided in the ICSConfig diagnostic tool online error message to resolve the problem.

The MRP200, MRP300, or core software SPE is inserted into a switch port that is not configured as "static access vlan 1."

The MRPs will not receive their configuration and IP addresses. The core software SPE will not receive its IP address.

Card types will be reported as "unknown." ICSConfig will report the error codes 400/301/200.

ICSConfig will detect the VLAN misconfiguration. Follow the links provided in the online error message to resolve the problem.

Misconfiguration of VLAN Settings on the Flash-Based MRP Cards

The IP address on the main interface (FE 0/0) is removed, and there is no other native VLAN subinterface.

The IP address on the other VLAN will be advertised by
Cisco Discovery Protocol (CDP).

ID will detect the different subnet addresses and report error code 301.

Do not remove the address under the main interface.

Follow the links provided in the online error message to resolve the problem

IP addresses under all subinterfaces are removed.

No IP addresses will be advertised by CDP.

ID will detect no IP address and will report error code 300.

Follow the links provided in the online error message to resolve the problem.

The IP address on the main interface is removed, and it is added under the subinterface of native vlan 1.

The IP address under the subinterface will be advertised by CDP.

ID will discover the IP address, but when it tries to change the IP address on the card, it will fail with an error code.

Follow the links provided in the online error message to resolve the problem.

Both the main interface and the subinterface for "native vlan 1" are allocated IP addresses.

The IP address under the main interface will be unreachable, and the IP address under the subinterface will be advertised by CDP.

If the IP address under the subinterface is in the same subnet as the SPE running System Manager, discovery will be successful but ID will fail when it tries to change the IP address. If the IP address under the subinterface is on a different subnet than System Manager, ID will report an error showing a "different subnet."

Follow the links provided in the online error message to resolve the problem.