Table of Contents

Initial Troubleshooting

This chapter describes the first steps you should take when you start troubleshooting your switch router, and contains the following sections:

Online Diagnostics

This section describes the online diagnostics available for troubleshooting your switch router. Online diagnostics provide the following types of tests:

The switch router displays an error message on the console when it detects a hardware failure or problem.

Note   Online diagnostic tests only run on the primary route processor.

Processor Loopback Test (Catalyst 8540 CSR)

The processor loopback test detects failures in the route processor to switch processor interface. The test sends a packet from the route processor to each switch processor which then loops back to the route processor. This test can run on the switch router without any port adapters or interface modules installed.

Note   The size of the packet and frequency of the test are configurable to minimize the impact on system performance.

Accessibility Test

The accessibility tests ensure connectivity, at a configurable interval, between all of the following:

OIR Test

Online insertion and removal (OIR) tests check the functioning of the switch fabric and interfaces on a per-port basis. The switch router performs these tests when the system boots up and when you insert an interface module into a slot. The OIR test sends a packet to the interface loopback and expects to receive it within a certain time period. If the packet does not reach the port within the expected time period, or the received packet is corrupted, an error is registered and the port is changed to an administrative down state. Packets that are 1000 bytes in size are used in the test.

The OIR tests support all the enhanced Gigabit Ethernet interface modules, with the exception of the OC-3c and OC12c ATM uplink interface module. In addition, OIR tests are not supported on the Fast Ethernet or Gigabit Ethernet interface modules.

OIR is enabled by default on the Catalyst 8540 CSR. To disable it, enter the no diag online oir command. To enable OIR, refer to commands in the "Configuring Online Diagnostics (Catalyst 8540 CSR)" section.

Snake Test

The snake test detects and reports port-to-port connectivity failures. The snake test establishes connections across all the active ports in the switch router, originating and terminating at the primary route processor. The route processor establishes a connection by sending a packet to each port in turn, which then terminates at the route processor. If the packet does not reach the route processor within the expected time period, or the received packet is corrupted, further testing is performed to isolate and disable the port causing the problem. The frequency of the test is configurable to minimize the impact on system performance.

The snake test supports all the enhanced Gigabit Ethernet interface modules on the Catalyst 8540 MSR and Catalyst 8540 CSR.

Configuring Online Diagnostics (Catalyst 8540 CSR)

To configure online diagnostics, use the following global configuration commands:

Examples

The following example shows how to enable all online diagnostic tests:

The following example shows how to change the frequency of the access test:

Displaying the Online Diagnostics Configuration and Results (Catalyst 8540 CSR)

To display the online diagnostics configuration and results, use the following EXEC command:

Example

The following example shows how to display detailed access test information:

Example

The following example shows how to display OIR test status and details:

Checking DDTs Database and Release Notes for Workarounds

This section describes different methods you can use to check for IOS software bugs (defect tracking tool numbers [DDTs]) in your version of IOS software. Often, your problems with the switch router have been fixed or a workaround has been determined in a more recent version of software.

There are two ways to check for known bugs in the IOS software:

Using Bug Navigator II

Bug Navigator II is a DDT search tool you can use to search the DDT database and ask either of two types of questions:

To search the DDT database, you can access Bug Navigator II on the World Wide Web at http://www.cisco.com/support/bugtools/bugtool.shtml . Then perform the following steps:

Step 2   Read the "Bug Navigator II Help" instructions.

Step 3   From the "Cisco Hardware" list select your switch router under "Catalyst 8500 Series Switches." The Bug Navigator search tool replaces "Bug Navigator II Help" (in the right frame of the page).

Step 4   Select the following from the drop down menus:

Step 5   Click the "Search" button.

The entire window will be replaced with a "Bug Search Results" window with a list of DDTs containing your search criteria. Look at the Bug reports listed in the "titles" column. An existing bug entry that describes the problem you are having may have been fixed in a more recent version of the IOS software. Look in the "Fixed-in" column for a later version of the IOS software. All you might have to do to solve your problem is upgrade your software.

If a software upgrade is not listed as a way to solve your problem, double-click on the bug title and read the DDT details; a workaround might be listed there.

Checking IOS Release Notes

Release notes describe the features and caveats for Cisco IOS software releases. The release notes are listed by both product and IOS release number.

Note   All information pertains to both the Catalyst 8540 CSR and Catalyst 8510 CSR platforms, unless differences between the platforms are noted in the text.

The "Caveats" section of the release note lists known caveats for the switch router by tracking the DDTS number and the release number, and indicates whether the caveat has been corrected.

The "Caveat Symptoms and Workarounds" section summarizes caveat symptoms and suggested workarounds for the switch routers. You can also search thorough this section online, using either a word string or the DDTS number.

Troubleshooting Hardware and Software Version Problems

A common error you may encounter is the incompatibility of hardware modules and the IOS software version needed to perform a particular function. Check the Hardware and Software Compatibility Matrix document, available online, to confirm that you are using IOS software that supports the various hardware components installed in your switch router.

Verifying Hardware and Software Versions

Display the hardware and software versions to ensure that they are the most recent. Very old hardware and software versions (two or three versions back) can have caveats that have been fixed in more recent versions. Use the following EXEC commands to display the version information:

To verify the hardware and software versions, use the following steps:

IOS (tm) L3 Switch/Router Software (C8540CSR-IN-M), Version 12.0(10)W5(18c)

Step 2   Use the show hardware command to display the hardware revision levels.

Slot Ctrlr-Type Part No. Rev Ser No Mfg Date RMA No. Hw Vrs Tst EEP

Step 3   Verify that the hardware version is listed in the Hw Vrs column.

Step 4   Use the show hardware detail command to display detailed information about the hardware, including the functional image versions.

Step 5   Use the show functional-image command to display detailed information about the functional images for the route processors, switch processors, and Fast Ethernet and Gigabit Ethernet interface modules for the switch router (in this example, the Catalyst 8540 CSR). The following example shows how to display the functional image for the route processor in slot 4:

Functional Version of the FPGA Image: 4.8

#HardwareRequired: 100(3.0-19,4.0-19,5.0-19)

Step 6   Verify the FunctionalVersion and #HardwareRequired fields to determine the FPGA version and the hardware version required for the FPGA. Compare this with the hardware version using the show hardware command output displayed in Step 2 and Step 4. If the FPGA version does not support the hardware version, download a new FPGA image, upgrade the hardware, or both.

Finding the Image

Use the Hardware and Software Compatibility Matrix (found on Cisco.com) to determine the correct IOS software image for your hardware configuration.

If you determine that you need to upgrade your IOS image, refer to the Cisco.com web page, and follow these steps to find and download the image you need:

Step 2   Under Service and Support, select Software Center.

Step 3   The Technical Assistance Center page is displayed. Under Tools, select IOS Upgrade Planner.

The Cisco IOS Planner page provides greater flexibility to browse for your preferred software. You can view all major releases, all platforms, and all software features from a single interface. Choosing a platform, a maintenance release, or a software feature the planner automatically limits the other menu choices, based on your selections until you arrive at your preferred software.

Step 4   In the Select Platform column, select either of the following:

Step 5   In the Select Release column, select the IOS software release you want to download.

Step 6   Read the requirements, and if your hardware configuration meets the requirements, click the agreement button.

Step 7   Select the file to download to your switch router.

To upgrade the IOS image on your platform, continue with the following section.

IOS Upgrade Procedures

If your IOS image is not the most recent, you can download the IOS image from Cisco.com to the switch router.

Refer to the Configuration Fundamentals Configuration Guide for details on the following frequently performed tasks:

Modifying, Downloading, and Maintaining System Images

The following tasks are performed frequently to maintain system image files:

Maintaining Functional Images

You can load functional images used by certain hardware controllers in the switch router. The following sections describe the function and maintenance of functional images:

Understanding Functional Images

Functional images provide the low-level operating functionality for various hardware controllers. On hardware controllers within system programmable devices, such as Field Programmable Gate Arrays (FPGAs) and Erasable Programmable Logic Devices (EPLDs), the hardware functional images can be reprogrammed independently of loading the system image, and without removing the devices from the controller.

All new hardware you purchase is shipped with the functional images loaded. Loading a different functional image is required only when upgrading or downgrading functional image versions.

FPGA Upgrade Procedures

If the functional image is not the most recent, you can download the functional image to the switch router from Cisco.com. Use the reprogram command to update the functional image to the processor or interface module. The following example shows how to reprogram the route processor in slot 4 with the functional image fi-c8540-rp.A.4-8.bin from the Flash PC card in slot 0:

Note   You can only enter the reprogram command from the console session prompt.

You can find the functional images and release notes for one of the following on Cisco.com, on the LAN Switching Products site:

Note   After you have determined the hardware and software versions on the switch router, check the release notes and DDTS database for symptoms resembling those you are observing. Often, the problem has already been discovered and a workaround has been provided.

Troubleshooting Switch Route Processor Redundancy and Enhanced High System Availability (Catalyst 8540 CSR and Catalyst 8540 MSR)

The Catalyst 8540 MSR and Catalyst 8540 CSR support, redundant route processor operation with dual route processors. In addition, Enhanced High System Availability (EHSA) is provided in the switching fabric when three switch processors are installed in the chassis. The troubleshooting processes for these features are described in the following sections:

Route Processor Redundant Operation (Catalyst 8540 CSR and Catalyst 8540 MSR)

The Catalyst 8540 MSR and Catalyst 8540 CSR supports fault tolerance by allowing a secondary route processor to take over if the primary fails. This secondary, or redundant, route processor runs in standby mode. In standby mode, the secondary route processor is partially booted with the Cisco IOS software; however, no configuration is loaded.

At the time of a switchover, the secondary route processor takes over as primary and loads the configuration as follows:

The former primary then becomes the secondary route processor.

Note   If the secondary route processor is unavailable, a major alarm is reported. Use the show facility-alarm status command to display the redundancy alarm status.

For detailed redundant route processor configuration information, refer to the "Initially Configuring the ATM Switch Router" chapter in the ATM Switch Router Software Configuration Guide .

Troubleshooting Route Processor Redundancy (Catalyst 8540 CSR and Catalyst 8540 MSR)

For redundant route processors to function correctly, your switch router's route processors must meet all of the following requirements:

A common error you may encounter is the incompatibility of hardware modules and the IOS software version needed to perform a particular function. Check the Hardware and Software Compatibility Matrix document, available on-line, to confirm that you are using IOS software that supports the various hardware components installed in your switch router.

Troubleshooting Hardware and Software Versions of Redundant Route Processors

To troubleshoot the route processor hardware and software versions for redundancy, use the following commands:

To confirm that your switch router route processors meets the redundancy requirements, complete the following steps:

ROM: System Bootstrap, Version 12.0(4.6)W5(13), RELEASE SOFTWARE

Step 2   Verify the ROM field. It indicates the release of IOS software loaded and running on the primary route processor.

Step 3   Use the show hardware detail command to compare the hardware versions of the primary and secondary route processors.

slot: 4/* Controller-Type : Route Proc

slot: 4/0 Controller-Type : Netclk Module

slot: 8/* Controller-Type : Route Proc

slot: 8/0 Controller-Type : Netclk Module

Step 4   In the slots labeled Controller-Type : Route Proc, compare the Part Number, FPGA, and H/W Version fields. These numbers must all match, or redundancy will not function correctly on your switch router.

Step 5   Use the show redundancy command to check the configuration and status of the route processors.

Primary

Slot: 8

    Last Running Config. Sync: 2 weeks, 4 days, 2 hours, 28 minutes

    Last Startup Config. Sync: 2 weeks, 4 days, 2 hours, 34 minutes

Last Restart Reason: Normal boot

Secondary

State:                         UP

Slot:                          4

Step 6   Verify the Primary, Secondary, and Slot fields. They indicate in which slot the primary route processor is configured.

Step 7   Verify the Last Running Config. Sync and Last Startup Config. Sync fields. They indicate the last time the running configuration and startup configuration were synchronized between the route processors.

Troubleshoot Redundant Route Processor Functions

To troubleshoot the route processor functions for redundancy, use the following commands:

Follow these steps to troubleshoot the route processor redundancy on the switch router:

ATM0 0 245 PVC ATM-SEC0 0 29 IPC UP

Step 2   Verify the ATM-SEC0 field and confirm the PVC is up.

Step 3   Use the show atm status command to confirm the status of the PVC and other states.

ATM0 UP up n/a UpAndNormal Idle n/a

ATM-SEC0 UP up done n/a none n/a

Step 4   Verify that the IF Status field is up.

Step 5   Verify that the ILMI Addr Reg State field is UpAndNormal.

If you determine that redundancy is configured incorrectly, refer to the "Initially Configuring the ATM Switch Router" chapter in the ATM  Switch Router Software Configuration Guide.

Troubleshooting Switch Processor Redundancy with HSRP (Catalyst 8540 CSR and Catalyst 8540 MSR)

Enhanced High System Availability (EHSA) is provided in the switching fabric when three switch processors are installed in the chassis. These features and their configuration are described in the "Initially Configuring the ATM Switch Router" chapter in the
ATM Switch Router Software Configuration Guide .

To troubleshoot the EHSA of the switch processor, use the following commands:

Follow these steps to troubleshoot the redundant switch processors on the switch router:

The currently preferred switch card slots are slot: 5 and slot: 7

The currently active switch card slots are slot: 5 and slot: 7

Step 2   Use the show switch fabric command to confirm the status of the switch processors and their location.

Number of Switch Cards present in the Chassis: 3

     5 EVEN ACTIVE

     6 STANDBY           STANDBY

     7 ODD ACTIVE

Step 3   Check the field Number of Switch Cards present in the Chassis. If you only have two switch processors, your switch router does not have EHSA. With three switch processors installed, if either of the two active switch processors fail, the third processor takes over.

Step 4   Check the SWC_STATUS field. This fields identifies the active and standby switch processors.

If you determine that redundancy is configured incorrectly, refer to the "Initially Configuring the ATM Switch Router" chapter in the ATM Switch Router Software Configuration Guide .