Initial Troubleshooting

Table of Contents

Initial Troubleshooting
Online Diagnostics
Checking DDTs Database and Release Notes for Workarounds
Troubleshooting Hardware and Software Version Problems
Troubleshooting Switch Route Processor Redundancy and Enhanced High System Availability (Catalyst 8540 CSR and Catalyst 8540 MSR)

Initial Troubleshooting


This chapter describes the first steps you should take when you start troubleshooting your switch router, and contains the following sections:

Online Diagnostics

This section describes the online diagnostics available for troubleshooting your switch router. Online diagnostics provide the following types of tests:

  • Processor Loopback Test (Catalyst 8540 CSR)
  • Accessibility tests between the route processor and the interface modules and the route processor and the switch processor.
  • Online insertion and removal (OIR) diagnostic tests.
  • Snake tests through the switch router to ensure connectivity between the ports.

The switch router displays an error message on the console when it detects a hardware failure or problem.


Note   Online diagnostic tests only run on the primary route processor.

Processor Loopback Test (Catalyst 8540 CSR)

The processor loopback test detects failures in the route processor to switch processor interface. The test sends a packet from the route processor to each switch processor which then loops back to the route processor. This test can run on the switch router without any port adapters or interface modules installed.


Note   The size of the packet and frequency of the test are configurable to minimize the impact on system performance.

Accessibility Test

The accessibility tests ensure connectivity, at a configurable interval, between all of the following:

  • Interface modules
  • Active switch processor
  • Standby switch processor, if it is present

OIR Test

Online insertion and removal (OIR) tests check the functioning of the switch fabric and interfaces on a per-port basis. The switch router performs these tests when the system boots up and when you insert an interface module into a slot. The OIR test sends a packet to the interface loopback and expects to receive it within a certain time period. If the packet does not reach the port within the expected time period, or the received packet is corrupted, an error is registered and the port is changed to an administrative down state. Packets that are 1000 bytes in size are used in the test.

The OIR tests support all the enhanced Gigabit Ethernet interface modules, with the exception of the OC-3c and OC12c ATM uplink interface module. In addition, OIR tests are not supported on the Fast Ethernet or Gigabit Ethernet interface modules.

OIR is enabled by default on the Catalyst 8540 CSR. To disable it, enter the no diag online oir command. To enable OIR, refer to commands in the "Configuring Online Diagnostics (Catalyst 8540 CSR)" section.

Snake Test

The snake test detects and reports port-to-port connectivity failures. The snake test establishes connections across all the active ports in the switch router, originating and terminating at the primary route processor. The route processor establishes a connection by sending a packet to each port in turn, which then terminates at the route processor. If the packet does not reach the route processor within the expected time period, or the received packet is corrupted, further testing is performed to isolate and disable the port causing the problem. The frequency of the test is configurable to minimize the impact on system performance.

The snake test supports all the enhanced Gigabit Ethernet interface modules on the Catalyst 8540 MSR and Catalyst 8540 CSR.

Configuring Online Diagnostics (Catalyst 8540 CSR)

To configure online diagnostics, use the following global configuration commands:

Command Purpose

diag online

Enables all of the online diagnostic tests.

diag online access

Enables only the accessibility diagnostic test.

diag online access freq [seconds]

Configures the frequency of the accessibility diagnostic tests. The default frequency is every 10 seconds.

diag online oir

Enables only the OIR test.

diag online oir pktsize [bytes]

Specifies the packet size for the OIR test. The default size is 1000 bytes.

diag online snake

Enables only the snake test.

diag online snake timer [seconds]

Specifies the time interval for the snake test. The default interval is 10 seconds.

no diag online [access | oir | snake]

Disables the online diagnostic tests.

clear counters diag online {access | oir | snake | all}

Clears the online diagnostic test counters.

debug diag online [access | oir | snake]

Enables debugging of online diagnostic tests.

no debug diag online [access | oir | snake]

Disables debugging of online diagnostic tests.

Examples

The following example shows how to enable all online diagnostic tests:

Router(config)# diag online
ONLINE-DIAG: Enabling all Online Diagnostics tests

The following example shows how to change the frequency of the access test:

Router(config)# diag online access freq 20
ONLINE-DIAG: Online Access Test Frequency set to 20 sec

Displaying the Online Diagnostics Configuration and Results (Catalyst 8540 CSR)

To display the online diagnostics configuration and results, use the following EXEC command:

Command Purpose

show diag online [details | status] [access | oir | snake]

Displays information about the online diagnostics tests and the test results.

Example

The following example shows how to display detailed access test information:

Switch# show diag online details access
======== Online Access Test Details ========
Current Test Status : Test is Enabled
Current Frequency of Access Test : 10 seconds
Slot Card-Type Iteration Success Failure Last Failure
---- ---------- ---------- ------- ------- ------------
0/* K1 GIGETHERNET 114536 114536 0 ----
1/* CMPM Card 114536 114536 0 ----
1/0 XPIF POS OC12 P 114536 114536 0 ----
2/* CMPM Card 114536 114536 0 ----
2/0 XPIF POS OC12 P 114536 114536 0 ----
2/1 XPIF GIGE PAM 114536 114536 0 ----
3/* CMPM Card 114536 114536 0 ----
3/0 XPIF ATM OC3 PM 114536 114536 0 ----
3/1 XPIF GIGE PAM 114536 114536 0 ----
5/* Switch Card 114536 114536 0 ----
6/* Switch Card 27126 0 27126 1w6d
7/* Switch Card 114536 114536 0 ----
10/* ETHERNET PAM 27119 27119 0 ----
11/* CMPM Card 114536 114536 0 ----
11/0 XPIF GIGE PAM 114536 114536 0 ----
11/1 XPIF GIGE PAM 114536 114536 0 ----
12/* CMPM Card 114537 114537 0 ----
12/0 XPIF ATM OC12 P 114537 114537 0 ----
12/1 XPIF GIGE PAM 114537 114537 0 ----
Example

The following example shows how to display OIR test status and details:

Switch# show diag online oir
======== OIR Test Status and Details ========
======== Online OIR Test Status ========
Current Test Status :Test is Enabled
-------- Bootup OIR status --------
Port Card Type Pkt Size Result Test Time LOOP
_______ ___________ _________ ___________________ ______________ ____
00/0/00 ETHERNET PA 1000 OIR_SUCCESS 00:01:54 PIF
00/0/01 ETHERNET PA 1000 OIR_SUCCESS 00:01:52 PIF
00/0/02 ETHERNET PA 1000 OIR_SUCCESS 00:01:50 PIF
00/0/03 ETHERNET PA 1000 OIR_SUCCESS 00:01:47 PIF
00/0/04 ETHERNET PA 1000 OIR_SUCCESS 00:01:55 PIF
00/0/05 ETHERNET PA 1000 OIR_SUCCESS 00:01:53 PIF
00/0/06 ETHERNET PA 1000 OIR_SUCCESS 00:01:51 PIF
00/0/07 ETHERNET PA 1000 OIR_SUCCESS 00:01:49 PIF
00/0/08 ETHERNET PA 1000 OIR_SUCCESS 00:02:04 PIF
00/0/09 ETHERNET PA 1000 OIR_SUCCESS 00:02:01 PIF
00/0/10 ETHERNET PA 1000 OIR_SUCCESS 00:01:59 PIF
00/0/11 ETHERNET PA 1000 OIR_SUCCESS 00:01:56 PIF
00/0/12 ETHERNET PA 1000 OIR_SUCCESS 00:02:05 PIF
00/0/13 ETHERNET PA 1000 OIR_SUCCESS 00:02:03 PIF
00/0/14 ETHERNET PA 1000 OIR_SUCCESS 00:02:00 PIF
00/0/15 ETHERNET PA 1000 OIR_SUCCESS 00:01:58 PIF
01/0/00 XPIF POS OC 1000 OIR_SUCCESS 00:01:48 PIF
01/1/00 XPIF GIGE P 1000 OIR_SUCCESS 00:01:57 PIF
02/0/00 GIGETHERNET 1000 OIR_SUCCESS 00:02:07 PIF
02/0/01 GIGETHERNET 1000 OIR_SUCCESS 00:02:10 PIF
03/0/00 XPIF GIGE P 1000 OIR_SUCCESS 00:02:08 PIF
03/1/00 XPIF GIGE P 1000 OIR_SUCCESS 00:02:11 PIF
-------- Latest OIR status --------
********No Other OIR tests not performed ********
======== Online OIR Test Details ========
Current Test Status :Test is Enabled
-------- Previous failure details ----------
******* No failures in OIR tests *******
-------- Complete details --------
Port Tx Pkt Rx Pkt Success Failure Total Tests
_______ _________ _________ _________ _________ ___________
00/0/00 1 1 1 0 1
00/0/01 1 1 1 0 1
00/0/02 1 1 1 0 1
00/0/03 1 1 1 0 1
00/0/04 1 1 1 0 1
00/0/05 1 1 1 0 1
00/0/06 1 1 1 0 1
00/0/07 1 1 1 0 1
00/0/08 1 1 1 0 1
00/0/09 1 1 1 0 1
00/0/10 1 1 1 0 1
00/0/11 1 1 1 0 1
00/0/12 1 1 1 0 1
00/0/13 1 1 1 0 1
00/0/14 1 1 1 0 1
00/0/15 1 1 1 0 1
01/0/00 1 1 1 0 1
01/1/00 1 1 1 0 1
02/0/00 1 1 1 0 1
02/0/01 1 1 1 0 1
03/0/00 1 1 1 0 1
03/1/00 1 1 1 0 1

Checking DDTs Database and Release Notes for Workarounds

This section describes different methods you can use to check for IOS software bugs (defect tracking tool numbers [DDTs]) in your version of IOS software. Often, your problems with the switch router have been fixed or a workaround has been determined in a more recent version of software.

There are two ways to check for known bugs in the IOS software:

Using Bug Navigator II

Bug Navigator II is a DDT search tool you can use to search the DDT database and ask either of two types of questions:

  • Symptom Diagnostics (for example, "What defect is causing my current symptoms?")
  • Upgrade Planning (for example, "What software release is best for the features I am interested in?")

To search the DDT database, you can access Bug Navigator II on the World Wide Web at http://www.cisco.com/support/bugtools/bugtool.shtml . Then perform the following steps:


Step 1   If you are not already logged in to Cisco.com, enter your user name and password at the login prompt.

Step 2   Read the "Bug Navigator II Help" instructions.

Step 3   From the "Cisco Hardware" list select your switch router under "Catalyst 8500 Series Switches." The Bug Navigator search tool replaces "Bug Navigator II Help" (in the right frame of the page).

Step 4   Select the following from the drop down menus:

  • Version
  • Revision
  • Severity

  • Note   As an option, you can enter words or phrases (separated by commas) in the data entry field to limit your search.

Step 5   Click the "Search" button.

The entire window will be replaced with a "Bug Search Results" window with a list of DDTs containing your search criteria. Look at the Bug reports listed in the "titles" column. An existing bug entry that describes the problem you are having may have been fixed in a more recent version of the IOS software. Look in the "Fixed-in" column for a later version of the IOS software. All you might have to do to solve your problem is upgrade your software.

If a software upgrade is not listed as a way to solve your problem, double-click on the bug title and read the DDT details; a workaround might be listed there.





Checking IOS Release Notes

Release notes describe the features and caveats for Cisco IOS software releases. The release notes are listed by both product and IOS release number.


Note   All information pertains to both the Catalyst 8540 CSR and Catalyst 8510 CSR platforms, unless differences between the platforms are noted in the text.

The "Caveats" section of the release note lists known caveats for the switch router by tracking the DDTS number and the release number, and indicates whether the caveat has been corrected.

The "Caveat Symptoms and Workarounds" section summarizes caveat symptoms and suggested workarounds for the switch routers. You can also search thorough this section online, using either a word string or the DDTS number.

Troubleshooting Hardware and Software Version Problems

A common error you may encounter is the incompatibility of hardware modules and the IOS software version needed to perform a particular function. Check the Hardware and Software Compatibility Matrix document, available online, to confirm that you are using IOS software that supports the various hardware components installed in your switch router.

Verifying Hardware and Software Versions

Display the hardware and software versions to ensure that they are the most recent. Very old hardware and software versions (two or three versions back) can have caveats that have been fixed in more recent versions. Use the following EXEC commands to display the version information:

Command Purpose

show version

Displays the software version information.

show hardware [detail]

Displays detailed hardware information including revision level and version.

show functional-image slot slot

Displays functional image information.

To verify the hardware and software versions, use the following steps:


Step 1   Display the system software version:

Switch# show version
Cisco Internetwork Operating System Software
IOS (tm) L3 Switch/Router Software (C8540CSR-IN-M), Version 12.0(10)W5(18c)
Copyright (c) 1986-2000 by cisco Systems, Inc.
Compiled Sat 19-Aug-00 00:11 by integ
Image text-base: 0x60010930, data-base: 0x608CA000
ROM: System Bootstrap, Version 12.0(4.6)W5(13) RELEASE SOFTWARE
8540CSR uptime is 2 minutes
System restarted by reload
System image file is "slot0:cat8540c-in-mz.120-10.W5.18c.bin"
cisco C8540CSR (R5000) processor with 262144K/256K bytes of memory.
R5000 processor, Implementation 35, Revision 2.1 (512KB Level 2 Cache)
Last reset from power-on
1 Ethernet/IEEE 802.3 interface(s)
16 FastEthernet/IEEE 802.3 interface(s)
13 Gigabit Ethernet/IEEE 802.3z interface(s)
3 ATM network interface(s)
2 Packet over SONET network interface(s)
505K bytes of non-volatile configuration memory.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
16384K bytes of Flash internal SIMM (Sector size 256K).
Configuration register is 0x0

Step 2   Use the show hardware command to display the hardware revision levels.

Switch# show hardware
C8540 named Router, Date: 13:59:13 UTC Mon Jan 3 2000
Slot Ctrlr-Type Part No. Rev Ser No Mfg Date RMA No. Hw Vrs Tst EEP
---- ------------ ---------- -- -------- --------- -------- ------- --- ---
0/* K1 GIGETHERN 73-3324-03 A0 0336441Y Oct 13 99 0 3.4
1/* CMPM Card 73-3944-03 09 03445724 Nov 09 99 3.0
1/0 XPIF POS OC1 73-4462-01 09 034558YP Nov 09 99 1.1
2/* CMPM Card 73-3944-03 A0 04087BW8 Mar 22 00 0 3.0
2/0 XPIF POS OC1 73-4462-01 A0 04046NRQ Mar 22 00 0 2.0
2/1 XPIF GIGE PA 73-4167-05 A0 04097GRJ Mar 22 00 0 1.0
3/* CMPM Card 73-3944-03 A0 04087BXK Mar 15 00 0 3.0
3/0 XPIF ATM OC3 73-3889-03 A0 040879AA Mar 15 00 0 1.0
3/1 XPIF GIGE PA 73-4167-05 A0 04097GQA Mar 15 00 0 1.0
4/* Route Proc 73-3775-04 A0 03201VCZ Oct 04 99 0 5.7
5/* Switch Card 73-3327-08 A0 032428ZR Jun 15 99 0 8.0
7/* Switch Card 73-3327-08 A0 032428ZE Jun 15 99 0 8.0
10/* ETHERNET PAM 73-3753-04 A0 03020FCA Sep 22 99 0 4.1
11/* CMPM Card 73-3944-03 A0 04087BY5 Mar 13 00 0 3.0
11/0 XPIF GIGE PA 73-4415-05 A0 04087AZE Mar 13 00 0 1.0
11/1 XPIF GIGE PA 73-4415-05 A0 04087AZL Mar 13 00 0 1.0
12/* CMPM Card 73-3944-03 A0 04087BWS Mar 14 00 0 3.0
12/0 XPIF ATM OC1 73-3889-03 A0 040879AO Mar 14 00 0 1.0
12/1 XPIF GIGE PA 73-4167-05 A0 04107N8R Mar 14 00 0 1.0
DS1201 Backplane EEPROM:
Model Ver. Serial MAC-Address MAC-Size RMA RMA-Number MFG-Date
------ ---- -------- ------------ -------- --- ---------- -----------
C8540 2 12237014 00D0BA1D3200 1024 0 0 Jun 18 1999
cubi version : 11
Power Supply:
Slot Part No. Rev Serial No. RMA No. Hw Vrs Power Consumption
---- ---------------- ---- ----------- ----------- ------- -----------------
0 34-0829-02 A000 APQ02450080 00-00-00-00 1.0 2746 cA

Step 3   Verify that the hardware version is listed in the Hw Vrs column.

Step 4   Use the show hardware detail command to display detailed information about the hardware, including the functional image versions.

Switch# show hardware detail
<Information deleted>
slot: 3/0 Controller-Type : XPIF ATM OC3 PM - 1 Port SM_IR
Part Number: 73-3889-03 Revision: A0
Serial Number: CAB040879AA Mfg Date: Mar 15 00
RMA Number: 0 H/W Version: 1.0
FPGA Version: 1.14
XPIF Version: 3001 CAM size: 64 KB
Ucode Version: 1.0 CAM Type: Private TCAM
Port Phy Setup
Port 0: DONE
Optical Line Daughter Card Serial EEPROM:
Part Number: 73-3975-02 Revision: A0
Serial Number: CAB0407768M Mfg Date: 2000/04/10
RMA Number: 0 HW Rever: 1.0
TCAM Daughter Card Serial EEPROM:
Part Number: 73-3970-02 Revision: A0
Serial Number: CAB04087BXK Mfg Date: 2000/03/04
RMA Number: 0 HW Rever: 2.0
slot: 3/1 Controller-Type : XPIF GIGE PAM
Part Number: 73-4167-05 Revision: A0
Serial Number: CAB04097GQA Mfg Date: Mar 15 00
RMA Number: 0 H/W Version: 1.0
FPGA Version: 20.72
XPIF Version: 3001 CAM size: 64 KB
Ucode Version: 1.0 CAM Type: Private TCAM
Port Phy Setup
Port 0: DONE GBIC Vendor: No vendor info.
slot: 4/* Controller-Type : Route Proc
Part Number: 73-3775-04 Revision: A0
Serial Number: CAB03201VCZ Mfg Date: Oct 04 99
RMA Number: 0 H/W Version: 5.7
FPGA Version: 4.8
slot: 5/* Controller-Type : Switch Card
Part Number: 73-3327-08 Revision: A0
Serial Number: CAB032428ZR Mfg Date: Jun 15 99
RMA Number: 0 H/W Version: 8.0
FPGA Version: 1.2
slot: 7/* Controller-Type : Switch Card
Part Number: 73-3327-08 Revision: A0
Serial Number: CAB032428ZE Mfg Date: Jun 15 99
RMA Number: 0 H/W Version: 8.0
FPGA Version: 1.2
<Information deleted>

Step 5   Use the show functional-image command to display detailed information about the functional images for the route processors, switch processors, and Fast Ethernet and Gigabit Ethernet interface modules for the switch router (in this example, the Catalyst 8540 CSR). The following example shows how to display the functional image for the route processor in slot 4:

Switch# show functional-image slot 4
Details for cpu Image on slot: 4
Functional Version of the FPGA Image: 4.8
#Jtag-Distribution-Format-B
#HardwareRequired: 100(3.0-19,4.0-19,5.0-19)
  #FunctionalVersion: 4.8
#Sections: 1
#Section1Format: MOTOROLA_EXORMAX
Copyright (c) 1996-00 by cisco Systems, Inc.
All rights reserved.
generated by: holliday
on: Mon Mar 6 13:59:17 PST 2000
using: /vob/cougar/bin/jtag_script Version 1.13
config file: cpu.jcf
Chain description:
Part type Bits Config file
10k50 10 ../cidrFpga2/max/cidr_fpga.ttf
xcs4062 3 ../cubiFpga2/xil/cubi.bit
xcs4062 3 ../cubiFpga2/xil/cubi.bit
generic 2
XC4005 3 /vob/cougar/custom/common/jtcfg/xil/jtcfg_r.bit
Number devices = 5
Number of instruction bits = 21
FPGA config file information:
Bitgen date/time Sum File
100/03/02 19:14:49 7068 ../cidrFpga2/max/cidr_fpga.ttf
1999/04/15 18:46:32 36965 ../cubiFpga2/xil/cubi.bit
1999/04/15 18:46:32 36965 ../cubiFpga2/xil/cubi.bit
98/06/11 16:56:44 49904 /vob/cougar/custom/common/jtcfg/xil/jtcfg_r.bit
#End-Of-Header

Step 6   Verify the FunctionalVersion and #HardwareRequired fields to determine the FPGA version and the hardware version required for the FPGA. Compare this with the hardware version using the show hardware command output displayed in Step 2 and Step 4. If the FPGA version does not support the hardware version, download a new FPGA image, upgrade the hardware, or both.





Finding the Image

Use the Hardware and Software Compatibility Matrix (found on Cisco.com) to determine the correct IOS software image for your hardware configuration.

If you determine that you need to upgrade your IOS image, refer to the Cisco.com web page, and follow these steps to find and download the image you need:


Step 1   Login to the Cisco.com; the window will change and display additional features.

Step 2   Under Service and Support, select Software Center.

Step 3   The Technical Assistance Center page is displayed. Under Tools, select IOS Upgrade Planner.

The Cisco IOS Planner page provides greater flexibility to browse for your preferred software. You can view all major releases, all platforms, and all software features from a single interface. Choosing a platform, a maintenance release, or a software feature the planner automatically limits the other menu choices, based on your selections until you arrive at your preferred software.

Step 4   In the Select Platform column, select either of the following:

  • "C8540C" (for Catalyst 8510 CSR and Catalyst 8540 CSR platforms)
  • "C8540M" for Catalyst 8510 MSR and Catalyst 8540 MSR platforms)

Step 5   In the Select Release column, select the IOS software release you want to download.

Step 6   Read the requirements, and if your hardware configuration meets the requirements, click the agreement button.

Step 7   Select the file to download to your switch router.





To upgrade the IOS image on your platform, continue with the following section.

IOS Upgrade Procedures

If your IOS image is not the most recent, you can download the IOS image from Cisco.com to the switch router.

Refer to the Configuration Fundamentals Configuration Guide for details on the following frequently performed tasks:

  • Formatting Flash memory on a new PCMCIA card or on any Flash memory device that has locked blocks or failed sectors.
  • Managing files on file systems, including setting the default file system, listing files on a file system and deleting and recovering files.

Modifying, Downloading, and Maintaining System Images

The following tasks are performed frequently to maintain system image files:

  • Copy images from Flash memory to a network server. You can store system images for backup, or other purposes, by copying them from a Flash memory device to a TFTP or rcp server.
  • Copy images from a network server to Flash memory. You perform this procedure when upgrading your system image or functional image.
  • Copy images between local Flash memory devices. You perform this procedure when moving a system image from one switch router to another or to synchronize the functional images on switch routers with redundant route processors.

Maintaining Functional Images

You can load functional images used by certain hardware controllers in the switch router. The following sections describe the function and maintenance of functional images:

Understanding Functional Images

Functional images provide the low-level operating functionality for various hardware controllers. On hardware controllers within system programmable devices, such as Field Programmable Gate Arrays (FPGAs) and Erasable Programmable Logic Devices (EPLDs), the hardware functional images can be reprogrammed independently of loading the system image, and without removing the devices from the controller.

All new hardware you purchase is shipped with the functional images loaded. Loading a different functional image is required only when upgrading or downgrading functional image versions.

FPGA Upgrade Procedures

If the functional image is not the most recent, you can download the functional image to the switch router from Cisco.com. Use the reprogram command to update the functional image to the processor or interface module. The following example shows how to reprogram the route processor in slot 4 with the functional image fi-c8540-rp.A.4-8.bin from the Flash PC card in slot 0:

Switch# reprogram slot0:fi-c8540-rp.A.4-8.bin 4

Note   You can only enter the reprogram command from the console session prompt.

You can find the functional images and release notes for one of the following on Cisco.com, on the LAN Switching Products site:






Note   After you have determined the hardware and software versions on the switch router, check the release notes and DDTS database for symptoms resembling those you are observing. Often, the problem has already been discovered and a workaround has been provided.

Troubleshooting Switch Route Processor Redundancy and Enhanced High System Availability (Catalyst 8540 CSR and Catalyst 8540 MSR)

The Catalyst 8540 MSR and Catalyst 8540 CSR support, redundant route processor operation with dual route processors. In addition, Enhanced High System Availability (EHSA) is provided in the switching fabric when three switch processors are installed in the chassis. The troubleshooting processes for these features are described in the following sections:

Route Processor Redundant Operation (Catalyst 8540 CSR and Catalyst 8540 MSR)

The Catalyst 8540 MSR and Catalyst 8540 CSR supports fault tolerance by allowing a secondary route processor to take over if the primary fails. This secondary, or redundant, route processor runs in standby mode. In standby mode, the secondary route processor is partially booted with the Cisco IOS software; however, no configuration is loaded.

At the time of a switchover, the secondary route processor takes over as primary and loads the configuration as follows:

  • If the running configurations on the primary and secondary route processors match, the new primary uses the running configuration file
  • If the running configurations on the primary and secondary route processors do not match, the new primary uses the last saved configuration file in its nonvolatile random-access memory (NVRAM) (not the NVRAM of the former primary)

The former primary then becomes the secondary route processor.


Note   If the secondary route processor is unavailable, a major alarm is reported. Use the show facility-alarm status command to display the redundancy alarm status.

For detailed redundant route processor configuration information, refer to the "Initially Configuring the ATM Switch Router" chapter in the ATM Switch Router Software Configuration Guide .

Troubleshooting Route Processor Redundancy (Catalyst 8540 CSR and Catalyst 8540 MSR)

For redundant route processors to function correctly, your switch router's route processors must meet all of the following requirements:

  • Route processors must have identical hardware configurations
  • ROMMON must be version 12.0(4.6)W5(13) or later
  • Both route processors must have identical releases of IOS software

A common error you may encounter is the incompatibility of hardware modules and the IOS software version needed to perform a particular function. Check the Hardware and Software Compatibility Matrix document, available on-line, to confirm that you are using IOS software that supports the various hardware components installed in your switch router.

Troubleshooting Hardware and Software Versions of Redundant Route Processors

To troubleshoot the route processor hardware and software versions for redundancy, use the following commands:

Command Purpose

show version

Displays the configuration register value.

show hardware detail

Displays the hardware and software configurations of the primary and secondary route processors.

show redundancy

Displays the hardware and software configurations of the primary and secondary route processors.

To confirm that your switch router route processors meets the redundancy requirements, complete the following steps:


Step 1   Use the show version command to confirm the system hardware and software status of the primary route processor.

Switch# show version
Cisco Internetwork Operating System Software
IOS (tm) PNNI Software (cat8540m-WP-M), Version 12.1(2.3)W6(1.33), CISCO DEVELOP
MENT TEST VERSION
Copyright (c) 1986-2000 by cisco Systems, Inc.
Compiled Fri 20-Oct-00 23:39 by
Image text-base: 0x60010958, data-base: 0x60D30000
ROM: System Bootstrap, Version 12.0(4.6)W5(13), RELEASE SOFTWARE
8540MSR uptime is 8 weeks, 3 days, 10 hours, 0 minutes
System returned to ROM by reload
System image file is "slot0:cat8540m-wp-mz.121-2.3.PE33"
cisco C8540MSR (R5000) processor with 262144K/256K bytes of memory.
R5000 CPU at 200Mhz, Implementation 35, Rev 2.1, 512KB L2 Cache
Last reset from power-on
3 Ethernet/IEEE 802.3 interface(s)
16 FastEthernet/IEEE 802.3 interface(s)
14 ATM network interface(s)
505K bytes of non-volatile configuration memory.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
8192K bytes of Flash internal SIMM (Sector size 256K).
Configuration register is 0x0
Switch#

Step 2   Verify the ROM field. It indicates the release of IOS software loaded and running on the primary route processor.

Step 3   Use the show hardware detail command to compare the hardware versions of the primary and secondary route processors.

8540MSR# show hardware detail
C8540 named 8540MSR, Date: 18:42:12 UTC Fri Jan 5 2001
.
(Information Deleted)
.
slot: 4/* Controller-Type : Route Proc
    Part Number: 73-2644-05 Revision: A0
Serial Number: MIC03140NXK Mfg Date: Apr 04 99
RMA Number: 0 H/W Version: 5.7
FPGA Version: 4.8
slot: 4/0 Controller-Type : Netclk Module
Part Number: 73-2868-03 Revision: A0
Serial Number: MIC03140NSU Mfg Date: Apr 04 99
RMA Number: 0 H/W Version: 3.1
  FPGA Version: 4.8
.
(Information Deleted)
.
slot: 8/* Controller-Type : Route Proc
    Part Number: 73-2644-05 Revision: A0
Serial Number: MIC03140NXH Mfg Date: Apr 04 99
    RMA Number: 0 H/W Version: 5.7
FPGA Version: 4.8
slot: 8/0 Controller-Type : Netclk Module
Part Number: 73-2868-03 Revision: A0
Serial Number: MIC03140NVT Mfg Date: Apr 04 99
    RMA Number: 0 H/W Version: 3.1
  FPGA Version: 4.8
.
(Information Deleted)
.

Step 4   In the slots labeled Controller-Type : Route Proc, compare the Part Number, FPGA, and H/W Version fields. These numbers must all match, or redundancy will not function correctly on your switch router.

Step 5   Use the show redundancy command to check the configuration and status of the route processors.

Switch# show redundancy
This CPU is the PRIMARY
Primary
-------
Slot: 8
Uptime: 8 weeks, 4 days, 11 hours, 31 minutes
Image: PNNI Software (cat8540m-WP-M), Version 12.1(2.3)W 12.0(4a)W5(11a) RELEASE SOFTWARE
Time Since :
    Last Running Config. Sync: 2 weeks, 4 days, 2 hours, 28 minutes
    Last Startup Config. Sync: 2 weeks, 4 days, 2 hours, 34 minutes
Last Restart Reason: Normal boot
Secondary
---------
State:                         UP
Slot:                          4
Uptime:                        1 day, 1 hour, 59 minutes
Image:                         PNNI Software (cat8540m-WP-M), Version 12.0(4a)W5(11a) RELEASE SOFTWARE

Step 6   Verify the Primary, Secondary, and Slot fields. They indicate in which slot the primary route processor is configured.

Step 7   Verify the Last Running Config. Sync and Last Startup Config. Sync fields. They indicate the last time the running configuration and startup configuration were synchronized between the route processors.

Troubleshoot Redundant Route Processor Functions

To troubleshoot the route processor functions for redundancy, use the following commands:

Command Purpose

show atm vc interface atm 0

Displays the VC status between the primary and secondary route processors.

show atm status

Displays the status of the primary and secondary route processors.

Follow these steps to troubleshoot the route processor redundancy on the switch router:


Step 1   Use the show atm vc interface atm 0 command to confirm that the permanent virtual circuit (PVC) between switch route processors (SRPs) is up.

Switch# show atm vc interface atm 0
Interface VPI VCI Type X-Interface X-VPI X-VCI Encap Status
ATM0 0 35 PVC ATM0/0/0 0 16 ILMI DO
.
(Information Deleted)
.
ATM0 0 245 PVC ATM-SEC0 0 29 IPC UP
Switch#

Step 2   Verify the ATM-SEC0 field and confirm the PVC is up.

Step 3   Use the show atm status command to confirm the status of the PVC and other states.

Switch# show atm status
NUMBER OF INSTALLED CONNECTIONS: (P2P=Point to Point, P2MP=Point to MultiPoint,
MP2P=Multipoint to Point)
Type PVCs SoftPVCs SVCs TVCs PVPs SoftPVPs SVPs Total
P2P 982 0 0 0 0 0 0 982
P2MP 36 0 0 0 0 0 0 36
MP2P 0 0 0 0 0 0 0 0
TOTAL INSTALLED CONNECTIONS = 1018
PER-INTERFACE STATUS SUMMARY AT 16:42:51 UTC Fri Dec 8 2000:
Interface IF Admin Auto-Cfg ILMI Addr SSCOP Hello
Name Status Status Status Reg State State State
------------- -------- ------------ -------- ------------ --------- --------
.
(Information Deleted)
.
ATM0 UP up n/a UpAndNormal Idle n/a
ATM-SEC0 UP up done n/a none n/a
.
(Information Deleted)
.

Step 4   Verify that the IF Status field is up.

Step 5   Verify that the ILMI Addr Reg State field is UpAndNormal.





If you determine that redundancy is configured incorrectly, refer to the "Initially Configuring the ATM Switch Router" chapter in the ATM  Switch Router Software Configuration Guide.

Troubleshooting Switch Processor Redundancy with HSRP (Catalyst 8540 CSR and Catalyst 8540 MSR)

Enhanced High System Availability (EHSA) is provided in the switching fabric when three switch processors are installed in the chassis. These features and their configuration are described in the "Initially Configuring the ATM Switch Router" chapter in the
ATM Switch Router Software Configuration Guide .

To troubleshoot the EHSA of the switch processor, use the following commands:

Command Purpose

show preferred-switch-card-slots

Displays the configuration of the switch processors.

show switch fabric

Displays the switch fabric details of the switch router.

Follow these steps to troubleshoot the redundant switch processors on the switch router:


Step 1   Use the show preferred-switch-card-slots command to confirm the configuration of the switch processors.

Switch# show preferred-switch-card-slots
The currently preferred switch card slots are slot: 5 and slot: 7
The currently active switch card slots are slot: 5 and slot: 7
Switch#

Step 2   Use the show switch fabric command to confirm the status of the switch processors and their location.

Switch# show switch fabric
swc_presence_mask: 0x5
Switch mode: NR_20G
Number of Switch Cards present in the Chassis: 3
SWC SLOT SWC_TYPE SWC_STATUS
=================================================
     5 EVEN ACTIVE
     6 STANDBY           STANDBY
     7 ODD ACTIVE
MMC Switch Fabric (idb=0x62146E7C)
.
(Information Deleted)
.

Step 3   Check the field Number of Switch Cards present in the Chassis. If you only have two switch processors, your switch router does not have EHSA. With three switch processors installed, if either of the two active switch processors fail, the third processor takes over.

Step 4   Check the SWC_STATUS field. This fields identifies the active and standby switch processors.





If you determine that redundancy is configured incorrectly, refer to the "Initially Configuring the ATM Switch Router" chapter in the ATM Switch Router Software Configuration Guide .