Cisco 7600 Series Router Supervisor Engine and Route Switch Processor Guide
Troubleshooting Route Switch Processors and Supervisor Engines
Downloads: This chapterpdf (PDF - 187.0KB) The complete bookPDF (PDF - 1.5MB) | Feedback

Troubleshooting Route Switch Processors and Supervisor Engines

Table Of Contents

Troubleshooting Route Switch Processors and Supervisor Engines

Getting Started

AC and DC Power Supply Information

Chassis and Supervisor Slot Installation

RSP720-1GE Supported Hardware and Features

RSP720-1GE Unsupported Hardware and Features

RSP720-10GE Supported Chassis, Line Cards, and Modules

RSP720-10GE Unsupported Chassis, Modules and Features

Obtaining Technical Assistance

Troubleshooting Supervisor Failures

Troubleshooting Back-Plane Data Corruption During OIR of a RP


Troubleshooting Chassis Overheat Conditions

Troubleshooting Line Card Module

Troubleshooting Interface Down Issues

Traceback Decode Procedure

Troubleshooting Image Reload Issues

Troubleshooting Supervisor 2T


Troubleshooting Route Switch Processors and Supervisor Engines


Getting Started

When the initial system boot is complete, verify the following:

Power supplies are supplying adequate power to the system.

The system fan assembly is operating normally.

System software boots successfully.

The supervisor engine and all switching modules are initialized and installed properly in their slots.

AC and DC Power Supply Information

AC power supply accepts a wide range of power inputs catering to different geographies. All AC power supplies relay a maximum output when connected to 230V input. For example, a 6000W AC power gives 6000W output when it is connected to 230V. When the AC power is connected to 110V, the power supply output is 3000W.

DC power supply draws power from more than one input. All the inputs of the supply is fed to receive the maximum rated power output. DC power supplies can work at lower capacity if all the inputs are not connected. For example, if a 6000W power supply takes 4 DC inputs, and if only 3 are connected, then output is 4500W, and if only 2 inputs are connected, the output is 3000W. However, if only one input is connected, 6000W Power supply output fails. The green LED indicates the DC power supply for all the power inputs. For more information on the various LED status, Cisco 7600 Series Router Supervisor Engine and Route Switch Processor Guide at http://www.cisco.com/en/US/docs/routers/7600/Hardware/Hardware_Guides/Supervisor_Engine_and_Route_Switch_Processor_Guide/SupE02_ps368_TSD_Products_Installation_Guide_Chapter.html#wp1135401 and Cisco 7600 Series Cisco IOS Software Configuration Guide, 12.2SR at http://www.cisco.com/en/US/docs/routers/7600/ios/12.2SR/configuration/guide/pwr_envr.html#wp1020518

Chassis and Supervisor Slot Installation

Table 4-1 lists the chassis and their slots where the supervisors can be installed.

Table 4-1

Chassis
Designated Supervisor Slots

7603S

1, 2

7604

1, 2

7606

5, 6

7606S

5, 6

7609

5, 6

7609S

5, 6

7613

7, 8


Chassis and Supervisor Slot Installation

RSP720-1GE Supported Hardware and Features

The RSP720 supports the following Cisco 7600 chassis, line cards and modules:

Supported on all Cisco 7600 routers (including enhanced chassis) except the Cisco 7603. At present 7603 is discontinued and only 7603S is shipped.

SPA interface processors (SIPs) and their shared port adapters (SPAs): 7600-SIP-600, 7600-SIP-400, and 7600-SIP-200.

Enhanced FlexWAN module (WS-X6582-2PA).

Ethernet services modules: 2-port 10 GE line card (7600-ESM-2X10GE) and 20-port 1 GE line card (7600-ESM-20X1GE).

All boards in the ES+ line cards family.

Distributed Forwarding Cards: DFC3C, DFC3CXL, DFC3B, DFC3BXL

LAN cards.

RSP720-1GE Unsupported Hardware and Features

These hardware and features are not supported by the RSP720:

Unsupported chassis: Cisco 7603.

Unsupported modules: Optical Service Modules (OSMs), FlexWAN module.

RSP720-10GE Supported Chassis, Line Cards, and Modules

The RSP720-10GE supports the following chassis and modules:

Cisco 7604 and 7609 chassis and the Cisco 7603-S, 7606-S, and 7609-S chassis.

If you insert an RSP720-10GE into an unsupported chassis, the RSP720-10GE drops to ROMMON and only the console is accessible.

SPA interface processors (SIPs) and their shared port adapters (SPAs): 7600-SIP-600, 7600-SIP-400, and 7600-SIP-200.

Enhanced FlexWAN module (WS-X6582-2PA).

Ethernet services modules: 2-port 10 GE line card (7600-ESM-2X10GE) and 20-port 1 GE line card (7600-ESM-20X1GE).

All boards in the ES+ family of line cards.

Distributed Forwarding Cards: DFC3C, DFC3CXL, DFC3B, DFC3BXL

All LAN cards.

RSP720-10GE Unsupported Chassis, Modules and Features

The RSP720-10GE does not support the following chassis and modules:

Cisco 7603, 7606, and 7613 chassis.

Services modules, Optical Service Modules (OSMs), FlexWAN module.

RSP720-10GE does not support High Availability features in 12.2(33)SRC and 12.2(33)SRD.

Obtaining Technical Assistance

If you are unable to solve a problem on your own based on the solutions provided, consult a Cisco customer service representative for assistance. When you call, ensure that you have the following information:

Chassis and module serial number.

Card information: Use the show module and show inventory command to determine which cards are installed.

Cisco IOS software release number: Use the show version command to determine this number.

Brief description of the symptoms and steps to isolate and solve the problem.

Maintenance agreement or warranty information.

ROM images. (Use the show version exec command.)

Programmable ROM labels. (This information is printed on the physical chip, and an example is shown in Figure 4-1.)

Figure 4-1 An Example of a Boot ROM Label—Boot ROM Version 12.2(33r)SRD5

NVRAM configurations for client and adjacent routers.

Debugging output from adjacent routers using the following privileged exec commands:

debug ip packet

debug arp

debug ip udp

debug tftp

Troubleshooting Supervisor Failures

Table 4-2 lists the solutions for supervisor issues.

Table 4-2 Troubleshooting Supervisor Failure  

Problem
Solution

Supervisor fails to operate or power up

1. Ensure that the card is inserted firmly in the slot.

2. Ensure that you have place the gold edge of the PCB in the guide rails of the chassis. This prevents damage to the connector placed at the rear side the board.

3. Check whether the ejector levers are latched and that the captive screws are fastened properly. If you are uncertain, unlatch the levers, loosen the screws, and attempt to reset the supervisor again.

4. Examine the power supply to see whether the chassis, as a whole, is receiving power.

5. Use the status LED on the supervisor to verify the correct installation of the card. If the card is properly installed, the status LED turns green.

Note If the system is running on reduced DC power supply, the system LED on a supervisor turns to orange from green. In this scenario, the system can run, however, not all the modules in the chassis may be supported.

Supervisor dual in-line memory module (DIMM) detection issue

1. Ensure that you have correctly installed DIMM in slot 1.

2. Ensure that you follow the standard ESD procedures while upgrading or replacing DIMMs.

RSP720 DIMM upgrade issues

1. Ensure that you remove the existing DIMM from slot 1.

2. Ensure that both the slot 1 and slot 2 DIMMs are from same vendor.

3. If you are upgrading the RP DIMMs to 2 Gigabytes by adding an extra Gigabyte DIMM in slot two, ensure that you insert the both the new 2 DIMMs in slot 1 and slot 2.

4. If you insert the DIMMs in both the slots for RP upgrade, and card does not initiate, then swap the DIMMs between the two slots and check again.


Troubleshooting Back-Plane Data Corruption During OIR of a RP

Table 4-3 lists the solutions for back-plane data corruption during Online Insertion and Removal of a route processor.

Table 4-3 Troubleshooting back-plane data corruption

Problem
Solution

Back-plane data corruption when cards are partially inserted

1. Ensure that the redundant supervisor is operational.

2. Ensure that the removed supervisor is reinserted into the chassis.

Note Though the card does not make full contact with the back part of chassis, it makes enough contact to receive power from the chassis,

3. The new active supervisor changes its LED appearance and the first two LEDs turns red.

4. If the reinserted board is well inserted or removed from chassis, the active supervisor's LED backs to normal appearance. If not, after 60 seconds approximately the system starts resetting.



Troubleshooting Chassis Overheat Conditions

Table 4-4 lists the solutions for a overheated chassis.

Table 4-4 Troubleshooting Overheated Chassis

Problem
Solution

Overheating in a chassis

1. Move the cards to the center of the chassis.

2. Close and open the slots with face plates.


Troubleshooting Line Card Module

If a c7600 line card module experiences issues, follow these instructions.

Leave the card in its impaired state to allow the Cisco TAC and escalation teams to collect additional information before resetting the card. If you are unable to leave the card in the impaired state, the card may ultimately need to be reset, but some minimal data should be collected before reloading the card. Before reloading a line card, capture the following CLI data to a text file.

!
term mon
term len 0
show logging
show tech                               ! issue twice 
remote command module <slot> show log
remote command module <slot> show tech  ! issue twice	

!

Once you collect the data, follow any of these options to move the traffic from the line card module without disturbing the card's state.

1. Costing: increase ip ospf cost and/or isis metric all interfaces on the module and the respective neighbors

2. Passive: add passive-interface under the routing protocols for all interfaces on the module

3. Shutdown: perform a shutdown on all the interfaces on the module

4. Reset: soft reset the line card module

5. Power-cycle: power-cycle the line card module

If (1), (2) and (3), in that order, fail to restore connectivity, perform step (4) as follows. Ensure wait a minute or two for the card to boot completely.

!
7600#hw-module module 4 reset 
Proceed with reload of module?[confirm]
% reset issued for module 4
7600#
*Feb 22 09:24:19.821: %C7600_PWR-SP-4-DISABLED: power to module in slot 4 set off (Reset)
*Feb 22 09:25:54.021: %DIAG-SP-6-BYPASS: Module 4: Diagnostics is bypassed
*Feb 22 09:25:56.513: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now 
online

If step (4) does not resolve the issue, perform step (5) by powering the card off completely. Wait about 10 seconds and then re-enable power to the card.

7600#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
7600(config)#no power enable module 4 
*Feb 22 09:27:43.149: %C7600_PWR-SP-4-DISABLED: power to module in slot 4 set off (admin 
request)
7600(config)#power enable module 4    
*Feb 22 09:39:23.071: %DIAG-SP-6-BYPASS: Module 4: Diagnostics is bypassed
*Feb 22 09:39:25.622: %OIR-SP-6-INSCARD: Card inserted in slot 4, interfaces are now 
online

Troubleshooting Interface Down Issues

Table 4-4 lists the solutions for a interface issues.

Table 4-5 Troubleshooting Interface Issues

Problem
Solution

Wrong or faulty cable

Use a straight cable in place of a cross cable or vice versa [for example, ethernet].

Use a single mode fiber in place of a multimode fiber or vice versa.

The Tx and Rx are reversed. The Tx of one end should connect to the Rx on the other side.

Clocking not synchronized

Though most interfaces work with internal clocking on both sides, it is advisable to use line clocking on one end. Execute the clock source <INTERNAL|LINE> command to configure the clocking. Execute the hssi internal-clock command to configure the internal clock on the HSSI interfaces.

Wrong transceiver or SFP connectors in use

If the SFP or transceivers are used, ensure that you use the correct SFP.

CRC mismatch at either end of the link

Check if the number of bits used for Cyclic Redundancy Check [CRC] on either end of the link are configured to be the same. CRC-16 and CRC-32 are the two options available.

Keepalive mismatch

Keepalives are link health polls that are periodically exchanged between routers over a link. A link is brought down if an end fails to respond to the keepalives for consecutive keepalive periods. If there is no response to the keepalives for three attempts, the link is UP momentarily as a recovery mechanism.

Mismatched timeslot configuration

For channelized interfaces, it is mandatory that the number of timeslots used for an interface on either end be the same i.e. if router A has used 10 timeslots for its interface, then router B should also use 10 timeslots. A mismatched timeslot configuration results in runts on either end and brings down the interface on either end.

Internal VLAN not allocated

Each interface/subinterface on the c7600 is allocated an internal VLAN by the SP. Sometimes the internal VLAN is not allocated or is lost or removed, the interface stays down and does not switch traffic after a router reload or an SSO switchover. To determine the internal/hidden VLAN allocated to an interface, use the show platform vlans command.


Traceback Decode Procedure

Tracebacks provide useful information to debug and fix problems. Tracebacks appear when a spurious memory access, alignment error, crash or memory corruption occurs.

This is a sample traceback message:

-Traceback 4018C538 4018A2A4 40338858 4034FDD0 403480B0 4017BC40 
 
   

It is important to decode the traceback message while filing DDTS. To decode a traceback symbol, files of the corresponding image is required. A c7600 image has a set of symbol files like RP symbol file, SP symbol file and a symbol file for each line card.

In general the messages before the traceback reveal the source of the traceback. So after determining whether it is SP/RP/linecard traceback, the corresponding symbol file has to be chosen to decode it.

These are some commonly used symbol files:

[ciscouser@blr-cde-001:/auto/nonrel_122S_PI/122SR_GDB-nightly/2006-01-08/sym]$ls 
cwpa-dw-m.rainier.symbols.gz    
cwpa-dwdbg-m.rainier.symbols.gz 
cwpa2-dw-m.rainier.symbols.gz 
cwpa2-dwdbg-m.rainier.symbols.gz                           //Flexwan2 symbol file 
cwtlc-dw-m.rainier.symbols.gz                              //OSM symbol file 
cwtlc-dwdbg-m.rainier.symbols.gz  
s3223-adventerprisek9_wan_dbg-mz.rainier.symbols.gz        //Sup32 SP symbol file 
s3223_rp-adventerprisek9_wan_dbg-m.rainier.symbols.gz      //Sup32 RP symbol file 
s72033-adventerprisek9_wan_dbg-mz.rainier.symbols.gz       //Sup720 SP symbol file 
s72033_rp-adventerprisek9_wan_dbg-m.rainier.symbols.gz     //Sup720 RP symbol file 
sip1-dwdbg-m.rainier.symbols.gz                            //SIP-200 symbol file
sip10g-dwdbg-m.rainier.symbols                             //SIP-600 symbol file  
sip2-dwdbg-m.rainier.symbols.gz 

[ciscouser@ blr-cde-001: ? /060108/sym]$ rsym sip10g-dwdbg-m.rainier.symbols

Reading sip10g-dwdbg-m.rainier_gdb_010806.symbols         
sip10g-dwdbg-m.rainier_gdb_010806.symbols read in  
Enter hex value: 4018C538 4018A2A4 40338858 4034FDD0 403480B0 4017BC40 
0x4018C538:abort(0x4018c530)+0x8 
0x4018A2A4:crashdump(0x4018a180)+0x124 
0x40338858:sip10g_fatal_error(0x40338800)+0x58 
0x4034FDD0:sip10g_external_error_interrupt_handler(0x4034fcb0)+0x120 
0x403480B0:sb_imr_lev5(0x40348010)+0xa0 
0x4017BC40:sip_lev5_intr(0x4017bb50)+0xf0 
Enter hex value: 

Troubleshooting Image Reload Issues

Table 4-5 describes the solutions for image reload issues.

Table 4-6 Troubleshooting Image Reload Issues

Problem
Solution

The following error message is displayed:

76k-eigrp-4#reload netboot 
tftp://171.69.17.19/tftpboot/kapilk/nightly
/Fri/s72033-jsv-mz
Copying 
tftp://171.69.17.19/tftpboot/kapilk/nightly
/Fri/s72033-jsv-mz to sup-image:0
%Error opening sup-image:0 (Unknown error 
0)

Ensure that you don't have a very large image loaded on the 7600 to begin with, which may cause resource problems. The solution is to load a default image first and then do reload netboot.


Troubleshooting Supervisor 2T

For more information on troubleshooting, see the following documents:

Troubleshooting Switch Port and Interface Problems at the following URL:

http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note09186a008015bfd6.shtml

How to Determine the Type of Supervisor Module That Is Installed in Catalyst 6500/6000 Series Switches at the following URL:

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801c0eb0.shtml

Catalyst Switch Cable, Connector, and AC Power Cord Guide at the following URL:

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00800a7af7.shtml

Troubleshooting Catalyst 6500/6000 Series Switches Running CatOS on the Supervisor Engine and Cisco IOS on the MSFC at the following URL:

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a008015504b.shtml

Hardware Requirements for Catalyst 6000/Catalyst 6500 Redundancy at the following URL:

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00800a2cee.shtml