Hardware Troubleshooting for the Cisco 7100 Series Router [Cisco 7100 Series VPN Routers]

Document ID: 23240

Introduction
Prerequisites
      Requirements
      Components Used
      Conventions
Platform Description
Identifying the Issue and Capturing Information
Step-by-Step Troubleshooting
      Booting Issues
      Post-booting Issues
Related Information

Introduction

Valuable time and resources are often wasted replacing hardware that actually functions properly. This document helps troubleshoot common hardware issues with Cisco 7100 series routers, and provides pointers for identifying whether or not the fault is in the hardware. This document does not cover any software-related failures except for those that are often mistaken as hardware issues.

Prerequisites

Requirements

Readers of this document should be knowledgeable of the following:

Components Used

The information in this document is based on the software and hardware versions:

Cisco 7100 Series Router
All Cisco IOS® software versions that run on the 7100 Series Router

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Platform Description

Cisco 7100 Series Virtual Private Network (VPN) routers offer specific hardware configurations optimized for VPN applications and network topologies. These include the Cisco 7120 Series and the Cisco 7140 Series Routers. The Cisco 7120 Series comes in six different models, and the Cisco 7140 Series comes in five different models. The models in both series are defined by a Wide Area Network (WAN) interface. Cisco 7100 Series Routers include the following:

Cisco 7120 series - Provides one fixed WAN port, two fixed 10BaseT/100BaseTX Fast Ethernet ports, one modular port adapter slot, and one service module slot
Cisco 7140 series - Provides two fixed WAN ports, two fixed 10BaseT/100BaseTX Fast Ethernet LAN ports, one modular port adapter slot, and one service module slot

For a detailed description of the 7100 platforms, see Cisco 7100 Series VPN Router Product Overview.

Identifying the Issue and Capturing Information

This section covers the most common symptoms resulting from a potential hardware issue seen on the 71XX.

The symptoms are organized into two sections:

Step-by-Step Troubleshooting: Booting Issues - This section lists the procedure to follow when a failure prevents the router from booting. This can result from a hardware failure (the procedure helps to identify which particular part, such as memory or modules), a configuration, or a software issue. For every symptom, appropriate actions are listed.

When a router is experiencing a booting issue, the only information to capture are the complete logs of messages that may be displayed on the console during the boot sequence.
Post-booting issues - These can be router crashes, router hangs, error messages on the console, packet drops, high CPU utilization, and memory allocation failures. The root cause in these situations is either hardware or software. The most important information to collect is:
- Console logs and/or syslog information
- The crashinfo file (if the router is crashing)
- Output from the show tech-support command, in enable mode

Step-by-Step Troubleshooting

Booting Issues

The table below lists symptoms and troubleshooting guidelines for booting issues.

Symptoms	Basic Troubleshooting	Advanced Troubleshooting
Power LED is OFF	Troubleshooting the Power Subsystem	n/a
Power LED is ON, fans not operating	Troubleshooting the Cooling Subsystem	n/a
Power LED is ON, Nothing appears on the console	Verify that the baud rate is set to 9600 bps. If that doesn't help, verify that the equipment used for connecting to the console is operating properly. You can do this by connecting to a known good router to check your console equipment. If the console equipment is successfully tested, but the problem remains: Remove the PCMCIA card Remove the modular port adapter and the service module. If the router boots correctly, try to test the cards in another router to determine if the fault comes from a card or the slot. If the problem persists, go to advanced troubleshooting.	Perform advanced troubleshooting using the internal diagnostic LEDs located on the motherboard
Power LED is ON, Hang after printing the "System Bootstrap" banner	Remove the PCMCIA card Remove the modular port adapter and the service module. If the router boots correctly, try to test the cards in another router to determine if the fault comes from a card or the slot. If the problem persists, go to advanced troubleshooting.	Perform advanced troubleshooting using the diagnostic mode
Power LED is ON, The router boots in ROMMON	If the 71xx is stuck in ROMmon mode, set the configuration register to 0x2102 and reload the router as follows: rommon 1 > confreg 0x2102 rommon 2 > reset If the router remains stuck in ROMmon, the reason is most likely a corrupt Cisco IOS software image or faulty Flash card. See ROMmon Recovery Procedure.	n/a
Power LED is ON, The router boots in BOOT mode	If the router is stuck in Bootmode, identified by the `Router_name(boot)>` prompt, check if the configuration register is 0x2102 from the show version output. If the configuration register shows a setting other than 0x2102, do the following: Router_name(boot)#configure terminal Enter configuration commands, one per line. End with CNTL/Z. Router_name(boot)(config)#config-register 0x2102 Router_name(boot)(config)#end Router_name(boot)#reload System configuration has been modified. Save? [yes/no]: no Proceed with reload? [confirm] Do NOT save the running configuration before reloading (you would lose the parts of the configuration that are not understood by the boot mode software). If the value of the configuration register is 0x2102 and you are still booting into Bootmode, it may be a boot failure from the PCMCIA card. Check if the "dir slot0:" or "dir disk0:" (for a Flash disk) works. If not, try to format the Flash (for instance, use the format slot0: command). If the dir command still doesn't work, replace the Flash card. If the dir command is working, the software image stored in the Flash is most likely corrupted. Delete the software image in Flash and download a new image (The Trivial File Transfer Protocol (TFTP) download procedure is the same as the Boot mode).	n/a
Power LED is ON, router is booting in a loop	The first step is to remove the modular cards (port adapter and service module). Then power-cycle the router. If it still fails, check if there is a valid Cisco IOS software image on the router into the bootflash and/or the PCMCIA Flash. In order to do this, you must be directly connected to the console port of the router. Send the break key within the first 60 seconds of bootup to go into ROMmon. From there, you can follow the procedures in ROMmon Recovery Procedure to try to recover. If it still fails, try to boot the router while bypassing the configuration. To do this, you need to issue the break key to enter the ROMmon. From there, change the value of the configuration register to 0x2142: rommon 1 > confreg 0x2142 rommon 2 > reset If the router boots properly after bypassing the configuration, this is either a misconfiguration or a software issue. Try to find error messages in the boot sequence that can help to identify the source of the issue, or try to load your configuration step-by-step until you identify which command is triggering the loop. Then, look for either a misconfiguration or a software bug using the Bug Toolkit (registered customers only) . If the router fails to boot without configuration and modular cards, then there is most probably a hardware failure in the chassis. In this case, replace the chassis.

Post-booting Issues

Port Adapter Issues

The router is booting correctly, but some interfaces are not available.

Symptom	Diagnostic
%PA-2-UNDEFPA: Undefined Port Adaptor type # in bay # %PA-3-DEACTIVATED: port adapter in bay [#] powered off.	The modular port adapter or service module is not recognized by the software. Use the Software Advisor (registered customers only) to find which software version is supporting that port adapter.
%PA-3-DEACTIVATED: port adapter in bay [#] powered off	If this is the modular port adapter or service module, try to reseat it and/or move it to another router to determine if the fault is on the card or on the slot. If this is a fixed port adapter, replace the chassis.

If the enable LED is ON, but you are still experiencing issues with an interface, then this is most likely not a hardware issue. Further troubleshooting needs to be done. Collect relevant error messages and then refer to the Error Message Decoder (registered customers only) .

Other Issues: Router crashes, router hangs, memory allocation failure, high CPU utilization, performance issue

If you are experiencing router crashes, router hangs, or console error messages, more specific troubleshooting needs to be done because these various issues can be the consequence of a hardware or software problem.

Memory allocation failures, high CPU utilization, and performance issues are usually software or configuration issues.

You can find detailed information on how to troubleshoot these issues at Troubleshooting Cisco 7100 Series Routers.

Advanced Troubleshooting

This is part of the Step-by-Step Troubleshooting, Booting Issue section.

Nothing on the console

The diagnostic LEDs located on the motherboard (see picture below) tell you which boot operation is actually being performed, and help to identify at which step the router is hanging.

The table below shows several steps in the boot process and the corresponding LED indication (1 = LED ON, 0 = LED OFF).

Boot Operation	LED numbers: 3 2 1 0
Power On Reset	1 1 1 1
Initialize R4K regs	1 1 1 1
Initialize gt64120 regs	1 1 1 1
Set up L2 cache	0 0 1 1
Start checking first 64K of DIMM1	0 1 0 0
DIMM1 Failure	0 1 0 1
Finished 64K DIMM1 Test	0 1 1 1
Clear GT64120 timer cntrl reg	1 0 0 0
Failed clearing timer cntrl reg	1 0 0 1
Check GT64120 revision	1 0 1 0
Read NVRAM Config Reg	1 0 1 1
Initialize UART	1 1 0 0
Turn off all LEDs and Print Banner	0 0 0 0

Actions to take depending on the LED display

LED Display	Action
All LEDs are ON	Remove SDRAM DIMM0 (packet memory) Replace the board
LEDs = 0100, 0101, 0110	Replace SDRAM DIMM1 and DIMM2 (system memory) Try to remove SDRAM DIMM0 (packet memory) Replace the board
LEDs = 1000, 1001, 1010	Replace the board
LEDs = 1011	Replace NVRAM Remove Flash memory SIMM (Bootflash) Remove SDRAM DIMM0 (packet memory) Replace the board
LEDs = 1100	Remove Flash memory SIMM (Bootflash) Remove SDRAM DIMM0 (packet memory) Replace the board
All LEDs are OFF	Go to Advanced troubleshooting using the diagnotic mode

No ROMMON Prompt

The router hangs after printing the System Bootstrap banner:

System Bootstrap, Version 12.0(19990324:032255) [100], DEVELOPMENT SOFTWARE
Copyright (c) 1999 by cisco Systems, Inc.

Advanced Troubleshooting using the Diagnostic Mode

The diagnostic mode is enabled using a specific setting of the configuration register. If the 71xx hangs after the bootstrap banner, you cannot configure anything on the 71xx as there is no prompt. To change the configuration register value, you must configure the diagnostic mode on the nonvolatile RAM (NVRAM) of a healthy 71xx, and then move this NVRAM into the appropriate 71xx.

The procedure to do this follows.

Enable the diagnostic mode on a healthy 71xx:

rommon 2 > confreg

    Configuration Summary
enabled are:
diagnostic mode
load rom after netboot fails
break/abort has effect
console baud: 9600
boot: image specified by the boot system commands
      or default to: cisco2-C7100

do you wish to change the configuration? y/n  [n]:  y
disable "diagnostic mode"? y/n  [n]: 
enable  "use net in IP bcast address"? y/n  [n]:  
disable "load rom after netboot fails"? y/n  [n]:  
enable  "use all zero broadcast"? y/n  [n]:  
disable "break/abort has effect"? y/n  [n]:  
enable  "ignore system config info"? y/n  [n]:  
change console baud rate? y/n  [n]:  
change the boot characteristics? y/n  [n]:  


    Configuration Summary
enabled are:
load rom after netboot fails
break/abort has effect
console baud: 9600
boot: image specified by the boot system commands
      or default to: cisco2-C7100

do you wish to change the configuration? y/n  [n]:  


You must reset or power cycle for new config to take effect
 
rommon 3 > reset

System Bootstrap, Version 11.3(2)XA4, RELEASE SOFTWARE (fc1)
Copyright (c) 1999 by cisco Systems, Inc.

Power off the router and remove the NVRAM. Look at the figure below to locate the NVRAM on the motherboard.

Place this NVRAM chip into the router under troubleshooting and switch ON the router.

Below is a sample output of the additional diagnostic messages that the diagnostic mode prints during the boot:

System Bootstrap, Version 12.0(19990324:032255) [100], DEVELOPMENT SOFTWARE
Copyright (c) 1999 by cisco Systems, Inc.

Testing lower main memory - data equals address
Testing lower main memory - checkerboard
Testing lower main memory - inverse checkerboard
Clearing lower 16K memory for cache initialization
Clearing bss 
Reading monitor variables from NVRAM 
Enabling interrupts 
Initializing TLB 
Initializing cache 
Initializing R7K L2/L3 caches
R7K CPU

Initializing required TLB entries
Initializing PCI subsystem
Initializing main memory
Config SDRAM parity check
Initializing shared memory
Sizing NVRAM
Initializing PCMCIA controller
Exiting init
C7100 platform with 65536 Kbytes of main memory

Depending on which diagnostic message the boot sequence is hanging, you can identify which part of the hardware may be the root cause of the booting issue.

The table below shows possible boot sequence hangs, and the related diagnostic.

If the boot sequence is hanging here...	Diagnostic
Testing lower main memory - data equals address Testing lower main memory - checkerboard Testing lower main memory - inverse checkerboard Clearing lower 16K memory for cache initialization Clearing bss	Synchronous Dynamic RAM (SDRAM) failure. Reseat the memory. If the problem persists, replace the SDRAM.
Initializing TLB Initializing cache Initializing R7K L2/L3 caches	CPU or External Cache Failure. Replace the chassis or router.
Initializing PCI subsystem	PCI Failure - Remove the modular port adapter and service module. If the router boots correctly, try to test the cards in another router to determine if the fault comes from a card or the slot.
Initializing main memory Config SDRAM parity check Initializing shared memory	SDRAM Failure - Reseat the memory. If the problem persists, replace the SDRAM
Initializing PCMCIA controller	PCMCIA Controller - Check the PCMCIA card. Try a spare card from another router (see PCMCIA Filesystem Compatibility Matrix). If the problem persists, replace the motherboard.

Related Information

Updated: Jul 07, 2005

Document ID: 23240

Hierarchical Navigation

Cisco 7100 Series VPN Routers