Guest

Cisco 7100 Series VPN Routers

Hardware Troubleshooting for the Cisco 7100 Series Router

Cisco - Hardware Troubleshooting for the Cisco 7100 Series Router

Document ID: 23240

Updated: Jul 07, 2005

   Print

Introduction

Valuable time and resources are often wasted replacing hardware that actually functions properly. This document helps troubleshoot common hardware issues with Cisco 7100 series routers, and provides pointers for identifying whether or not the fault is in the hardware. This document does not cover any software-related failures except for those that are often mistaken as hardware issues.

Prerequisites

Requirements

Readers of this document should be knowledgeable of the following:

Components Used

The information in this document is based on the software and hardware versions:

  • Cisco 7100 Series Router

  • All Cisco IOS® software releases that run on the 7100 Series Router

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Platform Description

Cisco 7100 Series Virtual Private Network (VPN) routers offer specific hardware configurations optimized for VPN applications and network topologies. These include the Cisco 7120 Series and the Cisco 7140 Series Routers. The Cisco 7120 Series comes in six different models, and the Cisco 7140 Series comes in five different models. The models in both series are defined by a Wide Area Network (WAN) interface. Cisco 7100 Series Routers include the following:

  • Cisco 7120 series—Provides one fixed WAN port, two fixed 10BaseT/100BaseTX Fast Ethernet ports, one modular port adapter slot, and one service module slot

  • Cisco 7140 series—Provides two fixed WAN ports, two fixed 10BaseT/100BaseTX Fast Ethernet LAN ports, one modular port adapter slot, and one service module slot

Refer to Cisco 7100 Series VPN Router Product Overview for more information and a detailed description of the 7100 platforms.

7140back.gif

Identifying the Issue and Capturing Information

This section covers the most common symptoms resulting from a potential hardware issue seen on the 71XX.

The symptoms are organized into two sections:

  • Step-by-Step Troubleshooting: Booting Issues—This section lists the procedure to follow when a failure prevents the router from booting. This can result from a hardware failure (the procedure helps to identify which particular part, such as memory or modules), a configuration, or a software issue. For every symptom, appropriate actions are listed.

    When a router is experiencing a booting issue, the only information to capture are the complete logs of messages that may be displayed on the console during the boot sequence.

  • Post-booting issues—These can be router crashes, router hangs, error messages on the console, packet drops, high CPU utilization, and memory allocation failures. The root cause in these situations is either hardware or software. The most important information to collect is:

    • Console logs and/or syslog information

    • The crashinfo file (if the router is crashing)

    • Output from the show tech-support command, in enable mode

Step-by-Step Troubleshooting

Booting Issues

The table below lists symptoms and troubleshooting guidelines for booting issues.

Symptoms Basic Troubleshooting Advanced Troubleshooting
Power LED is OFF Troubleshooting the Power Subsystem n/a
Power LED is ON, fans not operating Troubleshooting the Cooling Subsystem n/a
Power LED is ON, Nothing appears on the console
  • Verify that the baud rate is set to 9600 bps. If that doesn't help, verify that the equipment used for connecting to the console is operating properly. You can do this by connecting to a known good router to check your console equipment.
  • If the console equipment is successfully tested, but the problem remains:
    • Remove the PCMCIA card
    • Remove the modular port adapter and the service module. If the router boots correctly, try to test the cards in another router to determine if the fault comes from a card or the slot.
  • If the problem persists, go to advanced troubleshooting.
Perform advanced troubleshooting using the internal diagnostic LEDs located on the motherboard
Power LED is ON, Hang after printing the "System Bootstrap" banner
  • Remove the PCMCIA card
  • Remove the modular port adapter and the service module. If the router boots correctly, try to test the cards in another router to determine if the fault comes from a card or the slot.
  • If the problem persists, go to advanced troubleshooting.
Perform advanced troubleshooting using the diagnostic mode
Power LED is ON, The router boots in ROMMON If the 71xx is stuck in ROMmon mode, set the configuration register to 0x2102 and reload the router as follows:
rommon 1 > confreg 0x2102
rommon 2 > reset
If the router remains stuck in ROMmon, the reason is most likely a corrupt Cisco IOS software image or faulty Flash card. See ROMmon Recovery Procedure.
n/a
Power LED is ON, The router boots in BOOT mode If the router is stuck in Bootmode, identified by the Router_name(boot)> prompt, check if the configuration register is 0x2102 from the show version output. If the configuration register shows a setting other than 0x2102, do the following:
Router_name(boot)#configure terminal 
Enter configuration commands, one per line. End with CNTL/Z. 
Router_name(boot)(config)#config-register 0x2102 
Router_name(boot)(config)#end 
Router_name(boot)#reload 
System configuration has been modified. Save? [yes/no]: no 
Proceed with reload? [confirm]
Do NOT save the running configuration before reloading (you would lose the parts of the configuration that are not understood by the boot mode software). If the value of the configuration register is 0x2102 and you are still booting into Bootmode, it may be a boot failure from the PCMCIA card.
  • Check if the "dir slot0:" or "dir disk0:" (for a Flash disk) works. If not, try to format the Flash (for instance, use the format slot0: command). If the dir command still doesn't work, replace the Flash card.
  • If the dir command is working, the software image stored in the Flash is most likely corrupted. Delete the software image in Flash and download a new image (The Trivial File Transfer Protocol (TFTP) download procedure is the same as the Boot mode).
n/a
Power LED is ON, router is booting in a loop
  • The first step is to remove the modular cards (port adapter and service module). Then power-cycle the router.
  • If it still fails, check if there is a valid Cisco IOS software image on the router into the bootflash and/or the PCMCIA Flash. In order to do this, you must be directly connected to the console port of the router. Send the break key within the first 60 seconds of bootup to go into ROMmon. From there, you can follow the procedures in ROMmon Recovery Procedure to try to recover.
  • If it still fails, try to boot the router while bypassing the configuration. To do this, you need to issue the break key to enter the ROMmon. From there, change the value of the configuration register to 0x2142:
    rommon 1 > confreg 0x2142 
    rommon 2 > reset
    
    If the router boots properly after bypassing the configuration, this is either a misconfiguration or a software issue. Try to find error messages in the boot sequence that can help to identify the source of the issue, or try to load your configuration step-by-step until you identify which command is triggering the loop. Then, look for either a misconfiguration or a software bug using the Bug Toolkit (registered customers only) .
  • If the router fails to boot without configuration and modular cards, then there is most probably a hardware failure in the chassis. In this case, replace the chassis.

Post-booting Issues

Port Adapter Issues

The router is booting correctly, but some interfaces are not available.

Symptom Diagnostic
%PA-2-UNDEFPA: Undefined Port Adaptor type # in bay #
%PA-3-DEACTIVATED: port adapter in bay [#] powered off.
The modular port adapter or service module is not recognized by the software. Use the Software Advisor (registered customers only) to find which software version is supporting that port adapter.
%PA-3-DEACTIVATED: port adapter 
in bay [#] powered off
If this is the modular port adapter or service module, try to reseat it and/or move it to another router to determine if the fault is on the card or on the slot. If this is a fixed port adapter, replace the chassis.

If the enable LED is ON, but you are still experiencing issues with an interface, then this is most likely not a hardware issue. Further troubleshooting needs to be done. Collect relevant error messages and then refer to the Error Message Decoder (registered customers only) .

Other Issues: Router crashes, router hangs, memory allocation failure, high CPU utilization, performance issue

If you are experiencing router crashes, router hangs, or console error messages, more specific troubleshooting needs to be done because these various issues can be the consequence of a hardware or software problem.

Memory allocation failures, high CPU utilization, and performance issues are usually software or configuration issues.

You can find detailed information on how to troubleshoot these issues at Troubleshooting Cisco 7100 Series Routers.

Advanced Troubleshooting

This is part of the Step-by-Step Troubleshooting, Booting Issue section.

Nothing on the console

The diagnostic LEDs located on the motherboard (see picture below) tell you which boot operation is actually being performed, and help to identify at which step the router is hanging.

NP-71xx.gif

The table below shows several steps in the boot process and the corresponding LED indication (1 = LED ON, 0 = LED OFF).

Boot Operation LED numbers: 3 2 1 0
Power On Reset 1 1 1 1
Initialize R4K regs 1 1 1 1
Initialize gt64120 regs 1 1 1 1
Set up L2 cache 0 0 1 1
Start checking first 64K of DIMM1 0 1 0 0
DIMM1 Failure 0 1 0 1
Finished 64K DIMM1 Test 0 1 1 1
Clear GT64120 timer cntrl reg 1 0 0 0
Failed clearing timer cntrl reg 1 0 0 1
Check GT64120 revision 1 0 1 0
Read NVRAM Config Reg 1 0 1 1
Initialize UART 1 1 0 0
Turn off all LEDs and Print Banner 0 0 0 0

Actions to take depending on the LED display

LED Display Action
All LEDs are ON
  • Remove SDRAM DIMM0 (packet memory)
  • Replace the board
LEDs = 0100, 0101, 0110
  • Replace SDRAM DIMM1 and DIMM2 (system memory)
  • Try to remove SDRAM DIMM0 (packet memory)
  • Replace the board
LEDs = 1000, 1001, 1010
  • Replace the board
LEDs = 1011
  • Replace NVRAM
  • Remove Flash memory SIMM (Bootflash)
  • Remove SDRAM DIMM0 (packet memory)
  • Replace the board
LEDs = 1100
  • Remove Flash memory SIMM (Bootflash)
  • Remove SDRAM DIMM0 (packet memory)
  • Replace the board
All LEDs are OFF Go to Advanced troubleshooting using the diagnotic mode

No ROMMON Prompt

The router hangs after printing the System Bootstrap banner:

System Bootstrap, Version 12.0(19990324:032255) [100], DEVELOPMENT SOFTWARE
Copyright (c) 1999 by cisco Systems, Inc.

Advanced Troubleshooting using the Diagnostic Mode

The diagnostic mode is enabled using a specific setting of the configuration register. If the 71xx hangs after the bootstrap banner, you cannot configure anything on the 71xx as there is no prompt. To change the configuration register value, you must configure the diagnostic mode on the nonvolatile RAM (NVRAM) of a healthy 71xx, and then move this NVRAM into the appropriate 71xx.

The procedure to do this follows.

  1. Enable the diagnostic mode on a healthy 71xx:

    rommon 2 > confreg
    
        Configuration Summary
    enabled are:
    diagnostic mode
    load rom after netboot fails
    break/abort has effect
    console baud: 9600
    boot: image specified by the boot system commands
          or default to: cisco2-C7100
    
    do you wish to change the configuration? y/n  [n]:  y
    disable "diagnostic mode"? y/n  [n]: 
    enable  "use net in IP bcast address"? y/n  [n]:  
    disable "load rom after netboot fails"? y/n  [n]:  
    enable  "use all zero broadcast"? y/n  [n]:  
    disable "break/abort has effect"? y/n  [n]:  
    enable  "ignore system config info"? y/n  [n]:  
    change console baud rate? y/n  [n]:  
    change the boot characteristics? y/n  [n]:  
    
    
        Configuration Summary
    enabled are:
    load rom after netboot fails
    break/abort has effect
    console baud: 9600
    boot: image specified by the boot system commands
          or default to: cisco2-C7100
    
    do you wish to change the configuration? y/n  [n]:  
    
    
    You must reset or power cycle for new config to take effect
     
    rommon 3 > reset
    
    System Bootstrap, Version 11.3(2)XA4, RELEASE SOFTWARE (fc1)
    Copyright (c) 1999 by cisco Systems, Inc.
    
  2. Power off the router and remove the NVRAM. Look at the figure below to locate the NVRAM on the motherboard.

    nvram.gif

  3. Place this NVRAM chip into the router under troubleshooting and switch ON the router.

    Below is a sample output of the additional diagnostic messages that the diagnostic mode prints during the boot:

    System Bootstrap, Version 12.0(19990324:032255) [100], DEVELOPMENT SOFTWARE
    Copyright (c) 1999 by cisco Systems, Inc.
    
    Testing lower main memory - data equals address
    Testing lower main memory - checkerboard
    Testing lower main memory - inverse checkerboard
    Clearing lower 16K memory for cache initialization
    Clearing bss 
    Reading monitor variables from NVRAM 
    Enabling interrupts 
    Initializing TLB 
    Initializing cache 
    Initializing R7K L2/L3 caches
    R7K CPU
    
    Initializing required TLB entries
    Initializing PCI subsystem
    Initializing main memory
    Config SDRAM parity check
    Initializing shared memory
    Sizing NVRAM
    Initializing PCMCIA controller
    Exiting init
    C7100 platform with 65536 Kbytes of main memory

Depending on which diagnostic message the boot sequence is hanging, you can identify which part of the hardware may be the root cause of the booting issue.

The table below shows possible boot sequence hangs, and the related diagnostic.

If the boot sequence is hanging here... Diagnostic
Testing lower main memory - data equals address
Testing lower main memory - checkerboard
Testing lower main memory - inverse checkerboard
Clearing lower 16K memory for cache initialization
Clearing bss
Synchronous Dynamic RAM (SDRAM) failure. Reseat the memory. If the problem persists, replace the SDRAM.
Initializing TLB 
Initializing cache 
Initializing R7K L2/L3 caches
CPU or External Cache Failure. Replace the chassis or router.
Initializing PCI subsystem
PCI Failure - Remove the modular port adapter and service module. If the router boots correctly, try to test the cards in another router to determine if the fault comes from a card or the slot.
Initializing main memory
Config SDRAM parity check
Initializing shared memory
SDRAM Failure - Reseat the memory. If the problem persists, replace the SDRAM
Initializing PCMCIA controller
PCMCIA Controller - Check the PCMCIA card. Try a spare card from another router (see PCMCIA Filesystem Compatibility Matrix). If the problem persists, replace the motherboard.

Related Information

Updated: Jul 07, 2005
Document ID: 23240