Table Of Contents
Problem Solving Using a Subsystems Approach
Troubleshooting the Power Subsystem
Troubleshooting the Cooling Subsystem
Troubleshooting the I/O Subsystem
Troubleshooting the Processor Subsystem
Troubleshooting the Port Adapter or Service Adapter
Upgrading the Boot Helper (Boot Loader) Image
PXF Troubleshooting Information
Troubleshooting
Your Cisco 7401ASR router went through extensive testing before leaving the factory. However, if you encounter problems starting the router, use the information in this chapter to help isolate the cause of the problems. This chapter contains the following sections:
•
Problem Solving Using a Subsystems Approach
•
•
The procedures in this chapter assume that you are troubleshooting the initial system startup, and that your router is in the original factory configuration. If you have removed or replaced components or changed any default settings, the recommendations in this chapter might not apply. Make sure to review the safety warnings listed in the publication Cisco 7401ASR Regulatory Compliance and Safety Information that accompanied your Cisco 7401ASR router before using the troubleshooting procedures in this chapter.
Troubleshooting Overview
This section describes the troubleshooting methods used in this chapter and describes how the Cisco 7401ASR router is divided into subsystems for more efficient problem solving. If you are unable to easily solve the problem, contact a customer service representative for assistance and further instructions. Provide the representative with the following information:
•
•
•
•
•
•
The following table shows the general troubleshooting strategy described in this appendix. Refer to this table, as necessary, to follow the steps to isolate problems to a specific subsystem; resolve the problem if possible.
In addition to following the subsystems approach to troubleshooting, Cisco Registered Direct users can access troubleshooting tools at http://www.cisco.com/public/support/tac/tools_az.shtml.
Problem Solving Using a Subsystems Approach
The key to solving problems with the system is isolating the problem to a specific subsystem. The first step in solving startup problems is to compare what the system is doing to what it should be doing. Because a startup problem is usually caused by a single component, it is more efficient to first isolate the problem to a subsystem rather than troubleshoot each component in the system. For these troubleshooting procedures, consider the following subsystems:
•
•
•
The following sections help you isolate a problem to one of these subsystems and direct you to the appropriate troubleshooting section.
Identifying Startup Problems
Startup problems are commonly due to the source power or to a port adapter or service adapter that is dislodged from the system board. Although an over temperature condition is unlikely at initial startup, the environmental monitoring functions are included in this chapter because they also monitor internal voltages.
When you start up the router for the first time, you should observe the startup sequence described in the "Starting the System and Observing Initial Conditions" section in "Starting and Configuring." This section contains a more detailed description of the normal startup sequence and describes the steps to take if the system does not perform that sequence as expected.
LEDs indicate all system states in the startup sequence. By checking the state of the LEDs, you can determine when and where the system failed in the startup sequence. Use the following descriptions to isolate the problem to a subsystem, and then proceed to the appropriate sections to try to resolve the problem.
When you start up the system by turning on the power supply switch, the following should occur:
•
•
•
–
–
If this LED remains off when you start the router, it is probably a problem with the Gigabit Ethernet port. Consult the table on page 5-2.
–
–
•
•
Troubleshooting the Power Subsystem
Check the following to help isolate a problem with the power subsystem:
Note
Troubleshooting the Cooling Subsystem
Check the following to help isolate a problem with the cooling sub system:
Table 5-2 Troubleshooting the Cooling Subsystem
System displays the following message:
Router: 00:03:46:%ENVM-3-BLOWER:Fan 2 may have failedOne or more fans are not operating.
Contact a service representative.
System shuts down, some fans may or may not continue to rotate, and this error message is displayed:
Queued messages:%ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdownThis error message indicates that the system has detected an overtemperature condition or out-of-tolerance power condition inside the chassis.
•
•
•
To determine if the fans are operating, listen for them. In noisy environments, place your hand on the rear of the chassis to feel if air is being forced out the vents.
Contact a service representative.
Heated exhaust air from other equipment is entering the router's inlet vents.
Move other equipment or the router to ensure proper airflow.
System shuts down and this error message
is displayed:Queued messages:%ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdownThis error message indicates that the system has detected an overtemperature condition or out-of-tolerance power condition inside the chassis.
Note
The error message could indicate a faulty component or temperature sensor. Before the system shuts down, use the show environment or show environment table command to display the internal chassis environment. See "Starting and Configuring," the "Reporting Functions" section for descriptions of the show environment and show environment table commands.
Contact a service representative.
If an environmental shutdown results from an out-of-tolerance power condition, the system shuts down.
If the system still fails to come up when the power supply is connected to a different power source with a new power cable, the power supply is probably faulty. Contact a service representative.
Troubleshooting the I/O Subsystem
The procedures in this section assume that you have not made changes to your configuration file.
If the Cisco 7401ASR router I/O LEDs do not go on as expected (see the "Identifying Startup Problems" section), check the following items to help isolate the problem:
Table 5-3 Troubleshooting the I/O Subsystem
LEDs remain off when the system power switch is turned on.
Power supply or cooling subsystem is faulty.
If the LEDs stay off, first see the "Troubleshooting the Power Subsystem" section and the "Troubleshooting the Cooling Subsystem" section to ensure that both the fans and the power supply are functioning properly.
Power supply and fans appear operational, but none of the I/O LEDs are on.
The improperly connected port adapter has indefinitely paused the system.
Reseat the port adapter, tighten the screw of the port adapter latch, and then restart the system.
The prior configuration—native Gigabit Ethernet or the Fast Ethernet/Ethernet—LEDs remain on after configuring the ports to change the configuration from one Ethernet type to the other.
The media-type command was not used when changing the configuration of these ports.
See "Starting and Configuring," the "Configuring the Native Gigabit Ethernet and Fast Ethernet/Ethernet Interfaces" section.
Troubleshooting the Processor Subsystem
The processor subsystem comprises the system board and the port adapter. A port adapter that is partially connected to the system board sends incomplete signals to the processor, which faults the PCI bus and causes the system to pause indefinitely. Therefore, first ensure that the system software has initialized successfully and then check to see if the port adapter is fully connected.
Troubleshooting the Port Adapter or Service Adapter
Check the following to help isolate a problem with the port adapter or service adapter:
Upgrading the Boot Helper (Boot Loader) Image
The boot helper (boot loader) image resides in Flash memory and contains a subset of the Cisco IOS software. This image is used to boot your router from the network or to load Cisco IOS images onto the router. This image is also used if the system cannot find a valid system image.
Your boot helper (boot loader) image should correspond to the Cisco IOS release that is running on your router.
Note
To upgrade your boot helper (boot loader) image, obtain the most current boot helper image through Cisco.com and copy the new boot helper image to Flash memory on your router. For information on how to access Cisco.com, see the "Cisco.com" section in the Preface. Follow the Software Center link under Service and Support. You need to get a login code from a Cisco representative to retrieve files from the Software Center.
To obtain a boot helper (boot loader) image from Cisco.com and upgrade your bootflash, do the following:
Step 1
Step 2
router# format bootflash:Format operation may take a while. Continue? [confirm]Format operation will destroy all data in "bootflash:". Continue? [confirm]Formatting sector ...Format of bootflash:complete
Note
Step 3
router# copy tftp bootflash:Address or name of remote host []? biffSource filename []? c7200-boot-mz.120-5.SDestination filename [c7200-boot-mz.120-5.S]?Accessing tftp://biff/c7200-boot-mz.120-5.S...Loading c7200-boot-mz.120-5.S from 192.168.254.254 (via Ethernet4/0):!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!![OK - 3132516/6264832 bytes]3132516 bytes copied in 28.488 secs (111875 bytes/sec)This completes the procedure for upgrading your boot helper (boot loader) image. For more detailed instructions on loading and maintaining system images and microcode, including boot helper images, refer to the Cisco IOS Configuration Fundamentals Configuration Guide, which is available on Cisco.com.
PXF Troubleshooting Information
Note
Cisco IOS Statistics Not Supported by PXF
Some standard Cisco IOS statistics are not supported in the PXF path, including:
•
•
•
Features Not Supported by PXF
Features that are not supported by PXF are punted to the Route Processor (RP), which can cause high RP CPU usage.
High PXF CPU Usage
Enter the show pxf accounting summary command to display information about PXF CPU usage:
Router# show pxf accounting summary...10 second averages: PXF complex busy: 1% PXF read pipeline full: 0%The PXF has 16 processors, only one of which is in use at a time when there is a low traffic rate. Even though only one of the processors is working, all of the PXF processors are slightly slowed. At higher traffic rates, many of the PXF processors are working at the same time, but there is no additional slow down.
High Route Processor CPU Usage
High RP CPU usage may result from punting of packets from the PXF processors to the RP. Enter the show pxf accounting summary command to view the number and cause of punts.
Router# show pxf accounting summaryPkts Dropped RP Processed IgnoredTotal 0 48360 0PXF Statistic:Packets RP -> PXF:switch ip: 0switch raw: 30048360qos fastsend: 0qos enqueue: 1938Total: 30050298Packets PXF -> RP:qos pkts: 1938fast pkts: 30000000drops:total 0punts:total 48360" not IP : 40572" CEF no adjacency : 7788Total: 30050298Packets ignored: 0 | ring space:shadow ring full: 0 | shadow ring: 16384in ring full: 0 | inring: 968PXF inactive: 0tx credits: 16230330 | delayed credits: 0holdq enqueues: 0 | requeue drops: 0interrupts: 40538 | interrupt misses: 1947interrupt packets: 53326pending read bytes: 0Interface Pkts In Chars In Pkts Out Chars Out Punted DroppedFa0/0 0 0 30000000 1740000000 970 0Et1/0 0 0 0 0 21309 0Et1/1 0 0 0 0 0 0Et1/2 0 0 0 0 0 0Et1/3 0 0 0 0 0 0Se2/0 0 0 0 0 963 0Se2/1 0 0 0 0 0 0Se2/2 0 0 0 0 0 0Se2/3 0 0 0 0 0 0Fa3/0 0 0 0 0 963 0PO4/0 30000000 1440000000 0 0 963 0AT5/0 0 0 0 0 23192 0Vi1 0 0 0 0 0 0Vt1 0 0 0 0 0 0Vi2 0 0 0 0 0 0Vt2 0 0 0 0 0 0
Note
Ignored Packets
Packet ignores on an interface may result from high CPU usage. Enter the show interfaces command to display any input ignores:
Router# show interfaces ethernet 0/0Ethernet0/0 is up, line protocol is up...21 input errors, 0 CRC, 0 frame, 0 overrun, 21 ignoredPackets are ignored if there is no available CPU to accept the new packet. This can happen if the router is overloaded with traffic, but can also happen if the interface is faulty. If ignores are present on all interfaces, then the router is probably overloaded with traffic, or does not have sufficient free buffers in the pool that match the maximum transmission unit (MTU) on interfaces. In the latter case, an increment of the ignored counter is followed by an increment of the no buffer counter:
Router# show interfaces serial 0/0...1567 packets input, 0 bytes, 22 no buffer22 input errors, 0 CRC, 0 frame, 0 overrun, 22 ignored, 0 abortNo Buffers
If there are too many buffers configured for the output hold queue on an interface, this can use memory and result in dropped input packets. Enter the show running-config command and the show interfaces command to display the status of the incoming packet interfaces.
Priority and Policing Configured in the Class Policy
PXF does not support a configured priority and policing in the same class of a policy.
PXF Punts
If a feature that is supported in the PXF path appears not to be working, it could be the result of PXF failing to punt packets when it should. If you have only particular interfaces running the feature, you can configure a feature that PXF does not support on that interface to force PXF to punt packets.
QoS Fast-send and QoS Enqueue
PXF is responsible for managing the outbound traffic queues on an interface on which output QoS queueing is configured. All traffic destined for this interface must be processed by PXF, including both keepalive packets that originate in the router, known as QoS Fast-send packets, and packets that are switched in IOS, known as QoS Enqueue packets. If there is excessive traffic on the QoS Enqueue or on the QoS Fast-send path, QoS functionality can be disrupted. Use the show pxf accounting summary command to display the QoS Fast-send and QoS Enqueue packets:
Router# show pxf accounting summary...qos fastsend: 8qos enqueue: 5WFQ Queues
When there is an increase in ignored packets and in the no buffer count, it is possible that the output weighted fair queueing (WFQ) queues are too long. To specify the number of dynamic queues to be reserved for use by the class-default class as part of the default class policy, use the fair-queue policy-map class configuration command; for example:
Router# fair-queuepolicy-map policy9class class-defaultfair-queue 16queue-limit 20
