Your software release may not support all the features documented in this module. For the latest caveats and feature information, see Bug Search Tool and the release notes for your platform and software release. To find information about the features documented in this module, and to see a list of the releases in which each feature is supported, see the feature information table at the end of this module.

Use Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to www.cisco.com/go/cfn. An account on Cisco.com is not required.

To manage network faults, you need to discover, isolate, and correct problems. You can discover problems with the system monitoring commands, isolate problems with the system test commands, and resolve problems with other commands, including debug commands.

To perform general fault management, perform the tasks described in the following sections:

In addition to the material presented in this chapter, many chapters in the Cisco IOS software configuration guides include fault management tasks specific to certain technologies and features. You can find these tasks in the "Monitoring and Maintaining" sections.

To provide information about system processes, the Cisco IOS software includes an extensive list of show EXEC commands. Following is a partial list of system management show commands. To display the information described, use the following commands in EXEC mode, as needed:

Router# show c2600

Router# show c7200

Router# show context

Router# show controllers

Router# show controllers logging

Router# show controllers tech-support

Router# show controllers vip  slot-number  tech-support

Router# show diag

Router# show environment [all | last | table]

Router# show gsr

Router# show gt64010

Router# show memory [memory-type] [free] [summary] 

Router# show pci {hardware | bridge [register]}

Router# show processes [cpu]

Router# show processes memory

Router# show protocols

Router# show stacks

Router# show subsys [class class | name name]

Router# show tcp [line-number]

Router# show tcp brief [all]

Router# show tdm connections [motherboard | slot  number] 

Router# show tech-support [page] [password]

Refer to specific show commands in the tables of configuration commands found throughout the chapters in Cisco IOS software configuration guides. Refer to the Cisco IOS software command reference publications for detailed descriptions of the commands.

• Configuring the TCP Keepalive Packet Service
• Testing Connections with the ping Command
• Tracing Packet Routes

By default, routers send logging messages (including debug command output) a logging process. The logging process controls the distribution of logging messages to various destinations, such as the logging buffer, terminal lines, or a UNIX syslog server, depending on your configuration. The process also sends messages to the console. When the logging process is on, the messages are displayed on the console after the process that generated them has finished.

When the logging process is disabled, messages are sent only to the console. The messages are sent as they are generated, so error and debug output will be interspersed with prompts or output from the command.

You can set the severity level of the messages to control the type of messages displayed for the console and each destination. You can time-stamp log messages or set the syslog source address to enhance real-time debugging and management.

System logging messages are traditionally referred to as System Error Messages. Refer to the Cisco IOS Software System Error Messages publication for detailed information on specific system logging messages.

• Enabling System Message Logging
• Enabling Message Logging for a Slave Card
• Setting the Syslog Destination
• Configuring Synchronization of Logging Messages
• Enabling Time-Stamps on Log Messages
• Limiting the Error Message Severity Level and Facilities
• Defining the UNIX System Logging Facility
• Displaying Logging Information
• Logging Errors to a UNIX Syslog Daemon
• Setting the Syslog Source Address

Each line card on the Cisco 12000 series routers can perform field diagnostic testing to isolate faulty hardware without disrupting normal operation of the system. However, performing field diagnostic testing on a line card does halt all activity on the line card for the duration of the testing. After successful completion of the field diagnostic testing, the Cisco IOS software is automatically reloaded on the line card.

Note	The field diagnostic diag command must be executed from the Gigabit Route Processor (GRP) main console port.

To perform field diagnostic testing on a line card, use the following command in privileged EXEC mode:

Router# diag  slot-number [previous | post | verbose | wait]

Running Diags will halt ALL activity on the requested slot. [confirm]

To stop field diagnostic testing on a line card, use either of the following commands in privileged EXEC mode:

Router# diag  slot-number  halt

Router# no diag  slot-number 

Note

When you stop the field diagnostic test, the line card remains down (that is, in an unbooted state). In most cases, you stopped the testing because you need to remove the line card or replace the line card. If that is not the case and you want to bring the line card back up (that is, online), you must use the microcode reload global configuration command or power cycle the line card.

Cisco IOS provides the execute-on command to allow you to issue Cisco IOS commands (such as show commands) to a specific line card for monitoring and maintenance. For example, you could show which Cisco IOS image is loaded on the card in slot 3 of a Cisco 12012 Gigabit Switch Router (GSR) by issuing the execute-on slot 3 show versioncommand. You can also use this command for troubleshooting cards in the dial shelf of Cisco access servers.

This section explains how to enable storing of crash information for a line card and optionally specify the type and amount of information stored. Technical support representatives need to be able to look at the crash information from the line card to troubleshoot serious problems on the line card. The crash information contains all the line card memory information, including the main memory and transmit and receive buffer information.

Caution

Use the exception linecard global configuration command only when directed by a technical support representative, and only enable options that the technical support representative requests you to enable.

To enable and configure the crash information options for a line card, use the following command in global configuration mode.

Router(config)# exception linecard {all | slot  slot-number} [corefile filename | main-memory size [k | m] | queue-ram size [k | m] | rx-buffer  size [k | m] | sqe-register-rx | sqe-register-tx | tx-buffer size [k | m]]

"System exceptions" are any unexpected system shutdowns or reboots (most frequently caused by a system failure, commonly referred to as a "system crash"). When an exception occurs, it is sometimes useful to obtain a full copy of the memory image (called a core dump) to identify the cause of the unexpected shutdown. Not all exception types will produce a core dump.

Core dumps are generally useful only to your technical support representative. The core dump file, which is a very large binary file, can be transferred to a Trivial File Transfer Protocol (TFTP), File Transfer Protocol (FTP), or Remote Copy Protocol (RCP) server, or (on limited platforms) saved to the flash disk, and subsequently interpreted by technical personnel who have access to source code and detailed memory maps.

Caution

Use the exception commands only under the direction of a technical support representative. Creating a core dump while the router is functioning in a network can disrupt network operation.

• Specifying the Destination for the Core Dump File
• Creating an Exception Memory Core Dump

Your router includes hardware and software to aid in troubleshooting internal problems and problems with other hosts on the network. The debug privileged EXEC mode commands start the console display of several classes of network events. The following commands describe in general the system debug message feature. Refer to the Cisco IOS Debug Command Reference for all information regarding debug commands. Also refer to the Internetwork Troubleshooting Guide publication for additional information.

To enable debugging operations, use the following commands:

Router# show debugging 

Router# debug ?

Router# debug  command

Router# no debug  command

Caution

The system gives high priority to debugging output. For this reason, debugging commands should be turned on only for troubleshooting specific problems or during troubleshooting sessions with technical support personnel. Excessive debugging output can render the system inoperable.

You can configure time-stamping of system debug messages. Time-stamping enhances real-time debugging by providing the relative timing of logged events. This information is especially useful when customers send debugging output to your technical support personnel for assistance. To enable time-stamping of system debug messages, use either of the following commands in global configuration mode:

Router(config)# service timestamps debug uptime

Router(config)# service timestamps debug datetime [msec] [localtime] [show-timezone]

Normally, the messages are displayed only on the console terminal. Refer to the section "Setting the Syslog Destination" earlier in this chapter to change the output device.

When the Conditionally Triggered Debugging feature is enabled, the router generates debugging messages for packets entering or leaving the router on a specified interface; the router will not generate debugging output for packets entering or leaving through a different interface. You can specify the interfaces explicitly. For example, you may only want to see debugging messages for one interface or subinterface. You can also turn on debugging for all interfaces that meet specified condition. This feature is useful on dial access servers, which have a large number of ports.

Normally, the router will generate debugging messages for every interface, resulting in a large number of messages. The large number of messages consumes system resources, and can affect your ability to find the specific information you need. By limiting the number of debugging messages, you can receive messages related to only the ports you wish to troubleshoot.

Conditionally Triggered Debugging controls the output from the following protocol-specific debug commands:

• debug aaa {accounting | authorization | authentication}
• debug dialer {events | packets}
• debug isdn {q921 | q931}
• debug modem {oob | trace}
• debug ppp {all | authentication | chap | error | negotiation | multilink events | packet}

Although this feature limits the output of the commands listed, it does not automatically enable the generation of debugging output from these commands. Debugging messages are generated only when the protocol-specific debug command is enabled. The debug command output is controlled through two processes:

• The protocol-specific debug commands specify which protocols are being debugged. For example, the debug dialer events command generates debugging output related to dialer events.
• The debug condition commands limit these debugging messages to those related to a particular interface. For example, the debug condition username bob command generates debugging output only for interfaces with packets that specify a username of bob.

To configure Conditionally Triggered Debugging, perform the tasks described in the following sections:

• Enabling Protocol-Specific debug Commands
• Enabling Conditional Debugging Commands
• Specifying Multiple Debugging Conditions
• Conditionally Triggered Debugging Configuration Examples

Some routers and access servers have an environmental monitor that monitors the physical condition of the router. If a measurement exceeds acceptable margins, a warning message is printed to the system console. The system software collects measurements once every 60 seconds, but warnings for a given test point are printed at most once every 4 hours. If the temperature measurements are out of specification more than the shutdown, the software shuts the router down (the fan will remain on). The router must be manually turned off and on after such a shutdown. You can query the environmental monitor using the show environment command at any time to determine whether a measurement is out of tolerance. Refer to the Cisco IOS System Error Messages publication for a description of environmental monitor warning messages.

On routers with an environmental monitor, if the software detects that any of its temperature test points have exceeded maximum margins, it performs the following steps:

1. Saves the last measured values from each of the six test points to internal nonvolatile memory.
2. Interrupts the system software and causes a shutdown message to be printed on the system console.
3. Shuts off the power supplies after a few milliseconds of delay.

The system displays the following message if temperatures exceed maximum margins, along with a message indicating the reason for the shutdown:

Router#
%ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdown
%ENVM-2-TEMP: Inlet temperature has reached SHUTDOWN level at 64(C)

Refer to the hardware installation and maintenance publication for your router for more information about environmental specifications.