Cisco IOS Release 12.0 Configuration Fundamentals Configuration Guide
Troubleshooting the Router

Table Of Contents

Troubleshooting the Router

Understanding Fault Management

Display System Information Using Show Commands

Receiving Automatic Warning Messages

Receiving the Automatic Shutdown Message

Using Field Diagnostics

Storing Line Card Crash Information

Test Network Connectivity

Set Up the TCP Keepalive Packet Service

Test Connections with the Ping Command

Trace Packet Routes

Test Memory and Interfaces

Test Flash Memory

Test System Memory

Test Interfaces

Log System Error Messages

Enable Message Logging

Enable Message Logging for a Slave Card

Set the Error Message Display Device

Configure Synchronization of Logging Messages

Enable Timestamps on Log Messages

Define the Error Message Severity Level and Facilities

Define the UNIX System Logging Facility

Display Logging Information

Log Errors to a UNIX Syslog Daemon

Set the Syslog Source Address

Use Field Diagnostics

Store Line Card Crash Information

Create Core Dumps

Specify the Protocol for the Core Dump

Using TFTP

Using FTP

Using RCP

Using a Flash Disk

Specifying the Name of the Core Dump File

Creating an Exception Memory Core Dump

Creating a Spurious Interrupt Core Dump

Enable Debug Operations

Enable Conditionally Triggered Debugging

Enable Protocol-Specific Debug Commands

Enable Conditional Debugging Commands

Display Messages for One Interface

Display Messages for Multiple Interfaces

Limit Messages Based on Conditions

Specify Multiple Conditions

Conditionally Triggered Debugging Configuration Examples


Troubleshooting the Router


This chapter describes basic tasks that you can perform to troubleshoot your router and network. For detailed troubleshooting procedures and a variety of scenarios, see the Internetwork Troubleshooting Guide. For complete details on all debug commands, see the Debug Command Reference.

For a complete description of the troubleshooting commands in this chapter, refer to the "Troubleshooting Commands" chapter of the Configuration Fundamentals Command Reference. To locate documentation of other commands that appear in this chapter, use the command reference index or search online.

Understanding Fault Management

To manage network faults, you need to discover, isolate, and fix the problems. You can discover problems with the system's monitoring commands, isolate problems with the system's test commands, and resolve problems with other commands, including debug commands.

To perform general fault management, use the commands in the following sections:

Display System Information Using Show Commands

Receiving Automatic Warning Messages

Receiving the Automatic Shutdown Message

Test Network Connectivity

Test Memory and Interfaces

Log System Error Messages

Use Field Diagnostics

Store Line Card Crash Information

Create Core Dumps

Enable Debug Operations

Enable Conditionally Triggered Debugging

In addition, some chapters in the Cisco IOS software configuration guides include fault management tasks in a monitoring and maintaining section.

Display System Information Using Show Commands

To provide information about system processes, the Cisco IOS software includes an extensive list of EXEC commands that begin with the word show, which, when executed, display detailed tables of system information. Following is a list of the more common system management show commands. Use these commands in EXEC mode to display the information described:

Command
Purpose

show c2600

Display information about the Cisco 2600 platform, including interrupts, IOS Priority Masks, and IDMA status, for troubleshooting.

show c7200

Display information about the CPU and midplane for the Cisco 7200 series routers.

show context

Display information stored in NVRAM when the router crashes. This command is only useful to your technical support representative. This command is supported on the Cisco 2600 and 7000 series routers.

show controllers (GRP image)

Display information that is specific to the Cisco 12000 series Gigabit Switch Router hardware options listed with this command.

show controllers (line card image)

Display information specific to the hardware on a line card installed in the Cisco 12000 series Gigabit Switch Router.

show controllers logging

Display logging information about a VIP card.

show controllers tech-support

Display general information about a VIP card when reporting a problem.

show diag

Display hardware information including DRAM and SRAM on the line cards.

show environment [all | last | table]

Display a message indicating whether an environmental warning condition currently exists, the temperature and voltage information, the last measured value from each of the six test points stored in nonvolatile memory, or the environmental specifications. This command is supported on the Cisco 7000 series routers.

show gsr

Display hardware information on the Cisco 12000 series Gigabit Switch Router.

show gt64010

Display all GT64010 internal registers and interrupt status on the Cisco 7200 series routers.

show memory [memory-type] [free] [summary]

Display memory pool statistics including summary information about the activities of the system memory allocator and a block-by-block listing of memory use.

show pci {hardware | bridge [register]}

Display information about the peripheral component interconnect (PCI) hardware registers or bridge registers for the Cisco 2600 and 7000 series routers.

show processes [cpu]

Display information about all active processes.

show processes memory

Display information about memory usage.

show protocols

Display the configured protocols.

show stacks

Display stack usage of processes and interrupt routines, including the reason for the last system reboot. This command is only useful to your technical support representative.

show subsys [class class | name name]

Display subsystem information.

show tcp [line-number]

Display the status of TCP connections.

show tcp brief [all]

Display a concise description of TCP connection endpoints.

show tdm connections [motherboard | slot number]

Display a snapshot of the time-division multiplexing (TDM) bus connection or data memory in a Cisco AS5200 access server.

show tech-support [page] [password]

show controllers vip slot-number tech-support

Display general information about the router or VIP card when reporting a problem.


Look for specific show commands in the tables of configuration commands found throughout the chapters in Cisco IOS software configuration guides. See the Cisco IOS software command references for detailed descriptions of the commands.

Receiving Automatic Warning Messages

Some routers have an environmental monitor which monitors the physical condition of the router. If a measurement exceeds acceptable margins, a warning message is printed to the system console. The system software collects measurements once every 60 seconds, but warnings for a given test point are printed at most once every four hours. If the temperature measurements are out of specification more than the shutdown margin, the software shuts the router down but the fan will stay on. The router has to be manually turned off and on after such a shutdown. You can query the environmental monitor using the show environment command at any time to determine whether a measurement is out of tolerance. Refer to the System Error Messages publication for a description of environmental monitor warning messages.

Receiving the Automatic Shutdown Message

On routers with an environmental monitor, if the software detects that any of its temperature test points have exceeded maximum margins, it performs the following steps in this order:

1 Saves the last measured values from each of the six test points to internal nonvolatile memory.

2 Interrupts the system software and causes a shutdown message to be printed on the system console.

3 Shuts off the power supplies after a few milliseconds of delay.

The following is the message the system displays if temperatures exceed maximum margins, along with a message indicating the reason for the shutdown:

Router#
%ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdown
%ENVM-2-TEMP: Inlet temperature has reached SHUTDOWN level at 64(C)

Refer to the hardware installation and maintenance publication for your router for more information about environmental specifications.

Using Field Diagnostics

Each line card on the Cisco 12000 series can perform field diagnostic testing to isolate faulty hardware without disrupting normal operation of the system. However, performing field diagnostic testing on a line card does halt all activity on the line card for the duration of the testing. After successful completion of the field diagnostic testing, the Cisco IOS software is automatically reloaded on the line card.


Note   The field diagnostic diag command must be executed from the GRP main console port.


To perform field diagnostic testing on a line card, use the following commands in privileged EXEC mode:

Step
Command
Purpose

1

diag slot-number [previous | post | verbose | wait]

Specify the line card that you want to perform diagnostic testing on.

Optionally, specify that previous test results are displayed, that only extended power-on self-tests (POST) be performed, that the maximum messages are displayed, or that the Cisco IOS software not be reloaded on the line card after successful completion of the tests. The following prompt is displayed.

Running Diags will halt ALL activity on 
the requested slot. [confirm]

At the prompt, press Return to confirm that you want to perform field diagnostic testing on the specified line card, or type no to stop the testing.


To stop field diagnostic testing on a line card, use the following commands in privileged EXEC mode:

Command
Purpose

diag slot-number halt

or

no diag slot-number

Specify the line card that you want to stop perform diagnostic testing on.



Note   When you stop the field diagnostic test, the line card remains down (that is, in an unbooted state). In most cases, you stopped the testing because you need to remove the line card or replace the line card. If that is not the case and you want to bring the line card back up (that is, on-line), you must use the microcode reload global configuration command or power cycle the line card.


Storing Line Card Crash Information

This section explains how to enable storing of crash information for a line card and optionally specify the type and amount of information stored. Technical support representatives need to be able to look at the crash information from the line card to troubleshoot serious problems on the line card. The crash information contains all the line card memory information including the main memory and transmit and receive buffer information.


Caution   
Use the exception linecard global configuration command only when directed by a technical support representative and only enable options that the technical support representative requests you to enable.

To enable and configure the crash information options for a line card, use the following command in global configuration mode:

Command
Purpose

exception linecard {all | slot slot-umber} [corefile filename | main-memory size [k | m] | queue-ram size [k | m] | rx-buffer size [k | m] |
sqe-register-rx | sqe-register-tx |
tx-buffer size [k | m]]

Specify the line card that you want crash information for when a line card resets. Optionally, specify the type and amount of memory to be stored.


Test Network Connectivity

Use the commands in the following sections to test basic network connectivity:

Set Up the TCP Keepalive Packet Service

Test Connections with the Ping Command

Trace Packet Routes

Set Up the TCP Keepalive Packet Service

The TCP keepalive capability allows a router to detect when the host with which it is communicating experiences a system failure, even if data stops being transmitted (in either direction). This is most useful on incoming connections. For example, if a host failure occurs while talking to a printer, the router might never notice, because the printer does not generate any traffic in the opposite direction. If keepalives are enabled, they are sent once every minute on otherwise idle connections. If five minutes pass and no keepalives are detected, the connection is closed. The connection is also closed if the host replies to a keepalive packet with a reset packet. This will happen if the host crashes and comes back up again.

To set up the TCP keepalive packet service, use the following command in global configuration mode:

Command
Purposes

service {tcp-keepalives-in | tcp-keepalives-out}

Generate TCP keepalive packets on idle network connections, either incoming connections initiated by a remote host, or outgoing connections initiated by a user.


Test Connections with the Ping Command

As an aid to diagnosing basic network connectivity, many network protocols support an echo protocol. The protocol involves sending a special datagram to the destination host, then waiting for a reply datagram from that host. Results from this echo protocol can help in evaluating the path-to-host reliability, delays over the path, and whether the host can be reached or is functioning.

To use the echo protocol, use the following command in either user or privileged EXEC mode:

Command
Purposes

ping [protocol] {host | address}

Invoke a diagnostic tool for testing connectivity.


Look for specific ping commands in the tables of configuration commands found throughout the chapters in Cisco IOS software configuration guides. See the Cisco IOS software command references for detailed descriptions of the command.

Trace Packet Routes

To discover the routes that packets will actually take when traveling to their destinations, use the following command in either user or privileged EXEC mode:

Command
Purposes

trace [protocol] [destination]

Trace packet routes through the network (privileged level).


Test Memory and Interfaces

You can test the status of the following items:

Flash memory

System memory

Interfaces


Caution   
We do not recommend using these test commands; they are intended to aid manufacturing personnel in checking system functionality.

Test Flash Memory

To test the status of Flash memory, use the following command in privileged EXEC mode:

Command
Purposes

test flash

Test Flash memory on MCI and envm Flash EPROM interfaces.


Test System Memory

To test the status of system memory, use the following command in privileged EXEC mode:

Command
Purposes

test memory

Diagnose Multibus memory, including nonvolatile memory.


Test Interfaces


Caution   
Do not use this test to diagnose problems with an operational server.

To test the status of the interfaces, use the following command on a nonoperational server in privileged EXEC mode:

Command
Purposes

test interfaces

Check network interfaces.


Log System Error Messages

By default, routers send the output from the debug EXEC command and system error messages to a logging process. The logging process controls the distribution of logging messages to the various destinations, such as the logging buffer, terminal lines, or a UNIX syslog server, depending on your configuration. The process also sends messages to the console. When the logging process is on, the messages are displayed on the console after the process that generated them has finished.


Note   The syslog format is compatible with 4.3 BSD UNIX.


When the logging process is disabled, messages are sent only to the console. The messages are sent as they are generated, so error and debug output will be interspersed with prompts or output from the command.

You can set the severity level of the messages to control the type of messages displayed for the console and each of the destinations. You can timestamp log messages or set the syslog source address to enhance real-time debugging and management.

Refer to the System Error Messages publication for information on possible error messages.

Enable Message Logging

Message logging is enabled by default. It must be enabled in order to send messages to any destination other than the console.

To disable message logging, use the no logging on command. Disabling the logging process can slow down the router because a process must wait until the messages are written to the console before continuing.

To re-enable message logging after it has been disabled, use the following command in global configuration mode:

Command
Purposes

logging on

Enable message logging.


Enable Message Logging for a Slave Card

To enable slave Versatile Interface Processor (VIP) cards to log important messages to the console, use the following command in global configuration mode:

Command
Purposes

service slave-log

Enable slave message logging.


Set the Error Message Display Device

If message logging is enabled, you can send messages to specified locations, in addition to the console.

To specify the locations that receive messages, use one or more of the following commands in global configuration mode:

Command
Purposes

logging buffered [size]

Log messages to an internal buffer.

terminal monitor

Log messages to a nonconsole terminal.

logging host

Log messages to a UNIX syslog server host.


The logging buffered command copies logging messages to an internal buffer. The buffer is circular, so newer messages overwrite older messages after the buffer is full. To display the messages that are logged in the buffer, use the show logging EXEC command. The first message displayed is the oldest message in the buffer. To clear the current contents of the buffer, use the clear logging privileged EXEC command.

The EXEC command terminal monitor locally accomplishes the task of displaying the system error messages to a nonconsole terminal.

The logging command identifies a syslog server host to receive logging messages. The argument host is the name or Internet address of the host. By issuing this command more than once, you build a list of syslog servers that receive logging messages. The no logging command deletes the syslog server with the specified address from the list of syslogs.

Configure Synchronization of Logging Messages

You can configure the system to synchronize unsolicited messages and debug command output with solicited device output and prompts for a specific line. You can identify the types of messages to be output asynchronously based on the level of severity. You can also determine the maximum number of buffers for storing asynchronous messages for the terminal after which messages are dropped.

When synchronous logging of unsolicited messages and debug command output is turned on, unsolicited device output is displayed on the console or printed after solicited device output is displayed or printed. Unsolicited messages and debug command output is displayed on the console after the prompt for user input is returned. Therefore, unsolicited messages and debug command output are not interspersed with solicited device output and prompts. After the unsolicited messages are displayed, the console displays the user prompt again.

To configure for synchronous logging of unsolicited messages and debug command output with solicited device output and prompts, use the following commands beginning in global configuration mode:

Step
Command
Purposes

1

line [aux | console | vty] line-number [ending-line-number]

Specify the line to be configured for synchronous logging of messages.

2

logging synchronous [level severity-level | all] [limit number-of-buffers]

Enable synchronous logging of messages.


Enable Timestamps on Log Messages

By default, log messages are not timestamped. You can enable timestamping of log messages by using the following command in global configuration mode:

Command
Purposes

service timestamps log uptime

or

service timestamps log datetime [msec] [localtime] [show-timezone]

Enable log timestamps.


Define the Error Message Severity Level and Facilities

You can limit messages displayed to the selected device by specifying the severity level of the error message. To do so, use one of the following commands in global configuration mode:

Command
Purposes

logging console level

Limit messages logged to the console.

logging monitor level

Limit messages logged to the terminal lines.

logging trap level

Limit messages logged to the syslog servers.


If you have enabled syslog messages traps to be sent to an SNMP network management station with the snmp-server enable trap command, you can also change the level of messages sent and stored in a history table on the router. You can also change the number of messages that get stored in the history table.

Messages are stored in the history table because SNMP traps are not guaranteed to reach their destination. By default, one message of the level warning and above (see ) is stored in the history table even if syslog traps are not enabled.

To change the level and table size defaults, use the following commands in global configuration mode:

Step
Command
Purposes

1

logging history level

Change the default level of syslog messages stored in the history file and sent to the SNMP server.

2

logging history size number

Change the number of syslog messages that can be stored in the history table.



Note    lists the level keywords and severity level. For SNMP usage, the severity level values use +1. For example, emergency equals 1 not 0 and critical equals 3 not 2.


The logging console command limits the logging messages displayed on the console terminal to messages with a level number at or below the specified severity level, which is specified by the level argument. lists the error message level keywords and corresponding UNIX syslog definitions in order from the most severe level to the least severe level.

Table 54 Error Message Logging Keywords 

Level Keyword
Level
Description
Syslog Definition

emergencies

0

System unusable

LOG_EMERG

alerts

1

Immediate action needed

LOG_ALERT

critical

2

Critical conditions

LOG_CRIT

errors

3

Error conditions

LOG_ERR

warnings

4

Warning conditions

LOG_WARNING

notifications

5

Normal but significant condition

LOG_NOTICE

informational

6

Informational messages only

LOG_INFO

debugging

7

Debugging messages

LOG_DEBUG


The no logging console command disables logging to the console terminal.

The default is to log messages to the console at the debugging level and those level numbers that are lower, which means all levels. The logging monitor command defaults to debugging also. The logging trap command defaults to informational.

To display logging messages on a terminal, use the terminal monitor EXEC command.

Current software generates four categories of error messages:

Error messages about software or hardware malfunctions, displayed at levels warnings through emergencies

Output from the debug commands, displayed at the debugging level

Interface up/down transitions and system restart messages, displayed at the notifications level

Reload requests and low-process stack messages, displayed at the informational level

Define the UNIX System Logging Facility

You can log messages produced by UNIX system utilities. To do this, enable this type logging and define the UNIX system facility from which you want to log messages. lists the UNIX system facilities supported by the Cisco IOS software. Consult the operators manual for your UNIX operating system for more information about these UNIX system facilities.

Define UNIX system facility message logging by using the following command in global configuration mode:

Command
Purposes

logging facility facility-type

Configure system log facilities.


Table 55 Logging Facility Type Keywords 

Facility Type Keyword
Description

auth

Indicates the authorization system.

cron

Indicates the cron facility.

daemon

Indicates the system daemon.

kern

Indicates the Kernel.

local0-7

Reserved for locally defined messages.

lpr

Indicates line printer system.

mail

Indicates mail system.

news

Indicates USENET news.

sys9

Indicates system use.

sys10

Indicates system use.

sys11

Indicates system use.

sys12

Indicates system use.

sys13

Indicates system use.

sys14

Indicates system use.

syslog

Indicates the system log.

user

Indicates user process.

uucp

Indicates UNIX-to-UNIX copy system.


Refer also to your syslog manual pages.

Display Logging Information

To display logging information, use the following command in EXEC mode:

Step
Command
Purposes

1

show logging

show controllers vip slot-number logging

Display the state of syslog error and event logging, including host addresses, whether console logging is enabled, and other logging statistics.

2

show logging history

Display information in the syslog history table such as the table size, the status of messages, and the text of the messages stored in the table.


Log Errors to a UNIX Syslog Daemon

To set up the syslog daemon on a 4.3 BSD UNIX system, include a line such as the following in the /etc/syslog.conf file:

local7.debugging /usr/adm/logs/cisco.log

The debugging keyword specifies the syslog level; see for a general description of other keywords. The local7 keyword specifies the logging facility to be used; see for a general description of other keywords.

The syslog daemon sends messages at this level or at a more severe level to the file specified in the next field. The file must already exist, and the syslog daemon must have permission to write to it.

Set the Syslog Source Address

By default, a syslog message contains the IP address of the interface it uses to leave the router. To require that all syslog messages contain the same IP address, regardless of which interface they use, use the following command in global configuration mode:

Command
Purposes

logging source-interface type number

Set the syslog source address.


Use Field Diagnostics

Each line card on the Cisco 12000 series can perform field diagnostic testing to isolate faulty hardware without disrupting normal operation of the system. However, performing field diagnostic testing on a line card does halt all activity on the line card for the duration of the testing. After successful completion of the field diagnostic testing, the Cisco IOS software is automatically reloaded on the line card.


Note   The field diagnostic diag command must be executed from the GRP main console port.


To perform field diagnostic testing on a line card, perform the following commands in privileged EXEC mode:

Step
Command
Purpose

1

diag slot-number [previous | post | verbose | wait]

Specify the line card that you want to perform diagnostic testing on.

Optionally, specify that previous test results are displayed, that only extended power-on self-tests (POST) be performed, that the maximum messages are displayed, or that the Cisco IOS software not be reloaded on the line card after successful completion of the tests. The following prompt is displayed.

Running Diags will halt ALL activity on 
the requested slot. [confirm]

At the prompt, press Return to confirm that you want to perform field diagnostic testing on the specified line card, or type no to stop the testing.


To stop field diagnostic testing on a line card, use the following commands in privileged EXEC mode:

Command
Purpose

diag slot-number halt

or

no diag slot-number

Specify the line card that you want to stop perform diagnostic testing on.



Note   When you stop the field diagnostic test, the line card remains down (that is, in an unbooted state). In most cases, you stopped the testing because you need to remove the line card or replace the line card. If that is not the case and you want to bring the line card back up (that is, on-line), you must use the microcode reload global configuration command or power cycle the line card.


Store Line Card Crash Information

This section explains how to enable storing of crash information for a line card and optionally specify the type and amount of information stored. Technical support representatives need to be able to look at the crash information from the line card to troubleshoot serious problems on the line card. The crash information contains all the line card memory information including the main memory and transmit and receive buffer information.


Caution   
Use the exception linecard global configuration command only when directed by a technical support representative and only enable options that the technical support representative requests you to enable.

To enable and configure the crash information options for a line card, use the following command in global configuration mode:

Command
Purpose

exception linecard {all | slot slot-number} [corefile filename | main-memory size [k | m] | queue-ram size [k | m] | rx-buffer size [k | m] |
sqe-register-rx | sqe-register-tx |
tx-buffer size [k | m]]

Specify the line card that you want crash information for when a line card resets. Optionally, specify the type and amount of memory to be stored.


Create Core Dumps

When your router crashes, it is sometimes useful to obtain a full copy of the memory image (called a core dump) to identify the cause of the crash. Not all crash types will produce a core dump.

Core dumps are generally useful only to your technical support representative. The core dump file, which is a very large binary file, must be transferred to a a Trivial File Transfer Protocol (TFTP), File Transfer Protocol (FTP), or Remote Copy Protocol (RCP) server and subsequently interpreted by technical personnel who have access to source code and detailed memory maps.

The following commands are used to create core dumps:

Command
Purpose

ip ftp password [type] password

Specifies the password to be used for FTP connections.

ip ftp username username

Configures the username for FTP connections.

exception protocol {ftp | rcp | tftp}

Configures the protocol used for core dumps.

exception flash

Configures the router for a core dump using a flash disk.

exception core-file name

Specifies the name of the core dump file when your router has generated a core dump file after crashing.

exception dump ip-address

Configures the router to dump a core file to a particular server when the router crashes.

exception memory {fragment size | minimum size}

Causes the router to create a core dump and reboot when certain memory size parameters are violated.

exception spurious-interrupt [number]

Causes the router to create a core dump and crash after a specified number of spurious interrupts.


Specify the Protocol for the Core Dump

There are four protocols that can be used to set up the router to generate a core dump:

Using TFTP (trivial file transfer protocol)

Using FTP (file transfer protocol)

Using RCP (remote copy protocol)

Using a Flash Disk

Using TFTP

Due to a limitation of most TFTP applications, the router will only dump the first 16 MB of the core file. Therefore, if your router's main memory is larger than 16 MB, do not use TFTP.

To configure a router for a core dump using TFTP, use the following router configuration command:

Command
Purpose

exception dump ip-address

Configures the router to dump a core file to a particular server when the router crashes.


For example, the following command configures a router for a core dump using TFTP:

exception dump 172.17.92.2

where 172.17.92.2 is the IP address of the TFTP server.

The core dump is written to a file named "hostname-core" on the TFTP server, where hostname is the name of the router. You can change the name of the core file by adding the exception core-file filename configuration command.

Depending on the TFTP server application used, it may be necessary to create, on the TFTP server, the empty target file to which the router can write the core. Also, make sure there is enough memory on your TFTP server to hold the complete core dump.

Using FTP

To configure the router for a core dump using FTP, use the following configuration commands:

Command
Purpose

ip ftp username username

Configures the username for FTP connections.

ip ftp password [type] password

Specifies the password to be used for FTP connections.

exception protocol {ftp | rcp | tftp}

Configures the protocol used for core dumps.

exception dump ip-address

Configures the router to dump a core file to a particular server when the router crashes.

exception core-file name

Specifies the name of the core dump file when your router has generated a core dump file after crashing.


The example configures a router to use FTP to dump a core file named dumpfile to the FTP server at 172.17.92.2 when it crashes.

ip ftp username red
ip ftp password blue
exception protocol ftp
exception dump 172.17.92.2
exception core-file dumpfile

Using RCP

RCP can also be used to capture a core dump. Enabling RCP on a router is described in "Configure a Router to Use rcp".

Once RCP is enabled on the router, the following commands should be added to capture the core dump:

Command
Purpose

exception protocol {ftp | rcp | tftp}

Configures the protocol used for core dumps.

exception dump ip-address

Configures the router to dump a core file to a particular server when the router crashes.


The following example sets 172.17.92.2 as the IP address of the host on which RCP is enabled:

     exception protocol rcp
     exception dump a.b.c.d

Using a Flash Disk

Some router platforms support the flash disk as an alternative to the linear flash memory or PCMCIA flash card. The large storage capacity of these flash disks make them good candidates for another means of capturing a core dump. To configure a router for a core dump using a flash disk, use the following router configuration command:

Command
Purpose

exception flash

Configures the router for a core dump using a flash disk.


The show flash all command will give you a list of devices you can use for the exception flash command above.

Specifying the Name of the Core Dump File

The exception core-file global configuration command enables you to specify the name of the core dump file when the router crashes. To specify a filename of a core dump file, use the following command:

Command
Purpose

exception core-file name

Specifies the name of the core dump file when your router has generated a core dump file after crashing.


Creating an Exception Memory Core Dump

During the debugging process, you can cause the router to create a core dump and reboot when certain memory size parameters are violated. The following exception memory commands are used to trigger a core dump:

Command
Purpose

exception memory fragment size

The minimum contiguous block of memory in the free pool, in bytes.

exception memory minimum size

The minimum size of the free memory pool, in bytes.


The size parameter is expressed in bytes and is checked every 60 seconds. If you enter a size that is greater than the free memory and the exception dump command has been configured, a core dump and router reload is generated after 60 seconds. If the exception dump command is not configured, the router reloads without triggering a core dump. The following example configures the router to monitor the free memory. If the memory falls below 250000 bytes, the core dump is created and the router reloads.

exception dump 131.108.92.2
exception core-file memory.overrun
exception memory minimum 250000

Creating a Spurious Interrupt Core Dump

During the debugging process, you can configure the router to create a spurious interrupt core dump and reboot when a specified number of interrupts have occurred.


Caution   Use the exception spurious-interrupt global configuration command only when directed by a technical support representative and only enable options requested by the technical support representative.

To enable and configure the crash information for spurious interrupts, use the following command:

Command
Purpose

exception spurious-interrupt [number]

Specify a number between 1 and 4294967295 to set the maximum number of spurious interrupts to include in the core dump before crashing.


The following example configures a router to create a core dump with a limit of 2 spurious interrupts:

Router# exception spurious-interrupy 2

Enable Debug Operations

Your router includes hardware and software to aid in tracking down internal problems and problems with other hosts on the network. The privileged debug EXEC commands start the console display of several classes of network events. The following commands describe in general the system debug message feature. Refer to the Debug Command Reference for all information regarding debug commands. Also refer to the Internetwork Troubleshooting Guide publication.

Command
Purposes

show debugging

Display the state of each debugging option.

debug ?

Display a list and brief description of all the debug command options.

debug command

Begin message logging for the specified debug command.

no debug command

Turn message logging off for the specified debug command.



Caution   
The system gives high priority to debugging output. For this reason, debugging commands should be turned on only for troubleshooting specific problems or during troubleshooting sessions with technical support personnel. Excessive debugging output can render the system inoperable.

You can configure timestamping of system debug messages. Timestamping enhances real-time debugging by providing the relative timing of logged events. This information is especially useful when customers send debugging output to your technical support personnel for assistance. To enable timestamping of system debug messages, use the following command in global configuration mode:

Command
Purposes

service timestamps debug uptime

or

service timestamps debug datetime [msec] [localtime] [show-timezone]

Enable timestamping of system debug messages.


Normally, the messages are displayed only on the console terminal. See the section "Set the Error Message Display Device" earlier in this chapter to change the output device.

Enable Conditionally Triggered Debugging

When the Conditionally Triggered Debugging feature is enabled, the router generates debugging messages for packets entering or leaving the router on a specified interface; the router will not generate debugging output for packets entering or leaving through a different interface. You can specify the interfaces explicitly. For example, you may only want to see debugging messages for one interface or subinterface. You can also turn on debugging for all interfaces that meet specified condition. This feature is useful on dial access servers, which have a large number of ports.

Normally, the router will generate debugging messages for every interface, resulting in a large number of messages. The large number of messages consumes system resources, and can make it difficult to find the specific information you need. By limiting the debugging messages, you can receive messages related to only the ports you wish to troubleshoot.

Conditionally Triggered Debugging controls the output from the following protocol-specific debug commands:

debug aaa {accounting | authorization | authentication}

debug dialer {events | packets}

debug isdn {q921 | q931}

debug modem {oob | trace}

debug ppp {all | authentication | chap | error | negotiation | multilink events | packet}

While this feature limits the output of the above commands, it does not automatically enable the generation of debugging output from these commands. Debugging messages are generated only when the protocol-specific debug command is enabled. Debug command output is controlled through two processes:

The protocol-specific debug commands specify which protocols are being debugged. For example, the debug dialer events command generates debugging output related to dialer events.

The debug condition commands limit these debugging messages to those related to a particular interface. For example, the debug condition username bob command generates debugging output only for interfaces with packets that specify a username of bob.

To configure Conditionally Triggered Debugging, perform the following tasks:

Enable Protocol-Specific Debug Commands

Enable Conditional Debugging Commands

Specify Multiple Conditions

Enable Protocol-Specific Debug Commands

In order to generate any debugging output, the protocol-specific debug command for the desired output must be enabled. Use the show debugging command to determine which types of debugging are enabled. Use the show debug condition command to display the current debug conditions. Use the following commands in privileged EXEC mode to enable the desired protocol-specific debug commands:

Command
Purpose

show debugging

Determine which types of debugging are enabled.

show debug condition [condition-id]

Displays the current debug conditions.

debug protocol

Enable the desired debugging commands.

no debug protocol

Disable the debugging commands that are not desired.


If you wish to have no output, disable all the protocol-specific debug commands.

Enable Conditional Debugging Commands

If no debug condition commands are enabled, all debugging output, regardless of the interface, will be displayed for the enabled protocol-specific debug commands.

The first debug condition command you enter enables conditional debugging. The router will only display messages for interfaces that meet one of the specified conditions. If multiple conditions are specified, the interface must meet at least one of the conditions in order for messages to be displayed.

You can enable messages for interfaces specified explicitly or for interfaces that meet certain conditions, as described in the following sections:

Display Messages for One Interface

Display Messages for Multiple Interfaces

Limit Messages Based on Conditions

Display Messages for One Interface

To disable debugging messages for all interfaces except one, use the following command in privileged EXEC mode:

Command
Purpose

debug condition interface interface

Disable debugging messages for all interfaces except one.


If you enter the debug condition interface command, the debugging output will be turned off for all interfaces except the specified interface. To reenable debugging output for all interfaces, use the no debug interface command.

Display Messages for Multiple Interfaces

To enable debugging messages for multiple interfaces, use the following commands in privileged EXEC mode:

Command
Purpose

debug condition interface interface

Disable debugging messages for all interfaces except one.

debug condition interface interface

Enable debugging messages for additional interfaces. Repeat this task until debugging messages are enabled for all desired interfaces.


If you specify more than one interface by entering this command multiple times, debugging output will be displayed for all of the specified interfaces. To turn off debugging on a particular interface, use the no debug interface command. If you use the no debug interface all command or remove the last debug interface command, debugging output will be reenabled for all interfaces.

Limit Messages Based on Conditions

The router can monitor interfaces to see if any packets contain the specified value for one of the following conditions:

username

calling party number

called party number

If you enter a condition, such as calling number, debug output will be stopped for all interfaces. The router will then monitor every interface to see if a packet with the specified calling party number is sent or received on any interfaces. If the condition is met on an interface or subinterface, debug command output will be displayed for that interface. The debugging output for an interface is "triggered" when the condition has been met. The debugging output continues to be disabled for the other interfaces. If, at some later time, the condition is met for another interface, the debug output will become enabled for that interface as well.

Once debugging output has been triggered on an interface, the output will continue until the interface goes down. However, the session for that interface might change, resulting in a new username, called party number, or calling party number. Use the no debug interface command to reset the debug trigger mechanism for a particular interface. The debugging output for that interface will be disabled until the interface meets one of the specified conditions.

To limit debugging messages based on a specified condition, use the following command in privileged EXEC mode:

Command
Purpose

debug condition {username username | called dial-string | caller dial-string}

Enable conditional debugging. The router will only display messages for interfaces that meet this condition.


To reenable the debugging output for all interfaces, enter the no debug condition all command.

Specify Multiple Conditions

To limit debugging messages based on more than one condition, use the following commands in privileged EXEC mode:

Command
Purpose

debug condition {username username | called dial-string | caller dial-string}

Enable conditional debugging and specify the first condition.

debug condition {username username | called dial-string | caller dial-string}

Specify the second condition. Repeat this task until all conditions are specified.


If you enter multiple debug condition commands, debugging output will be generated if an interface meets at least one of the conditions. If you remove one of the conditions, using the no debug condition command, interfaces that meet only that condition will no longer produce debugging output. However, interfaces that meet a condition other than the removed condition will continue to generate output. Only if no active conditions are met for an interface will the output for that interface be disabled.

Conditionally Triggered Debugging Configuration Examples

In this example, four conditions have been set by the following commands:

debug condition interface serial 0

debug condition interface serial 1

debug condition interface virtual-template 1

debug condition username fred

The first three conditions have been met by one interface. The fourth condition has not yet been met.

Router# show debug condition

Condition 1: interface Se0 (1 flags triggered)
        Flags: Se0
Condition 2: interface Se1 (1 flags triggered)
        Flags: Se1
Condition 3: interface Vt1 (1 flags triggered)
        Flags: Vt1
Condition 4: username fred (0 flags triggered)

When any debug condition command is entered, debugging messages for conditional debugging are enabled. The following debugging messages show conditions being met on different interfaces as the serial 0 and serial 1 interfaces come up. For example, the second line of output indicates that serial interface 0 meets the username fred condition.

*Mar  1 00:04:41.647: %LINK-3-UPDOWN: Interface Serial0, changed state to up
*Mar  1 00:04:41.715: Se0 Debug: Condition 4, username fred triggered, count 2
*Mar  1 00:04:42.963: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0, changed 
state to up
*Mar  1 00:04:43.271: Vi1 Debug: Condition 3, interface Vt1 triggered, count 1
*Mar  1 00:04:43.271: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to up
*Mar  1 00:04:43.279: Vi1 Debug: Condition 4, username fred triggered, count 2
*Mar  1 00:04:43.283: Vi1 Debug: Condition 1, interface Se0 triggered, count 3
*Mar  1 00:04:44.039: %IP-4-DUPADDR: Duplicate address 172.27.32.114 on Ethernet 0, 
sourced by 00e0.1e3e.2d41
*Mar  1 00:04:44.283: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, 
changed state to up
*Mar  1 00:04:54.667: %LINK-3-UPDOWN: Interface Serial1, changed state to up
*Mar  1 00:04:54.731: Se1 Debug: Condition 4, username fred triggered, count 2
*Mar  1 00:04:54.735: Vi1 Debug: Condition 2, interface Se1 triggered, count 4
*Mar  1 00:04:55.735: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1, changed 
state to up

After a period of time, the show debug condition command displays the revised list of conditions.

Router# show debug condition

Condition 1: interface Se0 (2 flags triggered)
        Flags: Se0 Vi1
Condition 2: interface Se1 (2 flags triggered)
        Flags: Se1 Vi1
Condition 3: interface Vt1 (2 flags triggered)
        Flags: Vt1 Vi1
Condition 4: username fred (3 flags triggered)
        Flags: Se0 Vi1 Se1

Next, the serial 1 and serial 0 interfaces go down. When an interface goes down, conditions for that interface are cleared.

*Mar  1 00:05:51.443: %LINK-3-UPDOWN: Interface Serial1, changed state to down
*Mar  1 00:05:51.471: Se1 Debug: Condition 4, username fred cleared, count 1
*Mar  1 00:05:51.479: Vi1 Debug: Condition 2, interface Se1 cleared, count 3
*Mar  1 00:05:52.443: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1, changed 
state to down
*Mar  1 00:05:56.859: %LINK-3-UPDOWN: Interface Serial0, changed state to down
*Mar  1 00:05:56.887: Se0 Debug: Condition 4, username fred cleared, count 1
*Mar  1 00:05:56.895: Vi1 Debug: Condition 1, interface Se0 cleared, count 2
*Mar  1 00:05:56.899: Vi1 Debug: Condition 3, interface Vt1 cleared, count 1
*Mar  1 00:05:56.899: Vi1 Debug: Condition 4, username fred cleared, count 0
*Mar  1 00:05:56.903: %LINK-3-UPDOWN: Interface Virtual-Access1, changed state to down
*Mar  1 00:05:57.907: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0, changed 
state to down
*Mar  1 00:05:57.907: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, 
changed state to down

The final show debug condition output is the same as the output before the interfaces came up.

Router# show debug condition

Condition 1: interface Se0 (1 flags triggered)
        Flags: Se0
Condition 2: interface Se1 (1 flags triggered)
        Flags: Se1
Condition 3: interface Vt1 (1 flags triggered)
        Flags: Vt1
Condition 4: username fred (0 flags triggered)