-
Cisco IOS Configuration Fundamentals Configuration Guide, Release 12.1
-
Table of Contents
-
About the Cisco IOS Documentation Set
-
Configuration Fundamentals Overview
- Part 1: Cisco IOS User Interfaces
- Part 2: Cisco IOS File Management
-
Part 3: Cisco IOS System Management
-
Configuring SNMP Support
-
Configuring RMON Support
-
Configuring Cisco Discovery Protocol (CDP)
-
Network Monitoring Using Cisco Service Assurance Agent (SAA)
-
Troubleshooting the Router
-
Performing Basic System Management
-
System Management Using System Controllers
-
Configuring Web Cache Services Using WCCP
-
Managing Dial Shelves
-
-
Appendix: Cisco IOS Command Modes
-
Index: Cisco IOS Configuration Fundamentals Configuration Guide, Release 12.1
-
Table Of Contents
Understanding Fault Management
Displaying System Information Using Show Commands
Receiving Automatic Warning Messages
Receiving the Automatic Shutdown Message
Setting Up the TCP Keepalive Packet Service
Testing Connections with the Ping Command
Enabling System Message Logging
Enabling Message Logging for a Slave Card
Setting the Error Message Display Device
Configuring Synchronization of Logging Messages
Enabling Timestamps on Logging (Syslog) Messages
Defining the Logging Message Severity Level and Facilities
Defining the UNIX System Logging Facility
Displaying Logging Information
Logging Errors to a UNIX Syslog Daemon
Setting the Syslog Source Address
Troubleshooting Specific Line Cards
Storing Line Card Crash Information
Creating Core Dumps for System Exceptions
Specifying the Destination for the Core Dump File
Creating an Exception Memory Core Dump
Creating a Spurious Interrupt Core Dump
Enable Conditionally Triggered Debugging
Enable Protocol-Specific Debug Commands
Enable Conditional Debugging Commands
Display Messages for One Interface
Display Messages for Multiple Interfaces
Limit Messages Based on Conditions
Conditionally Triggered Debugging Configuration Examples
Troubleshooting the Router
This chapter describes basic tasks that you can perform to troubleshoot your router and network in Cisco IOS Release 12.1. For detailed troubleshooting procedures and a variety of scenarios, see the Internetwork Troubleshooting Guide. For complete details on all debug commands, see the Cisco IOS Debug Command Reference.
For a complete description of the troubleshooting commands in this chapter, refer to the "Troubleshooting Commands" chapter in "Cisco IOS System Management Commands" part of the Cisco IOS Configuration Fundamentals Command Reference, Release 12.1. To locate documentation of other commands that appear in this chapter, use the command reference index or search online.
Understanding Fault Management
To manage network faults, you need to discover, isolate, and fix the problems. You can discover problems with the system's monitoring commands, isolate problems with the system's test commands, and resolve problems with other commands, including debug commands.
To perform general fault management, use the commands in the following sections:
•
Displaying System Information Using Show Commands
•
•
•
•
•
•
•
•
In addition, some chapters in the Cisco IOS software configuration guides include fault management tasks in a monitoring and maintaining section.
Displaying System Information Using Show Commands
To provide information about system processes, the Cisco IOS software includes an extensive list of EXEC commands that begin with the word show, which, when executed, display detailed tables of system information. Following is a partial list of system management show commands. Use these commands in EXEC mode to display the information described:
Look for specific show commands in the tables of configuration commands found throughout the chapters in Cisco IOS software configuration guides. See the Cisco IOS software command references for detailed descriptions of the commands.
Receiving Automatic Warning Messages
Some routers have an environmental monitor which monitors the physical condition of the router. If a measurement exceeds acceptable margins, a warning message is printed to the system console. The system software collects measurements once every 60 seconds, but warnings for a given test point are printed at most once every four hours. If the temperature measurements are out of specification more than the shutdown margin, the software shuts the router down but the fan will stay on. The router has to be manually turned off and on after such a shutdown. You can query the environmental monitor using the show environment command at any time to determine whether a measurement is out of tolerance. Refer to the System Error Messages publication for a description of environmental monitor warning messages.
Receiving the Automatic Shutdown Message
On routers with an environmental monitor, if the software detects that any of its temperature test points have exceeded maximum margins, it performs the following steps in this order:
1.
2.
3.
The following is the message the system displays if temperatures exceed maximum margins, along with a message indicating the reason for the shutdown:
Router#%ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdown%ENVM-2-TEMP: Inlet temperature has reached SHUTDOWN level at 64(C)Refer to the hardware installation and maintenance publication for your router for more information about environmental specifications.
Testing Network Connectivity
Use the commands in the following sections to test basic network connectivity:
•
•
Setting Up the TCP Keepalive Packet Service
The TCP keepalive capability allows a router to detect when the host with which it is communicating experiences a system failure, even if data stops being transmitted (in either direction). This is most useful on incoming connections. For example, if a host failure occurs while talking to a printer, the router might never notice, because the printer does not generate any traffic in the opposite direction. If keepalives are enabled, they are sent once every minute on otherwise idle connections. If five minutes pass and no keepalives are detected, the connection is closed. The connection is also closed if the host replies to a keepalive packet with a reset packet. This will happen if the host crashes and comes back up again.
To set up the TCP keepalive packet service, use the following command in global configuration mode:
Testing Connections with the Ping Command
As an aid to diagnosing basic network connectivity, many network protocols support an echo protocol. The protocol involves sending a special datagram to the destination host, then waiting for a reply datagram from that host. Results from this echo protocol can help in evaluating the path-to-host reliability, delays over the path, and whether the host can be reached or is functioning.
To use the echo protocol, use the following command in either user or privileged EXEC mode:
ping [protocol] {host | address}
Invoke a diagnostic tool for testing connectivity.
Look for specific ping commands in the tables of configuration commands found throughout the chapters in Cisco IOS software configuration guides. See the Cisco IOS software command references for detailed descriptions of the command.
Tracing Packet Routes
To discover the routes that packets will actually take when traveling to their destinations, use the following command in either user or privileged EXEC mode:
trace [protocol] [destination]
Trace packet routes through the network (privileged level).
Testing Memory and Interfaces
You can test the status of the following items:
•
•
•
Caution
Testing Flash Memory
To test the status of Flash memory, use the following command in privileged EXEC mode:
Testing System Memory
To test the status of system memory, use the following command in privileged EXEC mode:
Testing Interfaces
Caution
To test the status of the interfaces, use the following command on a nonoperational server in privileged EXEC mode:
Logging System Error Messages
By default, routers send the output from the debug EXEC command and system error messages to a logging process. The logging process controls the distribution of logging messages to the various destinations, such as the logging buffer, terminal lines, or a UNIX syslog server, depending on your configuration. The process also sends messages to the console. When the logging process is on, the messages are displayed on the console after the process that generated them has finished.
Note
When the logging process is disabled, messages are sent only to the console. The messages are sent as they are generated, so error and debug output will be interspersed with prompts or output from the command.
You can set the severity level of the messages to control the type of messages displayed for the console and each of the destinations. You can timestamp log messages or set the syslog source address to enhance real-time debugging and management.
Refer to the System Error Messages publication for information on possible error messages.
Enabling System Message Logging
Message logging is enabled by default. It must be enabled in order to send messages to any destination other than the console.
To disable message logging, use the no logging on command. Disabling the logging process can slow down the router because a process must wait until the messages are written to the console before continuing.
To reenable message logging after it has been disabled, use the following command in global configuration mode:
Enabling Message Logging for a Slave Card
To enable slave Versatile Interface Processor (VIP) cards to log important messages to the console, use the following command in global configuration mode:
Setting the Error Message Display Device
If message logging is enabled, you can send messages to specified locations, in addition to the console.
To specify the locations that receive messages, use one or more of the following commands in global configuration mode:
logging buffered [size]
Log messages to an internal buffer.
terminal monitor
Log messages to a nonconsole terminal.
logging host
Log messages to a UNIX syslog server host.
The logging buffered command copies logging messages to an internal buffer. The buffer is circular, so newer messages overwrite older messages after the buffer is full. To display the messages that are logged in the buffer, use the show logging EXEC command. The first message displayed is the oldest message in the buffer. To clear the current contents of the buffer, use the clear logging privileged EXEC command.
The EXEC command terminal monitor locally accomplishes the task of displaying the system error messages to a nonconsole terminal.
The logging command identifies a syslog server host to receive logging messages. The argument host is the name or Internet address of the host. By issuing this command more than once, you build a list of syslog servers that receive logging messages. The no logging command deletes the syslog server with the specified address from the list of syslogs.
Configuring Synchronization of Logging Messages
You can configure the system to synchronize unsolicited messages and debug command output with solicited device output and prompts for a specific line. You can identify the types of messages to be output asynchronously based on the level of severity. You can also determine the maximum number of buffers for storing asynchronous messages for the terminal after which messages are dropped.
When synchronous logging of unsolicited messages and debug command output is turned on, unsolicited device output is displayed on the console or printed after solicited device output is displayed or printed. Unsolicited messages and debug command output is displayed on the console after the prompt for user input is returned. Therefore, unsolicited messages and debug command output are not interspersed with solicited device output and prompts. After the unsolicited messages are displayed, the console displays the user prompt again.
To configure for synchronous logging of unsolicited messages and debug command output with solicited device output and prompts, use the following commands beginning in global configuration mode:
Enabling Timestamps on Logging (Syslog) Messages
By default, logging messages are not timestamped. You can enable timestamping of logging messages by using the following command in global configuration mode:
service timestamps log uptime
or
service timestamps log datetime [msec] [localtime] [show-timezone]
Enable log timestamps.
Defining the Logging Message Severity Level and Facilities
You can limit messages displayed to the selected device by specifying the severity level of the error message. To do so, use one of the following commands in global configuration mode:
If you have enabled syslog messages traps to be sent to an SNMP network management station with the snmp-server enable trap syslog command, you can also change the level of messages sent and stored in a history table on the router. You can also change the number of messages that get stored in the history table.
Messages are stored in the history table because SNMP traps are not guaranteed to reach their destination. By default, one message of the level warning and above (see Table 20) is stored in the history table even if syslog traps are not enabled.
To change the level and table size defaults, use the following commands in global configuration mode:
Note
The logging console command limits the logging messages displayed on the console terminal to messages with a level number at or below the specified severity level, which is specified by the level argument. Table 20 lists the error message level keywords and corresponding UNIX syslog definitions in order from the most severe level to the least severe level.
The no logging console command disables logging to the console terminal.
The default is to log messages to the console at the debugging level and those level numbers that are lower, which means all levels. The logging monitor command defaults to debugging also. The logging trap command defaults to informational.
To display logging messages on a terminal, use the terminal monitor EXEC command.
Current software generates four categories of error messages:
•
•
•
•
Defining the UNIX System Logging Facility
You can log messages produced by UNIX system utilities. To do this, enable this type logging and define the UNIX system facility from which you want to log messages. Table 21 lists the UNIX system facilities supported by the Cisco IOS software. Consult the operators manual for your UNIX operating system for more information about these UNIX system facilities.
Define UNIX system facility message logging by using the following command in global configuration mode:
Refer also to your syslog manual pages.
Displaying Logging Information
To display logging information, use the following command in EXEC mode:
Logging Errors to a UNIX Syslog Daemon
To set up the syslog daemon on a 4.3 BSD UNIX system, include a line such as the following in the /etc/syslog.conf file:
local7.debugging /usr/adm/logs/cisco.logThe debugging keyword specifies the syslog level; see Table 20 for a general description of other keywords. The local7 keyword specifies the logging facility to be used; see Table 21 for a general description of other keywords.
The syslog daemon sends messages at this level or at a more severe level to the file specified in the next field. The file must already exist, and the syslog daemon must have permission to write to it.
Setting the Syslog Source Address
By default, a syslog message contains the IP address of the interface it uses to leave the router. To require that all syslog messages contain the same IP address, regardless of which interface they use, use the following command in global configuration mode:
Using Field Diagnostics
Each line card on the Cisco 12000 series can perform field diagnostic testing to isolate faulty hardware without disrupting normal operation of the system. However, performing field diagnostic testing on a line card does halt all activity on the line card for the duration of the testing. After successful completion of the field diagnostic testing, the Cisco IOS software is automatically reloaded on the line card.
Note
To perform field diagnostic testing on a line card, perform the following commands in privileged EXEC mode:
To stop field diagnostic testing on a line card, use the following commands in privileged EXEC mode:
diag slot-number halt
or
no diag slot-number
Specify the line card that you want to stop perform diagnostic testing on.
Note
Troubleshooting Specific Line Cards
Cisco IOS provides the execute-on command to allow you issue IOS commands (such as show commands) to a specific line card for monitoring and maintenance. For example, you could show what Cisco IOS image is loaded on the card in slot 3 of a Cisco 12012 GSR by issuing the command execute-on slot 3 show version. You can also use this command for troubleshooting cards in the dial shelf of Cisco Access Servers. For complete documentation of this command see the "Troubleshooting" chapter of the Cisco IOS Configuration Fundamentals Command Reference.
Storing Line Card Crash Information
This section explains how to enable storing of crash information for a line card and optionally specify the type and amount of information stored. Technical support representatives need to be able to look at the crash information from the line card to troubleshoot serious problems on the line card. The crash information contains all the line card memory information including the main memory and transmit and receive buffer information.
Caution
To enable and configure the crash information options for a line card, use the following command in global configuration mode:
Creating Core Dumps for System Exceptions
"System exceptions" are any unexpected system shutdowns or reboots (most frequently caused by a system failure, commonly referred to as a "system crash"). When an exception occurs, it is sometimes useful to obtain a full copy of the memory image (called a core dump) to identify the cause of the unexpected shutdown. Not all exception types will produce a core dump.
Core dumps are generally useful only to your technical support representative. The core dump file, which is a very large binary file, must be transferred to a Trivial File Transfer Protocol (TFTP), File Transfer Protocol (FTP), or Remote Copy Protocol (RCP) server and subsequently interpreted by technical personnel who have access to source code and detailed memory maps.
The following commands are used to create core dumps:
Specifying the Destination for the Core Dump File
Core dumps can be sent to a remote server using the trivial file transfer protoco (TFTP), the file transfer protocol (FTP), or the remote copy protocol (rcp). Additionally, core dumps can be written to a local flash disk. To configure the destination for a core dump file, perform one or more of the tasks in the following sections:
•
Core Dumps Using TFTP
Due to a limitation of most TFTP applications, the router will only dump the first 16 MB of the core file. Therefore, if your router's main memory is larger than 16 MB, use of TFTP is not recommended.
To configure a router for a core dump using TFTP, use the following commands in global configuration mode:
For example, the following command configures a router to send a core file to the server at the IP address 172.17.92.2. As the exception protocol is not specified, the default protocol of TFTP will be used.
exception dump 172.17.92.2The core dump is written to a file named "hostname-core" on the TFTP server, where hostname is the name of the router. You can change the name of the core file by adding the exception core-file filename configuration command.
Depending on the TFTP server application used, it may be necessary to create, on the TFTP server, the empty target file to which the router can write the core. Also, make sure there is enough memory on your TFTP server to hold the complete core dump.
Core Dumps Using FTP
To configure the router for a core dump using FTP, use the following configuration commands:
This example configures a router to use FTP to dump a core file named "dumpfile" to the FTP server at 172.17.92.2 if there is a system failure.
ip ftp username redip ftp password blueexception protocol ftpexception dump 172.17.92.2exception core-file dumpfileUsing RCP
The remote copy protocol can also be used to send a core dump file. To configure the router to send core dump files using rcp, use the following commands:
When an rcp username is not configured through the ip rcmd remote-username command, the rcp username defaults to the username associated with the current terminal (tty) connection. For example, if the user is connected to the router through Telnet and was authenticated through the username command, the router software sends the Telnet username as the rcp username. If the terminal username is not available, the router hostname will be used as the rcp username.
Core Dumps Using a Flash Disk
Some router platforms support the flash disk as an alternative to the linear flash memory or PCMCIA flash card. The large storage capacity of these flash disks make them good candidates for another means of capturing a core dump. To configure a router for a core dump using a flash disk, use the following router configuration command:
The show flash all command will give you a list of devices you can use for the exception flash command above.
Creating an Exception Memory Core Dump
As a debugging procedure, you can cause the router to create a core dump and reboot when certain memory size parameters are violated. The following exception memory commands are used to trigger a core dump:
Note that the exception memory minimum command is primarily useful if you anticipate running out of memory before a core dump can be triggered or other debugging can be performed (rapid memory leak); if the memory leak is gradual (slow drift), you have generally have time to perform debugging before the system runs out of memory and must be reloaded.
By default, the number of free memory bytes is checked every 60 seconds when these commands are configured. The frequency of this checking can be increased using the memory check-interval seconds command.
The exception dump ip-address command must be configured with these commands. If the exception dump command is not configured, the router reloads without triggering a core dump.
The following example configures the router to monitor the free memory. If the memory falls below 250000 bytes, the core dump is created and the router reloads.
exception dump 131.108.92.2exception core-file memory.overrunexception memory minimum 250000Creating a Spurious Interrupt Core Dump
During the debugging process, you can configure the router to create a spurious interrupt core dump and reboot when a specified number of interrupts have occurred.
Caution
To enable and configure the crash information for spurious interrupts, use the following command
The following example configures a router to create a core dump with a limit of 2 spurious interrupts:
exception spurious-interrupt 2exception dump 209.165.200.225Enable Debug Operations
Your router includes hardware and software to aid in tracking down internal problems and problems with other hosts on the network. The debug privileged EXEC mode commands start the console display of several classes of network events. The following commands describe in general the system debug message feature. Refer to the Debug Command Reference for detailed information on speicifc debug commands. Also refer to the Internetwork Troubleshooting Guide publication for additional information.
Caution
You can configure timestamping of system debug messages. Timestamping enhances real-time debugging by providing the relative timing of logged events. This information is especially useful when customers send debugging output to your technical support personnel for assistance. To enable timestamping of system debug messages, use the following command in global configuration mode:
service timestamps debug uptime
or
service timestamps debug datetime [msec] [localtime] [show-timezone]
Enable timestamping of system debug messages.
Normally, the messages are displayed only on the console terminal. See the section "Setting the Error Message Display Device" earlier in this chapter to change the output device.
Enable Conditionally Triggered Debugging
When the Conditionally Triggered Debugging feature is enabled, the router generates debugging messages for packets entering or leaving the router on a specified interface; the router will not generate debugging output for packets entering or leaving through a different interface. You can specify the interfaces explicitly. For example, you may only want to see debugging messages for one interface or subinterface. You can also turn on debugging for all interfaces that meet specified condition. This feature is useful on dial access servers, which have a large number of ports.
Normally, the router will generate debugging messages for every interface, resulting in a large number of messages. The large number of messages consumes system resources, and can make it difficult to find the specific information you need. By limiting the debugging messages, you can receive messages related to only the ports you wish to troubleshoot.
Conditionally Triggered Debugging controls the output from the following protocol-specific debug commands:
•
•
•
•
•
While this feature limits the output of the above commands, it does not automatically enable the generation of debugging output from these commands. Debugging messages are generated only when the protocol-specific debug command is enabled. Debug command output is controlled through two processes:
•
•
To configure Conditionally Triggered Debugging, perform the following tasks:
•
•
Enable Protocol-Specific Debug Commands
In order to generate any debugging output, the protocol-specific debug command for the desired output must be enabled. Use the show debugging command to determine which types of debugging are enabled. Use the show debug condition command to display the current debug conditions. Use the following commands in privileged EXEC mode to enable the desired protocol-specific debug commands:
If you wish to have no output, disable all the protocol-specific debug commands.
Enable Conditional Debugging Commands
If no debug condition commands are enabled, all debugging output, regardless of the interface, will be displayed for the enabled protocol-specific debug commands.
The first debug condition command you enter enables conditional debugging. The router will only display messages for interfaces that meet one of the specified conditions. If multiple conditions are specified, the interface must meet at least one of the conditions in order for messages to be displayed.
You can enable messages for interfaces specified explicitly or for interfaces that meet certain conditions, as described in the following sections:
•
•
