To troubleshoot problems with voice networks, the following resources are required:
•Other Troubleshooting Resources
This section presents information about the variety of tools available to assist you in troubleshooting your voice network, including information on using router diagnostic commands, Cisco network management tools, and third-party troubleshooting tools.
Using Router Diagnostic Commands
Cisco routers provide numerous integrated commands to assist you in monitoring and troubleshooting your internetwork.
•show commands help you monitor installation behavior and normal network behavior, and isolate problem areas.
•debug commands help you isolate protocol and configuration problems.
•ping commands help you determine connectivity between devices on your network.
•trace commands provide a method of determining the route by which packets reach their destination.
Using show Commands
show commands are powerful monitoring and troubleshooting tools. You can use show commands to perform a variety of functions:
•Monitor router behavior during initial installation.
•Monitor normal network operation.
•Isolate problem interfaces, nodes, media, or applications.
•Determine when a network is congested.
•Determine the status of servers, clients, or other neighbors.
The following are some of the most common show commands:
•show version—Displays the configuration of the system hardware, the software version, the names and sources of configuration files, and the boot images.
•show running-config—Displays the router configuration currently running.
•show startup-config—Displays the router configuration stored in NVRAM.
•show interfaces—Displays statistics for all interfaces configured on the router or access server. The resulting output varies, depending on the network for which an interface has been configured.
•show controllers—Displays statistics for interface card controllers.
•show flash—Displays the layout and contents of flash memory.
•show buffers—Displays statistics for the buffer pools on the router.
•show memory summary—Displays memory pool statistics and summary information about the activities of the system memory allocator, and gives a block-by-block listing of memory use.
•show process cpu—Displays information about the active processes on the router.
•show stacks—Displays information about the stack utilization of processes and interrupt routines, and the reason for the last system reboot.
•show cdp neighbors—Provides reachability information for directly connected Cisco devices. This is an extremely useful tool for determining the operational status of the physical and data link layers. Cisco Discovery Protocol (CDP) is a proprietary data link layer protocol.
•show debugging—Displays information about the type of debugging that is enabled for your router.
You can always use the ? at the command line for a list of subcommands.
Like the debug commands, some of the show commands listed are accessible only at the router's privileged EXEC mode. Debug command usage is explained further in the "Using debug Commands" section and also in the "Debug Command Output on Cisco IOS Voice Gateways" chapter.
Hundreds of other show commands are available. For details on using and interpreting the output of specific show commands, refer to the Cisco IOS command references.
Using debug Commands
The debug privileged EXEC commands can provide a wealth of information about the traffic seen (or not seen) on an interface, error messages generated by nodes on the network, protocol-specific diagnostic packets, and other useful troubleshooting data.
The following steps are a summary of debug command usage. For detailed debug command usage, see the "Debug Command Output on Cisco IOS Voice Gateways" chapter.
To access and list the privileged EXEC commands, enter this code:
Note the change in the router prompts here. The # prompt (instead of the normal > prompt) indicates that the router is in privileged EXEC mode (enable mode).
Exercise care when using
debug commands. Many
debug commands are processor-intensive and can cause serious network problems (such as degraded performance or loss of connectivity) if they are enabled on an already heavily loaded router. When you finish using a
debug command, remember to disable it with its specific
no debug command (or use the
no debug all command to turn off all debugging).
Use debug commands to isolate problems, not to monitor normal network operation. Because the high processor overhead of debug commands can disrupt router operation, you should use them only when you are looking for specific types of traffic or problems, and have narrowed your problems to a likely subset of causes.
Output formats vary with each debug command. Some generate a single line of output per packet, and others generate multiple lines of output per packet. Some generate large amounts of output, and others generate only occasional output. Some generate lines of text, and others generate information in field format.
To minimize the negative impact of using debug commands, follow this procedure:
Step 1 Use the no logging console global configuration command on your router. This command disables all logging to the console terminal.
Step 2 Telnet to a router port and enter the enable EXEC command. The enable EXEC command places the router in the privileged EXEC mode. After entering the enable password, you receive a prompt that consists of the router name with a # symbol.
Step 3 Use the terminal monitor command to copy debug command output and system error messages to your current terminal display.
By redirecting output to your current terminal display, you can view debug command output remotely, without being connected through the console port.
If you use debug commands at the console port, character-by-character processor interrupts are generated, maximizing the processor load already caused by the use of debug.
If you intend to keep the output of the debug command, spool the output to a file. The procedure for setting up such a debug output file is described in the "Debug Command Output on Cisco IOS Voice Gateways" chapter.
This book refers to specific debug commands that are useful when you are troubleshooting specific problems. Complete details regarding the function and output of debug commands are provided in the Cisco IOS Debug Command Reference.
In many situations, using third-party diagnostic tools can be more useful and less intrusive than using debug commands. For more information, see the "Third-Party Troubleshooting Tools" section.
Using the ping Commands
To check host reachability and network connectivity, use the ping command, which can be invoked from both user EXEC mode and privileged EXEC mode. After you log in to the router or access server, the router is automatically in user EXEC command mode. The EXEC commands available at the user level are a subset of those available at the privileged level. In general, the user EXEC commands enable you to connect to remote devices, change terminal settings on a temporary basis, perform basic tests, and list system information. The ping command can be used to confirm basic network connectivity on a variety of networks.
For IP, the ping command sends Internet Control Message Protocol (ICMP) echo messages. ICMP is the Internet protocol that reports errors and provides information relevant to IP packet addressing. If a station receives an ICMP Echo message, it sends an ICMP echo reply message back to the source.
The extended command mode of the ping command permits you to specify the supported IP header options. This mode allows the router to perform a more extensive range of test options. To enter ping extended command mode, enter yes at the extended commands prompt of the ping command.
It is a good idea to use the ping command when the network is functioning properly to see how the command works under normal conditions, so that you have a basis for comparison when you are troubleshooting.
For detailed information on using the ping and extended ping commands, refer to the Cisco IOS Configuration Fundamentals and Network Management Command Reference.
Using the trace Commands
The trace user EXEC command discovers the routes that a router's packets follow when traveling to their destinations. The trace privileged EXEC command permits the supported IP header options to be specified, allowing the router to perform a more extensive range of test options.
The trace command works by using the error message generated by routers when a datagram exceeds its time-to-live (TTL) value. First, probe datagrams are sent with a TTL value of 1. This value causes the first router to discard the probe datagrams and send back "time exceeded" error messages. The trace command then sends several probes and displays the round-trip time for each. After every third probe, the TTL is increased by 1.
Each outgoing packet can result in one of two error messages. A "time exceeded" error message indicates that an intermediate router has seen and discarded the probe. A "port unreachable" error message indicates that the destination node has received the probe and discarded it because it could not deliver the packet to an application. If the timer goes off before a response comes in, trace prints an asterisk (*).
The trace command terminates when the destination responds, when the maximum TTL is exceeded, or when you interrupt the trace with the escape sequence.
As with ping, it is a good idea to use the trace command when the network is functioning properly to see how the command works under normal conditions, so that you have a basis for comparison when you are troubleshooting.
For detailed information on using the trace and extended trace commands, refer to the Cisco IOS Configuration Fundamentals and Network Management Command Reference.
Using Cisco Network Management Tools
Cisco offers the CiscoWorks 2000 family of management products that provide design, monitoring, and troubleshooting tools to help you manage your internetwork.
The following tools are useful for troubleshooting internetwork problems:
•CiscoView provides dynamic monitoring and troubleshooting functions, including a graphical display of Cisco devices, statistics, and comprehensive configuration information.
•Internetwork Performance Monitor (IPM) empowers network engineers to proactively troubleshoot network response times utilizing real-time and historical reports.
•The TrafficDirector RMON application, a remote monitoring tool, enables you to gather data, monitor activity on your network, and find potential problems.
•The VlanDirector switch management application provides an accurate picture of your VLANs.
CiscoView graphical management features provide dynamic status, statistics, and comprehensive configuration information for Cisco internetworking products (switches, routers, hubs, concentrators, and access servers). CiscoView aids network management by displaying a physical view of Cisco devices and color-coding device ports for at-a-glance port status, allowing you to quickly grasp essential information. Features include the following:
•Graphical displays of Cisco products from a central location, giving network managers a complete view of Cisco products. You do not need to physically check devices at remote sites.
•A continuously updated physical view of routers, hubs, switches, or access servers in a network, regardless of physical location.
•Updated real-time monitoring and tracking of key information relating to device performance, traffic, and usage, with metrics such as utilization percentage, frames sent and received, errors, and a variety of other device-specific indicators.
•The capability to modify configurations such as trap, IP route, VLAN, and bridge configurations.
Internetwork Performance Monitor
The Internetwork Performance Monitor (IPM) is a network management application that enables you to monitor the performance of multiprotocol networks. IPM measures the response time and availability of IP networks on a hop-by-hop (router-to-router) basis. It also measures response time between routers and the mainframe in Systems Network Architecture (SNA) networks.
Use IPM to perform the following tasks:
•Troubleshoot problems by checking the network latency between devices.
•Send Simple Network Management Protocol (SNMP) traps and SNA alerts when a user-configured threshold is exceeded, when a connection is lost and reestablished, or when a timeout occurs.
•Analyze potential problems before they occur by accumulating statistics, which are used for modeling future network topologies.
•Monitor response time between two network endpoints.
The IPM product is composed of three parts: the IPM server, the IPM client application, and the response time reporter (RTR) feature of the Cisco IOS software.
TrafficDirector RMON Application
The TrafficDirector advanced packet filters let you monitor all seven layers of network traffic. Using Cisco IOS embedded RMON agents and SwitchProbe standalone probes, managers can view enterprise-wide network traffic from the link, network, transport, or application layer. The TrafficDirector multilayer traffic summary provides a quick, high-level assessment of network loading and protocol distributions. Network managers can "zoom in" on a specific segment, ring, switch port, or trunk link and apply real-time analysis and diagnostic tools to view hosts, conversations, and packet captures.
TrafficDirector threshold monitoring enables you to implement a proactive management environment. Thresholds for critical MIB variables are set within the RMON agent. When these thresholds are exceeded, traps are sent to the appropriate management station to notify the network administrator of an impending problem.
VlanDirector Switch Management Application
The VlanDirector switch management application simplifies VLAN port assignment and offers other management capabilities for VLANs. VlanDirector offers the following features for network administrators:
•Accurate representation of the physical network for VLAN design and configuration verification
•Capability to obtain VLAN configuration information on a specific device or link interface
•Discrepancy reports on conflicting configurations
•Capability to use system-level VLANs to troubleshoot and identify individual device configurations that are in error
•Quick detection of changes in VLAN status of switch ports
•User authentication and write protection security
Third-Party Troubleshooting Tools
In many situations, third-party diagnostic tools can be more useful than commands that are integrated into the router. For example, enabling a processor-intensive debug command can be extremely detrimental in an environment experiencing excessively high traffic levels. Attaching a network analyzer to the suspect network is less intrusive and is more likely to yield useful information without interrupting the operation of the router. The following are some typical third-party troubleshooting tools used for troubleshooting internetworks:
•Volt-ohm meters, digital multimeters, and cable testers are useful for testing the physical connectivity of your cable plant.
•Time domain reflectometers (TDRs) and optical time domain reflectometers (OTDRs) are devices that assist in the location of cable breaks, impedance mismatches, and other physical cable plant problems.
•Breakout boxes, fox boxes, bit error rate testers (BERTs), and block error rate testers (BLERTs) are useful for troubleshooting problems in peripheral interfaces.
•Network monitors provide an accurate picture of network activity over a period of time by continuously tracking packets crossing a network.
•Network analyzers such as sniffers decode problems at all seven OSI layers. The problems can be identified automatically in real-time and categorized by how critical they are, providing a clear view of network activity.
Volt-Ohm Meters, Digital Multimeters, and Cable Testers
Volt-ohm meters and digital multimeters are at the lower end of the spectrum of cable-testing tools. These devices measure parameters such as AC and DC voltage, current, resistance, capacitance, and cable continuity. They are used to check physical connectivity.
Cable testers (scanners) also enable you to check physical connectivity. Cable testers are available for shielded twisted-pair (STP), unshielded twisted-pair (UTP), 10BASE-T, and coaxial and twinax cables. A given cable tester might be capable of performing any of the following functions:
•Test and report on cable conditions, including near-end crosstalk (NEXT), attenuation, and noise
•Use TDR and perform traffic monitoring and wire map functions
•Display MAC-layer information about LAN traffic, provide statistics such as network utilization and packet error rates, and perform limited protocol testing (for example, TCP/IP tests such as ping)
Similar testing equipment is available for fiber-optic cable. Because of the relatively high cost of this cable and its installation, fiber-optic cable should be tested both before installation (on-the-reel testing) and after installation. Continuity testing of the fiber requires either a visible light source or a reflectometer. Light sources capable of providing light at the three predominant wavelengths—850 nanometers (nm), 1300 nm, and 1550 nm—are used with power meters that can measure the same wavelengths and test attenuation and return loss in the fiber.
TDRs and OTDRs
At the top end of the cable testing spectrum are time domain reflectometers (TDRs). These devices can quickly locate open and short circuits, crimps, kinks, sharp bends, impedance mismatches, and other defects in metallic cables.
A TDR works by bouncing a signal off the end of the cable. Opens, shorts, and other problems reflect the signal back at different amplitudes, depending on the problem. A TDR measures how much time it takes for the signal to reflect and calculates the distance to a fault in the cable. TDRs can be used to measure the length of a cable, and some TDRs can also calculate the propagation rate based on a configured cable length.
Fiber-optic measurement is performed by an optical TDR (OTDR). OTDRs can accurately measure the length of the fiber, locate cable breaks, measure the fiber attenuation, and measure splice or connector losses. An OTDR can be used to take the signature of a particular installation, noting attenuation and splice losses. This baseline measurement can then be compared with future signatures when a problem in the system is suspected.
Breakout Boxes, Fox Boxes, BERTs and BLERTs
Breakout boxes, fox boxes, and bit/block error rate testers (BERTs/BLERTs) are digital interface testing tools used to measure the digital signals present at PCs, printers, modems, the channel service unit/data service unit (CSU/DSU), and other peripheral interfaces. These devices can monitor data line conditions, analyze and trap data, and diagnose problems common to data communication systems. Traffic from data terminal equipment (DTE) through data communications equipment (DCE) can be examined to help isolate problems, identify bit patterns, and ensure that the proper cabling has been installed. These devices cannot test media signals such as Ethernet, Token Ring, or FDDI.
Network monitors continuously track packets crossing a network, providing an accurate picture of network activity at any moment, or a historical record of network activity over a period of time. They do not decode the contents of frames. Monitors are useful for baselining, in which the activity on a network is sampled over a period of time to establish a normal performance profile.
Monitors collect information such as packet sizes, the number of packets, error packets, overall usage of a connection, the number of hosts and their MAC addresses, and details about communications between hosts and other devices. This data can be used to create profiles of LAN traffic and to assist in locating traffic overloads, planning for network expansion, detecting intruders, establishing baseline performance, and distributing traffic more efficiently.
A network analyzer (also called a protocol analyzer or "sniffer") decodes the various protocol layers in a recorded frame and presents the frames as readable abbreviations or summaries. The analyzer indicates which layer is involved (physical, data link, and so forth) and what function each byte or byte content serves.
Most network analyzers can perform many of the following functions:
•Filter traffic that meets certain criteria so that, for example, all traffic to and from a particular device can be captured
•Time-stamp captured data
•Present protocol layers in an easily readable form
•Generate frames and send them onto the network
•Incorporate an "expert" system in which the analyzer uses a set of rules, combined with information about the network configuration and operation, to diagnose and solve, or offer potential solutions for network problems
Other Troubleshooting Resources
Refer to the following resources to assist in troubleshooting:
•Internetworking Troubleshooting Handbook
•TAC Troubleshooting Tools on Cisco.com
Internetworking Troubleshooting Handbook
The Internetworking Troubleshooting Handbook is a good resource for general network troubleshooting. It is available online:
TAC Troubleshooting Tools on Cisco.com
A variety of troubleshooting tools can be found at the following website:
Note You need to register to use these tools on Cisco.com.
This site has links to the following tools:
•2600/3600/3700 Memory Calculator—Compute the memory required for Cisco 2600 series, Cisco 3600 series, and Cisco 3700 series routers.
•BPX/IGX Firmware Compatibility Tool—Evaluates BPX and IGX firmware versions for switch software compatibility using the dspcds screen output.
•Cisco-centric Open Source Initiative (COSI)—An open source exchange community where Cisco-centric developers and customers publish, discuss, and release their open source tools, scripts, and utilities.
•Cisco Feature Navigator II—A web-based application that allows you to quickly find the right Cisco IOS software release for the features you want to run on your network.
•Command Lookup Tool—Look up a detailed description or configuration guidelines for a particular Cisco IOS, Cisco Catalyst, or PIX command. This tool is for Cisco IOS software releases 12.0, 12.1, and 12.2 only.
•Custom Document Generator for Cisco IOS—Generate customized command reference documents.
•Error Message Decoder—Look up explanations for console error messages listed in the Cisco Software System Messages guide.
•IOS Upgrade Planner—Browse for your preferred software.
•IP Subnet Calculator—Plan your subnetting and addressing strategy online.
•Networking Professionals Connection—Discussion forums, Tech Talks, and Ask the Expert forums are available in which you can share questions, suggestions, and information about networking solutions, products, and technologies.
•Output Interpreter—Receive instant troubleshooting analysis and course of action for your router, switch, or PIX device using collected show command output.
•RIF Decoder Tool—Use the RIF decoder to automatically interpret and decode a hex string.
•SNMP Object Navigator—Translate SNMP object identifiers (OIDs) into object names, search object names and descriptions, browse the OID tree, and download MIB files.
•Software Advisor—Choose software for your network device.
•Software Bug Toolkit—Search for software bugs based on version and feature sets or bug ID.
•Software Search Engine—Search for software images and image metadata.
•Stack Decoder for Cisco IOS—This link takes you to the Cisco Output Interpreter tool, which now offers all the functionality formerly available in the Stack Decoder.
•TAC Case Collection (Troubleshooting Assistant)—Interactively diagnose common problems involving hardware, configuration, and performance issues with solutions provided by TAC engineers.
•Voice Codec Bandwidth Calculator—Use the Voice Codec Bandwidth Calculator to determine the bandwidth used by different codecs with various voice protocols over different media.
•Wireless Troubleshooting Center—Interactive support for questions related to Cisco Aironet wireless LAN products.
CCDE, CCENT, CCSI, Cisco Eos, Cisco HealthPresence, Cisco IronPort, the Cisco logo, Cisco Nurse Connect, Cisco Pulse, Cisco SensorBase, Cisco StackPower, Cisco StadiumVision, Cisco TelePresence, Cisco Unified Computing System, Cisco WebEx, DCE, Flip Channels, Flip for Good, Flip Mino, Flipshare (Design), Flip Ultra, Flip Video, Flip Video (Design), Instant Broadband, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn, Cisco Capital, Cisco Capital (Design), Cisco:Financed (Stylized), Cisco Store, Flip Gift Card, and One Million Acts of Green are service marks; and Access Registrar, Aironet, AllTouch, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Lumin, Cisco Nexus, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, Continuum, EtherFast, EtherSwitch, Event Center, Explorer, Follow Me Browsing, GainMaker, iLYNX, IOS, iPhone, IronPort, the IronPort logo, Laser Link, LightStream, Linksys, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, PCNow, PIX, PowerKEY, PowerPanels, PowerTV, PowerTV (Design), PowerVu, Prisma, ProConnect, ROSA, SenderBase, SMARTnet, Spectrum Expert, StackWise, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.
All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0910R)
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
© 2007-2010 Cisco Systems, Inc. All rights reserved.