Troubleshoot Hardware for Catalyst 4000 Series Switches

Available Languages

Download Options

  • PDF     (388.1 KB)
    View with Adobe Reader on a variety of devices
Updated:September 30, 2009
Document ID:18935

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

This document provides troubleshooting procedures on how to diagnose hardware problems on Catalyst 4000 family switches. The Catalyst 4000 family includes the 4003 and 4006 modular chassis and the 2948G, 2980G, and 4912G fixed models. The naming conventions for the Catalyst 4000 and Catalyst 2900 can be very confusing. Refer to Understanding Catalyst 2900 and Catalyst 4000 Naming Conventions for more information on how to help clarify these issues.

The goal is to help Cisco customers identify and fix some basic hardware issues, or to perform more extensive troubleshooting before you contact Cisco Technical Support. An orderly troubleshooting process with the collection of specific diagnostics ensures that information necessary to the resolution of the problem is not lost. If you refine the scope of the problem, this saves valuable time in the search for a solution.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

Components Used

This document is not restricted to specific software and hardware versions.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Preparation for Troubleshooting Hardware on Catalyst Switches

Many hardware problems encountered during field installations or during normal operation can be prevented by a thorough product overview ahead of time. For those customers not already familiar with general system and power requirements, proper installation procedure, switch management and software considerations for these switches, Cisco recommends that you read documents in Cisco Catalyst 4000 Series Switches Troubleshooting TechNotes.

This document covers this important information:

  • Which supervisor is supported in which chassis?

  • How do I back up my configuration?

  • Which software version is General Deployment (GD) for the Catalyst 4000 Family?

This document assumes familiarity with the Catalyst 4000 Command Reference. You should also have a prior understanding of switching fundamentals, or have read How LAN Switches Work. Additional online documentation is referenced throughout this document in order to assist in troubleshooting.

Online Troubleshooting Tools

Cisco has a variety of troubleshooting tools and resources in order to help you interpret switch output, determine hardware software compatibility, track bugs, and search field notices. These tools and resources are referenced throughout this document:

Catalyst 4000 Family Troubleshooting Procedures

This section discusses troubleshooting procedures, symptoms, show commands, and diagnostics for the Catalyst 4000 family. This section assumes you have read the companion guide to this document, as described in the Introduction of this document, and that you understand your switch and its capabilities.

Note: If the switch is connected to the network, do not reset or reseat modules as a first troubleshooting step! In addition to the downtime that users experience, the internal buffer, which logs system messages are erased and potentially useful information in regards to hardware or software errors are lost. If the switch is offline, you have more freedom to monitor LED status, pull cables, reseat modules, or reset the switch as necessary. Troubleshooting LED status is discussed in more detail later in this document.

Hidden Commands

Some commands presented in this document are known as hidden, which means that they cannot be parsed with a "?", and you cannot Tab in order to complete. When a hidden command is suggested in this document, simply gather the output and send it to the TAC engineer, if you open a case. It is possible that this output is useful in solving your case. These commands are undocumented, and therefore the TAC engineer is not required to explain the output to the customer.

If you want to troubleshoot any problem, this requires a method or set of procedures which, if followed correctly, produces a solution. Begin by understanding general problem solving for LAN networks.

General Problem Solving Model

If you want to troubleshoot any problem, this requires a method or set of procedures which, if followed correctly, produces a solution. Begin by understanding general problem solving for LAN networks. Hardware failures in LAN networks are characterized by certain symptoms. These symptoms can be general such as the inability to Telnet between switches, more specific such as link flapping, or perhaps the switch is resetting itself. Each symptom can be traced to one or more causes if you use specific troubleshooting techniques. A systematic approach works best. Define the specific symptoms, identify all potential problems that could be causing the symptoms, and then eliminate each potential problem, from most likely to least likely, until the symptoms disappear.

General Problem Solving Flow Chart

This diagram outlines the steps that detail the problem-solving process:

121-b.gif

Complete these steps:

  1. Define the problem.

    It is important to first identify the problem being experienced. This allows you to identify what kinds of causes can result in these symptoms. In order to help determine the problem, ask yourself these questions:

    • What is the primary symptom?

    • Is the problem specific to this switch or does it affect other switches on the network as well?

    • Is this a problem with one or more ports on a specific module? What type of ports: 10/100, Multimode Fiber (MMF), Singlemode Fiber (SMF), GigabitEthernet, and so forth?

    • What device is connected to the switch ports that experiences the problem?

    • When did this problem first occur and has it occurred more than once?

    • What happened at the time the problem was first noticed? Is there anything unique about traffic conditions at that time of day? For example, was this a peak time for traffic?

    • Did you run any particular commands at the time or make any configuration changes?

  2. Gather the facts.

    Gather diagnostics and show commands output from the switch to isolate the scope of the problem. If physical access to the equipment is possible, locate and list any modules with red or yellow LEDs, disconnected cables, or loose connections.

  3. Consider the possible causes.

    Consider possible problems based on the information you gathered. With certain data, you are able, for example, to eliminate hardware as a problem, so that you can focus on software problems. At every opportunity, try to narrow the number of potential problems so that you can create an effective plan of action.

  4. Create and implement an action plan.

    Create an action plan based on the potential problems. Focus on only one potential problem at a time. If you alter more than one variable simultaneously, you can solve the problem, but the identification of the specific change that eliminated the symptom becomes far more difficult and does not help you solve the same problem if it occurs in the future.

  5. Observe the results.

    Be sure to gather and analyze the results each time a variable is changed to determine if the problem has been fixed.

  6. Repeat the process.

    Repeat testing for possible causes until the problem is resolved.

Common Problems

As described in the Problem Solving Model, the first step in resolving a problem is to identify the symptom. Refer to Catalyst Troubleshooting Tips for more information on some common problems associated with all Catalyst switches that can be resolved.

Most hardware problems with LAN networks fall into these categories and each category has various symptoms related to it:

  • Connectivity Problems

  • System/Supervisor/Module Problems

  • Supervisor Crashes

Connectivity Problems

These problems can occur when communication with the supervisor, module, or hosts connected to the module is intermittent or has been lost.

System/Supervisor/Module Problems

These problems can occur when system status LEDs indicate a problem, the supervisor or modules are not recognized or show faulty, or when users experience poor performance.

Supervisor Crashes

These problems can occur when the switch has reset, continually resets, or is down completely.

Symptom Description

This section discusses symptoms, troubleshooting procedures, and commands for Catalyst 4000 family switches. This section assumes you are able to identify your switch chassis, supervisor engine, modules, and feature cards, and that you understand the system specifications, cabling, power, and software requirements as described for Cisco Catalyst 4500 Series Switches Install and Upgrade Guides.

If you have not determined what your primary symptom is, see the General Problem Solving Model section of this document and apply the steps to your problem.

Connectivity Problems and Steps to Resolve Them

This section covers common connectivity issues that the customer can encounter with the Catalyst 4000.

These commands are supported by the Output Interpreter tool for CatOS and can be used to assist in troubleshooting switch port problems:

  • show version

  • show module

  • show system

  • show port

  • show mac

  • show counters

  • show cdp neighbors detail

If you have the output of the supported commands from your Cisco device, you can use the Output Interpreter (registered customers only) to display potential issues and fixes. In order to use Output Interpreter (registered customers only) , you must be a registered user, be logged in, and have JavaScript enabled.

Can not console/Telnet into the supervisor

Both of these problems are covered in the Catalyst Troubleshooting Tips document that is mentioned earlier.

Receiving the "Failed to allocate session block" error message

If you receive the Failed to allocate session block error message while you access the switch on the Telnet, the problem occurs because the switch cannot allocate the required memory for the Telnet application. The available free memory is low because of some process that uses more memory or because of a memory leakage in the switch.

In order to avoid the error, issue the show proc mem command and verify the process that uses more memory in the switch. In order to resolve the problem, either add more memory to the system or disable some features in order to free some of the existing memory.

If there is memory leak in the switch, reset the switch in order to release all the process in the memory. If the error message still appears even after you reboot, upgrade the software version of the switch.

Can not connect to a remote host, router, or another switch

Complete these steps:

  1. Verify that the port LED status is green. If the link LED is solid orange, it has been disabled by the software. If it is blinking orange after supervisor bootup and module initialization, this is a hardware failure. If there is no link LED, check and swap the cables. Verify operation of the end device and NIC.

    Refer to Troubleshooting Cisco Catalyst Switches to NIC Compatibility Issues for more information on NIC troubleshooting.

  2. What type of media is involved? Fiber? Gigabit Interface Converter (GBIC)? Gigabit Ethernet? 10/100 BaseTX? If this a physical layer issue, refer to the Physical Layer Troubleshooting section of Troubleshooting Switch Port Problems for more information.

  3. Issue the show port <mod/port> command in order to verify that the status is connected, which means that the port is operational. If any other status is displayed, see the Port status shows not connected, faulty, disabled, inactive, or errdisable section for troubleshooting steps.

    If the end device is a Cisco router or switch, and Cisco Discovery Protocol (CDP) is enabled, issue the show cdp neighbor detail command in order to identify the device, remote interface type, and remote IP address.

    Note: A status of connected does not mean that the ports are free of errors. If there are errors on the ports, proceed to the Seeing errors on the ports section of this document.

  4. Swap the cables. Move the cable to a different port. Eliminate patch panels. Patch panels are a common source of connectivity failures, so attempt to connect directly to the end device. Verify the operation of the end device.

  5. Capture the output of the show config , show module , and show test 0 commands.

    1. Issue the show module command in order to verify that the status is ok for that module and not disabled or faulty.

      • If the status is disabled, issue the set module enable <mod> command.

      • If the status is faulty, establish a console connection to capture bootup Power On Self Test (POST) diagnostics and any system error messages. Issue the reset <mod> command in order to reset the module. Issue the show test 0 command in order to determine if this module passed all of it's diagnostic tests on bootup.

      • Remove the module and inspect for bent pins. Reseat the module, firmly press down the ejector levers, and tighten the captive installation screws. If the output of the show module command status is still faulty, try the module in another slot. Slot 2 accepts line cards or a supervisor engine. If necessary, power off/on the switch. If the status is still faulty, the module has failed.

    2. Issue the show test 0 command in order to verify that the port has passed its last diagnostic test on bootup. If F is indicted for that port, proceed as in step a.

  6. Verify whether this device is on the same or a different VLAN. Remember that this is a Layer 2 (L2) device and a router is required to route between VLANs.

  7. If you connect to another switch, ask yourself these questions:

    • What type of port is this? A trunk port?

    • If it is a trunk port, what trunk encapsulations does it support?

    • Is the port capable of EtherChannel?

    Issue the show port capabilities command for a quick look at port capabilities. Refer to LAN Technical Tips for more information on how to troubleshoot issues with trunking or EtherChannel.

Port status shows not connected, faulty, disabled, inactive, or errdisable

Possible Port Status

Status Description and Work Around 
connected
 Port is operational and connected to end device. A status of connected does not mean the ports are error-free. If there are errors on the ports, proceed to the Seeing errors on the ports section of this document. 
notconnect
 Nothing is connected to the port. Check or swap cables. Verify the operation of the end device. 
faulty
 Possible hardware failure. Issue the show test command in order to verify. If F displays for a port, proceed as in step 5 of the Can not connect to a remote host on the switch section of this document. 
disabled
 Manually disabled. Issue the set port enable <mod/port> command in order to enable the port. If the port status does not change to enable, issue the show module command in order to determine if the module is disabled. 
inactive
 Port belongs to a VLAN that does not exist. Issue the set vlan <vlan> command in order to add a VLAN. 
errdisable
 Port had been shut down due to errors. Refer to the Recovering From errDisable Port State on the CatOS Platforms document for more information.

Seeing errors on the ports

Complaints of poor performance by users can sometimes translate to errors on switch ports. Output from the port error counters command help you troubleshoot connectivity problems.

  1. Verify port status and troubleshoot accordingly. Refer to the Port status shows not connected, faulty, disabled, inactive, or errdisable section of this document.

  2. Capture the output of the show port <mod/port> , show mac <mod/port> , and show counters <mod/port> commands.

    These are common causes for data link errors on ports:

    The show port <mod/port> command can show Late-Coll, Align-Err, FCS-Err, Xmit-Err, and Rcv-Err errors. Refer to the the Show Port for CatOS and Show Interfaces for Cisco IOS section of Troubleshooting Switch Port Problems for more informaiton on these errors and possible causes.

    The show mac <mod/port> command shows the number of unicast, multicast, and broadcast frames transmitted. Issue this command in order to verify if frames are received and transmitted.

    In-Discards show frames that do not need to be switched. This is normal if the port was connected to a hub and two devices exchanged data. Lrn-Discards indicate that Content Addressable Memory (CAM) entries are being discarded. In-Lost counter displays the sum of all error packets received on the port. The Out-Lost counter indicates egress port buffer overflows. Refer to the Show Mac for CatOS and Show Interfaces Counters for Cisco IOS section of Troubleshooting Switch Port Problems for more information on these errors and possible causes.

    The show counters <mod/port> command is useful in particular for troubleshooting port problems.

    For example, this counter results if you issue the command: 

    5 badTxCRC = 0

    If badTxCRC were incrementing, this can be bad hardware corrupting packets. Capture the output of the show counters <mod/port> command and open a case with the Cisco Technical Support.

  3. Issue the clear counters command in order to reset the output of the show port <mod/port>, show mac <mod/port>, and show counters <mod/port> commands. View the command outputs several times in order to see if errors are incrementing.

    If you have not been able to track down any reason for intermittent connectivity loss on the switch in the previous steps mentioned, capture the output of the show nvramenv 1 command, as well as the other commands in the previous steps, and open a case with the Cisco Technical Support.

  4. Refer to these documents for more information on how to troubleshoot the other causes of port errors:

Experiencing poor performance

Poor performance is often perceived to be a hardware problem, when in fact it can be attributed most often to connectivity problems. See the Seeing errors on the ports section for troubleshooting steps.

Getting continuous %PAGP-5 left/joined bridge messages

Complete these steps:

  1. Capture the show port <mod/port>, show mac <mod/port>, and show spantree summary command output.

    System messages similar to these messages are informational, although if the errors continue to repeat, the link can be flapping. 

    2002 Jan 19 14:59:05 %PAGP-5-PORTFROMSTP:Port 2/11 left bridge port 2/11
2002 Jan 19 14:59:23 %PAGP-5-PORTTOSTP:Port 2/11 joined bridge port 2/11

  2. If these messages occur repeatedly on certain ports, refer to these document for possible causes:

  3. If you also see errors on the port in the show port <mod/port> and show mac<mod/port> command output, see the Seeing errors on the ports section for troubleshooting steps.

  4. Issue the show spantree summary command in order to verify how many ports are in each VLAN, if any ports on the switch are blocking, and which VLANs are being blocked. Since Spanning-Tree Protocol (STP) loops can cause link flaps or actually bring down a switch or network, with the appearance of a hardware failure, this is vital information to capture, whether troubleshooting hardware or software. Refer to LAN Technical Tips for more information on how to troubleshoot STP.

Can not autonegotiate or speed/duplex mismatch

Complete these steps:

  1. Make sure you have speed and duplex configured identically on both sides of the link. Catalyst 4000 switchports are set to auto by default. When both sides of a 100 BaseTX link autonegotiate correctly, the show port <mod/port> command output is as follows: 

    Duplex   Speed
-------   -------
a-full    a-100

    Hardcode both sides. Remember when hardcoding the port, the port speed must be set first and then the duplex setting must be set. Issue the show port <mod/port> command. The switch output is as follows: 

    Duplex   Speed
-------   -------
full       100

    Note: Even though the switch has been hard coded, the connecting device must still be hardcoded to eliminate problems.

  2. If there is an autonegotiation problem caused by a speed/duplex mismatch or NIC incompatibility, errors show up on the ports. Refer to these documents for more information:

System/Supervisor/Module Problems and Steps to Resolve Them

System, supervisor, and module problems occur when either system status LEDs indicate a problem, the supervisor or modules are not recognized or show faulty, or when users are experiencing poor performance.

The following commands are supported by the Output Interpreter and can be used to assist in troubleshooting system, supervisor, and module problems: show version, show module, or show system.

If you have the output of the supported commands from your Cisco device, you can use the Output Interpreter (registered customers only) in order to display potential issues and fixes. In order to use Output Interpreter (registered customers only) , you must be a registered user, be logged in, and have JavaScript enabled.

Having problems upgrading software

Complete these steps:

  1. Most customer problems that have to do with software upgrades are the result of not understanding the copy tftp procedure, the boot process, or the Flash system for the supervisor.

    Refer to Working with System Software Images for more information, specifically, on the copy tftp procedure for your supervisor.

    Refer to Using the Flash File System for more information on the Flash file system for your supervisor.

    Refer to Recovering Catalyst Switches Running CatOS from Booting Failures for more information on rommon recovery information.

    Capture the show version, show flash, or dir bootflash command output, which depends on the type of supervisor you have. Verify that you have enough DRAM and Flash for the image to which you attempt to upgrade, and then perform the copy tftp procedure.

  2. Set the boot environment variable and the config-register. Refer to Modifying the Switch Boot Configuration for more information on these settings.

    Cat4000-c> (enable) set boot ?
auto-config                Set auto config file
config-register            Set configuration register
sync                       Set sync parameters
system                     Set BOOT environment variable

    Cisco recommends that you set the boot environment variable and config-register in this way:

    1. Verify the image that you want to boot, currently installed in Flash. Issue the dir bootflash: command. 

      Cat4000-c> (enable) dir bootflash:
-#- -length- -----date/time------ name
1  4106492 Aug 17 2001 16:22:52 cat4000.6-3-1.bin
2  3554592 Nov 28 2001 10:38:33 cat4000.5-5-11.bin
3  4199168 Dec 07 2001 10:30:01 cat4000-k9.6-3-3.bin
4  3651336 DEC 11 2001 12:26:20 cat4000.5-5-8.bin

216540 bytes available (15512100 bytes used)

    2. Set the boot environment variable for the image in Flash from which you want to boot. 

      Cat4000-c> (enable) set boot system flash  bootflash:cat4000.6-3-1.bin
BOOT variable = bootflash:cat4000.6-3-1.bin,1;

    3. Set the config-register to boot from Flash.

      Cat4000-c> (enable) set boot config-register 0x2102
Configuration register is 0x2102
ignore-config: disabled
auto-config: non-recurring
console baud: 9600
boot: image specified by the boot system commands

  3. If you end in rommon or boot mode during the upgrade, refer to Recovering Catalyst Switches Running CatOS from Booting Failures for more information.

  4. Use the Bug Toolkit to track down bugs, or refer to Release Notes for Catalyst 4000 Family Software Release 5.x for Caveats.

Supervisor is not online or is stuck in boot or rommon

The most common causes for a Catalyst 4000 family supervisor not to be recognized is when it is stuck in boot or rommon mode due to a missing or corrupt image. In these modes, you are not able to Telnet to the supervisor and must have a console session open.

  1. If the supervisor is stuck in either boot or rommon mode, complete the troubleshooting steps in Recovering Catalyst Switches Running CatOS from Booting Failures.

  2. If the supervisor is not in either boot or rommon mode but is still not online, complete the troubleshooting steps for the Supervisor Engine in the System component LEDs are orange/red section of this document.

System component LEDs are orange/red or supervisor not online

Complete these steps:

  1. If you observe orange or red LEDs on startup, wait until the system boots up completely before concluding that there is a problem. The system status LED on the supervisor will stay orange until bootup is complete, then turn green if bootup is successful. One cause of an orange sys-status LED is a fan failure.

    Next, the supervisor initializes the switching modules, which operate differently depending on the module; some flash on and off, and others stay orange until initialization is complete. At this point, the link (port) LEDs turns off altogether until a signal is detected.

  2. Understand the Catalyst 4000 family components and what the LEDs tell you. As a starting place, refer to Troubleshooting the Installation for more information:

    Look at the front panel LEDs for your supervisor. Refer to these documents for more information:

    Look at the front panel LEDs for your switching module. Refer to Catalyst 4500 E-Series Module Installation Note for more information:

  3. Capture the show version, show system, show module, and show test 0 command output.

    Power supply—includes the power supplies and power supply fans. The PS1, PS2, and PS3, for the Catalyst 4006, status LEDs should be green. If one or both are red, this can indicate a power supply failure.

    1. When you issue the show system command, determine if PS1 or PS2 status is faulty.

      Note: The Catalyst 4006 requires two power supplies installed to operate the switch and the third is for redundancy. Refer to Module Overview for more information.

    2. Inspect the power supplies. Make sure there is power applied to both units. If a redundant power supply is installed but has no power, the show system command output shows that the power supply status and sys-status is faulty.

    3. Reseat the power supply. Try a different circuit or swap power cords. If the status is still red, or the show system command output shows faulty, this is a power supply failure. Refer to Removal and Replacement Procedures for more information.

      Fan assembly—Whenever system power is on, the system fan assembly should operate. You should be able to hear the fan assembly to determine if it operates.

    1. Inspect the fan assembly and power supplies to verify if power is being applied to the system.

    2. Issue the show system command to determine if the fan-status is faulty.

    3. Reseat the fan assembly and tighten the captive installation screws. If necessary, reset the switch. If the show system command output still shows faulty, this is a fan failure. Refer to Removal and Replacement Procedures for more information.

      Supervisor engine—The supervisor engine contains the system operating software. Check the supervisor engine if you have trouble with the system software. The status LED on the supervisor engine indicates whether the supervisor engine has passed all diagnostic tests. Have a console session open and determine whether the supervisor is in boot or rommon mode. If this is the case, see the Supervisor is not online or stuck in rommon section for troubleshooting steps.

    1. Issue the show system command in order to determine if the sys-status is faulty. Issue the show test 0 command in order to determine if the supervisor has passed all diagnostic tests as of the last bootup of the switch. Note any F for fail results.

    2. Inspect the fan assembly and power supplies for any problems.

    3. Have a console session open and capture bootup POST diagnostics and system error messages. Reset the switch and issue the show test 0 command in order to determine if the diagnostic test on bootup has been passed.

    4. Remove the supervisor and inspect for bent pins. Reseat the supervisor, firmly press down the ejector levers, and tighten the captive installation screws. Wait for the supervisor to initialize. If the show system command sys-status is still faulty, the supervisor has failed.

      Switching modules—The status LEDs on each switching module indicate whether the switching module has been initialized correctly. The supervisor engine must be operating properly before the switching module initializes. If a switching module is improperly installed in the switch, it does not function.

    1. If a link (port) LED is solid orange or is blinking orange after the supervisor bootup and module initialization, see the Can not connect to a remote host, router, or another switch section.

    2. Capture the show version and show module command output. Determine if the software version you are running supports this module. Refer to the Module Overview section of Catalyst 4500 E-Series Module Installation Note for more information.

    3. Determine if the status is disable. This indicates that the module was administratively disabled. The status LED is orange in this case. Issue the set module enable <mod> command.

    4. View the output of the show module command in order to determine if status is faulty for that module. View the output of the show test 0 command in order to determine if this module passed all its diagnostic tests as of the last bootup of the switch. Note any F for fail results.

    5. Have a console session open and capture bootup POST diagnostics and any system error messages. Issue the reset <mod> command in order to reset the module. Issue the show test 0 command in order to determine if this module has passed all of its diagnostic tests on bootup. Note any F for fail results.

    6. Remove the module and inspect for bent pins. Reseat the module, firmly press down the ejector levers, and tighten the captive installation screws. If the show module status is still faulty, try the module in another slot. If necessary, power off/on the switch. If status is still faulty, the module has failed.

Switching module is not recognized

The most common cause for a switching module or a line card to not be recognized is due to the wrong version of software.

  1. Determine that this is a problem with just one module and not all modules. If all modules are affected, complete the steps in the System component LEDs are orange/red or supervisor not online section. Capture the output the show version , show module , and show test 0 commands.

  2. Issue the show version command in order to check the model number of the module you have problems with and the software version you use. Determine the total DRAM and total Flash. Refer to the Module Overview section of Catalyst 4500 E-Series Module Installation Note in order to determine if the hardware is compatible with the software.

    If an upgrade is required, always check the release notes first. Refer to the to Catalyst 4500 Family Release Notes and choose the version to which you need to upgrade. For example, choose the Release Notes for Catalyst 4000 Family Software Release 5.x and search on any information in regards to your hardware.

    Refer to Managing Software Images and Working with Configuration Files on Catalyst Switches for more information.

  3. If the supervisor is not stuck in boot or rommon and you have determined that the module is supported by the current version of software, complete the steps for troubleshooting the Switching Module in the System component LEDs are orange/red or supervisor not online section.

Module status is showing faulty or not ok

Complete these steps:

  1. Capture the show module and show test 0 command output.

  2. For any status other than ok in the output of these two commands, complete the troubleshooting steps for the Switching Module in the System component LEDs are orange/red or supervisor not online section.

Experiencing poor performance

Poor performance is often perceived to be a hardware issue, but this is usually not the case. When customers describe to Cisco Technical Support that users on a particular switch experience slow performance, this often turns out to be related to connectivity problems, software misconfiguration, or problems elsewhere on the network.

  1. Identify whether performance issues occur for users connected to all switching modules, one module in particular, or just users on one or more ports. Capture the show module and show test 0 command output. Make sure that the supervisor and modules have an ok status. If there is a faulty status, complete the troubleshooting steps for the Switching Module in the System component LEDs are orange/red or supervisor not online section.

  2. Capture the show port <mod/port> , show Mac<mod/port> , and show counters <mod/port> command output. If you see incrementing errors on port counters, troubleshoot this performance issue as a connectivity issue. See the Seeing errors on the ports section for troubleshooting steps.

  3. Capture the show config and show logging buffer 1023 command output. The show config command shows only the non-default configuration changes. Ideally, every time you make a change, you should have backed-up the configuration to use as a comparison. Issue the show config command in order to possibly associate a configuration change with the behavior you experience.

    If you see any system messages other than informational messages that can indicate a hardware or some other problem, issue the show logging buffer 1023 command in order to capture these messages. This command displays the last 1023 system messages with timestamps, by default. Also, refer to Messages and Recovery Procedures well as Common CatOS Error Messages on Catalyst 4000 Series Switches in order to see if you can rule out any harmless system messages from those that can indicate a problem.

  4. Many performance related problems are related to network traffic conditions. Capture the show system command output in order to see if this is a network traffic problem.

    The show system command can be used to check the current backplane utilization, which is typically less than ten percent. If you believe that you are having performance related issues on a particular switch, look at the Peak field, which is the peak backplane utilization on the switch since it was last booted, and note the timestamp indicated by Peak-Time. Keep in mind that spikes in traffic percentage on the backplane can be a STP loop or broadcast storm. Refer to Spanning Tree Protocol Problems and Related Design Considerations for more information.

  5. Capture the show proc cpu command output. This command helps identify a process that can cause high CPU utilization on the supervisor. This is an excerpt of show proc cpu command output: 

    Cat4000-c> (enable) show proc cpu
CPU utilization for five seconds:  11.62%                      
                      one minute:  12.00%                    
                    five minutes:  12.00%
PID Runtime(ms) Invoked    uSecs    5Sec    1Min    5Min    TTY Process
--- ----------- ---------- -------- ------- ------- ------- --- ---------------
1   20176816    0          0         88.38%  88.00%  88.00% -2  Kernel and Idle

    When you view the output of this command, remember that the CPU utilization is the first thing shown. Do not confuse the Kernel and Idle amount as CPU utilization. The Kernel and Idle is the percentage of CPU that was idle for that time frame. Therefore, in the past five minutes, only 11.62 percent of the CPU was utilized, which is within typical boundaries.

    Refer to Understanding CPU Utilization on Catalyst 4000, 2948G, 2980G, and 4912G Switches for more information and a complete understanding of how the CPU is utilized on Catalyst 4000, 2948G, 2980G, and 4912G switches.

    Complete these steps in order to get a baseline of your switch and help identify which process can cause a problem:

    • Issue the show proc cpu command during a time of normal activity for your network. Save the results.

    • Run this command again if you experience any performance related issues.

    • Compare the two outputs. Is there a process you can identify that is unusually high in comparison?

    • Run the command multiple times. Is there a significant increase or decrease in CPU utilization or spikes? Or, does CPU utilization remain consistently high?

    The answer is most likely not a hardware problem, but points elsewhere.

  6. One performance related issue that results from misconfiguration is when the inband channel, which is used for any control traffic terminating on the switch such as ping, Telnet, VLAN Trunk Protocol (VTP), STP, CDP, and so forth, is not put in a separate VLAN from user data.

    It is always recommended to keep the management or sc0 interface of the switch in a separate VLAN from the user data. Otherwise, any broadcast or multicast storm can flood the inband channel to the Network Management Processor (NMP), which needs to be free to handle the protocols just mentioned.

    If you have not been able to track down any reason for performance issues on the switch in the previous steps mentioned, capture the output of these commands, as well as the other commands in the previous steps, and open a case with the Cisco Technical Support:

    • show nvramenv 1 (hidden)

    • show interposition 1 (hidden)

    These are hidden commands, which means they cannot be parsed with a "?" and you cannot Tab to complete. Type the command out in its entirety. It is possible that this output is not useful in the resolution your case. These commands are undocumented, and therefore the TAC engineer is not required to explain the output to the customer.

  7. Although fairly rare, memory leaks do occur and can cause what seem naturally to be poor performance and other symptoms. If you have not been able to track down any reason for performance issues on the switch in the previous steps mentioned, capture the output of the show mbuf total (hidden) command, as well as the other commands in the previous steps, and open a case with the Cisco Technical Support.

    There are two things to consider when you look at the output of this command in order to help determine if you have a memory leak issue:

    • Look at the output and if the free mbufs or clusters values decrease but never increase, this can indicate a possible memory leak.

    • Look at the output, and if the lowest free memory has ever approached zero or was at zero, this indicates the switch either runs low on or has ran out of memory.

    Both of these issues indicate a memory issue that obviously affects the protocols/processes that require this memory. 

    Cat4000-c> (enable) show mbuf total        
mbufs                   9280    clusters        3660        
free mbufs              9256    clfree          3659        
lowest free mbufs       9235    lowest clfree   3638

    These are hidden commands, which means they cannot be parsed with a "?" and you cannot Tab to complete. Type the command out in its entirety. It is possible that this output is not useful in the resolution your case. These commands are undocumented, and therefore the TAC engineer is not required to explain the output to the customer.

Getting system error messages

As mentioned in the Introduction of this document, Cisco has a suite of online diagnostic tools to help you determine hw/sw compatibility, interpret output, and decode errors.

  1. System messages have timestamps by default, which can help in isolating a timeframe for your problem. by Issue the show time command in order to make sure your system clock is set correctly. Also, verify that your connecting devices are set so that the logs match.

  2. Capture the output of any system messages with the show logging buffer 1023 command. Many system messages are informational in nature while others can indicate a problem. Refer to these documents for more information:

Supervisor Crashes and Steps to Resolve Them

Supervisor crashes occur when the switch has reset, continually resets, or is down completely.

These commands are supported by the Output Interpreter and can be used to assist in troubleshooting supervisor crashes: show version or show system.

If you have the output of the supported commands from your Cisco device, you can use Output Interpreter (registered customers only) in order to display potential issues and fixes. In order to use Output Interpreter (registered customers only) , you must be a registered user, be logged in, and have JavaScript enabled.

Getting system error messages

System error messages can be useful if you experience a switch reset. See the Getting system error messages section for more information.

Switch has reset or is continually resetting

If the switch has reset or crashed due to a reason related to hardware or software, it is important to capture the output of certain show commands as quickly as possible.

  1. Capture the show log, show version, show test 0,and show logging buffer 1023 command output.

    The show log command output has a number of important indications of problems that can be related to a crash.

    • It keeps track of the last ten system resets with timestamps that show when the reboot occurred. This is a snapshot of the Reboot History output: 

      Reboot History:   Jan 23 2002 11:14:16 0, Jan 22 2002 14:57:21 0
                   DEC 24 2001 13:56:38 0, DEC 24 2001 13:52:30 0
                   DEC 11 2001 12:31:59 0, DEC 07 2001 13:26:48 0
                   DEC 07 2001 10:42:19 0, DEC 07 2001 10:36:16 0
                   Nov 28 2001 11:03:10 0, Oct 26 2001 16:04:26 0

      The Reboot History only indicates that the switch was reset. It can have been reset manually by the user or due to a crash. But, the most recent manual reset of the switch is recorded further down in the output. 

      Last software reset by user: Jan 23 2002 11:14:16 0

      Notice that the timestamp of the last manual reset, 1/23/2002,11:13:13, matches the most recent entry in the Reboot History.

    • It shows if there have been any exceptions. Exceptions are CPU dumps that occur immediately after a crash. For example: 

      MCP Exceptions/Hang:    0

      In this case, there were no exceptions recorded. If there were an exception, it includes a timestamp that can be matched with the Reboot History, and also includes a HEX dump or stack, which can be decoded by a TAC engineer in order to determine whether this was a software forced exception or due to hardware.

    The show version command provides software version information to use for a bug search. For example, if you identify an exception in the show log command output, use the Bug Toolkit in order to search for bugs on the Catalyst 4000 and the exception. Also, the show version command gives you a quick snapshot of how long the switch has been up. For example: 

    Uptime is 28 days, 11 hours, 42 minutes

    The show test 0 command output indicates an F status on the supervisor or module if any of the diagnostics failed. An improperly seated module can cause the switch to crash. If the supervisor or module shows failed, proceed with the troubleshooting steps in the System component LEDs are orange/red or supervisor not online section of this document.

    The show logging buffer 1023 command displays all system messages, which includes possible error messages that can relate to the crash. See the Getting system error messages section for troubleshooting suggestions.

  2. Issue the show commands and troubleshooting procedures in the preceding steps first. If these steps fail, capture the show tech-support command output. This command displays output for all these commands continuously, which means the output continues to scroll until complete or until the display is ended with the Ctrl + C keystrokes:

    sh version, sh flash, sh microcode, sh system, sh module, sh port, sh Mac, sh trunk, sh vlan, sh vtp domain, sh spantree active, sh spantree summary, sh test, sh arp, sh ip route, sh cdp neighbor detail, sh netstat stats, show memory buffers, show out-of-band stats, sh inband stats, show cam static, sh cam count dynamic, sh cam system, sh config, sh log, sh proc, sh proc mem, sh proc CPU, ps, Ps -c

    Often, the output from all these commands is not necessary to resolve a specific problem, so TAC engineers cannot ask for it. But, it is beneficial to have this output should other show commands or troubleshooting steps fail to resolve the problem.

  3. Should all the previous troubleshooting steps fail to diagnose the problem, capture these hidden commands, as well as the other commands in the previous steps, and open a case with the Cisco Technical Support:

    • ps-c (capture multiple times)

    • show mbuf all (hidden)

    • show nvramenv 1 (hidden)

    • show interposition 1 (hidden)

    These are hidden commands, which means they cannot be parsed with a "?" and you cannot Tab to complete. Type the command out in its entirety. This output can or cannot be useful in the resolution of your case. These commands are undocumented, and therefore the TAC engineer is not required to explain the output to the customer.

Misleading Problems

There are many misleading problems that are thought to be caused by faulty hardware. This section lists a few issues that are often confused as a hardware failure.

  • One common customer issue is for the system LED to show faulty when additional power supplies are added, but not plugged in. When this happens, both the ps#-status and sys-status shows faulty. This is because the switch senses an additional power supply is installed but is not active. Since this can also mean that the additional power supply has actually failed, an onsite inspection is required.

  • A common misconception when you view output of the show proc cpu command is that the Kernel and Idle percentage is interpreted to be the CPU utilization for that time period. The Kernel and Idle is the percentage of CPU that was idle for that time frame.

show Command Descriptions

These table breaks down what show commands are used to help troubleshoot the different symptom types.

Connectivity Problems System/Supervisor/Module Problems Supervisor Resets/Crashes
show version show config show module show system show port capabilities show port <mod/port> show Mac<mod/port> show counters <mod/port> clear counters show cdp neighbors detail show spantree summary show version show module show flash show config show test 0 show system show time show logging buffer 1023 show proc CPU or Ps -c show mbuf all (hidden) show nvramenv 1 (hidden) show interposition 1 (hidden) show log show logging buffer 1023 show version show test 0 show system show tech support ps-c (multiple times) (hidden) show mbuf all (hidden) show nvramenv 1 (hidden) show interposition 1 (hidden)

Capture these show commands that depends on your symptom(s).

Notice that many of the commands in each previous symptom category overlap. This is because the same symptom can occur in different levels of severity; one can cause a performance issue and the other can cause a crash.

Notice also that some of the commands seem meant more for software troubleshooting or configuration problems. For instance, the show spantree summary command shows which VLANs run STP, how many ports are in each VLAN, if any ports on the switch are blocking, and for which VLANs that they are blocking. Since STP loops can actually bring down a switch or network that gives the appearance of a hardware failure, this is vital information to capture, whether troubleshooting hardware or software.

show version

This command verifies the version of software you are running. This command also has information about the size of Flash and DRAM. This is useful information should you need to upgrade. If an upgrade is required, always check the release notes first. Refer to the to Catalyst 4500 Family Release Notes and choose the version you need to upgrade. For example, choose the Release Notes for Catalyst 4000 Family Software Release 5.x and search on any information in regards to your hardware.

Refer to Table 2-82: show version Command Output Fields for more information.

Cat4000-c> (enable) show version
WS-C4006 Software, Version NmpSW: 6.3(1)
Copyright (c) 1995-2001 by Cisco Systems, Inc.
NMP S/W compiled on Jul 24 2001, 12:55:29
GSP S/W compiled on Jul 24 2001, 10:36:29

System Bootstrap Version: 5.4(1)

Hardware Version: 2.0  Model: WS-C4006  Serial #: JAB04380209

Mod Port Model      Serial #              Versions
--- ---- ---------- -------------------- ---------------------------------
1   2    WS-X4013   JAB04380209          Hw : 2.0
                                         Gsp: 6.3(1.0)
                                         Nmp: 6.3(1)
2   34   WS-X4232-L3 JAB045004AA          Hw : 1.5
3   24   WS-X4424-GB-RJ45 JAB0514071N          Hw : 0.7
5   6    WS-X4306   JAB02400048          Hw : 0.2

       DRAM                    FLASH                   NVRAM
Module Total   Used    Free    Total   Used    Free    Total Used  Free
------ ------- ------- ------- ------- ------- ------- ----- ----- -----
1       65536K  33235K  32301K  16384K  16173K    211K  480K  180K  300K

Uptime is 28 days, 11 hours, 42 minutes

show module

This command displays information about the modules installed in the switch. In particular, note the status of the module. If the status is faulty, this can be a hardware failure.

Cat4000-c> (enable) show module
 Mod Slot Ports Module-Type               Model               Sub Status
 --- ---- ----- ------------------------- ------------------- --- --------
 1   1    2     1000BaseX Supervisor      WS-X4013            no  OK
 2   2    34    Router Switch Card        WS-X4232-L3         no  OK
 3   3    24    10/100/1000 Ethernet      WS-X4424-GB-RJ45    no  disable
 5   5    6     1000BaseX Ethernet        WS-X4306            no  OK
   
 Mod Module-Name Serial-Num 
      --- -------------------- -------------------- 
      1 JAB04380209  
      2 JAB045004AA  
      3 JAB0514071N  
      5 JAB02400048 
   
 Mod MAC-Address(es) Hw Fw SW 
      --- -------------------------------------- ------ ---------- ----------------- 
      1 00-02-b9-83-ac-00 to 00-02-b9-83-af-ff 2.0 5.4(1) 6.3(1) 
      2 00-02-16-f6-64-5c to 00-02-16-f6-64-7d 1.5 12.0(7)W5( 12.0(14)W5(20) 
      3 00-30-85-0e-2c-18 to 00-30-85-0e-2c-2f 0.7  
      5 00-10-7b-f6-9c-e4 to 00-10-7b-f6-9c-e9 0.2  
      Cat4000-c> (enable)

Refer to Table 2-35: show module Command Output Fields for more information.

show flash

This command displays the the contents of the Flash file system. Flash file systems differ between Catalyst supervisors. Some supervisors use the show flash command to display the contents, while others use the dir bootflash: command. When you copy an image to the SupIIIG, for example, you use the download command and the Flash is completely erased in the process of installing the image. With other sups, you can use the copy tftp flash command in order to add one or more images.

Many problems, both hardware and software related, can be avoided if you understand the Flash system for your supervisor.

Refer to the show flash or dir bootflash: command for more information.

Cat4000-c> sh flash
-#- ED --type-- --crc--- -seek-- nlen -length- -----date/time------ name
  1 .. ffffffff 4e88958b  42a97c   17  4106492 Aug 17 2001 16:22:52 cat4000.6-3n
  2 .. ffffffff b965ace8  78e71c   18  3554592 Nov 28 2001 10:38:33 cat4000.5-5n
  3 .. ffffffff 70a608c8  b8fa9c   20  4199168 DEC 07 2001 10:30:01 cat4000-k9.n
  4 .. ffffffff e873ea40  f0b224   17  3651336 DEC 11 2001 12:26:20 cat4000.5-5n

216540 bytes available (15512100 bytes used)
Cat4000-c>

show config

This command displays the non-default system configuration. This is useful to capture every time you make a configuration change as a way to possibly associate changes to hardware or software problems. Notice there is is a timestamp for each output. Compare the output to the show config all command output, which shows the entire system configuration and can be quite lengthy. Refer to the show config command for more information.

Cat4000-c> (enable) show config
This command shows non-default configurations only.
Use 'show config all' to show both default and non-default configurations.
.............
..................

....................

..

begin
!
# ***** NON-DEFAULT CONFIGURATION *****
!
!
#time: Tue Jan 22 2002, 11:20:05
!
#version 6.3(1)
!
!
#system web interface version(s)
!
#test
!
#system
set system name  Cat4000-c
!
#frame distribution method
set port channel all distribution Mac both
!
#vtp
set vtp domain blah
!
#ip
set interface sc0 1 172.16.84.200/255.255.255.0 172.16.84.255

set interface sl0 down
set interface me1 1.1.1.1 255.255.255.0 1.1.1.255

set ip route 0.0.0.0/0.0.0.0         172.16.84.1
!
#syslog
set logging level cops 2 default
!
#set boot command
set boot config-register 0x2102
clear boot system all
!
#mls
set mls nde disable
!
#port channel
set port channel 1/1-2 100
!
#module 1 : 2-port 1000BaseX Supervisor
set udld enable 1/1
set port channel 1/1-2 mode desirable silent
!
#module 2 : 34-port Router Switch Card
!
#module 3 : 24-port 10/100/1000 Ethernet
set vlan 150  3/9
!
#module 4 empty
!
#module 5 : 6-port 1000BaseX Ethernet
!
#module 6 empty
!
#cam
set cam permanent 01-00-5e-01-01-01 1/1 1

end
Cat4000-c> (enable)

show test 0

This command displays the results of diagnostic tests for the supervisor and all modules. It is very important to understand that the show test command only displays the results of diagnostics on the last bootup of the switch or a reset of the supervisor or modules. If the diagnostics for one module are required, issue the show test <mod #> command for this information.

If you are running 5.4.1 or later, check the status of the diaglevel by issuing the show test diaglevel command. A complete status test of the Encoded Address Recognition Logic (EARL), port loopback/bundle/inline rewrite, and DRAM/NVRAM/External cache is recommended. This test takes about one minute versus 30 seconds for a test level of minimal. But, it is more thorough. Results are output with a . for pass or F for fail, which indicates a hardware failure.

Display and/or change the diaglevel as follows:

Cat4000-c> (enable) show test diaglevel     
 Diagnostic mode at next reset  : minimal 
Cat4000-c> (enable) set test diaglevel ? 
  complete                   Complete diagnostics
   minimal                    Minimal diagnostics 
  bypass                     Bypass diagnostics

Diagnostic level set to complete. 

Cat4000-c> (enable) show test diaglevel     
 Diagnostic mode at next reset  : complete

Refer to the show test command for more information.

Cat4000-c> (enable) show test 0

Diagnostic mode at next reset: complete 
System Diagnostic Status : (. = Pass, F = Fail, N = N/A)
  Module 1 : 2-port 1000BaseX Supervisor
 Status: (. = Pass, F = Fail, U = Unknown)
  Module 2 : 34-port Router Switch Card
 Status: (. = Pass, F = Fail, U = Unknown)
   Eeprom: .
   CX1000 Regs:
     Ports 3-11  : .    Ports 12-19    : .    Ports 20-27 : .
     Ports 28-34 : .
   CX1000 Sram:
     Ports 3-11  : .    Ports 12-19    : .    Ports 20-27 : .
     Ports 28-34 : .
   10/100Base-TX Loopback Status:
   Ports   3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
          -----------------------------------------------------------------------
           .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .


            27 28 29 30 31 32 33 34
          -----------------------
           .  .  .  .  .  .  .  .
   1000Base-X Loopback Status:
   Ports   1  2
          -----
           .  .

  Router CPU board Status:

  Module 3 : 24-port 10/100/1000 Ethernet
 Status: (. = Pass, F = Fail, U = Unknown)
   Eeprom: .
   Lemans Regs:
     Ports 1-4   : .    Ports 5-8   : .    Ports 9-12  : .
     Ports 13-16 : .    Ports 17-20 : .    Ports 21-24 : .
   Lemans SRAM:
     Ports 1-4   : .    Ports 5-8   : .    Ports 9-12  : .
     Ports 13-16 : .    Ports 17-20 : .    Ports 21-24 : .

  10/100/1000Base-TX Loopback Status:
   Ports   1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
          -----------------------------------------------------------------------
           .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .  .

   Module 5 : 6-port 1000BaseX Ethernet
 Status: (. = Pass, F = Fail, U = Unknown)
   Eeprom: .
   Alpheratz: .

    1000BaseX Loopback Status:
     Ports  1  2  3  4  5  6
     -----------------------
            .  .  .  .  .  .
 Cat4000-c> (enable)

show system

This command displays system information. The status fields relate to the various LEDs on the system components. Take note of the uptime or how long the switch has been up and running. This would be useful information to know in the event of a switch crash. Refer to the show system command for more information.

Cat4000-c> (enable) show system
 PS1-Status PS2-Status PS3-Status PEM Installed PEM Powered
 ---------- ---------- ---------- ------------- -----------
 OK         OK         none       no           no           

  Fan-Status Temp-Alarm sys-status Uptime d,h:m:s Logout
 ---------- ---------- ---------- -------------- ---------
 OK         off        OK         28,15:10:39    20 min

  PS1-Type     PS2-Type     PS3-Type     
 ------------ ------------ ------------ 
 WS-C4008     WS-C4008     none     

  Modem   Baud  Traffic Peak Peak-Time
 ------- ----- ------- ---- -------------------------
 disable  9600   0%      0% Fri Jan 11 2002, 13:37:07

  Power Capacity of the Chassis: 2 supplies


   System Name              System Location          System Contact           CC
 ------------------------ ------------------------ ------------------------ ---
 Cat4000-c

show time

This command displays the day of the week/month/year and the time in a 24-hour format. This verifies the operation of the system clock, but also is a reminder that system log messages carry a timestamp. Make sure to set the time accurately or sync the switch to Network Time Protocol (NTP). 

Cat4000-c> (enable) show time
 Wed Jan 23 2002, 10:41:22 
 Cat4000-c> (enable)

Refer to the show time command for more information.

show logging buffer 1023

This command displays system messages from the internal buffer. The show logging buffer command only gives you the last 20 system messages, while if you add the 1023 keyword, this gives you the last 1023 messages. Many of these messages are strictly informational. Others can contain clues as to the nature of the problem, whether it is a hardware problem, switch crash, or software problem. When you compare the logs on several pieces of equipment, verify that the time stamps are correct and issue the show time command.

For example, these types of messages are informational:

2002 Jan 06 16:07:04 %DTP-5-TRUNKPORTON:Port 2/23 has become dot1q trunk
2002 Jan 06 16:07:08 %PAGP-5-PORTTOSTP:Port 2/21 joined bridge port 2/21-24

A message like this one indicates a hw/sw incompatibility:

Module 6 is not supported (46)

A message like this one can indicate a hardware failure:

EARL-3-LTL: Failure to set LTL for module [DEC]

Refer to Messages and Recovery Procedures for a listing of system messages. Use the Bug Toolkit and other resources described under the Prerequisites section in this document. Also, refer to Common CatOS Error Messages on Catalyst 4000 Series Switches for more information.

Refer to the show logging buffer 1023 command for more information:

Cat4000-c> sh logging buffer 1023
2002 Jan 23 11:14:23 %SYS-5-MOD_OK:Module 1 is online
2002 Jan 23 11:14:32 %SYS-5-MOD_OK:Module 5 is online
2002 Jan 23 11:14:35 %SYS-5-MOD_OK:Module 3 is online
2002 Jan 23 11:14:54 %PAGP-5-PORTTOSTP:Port 3/9 joined bridge port 3/9
2002 Jan 23 11:15:14 %SYS-5-MOD_OK:Module 2 is online
2002 Jan 23 11:15:23 %PAGP-5-PORTFROMSTP:Port 3/9 left bridge port 3/9
2002 Jan 23 11:15:30 %PAGP-5-PORTTOSTP:Port 2/1 joined bridge port 2/1
2002 Jan 23 11:15:30 %PAGP-5-PORTTOSTP:Port 2/2 joined bridge port 2/2
2002 Jan 23 11:15:41 %PAGP-5-PORTTOSTP:Port 3/9 joined bridge port 3/9
2002 Jan 23 11:17:19 %PAGP-5-PORTFROMSTP:Port 3/9 left bridge port 3/9
2002 Jan 23 11:17:37 %PAGP-5-PORTTOSTP:Port 3/9 joined bridge port 3/9
Cat4000-c>

show proc cpu

This command displays information about CPU usage. Issue the ps-c command in order to format this information differently.

Refer to these documents for more information on how the CPU is utilized on Catalyst 4000, 2948G, 2980G, and 4912G switches

Cat4000-c> (enable) show proc cpu

CPU utilization for five seconds:  11.62%
                       one minute:  12.00%
                     five minutes:  12.00%

  PID Runtime(ms) Invoked    uSecs    5Sec    1Min    5Min    TTY Process
 --- ----------- ---------- -------- ------- ------- ------- --- ---------------
 1   20176816    0          0         88.38%  88.00%  88.00% -2  Kernel and Idle
 2   8           131        1000       0.00%   0.00%   0.00% -2  Flash MIB Updat
 3   97245       176675     40000      0.25%   0.00%   0.00% -2  SynConfig      
 4   33358       34879      2000       0.96%   0.00%   0.00% -2  Statuspoll     
 5   6254        87069      1000       0.00%   0.00%   0.00% -2  PwrDevMsgUpd   
 6   376         5258       1000       0.00%   0.00%   0.00% -2  StatusPoll 5s  
 8   5           2          5000       0.00%   0.00%   0.00% -2  SecurityRx     
 9   106         1092       1000       0.00%   0.00%   0.00% -2  SWPoll64bCnt   
 10  1713        26229      1000       0.00%   0.00%   0.00% -2  Earl           
 11  172         2613       1000       0.00%   0.00%   0.00% -2  ProtocolFilter 
 12  0           1          0          0.00%   0.00%   0.00% -2  telnetd        
 13  0           1          0          0.00%   0.00%   0.00% -2  llcSSTPFlood   
 14  441829      9511273    1000       1.47%   1.00%   1.00% -2  gsgScpAggregati
 15  347         444        1000       0.00%   0.00%   0.00% -2  cdpd           
 16  58134       26267      5000       0.57%   0.00%   0.00% -2  cdpdtimer      
 17  29751       26913      9000       0.96%   0.00%   0.00% -2  SptTimer       
 18  1           1          1000       0.00%   0.00%   0.00% -2  SptBpduRx      
 19  40610       26227      3000       0.28%   0.00%   0.00% -2  SptBpduTx      
 20  2230        26227      1000       0.16%   0.00%   0.00% -2  VtpTimer       
 21  0           1          0          0.00%   0.00%   0.00% -2  RMON AlarmTimer
 22  22352       257353     9000       0.28%   0.00%   0.00% -2  ProtocolTimer  
 23  2024        2305       2000       0.00%   0.00%   0.00% -2  DTP_Rx         
 24  649         1200       16000      0.00%   0.00%   0.00% -2  EthChnlRx      
 25  901         1745       2000       0.00%   0.00%   0.00% -2  EthChnlConfig  
 26  15943       260008     1000       0.28%   0.00%   0.00% -2  sptHelper      
 27  0           1          0          0.00%   0.00%   0.00% -2  sptTraps       
 28  154         2629       1000       0.00%   0.00%   0.00% -2  ciscoRmonTimer 
 29  167         2629       1000       0.00%   0.00%   0.00% -2  ciscoUsrHistory
 30  1           1          1000       0.00%   0.00%   0.00% -2  rmonMediaIndep 
 31  0           1          0          0.00%   0.00%   0.00% -2  SnmpTraps      
 32  0           1          0          0.00%   0.00%   0.00% -2  Acct Send Bkg  
 34  0           1          0          0.00%   0.00%   0.00% -2  l2t_server     
 36  164         504        1000       0.00%   0.00%   0.00% -2  SysLogTask     
 37  8188        26039      1000       0.80%   0.00%   0.00% -2  pinggateA      
 38  43007       876770     1000       0.44%   0.00%   0.00% -2  Authenticator_S
 39  0           1          0          0.00%   0.00%   0.00% -2  dot1x_rx       
 40  3423        57501      1000       0.32%   0.00%   0.00% -2  Backend_Rx     
 41  39173       577158     1000       0.09%   0.00%   0.00% -2  Backend_SM     
 143 642792      9511281    34000      2.28%   2.00%   2.00% 0   Console        
 144 199         1          199000     0.00%   0.00%   0.00% -2  snmpdm         
 145 1           2          1000       0.00%   0.00%   0.00% -2  VtpRx          
 193 591423      783586     10730      2.26%   2.27%   2.22% 0   Packet forwardi
 194 353123      359502     6164       1.33%   1.35%   1.36% 0   Switching overh
 195 727712      633244     57354      2.83%   2.85%   2.77% 0   Admin overhead 
 Cat4000-c> (enable)

show port capabilities

This command displays the capabilities of the modules and ports in a switch. Think of this command as a quick way to display hardware/software features without the need to search the release notes. This command can answer question, such as what trunk encapsulation types are supported and can the ports etherchannel. Refer to Table 2-49: show port capabilities Command Output Fields for more information.

Cat4000-c> (enable) show port capabilities 2/1
 Model                    WS-X4232-L3
 Port                     2/1
 Type                     No Connector
 Speed                    1000
 Duplex                   full
 Trunk encap type         802.1Q
 Trunk mode               on,off
 Channel                  2/1-2
 Flow control             no
 Security                 yes
 Dot1x                    yes
 Membership               static,dynamic
 Fast start               yes
 QOS scheduling           rx-(none),tx-(2q1t)
 CoS rewrite              no
 ToS rewrite              no
 Rewrite                  no
 UDLD                     yes
 Inline power             no
 AuxiliaryVlan            no
 SPAN                     source
 Link debounce timer      yes
 Cat4000-c> (enable)

show port <mod/port>

This command displays port status and counters. If the status is anything other than connected, see the troubleshooting steps in the Port status shows not connected, faulty, disabled, inactive or errdisable section of this document . If the port counters show incrementing errors, see the troubleshooting steps in the Seeing errors on ports section.

Refer to the show port command for more information.

Cat4000-c> (enable) show port 3/9
 Port  Name               Status     Vlan       Level  Duplex Speed Type
 ----- ------------------ ---------- ---------- ------ ------ ----- ------------
  3/9                     connected  1          normal a-full a-100 10/100/1000

  Port  AuxiliaryVlan AuxVlan-Status     InlinePowered     PowerAllocated
                                  Admin Oper   Detected mWatt mA @51V
 ----- ------------- -------------- ----- ------ -------- ----- --------
  3/9  none          none           -     -      -        -     -

  Port  Security Violation Shutdown-Time Age-Time Max-Addr Trap     IfIndex
 ----- -------- --------- ------------- -------- -------- -------- -------
  3/9  disabled  shutdown             0        0        1 disabled      64

  Port  Num-Addr Secure-Src-Addr   Age-Left Last-Src-Addr     Shutdown/Time-Left
 ----- -------- ----------------- -------- ----------------- ------------------
  3/9         0                 -        -                 -        -         -

  Port   Send FlowControl    Receive FlowControl   RxPause TxPause Unsupported
        admin    oper       admin    oper                           opcodes
 -----  -------- --------   -------- --------     ------- ------- -----------
  3/9   on       disagree   desired  off          0       0       0

  Port  Status     Channel              Admin Ch
                  Mode                 Group Id
 ----- ---------- -------------------- ----- -----
  3/9  connected  auto silent             40     0


   Port  Align-Err  FCS-Err    Xmit-Err   Rcv-Err    UnderSize
 ----- ---------- ---------- ---------- ---------- ---------
  3/9           -          0          0          0         0

  Port  Single-Col Multi-Coll Late-Coll  Excess-Col Carri-Sen Runts     Giants
 ----- ---------- ---------- ---------- ---------- --------- --------- ---------
  3/9           0          0          0          0         0         0         0

  Last-Time-Cleared
 --------------------------
 Tue Jan 22 2002, 14:57:21

show mac <mod/port>

This command displays the MAC counters, and is useful in the determination of whether counters are incrementing as expected. This command shows the total unicast, multicast, and broadcast frames received on a port. The In-lost counter on the Catalyst 4000 reflects the sum of all error packets received on the port. This is different then the behavior of the In-Lost counter on the Catalyst 5000 switches; which reflects the sum of all receive buffer failures. The out-Lost counter on both the Catalyst 4000 and 5000, reflect outgoing frames that were lost before forwarded due to insufficient buffer space. This is commonly caused if you oversubscribe the interface.

See the Seeing errors on the ports section of this document for troubleshooting steps. Refer to the show mac command for more information.

Cat4000-c> (enable) show mac 2/1

Port     Rcv-Unicast          Rcv-Multicast        Rcv-Broadcast
-------- -------------------- -------------------- --------------------
 2/1                        6                  446                    0

Port     Xmit-Unicast         Xmit-Multicast       Xmit-Broadcast
-------- -------------------- -------------------- --------------------
 2/1                        6                16041                26236

Port     Rcv-Octet            Xmit-Octet
-------- -------------------- --------------------
 2/1                   149408              2901773

MAC      Dely-Exced MTU-Exced  In-Discard Lrn-Discrd In-Lost    Out-Lost
-------- ---------- ---------- ---------- ---------- ---------- ----------
 2/1              0          0          0          0          0          0

Last-Time-Cleared
--------------------------
Tue Jan 22 2002, 14:57:21

show counters <mod/port>

This command displays hardware counters for the port and will vary depending on the type of port. See the Seeing errors on the ports section of this document for troubleshooting steps. Refer to the show counters command for more information.

Cat4000-c> (enable) show counters 2/1
2  rxUnicastPacketCount       = 6
3  txUnicastPacketCount       = 6
4  rxMulticastPacketCount     = 447
5  txMulticastPacketCount     = 16078
6  rxBroadcastPacketCount     = 0
7  txBroadcastPacketCount     = 26296
8  rxByteCount                = 149742
9  txByteCount                = 2908424
10 pkts64                     = 40611
11 pkts65to127                = 890
12 pkts128to255               = 441
13 pkts256to511               = 891
14 pkts512to1023              = 0
15 pkts1024to1522             = 0
16 rxNoPacketBufferCount      = 0
17 rxCRCAlignErrorPacketCount = 0
18 rxUndersizedPacketCount    = 0
19 rxOversizedPacketCount     = 0
20 rxFragmentPacketCount      = 0
21 rxJabberPacketCount        = 0
22 pauseControlFramesRx       = 0
23 pauseControlFramesTx       = 0
24 unsupportedOpcodesRx       = 0
25 txQueueNotAvailable        = 0
26 totalCollisionCount        = 0
27 lateCollisionCount         = 0
28 singleCollisionFrames      = 0
29 multipleCollisionFrames    = 0
30 excessiveCollisionFrames   = 0
31 deferredTransmissions      = 0
32 carrierSenseErrors         = 0
33 falseCarrierDuringIdle     = 0
34 symbolErrorDuringCarrier   = 0
35 sequenceErrorDuringCarrier = 0

clear counters

This command is used to reset the show port, show mac, and show counter statistics. It is useful for the determination of errors that continue to increment or have been resolved.

Refer to the clear counters command for more information.

show cdp neighbors detail

This command shows details about remote Cisco devices using CDP. This is one quick way to get the IP address and interface of a Cisco device on any given switchport. Refer to the show cdp neighbors detail commands for more information.

Cat4000-c> (enable) show cdp neighbors detail 
Port (Our Port): 2/1
Device-ID: 8-4006-L3
Device Addresses:
IP Address: 127.0.0.3
Holdtime: 170 sec
Capabilities: ROUTER
Version:
Cisco Internetwork Operating System Software
IOS (tm) L3 Switch/Router Software (CAT4232-IN-M), Version 12.0(14)W5(20)  RE
  Copyright (c) 1986-2001 by cisco Systems, Inc.
  Compiled Thu 01-Mar-01 18:18 by integ
Platform: cisco Cat4232L3
Port-ID (Port on Neighbors's Device): GigabitEthernet3
VTP Management Domain: unknown
Native VLAN: unknown
Duplex: unknown
System Name: unknown
System Object ID: unknown
Management Addresses: unknown
Physical Location: unknown
___________________________________________________________________________
Port (Our Port): 2/2
Device-ID: 8-4006-L3
Device Addresses:
  IP Address: 127.0.0.3
Holdtime: 170 sec
Capabilities: ROUTER
Version:
  Cisco Internetwork Operating System Software
  IOS (TM) L3 Switch/Router Software (CAT4232-IN-M), Version 12.0(14)W5(20)  RE
  Copyright (c) 1986-2001 by cisco Systems, Inc.
  Compiled Thu 01-Mar-01 18:18 by integ
Platform: cisco Cat4232L3
Port-ID (Port on Neighbors's Device): GigabitEthernet4
VTP Management Domain: unknown
Native VLAN: unknown
Duplex: unknown
System Name: unknown
System Object ID: unknown
Management Addresses: unknown
Physical Location: unknown
Cat4000-c> (enable)

show spantree summary

This command provides a summary of STP information useful in troubleshooting link flaps and other network issues masquerading as hardware issues. Refer to the show spantree summary and the show spantree commands for more information.

Cat4000-c> (enable) show spantree summary
MAC address reduction: disabled
Root switch for vlans: 1.
BPDU skewing detection disabled for the bridge
BPDU skewed for vlans:  none.
Portfast bpdu-guard disabled for bridge.
Portfast bpdu-filter disabled for bridge.
Uplinkfast disabled for bridge.
Backbonefast disabled for bridge.

Summary of connected spanning tree ports by vlan

VLAN  Blocking Listening Learning Forwarding STP Active
----- -------- --------- -------- ---------- ----------
   1         0         0        0          3          3

      Blocking Listening Learning Forwarding STP Active
----- -------- --------- -------- ---------- ----------
Total        0         0        0          3          3
Cat4000-c> (enable)

show log

This command displays the error log for the system or a specific module. If there has been a switch reset or crash, the stack information necessary to determine the cause of the switch crash is displayed here. Refer to the show log command for more information.

Cat4000-c> show log

Network Management Processor (ACTIVE NMP) Log:
  Reset count:   15
  Reboot History:   Jan 23 2002 11:14:16 0, Jan 22 2002 14:57:21 0
                     DEC 24 2001 13:56:38 0, DEC 24 2001 13:52:30 0
                     DEC 11 2001 12:31:59 0, DEC 07 2001 13:26:48 0
                     DEC 07 2001 10:42:19 0, DEC 07 2001 10:36:16 0
                     Nov 28 2001 11:03:10 0, Oct 26 2001 16:04:26 0
  Bootrom Checksum Failures:      0   UART Failures:                  0
  Flash Checksum Failures:        0   Flash Program Failures:         0
  Power Supply 1 Failures:        0   Power Supply 2 Failures:        0
  DRAM Failures:                  0

  Exceptions:                     0

  Loaded NMP version:            6.3(1)
  Reload same NMP version count: 2

  Last software reset by user: 1/23/2002,11:13:13

  MCP Exceptions/Hang:            0

Heap Memory Log:
Corrupted Block = none

NVRAM log:

01. 11/28/2001,11:03:11: convert_post_SAC_CiscoMIB:Nvram block 0 unconvertible:)
02. 11/28/2001,11:03:11: convert_post_SAC_CiscoMIB:Nvram block 1 unconvertible:)
03. 11/28/2001,11:03:11: convert_post_SAC_CiscoMIB:Nvram block 2 unconvertible:)
04. 11/28/2001,11:03:11: check_block_and_log:Block 3 has been deallocated: (0x1)
05. 11/28/2001,11:03:11: convert_post_SAC_CiscoMIB:Nvram block 5 unconvertible:)
06. 11/28/2001,11:03:11: check_block_and_log:Block 35 has been deallocated: (0x)
07. 11/28/2001,11:03:11: convert_post_SAC_CiscoMIB:Nvram block 44 unconvertible)
08. 11/28/2001,11:03:11: convert_post_SAC_CiscoMIB:Nvram block 62 unconvertible)
09. 11/28/2001,11:03:14: supVersion:Nmp version 5.5(11)
10. 12/7/2001,10:36:16: convert_post_SAC_CiscoMIB:Block 0 converted from versio5
11. 12/7/2001,10:36:20: supVersion:Nmp version 6.3(3)
12. 12/11/2001,12:32:00: convert_post_SAC_CiscoMIB:Nvram block 0 unconvertible:)
13. 12/11/2001,12:32:00: convert_post_SAC_CiscoMIB:Nvram block 1 unconvertible:)
14. 12/11/2001,12:32:00: convert_post_SAC_CiscoMIB:Nvram block 2 unconvertible:)
15. 12/11/2001,12:32:00: convert_post_SAC_CiscoMIB:Nvram block 5 unconvertible:)
16. 12/11/2001,12:32:00: check_block_and_log:Block 35 has been deallocated: (0x)
17. 12/11/2001,12:32:00: convert_post_SAC_CiscoMIB:Nvram block 44 unconvertible)
18. 12/11/2001,12:32:00: convert_post_SAC_CiscoMIB:Nvram block 62 unconvertible)
19. 12/11/2001,12:32:04: supVersion:Nmp version 5.5(8)
20. 12/24/2001,13:56:38: convert_post_SAC_CiscoMIB:Block 0 converted from versi5
21. 12/24/2001,13:56:42: supVersion:Nmp version 6.3(1)

Module 2 Log:
  Reset Count:   16
  Reset History: Wed Jan 23 2002, 11:15:13
                 Tue Jan 22 2002, 14:58:18
                 Tue Jan 15 2002, 17:03:35
                 Tue DEC 11 2001, 12:32:58


Module 3 Log:
  Reset Count:   12
  Reset History: Wed Jan 23 2002, 11:14:34
                 Tue Jan 22 2002, 14:57:39
                 Mon DEC 24 2001, 13:56:53
                 Fri DEC 7 2001, 13:27:07


Module 5 Log:
  Reset Count:   15
  Reset History: Wed Jan 23 2002, 11:14:31
                 Tue Jan 22 2002, 14:57:36
                 Mon DEC 24 2001, 13:56:51
                 Mon DEC 24 2001, 13:52:43

show tech-support

This command displays this as continuous output:

show version, sh flash, sh microcode, sh system, sh module, sh port, sh mac, sh trunk, sh vlan, sh vtp domain, sh spantree active, sh spantree summary, sh test, sh arp, sh ip route, sh cdp neighbor detail, sh netstst ststs, show memory buffers, show out-of-band stats, sh inband stats, show cam static, sh cam count dynamic, sh cam system, sh config, sh log, sh proc, sh proc mem, sh proc cpu, ps, ps -c

Refer to the show tech-support command for more information.

Related Information

Revision History

Revision Publish Date Comments
1.0
02-Dec-2013
Initial Release

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products