Many hardware problems encountered during field installations or during normal operation can be prevented by a thorough product overview ahead of time. For Catalyst 5000 series switches, this means being able to identify the switch chassis, Supervisor Engine and switching modules, and understanding system specifications, cabling, and power requirements. Basic knowledge of switch architecture is helpful, as is an understanding of software considerations. Therefore, for those customers not already familiar with the Catalyst 5000 series, it is not required but highly recommended that you complete this document before proceeding to Hardware Troubleshooting for Catalyst 5500/5000/2926G/2926 Series Switches.
These major topics are covered:
Product Overview - an introduction to Catalyst 5500/5000/2926 series switches with links to detailed chassis, Supervisor Engine, and switching module documentation.
Switch Management - establishing console and Telnet connections and working with configuration files.
Software Considerations - software versions and naming conventions.
Basic Architecture Overview - understanding the differences and similarities between various Catalyst switch architectures and how frames are forwarded.
The following topics are not covered:
Software configuration and troubleshooting. Refer to LAN Product Support Pages.
Recovering from ROMmon mode. Refer to the document Recovering Catalyst Switches Running CatOS from Booting Failures.
Password recovery. Refer to the Password Recovery Procedures.
There are no specific requirements for this document.
This document is not restricted to specific software and hardware versions.
For more information on document conventions, refer to the Cisco Technical Tips Conventions.
The Catalyst 5000 series features five modular chassis: Catalyst 5000, 5002, 5505, 5509, and 5500 (13-slot chassis). The 2900/2926 series has six non-modular or fixed-configuration switches: the 2901, 2902, 2926T, 2926F, 2926GS, and 2926GL, which are in the same switch series as the 5000/5500 series and share the same architecture. A good reference for this subject is the Catalyst 2900 Platforms section of the document Understanding Catalyst 2900 and Catalyst 4000 Naming Conventions.
There are a number of Supervisor Engine, feature card, and uplink module options in the Catalyst 5000 series. Some have internal, non-replaceable power supplies, while others have external, removable power supplies. All Catalyst 5000/5500/2900/2926 Supervisor Engines have a centralized processing engine called the Encoded Address Recognition Logic (EARL) for learning and forwarding packets based on MAC addresses, but each Supervisor Engine has a different version of the EARL. For the 5000/5500 series, certain Supervisor Engines can only be used in certain switch chassis, and there are requirements for redundant Supervisor Engines. The information below helps you understand your switch and its capabilities.
Note: Different versions of Catalyst 5000 Supervisor Engines have different versions of the centralized processing engines. The discovery of an IEEE 802.1x security vulnerability makes it important to understand which version of EARL your Supervisor Engine has. Refer to the document Identifying Catalyst 5000 EARL Version and Other Common EARL Questions for more information.
Refer to the document Catalyst 5000 Family Installation Guide (May 2000). Cisco Technical Support recommends reviewing at least the following:
Start by referring to the Product Overview, which describes the features and functionality of the Catalyst 5000 series switches—specifically, which chassis supports which Supervisor Engines.
Refer to Site Planning, which talks about site power and environmental requirements.
Refer to Specifications, which lists the Catalyst 5000 series system specifications in detail.
Refer to the document Catalyst 2900 User Guide. Cisco Technical Support recommends reviewing at least the following:
Start by referring to the introduction, What Is the Catalyst 2900?, which describes the features and functionality of the Catalyst 2900.
Refer to Installation Overview, which talks about site power and environmental requirements.
Refer to System Specifications, which lists the Catalyst 2900 system specifications in detail.
Refer to the document Catalyst 2926 and 2926G Series Installation Guide. Cisco Technical Support recommends reviewing at least the following:
Start by referring to the Product Overview, which describes the features and functionality of this series.
Refer to Switch Description, which gives a more specific overview of system features and components.
Refer to Technical Specifications, which lists Catalyst 2926 series system specifications in detail.
Refer to the document Catalyst 5000 Supervisor Engine Install Guide (September 2000). Cisco Technical Support recommends reviewing at least the following:
Start by referring to the Product Overview, which describes the Catalyst 5000 series Supervisor Engine models. Pay particular attention to the section Supervisor Engine Redundancy, which lists the requirements for redundant Supervisor Engines.
Refer to Preparing for Installation, which talks about maximum distances for a variety of cables.
Refer to Installing the Supervisor Engine, which discusses proper insertion and removal of Supervisor Engines, modules, and feature cards.
Refer to the document Catalyst 5000 Family Module Install Guide (January 2000). Cisco Technical Support recommends reviewing at least the following:
Start by referring to the Product Overview, which describes the Catalyst 5000 series switching modules.
Refer to Installation Preparation, which talks about cable specifications and standards compliance per module.
Refer to Switching Module Installation, which discusses proper insertion and removal of modules.
Refer to Catalyst 5000 Family Installation & Configuration Notes. Cisco Technical Support recommends you review the relevant information for your specific Supervisor Engine module or line card, as time permits.
"Switch management" in this document describes configuration management, flash file system management, syslogging, and Simple Network Management Protocol (SNMP). This is one of the most critical elements in preparing to troubleshoot hardware. Simply put: have a backup plan in place. Back up your configuration on a regular basis, save passwords in a secure location, and log system messages either manually or automatically to a server. This principle applies to your network in general.
Take these steps in preparation for possible failures:
Keep a secure record of the Ethernet port and Gateway IP addresses of the switch. Refer to Configuring the Switch IP Address and Default Gateway for the Catalyst 5000/5500 series switches, Setting the Interface Type for the 2900, and Configuring the Switch for the 2926 series.
Keep a secure record of the default user login, password, and enable password. This document does not cover password recovery.
Routinely save a copy of the up-to-date switch configuration to flash or an external TFTP server. Refer to Working with Configuration Files for the Catalyst 5000/5500 series and 2926 series, and to Creating a Configuration File for the 2900.
Enable system message logging for alerts and critical messages on your switch. Refer to Configuring System Message Logging and Configuring SNMP for the Catalyst 5000/5500 series and 2926 series, and to Configuring SNMP Management for the 2900.
Cisco Technical Support highly recommends reading Best Practices for Catalyst 4000, 5000 and 6000 Series Switch Configuration and Management, which represents years of field experience from Cisco engineers working with many of their largest customers and complex networks.
Customers frequently have questions regarding software for Catalyst 4000, 5000, and 6500/6000 switches. Each Catalyst 4000, 5000, and 6500/6000 software release is a member of a release train (Catalyst 2900/2926 series switches run Catalyst 5000 software, as well). Part of troubleshooting hardware is understanding the software release numbering scheme, the software train life cycle, train types and definitions, time frames, and guidelines for releases.
The following documents are accessed from the Product Literature page for the Catalyst 5000 and are essential for answering questions of this type:
Notice from the links above that 5.5(7) is currently General Deployment (GD) for Catalyst 4000, 5000, and 6500/6000 switches in the 5.x train, while 4.5(4) is GD for the 4.x train.
For new features, memory requirements, caveats (bugs), or upgrade paths from early versions to later versions, always check the release notes. Depending on the switch, the release notes may also contain a condensed list of Supervisor Engine requirements and redundancy information.
For Catalyst 5000 series switches, refer to Supervisor Engine Module Software Release Notes. Check the release notes for the version of software you are currently running or to which you plan to upgrade. Pay particular attention to the sections Memory Requirements, Software Upgrade Path and Supervisor Engine Support Matrix.
For example, in Release Notes for Catalyst 5000 Family Software Release 5.x, check the sections Release 5.x Memory Requirements, Release 5.x Software Upgrade Path and Release 5.x Supervisor Engine Support Matrix. If you had a Supervisor Engine I and were planning to upgrade, you would see that a Supervisor Engine I for the Catalyst 5000 does not support 5.x. You could then check the Release Notes for Catalyst 5000 Family Software Release 4.x and determine that a Supervisor Engine I is supported in 4.x.
Note: The Catalyst 2900/2926 runs Catalyst 5000 software, as mentioned above in Software Considerations. The latest release supported by the Catalyst 2900/2926 series is 4.x. Only the Catalyst 2926Gs supports 5.x. None of the 2900/2926 series switches support 6.x.
The next section provides a basic overview of system architecture in preparation for detailed troubleshooting of Catalyst 5000 series switches.
The Catalyst 5000 series features five modular chassis: Catalyst 5002, 5000, 5505, 5500 (13-slot chassis), and 5509. The 2900/2926 series shares the same architecture.
The Catalyst 5000 series architecture has three basic components: the bus arbiter and EARL, the port interface, and the Network Management Processor (NMP). The bus arbiter and EARL (used by all ports) govern access to the data-switching bus and control packet transfer destination. Each Ethernet port interface comprises a custom application-specific integrated circuit (ASIC) called a SAINT (Synergy Advanced Interface and Network Termination) that has an integrated 10/100 Mbps Ethernet MAC controller. Other media ports make use of a second custom ASIC called a SAGE (Synergy Advanced Gate Array Engine) that does not have an integrated MAC controller.
The switch uses central bus arbitration and address recognition logic. Each port fully buffers each frame received from the network before issuing a request to the central bus arbiter of the switch to access the switching bus to transmit its stored frames. When given the go-ahead, the port forwards the frame across the backplane, and all ports on all modules, as well as the Supervisor Engine, receive a copy of it. The central EARL of the Supervisor Engine determines the destination ports that should forward the frame and instructs each port to either flush or forward the frame as appropriate.
Each frame traversing the switching bus may be destined to a single port or to multiple ports. Ensuring that every port gets a copy of each frame, and then using the "flush or forward" model, allows for high-speed multicast and broadcast forwarding without the need for frame copies. Frames cross the backplane just one time, thereby optimizing performance. The switching bus operates at 1.2 Gbps and resides on the backplane with interfaces to each line module.
The Catalyst 5000 series switch architecture uses a store-and-forward model for input and output. Each switch port maintains its own frame buffer memory. Each frame is stored in a frame buffer before it is forwarded.
Using the bus arbiter, the bus supports a three-level priority request scheme. The bus also allows each port to perform a local flush and maintains a packet retry mechanism used during outbound port congestion. The Catalyst 5000 architecture allows media-rate performance for all traffic types (unicast, broadcast, and multicast).
In addition to the backbone switching bus, the Catalyst switch maintains the management bus and the index bus.
The management bus carries configuration information from the NMP to each module and statistical information from each module to the NMP. The index bus carries port-select information from the central EARL to the ports. This information determines which ports forward the packet and which flush it from the buffer.
The Supervisor Engine II can forward up to 1 million packets per second (Mpps), and the Supervisor Engine III can forward up to 2 to 3 Mpps with the NetFlow feature card. The NetFlow feature card is a modular feature card upgrade to the Supervisor Engine that provides Layer 3 switching in hardware for the Catalyst 5000 series switches.
The Supervisor Engine controls data access to the backplane, prioritizes the interfaces, maintains up to 16,000 active MAC addresses in the bridge lookup table, monitors system status, delivers configuration information for all modules, gathers performance information, and updates operational software information. The Supervisor Engine also supports VLANs on a network-wide basis using any Fast Ethernet (with Inter-Switch Link Protocol [ISL] or 802.1Q), FDDI (with 802.10), or ATM (with LAN Emulation [LANE]) interface between multiple Catalyst 5000 switches and Cisco routers.
The NMP on the Supervisor Engine uses system software and performs system control and configuration, including the loading of run-time code to the line modules and system diagnostics. The NMP executes a separate instance of the Spanning Tree Protocol (STP) on each VLAN and runs a wide range of other switch protocols, as well as VLAN Trunking Protocol (VTP), Cisco Discovery Protocol (CDP), and others. The NMP is also responsible for controlling SNMP, Telnet, and console access to the switch.
The primary function of the master communication processor (MCP) on the Supervisor Engine is to communicate information between the NMP and the line module communication processors (LCPs) distributed on Catalyst 5000 line modules. This intermodule communication occurs across the management bus, which is a serial bus.
Each LCP processes information sent to it by the MCP across the management bus. The Serial Control Protocol (SCP) is an internal communications protocol used for this communication between modules. The LCP runs diagnostics on all the local module hardware upon power up or reset of the line module.
The switching bus is 48 bits wide and operates at 25 MHz, which gives a data-transfer rate of 1.2 Gbps. A bus access arbitration scheme is implemented on the Supervisor Engine. All line modules and the Supervisor Engine have access to the switching bus. The Catalyst 5500 employs three such buses. The Supervisor Engine II ties these buses together as a single data bus. The Supervisor Engine III enables independent 1.2-Gbps input buses to be bridged together by Phoenix ASICs, which are described below in the section PHOENIX ASIC on Supervisor Engine III.
Switching bus encapsulations are added and removed by the SAINT and SAGE ASICs.
Each dedicated Ethernet and Fast Ethernet switch port has its own SAINT ASIC and 192 KB of dedicated frame buffer. ATM and FDDI MAC layers are not IEEE 802.3-compliant, so these interfaces use the SAGE (non-Ethernet) ASIC and 1 to 2 MB of additional buffering.
The SAINT ASIC is part of the port interface that also includes a dual-channel direct memory access (DMA) controller for getting packets in and out of the buffer and onto and off the switching backplane. The SAINT is a highly integrated, high-performance, custom 10/100 Ethernet MAC-compliant (IEEE 802.3) ASIC. It has built-in support for 10-Mbps Ethernet, 4B/5B Fast Ethernet, and the media-independent interface (MII). It permits the use of half- and full-duplex Ethernet and two priority request levels. (A third priority, which is hardware-initiated, is reserved to prevent per-port buffer overflow conditions. The hardware-initiated priority has the highest priority and is set when the use of the input buffer exceeds 90 percent). SAINT also performs ISL and 802.1Q encapsulation and de-encapsulation on frames outbound from ports configured as trunks.
The external packet buffer stores data coming from the network before it is forwarded to the switching bus. Likewise, it is used to store data coming from the switching bus before it is forwarded to the network. The packet buffer is divided into two sections, one for packets going to the network (the output, or transmit buffer), and the other for packets going to the switching bus (the input, or receive buffer). The switching bus is very efficient in processing the incoming packets. The output buffer, therefore, is larger than the input buffer because it is more likely that the switch needs more output buffer than input buffer.
As a frame is received from the network and stored in the frame buffer of the port, ASICs on each port encapsulate Ethernet frames with 12 bytes of information to indicate VLAN ID (color), the port of origin for the frame, and a frame check sequence (FCS). This FCS is checked at the port receiving the frame from the switching bus. The same ASIC on each port also strips off that encapsulated information and extracts the Ethernet frame before sending it out the destination ports. This encapsulation is different from the 30-byte ISL encapsulation. Ports configured as ISL trunks also strip off the 12-byte encapsulation used across the Catalyst 5000 series switch backplane before forwarding the frame to the network. In addition, they add a 30-byte header on the outbound frame.
The SAGE ASIC design is similar to that of the SAINT, but without the 10/100-Mbps Ethernet MAC-layer functionality. It is used for non-Ethernet applications, such as the FDDI module, ATM LANE module, route switch module, Token Ring, and the NMP on the Supervisor Engine.
The 48-port Ethernet switching module for the Catalyst 5000 series switch uses a new ASIC called Bodega. Each Bodega ASIC supports 24 10-Mbps Ethernet ports. The Bodega ASIC uses a shared-memory buffer, and buffer space is allocated dynamically according to packet size. The Bodega ASIC helps reduce the cost of high-performance, switched networks.
The SAMBA ASIC is located on both line modules and the Supervisor Engines. It provides arbitration to the switching bus both among the ports and among the line modules in the chassis. Its dual usage is accomplished by strapping on external device pins.
The SAMBA can be strapped in either master mode or slave mode. The SAMBA in master mode is located on the Supervisor Engine, and the SAMBA in slave mode is on the line module and the Fast Ethernet ports on the Supervisor Engine. A master can support up to 13 line cards and a slave can support up to 48 ports on a single device.
An arbitration process starts with ports on the line cards requesting access to the bus through the slave SAMBA, which forwards the requests to the master SAMBA on the Supervisor Engine. The slave SAMBA then waits for permission from the master SAMBA before issuing grants to the ports.
Hardware broadcast/multicast suppression capability is available when SAMBA is in slave mode. This capability allows SAMBA to monitor the number of broadcast packets going through every port on the line card and removes the entire broadcast packet if the total number of broadcast packet words reaches a threshold within a given time period. The threshold number and the time period are both initialized by the CPU.
The EARL is a custom Catalyst 5000 series switch component that is similar in its function to the learning bridges or content-addressable memory (CAM) used on other systems. The EARL automatically learns source MAC addresses and associated VLANs, and saves them in a RAM address table with VLAN and port information. You can also configure static entries in the EARL table. The EARL stores up to 16,000 addresses.
The EARL then uses the learned entries to perform lookup operations on destination addresses (DAs) to get VLAN and port information to be used to direct the frame. It stores these addresses for 300 seconds (5 minutes) by default, or 60 to 1200 seconds (1 to 20 minutes), if so configured by the user.
Local target logic (LTL) on each line module helps the EARL to find the destination port(s) for each frame. Index values generated by the EARL can select a single port, multiple ports, or all ports in a VLAN. LTL memory is segmented to support unicast, multicast, and flood cases.
The color blocking logic (CBL) blocks traffic that is part of a VLAN different from that configured on a port from coming into or going out of that port and assists spanning tree in blocking ports to prevent loops.
The PHOENIX gate array allows you to continue to use all the Catalyst 5000 switch modules, while also providing a migration path to switching modules that are capable of 3.6-Gbps switching. It provides intelligent forwarding of packet data from one bus to another, which allows the Catalyst 5500 switch to increase the bandwidth support to 3.6 Gbps by interconnecting three 1.2-Gbps switching buses.
The PHOENIX ASICs on the Supervisor Engine III are a gigabit bridge used to create the crossbar fabric interconnect between the three 1.2-Gbps buses. Most Catalyst 5000 series switch modules connect only to one bus in the fabric. However, some modules (such as the Supervisor Engine III itself) connect to all three buses.
To understand how the major components that comprise the architecture of the Catalyst 5000 series of switches work, you need to follow the path of an Ethernet frame through the switch. When an Ethernet frame arrives at a port, the port DMA controller stores it in its receive buffer. The port fully receives, buffers, and checks the FCS for the frame, adds the 12-byte frame header containing the port number and VLAN number, and increments the Ethernet Management Information Base (MIB) and Remote Monitoring (RMON) counters before storing the frame in the frame buffer and requesting access to the switching bus to forward the frame.
After the entire frame is received and stored in the frame buffer, the SAINT ASIC posts a request to the bus arbiter to transmit the frame across the high-speed switching bus. The Supervisor Engine performs the necessary bus arbitration and grants bus access to the line module as soon as it becomes available, using a round-robin method. The local arbiter on the line module then uses a round-robin method to grant each port permission to transmit one frame at the priority level being serviced.
The bus arbiter issues a grant to the local arbiter, which, at the appropriate time, signals the SAINT ASIC to initiate the data transfer. The frame that is stored in the buffer is then transmitted across the high-speed switching backplane by the DMA controller.
As the frame is transmitted across the switching backplane, all ports receive the frame and store it in their frame output buffers.
As with all bus architectures, because all ports on the bus receive the frame, there is no need for multiple copies of broadcast and multicast frames.
The last step in the frame flow process varies, depending on these factors:
Unicast and Multicast
When the EARL, in conjunction with LTL and CBL, identifies a specific port or set of ports as the destination, it instructs the SAINT ASICs on the destination ports through the control bus to continue to receive the frame and subsequently transmit it out its port. The SAINT ASICs on the other ports flush their buffers.
Broadcasts and Unknown Destination Addresses
If the destination address is unknown to the EARL, or is a broadcast or multicast address (where multicasting is not configured), the frames are forwarded to all ports on the same VLAN as the source address.
If the EARL determines that the frame does not require switching into the system, it issues a flush to terminate frame transfer on the bus from the source interface and causes all ports to flush their buffers. However, only a few bits of the packet are stored in each buffer before the EARL orders the flush. The EARL then flushes the buffer of only that packet.
If the destination interface has no buffer to accommodate a frame, it issues a retry to the source. The source terminates the frame transfer and retries the frame at a later time.
The EARL Plus (EARL+) is a logic based on the original EARL used for Ethernet switching. The Plus part of EARL is the added logic required to perform Token Ring switching in the Catalyst 5000. These EARL changes are required by ISL for Token Ring and by the Token Ring card for the Catalyst 5000.
As data traverses the switching bus of the Catalyst 5000 backplane, EARL+ monitors the type field of the switching bus header. If the packet is not of type B"0000", then EARL performs its current Ethernet switching function. If the packet type is B"0001," then EARL+ latches data from an additional switching bus header word (48 bits) added for Token Ring.
The SAGE and SAINT ASICs take the 48-bit MAC address along with the 10 bits of VLAN information and encode them into 15 bits. The 15 bits are presented to EARL and used to access a lookup table that contains the actual MAC address and its associated port index and VLAN information.
Although Token Ring addresses are noncanonical and Ethernet addresses are canonical, there can never be a match conflict within the EARL+ lookup tables because a Token Ring address never has the same VLAN ID as an Ethernet address. The VLAN ID acts as an additional qualifier for the EARL+ lookup/compare step.
The Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers.
Refer to Cisco Technical Tips Conventions for information on conventions used in this document.