Guest

Cisco Catalyst G-L3 Series Switches

Hardware Troubleshooting for Catalyst 2948G-L3/4908G-L3 Series Switches

Cisco - Hardware Troubleshooting for Catalyst 2948G-L3/4908G-L3 Series Switches

Document ID: 18802

Updated: Aug 30, 2005

   Print

Introduction

This document helps troubleshoot potential hardware issues with the Catalyst 2948G-L3 and the Catalyst 4908G-L3 switches. Both the Catalyst 2948G-L3 and the Catalyst 4908G-L3 are wire-rate Layer 3 (L3) switches. Because both switches share the same architecture and software, the remainder of this document applies to both devices, unless specifically stated otherwise.

Before You Begin

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Prerequisites

Readers of this document should be knowledgeable of the following:

Preparation for Troubleshooting Hardware on Catalyst Switches

Many hardware problems encountered during field installations or during normal operation could be prevented by a thorough product overview ahead of time.

This document covers the following important information:

  • What are the installation procedures and technical specifications for 2948G-L3 and 4908G-L3 switches? See the Switch Chassis Information sections.

  • How do I configure a management interface? How do I back up my configuration? See the Switch Management section.

  • Are there any software requirements I should be aware of? See the Software Considerations section.

Components Used

The information in this document applies to:

  • all Catalyst 2948G-L3

  • Catalyst 4908G-L3 switches

The information presented in this document was created from devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If you are working in a live network, ensure that you understand the potential impact of any command before using it.

Understanding the Catalyst 2948G-L3 and 4908G-L3 Architecture

These Catalyst switches achieve high performance by implementing distributed Cisco Express Forwarding (CEF) on specialized hardware network processor application-specific integrated circuits (ASICs). For more information on CEF, see the Overview of Layer 3 Switching and Software Features documentation.

Catalyst 2948G-L3 and 4908G-L3 switches consist of three major components:

  • high performance network processor ASICs - data plane

  • switching fabric - data plane

  • route switch processor (RSP) - control plane

The data plane and the control plan are separated, which means that the majority of the packets should be switched by network processor ASICs. The RSP needs only to see control protocols packets such as Routing Information Protocol (RIP), Open Shortest Path First (OSPF), Cisco Discovery Protocol (CDP), Spanning Tree Protocol (STP), and so on.

The switching fabric transfers packets between ports as well as to and from the route switch processor.

The RSP has the following responsibilities:

  • Running control plane protocols, building the routing table, compiling the routing table onto the Forwarding Information Base (FIB) table, and downloading the resulting FIB table to network processors

  • Collecting statistics, monitoring, controlling, and downloading appropriate microcode to network processors

  • Providing a user interface to the device for administration, configuration, and maintenance

There are two types of network processors used in the Catalyst 2948G-L3: one is serving four adjacent Fast Ethernet ports (E-PIF) and the other is serving one Gigabit Ethernet port (X-PIF). The Catalyst 2948G-L3 has twelve E-PIFs and two X-PIFs. The Catalyst 4908G-L3 has eight X-PIFs and does not have any E-PIFs. This is useful to know when troubleshooting network processor-related hardware issues.

It is also important to understand that both the forwarding decision (that is, the decision where to send the packet) and the rewrite decision is done on the incoming side, which is before the packet is sent over the switching fabric.

Packets can be forwarded within the device along two paths:

  • the hardware switching path, which is the fast path (packets are switched by network processors)

  • the software switching path, which is the slower path (packets are switched by the RSP CPU)

When packets are switched by the fast path, the switch can forward traffic at up to line rate. The software path is meant as a fallback mechanism, when the fast path is not capable of switching the packet or if the packet is destined to the device itself (control plane packets). IP, IP multicast, IPX, and bridging are supported on the fast path. AppleTalk protocol packets are always switched by software path. While the software path is more flexible, it is considerably slower than the fast path. Every packet forwarded by the software path requires CPU attention, meaning that the software path scales only as high as the CPU permits. Constantly overloading the CPU might result in dropping control plane protocol packets, which could potentially cause network instability.

Four specialized ASICs comprise the switching fabric. Each interface (or group of interfaces) of the switch is serviced by one of switching fabric ASICs. See the table below for mapping between the interfaces and fabric ASICs. It is useful to know this mapping in order to be able to isolate the problem down to one of the fabric ASICs.

Interface to Switching Fabric ASIC Mapping

Catalyst 2948G-L3 Catalyst 4908G-L3 Switching Fabric ASIC
FastEthernet1-16 GigabitEthernet1-2 0
FastEthernet17-32 GigabitEthernet3-4 1
FastEthernet33-40 GigabitEthernet49 GigabitEthernet5-6 2
FastEthernet41-48 GigabitEthernet50 GigabitEthernet7-8 3

Inside the switch, the incoming packet will be received by the incoming port MAC circuitry, network processor microcode of the incoming interface will perform forwarding lookups, then the frame will be enqueued to the switching fabric to be sent over to either the RSP (for software path forwarding if network processor lookup will not produce a result, or for processing if the frame is a control protocol frame) or the outgoing interface processor.

All the forwarding lookups are done on the incoming port network processor. Frame rewrite (in case of routing) is done on the ingress of the switching fabric. Final encapsulation before sending the packet out to the network is done on the egress interface processor.

Troubleshooting Boot Issues

The booting process of the Catalyst 2948G-L3 and the Catalyst 4908G-L3 is very similar to the booting process of other Cisco IOS routers. One issue is that prior to bootstrap version 12.0(7)W5.15a, xmodem download of the IOS was not supported. Therefore, if there is no IOS image in the bootflash: and the device is in ROMMON mode, then there is no way to recover and the complete device needs to be replaced. ROMMON code is not upgradable in the field. Beginning with bootstrap version 12.0(7)W5.15a, xmodem download is supported.

To find the bootstrap version of the switch in IOS mode, issue a show version command. An example is below:

cat2948g-l3#show version
Cisco Internetwork Operating System Software 
IOS (tm) L3 Switch/Router Software (CAT2948G-IN-M), Version 12.0(14)W5(20)  RELEASE SOFTWARE 
Copyright (c) 1986-2001 by cisco Systems, Inc.
Compiled Thu 01-Mar-01 17:55 by integ
Image text-base: 0x60010928, data-base: 0x605EA000

ROM: System Bootstrap, Version 12.0(7)W5(15) RELEASE SOFTWARE 

cat2948g-l3 uptime is 6 days, 3 hours, 54 minutes
System restarted by power-on
System image file is "bootflash:cat2948g-in-mz.120-14.W5.20.bin"

cisco Cat2948G (R5000) processor with 49152K/16384K bytes of memory.
R5000 processor, Implementation 35, Revision 2.1
Last reset from power-on
48 FastEthernet/IEEE 802.3 interface(s)
2 Gigabit Ethernet/IEEE 802.3z interface(s)
121K bytes of non-volatile configuration memory.
16384K bytes of processor board Boot flash (Read/Write)

Configuration register is 0x2102

For more information on boot issues, please see the following documents:

Troubleshooting Platform-Specific Issues

This section contains troubleshooting examples that use the output of some low-level commands. Some parts of the command output of these low-level commands are for engineering purposes only, and will not be useful to you. Please use these commands only as indicated by this document.

Begin troubleshooting by inspecting the device log. To access the log, issue a show log command. Within the log, look for messages issued at around the time the problem has occurred or for other syslog messages indicating possible a platform-related problem. An example is below:

Dec 26 14:21:11: xpif_port_read_mac: Timeout while waiting for response message
Dec 26 14:21:23: xpif_port_read_mii: Timeout for response message

If the problem is with connectivity across a specific port, then port troubleshooting needs to be performed, as described below:

  1. Check if the corresponding interface is showing "up/up" status in show interface command output:

    cat2948g-l3#sh int fastEthernet1 
    FastEthernet1 is up, line protocol is up 
      Hardware is epif_port, address is 0001.4391.8c07 (bia 0001.4391.8c07)
      Internet address is 10.48.72.124/24
      MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, rely 255/255, load 1/255
      Encapsulation ARPA, loopback not set, keepalive set (10 sec)
      Auto-duplex, Auto Speed, 100BaseTX
      ARP type: ARPA, ARP Timeout 04:00:00
    [output truncated]
    
    

    If the port is administratively enabled, but the protocol is down or the interface is flapping, check the cable to ensure it is not faulty, and check if the connected device has its interface enabled.

    Another option is to try another cable and/or Gigabit Interface Converter (GBIC), for Gigabit Ethernet ports, and try to connect to a different port on the switch. If the port is a FastEthernet port, it makes sense to try to connect to a port that is at least four ports away from the current port. This is to make sure that the new port is serviced by a different ASIC. For example, if a link does not come up with port FastEthernet1, try port FastEthernet5.

    If the link parameters are not auto-negotiated, verify if these parameters are configured consistently on both sides; this applies to both Fast Ethernet and Gigabit Ethernet ports.

    For more information on troubleshooting link issues, refer to the "Auto-Negotiation on Catalyst IOS Switches" section of the following document:

    Configuring and Troubleshooting Ethernet 10/100Mb Half/Full Duplex Auto-Negotiation

    In summary, look for the following port problems:

    • Is connectivity fine across all ports except one or a group of adjacent ports?

    • Are failing ports configured exactly same way as working ports?

    • Do the ports still malfunction after a device reload?

    If the above items are all true, then it is likely that those failing ports are experiencing a hardware problem.

  2. If the interface status is reported "up/up," but still no packets go through, check the output of the sh controllers interface command.

    The important fields are indicated by bold in the example below. This command output needs to be collected several times to see if marked counters are increasing:

    cat2948g-l3#sh controllers fastEthernet1 
    IF Name: FastEthernet1    
    Port Status UP
    
    Slicer registers
    SMDR 0x0060 SSTR 0x0000 SSMR 0x0000 EVER 0x0000 
    SIMR 0x0000 MBXW 0x0000 MBXR 0x0000 SPER 0xF000 
    
    MAC registers
    CMCR : 0x00000433 CMPR : 0x140A0E60
    
    [output skipped]
    
    Auto-Negotiation Status        Complete
                     Speed         10Base-X    
                     Duplex        Half
    Link Partner Auto-Negotiation  Not Capable
    
    
    Counters :
    MR1  1831782732 MR2  59044      MR3  37165      MR4  25273      MR5  33652     
    MR6  55259      MR7  13743      MR8  0          MR9  0          MR10 0         
    MR11 0          MR12 3449       MR13 12216      MR14 11863      MR15 0         
    MR16 0          MR17 0         
    SR1  11670524   SR2  38472      SR3  0          SR4  0          SR5  0          
    MT1  2901463    MT2  95         MT3  5          MT4  0          MT5  9062      
    MT6  0          MT7  0          MT8  97         MT9  9062       MT10 3         
    MT11 0          MT12 0          MT13 0          MT14 9          MT15 26        
    MT16 0         
    ST1  6055469    ST2  62809      IM   0

    The explanation of MR, MT, SR, and ST counters are given in the Appendix A - show controller interface Command Counter Description section of this document.

    The counters in bold should increase over time. These counters indicate the port MAC layer circuitry is receiving packets from the network (MR1) and sending them towards the switching fabric (ST2). Counters responsible for receiving packets from the switching fabric and sending them out to the network are SR2 and MT1, respectively. Note that MR1 and MT1 represent byte counters, not packet counters.

    If SR4 and/or SR5 counters are increasing or are non-zero on any interface, there is likely a hardware problem with the device.

    Another set of counters is maintained by the network processor microcode. These counters can be displayed by the sh epc counters command:

    cat2948g-l3#sh epc counters
    Interface   Input     Runts Giants  Input       CRC  Frame Output      Output
        State   Packets                 Errors                 Packets     Errors
    -----------------------------------------------------------------------------
    F1      U   12538448    0     0     0           0     0     9361        0     
    F2      D   0           0     0     0           0     0     3           0     
    
    [output skipped]
    
    
    F47     D   11040       0     0     0           0     0     8643        0     
    F48     D   0           0     0     0           0     0     0           0     
    G49     D   0           0     0     0           0     0     1           0     
    G50     D   0           0     0     0           0     0     0           0     
    -----------------------------------------------------------------------------
    AD - Admin Down, D - Down, F - Fail, U - Up
    

    This command output should be inspected several times. Over time, packet counters should increase and error counters should not increase. Also take notice of the interface state. If the state indicated is F, the port may possibly be in a "stuck" condition. See below for an explanation of "stuck" ports.

    If the sh controllers interface command output contains a message similar to:

    xpif_port_read_mac: Timeout while waiting for response message
    
    or 
    
    
    xpif_port_read_mii: Timeout for response message
    

    then the network processor may not be responding to the CPU. In such a case, counters cannot be retrieved. If the interface is not responding to the CPU it is likely not able to perform its functions correctly and considered "stuck." In Cisco IOS version 12.0(10)W5.18g and later there is a detection and recovery mechanism implemented for stuck ports.

    By default, stuck ports will be shut down. There is a configuration command to automatically reset a stuck port attempting to restore the connectivity via the port. To enable this option, in global configuration mode, use the epc port-reload command. This is global command and affects all ports. Note that reloading a stuck port will affect all ports serviced by the same ASIC.

    Shutting down the port and then re-enabling it will not fix the port-stuck issue. A stuck port can be recovered by either configuring the epc port-reload command prior to the port being stuck or by reloading the device.

    Note: The epc port-reload command is not supported on the Catalyst 2948G-L3 Gig50 and the Catalyst 4908G-L3 Gig8 interfaces.

    Another way to see if the port is possibly stuck is to use the sh epc status command.

    cat2948g-l3#sh epc status
    Status of FastEthernet1: OK
    Status of FastEthernet2: OK
    Status of FastEthernet3: OK
    Status of FastEthernet4: OK
    Status of FastEthernet5: not OK
    Status of FastEthernet6: OK
    Status of FastEthernet7: OK
    Status of FastEthernet8: OK
    
    [output truncated]

    Interface FastEthernet5 is likely to be stuck. When the interface is stuck, this command might freeze your Telnet/console session for up to 15 minutes. You can use the sh epc status command output along with the sh epc queuing command output to see the packets queued to the interface, which allows you to verify whether the port is stuck. An example is below:

    2948gl3_2#sh epc queuing
    np54[0].Stream[0x007D].ECNT. =   0x0002
    np54[0].Stream[0x0087].ECNT. =   0x0002
    

    The output above shows that ports connected to Streams hex 7D (decimal 125) and Stream hex 87 (decimal 135) have two cells queued.

    To determine which interface is associated with a given stream, use the following hidden command:

    2948gl3_2#sh mmc np5400 streams 135 
    !-- This is the decimal value for the Stream ID
    
    
    Port 11 - NP54:0, stream:0x0087, local port:14, sched_id:2, ovc_header:0x0005, 
    OVC_id:0x005E
    np54[0].Stream[0x0087].ECNT. =   0x0002

    The output above shows Port 11 or interface FastEthernet 11 is attached to this stream. If the queue count (0x0002 above) is close to 800 cells (0x0320, maximum queue size) and not decrementing, this could indicate a port stuck condition.

    If a certain interface or group of interfaces become stuck repeatedly, try moving the configuration and connection from such an interface to another interface serviced by a different ASIC and see if the issue will follow the connection. If the new port will not become stuck, then there is a good possibility of a hardware problem with the former port.

  3. Verify with the show hardware command that the microcode for the EPIFs (ASICs for the 10/100 ports) and XPIFs (ASICs for the GigabitEthernet ports) has downloaded correctly.

    Ucode Image      : EPIF_UCODE_RUNTIME
    Port Phy Setup
         Port1 :DONE      Port2 :DONE      Port3 :DONE      Port4 :DONE
    

    If DONE is not indicated in the output, try rebooting the switch.

    As a summary for troubleshooting port issues: it is necessary to establish whether the problem is specific to a port, port configuration, and traffic pattern/network event. If failure is consistent to the port, it is likely that the port is experiencing a hardware problem.

Troubleshooting Switch Fabric Issues

To verify if the switching fabric is experiencing hardware problems, consider the output of the commands described in this section. Please note that some of the commands described below are classified as hidden commands. A hidden command is one that cannot be parsed with a "?" and the Tab key cannot be used to auto-complete the command. Hidden commands are not documented and some of the output is used strictly for engineering purposes. Command output not relevant to troubleshooting will not be described. The sh mmc np statistics hidden command is presented below:

cat2948g-l3#sh mmc np statistics
Streams opened  :   0x0191
Flows created   :   0x01AF
-----------------------------------------------------------
|                    || NP54 0 | NP54 1 | NP54 2 | NP54 3 |
-----------------------------------------------------------
| Flows per module   || 0x0092 | 0x0080 | 0x004B | 0x0052 |
-----------------------------------------------------------
| Discarded frames   || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------
| Invalid frames     || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------
| Bad frames         || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------
| Cell Stack Top     || 0x9905 | 0x8428 | 0xCE6F | 0x502D |
-----------------------------------------------------------
| Cell Stack Bottom  || 0x9904 | 0x8427 | 0xCE6E | 0x4D1F |
-----------------------------------------------------------
| Free cells         || 0xFDFF | 0xFFFF | 0xFFFF | 0xFFFF |
-----------------------------------------------------------
| Frame Stack Top    || 0x035A | 0x05EE | 0x06AB | 0x0536 |
-----------------------------------------------------------
| Frame Stack Bottom || 0x0357 | 0x05ED | 0x06AA | 0x0788 |
-----------------------------------------------------------
| Free frames        || 0x07FE | 0x07FE | 0x07FE | 0x07FE |
-----------------------------------------------------------
| Frames to remove   || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------
| Invalid stream id  || 0x0000 | 0x0000 | 0x0000 | 0x0000 |

If the "Free frames" output becomes 1, than there is likely to be a port stuck issue. For troubleshooting stuck ports, refer to the Troubleshooting Platform-Specific Issues section of this document.

The output Discarded frames indicates that packets are being discarded because of lack of resources. Packets can be discarded due to the occurrence of any of the following conditions: data memory full, data memory almost full, and stream full.

The Bad frames counter is incremented for every packet marked "bad" by the EPIF or XPIF because of an error in the header; for example, a CRC error.

The Invalid frames counter represents the number of packets that are discarded because of an Invalid Stream ID. In some momentary conditions, such as removing a port from a bridge group, this counter can increase but does not necessarily indicate a problem. If, however, the Invalid frames counter increments repeatedly over time, save the output and contact the Cisco TAC.

The sh mmc np indications hidden command is presented below:

cat2948g-l3#sh mmc np indications
-----------------------------------------------------------------
|                          || NP54 0 | NP54 1 | NP54 2 | NP54 3 |
-----------------------------------------------------------------
| CPU cells received       || 0xFE5ED4A | 0x0001 | 0x0001 | 0x0001 |
-----------------------------------------------------------------
| Marker send count        || 0x0002 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Marker done interrupts   || 0x0002 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Invalid frames           || 0x0001 | 0x0001 | 0x0001 | 0x0001 |
-----------------------------------------------------------------
| ICP Parity Err bit 0     || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| ICP Parity Err bit 1     || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| ICP Parity Err bit 2     || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 00    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 01    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 02    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 03    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 04    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 05    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 06    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 07    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 08    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 09    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 10    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Mem Parity Err bit 11    || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Group 0 full             || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Group 1 full             || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Group 2 full             || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Group 3 full             || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Group 4 full             || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Group 5 full             || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Group 6 full             || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Group 7 full             || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Frame Stack Almost Empty || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Frame Stack Empty        || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Memory Almost Full       || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------
| Memory Full              || 0x0000 | 0x0000 | 0x0000 | 0x0000 |
-----------------------------------------------------------------

In the above output example, if any parity errors are considerably above 0, it is likely there is a hardware problem with the device.

Troubleshooting Control Plane Issues

The following commands can help in identifying hardware-related problems affecting the control plane of the device:

Here is a list of the different types of control plane/data plane traffic:

  • Control Traffic - MAC Learn/Age, L2/L3 Statistics

    • Transmit via FX1000 path

    • Receive via 8 bit path

  • Data Traffic - BPDUs, Routing Updates, To Me Traffic

    • Transmit via FX1000 path

    • Receive via FX1000 path

  • Low Priority Data - Unroutable Traffic

    • Receive via 8 Bit path

Below is an example of the sh mmc np fx1000 command:

cat2948g-l3#sh mmc np fx1000

Interface FX1000(idb 0x608DF920)
Hardware is WISEMAN 2.1, network connection mode is force
  network link is up
  loopback type is none
GBIC is missing
FX1000 Statistics :(PA0)
Error Counters:
  CRC errors           0             Symbol errors        3           
  Missed Packets       0             Single Collision     0           
  Excessive Collision  0             Multiple Collision   0           
  Late Collision       0             Total TX Collisions  0           
  TX Defer count       0             Receive Length Error 0           
  Sequence Errors      0             XON Receive count    0           
  XON Transmit count   0             XOFF Receive count   0           
  XOFF Transmit count  0             FC RX Unsupported    0           
Receive Counters:
  Packet RX 64         0             Packet RX 65-127     100246      
  Packet RX 128-255    27601         Packet RX 256-511    84866       
  Packet RX 512-1023   3             Packet RX 1024-1522  0           
  RX Good Packet       212716        RX Broadcast         0           
  RX Multicast         0             Good Packet TX       0           
  Good Octets RX.H     0             Good Octets RX.L     44376731    
  RX No Buffers        0             RX Undersize         0           
  RX Fragment          0             RX Oversize          0           
  RX Octets High       0             RX Octets Low        44376731    
  TOTAL RX PACKETS     212716      
Transmit Counters:
  Packet TX 64         9             Packet TX 65-127     301         
  Packet TX 128-255    3120682       Packet TX 256-511    88193960    
  Packet TX 512-1023   554212        Packet TX 1024-1522  0           
  Good Octets TX.H     5             Good Octets TX.L     656639513   
  TX Octets High       5             TX Octets Low        656639513   
  TX Broadcast         0             TX Multicast         0           
  TOTAL TX PACKETS     91869164    

[output truncated]

In the above output, if any of the error counters are constantly increasing at a considerable rate, then there is a potential hardware problem. Note that TX and RX packet counters should increase over time, which indicates the CPU inband port is receiving and transmitting the packets.

Below is an example of the sh mmc np5400 8 command:

cat2948g-l3#sh mmc np5400 8 
8-bit Path Statistics:
MAC Learn IPC Dropped              =                                0
Non MAC Learn IPC Dropped          =                                0
Cell Dropped                       =                          1768714
IPC Received                       =                         64509090
Data Received                      =                          7007941
Download Resp Received             =                               28
Invalid Packet Received            =

In the above output, if you are seeing a non-zero value in the "MAC Learn IPC Dropped field," this indicates an inconsistency between the L2 table in the CPU and what is in the CAM/TCAM. Issue a clear bridge command as a remedy.

Note: The clear bridge command action results in temporary flooding as L2 entries are re-learned and re-populated.

Also note that the "Cell Drop" field is the total number of cells dropped on the 8-bit path, where low priority data is received from switch fabric. These could be drops of low priority data traffic such as unrouteable packets.

Below is an example of the sh epc lsipc command:

cat2948g-l3#sh epc lsipc 
LSIPC requested: Total: 91897715 Mlet: 91897715
Sent:Total: 91897715 Mlet: 91897715 No-resp: 91896608 Resp-required: 1107
Broadcast IPCs:Requested: 25 Sent: 25
  Queued: 25 Current qsize: 0 Max qsize-reached: 10

Received:Total: 105756307 Unsolicited: 91892343 Response: 1107
Recv Q size: 0

LSIPC Failures:
Toobig: 0 Memory Fail: 0 Packet fail: 0 Invalid VC: 0
Invalid resp: 0 Retries: 0 Timeouts: 0 Ack timeouts: 0
Bcast: Failed: 0 Pkt failed: 0 enq failed: 0 discard: 0
Unicast: Enq failed: 0

In the above output, the LSIPC failures are important to monitor; if retries or timeouts are increasing, there is a possible port stuck issue. For troubleshooting stuck ports, refer to the Troubleshooting Platform-Specific Issues section of this document.

Because these devices run Cisco IOS, all the software-related Cisco IOS problems can be diagnosed as they are done on any Cisco IOS routers. Refer to the following documents for general Cisco IOS troubleshooting:

When troubleshooting memory problems, note that network processor and switching fabric ASICs have their own dedicated buffers for fast path traffic switching. Only control plane protocol packets and software path switched packet use Cisco IOS buffers.

Collecting Information for the Cisco TAC

If you need to escalate your issue to the Cisco Technical Assistance Center (TAC), then collecting the following information will aid the TAC in providing you with a speedy resolution.

  • Collect sh tech and sh log command output from the relevant device at the time the problem was happening. Reloading the device prior to collecting the information about the problem often clears all the traces of failure and makes it difficult to diagnose and troubleshoot.

  • A detailed description of the problem including affected ports, device addresses, whether the issue is consistent or intermittent, if it can be reproduced at will, a network topology map (if relevant), and device access details (if remote access is available) are all extremely helpful.

  • A description of troubleshooting you have already completed.

Appendix A - show controller interface Command Counter Description

Counter Group/Name Description
MAC Receive Counters

MR1
MR2
MR3
MR4
MR5
MR6
MR7
MR8
MR9
MR10
MR11
MR12
MR13
MR14
MR15
MR16
MR17
MR18
MR19
MR20
MR21
Bytes received
Frames received of size 64
Frames received of size 65-127
Frames received of size 128-255
Frames received of size 256-511
Frames received of size 512-1023
Frames received of size 1024-1518
Frames received of size 1519-1530
Oversize (>1530, good FCS) frames received
Giant (>1530, bad FCS) frames received
Undersize (<64, good FCS) frames received
Runt (<64, bad FCS) frames received
Unicast frames received
Multicast frames received
Broadcast frames received
RX SYNC loss errors
Overrun errors
FCS errors
RX Delimiter Sequence errors
GMAC drop count
Symbol errors
MAC Transmit Counters

MT1
MT2
MT3
MT4
MT5
MT6
MT7
MT8
MT9

Bytes transmitted
Frames transmitted of size 64
Frames transmitted of size 65-127
Frames transmitted of size 128-255
Frames transmitted of size 256-511
Frames transmitted of size 512-1023
Frames transmitted of size 1024-1518
Frames transmitted of size 1519-1530
Oversize frames transmitted
Slicer Receive Counters

SR1
SR2
SR3
SR4
SR5
Cells received from switching fabric
Frames received from switching fabric
header sequence errors received
FCS_errors received from switching fabric
length errors from switching fabric
Slicer Transmit Counters

ST1
ST2

Cells sent to switching fabric
Frames sent to switching fabric

Related Information

Updated: Aug 30, 2005
Document ID: 18802