Table of Contents
Troubleshooting Ethernet, ATM Uplink, and POS Uplink Interfaces
Troubleshooting General Ethernet Interface Problems
Troubleshooting 10/100 Ethernet Interface Modules
Troubleshooting Gigabit Ethernet Interface Modules
Troubleshooting ATM Uplink with Enhanced Gigabit Ethernet Interface Modules
Troubleshooting Packet-over-SONET Uplink with Enhanced Gigabit Ethernet Interface Modules
Troubleshooting ACL Daughter Card
Troubleshooting Ethernet, ATM Uplink, and POS Uplink Interfaces
This chapter provides troubleshooting information about connectivity and performance problems in the Ethernet, ATM uplink, and POS uplink physical interfaces of a Layer 3 enabled ATM switch router.
The chapter includes the following sections:
Troubleshooting General Ethernet Interface Problems
You might see problems of cell transmission through the switch router, detected by a buildup of cells on an internal virtual channel (VC). These problems occur for the following reasons:
Troubleshooting Switch Card Failures
A switch processor can have a cell stuck problem in internal virtual channels (VCs), resulting from timing issues in the hardware and software on the Catalyst 8540 CSR. You might see more than one port affected on one or more interface modules. Online insertion and removal (OIR) of the interface module will temporarily fix the problem.
Follow these steps to troubleshoot cell stuck problems:
Step 1 Under the lightest possible traffic, issue the show switch fabric command on the switch router to clear the counters.
Switch#
show switch fabric
MMC Switch Fabric (idb=0x61DD8F0C)
Key: Rej. Cells - # cells rejected due to lack of resources
Inv. Cells - # good cells that came in on a non-existent conn.
Mem Buffs - # cell buffers currently in use
RX Cells - # rx cells (16-bit)
TX Cells - # tx cells (16-bit)
Rx HEC - # cells Received with HEC errors
Tx PERR - # cells with memory parity errors
MSC# Rej. Cells Inv. Cells Mem. Buffs Rx Cells Tx Cells Rx HEC Tx PErr
----- ----------- ------------ ----------- ----------- ---------- ---------- ----------
MSC 0: 0 110018 0 0 0 0 0
MSC 1: 0 231044 0 0 0 0 0
MSC 2: 0 234283 0 0 0 0 0
MSC 3: 0 232492 0 0 0 0 0
MSC 4: 0 242004 0 0 0 0 0
MSC 5: 0 120995 345 0 0 0 0
MSC 6: 0 111466 0 0 0 0 0
MSC 7: 0 334398 0 0 0 0 0
Rejected Cells: 0
Invalid Cells: 1616700
The show switch fabric command clears the counters after it displays. Entering the command again shows the current activity on the switch router.
Step 2 Issue the show switch fabric command again to show new activity.
Switch#
show switch fabric
MMC Switch Fabric (idb=0x60CF1788)
Key: Rej. Cells - # cells rejected due to lack of resources
Inv. Cells - # good cells that came in on a non-existent conn.
Mem Buffs - # cell buffers currently in use
RX Cells - # rx cells (16-bit)
TX Cells - # tx cells (16-bit)
Rx HEC - # cells Received with HEC errors
Tx PERR - # cells with memory parity errors
MSC# Rej. Cells Inv. Cells Mem. Buffs Rx Cells Tx Cells Rx HEC Tx PErr
----- ----------- ------------ ----------- ----------- ---------- ---------- ----------
MSC 0: 2189 6 14177 0 0 0 0
Rejected Cells: 2189
Invalid Cells: 58
Look at the values in the Rejected Cells and Invalid Cells fields. Note that the Rejected Cells and Invalid Cells field counters are increasing. This means there might be a problem in the switch fabric.
Step 3 Verify that no ports are involved by issuing the show epc queuing and show epc status commands.
INT X-INT VCI QCNT VCI QCNT
Status of GigabitEthernet0/0/0: OK
Status of GigabitEthernet0/0/1: OK
Status of GigabitEthernet1/0/0: OK
Status of GigabitEthernet1/0/1: OK
Status of GigabitEthernet2/0/0: OK
Status of GigabitEthernet2/0/1: OK
Status of GigabitEthernet3/0/0: OK
Status of GigabitEthernet3/0/1: OK
Status of GigabitEthernet9/0/0: OK
Status of GigabitEthernet9/0/1: OK
Status of GigabitEthernet10/0/0: OK
Status of GigabitEthernet10/0/1: OK
Status of GigabitEthernet11/0/0: OK
Status of GigabitEthernet11/0/1: OK
Status of GigabitEthernet12/0/0: OK
Status of GigabitEthernet12/0/1: OK
If the queues are empty and all of the ports show OK status, then the problem is not the ports, it is the switch processor.
You can resolve this problem by upgrading your system software image to release Cisco IOS Release version 12.0(4a)WX5(11) or later, by replacing the switch processors, or by doing both.
Troubleshooting Port Stuck Problems
If one or more Fast Ethernet or Gigabit Ethernet ports are not transmitting cells, then the failure might be a port stuck problem.
Follow these steps to troubleshoot a port stuck problem:
Step 1 Use the show switch fabric command to display the activity in the switch processors.
 |
Note Be sure to use the show switch fabric command during the lightest possible traffic
conditions because actual traffic might be using the memory buffers.
|
Switch#
show switch fabric
MMC Switch Fabric (idb=0x60CF1788)
Key: Rej. Cells - # cells rejected due to lack of resources
Inv. Cells - # good cells that came in on a non-existent conn.
Mem Buffs - # cell buffers currently in use
RX Cells - # rx cells (16-bit)
TX Cells - # tx cells (16-bit)
Rx HEC - # cells Received with HEC errors
Tx PERR - # cells with memory parity errors
MSC# Rej. Cells Inv. Cells Mem. Buffs Rx Cells Tx Cells Rx HEC Tx PErr
----- ----------- ------------ ----------- ----------- ---------- ---------- ----------
MSC 0: 389023 7896 14177 0 0 0 0
MSC 1: 0 32709 2070 0 0 0 0
MSC 6: 0 6170 1351 0 0 0 0
MSC 7: 0 9624 1280 0 0 0 0
Rejected Cells: 389023
Invalid Cells: 56399
The show switch fabric command clears the counters after it displays. Entering the command again shows the current activity on the switch router.
Step 2 Enter the show switch fabric command again.
Switch#
show switch fabric
MMC Switch Fabric (idb=0x60CF1788)
Key: Rej. Cells - # cells rejected due to lack of resources
Inv. Cells - # good cells that came in on a non-existent conn.
Mem Buffs - # cell buffers currently in use
RX Cells - # rx cells (16-bit)
TX Cells - # tx cells (16-bit)
Rx HEC - # cells Received with HEC errors
Tx PERR - # cells with memory parity errors
MSC# Rej. Cells Inv. Cells Mem. Buffs Rx Cells Tx Cells Rx HEC Tx PErr
----- ----------- ------------ ----------- ----------- ---------- ---------- ----------
MSC 0: 2189 6 14177 0 0 0 0
Rejected Cells: 2189
Invalid Cells: 58
Look at the values in the Rejected Cells and Invalid Cells fields. Note that the Rejected Cells and Invalid Cells field counters are increasing. This means there might be a problem in the switch fabric.
Step 3 Use the show epc queuing and show epc status command to display interface queues and status.
INT X-INT VCI QCNT VCI QCNT
Gi0/0/0 Gi1/0/0 67 640 62 0
Gi0/0/0 Gi1/0/0 71 546 66 0
Gi0/0/1 Gi1/0/0 67 135 147 0
Gi0/0/1 Gi1/0/0 69 18 149 0
Gi1/0/0 SRP 35 0 342 1791
Gi1/0/0 Gi0/0/0 62 0 67 640
Gi1/0/0 Gi0/0/0 66 0 71 546
Gi1/0/0 Gi0/0/1 147 0 67 135
Gi1/0/0 Gi0/0/1 149 0 69 18
Gi1/0/0 Gi1/0/1 152 0 67 639
Gi1/0/0 Gi12/0/0 577 0 67 640
Gi1/0/0 Gi12/0/0 578 0 68 16
Gi1/0/0 Gi12/0/0 579 0 69 38
Gi1/0/0 Gi12/0/0 580 0 70 16
Gi1/0/0 Gi12/0/1 662 0 67 640
Gi1/0/0 Gi12/0/1 666 0 71 640
Gi1/0/1 Gi1/0/0 67 639 152 0
Gi12/0/0 Gi1/0/0 67 640 577 0
Gi12/0/0 Gi1/0/0 68 16 578 0
Gi12/0/0 Gi1/0/0 69 38 579 0
Gi12/0/0 Gi1/0/0 70 16 580 0
Gi12/0/1 Gi1/0/0 67 640 662 0
Gi12/0/1 Gi1/0/0 71 640 666 0
Status of GigabitEthernet0/0/0: OK
Status of GigabitEthernet0/0/1: OK
Status of GigabitEthernet1/0/0: not OK
Status of GigabitEthernet1/0/1: OK
Status of GigabitEthernet2/0/0: OK
Status of GigabitEthernet2/0/1: OK
Status of GigabitEthernet12/0/0: OK
Status of GigabitEthernet12/0/1: OK
The show epc queuing command output shows that no activity is going across interface GigabitEthernet 1/0/0. This is verified in the show epc status command output, which indicates that interface GigabitEthernet 1/0/0 is "not OK." You have confirmed that the problem is a stuck port.
|
Note You might see a few cells in the QCNT column in the show epc queuing command
output. That is normal. Issue the command several times to verify that traffic is
moving through the queues. If the QCNT column values are incrementing and
incrementing for the VCIs belonging to a particular interface, the problem is
probably a stuck port.
|
You can remedy the port stuck condition by removing and reinserting the interface module. A shutdown/no shutdown command sequence on the problem interface does not resolve the problem.
Configuring Automatic Port Stuck Failure Recovery
To recover from a port stuck failure, perform the following tasks:
- Detect port stuck failure.
- Isolate the cell stuck failure.
- If it is only a port stuck failure, isolate the port from the other functional ports.
- Depending on the configuration option for reset of the stuck port, one of the following actions will occur:
If the switch router is not configured to reset the port upon detecting a port stuck failure, the port will be isolated, thus preserving the integrity of the switch router.
If the switch router is configured to reset the port upon detection of a stuck port failure, the switch router will isolate the port from the rest of the functioning ports, and reset the port. This might affect other ports on the interface module.
|
Note If you configure the switch router as described in the nondefault behavior section after a
port stuck failure is detected, the switch router will not reset the Ethernet ports. The
Ethernet interface must be configured to reset before the port stuck failure occurs. Also,
the default behavior is to not reset the port if a port stuck failure is detected. If the Ethernet
interface is not configured to reset when a port stuck failure is detected, schedule the
switch router for downtime to remove and reinsert the module.
|
To configure the switch router to automatically recover from port stuck failures, use the following interface configuration commands:
| Command
|
Purpose
|
Switch(config-if)# epc port-reload
|
Enables automatic resetting and reloading of the interface module microcode after detecting a port stuck failure.
|
Switch(config-if)# epc portstuck-wait seconds
|
Specifies the delay before signalling a port stuck failure (from the time the failure is detected). The default is 180 seconds. The range for seconds is 0 to 200. A value of 0 seconds causes a port stuck failure to not be detected.
|
 |
Caution Because of the nature of the microcode architecture, do not configure low values for the wait time in the epc portstuck-wait command. The default value of 180 seconds has been carefully chosen, allowing for the hello intervals of protocols such as HSRP, EIGRP, and OSPF. Configuring a low value might lead to incorrectly detecting temporary port stuck failures as real port stuck failures, and can cause a temporary loss of connectivity. It is highly recommended to keep this value to at 60 seconds, at a minimum. Lower values are provided to allow for some specific network designs when you can absolutely rule out temporary port stuck failure scenarios, and also as a debugging aid. For most networks, 180 seconds works very well. |
Connectivity Troubleshooting Commands
To troubleshoot a connectivity problem between a port and another port or end-station, use the following commands:
| Command
|
Purpose
|
show interfaces {fastethernet | gigabitethernet} card/subcard/port
|
Displays interface configuration, status, and statistics.
|
show controllers {fastethernet | gigabitethernet} card/subcard/port
|
Displays controller status for the specified interface.
|
show epc if-entry interface {fastethernet | gigabitethernet} card/subcard/port all
|
Displays all interface entry information for the specific interface.
|
show epc ip-prefix interface {fastethernet | gigabitethernet} card/subcard/port all-entries
|
Displays all ip prefix entries for the specified interface.
|
show epc ip-address interface {fastethernet | gigabitethernet} card/subcard/port all-entries
|
Displays all adjacent IP addresses for the specified interface.
|
show epc patricia interface {fastethernet | gigabitethernet} card/subcard/port ipucast detail
|
Displays IP unicast patricia tree for the specified interface.
|
show epc patricia interface {fastethernet | gigabitethernet} card/subcard/port mac detail
|
Displays the MAC patricia tree for the specified interface.
|
Troubleshooting 10/100 Ethernet Interface Modules
This section describes specific processes and commands used to troubleshoot the 10/100BASE-T and BASE-FX Ethernet interface modules.
The Catalyst 8500 CSR supports two different interface modules. The 10/100BASE-T Ethernet interface module supports 100-Mbps Layer 2 or Layer 3 UTP connections. The 100BASE-FX Ethernet interface module supports 100-Mbps Layer 2 or Layer 3 multimode fiber connections.
This section includes the following:
10/100BASE-T Interface Modules
The 10/100BASE-T Ethernet interface module supports 16 10-Mbps or 100-Mbps Layer 2 or Layer 3 unshielded twisted-pair (UTP) ports. This module supports full-duplex or half-duplex connections and Fast EtherChannel operation. The 10/100BASE-T interface module is available with 16K or 64K of memory. Routing tables use this memory.
10/100BASE-T Interface Module LEDs
Table 10-1 describes the LEDs used to confirm and troubleshoot the operation of interface modules. The LEDs on interface modules indicate the status of the modules and their ports.
Table 10-1 10/100BASE-T Interface Module LED Descriptions
| LED
|
State
|
Description
|
Lk
|
Green
|
Port is operational (a signal is detected).
|
|
Off
|
No signal is detected.
|
Sp
|
Green
|
Port is operating at 100 Mbps.
|
|
Off
|
Port is operating at 10 Mbps.
|
100BASE-FX Interface Modules
The 100BASE-FX Ethernet interface module supports 100-Mbps Layer 2 or Layer 3 multimode fiber connections. This module supports full-duplex connections and Fast EtherChannel operation. It provides 16 multimode fiber ports that have MT-RJ connectors. The 100BASE-FX interface module is available with 16K or 64K of memory. Routing tables use this memory.
100BASE-FX Interface Module LEDs
Table 10-2 describes the LEDs used to confirm and troubleshoot the operation of interface modules. The LEDs on interface modules indicate the status of the modules and their ports.
Table 10-2 100BASE-FX Interface Module LED Descriptions
| LED
|
State
|
