Troubleshoot Interface Overrun caused by Distributed Etherchannel
Available Languages
Download Options
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
Contents
Introduction
EtherChannel bundle is used to provide high bandwidth interconnects. This article discusses a limitation which applies to Cisco EtherChannels on Catalyst 6500 switches running Supervisor 720 with PFC3A, PFC3B or PFC3BXL that can cause overrun to increment on Etherchannel member interfaces. This limitation is related to Layer 2 Forwarding Engine and hence applies only to layer 2 EtherChannels.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
The information in this document is based on Cisco Catalyst 6500 Series switches that run Supervisor Engine 720. WS-X6704-10GE has been used in this lab setup.WS-X6704-10GE is a Catalyst 6500 module with no oversubscription & has 2 fabric channel connections of 20 Gbps each.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Problem
A Catalyst 6500 might experience interface overrun when a Layer 2 Distributed Etherchannel (DEC) is configured. DEC is an etherchannel across 2 or more Distriubuted Forwarding Card (DFC) modules. An Etherchannel with 2 member interfaces connected on the same linecard but on different fabric channels is not considered a DEC.
Overrun counter accounts number of times the receiver hardware was unable to hand received data to a hardware buffer. In other words, the input rate of traffic exceeded the ability of the receiver to handle the data.
In certain cases, DFC provides the capability to recirculate the packets. Recirculation can be used to perform additional lookups in the ACL or QoS Ternary Content Addressable memory (TCAM), the NetFlow table, or the Forwarding Information Base (FIB) TCAM table. Packet recirculation occurs only on a particular packet flow; other packet flows are not affected. The rewrite of the packet occurs on the modules; the packets are then forwarded back to the Policy Feature Card (PFC) for additional processing.
When using a layer 2 DEC, packet recirculation at ingress module is required during packet forwarding. Recirculation is also required for multi-module L2 EtherChannel if Catalyst 6500 is running in flow-through mode along with 3B/3BXL PFC mode.
More information about flow-through mode is available here.
Overrun counter can start incrementing when the fabric utilization reaches about 50%.
Troubleshoot & Verify
1) Find out the member interfaces in the Etherchannel experiencing incrementing overruns.
6500#show etherchannel summary
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use N - not in use, no aggregation
f - failed to allocate aggregator
M - not in use, no aggregation due to minimum links not met
m - not in use, port not aggregated due to minimum links not met
u - unsuitable for bundling
d - default port
w - waiting to be aggregated
Number of channel-groups in use: 2
Number of aggregators: 2
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
10 Po10(SU) - Te2/1(P) Te3/1(P)
20 Po20(SU) - Te2/2(P) Te3/2(P)
2) Verify input rate and overrun counters on member interfaces.
6500#show interfaces tenGigabitEthernet 2/1
TenGigabitEthernet2/1 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 0002.fcc1.21ac (bia 0002.fcc1.21ac)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 251/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 10Gbase-SR
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:51, output hang never
Last clearing of "show interface" counters 00:00:13
Input queue: 0/2000/5597178/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 9868906000 bits/sec, 822409 packets/sec
30 second output rate 3000 bits/sec, 5 packets/sec
10968368 packets input, 16452552000 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 5597195 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
79 packets output, 5596 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
6500#show interfaces tenGigabitEthernet 2/2
TenGigabitEthernet2/2 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 0002.fcc1.21ad (bia 0002.fcc1.21ad)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 251/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 10Gbase-SR
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:26, output hang never
Last clearing of "show interface" counters 00:00:03
Input queue: 0/2000/45043/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 9868149000 bits/sec, 822345 packets/sec
30 second output rate 2000 bits/sec, 4 packets/sec
8823464 packets input, 13233496000 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 4575029 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
3) Find out the modules on which these interfaces are present.
6500#show module
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
2 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAD07430301
3 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SAL1316NJD4
5 2 Supervisor Engine 720 (Active) WS-SUP720-3B JAF1224BFSQ
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
2 0002.fcc1.21ac to 0002.fcc1.21af 1.2 12.2(14r)S5 12.2(33)SXI5 Ok
3 0024.c4f5.b2f4 to 0024.c4f5.b2f7 2.9 12.2(14r)S5 12.2(33)SXI5 Ok
5 001e.be6e.99b4 to 001e.be6e.99b7 5.6 8.5(4) 12.2(33)SXI5 Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
2 Centralized Forwarding Card WS-F6700-CFC SAD0743039A 1.1 Ok
3 Distributed Forwarding Card WS-F6700-DFC3B SAL1408BP0Y 4.8 Ok
5 Policy Feature Card 3 WS-F6K-PFC3B JAF1223BAPB 2.3 Ok
5 MSFC3 Daughterboard WS-SUP720 JAF1223BACM 3.1 Ok
Mod Online Diag Status
---- -------------------
2 Pass
3 Pass
5 Pass
4) Find out the fabric interface utilization corresponding to these modules.
6500#show fabric utilization
slot channel speed Ingress % Egress %
2 0 20G 0 0
2 1 20G 49 0
3 0 20G 0 0
3 1 20G 0 50
5 0 20G 0 0
6500#show platform hardware capacity rewrite-engine performance slot 2
slot channel perf_id description packets total overruns
----+-------+-------+--------------------+--------------------+--------------+
2 0 0x235 FAB RX 0 41083 0
2 0 0x237 FAB RX 1 0 0
2 0 0x27B FAB TX 0 1904 0
2 0 0x27F FAB TX 1 0 0
2 0 0x350 REPLICATION ML3 0 0
2 0 0x351 REPLICATION ML2 0 0
2 0 0x352 RECIRC L2 0 0
2 0 0x353 RECIRC L3 0 0
2 0 0x34C SPAN TX 0 0 0
2 0 0x34D SPAN TX 1 0 0
2 0 0x34E SPAN RX 0 0 0
2 0 0x34F SPAN RX 1 0 0
2 0 0x354 SPAN TERMINATION 0 0
2 1 0x235 FAB RX 0 759500888 0
2 1 0x237 FAB RX 1 0 0
2 1 0x27B FAB TX 0 4545890286 0
2 1 0x27F FAB TX 1 0 0
2 1 0x350 REPLICATION ML3 0 0
2 1 0x351 REPLICATION ML2 0 0
2 1 0x352 RECIRC L2 68615145 1047 <<< L2 Recirculation
2 1 0x353 RECIRC L3 0 0
2 1 0x34C SPAN TX 0 0 0
2 1 0x34D SPAN TX 1 0 0
2 1 0x34E SPAN RX 0 0 0
2 1 0x34F SPAN RX 1 0 0
2 1 0x354 SPAN TERMINATION 0 0
Solution
1) Configure a non distributed Etherchannel which is not affected by this limitation.
To validate this theory, an Etherchannel was configured on interfaces on same module (non DEC) and it was observed that at same packet rate as above, interfaces did not see any overruns incrementing. This can be a workaround to bypass this problem.
6500#show fabric utilization
slot channel speed Ingress % Egress %
2 0 20G 0 0
2 1 20G 99 0
3 0 20G 0 0
3 1 20G 0 99
5 0 20G 0 0
TenGigabitEthernet2/1 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 0002.fcc1.21ac (bia 0002.fcc1.21ac)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 251/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 10Gbase-SR
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:06, output hang never
Last clearing of "show interface" counters 00:36:12
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 9868487000 bits/sec, 822374 packets/sec
30 second output rate 3000 bits/sec, 6 packets/sec
1783710310 packets input, 2675565466500 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
13115 packets output, 946206 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
TenGigabitEthernet2/2 is up, line protocol is up (connected)
Hardware is C6k 10000Mb 802.3, address is 0002.fcc1.21ad (bia 0002.fcc1.21ad)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 251/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, media type is 10Gbase-SR
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:11, output hang never
Last clearing of "show interface" counters 00:37:31
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 9868462000 bits/sec, 822371 packets/sec
30 second output rate 3000 bits/sec, 6 packets/sec
1849499775 packets input, 2774249662500 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
13599 packets output, 980928 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
2) Use Catalyst 6500 switch in PFC 3C/3CXL mode in case L2 DEC is required.
Note: DFC hardware upgrade would be required in case existing modules are running DFC3A/ DFC3B/ DFC3BXL.
3) Upgrade IOS version if your design and configuration applies to the conditions in CSCti23324.
This bug fix relaxes the recirculation requirement for L2 DEC or multi-module EtherChannel for Catalyst 6500 switches with 67xx modules only. This bug is resolved in Cisco IOS release 12.2(33)SXJ1 and later. Be aware of following points that apply to this bug.
a) The bug fix relaxes the recirculation requirement for L2 DEC or multi-module EC for Catalyst 6500 switches 67xx modules only. In case the Catalyst 6500 switch has at least one L2 DEC across any older DFC module (e.g. 6516/6816) or combination of 67xx and 6516/6818 module, recirculation will be imposed for all L2 DECs configured in the system. In case Catalyst 6500 switch has any older module and is configured with L2 DEC on 67xx modules only, recirculation will not be imposed.
b) Presence of all 67xx line cards is not enough to remove the recirculation requirement for DECs. For example, if you have a DEC across 2 6704 DFCs and another port-channel configured on a 6748 CFC, the system will check the forwarding engine of supervisor (for the CFC module) and start using recirculation.
c) For VS-SUP720-10G, this bug fix does not work in scenarios where at least one port of L2 DEC is on CFC linecard / supervisor. In this scenario recirculation still happens. In addition, adjacency is not upgraded and recirculation is still in place even if you remove supervisor/CFC enabled port from the port-channel. In such scenario, reload is required to reprogram the hardware and removing & reconfiguring port-channel / redundancy switchover / removing L2 VLAN, etc. do not help.
Contributed by Cisco Engineers
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)