Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Problem Summary

Software Changes

Workarounds

Introduction

This document describes how to understand errors from devices connected to Multigigabit Ethernet (mGig) ports on Catalyst 9000 Series Switches.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these platforms: Catalyst 9000 series switches with mGig capable ports.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

This document describes why you can encounter frame check sequence (FCS) errors, input errors, or packet loss with devices that connect to Multigigabit Ethernet (mGig) ports on Catalyst 9000 series switches due to interpacket gap (IPG) or interframe gap (IFG) tolerance.

In networking, a pause could be required between network packets or network frames. This time between packets is known as the IPG or IFG. This pause is necessary to allow for receiver clock recovery, which permits the receiver to prepare for another packet. The IFG/IPG standard value for Gigabit Ethernet is 12 bytes. However, from  IEEE Standard 802.3, the minimum value for the IFG can be as low as 8 bytes or 64 BT (bit times). For reference, this is documented in 802.3-2000 - IEEE Standard for Information Technology - LAN/MAN - Specific Requirements.

Problem Summary

Multigigabit Ethernet technology is implemented on 10Gig PHYs on Cat9000 architecture. For example, when a connection is established through an mGig port at 1Gbps, if traffic bursts higher than the bandwidth of the interface, the C9600 utilizes port buffers to accommodate that excess traffic and dynamically decreases the IFG/IPG size to avoid any impact and ensure traffic throughput and switch performance. The issue arises when some peer devices are unable to handle the smaller IFG/IPG sizes and no longer recognize legitimate packets and drop this traffic, which results in input errors on their NIC or PHY, such as Cyclic Redundancy Check (CRC) or FCS errors. In certain scenarios the local mGig port (an interface from the mGig linecard C9600-LC-48TX) can also experience the same type of loss in the form of input errors (CRC, FCS) on the interface.

As shown in the table, the structure of an Ethernet packet, which includes the IPG/IFG field:

Layer

Preamble

Start Frame Delimeter

Destination MAC

Source MAC

802.1Q Tag

Ethertype (Ethernet II) or length (IEEE 802.3)

Payload

Frame Check Sequence (32 bit CRC)

IPG/IFG

7 octets

1 octet

6 octets

6 octets

4 octets

2 octets

46-1500 octets

4 octets

≥ 8 octets

Layer 2 Ethernet Frame

64-1522 octets

Layer 1 Bits

72-1530 octets

≥ 8 octets

Software Changes

Cisco has made changes to software for mGig capable Catalyst switches to accommodate devices that do not tolerate variance in the IPG/IFG. These changes are documented in various Cisco bug IDs.

Platform(s) Affected

Bug ID and Resolution Status

C9200L

Fully resolved, see 'Cisco bug ID CSCvy72944' for more information.

C9300-48UN

Fully resolved, see Cisco bug ID CSCvw65866 for more information.

C9300-48UXM

Fully resolved, see 'Cisco bug ID CSCvr95643' for more information.

C9300-48UXM

Fully resolved, see 'Cisco bug ID CSCvr13950 'for more information.

C9600-LC-48TX

Resolution in progress:

Under rare circumstances, customers can still encounter issues that would have been resolved, see 'Cisco bug ID CSCvz67689' for more information.

As a result of the rare issues documented previously, additional fixes are required, see 'Cisco bug ID CSCwb31319' for more information.

Note: Only registered Cisco clients can access the bugs listed in this document.

Workarounds

In some cases, these interoperability issues can be mitigated through hard-coding the mGig port to a lower speed (100Mbps vs 1Gbps), utilization of a different speed (100Mbps or 10Gbps vs 1Gbps), or the affected device is moved to a non-mGig capable port.