Identify and Trace Nexus 9000 Cloud Scale ASIC CRC
Available Languages
Contents
Introduction
This document describes the steps used to trace the source of CRC errors observed on Cisco Nexus 9000 Cloud Scale ASIC modules.
Prerequisites
Requirements
Cisco recommends that you understand the basics of cut-through and store-and-forward switching. Cisco also recommends that you understand the basics of the Ethernet Frame Check Sequence (FCS) field and the Cyclic Redundancy Check (CRC) algorithm used by the FCS field. For more information, refer to this document:
Components Used
The information in this document is based on Cisco Nexus 9000 series switches with the Cloud Scale ASIC running NX-OS software release 7.0(3)I7(8).
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
This document also describes the procedure used to differentiate stomped and non-stomped CRC errors observed on physical interfaces and internal fabric links of modular Nexus switches.
Cisco Nexus 9000 series switches use cut-through switching by default. Cut-through switching is where a switch makes a forwarding decision on a frame and begins forwarding the frame out of an egress interface as soon as the switch has processed enough of the frame's header to make a valid forwarding decision. This differs from store-and-forward switching, where a switch buffers the entire frame prior to forwarding the frame out of an egress interface.
The FCS field of an Ethernet frame validates the integrity of the frame and ensures that the frame has not been corrupted in transit. The FCS field of an Ethernet frame is located at the end of the Ethernet frame behind the frame's payload. A switch operating in a store-and-forward switching mode is able to verify the integrity of an Ethernet frame with the FCS field prior to forwarding the frame out of an egress interface (or drop the frame if the FCS field has invalid contents). However, a switch operating in a cut-through switching mode is not able to verify the integrity of an Ethernet frame with the FCS field prior to forwarding the frame out of an egress interface; in other words, by the time a cut-through switch is able to verify the integrity of an Ethernet frame, the majority of the Ethernet frame has already been forwarded out of an egress interface.
If a switch operating in a cut-through switching mode receives an Ethernet frame with an invalid FCS field, the switch can take these actions:
- Rewrite the FCS field of the Ethernet frame with the bitwise inverse of the current (incorrect) FCS field's value. If the frame needs to be routed, the current (incorrect) FCS field's value would be computed after the frame's Ethernet header is rewritten. This action is known as stomping the CRC.
- Forward the remainder of the Ethernet frame (along with the stomped CRC) out of the egress interface according to the forwarding decision made on the frame.
- Increment the input errors counter and/or CRC errors counter on the ingress interface.
This document describes the steps to verify whether CRC counters associated with an ingress interface are normal CRCs (which typically indicate physical layer issues on the link connected to the ingress interface) or stomped CRCs (which indicate that the device connected to the ingress interface is also operating in a cut-through switching mode and received a malformed Ethernet frame).
Applicable Hardware
The procedure covered in this document is applicable to this hardware only:
Nexus 9200/9300 Fixed Switches
- N9K-C92160YC-X
- N9K-C92300YC
- N9K-C92304QC
- N9K-C92348GC-X
- N9K-C9236C
- N9K-C9272Q
- N9K-C9332C
- N9K-C9364C
- N9K-C93108TC-EX
- N9K-C93108TC-EX-24
- N9K-C93180LC-EX
- N9K-C93180YC-EX
- N9K-C93180YC-EX-24
- N9K-C93108TC-FX
- N9K-C93108TC-FX-24
- N9K-C93180YC-FX
- N9K-C93180YC-FX-24
- N9K-C9348GC-FXP
- N9K-C93240YC-FX2
- N9K-C93216TC-FX2
- N9K-C9336C-FX2
- N9K-C9336C-FX2-E
- N9K-C93360YC-FX2
- N9K-C93180YC-FX3
- N9K-C93108TC-FX3P
- N9K-C93180YC-FX3S
- N9K-C9316D-GX
- N9K-C93600CD-GX
- N9K-C9364C-GX
- N9K-C9364D-GX2A
- N9K-C9332D-GX2B
Nexus 9500 Modular Switch Line Cards
- N9K-X97160YC-EX
- N9K-X9732C-EX
- N9K-X9736C-EX
- N9K-X97284YC-FX
- N9K-X9732C-FX
- N9K-X9788TC-FX
- N9K-X9716D-GX
Cisco Nexus 9200 and 9300 Cloud Scale CRC Identification and Tracing Procedure
This section of the document describes step-by-step instructions to identify the source of CRC errors observed on a specific physical interface Ethernet1/1 on Cisco Nexus 9200 and 9300 series switches.
NX-OS Software Release 10.2(1) and Later
Starting with NX-OS software release 10.2(1), Nexus switches equipped with the Cloud Scale ASIC have a new interface counter for packets with a stomped CRC in the FCS field of Ethernet frames that traverse the switch. You can use the show interface command to identify physical interfaces with incrementing non-zero CRC and stomped CRC counters. An example of this is shown here, where physical interface Ethernet1/1 has a zero CRC counter and non-zero stomped CRC counter, which indicates that frames with an invalid and stomped CRC were received on this interface.
switch# show interface
<snip>
Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000/25000 Ethernet, address: 00d7.8f86.2bbe (bia 00d7.8f86.2bbe)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 04:09:21
Last clearing of "show interface" counters 00:50:37
0 interface resets
RX
8 unicast packets 253 multicast packets 2 broadcast packets
1832838280 input packets 2199405650587 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 1832838019 CRC 0 no buffer
1832838019 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
1832838019 Stomped CRC
TX
908 unicast packets 323 multicast packets 3 broadcast packets
1234 output packets 113342 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
Note: An incrementing CRC counter indicates a frame was received with either stomped CRC or an invalid, but non-stomped CRC. An incrementing stomped CRC counter increase indicates a frame with a stomped CRC was received.
Alternatively, the show interface counters errors non-zero command can be used to see interface errors counters. An example of this is shown here.
switch# show interface counters errors non-zero
--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
Eth1/1 1790348828 1790348828 0 1790348828 0 0
--------------------------------------------------------------------------------
Port Single-Col Multi-Col Late-Col Exces-Col Carri-Sen Runts
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Port Giants SQETest-Err Deferred-Tx IntMacTx-Er IntMacRx-Er Symbol-Err
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Port InDiscards
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Port Stomped-CRC
--------------------------------------------------------------------------------
Eth1/1 1790348828
You can pipe the show interface command to the json or json-pretty commands to obtain CRC and stomped CRC counter statistics in a structured format. An example of this is shown here.
switch# show interface Ethernet1/1 | json-pretty | include ignore-case crc
"eth_crc": "828640831",
"eth_stomped_crc": "828640831",
The NX-API REST API can be used to retrieve these same statistics using the sys/intf/phys-[intf-id]/dbgEtherStats.json object model. An example of this is shown here.
/api/node/mo/sys/intf/phys-[eth1/1]/dbgEtherStats.json
{
"totalCount": "1",
"imdata": [
{
"rmonEtherStats": {
"attributes": {
"cRCAlignErrors": "26874272810",
"dn": "sys/intf/phys-[eth1/1]/dbgEtherStats",
"dropEvents": "0",
"rXNoErrors": "26874276337",
"stompedCRCAlignErrors": "26874272810",
...
}
}
}
]
}
NX-OS Software Release 10.1(2) and Earlier
For NX-OS software releases prior to 10.2(1), the stomped CRC counter is not available on interfaces. Several steps are needed to determine the ingress interface where invalid CRCs are observed and validate whether the CRCs are invalid or stomped.
Step 1. Identify Incrementing CRC Counters on Physical Interface(s)
Use the show interface command to identify physical interfaces with incrementing non-zero CRC counters. An example of this shown here, where physical interface Ethernet1/1 has a non-zero CRC counter.
switch# show interface
<snip> Ethernet1/1 is up admin state is up, Dedicated Interface Hardware: 100/1000/10000/25000 Ethernet, address: 00d7.8f86.2bbe (bia 00d7.8f86.2bbe) MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, medium is broadcast Port mode is trunk full-duplex, 10 Gb/s, media type is 10G Beacon is turned off Auto-Negotiation is turned on FEC mode is Auto Input flow-control is off, output flow-control is off Auto-mdix is turned off Rate mode is dedicated Switchport monitor is off EtherType is 0x8100 EEE (efficient-ethernet) : n/a admin fec state is auto, oper fec state is off Last link flapped 04:09:21 Last clearing of "show interface" counters 00:50:37 0 interface resets RX 3 unicast packets 3087 multicast packets 0 broadcast packets 3097 input packets 244636 bytes 7 jumbo packets 0 storm suppression bytes 0 runts 7 giants 7 CRC 0 no buffer 7 input error 0 short frame 0 overrun 0 underrun 0 ignored 0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop 0 input with dribble 0 input discard 0 Rx pause
Alternatively, you can use the show interface counters errors non-zero command to display all interface with non-zero error counters (which includes non-zero CRC counters). An example of this shown here, where physical interface Ethernet1/1 has a non-zero CRC counter displayed by the FCS-Err column.
switch# show interface counters errors non-zero <snip> -------------------------------------------------------------------------------- Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards -------------------------------------------------------------------------------- Eth1/1 7 7 0 7 0 0
Step 2. Map the Physical Interface to ASIC, MAC Block, and Mac Block Sub-Port
Use the show interface hardware-mappings command to identify three key characteristics:
- Unit - The identifier of the Cloud Scale ASIC that the physical interface connects to. This uses a zero-based numbering system (for example, the first ASIC is 0, the second ASIC is 1, and so on)
- MacId - The identifier of the MAC block that the physical interface connects to. This uses a zero-based numbering system (for example, the first MAC block is 0, the second MAC block is 1, and so on)
- MacSP - The identifier of the MAC block sub-port that the physical interface connects to. Each MAC block has four sub-ports associated with it, which use a zero-based numbering system and increment by a value of 2. Therefore, the first sub-port has an index of 0, the second sub-port has an index of 2, the third sub-port has an index of 4, and the fourth sub-port has an index of 6.
This is demonstrated in the example here, where physical interface Ethernet1/1 is associated with Cloud Scale ASIC 0, MAC block 4, and MAC block sub-port 0.
switch# show interface hardware-mappings
<snip>
-------------------------------------------------------------------------------------------------------
Name Ifindex Smod Unit HPort FPort NPort VPort Slice SPort SrcId MacId MacSP VIF Block BlkSrcID
-------------------------------------------------------------------------------------------------------
Eth1/1 1a000000 1 0 16 255 0 -1 0 16 32 4 0 1 0 32
Eth1/2 1a000200 1 0 17 255 4 -1 0 17 34 4 2 5 0 34
Eth1/3 1a000400 1 0 18 255 8 -1 0 18 36 4 4 9 0 36
Eth1/4 1a000600 1 0 19 255 12 -1 0 19 38 4 6 13 0 38
Eth1/5 1a000800 1 0 12 255 16 -1 0 12 24 3 0 17 0 24
Step 3. Check Cloud Scale ASIC Registers for CRC-Related Counters
Use the slot {x} show hardware internal tah counters asic {y} command to view register counters for the Cloud Scale ASIC. This command contains two variables:
- {x} - Replace this value with the line card slot number. For top-of-rack switches, this is always a value of 1. For end-of-row modular switches, the line card slot number is the first number in the physical interface's name. For example, physical interface Ethernet1/1 would have a line card slot number of 1, while physical interface Ethernet4/24 would have a line card slot number of 4.
- {y} - Replace this value with the Cloud Scale ASIC identifier identified in Step 2. For example, if the Unit column for physical interface Ethernet1/1 had a value of 0, then the value of this variable would be 0. If the Unit column for physical interface Ethernet4/24 had a value of 3, then the value of this variable would be 3.
This output can display a table. Each row of the table is a different ASIC register. Each column of the table corresponds with a physical interface on the switch. The name used for each column is not the name of the physical interface, but is a combination of the MAC block and MAC block sub-port. The format used for the column header is this:
M{A},{B}-{InterfaceSpeed}There are three variables in this format, which are:
- {A} - Replace this value with the MAC block number.
- {B} - Replace this value with the MAC block sub-port number.
- {InterfaceSpeed} - This value can correspond with the physical speed of the interface (for example, 10G, 25G, 40Gx4, and so on)
This is demonstrated in the example here. Recall that physical interface Ethernet1/1 is associated with line card slot number 1 and Cloud Scale ASIC 0, which means the command you must run is slot 1 show hardware internal tah counters asic 0. The MAC block associated with physical interface Ethernet1/1 is 4, the MAC block sub-port associated with physical interface Ethernet1/1 is 0, and physical interface Ethernet1/1 is a 10G interface. Therefore, the column header we are looking for is M4,0-10G.
switch# slot 1 show hardware internal tah counters asic 0 <snip> *************** PER MAC/CH SRAM COUNTERS **************** REG_NAME M4,0-10G M4,2-10G M4,4-10G M4,6-10G M5,0-40Gx4 M6,0-40Gx4 M7,0-40Gx4 M8,0-10G ---------------------------------------------------------------------------------------------------------------------------- 02-RX Frm with FCS Err .... .... .... .... .... .... .... .... 16-RX Frm CRC Err(Stomp) c .... .... .... .... .... .... ....
The output of this command contains several dozen register counters. There are two key register counters that are related to differentiating natural CRC errors from stomped CRCs:
- 02-RX Frm with FCS Err - Indicates a frame with an invalid, but non-stomped CRC was received.
- 16-RX Frm CRC Err(Stomp) - Indicates a frame with a stomped CRC was received.
The value of these counters is in hexadecimal. The dec NX-OS command can convert a hexadecimal value to a decimal value, as shown here.
N9K-C93180YC-EX-2# dec 0xc
12
The combined values of both register counters is equivalent to the number of CRCs observed on the physical interface through the output of show interface or show interface counters errors non-zero.
Cisco Nexus 9500 Cloud Scale - CRC Identification and Tracing Procedure on Modular Switches
This section of the document describes step-by-step instructions to identify the source of CRC errors observed on a specific physical interface Ethernet1/1 on Cisco Nexus 9500 series switches.
Each line card on a Nexus 9500 series switch is connected via internal link (iEth) to the fabric modules. Each ASIC of each line card has full-mesh connectivity to all fabric modules. The example here shows a line card with four Sugarbowl ASICs with internal links connecting to four fabric modules within a modular Nexus 9500 switch.
When traffic received by an ASIC needs to egress another ASIC or line card, that traffic needs to be sent to the fabric through a fabric module. The ingress ASIC selects one of the iEth links to the fabric modules based on a hash of the packet's headers and the number of iEth links available to the ASIC.
Step 1. Map Internal Links Between Line Cards and Fabric Modules.
Use the show system internal fabric connectivity module {x} command (where {x} is the line card or fabric module slot number) to display the internal links between the specified line card and all fabric modules. This output displays a table where each row shows a one-to-one mapping between line card internal links (under the LC-iEthLink column) to each fabric module's internal links (under the FM-iEthLink column). An example of this is shown here, taken from a Nexus 9508 switch with 8 line cards and 4 fabric modules inserted. The output here shows that each ASIC instance of the line card inserted in slot 8 of the switch is connected to each of the 4 installed fabric modules (inserted in slots 22, 23, 24, and 26) through 2 internal links.
Nexus9500# show system internal fabric connectivity module 8
Internal Link-info Linecard slot:8
------------------------------------------------------------------------
LC-Slot LC-Unit LC-iEthLink MUX FM-Slot FM-Unit FM-iEthLink
------------------------------------------------------------------------
8 0 iEth01 - 22 0 iEth18
8 0 iEth02 - 22 1 iEth50
8 0 iEth03 - 23 0 iEth18
8 0 iEth04 - 23 1 iEth50
8 0 iEth05 - 24 0 iEth18
8 0 iEth06 - 24 1 iEth50
8 0 iEth07 - 26 0 iEth18
8 0 iEth08 - 26 1 iEth50
8 1 iEth09 - 22 0 iEth03
8 1 iEth10 - 22 1 iEth35
8 1 iEth11 - 23 0 iEth03
8 1 iEth12 - 23 1 iEth35
8 1 iEth13 - 24 0 iEth03
8 1 iEth14 - 24 1 iEth35
8 1 iEth15 - 26 0 iEth03
8 1 iEth16 - 26 1 iEth35
8 2 iEth17 - 22 0 iEth32
8 2 iEth18 - 22 1 iEth53
8 2 iEth19 - 23 0 iEth32
8 2 iEth20 - 23 1 iEth53
8 2 iEth21 - 24 0 iEth32
8 2 iEth22 - 24 1 iEth53
8 2 iEth23 - 26 0 iEth32
8 2 iEth24 - 26 1 iEth53
8 3 iEth25 - 22 0 iEth31
8 3 iEth26 - 22 1 iEth54
8 3 iEth27 - 23 0 iEth31
8 3 iEth28 - 23 1 iEth54
8 3 iEth29 - 24 0 iEth31
8 3 iEth30 - 24 1 iEth54
8 3 iEth31 - 26 0 iEth31
8 3 iEth32 - 26 1 iEth54
Similarly, iEth link mapping can be checked from a fabric module's perspective. An example of this is shown here, where internal links between the fabric module inserted in slot 22 and each of the 8 line cards installed in the Nexus 9508 chassis are displayed.
Nexus9500# show system internal fabric connectivity module 22
Internal Link-info Fabriccard slot:22
------------------------------------------------------------------------
FM-Slot FM-Unit FM-iEthLink LC-Slot LC-Unit LC-EthLink MUX
------------------------------------------------------------------------
22 0 iEth09 1 0 iEth01 -
22 0 iEth06 1 1 iEth11 -
22 0 iEth25 1 2 iEth21 -
22 0 iEth26 1 3 iEth31 -
22 0 iEth10 2 0 iEth01 -
22 0 iEth05 2 1 iEth11 -
22 0 iEth23 2 2 iEth21 -
22 0 iEth24 2 3 iEth31 -
22 0 iEth12 3 0 iEth01 -
22 0 iEth11 3 1 iEth11 -
22 0 iEth21 3 2 iEth21 -
22 0 iEth22 3 3 iEth31 -
22 0 iEth14 4 0 iEth01 -
22 0 iEth13 4 1 iEth11 -
22 0 iEth07 4 2 iEth21 -
22 0 iEth08 4 3 iEth31 -
22 0 iEth16 5 0 iEth01 -
22 0 iEth15 5 1 iEth11 -
22 0 iEth01 5 2 iEth21 -
22 0 iEth04 5 3 iEth31 -
22 0 iEth20 6 0 iEth01 -
22 0 iEth17 6 1 iEth11 -
22 0 iEth28 6 2 iEth21 -
22 0 iEth27 6 3 iEth31 -
22 0 iEth19 7 0 iEth01 -
22 0 iEth02 7 1 iEth09 -
22 0 iEth30 7 2 iEth17 -
22 0 iEth29 7 3 iEth25 -
22 0 iEth18 8 0 iEth01 -
22 0 iEth03 8 1 iEth09 -
22 0 iEth32 8 2 iEth17 -
22 0 iEth31 8 3 iEth25 -
22 1 iEth41 1 0 iEth02 -
22 1 iEth38 1 1 iEth12 -
22 1 iEth59 1 2 iEth22 -
22 1 iEth60 1 3 iEth32 -
22 1 iEth42 2 0 iEth02 -
22 1 iEth37 2 1 iEth12 -
22 1 iEth62 2 2 iEth22 -
22 1 iEth61 2 3 iEth32 -
22 1 iEth44 3 0 iEth02 -
22 1 iEth43 3 1 iEth12 -
22 1 iEth64 3 2 iEth22 -
22 1 iEth63 3 3 iEth32 -
22 1 iEth46 4 0 iEth02 -
22 1 iEth45 4 1 iEth12 -
22 1 iEth39 4 2 iEth22 -
22 1 iEth40 4 3 iEth32 -
22 1 iEth48 5 0 iEth02 -
22 1 iEth47 5 1 iEth12 -
22 1 iEth36 5 2 iEth22 -
22 1 iEth33 5 3 iEth32 -
22 1 iEth52 6 0 iEth02 -
22 1 iEth49 6 1 iEth12 -
22 1 iEth57 6 2 iEth22 -
22 1 iEth58 6 3 iEth32 -
22 1 iEth34 7 0 iEth02 -
22 1 iEth51 7 1 iEth10 -
22 1 iEth55 7 2 iEth18 -
22 1 iEth56 7 3 iEth26 -
22 1 iEth50 8 0 iEth02 -
22 1 iEth35 8 1 iEth10 -
22 1 iEth53 8 2 iEth18 -
22 1 iEth54 8 3 iEth26 -
Use the show system internal fabric link-state module {x} command to check whether the internal port is up or not (under the ST columns), and what the corresponding ASIC slice and MAC identifier of a particular internal link (under the MAC column) are. An example of this is shown here.
Nexus9500# show system internal fabric link-state module 8
cli : mod = 8
module number = 8
========================================================================================================================
Module number = 8
========================================================================================================================
[LC] [ INST:SLI:MAC:GLSRC] [IETH] [ST] <========> [FM] [ INST:SLI:MAC:GLSRC] [IETH] [ST]
========================================================================================================================
[ 8] [ 0 : 0 : 7 : 0x38] [iEth01] [UP] <========> [22] [ 0 : 3 : 21 : 0x18] [iEth18] [UP]
[ 8] [ 0 : 1 : 9 : 0x0] [iEth02] [UP] <========> [22] [ 1 : 3 : 21 : 0x18] [iEth50] [UP]
[ 8] [ 0 : 0 : 6 : 0x30] [iEth03] [UP] <========> [23] [ 0 : 3 : 21 : 0x18] [iEth18] [UP]
[ 8] [ 0 : 1 : 16 : 0x38] [iEth04] [UP] <========> [23] [ 1 : 3 : 21 : 0x18] [iEth50] [UP]
[ 8] [ 0 : 0 : 8 : 0x40] [iEth05] [UP] <========> [24] [ 0 : 3 : 21 : 0x18] [iEth18] [UP]
[ 8] [ 0 : 1 : 15 : 0x30] [iEth06] [UP] <========> [24] [ 1 : 3 : 21 : 0x18] [iEth50] [UP]
[ 8] [ 0 : 0 : 5 : 0x28] [iEth07] [UP] <========> [26] [ 0 : 3 : 21 : 0x18] [iEth18] [UP]
[ 8] [ 0 : 1 : 17 : 0x40] [iEth08] [UP] <========> [26] [ 1 : 3 : 21 : 0x18] [iEth50] [UP]
[ 8] [ 1 : 0 : 7 : 0x38] [iEth09] [UP] <========> [22] [ 0 : 0 : 4 : 0x20] [iEth03] [UP]
[ 8] [ 1 : 1 : 9 : 0x0] [iEth10] [UP] <========> [22] [ 1 : 0 : 4 : 0x20] [iEth35] [UP]
[ 8] [ 1 : 0 : 6 : 0x30] [iEth11] [UP] <========> [23] [ 0 : 0 : 4 : 0x20] [iEth03] [UP]
[ 8] [ 1 : 1 : 16 : 0x38] [iEth12] [UP] <========> [23] [ 1 : 0 : 4 : 0x20] [iEth35] [UP]
[ 8] [ 1 : 0 : 8 : 0x40] [iEth13] [UP] <========> [24] [ 0 : 0 : 4 : 0x20] [iEth03] [UP]
[ 8] [ 1 : 1 : 15 : 0x30] [iEth14] [UP] <========> [24] [ 1 : 0 : 4 : 0x20] [iEth35] [UP]
[ 8] [ 1 : 0 : 5 : 0x28] [iEth15] [UP] <========> [26] [ 0 : 0 : 4 : 0x20] [iEth03] [UP]
[ 8] [ 1 : 1 : 17 : 0x40] [iEth16] [UP] <========> [26] [ 1 : 0 : 4 : 0x20] [iEth35] [UP]
[ 8] [ 2 : 0 : 7 : 0x38] [iEth17] [UP] <========> [22] [ 0 : 5 : 35 : 0x28] [iEth32] [UP]
[ 8] [ 2 : 1 : 9 : 0x0] [iEth18] [UP] <========> [22] [ 1 : 4 : 24 : 0x0] [iEth53] [UP]
[ 8] [ 2 : 0 : 6 : 0x30] [iEth19] [UP] <========> [23] [ 0 : 5 : 35 : 0x28] [iEth32] [UP]
[ 8] [ 2 : 1 : 16 : 0x38] [iEth20] [UP] <========> [23] [ 1 : 4 : 24 : 0x0] [iEth53] [UP]
[ 8] [ 2 : 0 : 8 : 0x40] [iEth21] [UP] <========> [24] [ 0 : 5 : 35 : 0x28] [iEth32] [UP]
[ 8] [ 2 : 1 : 15 : 0x30] [iEth22] [UP] <========> [24] [ 1 : 4 : 24 : 0x0] [iEth53] [UP]
[ 8] [ 2 : 0 : 5 : 0x28] [iEth23] [UP] <========> [26] [ 0 : 5 : 35 : 0x28] [iEth32] [UP]
[ 8] [ 2 : 1 : 17 : 0x40] [iEth24] [UP] <========> [26] [ 1 : 4 : 24 : 0x0] [iEth53] [UP]
[ 8] [ 3 : 0 : 7 : 0x38] [iEth25] [UP] <========> [22] [ 0 : 5 : 34 : 0x20] [iEth31] [UP]
[ 8] [ 3 : 1 : 9 : 0x0] [iEth26] [UP] <========> [22] [ 1 : 4 : 25 : 0x8] [iEth54] [UP]
[ 8] [ 3 : 0 : 6 : 0x30] [iEth27] [UP] <========> [23] [ 0 : 5 : 34 : 0x20] [iEth31] [UP]
[ 8] [ 3 : 1 : 16 : 0x38] [iEth28] [UP] <========> [23] [ 1 : 4 : 25 : 0x8] [iEth54] [UP]
[ 8] [ 3 : 0 : 8 : 0x40] [iEth29] [UP] <========> [24] [ 0 : 5 : 34 : 0x20] [iEth31] [UP]
[ 8] [ 3 : 1 : 15 : 0x30] [iEth30] [UP] <========> [24] [ 1 : 4 : 25 : 0x8] [iEth54] [UP]
[ 8] [ 3 : 0 : 5 : 0x28] [iEth31] [UP] <========> [26] [ 0 : 5 : 34 : 0x20] [iEth31] [UP]
[ 8] [ 3 : 1 : 17 : 0x40] [iEth32] [UP] <========> [26] [ 1 : 4 : 25 : 0x8] [iEth54] [UP]
Step 2. Check the CRC counters on iEth Links and Track the Source of Corrupted Frames
On a modular Nexus 9500 switch, you can see CRC errors on one or more iEth links in these scenarios:
- When the switch is operating in a a cut-through switching mode, a line card that receives a corrupted Ethernet frame with an incorrect CRC value in the FCS field does not drop the frame locally. Instead, the line card forwards the packet as normal. If the egress interface for the packet belongs to another ASIC or line card, the ingress line card forwards the packet towards a fabric module. The fabric modules also operate in a cut-through switching mode, so the fabric module forwards the packet to the egress line card. The egress line card forwards the packet towards the next-hop and increment the output errors counter on the egress interface.
- If an internal link is failing due to faulty hardware, packets that traverse the internal link can be corrupted between a line card and the fabric module.
Use the show system internal fabric connectivity stats module {x} command to check the CRC counter of the corresponding internal links. An example of this is shown here, where the fabric module inserted in slot 22 receives packets with an invalid CRC on iEth56 connected to iEth26 of the line card inserted in slot 7 of the switch. This indicates that corrupted Ethernet frames are received by the fabric module from the line card inserted in slot 7 of the switch.
Nexus9500# show system internal fabric connectivity stats module 22
Internal Link-info Stats Fabriccard slot:22
-------------------------------------------------------------------------------------
FM-Slot FM-Unit FM-iEthLink LC-Slot LC-Unit LC-EthLink MUX CRC
-------------------------------------------------------------------------------------
22 0 iEth09 1 0 iEth01 - 0
22 0 iEth06 1 1 iEth11 - 0
22 0 iEth25 1 2 iEth21 - 0
22 0 iEth26 1 3 iEth31 - 0
22 0 iEth10 2 0 iEth01 - 0
22 0 iEth05 2 1 iEth11 - 0
22 0 iEth23 2 2 iEth21 - 0
22 0 iEth24 2 3 iEth31 - 0
22 0 iEth12 3 0 iEth01 - 0
22 0 iEth11 3 1 iEth11 - 0
22 0 iEth21 3 2 iEth21 - 0
22 0 iEth22 3 3 iEth31 - 0
22 0 iEth14 4 0 iEth01 - 0
22 0 iEth13 4 1 iEth11 - 0
22 0 iEth07 4 2 iEth21 - 0
22 0 iEth08 4 3 iEth31 - 0
22 0 iEth16 5 0 iEth01 - 0
22 0 iEth15 5 1 iEth11 - 0
22 0 iEth01 5 2 iEth21 - 0
22 0 iEth04 5 3 iEth31 - 0
22 0 iEth20 6 0 iEth01 - 0
22 0 iEth17 6 1 iEth11 - 0
22 0 iEth28 6 2 iEth21 - 0
22 0 iEth27 6 3 iEth31 - 0
22 0 iEth19 7 0 iEth01 - 0
22 0 iEth02 7 1 iEth09 - 0
22 0 iEth30 7 2 iEth17 - 0
22 0 iEth29 7 3 iEth25 - 0
22 0 iEth18 8 0 iEth01 - 0
22 0 iEth03 8 1 iEth09 - 0
22 0 iEth32 8 2 iEth17 - 0
22 0 iEth31 8 3 iEth25 - 0
22 1 iEth41 1 0 iEth02 - 0
22 1 iEth38 1 1 iEth12 - 0
22 1 iEth59 1 2 iEth22 - 0
22 1 iEth60 1 3 iEth32 - 0
22 1 iEth42 2 0 iEth02 - 0
22 1 iEth37 2 1 iEth12 - 0
22 1 iEth62 2 2 iEth22 - 0
22 1 iEth61 2 3 iEth32 - 0
22 1 iEth44 3 0 iEth02 - 0
22 1 iEth43 3 1 iEth12 - 0
22 1 iEth64 3 2 iEth22 - 0
22 1 iEth63 3 3 iEth32 - 0
22 1 iEth46 4 0 iEth02 - 0
22 1 iEth45 4 1 iEth12 - 0
22 1 iEth39 4 2 iEth22 - 0
22 1 iEth40 4 3 iEth32 - 0
22 1 iEth48 5 0 iEth02 - 0
22 1 iEth47 5 1 iEth12 - 0
22 1 iEth36 5 2 iEth22 - 0
22 1 iEth33 5 3 iEth32 - 0
22 1 iEth52 6 0 iEth02 - 0
22 1 iEth49 6 1 iEth12 - 0
22 1 iEth57 6 2 iEth22 - 0
22 1 iEth58 6 3 iEth32 - 0
22 1 iEth34 7 0 iEth02 - 0
22 1 iEth51 7 1 iEth10 - 0
22 1 iEth55 7 2 iEth18 - 0
22 1 iEth56 7 3 iEth26 - 1665601166
22 1 iEth50 8 0 iEth02 - 0
22 1 iEth35 8 1 iEth10 - 0
22 1 iEth53 8 2 iEth18 - 0
22 1 iEth54 8 3 iEth26 - 0
Use the slot {x} show hardware internal tah counters asic {y} command on a line card or fabric module to determine if CRC errors are invalid or stomped CRCs. The two register counters that differentiate invalid CRC errors from stomped CRC errors are:
- 02-RX Frm with FCS Err - Indicates a frame with an invalid, but non-stomped CRC was received.
- 16-RX Frm CRC Err(Stomp) - Indicates a frame with a stomped CRC was received.
An example of this is shown here, where corrupted frames are received on the fabric module inserted in slot 22 of the chassis through internal link iEth54 connect to the line card inserted in slot 8of the chassis are received with stomped CRC:
Nexus9500# slot 22 show hardware internal tah counters asic 1
REG_NAME M24,0-100Gx4 M25,0-100Gx4
--------------------------------------------------------------------------------------------------------------------------
02-RX Frm with FCS Err .... ....
03-RX Frm with any Err .... 144912733e
16-RX Frm CRC Err(Stomp) .... 14491277d7
Alternatively, use the show hardware internal errors module {x} command to see ASIC error counters for a specific module. An example of this is shown here.
Note: In this output, the Interface Inbound Errors (CRC,len,Algn Err) counter increments for both invalid CRCs and stomped CRCs, while the Interface Inbound CRC Error Stomped counter increments for only stomped CRCs.
Nexus9500# show hardware internal errors module 22
|------------------------------------------------------------------------|
| Device:Lacrosse Role:MAC Mod:22 |
| Last cleared @ Tue Jul 6 04:10:45 2021
| Device Statistics Category :: ERROR
|------------------------------------------------------------------------|
Instance:0
ID Name Value Ports
-- ---- ----- -----
Instance:1
ID Name Value Ports
-- ---- ----- -----
196635 Interface Inbound Errors (CRC,len,Algn Err) 0000053053264536 27:0
1048603 Interface Inbound CRC Error Stomped 0000053053264535 27:0
After identifying the ingress line card from which the corrupted frames are being received, use the slot {x} show hardware internal tah counters asic {y} or show hardware internal errors module {x} commands in a similar way to identify the ingress interface that the errors are received on, as well as if errors are received as invalid CRCs or stomped CRCs.
A rare scenario is possible where a fabric module or egress line card shows CRC errors on an iEth link, but the connected line card has no signs of ingress CRCs. The root cause of this issue is typically hardware failure of the fabric module. Cisco recommends opening a support case with Cisco TAC to troubleshoot this issue further and replace the fabric module if necessary.
Examples
This section of the document walks through the previous procedure with some examples.
Scenario 1. Physical Interface Receiving Stomped CRCs
This example demonstrates how to identify that CRC errors on a physical interface are stomped CRCs.
Consider this topology:
In this example, purposefully-stomped CRC errors are generated on switch N9K-1 through jumbo-sized 8000 byte ICMP packets sourced from interface SVI 10 (which owns IP address 192.0.2.1) destined to N9K-3's interface SVI 10 (which owns IP address 192.0.2.3), which has an MTU of 1500 bytes. N9K-1, N9K-2, and N9K-3 are all Nexus 93180YC-EX model switches.
N9K-1# ping 192.0.2.3 count 5 packet-size 8000 PING 192.0.2.3 (192.0.2.3): 8000 data bytes Request 0 timed out Request 1 timed out Request 2 timed out Request 3 timed out Request 4 timed out Request 5 timed out --- 192.0.2.3 ping statistics --- 5 packets transmitted, 0 packets received, 100.00% packet loss
In this example, incrementing CRC errors are observed on physical interface Ethernet1/1 of switch N9K-3.
N9K-3# show interface Ethernet1/1
<snip>
Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000/25000 Ethernet, address: 00d7.8f86.2bbe (bia 00d7.8f86.2bbe)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 06:13:44
Last clearing of "show interface" counters 02:55:00
0 interface resets
RX
9 unicast packets 10675 multicast packets 0 broadcast packets
10691 input packets 816924 bytes
7 jumbo packets 0 storm suppression bytes
0 runts 7 giants 7 CRC 0 no buffer
7 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
Step 1. Confirm Incrementing CRCs
Confirm that CRCs are incrementing on physical interface Ethernet1/1 by generating jumbo-sized 8000 byte ICMP packets sourced from N9K-1's interface SVI 10 (which owns IP address 192.0.2.1) destined to N9K-3's interface SVI 10 (which owns IP address 192.0.2.3).
N9K-1# ping 192.0.2.3 count 5 packet-size 8000
PING 192.0.2.3 (192.0.2.3): 8000 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out
Request 5 timed out
--- 192.0.2.3 ping statistics ---
5 packets transmitted, 0 packets received, 100.00% packet loss
N9K-3# show interface Ethernet1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000/25000 Ethernet, address: 00d7.8f86.2bbe (bia 00d7.8f86.2bbe)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 06:52:57
Last clearing of "show interface" counters 03:34:13
0 interface resets
RX
11 unicast packets 13066 multicast packets 0 broadcast packets
13089 input packets 1005576 bytes
12 jumbo packets 0 storm suppression bytes
0 runts 12 giants 12 CRC 0 no buffer
12 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
Step 2. Map Physical Interface to ASIC, MAC Block, and MAC Block Sub-Port
Use the show interface hardware-mappings command on N9K-3 to map physical interface Ethernet1/1 to ASIC number 0, MAC block 4, and MAC block sub-port 0.
N9K-3# show interface hardware-mappings
<snip>
-------------------------------------------------------------------------------------------------------
Name Ifindex Smod Unit HPort FPort NPort VPort Slice SPort SrcId MacId MacSP VIF Block BlkSrcID
-------------------------------------------------------------------------------------------------------
Eth1/1 1a000000 1 0 16 255 0 -1 0 16 32 4 0 1 0 32
Eth1/2 1a000200 1 0 17 255 4 -1 0 17 34 4 2 5 0 34
Eth1/3 1a000400 1 0 18 255 8 -1 0 18 36 4 4 9 0 36
Eth1/4 1a000600 1 0 19 255 12 -1 0 19 38 4 6 13 0 38
Eth1/5 1a000800 1 0 12 255 16 -1 0 12 24 3 0 17 0 24
Step 3. Check Cloud Scale ASIC Registers for CRC-Related Counters
Based on the information from Step 2, you know these facts:
- Physical interface Ethernet1/1 is mapped to ASIC number 0.
- Physical interface Ethernet1/1 is mapped to MAC block sub-port 0 of MAC block 4
- Because N9K-3 is a top-of-rack Nexus 93180YC-EX model switch, you know that the only possible line card slot number is 1
- From the output of show interface gathered in Step 1, you know the speed of physical interface Ethernet1/1 is 10G.
Using this information, you can use the slot 1 show hardware internal tah counters asic 0 command to view the ASIC register counters for all physical interfaces. Specifically, you are looking for ASIC register counters associated with M4,0-10G.
N9K-3# slot 1 show hardware internal tah counters asic 0 <snip> *************** PER MAC/CH SRAM COUNTERS **************** REG_NAME M4,0-10G M4,2-10G M4,4-10G M4,6-10G M5,0-40Gx4 M6,0-40Gx4 M7,0-40Gx4 M8,0-10G ---------------------------------------------------------------------------------------------------------------------------------- 02-RX Frm with FCS Err .... .... .... .... .... .... .... .... 16-RX Frm CRC Err(Stomp) c .... .... .... .... .... .... ....
You can see a non-zero hexadecimal value of 0xc for register 16, which indicates frames with a stomped CRC were received on this physical interface. You can use the dec 0xc command to translate this to a decimal value of 12, which matches the number of CRC errors on physical interface Ethernet1/1.
N9K-3# dec 0xc
12
Conclusion
You have confirmed that N9K-3 is receiving frames with a stomped CRC on physical interface Ethernet1/1. This means that the device on the remote side of the Ethernet1/1 link (in this case, N9K-2) is stomping the CRC of these frames; the root cause of the malformed frames is not the link directly connected to Ethernet1/1, but is further downstream. Additional troubleshooting can be performed on the downstream network device to determine the source of these malformed frames.
Scenario 2. Physical Interface Received Malformed Frames with Invalid CRC
This example demonstrates how to identify that CRC errors on a physical interface are incrementing due to malformed frames caused by a physical layer issue on a directly-connected link.
Consider this topology:
In this example, a traffic generator connected to physical interface Ethernet1/40 of switch N9K-1 is purposefully generating frames with an incorrect CRC. This simulates a physical layer issue on the link connected to Ethernet1/40, such as a faulty transceiver or damaged cable. N9K-1 receives these frames, recognizes that the CRC is invalid, and increments the CRC error counter on the Ethernet1/40 physical interface. N9K-1 is a Nexus 93180YC-EX model switch.
N9K-1# show interface Ethernet1/40
Ethernet1/40 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000/25000 Ethernet, address: 00d7.8f86.2bbe (bia 00d7.8f86.2c02)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 06:13:44
Last clearing of "show interface" counters 02:55:00
0 interface resets
RX
1710 unicast packets 9873 multicast packets 0 broadcast packets
11583 input packets 886321 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 1683 CRC 0 no buffer
1683 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
Step 1. Confirm Incrementing CRCs
Confirm that CRCs are incrementing on physical interface Ethernet1/40 of N9K-1 through the show interface or show interface counters non-zero commands.
N9K-1# show interface Ethernet1/40
<snip>
Ethernet1/40 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000/25000 Ethernet, address: 00d7.8f86.2bbe (bia 00d7.8f86.2c02)
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
admin fec state is auto, oper fec state is off
Last link flapped 06:13:44
Last clearing of "show interface" counters 02:55:00
0 interface resets
RX
14055 unicast packets 9873 multicast packets 0 broadcast packets
23928 input packets 1676401 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 14028 CRC 0 no buffer
14028 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
N9K-1# show interface counters errors non-zero
<snip>
--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
Eth1/40 26373 26373 0 26373 0 0
Step 2. Map Physical Interface to ASIC, MAC Block, and MAC Block Sub-Port
Use the show interface hardware-mappings command on N9K-1 to map physical interface Ethernet1/40 to ASIC number 0, MAC block 10, MAC block sub-port 6.
N9K-1# show interface hardware-mappings <snip> --------------------------------------------------------------------------------------------- Name Ifindex Smod Unit HPort FPort NPort VPort Slice SPort SrcId MacId MacSP VIF Block BlkSrcID --------------------------------------------------------------------------------------------- Eth1/38 1a004a00 1 0 45 255 148 -1 1 5 10 10 2 149 0 10 Eth1/39 1a004c00 1 0 46 255 152 -1 1 6 12 10 4 153 0 12 Eth1/40 1a004e00 1 0 47 255 156 -1 1 7 14 10 6 157 0 14 Eth1/41 1a005000 1 0 76 255 160 -1 1 36 64 17 0 161 0 64 Eth1/42 1a005200 1 0 77 255 164 -1 1 37 66 17 2 165 0 66
Step 3. Check Cloud Scale ASIC Registers for CRC-Related Counters
Based on the information from Step 2, you know these facts:
- Physical interface Ethernet1/40 is mapped to ASIC number 0.
- Physical interface Ethernet1/40 is mapped to MAC block sub-port 6 of MAC block 10.
- Because N9K-1 is a top-of-rack Nexus 93180YC-EX model switch, you know that the only possible line card slot number is 1.
- From the output of show interface gathered in Step 1, you know the speed of physical interface Ethernet1/40 is 10G.
Using this information, you can use the slot 1 show hardware internal tah counters asic 0 command to view the ASIC register counters for all physical interfaces. Specifically, you are looking for ASIC register counters associated with M10,6-10G.
N9K-1# slot 1 show hardware internal tah counters asic 0 *************** PER MAC/CH SRAM COUNTERS **************** REG_NAME M8,2-10G M8,4-10G M8,6-10G M9,0-40Gx4 M10,0-10G M10,2-10G M10,4-10G M10,6-10G ------------------------------------------------------------------------------------------------------------------------------------- 02-RX Frm with FCS Err .... .... .... .... .... .... .... 973e 16-RX Frm CRC Err(Stomp) .... .... .... .... .... .... .... ....
You can see a non-zero hexadecimal value of 0x973e for register 2, which indicates frames with an invalid, but non-stomped CRC were received on this physical interface.
You can use the dec 0x973e command to translate this to a decimal value of 38,718, which matches (or is less than, since the CRCs are constantly incrementing) the number of CRC errors on physical interface Ethernet1/40.
N9K-1# dec 0x973e
38718
Conclusion
You have confirmed that N9K-1 is receiving frames with an invalid, but non-stomped CRC on physical interface Ethernet1/40. This means that the link directly connected to Ethernet1/40 (or the device on the remote end of the link) is the most likely source of the malformed frames. Further troubleshooting can be performed on the physical layer of this link to isolate the root cause of the malformed frames (such as checking for damaged cabling, replacing the current transceivers with known good transceivers, and so on).
Scenario 3. Nexus 9500 iEth CRC Errors Syslog
This example demonstrates how to identify the source of CRC errors on an iEth internal link when a syslog reporting errors on an internal interface is generated by a Nexus 9500 series switch. An example of this syslog is shown here.
Nexus9500# show logging logfile
<snip>
2021 Jul 9 05:51:19 Nexus9500 %DEVICE_TEST-SLOT22-3-INTERNAL_PORT_MONITOR_CRC_ERRORS_DETECTED: Module 22 received tx errors on internal interface ii22/1/56 since last run TXErr=36836897 TotalTXErr=50781987904
This syslog indicates that errors were detected on the iEth56 internal link of the fabric module inserted in slot 22 of the switch.
Step 1. Map iEth Link on Fabric Module to Connected Line Card
Use the show system internal fabric connectivity stats module {x} command to identify which line card the affected iEth internal link connects to. In this example, iEth56 of the fabric module inserted in slot 22 of the switch has errors. An example of this is shown here, where iEth56 of the fabric module inserted in slot 22 is connected to iEth26 of the line card inserted in slot 7 of the switch.
Nexus9500# show system internal fabric connectivity stats module 22 | include Eth56|FM-Slot
FM-Slot FM-Unit FM-iEthLink LC-Slot LC-Unit LC-EthLink MUX CRC
22 1 iEth56 7 3 iEth26 - 603816174
Use the show system internal fabric link-state module {x} command to locate the ASIC instance and MAC identifier associated with the fabric module's iEth56 internal link. An example of this is shown here, where the ASIC instance is 1 and the MAC identifier is 27.
Nexus9500# show system internal fabric link-state module 22 | include MAC|iEth56
[FM] [ INST:SLI:MAC:GLSRC] [IETH] [ST] <========> [LC] [ INST:SLI:MAC:GLSRC] [IETH] [ST]
[22] [ 1 : 4 : 27 : 0x18] [iEth56] [UP] <========> [ 7] [ 3 : 1 : 9 : 0x0] [iEth26] [UP]
Step 2. Check if CRCs Received on iEth Link are Invalid or Stomped
The previous step shows that our ASIC instance identifier is 1 and our MAC identifier is 27 for iEth56 connected to the fabric module inserted in slot 22. Use the slot {x} show hardware internal tah counters asic {y} command to identify whether the CRCs reported by the syslog are invalid CRCs or stomped CRCs. An example of this is shown here, where the M27,0-100Gx4 column is associated with our MAC identifier of 27 and indicates the CRCs are stomped.
Nexus9500# slot 22 show hardware internal tah counters asic 1
REG_NAME M27,0-100Gx4
-----------------------------------------
02-RX Frm with FCS Err ....
16-RX Frm CRC Err(Stomp) be9cb9bd6
Alternatively, use the show hardware internal errors module {x} command to glean this same information. An example of this is shown here.
Nexus9500# show hardware internal errors module 22 | include CRC|Stomp|Inst
Instance:1
196635 Interface Inbound Errors (CRC,len,Algn Err) 0000051587084851 27:0
1048603 Interface Inbound CRC Error Stomped 0000051587084850 27:0
Recall that in this output, the Interface Inbound Errors (CRC,len,Algn Err) counter increments for both invalid CRCs and stomped CRCs, while the Interface Inbound CRC Error Stomped counter increments for only stomped CRCs.
Step 3. Track Source of Frames with Invalid CRCs on Ingress Line Card
You now know that the CRCs entering the fabric module inserted in slot 22 of the switch are entering the switch from the line card inserted in slot 7. With this information, you can use the show interface counters errors module {x} non-zero command to identify non-zero CRC counters on interfaces belonging to the relevant line card. An example of this is shown here.
Nexus9500# show interface counters errors module 7 non-zero
<snip>
--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
Eth7/32 0 0 0 1195309745 0 0
You can repeat Step #2 of this scenario on the relevant line card to verify whether the line card is receiving invalid CRCs or stomped CRCs.
Nexus9500# show hardware internal errors module 7 | include ignore-case CRC|Stomp|Inst
Instance:3
196619 Interface Inbound Errors (CRC,len,Algn Err) 0000051801011139 11:0
1048587 Interface Inbound CRC Error Stomped 0000051801011140 11:0
Use the show interface hardware-mappings command to identify the front-panel port that the MacId:MacSP value of 11:0 in the previous output is mapped to. An example of this is shown here, where 11:0 maps to front-panel port Eth7/32.
Nexus9500# show interface hardware-mappings | include Name|Eth7
<snip>
Name Ifindex Smod Unit HPort FPort NPort VPort Slice SPort SrcId MacId MacSP VIF Block BlkSrcID
Eth7/1 1a300000 25 0 16 255 0 -1 0 16 32 4 0 1 0 32
Eth7/2 1a300200 25 0 12 255 4 -1 0 12 24 3 0 5 0 24
Eth7/3 1a300400 25 0 8 255 8 -1 0 8 16 2 0 9 0 16
Eth7/4 1a300600 25 0 4 255 12 -1 0 4 8 1 0 13 0 8
Eth7/5 1a300800 25 0 60 255 16 -1 1 20 40 14 0 17 0 40
Eth7/6 1a300a00 25 0 56 255 20 -1 1 16 32 13 0 21 0 32
Eth7/7 1a300c00 25 0 52 255 24 -1 1 12 24 12 0 25 0 24
Eth7/8 1a300e00 25 0 48 255 28 -1 1 8 16 11 0 29 0 16
Eth7/9 1a301000 26 1 12 255 32 -1 0 12 24 3 0 33 0 24
Eth7/10 1a301200 26 1 8 255 36 -1 0 8 16 2 0 37 0 16
Eth7/11 1a301400 26 1 4 255 40 -1 0 4 8 1 0 41 0 8
Eth7/12 1a301600 26 1 0 255 44 -1 0 0 0 0 0 45 0 0
Eth7/13 1a301800 26 1 60 255 48 -1 1 20 40 14 0 49 0 40
Eth7/14 1a301a00 26 1 56 255 52 -1 1 16 32 13 0 53 0 32
Eth7/15 1a301c00 26 1 52 255 56 -1 1 12 24 12 0 57 0 24
Eth7/16 1a301e00 26 1 48 255 60 -1 1 8 16 11 0 61 0 16
Eth7/17 1a302000 27 2 16 255 64 -1 0 16 32 4 0 65 0 32
Eth7/18 1a302200 27 2 12 255 68 -1 0 12 24 3 0 69 0 24
Eth7/19 1a302400 27 2 8 255 72 -1 0 8 16 2 0 73 0 16
Eth7/20 1a302600 27 2 4 255 76 -1 0 4 8 1 0 77 0 8
Eth7/21 1a302800 27 2 60 255 80 -1 1 20 40 14 0 81 0 40
Eth7/22 1a302a00 27 2 56 255 84 -1 1 16 32 13 0 85 0 32
Eth7/23 1a302c00 27 2 52 255 88 -1 1 12 24 12 0 89 0 24
Eth7/24 1a302e00 27 2 48 255 92 -1 1 8 16 11 0 93 0 16
Eth7/25 1a303000 28 3 12 255 96 -1 0 12 24 3 0 97 0 24
Eth7/26 1a303200 28 3 8 255 100 -1 0 8 16 2 0 101 0 16
Eth7/27 1a303400 28 3 4 255 104 -1 0 4 8 1 0 105 0 8
Eth7/28 1a303600 28 3 0 255 108 -1 0 0 0 0 0 109 0 0
Eth7/29 1a303800 28 3 60 255 112 -1 1 20 40 14 0 113 0 40
Eth7/30 1a303a00 28 3 56 255 116 -1 1 16 32 13 0 117 0 32
Eth7/31 1a303c00 28 3 52 255 120 -1 1 12 24 12 0 121 0 24
Eth7/32 1a303e00 28 3 48 255 124 -1 1 8 16 11 0 125 0 16
Conclusion
You have confirmed that the Nexus 9500 is receiving frames with a stomped CRC on physical interface Ethernet7/32. This means that the device on the remote side of the Ethernet7/32 link is stomping the CRC of these frames; the root cause of the malformed frames is not the link directly connected to Ethernet7/32, but is further downstream. Additional troubleshooting can be performed on the downstream network device to determine the source of these malformed frames.
Scenario 4. Track Source of Invalid CRC Frames with Egress Interface
This example demonstrates how to track the source of frames with invalid CRCs on a Nexus 9500 switch when an upstream switch reports that the Nexus 9500 is generating frames with stomped CRCs. In this scenario, the upstream switch is connected via front-panel port Ethernet8/9.
Step 1. Identify Fabric Module Sending Invalid CRC Frames to Egress Line Card
You know that the egress interface sending frames with stomped CRCs towards the upstream switch is Ethernet8/9. First, you need to determine the fabric module that is sending frames with stomped CRCs to the line card inserted in slot 8 of the chassis. You start this process with the show hardware internal errors module {x} command. An example of this is shown here.
Nexus9500# show hardware internal errors module 8 | i CRC|Inst
<snip>
Instance:1
196617 Interface Inbound Errors (CRC,len,Algn Err) 0000091499464650 9:0
1048585 Interface Inbound CRC Error Stomped 0000091499464651 9:0
MacID:MacSP 9:0 in the prevous output can be mapped to the source fabric module with the show system internal fabric link-state module 8 command. An example of this is shown here.
Nexus9500# show system internal fabric link-state module 8
cli : mod = 8
module number = 8
========================================================================================================================
Module number = 8
========================================================================================================================
[LC] [ INST:SLI:MAC:GLSRC] [IETH] [ST] <========> [FM] [ INST:SLI:MAC:GLSRC] [IETH] [ST]
========================================================================================================================
...
[ 8] [ 1 : 1 : 9 : 0x0] [iEth10] [UP] <========> [22] [ 1 : 0 : 4 : 0x20] [iEth35] [UP]
You see that MAC identifier 9 on the line card inserted in slot 8 is mapped to the fabric module inserted in slot 22 of the chassis. You expect to see CRC errors on internal link iEth10. You can validate this with the show system internal fabric connectivity stats module 8 command. An example of this is shown here.
Nexus9500# show system internal fabric connectivity stats module 8
Internal Link-info Stats Linecard slot:8
-------------------------------------------------------------------------------------
LC-Slot LC-Unit LC-iEthLink MUX FM-Slot FM-Unit FM-iEthLink CRC
-------------------------------------------------------------------------------------
8 0 iEth01 - 22 0 iEth18 0
8 0 iEth02 - 22 1 iEth50 0
8 0 iEth03 - 23 0 iEth18 0
8 0 iEth04 - 23 1 iEth50 0
8 0 iEth05 - 24 0 iEth18 0
8 0 iEth06 - 24 1 iEth50 0
8 0 iEth07 - 26 0 iEth18 0
8 0 iEth08 - 26 1 iEth50 0
8 1 iEth09 - 22 0 iEth03 0
8 1 iEth10 - 22 1 iEth35 1784603561
Step 2. Map iEth Link on Fabric Module to the Connected Linecard and Check for Stomped CRCs
Next, you use the same process as in Scenario 3 by checking the iEth internal links that receive CRCs, whether those CRCs are stomped or not according to the fabric module's ASIC, and what line card is connected to the fabric module's iEth internal link. An example of this is shown here using the show system internal fabric connectivity stats module {x} command, the show hardware internal errors module {x} command, and the show system internal fabric link-state module {x} command, respectively.
Nexus9500# show system internal fabric connectivity stats module 22
Internal Link-info Stats Fabriccard slot:22
-------------------------------------------------------------------------------------
FM-Slot FM-Unit FM-iEthLink LC-Slot LC-Unit LC-EthLink MUX CRC
22 1 iEth56 7 3 iEth26 - 1171851894
Nexus9500# show hardware internal errors module 22 | i CRC|Stomp|Inst
Instance:1
196635 Interface Inbound Errors (CRC,len,Algn Err) 0000054593935847 27:0
1048603 Interface Inbound CRC Error Stomped 0000054593935846 27:0
Nexus9500# show system internal fabric link-state module 22 | i MAC|iEth56
[FM] [ INST:SLI:MAC:GLSRC] [IETH] [ST] <========> [LC] [ INST:SLI:MAC:GLSRC] [IETH] [ST]
[22] [ 1 : 4 : 27 : 0x18] [iEth56] [UP] <========> [ 7] [ 3 : 1 : 9 : 0x0] [iEth26] [UP]
Step 3. Track Source of Frames with Invalid CRCs on Ingress Module
After determining the ingress line card (in this scenario, the line card inserted in slot 7 connected by iEth26 to iEth56 of the fabric module inserted in slot 22), you identify which ingress port the corrupted frames enter the switch. This is done with the show interface counters errors module {x} non-zero command. The output of the show hardware internal errors module {x} command and the show interface hardware-mappings command can validate whether the received frames are invalid or stomped CRCs.
An example of this is shown here, where corrupted frames are entering the switch through front-panel interface Ethernet7/32.
Nexus9500# show interface counters errors module 7 non-zero
<snip>
--------------------------------------------------------------------------------
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
--------------------------------------------------------------------------------
Eth7/32 0 0 0 4128770335 0 0
--------------------------------------------------------------------------------
Port Stomped-CRC
--------------------------------------------------------------------------------
Eth7/32 4129998971
Nexus9500# show hardware internal errors module 7 | i i CRC|Stomp|Inst
<snip>
Instance:3
196619 Interface Inbound Errors (CRC,len,Algn Err) 0000054901402307 11:0
1048587 Interface Inbound CRC Error Stomped 0000054901402308 11:0
Nexus9500# show interface hardware-mappings | i Name|Eth7
<snip>
Name Ifindex Smod Unit HPort FPort NPort VPort Slice SPort SrcId MacId MacSP VIF Block BlkSrcID
...
Eth7/32 1a303e00 28 3 48 255 124 -1 1 8 16 11 0 125 0 16
Conclusion
You have confirmed that the Nexus 9500 is receiving frames with a stomped CRC on physical interface Ethernet7/32. This means that the device on the remote side of the Ethernet7/32 link is stomping the CRC of these frames; the root cause of the malformed frames is not the link directly connected to Ethernet7/32, but is further downstream.
Additional troubleshooting can be performed on the downstream network device to determine the source of these malformed frames.
Related Information
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
3.0 |
08-Nov-2023 |
Updated Title, Introduction, Tech Content Terminology, SEO, Style Requirements, Machine Translation and Formatting. |
2.0 |
03-Oct-2021 |
Update Applicable Hardware section to include new hardware. |
1.0 |
13-Nov-2020 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)