Troubleshoot Network Connectivity, Drops and CRC Errors

Available Languages

Download Options

PDF (31.8 KB)
View with Adobe Reader on a variety of devices
ePub (87.2 KB)
View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle) (79.9 KB)
View on Kindle device or Kindle app on multiple devices

Updated:October 6, 2023

Document ID:215449

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Background Information

Reasons for Bad Frames and CRC Errors

Forwarding Mode Behavior (Cut-Through or Store-and-Forward)

Main Forwarding ASICs Commands for UCS FIs, IOMs, and VIC Cards

Related Information

Introduction

This document describes commands used to troubleshoot network connectivity, drops, and CRC errors within different UCS, FIs, IOMs, and VIC adapters.

Prerequisites

Requirements

This document assumes that you have knowledge of these topics:

Cisco Unified Computing Systems (UCS) Virtual Interface Card (VIC)
Cisco UCS B-Series and C-Series servers
Cisco UCS Fabric extender I/O Module (IOM)
Cisco UCS Fabric Interconnect (FI)
Cisco Unified Computing System Manager (UCSM)
Cisco Unified Computing System Manager (UCSM) Command Line Interface (CLI)
Intersight Management Mode (IMM)
Cut-through and store-and-forward switches
Stomps

Components Used

The information in this document is based on these software and hardware versions:

Cisco UCS Manager version 2. x and later
Cisco UCS 6200, 6300, 6400, and 6500 Series Fabric Interconnect
Cisco UCS 2200, 2300, and 2400 Series Fabric extender I/O Module
Cisco UCS 1200, 1300, 1400, and 1500 Series Virtual Interface Card (VIC)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The Cisco UCS Fabric Interconnect is a cut-through switch like the Cisco Nexus 5000 Series Switches. It forwards bad frames like good frames. Bad frames get dropped by the destination server or when they pass through a piece of network equipment that is not cut-through. A CRC check is performed at the end of the frame to determine whether or not a frame has become corrupted. Some switches can drop the frame once they detect a frame is corrupted. Cut through switches make the forwarding decision before they can perform the CRC check. Because of these frames that fail, a CRC check can still be switched by a cut-through switch. Other switches like the N7K are store and forward switches. Store and forward switches look at the entire frame before they make a forwarding decision. A store and forward switch would drop a frame that failed a CRC check. To understand the architecture (cut-through vs store-forward) of the switches in the topology is imperative. Many times, you need to work your way backward to the source of the CRC error. Refer to this article for more details about cut-through and store-and-forward switches: Cloud Networking Switches

Note: . If you experience CRC errors on an interface, it does not mean that the interface is the source of the problem.

Reasons for Bad Frames and CRC Errors

Some of the reasons when you get bad frames and CRC errors can be:

Bad physical connection; transceiver, copper, fiber, adapter, port expander, and so on.
MTU Violation
Received bad CRC stomped from neighboring cut-through switch.

Forwarding Mode Behavior (Cut-Through or Store-and-Forward)

UCS Fabric Interconnects (similar to Nexus 5000) utilizes both cut-through and store-and-forward switching. Forwarding mode is dependent on the ingress and egress data rate, as shown in Table 1.

Note: Cut-through switching can be performed only when the ingress data rate is equivalent to or faster than the egress data rate.

Table 1 - Forwarding Mode Behavior (Cut-Through or Store-and-Forward) for UCS Fabric Interconnect

Ingress/ Source Interface	Egress/ Destination Interface	Forwarding mode
10 Gigabit Ethernet	10 Gigabit Ethernet	Cut-through
10 Gigabit Ethernet	1 Gigabit Ethernet	Cut-through
1 Gigabit Ethernet	1 Gigabit Ethernet	Store-and-forward
1 Gigabit Ethernet	10 Gigabit Ethernet	Store-and-forward
10 Gigabit Ethernet	40 Gigabit Ethernet	Store-and-forward
40 Gigabit Ethernet	10 Gigabit Ethernet	Cut-through
40 Gigabit Ethernet	40 Gigabit Ethernet	Cut-through
40 Gigabit Ethernet	100 Gigabit Ethernet	Store-and-forward
100 Gigabit Ethernet	40 Gigabit Ethernet	Cut-through
100 Gigabit Ethernet	100 Gigabit Ethernet	Cut-through
FCoE	Fibre Channel	Cut-through
Fibre Channel	FCoE	Store-and-forward
Fibre Channel	Fibre Channel	Store-and-forward
FCoE	FCoE	Cut-through

Main Forwarding ASICs Commands for UCS FIs, IOMs, and VIC Cards

Tables 2 and 3 show the different commands that can be run from the different management endpoints in UCS to determine where the drops are coming from and why they are occurring.

In addition to the ASIC-specific commands mentioned in Table 2, these commands can be run from the UCS FI NXOS shell to look for errors at the receive direction of interfaces:

show interface counters errors

Table 2 - Main Forwarding ASICs commands for UCS FIs and IOMs

UCS FI / IOM	Main FW ASIC Name	Commands	Purpose
Cisco UCS Fabric Interconnects
Cisco UCS 6200 Series (Gen 2 FIs 62xx)	Carmel	(nxos)# show hardware internal carmel	This command shows the Carmel ASIC internals and driver information. 5th column shows how many ports/interfaces are mapped to each Carmel ASIC.
		(nxos)# show hardware internal carmel all-ports	This command shows driver information for all ports and front panel physical ports to ASIC mapping.
		(nxos)# show hardware internal carmel crc	This command shows information if any frames were received or transmitted with CRC errors or stomped for all ports.
		(nxos)# show platform fwm info asic-errors X	This command shows non zero Carmel drop reasons error registers (where X is the Carmel ASIC number from 0-4).
		(nxos)# show platform fwm info pif e1/X \| grep asic	Use this command and you can map your interface to the Carmel ASIC ID “global_asic_num” (where X is the interface number).
		(nxos)# show platform fwm info pif e1/X \| grep drop	This command shows the number of frames, and it filters for the drop counter for a certain interface (where X is the interface number).
		(nxos)# show hardware internal carmel all-ports detail \| egrep -i "Carmel port\|crc\|frame_error"	This command filters for CRC and frame error counters for all ports.
Cisco UCS 6300 Series (Gen 3 FIs 63xx)	Trident2 (Broadcom ASIC)	(nxos)# show hardware internal bcm-usd info port-info	This command shows the mapping between each physical port to a front port on the Broadcom ASIC and this mapping is different between 6332 and 6332-16UP FIs.
Cisco UCS 6300 Series (Gen 3 FIs 63xx)	Trident2 (Broadcom ASIC)	(nxos)# show hard internal interface indiscard-stats front-port X	This command shows port internal discard counters for a certain front port on the Broadcom ASIC after the mapping is done that uses the previous command.
Cisco UCS Mini (6324 Fabric Interconnect)	Monticello ASIC	(nxos)# show hardware internal mtc-usd port-status	This command shows the status of the ports for the Monticello ASIC. (nxos)# show hardware internal inband-mtc ? ASIC Show Monticello ASIC information info. Show Monticello inband driver info stats. Show Monticello inband driver statistics. Please note KRs mapping for the blades within UCS Mini uses a different Port Mapping compared to a chassis with UCS IOMs, refer to TAC for more details.
Cisco UCS 6400 (Gen 4 FIs 64xx)	Homewood ASIC	FI # connect nxos (nx-os)# show hardware internal interface asic counters module 1	This command shows the reason for the forwarding drops if reported on the interface.
		FI # attach module 1 module-1# show hardware internal tah counters asic 0	This command shows the different counters of information tha use ASIC library. There is only one ASIC within this UCS Fabric interconnect model so always ASIC number 0.
		FI # attach module 1 module-1# show hardware internal tah drop-reason counters module 0	This command shows the drop reasons and number of dropped packets.
Cisco UCS 64108 Gen 4 FIs	Cisco ASIC Heavenly	FI # connect nxos (nx-os)# show hardware internal interface asic counters module 1	This command shows the reason for the forwarding drops if reported on the interface
		(nxos) # attach module 1 module-1# show hardware internal tah counters asic 0	This command shows the different counters of information that use the ASIC library. There is only one ASIC within this UCS Fabric interconnect model so always ASIC number 0.
		FI # attach module 1 module-1# show hardware internal tah drop-reason counters module 0	This command shows the reason for the forwarding drops if reported on the interface.
Cisco UCS 6500 Gen 5 FIs	Cisco ASIC Heavenly	FI # connect nxos (nx-os)# show hardware internal interface asic counters module 1	This command shows the reason for the forwarding drops if reported on the interface
		(nxos) # show hardware internal errors module 1	This command shows the output Drop Packets for the module on the ports.
		(nxos) # attach module 1 module-1# show hardware internal tah counters asic 0 FI # attach module 1 module-1# show hardware internal tah drop-reason counters module 0	This command shows the different counters of information that use the ASIC library. This second command shows the drop reasons and number of dropped packets.³
Cisco UCS Fabric extender I/O Modules (IOMs)
Cisco UCS 2200 IOM (Gen 2)	Woodside	FI # connect IOM <chassis ID> fex-1# show platform software woodside sts	This command shows the interface status of the HIFs and NIFs within the Woodside ASIC and which HIFs are used by each blade. Note: There are two numberings for the HIFs, one is used when you troubleshoot from the IOM (after you connect to IOM) and the other is used when you troubleshoot the same HIF and run the commands from UCSM nxos. For example, blade 1 uses HIF numbers 28-31. You can use these numbers after you connect to IOM and run the related commands to that HIF. These correspond to Eth1/1/1 - 4 from UCSM NXOS as per the shown FEX detail.
		FI # connect IOM <chassis ID> fex-1# show platform software woodside rate	This command shows the packet rates for active HIF or NIF ports.
		FI # connect IOM <chassis ID> fex-1# show platform software woodside rmon 0 [NIx/HIx] For example, you can filter some error counters using grep for all NIFs as below: fex-1# show platform software woodside rmon 0 nif_all \| egrep -i	This command shows the received and transmitted packet sizes for a certain HIF or NIF and packet types like unicast, broadcast, or multicast. The Diff column is useful while you do live troubleshooting as it resets after each time you run the command to show you if packets are incrementing when you run the command again. You can also check if the Diff column shows new packets for these: RX_CRC_NOT_STOMPED RX_CRC_STOMPED TX_FRM_ERROR
		FI # connect IOM <chassis ID> fex-1# show platform software woodside drops 0 [NIx/HIx]	This command shows the drop counters for a certain NIF or HIF.
		FI # connect IOM <chassis ID> fex-1# show platform software woodside oper	This command shows the administrative control, MAC, and physical status, in addition to detected SFPs within the NIFs.
		FI # connect iom <chassis ID> fex-1# show platform software woodside sfp 0 ni0 fex-1# show platform software woodside sfp 0 ni1 fex-1# show platform software woodside sfp 0 ni2 fex-1# show platform software woodside sfp 0 ni3	This command shows the transceiver details within the woodside IOM NIF ports.
Cisco UCS 2300 IOM (Gen 3) and Cisco UCS 2300 IOM version 2 (UCS-IOM-2304V2)	Tiburon (Broadcom ASIC)	# connect IOM <chassis ID> Fex-1# show platform software tiburon sts	This command shows the interface status of the HIFs and NIFs within the Tibrun ASIC and which HIFs are used by each blade. Tibrun ASIC comes from the 2248 FEX which has 48 HIF ports, so for UCS, there are some unused ports on the ASIC (NI0-7 and HI0-9 are unused). Note: in regards to the 40G backplane ports within Gen 3 IOMs, HIF status can normally be with the 40 Gig primary ports marked as UP, and the 40 Gig member ports are marked Down.
		# connect IOM <chassis ID> fex-1# show platform software tiburon rate	This command shows the packet rates for active HIF or NIF ports.
		FI # connect IOM <chassis ID> fex-1# show platform software tiburon rmon 0 [NIx/HIx] For example, you can filter some error counters using grep for all NIFs as shown: fex-1# show platform software tiburon rmon 0 nif_all \| egrep -i 'crc\|ni\|stomp\|pause\|err'	This command shows the received and transmitted packet sizes for a certain HIF or NIF and packet types like unicast, broadcast, or multicast. The Diff column is useful while you do live troubleshooting as it resets after each time you run the command to show you if any new packets are coming when you run the command again. You can also check if the Diff column shows new packets for these: RX_CRC_NOT_STOMPED RX_CRC_STOMPED TX_FRM_ERROR
Cisco UCS 2408 (fourth-generation I/O Module) "Summerville" UCS-IOM-2408	Sundown	FI # connect iom <chassis ID> fex-1# show hardware internal tah sts	This command shows the interface status of the HIFs and NIFs within the Tahoe ASIC and which HIFs are used by each blade.
		fex-1# show hardware internal tah sts detail	This command shows the NXOS to HIF port mapping, link-state, and operational speed.
		fex-1# show hardware internal tah counters asic 0 nxos-port ? <0-44> Nxos-port num 0-31 hif/35 bif/36-43 nif	This command shows the detailed per-port counters The detailed interface counters can be viewed by referring to the NXOS port number. NXOS Ports 0-31 correspond to 32 HIF Ports NXOS ports 36-43 correspond to the 8 NIF ports.

1 End-of-Sale and End-of-Life Announcement for the Cisco UCS 6200 Series Fabric Interconnects: Cisco UCS 6200 Series Fabric Interconnects

2 End-of-Sale and End-of-Life Announcement for the Cisco UCS 2100 Series IO Modules: Cisco UCS 2200 Series IO Modules

3 Mode details on some columns of show hardware internal carmel crc command:

MM rx CRC = CRC on this link; Problem is L1 issue; Check eye height; shut, no shut; replace cable;
MM Rx Stomp = STOMP on the remote switch; Go check the same output on the switch across this link;
FI Rx Stomp = If MM Rx CRC and MM Rx Stomp are blank; L2/policy violation, most commonly MTU violation; Check QoS MTU settings.

4 Sometimes to see the Port mapping on FI 6400/6500 the attach module does not work, to have this addressed please contact a Cisco TAC Engineer.

Table 3 - The main commands to troubleshoot connectivity, drops, and CRC errors for Cisco UCS VIC cards.

UCS VIC Generation	Example of the VIC card model	Commands	Purpose
Cisco UCS 1200 VIC (Gen 2)	an example is the 1225 VIC, 1240 VIC, 1280 VIC, etc	Blades example: FI# connect adapter 1/1/1 adapter 1/1/1 # connect adapter 1/1/1 (top):1# show-log adapter 1/1/1 (top):1# attach-mcp adapter (mcp):1# uifportstatus adapter (mcp):3# dcem-macstats 0 <<<< Stats for port-1 adapter (mcp):3# dcem-macstats 1 <<<< Stats for port-2 adapter 1/1/1 (mcp):1# vnic adapter 1/1/1 (mcp):1# lifstats For Standalone C-Series UCS: # scope chassis /chassis # show adapter (get the PCIe slot #) /chassis # connect debug-shell <PCIe slot #> (this command can only work when Server is powered on) adapter (top):1# attach-mcp adapter (mcp):1# uifportstatus	These commands can be run after connecting to the adapter of a Cisco UCS B or C series servers. The macstats command gives information about the status of the physical ports, packet sizes, and if there are any stomped or non-stomped frames received.
Cisco UCS 1300 VIC (Gen 3)	an example is the 1380 VIC
Cisco UCS 1400 VIC (Gen 4)	Example is: (VIC1440): PCIe-based mLOM card for M5 blades (UCSB-MLOM-40G-04) · (VIC1480): PCIe based MEZZ card for M5 blades (UCSB-VIC-M84-4P) · (VIC1455): PCIe card for M5 Rack servers (UCSC-PCIE-C25Q-04) · (VIC1457): PCIe based mLOM card for M5 Rack servers (UCSC-MLOM-C25Q-04)	- Check PCIe Link status adapter (top):1# attach-mcp adapter (mcp):1# pcie_links pp,pps type link config link status state 0,0 host gen3x16 gen3x16 UP adapter (mcp):2# exit - Check the Ethernet Link status adapter (top):2# attach-mcp adapter (mcp):1# uifportstatus ASIC Port UIF Port State Speed 0 0 UP 25g 0 1 UP 25g 1 0 UP 25g 1 1 UP 25g adapter (mcp):2# exit - Check Ethernet error counters adapter (top):3# attach-macd adapter (macd):1# macstats 0 DELTA TOTAL DESCRIPTION 112 112 Rx good packets 112 112 Rx total received packets 14574 14574 Rx bytes for good packets 14574 14574 Rx bytes 104 104 Rx multicast frames <snip>	These commands can be run after connecting to the adapter of a Cisco UCS B or C series servers deployed with 4th Gen VIC adapter.
Cisco UCS 1500 VIC (Gen 5)	Example is: (VIC15231): PCIe-based mLOM card for X210 (UCSX-ML-V5D200G) (VIC15420): PCIe based mLOM card for X210C M6/M7 (UCSX-ML-V5Q50G) (VIC15422): PCIe based MEZZ card for X210C M6/M7 (UCSX-ME-V5Q50G) (VIC15428): PCIe based mLOM card for M6/M7 Rack servers (UCSC-M-V5Q50G) (VIC15238): PCIe based mLOM card for M6/M7 Rack servers (UCSC-M-V5D200G) (VIC15411): PCIe-based mLOM card for B-Series M6 (UCSB-ML-V5Q10G) (VIC15235): PCIe card for M6/M7 Rack servers (UCSC-P-V5Q50G) (VIC15425): PCIe card for M6/M7 Rack servers (UCSC-P-V5Q50G)	Blades example: FI# connect adapter 1/1/1 adapter 1/1/1 # connect adapter 1/1/1 (top):1# For Standalone C-Series UCS: # scope chassis /chassis # show adapter (get the PCIe slot #) /chassis # connect debug-shell <PCIe slot #> (this command can only work when Server is powered on) - Check PCIe Link status adapter (top):1# attach-mcp adapter (mcp):1# pcie_links adapter (mcp):1# exit - Check the Ethernet Link status adapter (top):1# attach-mcp adapter (mcp):1# uifportstatus adapter (mcp):1# exit - Check Ethernet error counters adapter (top):1# attach-macd adapter (macd):1# macstats 0	These commands can be run after connecting to the adapter of a Cisco UCS B or C series server deployed with a 5th Gen VIC adapter.

Related Information

Revision History

Revision	Publish Date	Comments
3.0	06-Oct-2023	Updated with FI 5th Generation
2.0	14-Mar-2023	Updated Title, Introduction, SEO, Machine Translation, Style Requirements, Gerunds, Grammar and Formatting.
1.0	27-Apr-2020	Initial Release

Contributed by Cisco Engineers

Razan Alhalaseh
Technical Consulting Engineer
Iridian Fernandez
Technical Consulting Engineer

Was this Document Helpful?

Feedback

Contact Cisco

Open a Support Case
(Requires a Cisco Service Contract)

Troubleshoot Network Connectivity, Drops and CRC Errors

Available Languages

Download Options

Bias-Free Language

Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Reasons for Bad Frames and CRC Errors

Forwarding Mode Behavior (Cut-Through or Store-and-Forward)

Main Forwarding ASICs Commands for UCS FIs, IOMs, and VIC Cards

Related Information

Revision History

Contributed by Cisco Engineers

Was this Document Helpful?

Contact Cisco

This Document Applies to These Products