Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Reasons for Bad Frames and CRC Errors

Forwarding Mode Behavior (Cut-Through or Store-and-Forward)

Main Forwarding ASICs Commands for UCS FIs, IOMs, and VIC Cards

Related Information

Introduction

This document describes commands used to troubleshoot network connectivity, drops, and CRC errors within different UCS, FIs, IOMs, and VIC adapters.

Prerequisites

Requirements

This document assumes that you have knowledge of these topics:

- Cisco Unified Computing Systems (UCS) Virtual Interface Card (VIC)

- Cisco UCS B-Series and C-Series servers

- Cisco UCS Fabric extender I/O Module (IOM)

- Cisco UCS Fabric Interconnect (FI)

- Cisco Unified Computing System Manager (UCSM)

- Cisco Unified Computing System Manager (UCSM) Command Line Interface (CLI)

- Intersight Management Mode (IMM)

- Cut-through and store-and-forward switches

- Stomps

Components Used

The information in this document is based on these software and hardware versions:

- Cisco UCS Manager version 2. x and later

- Cisco UCS 6200, 6300, 6400, and 6500 Series Fabric Interconnect

- Cisco UCS 2200, 2300, and 2400 Series Fabric extender I/O Module

- Cisco UCS 1200, 1300, 1400, and 1500 Series Virtual Interface Card (VIC)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The Cisco UCS Fabric Interconnect is a cut-through switch like the Cisco Nexus 5000 Series Switches. It forwards bad frames like good frames. Bad frames get dropped by the destination server or when they pass through a piece of network equipment that is not cut-through. A CRC check is performed at the end of the frame to determine whether or not a frame has become corrupted.  Some switches can drop the frame once they detect a frame is corrupted. Cut through switches make the forwarding decision before they can perform the CRC check. Because of these frames that fail, a CRC check can still be switched by a cut-through switch. Other switches like the N7K are store and forward switches. Store and forward switches look at the entire frame before they make a forwarding decision.  A store and forward switch would drop a frame that failed a CRC check. To understand the architecture (cut-through vs store-forward) of the switches in the topology is imperative. Many times, you need to work your way backward to the source of the CRC error. Refer to this article for more details about cut-through and store-and-forward switches: Cloud Networking Switches

Note: . If you experience CRC errors on an interface, it does not mean that the interface is the source of the problem.

Reasons for Bad Frames and CRC Errors

Some of the reasons when you get bad frames and CRC errors can be:

- Bad physical connection; transceiver, copper, fiber, adapter, port expander, and so on.

- MTU Violation

- Received bad CRC stomped from neighboring cut-through switch.

Forwarding Mode Behavior (Cut-Through or Store-and-Forward)

UCS Fabric Interconnects (similar to Nexus 5000) utilizes both cut-through and store-and-forward switching. Forwarding mode is dependent on the ingress and egress data rate, as shown in Table 1.

Note: Cut-through switching can be performed only when the ingress data rate is equivalent to or faster than the egress data rate.

Table 1 - Forwarding Mode Behavior (Cut-Through or Store-and-Forward) for UCS Fabric Interconnect

Ingress/ Source Interface

Egress/ Destination Interface

Forwarding mode

10 Gigabit Ethernet

10 Gigabit Ethernet

Cut-through

10 Gigabit Ethernet

1 Gigabit Ethernet

Cut-through

1 Gigabit Ethernet

1 Gigabit Ethernet

Store-and-forward

1 Gigabit Ethernet

10 Gigabit Ethernet

Store-and-forward

10 Gigabit Ethernet

40 Gigabit Ethernet

Store-and-forward

40 Gigabit Ethernet

10 Gigabit Ethernet

Cut-through

40 Gigabit Ethernet

40 Gigabit Ethernet

Cut-through

40 Gigabit Ethernet

100 Gigabit Ethernet

Store-and-forward

100 Gigabit Ethernet

40 Gigabit Ethernet

Cut-through

100 Gigabit Ethernet

100 Gigabit Ethernet

Cut-through

FCoE

Fibre Channel

Cut-through

Fibre Channel

FCoE

Store-and-forward

Fibre Channel

Fibre Channel

Store-and-forward

FCoE

FCoE

Cut-through

Main Forwarding ASICs Commands for UCS FIs, IOMs, and VIC Cards

Tables 2 and 3 show the different commands that can be run from the different management endpoints in UCS to determine where the drops are coming from and why they are occurring.

In addition to the ASIC-specific commands mentioned in Table 2, these commands can be run from the UCS FI NXOS shell to look for errors at the receive direction of interfaces:

show interface counters errors

Table 2 - Main Forwarding ASICs commands for UCS FIs and IOMs

UCS FI / IOM

Main FW ASIC Name

Commands

Purpose

Cisco UCS Fabric Interconnects

Cisco UCS 6200 Series (Gen 2 FIs 62xx)

Carmel

(nxos)# show hardware internal carmel

This command shows the Carmel ASIC internals and driver information. 5th column shows how many ports/interfaces are mapped to each Carmel ASIC.

(nxos)# show hardware internal carmel all-ports

This command shows driver information for all ports and front panel physical ports to ASIC mapping.

(nxos)# show hardware internal carmel crc

This command shows information if any frames were received or transmitted with CRC errors or stomped for all ports.

(nxos)# show platform fwm info asic-errors X

This command shows non zero Carmel drop reasons error registers (where X is the Carmel ASIC number from 0-4).

(nxos)# show platform fwm info pif e1/X | grep asic

Use this command and you can map your interface to the Carmel ASIC ID “global_asic_num” (where X is the interface number).

(nxos)# show platform fwm info pif e1/X | grep drop

This command shows the number of frames, and it filters for the drop counter for a certain interface (where X is the interface number).

(nxos)# show hardware internal carmel all-ports detail | egrep -i "Carmel port|crc|frame_error"

This command filters for CRC and frame error counters for all ports.

Cisco UCS 6300 Series (Gen 3 FIs 63xx)

Trident2 (Broadcom ASIC)

(nxos)# show hardware internal bcm-usd info port-info

This command shows the mapping between each physical port to a front port on the Broadcom ASIC and this mapping is different between 6332 and 6332-16UP FIs.

(nxos)# show hard internal interface indiscard-stats front-port X

This command shows port internal discard counters for a certain front port on the Broadcom ASIC after the mapping is done that uses the previous command.

Cisco UCS Mini (6324 Fabric Interconnect)

Monticello ASIC

(nxos)# show hardware internal mtc-usd port-status

This command shows the status of the ports for the Monticello ASIC.

(nxos)# show hardware internal inband-mtc ?

ASIC   Show Monticello ASIC information

info. Show Monticello inband driver info

stats.  Show Monticello inband driver statistics.

- Please note KRs mapping for the blades within UCS Mini uses a different Port Mapping compared to a chassis with UCS IOMs, refer to TAC for more details.

Cisco UCS 6400 (Gen 4 FIs 64xx)

Homewood ASIC

FI # connect nxos

(nx-os)# show hardware internal interface asic counters module 1

This command shows the reason for the forwarding drops if reported on the interface.

FI # attach module 1

module-1# show hardware internal tah counters asic 0

This command shows the different counters of information tha use ASIC library.

There is only one ASIC within this UCS Fabric interconnect model so always ASIC number 0.

FI # attach module 1

module-1# show hardware internal tah drop-reason counters module 0

This command shows the drop reasons and number of dropped packets.

Cisco UCS 64108 Gen 4 FIs

Cisco ASIC Heavenly

FI # connect nxos

(nx-os)# show hardware internal interface asic counters module 1

This command shows the reason for the forwarding drops if reported on the interface

(nxos) # attach module 1

module-1# show hardware internal tah counters asic 0

This command shows the different counters of  information that use the ASIC library.

There is only one ASIC within this UCS Fabric interconnect model so always ASIC number 0.

FI # attach module 1

module-1# show hardware internal tah drop-reason counters module 0

This command shows the reason for the forwarding drops if reported on the interface.

Cisco UCS 6500 Gen 5 FIs

Cisco ASIC Heavenly

FI # connect nxos

(nx-os)# show hardware internal interface asic counters module 1

This command shows the reason for the forwarding drops if reported on the interface

(nxos) # show hardware internal errors module 1

This command shows the output Drop Packets for the module on the ports.

(nxos) # attach module 1

module-1# show hardware internal tah counters asic 0

FI # attach module 1

module-1# show hardware internal tah drop-reason counters module 0

This command shows the different counters of  information that use the ASIC library.

This second command shows the drop reasons and number of dropped packets.3

Cisco UCS Fabric extender I/O Modules (IOMs)

Cisco UCS 2200 IOM (Gen 2)

Woodside

FI # connect IOM <chassis ID>

fex-1# show platform software woodside sts

This command shows the interface status of the HIFs and NIFs within the Woodside ASIC and which HIFs are used by each blade.

Note: There are two numberings for the HIFs, one is used when you troubleshoot from the IOM (after you connect to IOM) and the other is used when you troubleshoot the same HIF and run the commands from UCSM nxos.

For example, blade 1 uses HIF numbers 28-31. You can use these numbers after you connect to IOM and run the related commands to that HIF. These correspond to Eth1/1/1 - 4 from UCSM NXOS as per the shown FEX detail.

FI # connect IOM <chassis ID>

fex-1# show platform software woodside rate

This command shows the packet rates for active HIF or NIF ports.

FI # connect IOM <chassis ID>

fex-1# show platform software woodside rmon 0 [NIx/HIx]

For example, you can filter some error counters using grep for all NIFs as below:

fex-1# show platform software woodside rmon 0 nif_all | egrep -i

This command shows the received and transmitted packet sizes for a certain HIF or NIF and packet types like unicast, broadcast, or multicast.

The Diff column is useful while you do live troubleshooting as it resets after each time you run the command to show you if packets are incrementing when you run the command again.

You can also check if the Diff column shows new packets for these:

RX_CRC_NOT_STOMPED

RX_CRC_STOMPED

TX_FRM_ERROR

FI # connect IOM <chassis ID>

fex-1# show platform software woodside drops 0 [NIx/HIx]

This command shows the drop counters for a certain NIF or HIF.

FI # connect IOM <chassis ID>

fex-1# show platform software woodside oper

This command shows the administrative control, MAC, and physical status, in addition to detected SFPs within the NIFs.

FI # connect iom <chassis ID>

fex-1# show platform software woodside sfp 0 ni0

fex-1# show platform software woodside sfp 0 ni1

fex-1# show platform software woodside sfp 0 ni2

fex-1# show platform software woodside sfp 0 ni3

This command shows the transceiver details within the woodside IOM NIF ports.

Cisco UCS 2300 IOM (Gen 3)

and

Cisco UCS 2300 IOM version 2 (UCS-IOM-2304V2)

Tiburon (Broadcom ASIC)

# connect IOM <chassis ID>

Fex-1# show platform software tiburon sts

This command shows the interface status of the HIFs and NIFs within the Tibrun ASIC and which HIFs are used by each blade.

Tibrun ASIC comes from the 2248 FEX which has 48 HIF ports, so for UCS, there are some unused ports on the ASIC (NI0-7 and HI0-9 are unused).

Note: in regards to the 40G backplane ports within Gen 3 IOMs, HIF status can normally be with the 40 Gig primary ports marked as UP, and the 40 Gig member ports are marked Down.

# connect IOM <chassis ID>

fex-1# show platform software tiburon rate

This command shows the packet rates for active HIF or NIF ports.

FI # connect IOM <chassis ID>

fex-1# show platform software tiburon rmon 0 [NIx/HIx]

For example, you can filter some error counters using grep for all NIFs as shown:

fex-1# show platform software tiburon rmon 0 nif_all | egrep -i 'crc|ni|stomp|pause|err'

This command shows the received and transmitted packet sizes for a certain HIF or NIF and packet types like unicast, broadcast, or multicast.

The Diff column is useful while you do live troubleshooting as it resets after each time you run the command to show you if any new packets are coming when you run the command again.

You can also check if the Diff column shows new packets for these:

RX_CRC_NOT_STOMPED

RX_CRC_STOMPED

TX_FRM_ERROR

Cisco UCS 2408 (fourth-generation I/O Module)

"Summerville"  UCS-IOM-2408

Sundown

FI # connect iom <chassis ID>

fex-1# show hardware internal tah sts

This command shows the interface status of the HIFs and NIFs within the Tahoe ASIC and which HIFs are used by each blade.

fex-1# show hardware internal tah sts detail

This command shows the NXOS to HIF port mapping, link-state, and operational speed.

fex-1# show hardware internal tah counters asic 0 nxos-port ?

<0-44> Nxos-port num 0-31 hif/35 bif/36-43 nif

This command shows the detailed per-port counters

The detailed interface counters can be viewed by referring to the NXOS port number.

NXOS Ports 0-31 correspond to 32 HIF Ports

NXOS ports 36-43 correspond to the 8 NIF ports.

1 End-of-Sale and End-of-Life Announcement for the Cisco UCS 6200 Series Fabric Interconnects: Cisco UCS 6200 Series Fabric Interconnects

2 End-of-Sale and End-of-Life Announcement for the Cisco UCS 2100 Series IO Modules: Cisco UCS 2200 Series IO Modules

3 Mode details on some columns of show hardware internal carmel crc command:

- MM rx CRC = CRC on this link; Problem is L1 issue; Check eye height; shut, no shut; replace cable;

- MM Rx Stomp = STOMP on the remote switch; Go check the same output on the switch across this link;

- FI Rx Stomp = If MM Rx CRC and MM Rx Stomp are blank; L2/policy violation, most commonly MTU violation; Check QoS MTU settings.

4 Sometimes to see the Port mapping on FI 6400/6500 the attach module does not work, to have this addressed please contact a Cisco TAC Engineer.

Table 3 - The main commands to troubleshoot connectivity, drops, and CRC errors for Cisco UCS VIC cards.

UCS VIC Generation

Example of the VIC card model

Commands

Purpose

Cisco UCS 1200 VIC (Gen 2)

an example is the  1225 VIC, 1240 VIC, 1280 VIC, etc

Blades example:

FI# connect adapter 1/1/1

adapter 1/1/1 # connect

adapter 1/1/1 (top):1# show-log

adapter 1/1/1 (top):1# attach-mcp

adapter (mcp):1# uifportstatus

adapter (mcp):3# dcem-macstats 0       <<<< Stats for port-1

adapter (mcp):3# dcem-macstats 1       <<<< Stats for port-2

adapter 1/1/1 (mcp):1# vnic

adapter 1/1/1 (mcp):1# lifstats

For Standalone C-Series UCS:

# scope chassis

/chassis # show adapter  (get the PCIe slot #)

/chassis # connect debug-shell <PCIe slot #> (this command can only work when Server is powered on)

adapter (top):1# attach-mcp

adapter (mcp):1# uifportstatus

These commands can be run after connecting to the adapter of a Cisco UCS B or C series servers.

The macstats command gives information about the status of the physical ports, packet sizes, and if there are any stomped or non-stomped frames received.

Cisco UCS 1300 VIC (Gen 3)

an example is the 1380 VIC

Cisco UCS 1400 VIC (Gen 4)

Example is:

(VIC1440):

PCIe-based mLOM card for M5 blades (UCSB-MLOM-40G-04)

·        (VIC1480):

PCIe based MEZZ card for M5 blades (UCSB-VIC-M84-4P)

·        (VIC1455):

PCIe card for M5 Rack servers (UCSC-PCIE-C25Q-04)

·        (VIC1457):

PCIe based mLOM card for M5 Rack servers (UCSC-MLOM-C25Q-04)

-  Check PCIe Link status

adapter (top):1# attach-mcp

adapter (mcp):1# pcie_links

pp,pps type link config link status state

0,0   host gen3x16     gen3x16     UP

adapter (mcp):2# exit

-  Check the Ethernet Link status

adapter (top):2# attach-mcp

adapter (mcp):1# uifportstatus

ASIC        Port

UIF Port State Speed

0   0    UP      25g

0   1    UP      25g

1   0    UP      25g

1   1    UP      25g

adapter (mcp):2# exit

-  Check Ethernet error counters

adapter (top):3# attach-macd

adapter (macd):1# macstats 0

DELTA                TOTAL DESCRIPTION

112                  112 Rx good packets

112                  112 Rx total received packets

14574                14574 Rx bytes for good packets

14574                14574 Rx bytes

104                  104 Rx multicast frames

<snip>

These commands can be run after connecting to the adapter of a Cisco UCS B or C series servers deployed with 4th Gen VIC adapter.

Cisco UCS 1500 VIC (Gen 5)

Example is:

- (VIC15231):

PCIe-based mLOM card for X210 (UCSX-ML-V5D200G)

- (VIC15420):

PCIe based mLOM card for X210C M6/M7 (UCSX-ML-V5Q50G)

- (VIC15422):

PCIe based MEZZ card for X210C M6/M7 (UCSX-ME-V5Q50G)

- (VIC15428):

PCIe based mLOM card for M6/M7 Rack servers (UCSC-M-V5Q50G)

- (VIC15238):

PCIe based mLOM card for M6/M7 Rack servers (UCSC-M-V5D200G)

- (VIC15411):

PCIe-based mLOM card for B-Series M6 (UCSB-ML-V5Q10G)

- (VIC15235):

PCIe card for M6/M7 Rack servers (UCSC-P-V5Q50G)

- (VIC15425):

PCIe card for M6/M7 Rack servers (UCSC-P-V5Q50G)

Blades example:

FI# connect adapter 1/1/1

adapter 1/1/1 # connect

adapter 1/1/1 (top):1#

For Standalone C-Series UCS:

# scope chassis

/chassis # show adapter  (get the PCIe slot #)

/chassis # connect debug-shell <PCIe slot #> (this command can only work when Server is powered on)

-  Check PCIe Link status

adapter (top):1# attach-mcp

adapter (mcp):1# pcie_links

adapter (mcp):1# exit

-  Check the Ethernet Link status

adapter (top):1# attach-mcp

adapter (mcp):1# uifportstatus

adapter (mcp):1# exit

-  Check Ethernet error counters

adapter (top):1# attach-macd

adapter (macd):1# macstats 0

These commands can be run after connecting to the adapter of a Cisco UCS B or C series server deployed with a 5th Gen VIC adapter.

Related Information

- Technical Support & Documentation - Cisco Systems

- Need reload mechanism for 2348 FEX when CRC errors are seen

- Cisco UCS Manager CLI User Guides List

- UCS B Series Blade Servers 6200 Spec Sheet

- UCS B Series Blade Servers 6332 Spec Sheet

- UCS B Series Blade Servers 6454 Spec Sheet

- UCS B Series Blade Servers 6536 Spec Sheet