Troubleshoot Link Flap Issue on Nexus 9000

Available Languages

Download Options

PDF (44.7 KB)
View with Adobe Reader on a variety of devices
ePub (84.3 KB)
View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle) (75.6 KB)
View on Kindle device or Kindle app on multiple devices

Updated:January 31, 2024

Document ID:221572

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

This document describes how to troubleshoot the layer 1 link flap issue on Nexus 9000 switches.

Prerequisites

Requirements

Cisco recommends that you have a familiarity with the Cisco Nexus Operating System (NX-OS) and basic Nexus architecture before you proceed with the information that is described in this document.

Components Used

The information in this document is based on these software and hardware versions:

N9K-C93180YC-FX
nxos64-cs.10.2.6.M

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Understand Link Flap Causes

A link flap is a networking issue where a physical interface on a switch, such as the Nexus 9000, continuously alternates between being up and down. This disruptive behavior can degrade network performance, destabilize the network, and interrupt communication, thereby causing significant inconveniences. Link flaps generally arise from faulty physical layers or protocol synchronization issues.

Protocol Triggered Link Flap

Protocol-triggered link flaps occur when there is an issue with the protocol synchronization. This could involve protocols such as the Link Aggregation Control Protocol (LACP), Virtual Port-Channel and others. The issue can arise from protocol misconfiguration or packets lost, leading to link instability. Regular monitoring and timely software updates can help prevent this type of link flap.

Layer 1 Physical Issue

Link flaps can also stem from Layer 1, the physical layer of the network. This often involves physical components such as cables and interfaces. Damaged, loose, or aging cables and malfunctioning interfaces can cause the link to flap. Regular physical inspections and maintenance, including cable checks and interface testing, can help identify and rectify these issues before they lead to link flaps.

This article focuses on layer 1 physical issue troubleshooting.

Identify Link Flap

Link flaps can be easily identified from logs. The example displays a link flap event on port E1/5, where the port goes down and then comes back up later.

2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel100: first operational port changed from Ethernet1/5 to none
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel100: Ethernet1/5 is down
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel100 is down (No operational members)
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel100,bandwidth changed to 100000 Kbit
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/5 is down (Link failure)
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel100 is down (No operational members)
2024 Jan 21 05:27:58 N9K-C93180YC-FX %ETHPORT-5-SPEED: Interface Ethernet1/5, operational speed changed to 10 Gbps
2024 Jan 21 05:27:58 N9K-C93180YC-FX %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/5, operational duplex mode changed to Full
2024 Jan 21 05:27:58 N9K-C93180YC-FX %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/5, operational Receive Flow Control state changed to off
2024 Jan 21 05:27:58 N9K-C93180YC-FX %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/5, operational Transmit Flow Control state changed to off
2024 Jan 21 05:27:58 N9K-C93180YC-FX %ETHPORT-5-SPEED: Interface port-channel100, operational speed changed to 10 Gbps
2024 Jan 21 05:27:58 N9K-C93180YC-FX %ETHPORT-5-IF_DUPLEX: Interface port-channel100, operational duplex mode changed to Full
2024 Jan 21 05:27:58 N9K-C93180YC-FX %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel100, operational Receive Flow Control state changed to off
2024 Jan 21 05:27:58 N9K-C93180YC-FX %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel100, operational Transmit Flow Control state changed to off
2024 Jan 21 05:28:02 N9K-C93180YC-FX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel100: Ethernet1/5 is up
2024 Jan 21 05:28:02 N9K-C93180YC-FX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel100: first operational port changed from none to Ethernet1/5
2024 Jan 21 05:28:02 N9K-C93180YC-FX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel100,bandwidth changed to 10000000 Kbit
2024 Jan 21 05:28:02 N9K-C93180YC-FX %ETHPORT-5-IF_UP: Interface Ethernet1/5 is up in mode access
2024 Jan 21 05:28:02 N9K-C93180YC-FX %ETHPORT-5-IF_UP: Interface port-channel100 is up in mode access

Identify Layer 1 Link Flap or Protocol-triggered Link Flap

The Ethernet Port Manager (Ethpm) is a process that manages Ethernet interfaces. The Ethpm event history can be utilized to identify the cause of a link flap.

Layer 1 Flap Example

E1/5 experiences a link failure at 05:28:35, with the ethpm transition triggered by ETH_PORT_FSM_EV_LINK_DOWN. This indicates a Layer 1 flap.

2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel100 is down (No operational members)
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel100,bandwidth changed to 100000 Kbit
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/5 is down (Link failure)
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel100 is down (No operational members)

N9K-C93180YC-FX# show system internal ethpm event-history interface e1/5
[143] 2024-01-21T05:26:02.100255000+00:00 [-] FSM:<Ethernet1/5> Transition:
Previous state: [ETH_PORT_FSM_ST_WAIT_BUNDLE_MEMBER_BRINGUP]
Triggered event: [ETH_PORT_FSM_EV_FIRST_BRINGUP_BUNDLE_MEMBER_DONE]
Next state: [ETH_PORT_FSM_ST_BUNDLE_MEMBER_UP]
[144] 2024-01-21T05:27:35.783495000+00:00 [-] FSM:<Ethernet1/5> Transition:
Previous state: [ETH_PORT_FSM_ST_BUNDLE_MEMBER_UP]
Triggered event: [ETH_PORT_FSM_EV_LINK_DOWN]
Next state: [FSM_ST_NO_CHANGE]

LACP Triggered Flap Example

E1/8 enters an initializing down state at 07:40:07, with the ethpm transition triggered by ETH_PORT_FSM_EV_EXTERNAL_REINIT_NO_FLAP_REQ. This indicates a link flap triggered by Link Aggregation Control Protocol (LACP).

2024 Jan 21 07:37:20 N9K-C93180YC-FX %ETHPORT-5-IF_UP: Interface port-channel200 is up in Layer3
2024 Jan 21 07:40:07 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel200 is down (No operational members)
2024 Jan 21 07:40:07 N9K-C93180YC-FX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel200: first operational port changed from Ethernet1/8 to none
2024 Jan 21 07:40:07 N9K-C93180YC-FX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel200: Ethernet1/8 is down
2024 Jan 21 07:40:07 N9K-C93180YC-FX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel200,bandwidth changed to 100000 Kbit
2024 Jan 21 07:40:07 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/8 is down (Initializing)

N9K-C93180YC-FX# show system internal ethpm event-history interface e1/8
[218] 2024-01-21T07:37:20.551880000+00:00 [-] FSM:<Ethernet1/8> Transition:
Previous state: [ETH_PORT_FSM_ST_WAIT_BUNDLE_MEMBER_BRINGUP]
Triggered event: [ETH_PORT_FSM_EV_FIRST_BRINGUP_BUNDLE_MEMBER_DONE]
Next state: [ETH_PORT_FSM_ST_BUNDLE_MEMBER_UP]

[219] 2024-01-21T07:40:07.104339000+00:00 [-] FSM:<Ethernet1/8> Transition:
Previous state: [ETH_PORT_FSM_ST_BUNDLE_MEMBER_UP]
Triggered event: [ETH_PORT_FSM_EV_EXTERNAL_REINIT_NO_FLAP_REQ]
Next state: [FSM_ST_NO_CHANGE]

Troubleshoot Layer 1 Link Flap

Cisco offers an extensive array of optical modules to accommodate a wide range of speeds, media, and distances. Before connecting a link to the Nexus 9000, ensure that the SFP and cable are compatible with your current software and hardware. You can verify this by:

Cisco Optics-to-Device Compatibility Matrix

Cisco Optics-to-Optics Interoperability Matrix

Layer 1 Issue on NX-OS 10.2.1 and Later Releases

Starting with NX-OS 10.2.1, the Platform Insights Engine (PIE) is supported on all Cloudscale ToR and EoR platforms. PIE is an on-switch real-time root cause analysis application.

Three PIEs can assist you in addressing the Layer 1 link flap issue.

Link Flap PIE

The link flap PIE analyzes link flap events published by user space drivers (USDs) and determines the root cause for a link flap. The PIE publishes the root cause analysis insight to the broker. Link flap events are published by the USDs (PIE client) when a link flaps. The USDs collect all of the relevant data from the ASIC and USD that is required for root cause analysis and publish the data to the broker. The link flap PIE analyzes the data and arrives at the most probable root cause for the flap.

Link Down PIE

The link down PIE finds the root cause for a link not coming up. The USD collects data about an interface when the interface is configured to be up, but the interface operating state is not up. This data is published to the PIE application. The link-down PIE subscribes to these events, receives the data from the broker, and analyzes the data to find the root cause.

Optics PIE

The optics PIE is a continuous monitoring engine that performs a time series analysis of the DOM data collected at regular intervals. By tracking various parameters in the DOM over a period, the PIE arrives at a metric to describe the state of optics for each optical port. The metric is an insight about the trending health of an optical transceiver.

For more information, refer to this PIE document:

Cisco Nexus 9000 Series NX-OS Platform Insights Engine Guide, Release 10.2(x)

PIE Example: Link Flap Caused by Shutting Down and then Re-Enabling the Port on the Peer Side

2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel100: first operational port changed from Ethernet1/5 to none
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel100: Ethernet1/5 is down
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel100 is down (No operational members)
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel100,bandwidth changed to 100000 Kbit
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/5 is down (Link failure)
2024 Jan 21 05:27:35 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel100 is down (No operational members)
2024 Jan 21 05:27:58 N9K-C93180YC-FX %ETHPORT-5-SPEED: Interface Ethernet1/5, operational speed changed to 10 Gbps
<snip>
2024 Jan 21 05:28:02 N9K-C93180YC-FX %ETH_PORT_CHANNEL-5-PORT_UP: port-channel100: Ethernet1/5 is up

N9K-C93180YC-FX# show pie interface ethernet 1/5 link-flap-rca 
2024-01-21 05:27:35 Event Id: 00000068 Ethernet1/5 Source Id: 436209664 RCA Code: 41   >>>PIE event time
Reason: Link flapped/down due to Local Fault, check peer                               >>>PIE link flap reason

N9K-C93180YC-FX# show pie interface ethernet 1/5 transceiver-insights 
2024-01-21 05:30:12 Event Id: 00000080 Event Class: xcvr DOM DB Event Interface: Ethernet1/5 Health Metric: --------GOOD------- Mod: 01
2024-01-21 05:28:12 Event Id: 00000072 Event Class: xcvr DOM DB Event Interface: Ethernet1/5 Health Metric: --------GOOD------- Mod: 01

PIE Example: Link Down Caused by Shutting Down the Port on the Peer Side

2024 Jan 21 05:48:38 N9K-C93180YC-FX %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel100: first operational port changed from Ethernet1/5 to none
2024 Jan 21 05:48:38 N9K-C93180YC-FX %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel100: Ethernet1/5 is down
2024 Jan 21 05:48:38 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel100 is down (No operational members)
2024 Jan 21 05:48:38 N9K-C93180YC-FX %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel100,bandwidth changed to 100000 Kbit
2024 Jan 21 05:48:38 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/5 is down (Link failure)
2024 Jan 21 05:48:38 N9K-C93180YC-FX %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel100 is down (No operational members)

N9K-C93180YC-FX# show pie interface ethernet 1/5 link-down-rca 
2024-01-21 05:48:48 Event Id: 00000197 Ethernet1/5 Source Id: 436209664 RCA Code: 16           >>>PIE event time
Reason: No PCS alignment detected. Please check Fec, speed, Autoneg configurations with peer   >>>Physical layer failed

N9K-C93180YC-FX# show pie interface ethernet 1/5 transceiver-insights
2024-01-21 05:50:12 Event Id: 00000199 Event Class: xcvr DOM DB Event Interface: Ethernet1/5 Health Metric: ********BAD******** Mod: 01
2024-01-21 05:48:12 Event Id: 00000187 Event Class: xcvr DOM DB Event Interface: Ethernet1/5 Health Metric: --------GOOD------- Mod: 01

Replacing Faulty Parts

Based on the PIE output, it is recommended to replace the potentially faulty component and continue monitoring. If the link flap persists, then a swap test is needed to narrow down the faulty part. A swap test can be conducted by changing one component at a time while keeping everything else unchanged. Ultimately, the link stabilizes after the specific faulty component has been swapped out.

Layer 1 Issue on NX-OS 10.1.2 and Earlier Releases

For NX-OS software releases prior to 10.2(1), PIE support is not available. Several manual steps are required for checking layer 1 link flap.

Verifying Port-Client Event History

This lists all link events on the attached module. Debounce time refers to the duration that an interface waits before notifying the supervisor of a link going down. During this period, the interface waits to see if the link comes back up. This is used to determine whether the link has gone down or is just experiencing a minor flap.

N9K-C93180YC-FX# attach module 1
module-1# show system internal port-client link-event 
*************** Port Client Link Events Log ***************
---- ------ ----- ----- ------
Time PortNo Speed Event Stsinfo
---- ------ ----- ----- ------
Jan 21 05:48:38 2024 00122142 Ethernet1/5 ---- DOWN Link down debounce timer stopped and link is down
Jan 21 05:48:37 2024 00993003 Ethernet1/5 ---- DOWN Link down debounce timer started(0x40e50006)
Jan 21 05:45:14 2024 00432606 Ethernet1/5 10G UP SUCCESS(0x0)

Verifying the ASIC Events

These events provide detailed information about each link event.

N9K-C93180YC-FX# attach module 1
module-1# show hardware internal tah link-events fp-port 5
324) Jan 21 05:48:37 2024 uSec 992843: Fp 5 : tahusd_isr.c #8469
Port Down with an ASIC interrupt                            
------------- ASIC MAC/PCS/Serdes REGS (Mac Channel 0) -------------
Link flapped due to Local Fault, check peer          >>>Local Fault means the local device detected the issue on the receive path.
                                                     >>>Remote Fault means a Local Fault is detected across the link. 
              
Intr Regs 00:0x0000, 01:0x0000, 02:0x0000, 03:0x0010, 07:0x0000, 11:0x0000, 15:0x0000
sts2.bercount : 0x0f00 sts2.erroredblocks : 0x0000
bercounthi : 0x0000 erroredblockhi : 0x0000
counters0.syncloss : 0x0001 counters0.blocklockloss: 0x0001
counters1.highber : 0x0000 counters1.vlderr : 0x0000
counters2.unkerr : 0x0012 counters2.invlderr : 0x0000

Error Code	Explanation
sts2.erroredblocks	Counts errored blocks (higher order bits).
sts2.bercount	Counts bad sync headers (lower order bits).
bercounthi	Counts bad sync headers (higher order bits).
erroredblockhi	Counts errored blocks (higher order bits).
counters0.syncloss	Sync loss
counters0.blocklockloss	Block lock loss
counters1.highber	High BER
counters1.vlderr	Valid Error
counters2.unkerr	Unknown Error
counters2.invlderr	Invalid Error

Checking Digital Optical Monitoring (DOM) Information on Both Sides

There are several pieces of Small Form-factor Pluggable (SFP) information in this output. If any value falls outside the acceptable range in the SFP diagnostic, then the SFP is considered a potentially damaged component and needs to be replaced. In this example, everything is in good order.

N9K-C93180YC-FX# show interface e1/5 transceiver details 
Ethernet1/5
transceiver is present
type is 10Gbase-SR                      >>>SFP type
name is CISCO-OPLINK                    >>>SFP vendor
part number is TPP4XGDS0CCISE2G
revision is 02
serial number is OPMXXXXXXXX            >>>SFP SN
nominal bitrate is 10300 MBit/sec       >>>SFP bitrate
Link length supported for 50/125um OM2 fiber is 82 m
Link length supported for 62.5/125um fiber is 26 m
Link length supported for 50/125um OM3 fiber is 300 m
cisco id is 3
cisco extended id number is 4
cisco part number is 10-2415-03
cisco product id is SFP-10G-SR          >>>SFP PID
cisco version id is V03

SFP Detail Diagnostics Information (internal calibration)
----------------------------------------------------------------------------
            Current                Alarms       Warnings
          Measurement           High    Low    High    Low
----------------------------------------------------------------------------
Temperature 36.52 C           75.00 C -5.00 C 70.00 C 0.00 C
Voltage     3.28 V             3.63 V  2.97 V  3.46 V 3.13 V
Current     6.61 mA         12.00 mA 0.50 mA 11.50 mA 1.00 mA
Tx Power   -2.70 dBm        1.99 dBm -11.30 dBm -1.00 dBm -7.30 dBm
Rx Power   -2.40 dBm        1.99 dBm -13.97 dBm -1.00 dBm -9.91 dBm
Transmit Fault Count = 0
----------------------------------------------------------------------------
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning
peer side information is snipped.

Swap Test and Replacement of Faulty Parts

If everything appears fine with the previous checks, then a swap test is needed to narrow down the faulty part. A swap test can be conducted by changing one component at a time while keeping everything else unchanged. Eventually, the link stabilizes after the specific faulty component has been swapped out.

Related Information

Nexus 9000 Datasheet

Nexus 9000 Interfaces Configuration Guide

Nexus 9000 Series NX-OS Platform Insights Engine Guide

Revision History

Revision	Publish Date	Comments
1.0	31-Jan-2024	Initial Release

Contributed by Cisco Engineers

Bowen Ji
Cisco Technical Consulting Engineer