Troubleshoot Fiber Links on Catalyst 9000 Series Switches
Available Languages
Contents
Introduction
This document describes how to troubleshoot fiber optic interfaces by addressing some of the fiber optic module and cabling specifications.
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
The information in this document is based on all Catalyst 9000 Series switches. This includes Doppler Based and Silicon One (S1) switches.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background Information
The intricacies of dealing with fiber optics are commonly under-estimated, and therefore, mistakes can be committed while implementing new fiber links; low performance, interface errors, and connectivity problems can arise out of choosing the wrong fiber cable.
Troubleshoot
The intention of this document is to explain some of the specifications that appear in the Cisco Optics-to-Device Compatibility Matrix and the importance of paying specific attention to the transceiver and cable specification while implementing and troubleshooting fiber optic link.
Cisco Compatible Small Form Plugable (SFP) Transceivers
Insertion of non-compatible and/or third-party SFPs can lead to unpredictable behavior, and thus the stability of the link is not guaranteed in the absence of Cisco original compatible transceivers. Thus, it is recommended to connect only Cisco-compatible transceivers to Cisco equipment. You can obtain the list of compatible transceivers by visiting the Cisco Optics-to-Device Compatibility Matrix or run the show interface show interface transceiver supported-list command.
Switch#show interfaces transceiver supported-list
Transceiver Type Cisco p/n min version
supporting DOM
------------------ -------------------------
GLC-T NONE
GLC-TE NONE
GLC-SX-MM NONE
GLC-LH-SM NONE
GLC-ZX-SM NONE
GLC-SX-MM-RGD CPN 2274-02
GLC-LX-SM-RGD CPN 10-2293-02
GLC-ZX-SM-RGD CPN 10-2366-02
GLC-SX-MMD ALL
GLC-LH-SMD ALL
!----Lines omitted for summarization---In order to consult the SPF model connected to an interface, run the show idprom interface <interface-id> command.
Switch#show idprom interface twentyFiveGigE 1/0/24 | include PID Product Identifier (PID) = SFP-10G-LR-S Switch#
SFP Standard
SFP standards tend to be backwards compatible, but higher standards cannot be supported by a given interface. SFP standards can vary from port to port, even on the same switch front panel. This is the case for the C9500-32QC switch model. Therefore, the presence of an SFP in the compatibility matrix does not guarantee the SFP compatibility with a given interface, so it must be verified against the hardware installation notes. In order to obtain the SFP standard of the transceiver navigate to the Cisco Optics-to-Device Compatibility Matrix or run the show idprom interface <interface-id> command.
Switch#show idprom interface twentyFiveGigE 1/0/24 | include Transceiver Type Transceiver Type: = SFP+ 10GBASE-LR (274) Switch#
Speed Capabilities
It is important to always ensure the SFP connected at both sides of the link supports the same speed. Supported speeds can be verified with the show interface <interface-id> capabilities command. The speed and duplex setting for multigigabit links is considered best practice, and in some scenarios necessary for the links to come up.
Switch#show interfaces twentyFiveGigE 1/0/24 capabilities | include Speed Speed: 10000
Connector Type
This aspect is hard to get wrong, as using the wrong connector type does not allow the cable to be connected to the corresponding SFP slot. However, it is still important to consider when choosing the SFP and cabling. In order to consult the connector type of the transceiver navigate to Cisco Optics-to-Device Compatibility Matrix or run the show idprom interface <interface-id> command.
Switch#show idprom interface twentyFiveGigE 1/0/24 | include Connector type Connector type = LC
Polish Type
The polish type is one of the most ignored aspects when acquiring SFPs, but it is one of the most important aspects to consider. This is the termination form of the core of the cable, the actual medium that carries the signal. The polish type is designed to provide acceptable levels of Optical Return Loss (ORL); the light reflected back to the laser/LED transmitter.
| Polish type | Back reflection |
| Flat | -30dB |
| Physical Contact (PC) connector | -35dB |
| Ultra-Physical Contact (UPC) connector | -55dB |
| Angled Physical contact (APC) connector | -65dB |
Choosing the wrong polish type can damage the transceiver due to the ORL level, and at best, it can lead to link instability and Layer 1 errors. In order to consult the polish type of the cable, navigate to the Cisco Optics-to-Device Compatibility Matrix. Ensure the SFPs and cable match the required core polish type.
UPC, PC, and FLAT connectors cannot always be visually identified, so refer to the specifications provided by the cable vendor.
Single Mode Fiber (SMF) vs. Multi Mode Fiber (MMF)
In Multi-Mode fiber cables, different paths exist for the light to reach its destination. On the other hand, Single Mode cables allow only one path for the laser light.
Single Mode Fiber (SMF) vs. Multi-Mode Fiber (MMF)
There is a clear outline in the infrastructure required to support Multi Mode Fiber vs. Single Mode Fiber. For example, SMF cabling uses a 9-micron core width, which allows the light to be transmitted over a single path, and the wavelength is optimized to a range between 1300nm and 1500nm. Therefore, ensure both SFPs and cabling are either MMF or SMF fiber components. In order to consult the MMF/SMF mode navigate to Cisco Optics-to-Device Compatibility Matrix.
Parallel/Single-Strand/Duplex Cabling
| Type of Cable | Explanation |
| Single-strand | Permits to send and receive data over the same core. |
| Duplex | Allows to send data over one core and receive over a secondary core. |
| Parallel | Sends data over multiple parallel cores and receives it over a symmetrical number of cores. |
.
Special attention has to be paid to duplex cables. Ensure the sender transceiver is connected to the receiver at the other side of the link for proper polarization. Worst case scenario; the sender slot is connected to the sender of the pairing device and therefore not coming up.
Duplex Mode
Connection polarization complexity increases with parallel links as there are multiple solutions to address this problem depending on the Multi-fiber Push On (MPO) Standard. Thus, please consider investigating in dedicated documentation when troubleshooting parallel fiber links.
Wavelength
Photo-detectors of the transceivers are calibrated to interpret certain infrared wavelength of electromagnetic fields. These wavelengths range between 850nm and 1300nm for MMF fiber links, and between 1300nm and 1500nm for SMF.
Just as our eyes are only capable of seeing a certain range of the electromagnetic spectrum, and no other, the photoreceptors are calibrated to detect certain wavelengths of the infrared spectrum. Choosing the wrong laser/LED wavelength leads to miscommunication between transceivers, if communication is even possible. Both the SFPs must be capable of reading, transmitting on the same wavelength. In order to consult the wavelength to be used navigate to Cisco Optics Product Information or run the show idprom interface <intercace-id > detail command.
Switch#show idprom interface twentyFiveGigE 1/0/24 detail | include laser wave Nominal laser wavelength = 1310 nm
Special attention must be paid to asymmetrical Receive (RX)/Transmit (TX) transceivers where the TX and RX values differ from each other, and they must be conversely matched at the other side of the link.
Transmit/Receive Power
In order to guarantee the SFP signal is understood by the other side of the link, the electromagnetic signal strength must fall within certain thresholds. This signal is measured in decibel milliwatts (dBm), and the thresholds within which the operational values reside depend on the SFP in use. In order to obtain the current TX and RX dBm values and their upper and lower thresholds run theshow interfaces <interface-id> transceiver detail command.
Switch#show interfaces twentyFiveGigE 1/0/24 transceiver detail
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.
High Alarm High Warn Low Warn Low Alarm
Temperature Threshold Threshold Threshold Threshold
Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)
--------- ----------------- ---------- --------- --------- ---------
Twe1/0/24 20.6 75.0 70.0 0.0 -5.0
High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
--------- ----------------- ---------- --------- --------- ---------
Twe1/0/24 3.30 3.63 3.46 3.13 2.97
High Alarm High Warn Low Warn Low Alarm
Current Threshold Threshold Threshold Threshold
Port Lane (milliamperes) (mA) (mA) (mA) (mA)
--------- ---- --------------- ---------- --------- --------- ---------
Twe1/0/24 N/A 26.7 75.0 70.0 18.0 15.0
Optical High Alarm High Warn Low Warn Low Alarm
Transmit Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Twe1/0/24 N/A -2.2 3.5 0.5 -8.2 -12.2
Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Twe1/0/24 N/A -2.0 3.5 0.5 -14.1 -18.4In this scenario, the current receive power is equal to -2.0 dBm, which is an acceptable value based on the thresholds to the right. Any value under -14.1 dBm or over 0.5 dBm (the warning thresholds) must be considered a problem, as it can potentially affect data quality and cause link flaps.
Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Twe1/0/24 N/A -2.0 3.5 0.5 -14.1 -18.4Receive Power values beneath the Low Warn Threshold most of the time indicate a problem with the transceiver at the other side, the device hosting the transceiver at the opposite end of the link, or the cable connecting the transceivers together. The same applies to high Receive Power values exceeding the High Warn Threshold. A defect in the values provided by the Digital Optical Monitoring (DOM) sensors is also plausible.
In contrast, problems with the Transmit Power measurements indicate an issue with the transceiver providing these values or the switch hosting the transceiver. A defect in the values provided by the DOM sensors is also plausible.
When troubleshooting, consider that the transceiver output could provide acceptable values when the interface is in an operational state, and yet provide significantly different values when the interface goes down unexpectedly, precisely because of a sudden change in these values over or under the accepted thresholds. Even when the switch can notify about the violation of these thresholds, this is not always the case, making the problem harder to detect. To circumvent this, creating an Embedded Event Manager (EEM) script to monitor such values at the time the interface goes down is one way to address this problem. Please note that Cisco Digital Network Architecture (Cisco DNA) subscription is required in order to configure EEM scripts on Catalyst 9000 series switches.
EEM is a software component of Cisco IOS® XE that makes life easier for administrators by tracking and classifying events that occur on the switch and providing notification options for those events. EEM allows you to automate tasks, perform minor enhancements, and create workarounds.
In this example, the script is triggered when interface 1/0/24 goes down. It records the timestamp and the DOM values at the time the interface goes down, then saves that information to the logs.txt file located in the flash memory of the switch.
event manager applet connection_monitoring authorization bypass event syslog pattern "Line protocol on Interface TwentyFiveGigE1/0/24, changed state to down" maxrun 10 ratelimit 600 action 010 syslog msg "EEM trigger event received: Int Twe1/0/24 is down. EEM INIT" action 020 file open logs flash:logs.txt a+ action 030 cli command "enable" action 040 cli command "terminal length 0" action 050 cli command "terminal exec prompt expand" action 060 comment "Capturing time stamp" action 062 cli command "show clock" action 064 file write logs "$_cli_result" action 070 comment "capturing DOM values" action 080 cli command "show interfaces twentyFiveGigE 1/0/24 transceiver detail" action 090 file write logs "$_cli_result" action 100 file close logs action 120 syslog msg "EEM Successfully executed: DOM values for int Twe1/0/24 captured. EEM FIN"
Voltage and Current
These are exponentially related characteristics of the electrical input required for the diode to push electrons to low-energy stages that convert this energy into photons used as laser/LED output in the form of infrared electromagnetic waves. This electrical input must fall within given thresholds in order to guarantee the operability of the SFP. In order to obtain the current and voltage values and their upper and lower thresholds run the show interfaces <interface-id> transceiver detail command.
Switch#show interfaces twentyFiveGigE 1/0/24 transceiver detail
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.
High Alarm High Warn Low Warn Low Alarm
Temperature Threshold Threshold Threshold Threshold
Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)
--------- ----------------- ---------- --------- --------- ---------
Twe1/0/24 20.6 75.0 70.0 0.0 -5.0
High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
--------- ----------------- ---------- --------- --------- ---------
Twe1/0/24 3.30 3.63 3.46 3.13 2.97
High Alarm High Warn Low Warn Low Alarm
Current Threshold Threshold Threshold Threshold
Port Lane (milliamperes) (mA) (mA) (mA) (mA)
--------- ---- --------------- ---------- --------- --------- ---------
Twe1/0/24 N/A 26.7 75.0 70.0 18.0 15.0
Optical High Alarm High Warn Low Warn Low Alarm
Transmit Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Twe1/0/24 N/A -2.2 3.5 0.5 -8.2 -12.2
Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Twe1/0/24 N/A -2.0 3.5 0.5 -14.1 -18.4In this output, the Current right now is 26.7 milliamperes, and the Voltage is currently 3.30 volts. In this scenario any Current value over 70 milliamperes or under 18 milliamperes, based on the warning thresholds to the right, is considered a problem.
High Alarm High Warn Low Warn Low Alarm
Current Threshold Threshold Threshold Threshold
Port Lane (milliamperes) (mA) (mA) (mA) (mA)
--------- ---- --------------- ---------- --------- --------- ---------
Twe1/0/24 N/A 26.7 75.0 70.0 18.0 15.0On the other hand, any value over 3.46 volts or under 3.13 volts, based on the warning thresholds to the right, is considered a problem.
High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
--------- ----------------- ---------- --------- --------- ---------
Twe1/0/24 3.30 3.63 3.46 3.13 2.97Low or high measurements of these values are related to a problem either in the SFP or the switch hosting the SFP.
Non-Return-to-Zero (NRZ) vs. Pulse Amplitude Modulation Level-4 (PAM4)
In order to communicate 0s and 1s through electromagnetism, the transceiver varies the strength of the signal, increasing or decreasing the range of the electromagnetic waves. Thus, binarily splitting the range. This is known as Non-Return-to-Zero (NRZ) signaling.
Non-Return-to-Zero (NRZ) Signaling
For high-performance links (for example: 100G per second), this communication method can be deprecated in favor of the optimized PAM4 (See this downloadable table), which expresses 2 binary digits instead of 1, splitting the strength range into 4 parts. Therefore, a mismatch between these two methods can lead to miscommunication between the fiber optic transceivers. Ensure both sides have the proper signaling method implemented for high-performance links.
Forward Error Correction (FEC)
FEC is a technique used to detect and correct a certain number of errors in a bitstream and appends redundant bits and Error-Correcting Code (ECC) to the message block before transmission for high-speed fiber links (for example: 25G, 100G, and 400G). As a module manufacturer, Cisco designs its transceivers to comply with specifications. When the optical transceiver operates in a Cisco host platform, FEC is enabled by default based on the optical module type detected by the host software (see this downloadable table). In the vast majority of cases, FEC implementation is dictated by the industry standard supported by the optic type.
FEC capable transceivers list a special field to identify this attribute in the output of the show interface <interface-id> capabilities command
Switch#show interfaces hundredGigE 1/0/26 capabilities | in FEC FEC: auto/off/cl91 Switch#
The example shows how to configure FEC and some of the available options:
switch(config-if)#fec? auto Enable FEC Auto-Neg cl108 Enable clause108 with 25G cl74 Enable clause74 with 25G off Turn FEC off <p/re>
Use the show interface command to verify FEC configuration:
TwentyFiveGigE1/0/13 is up, line protocol is up (connected)
Hardware is Twenty Five Gigabit Ethernet, address is xxxx.xxxx.xxxx (bia xxxx.xxxx.xxxx)
MTU 9170 bytes, BW 25000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 25Gb/s, link type is force-up, media type is SFP-25GBase-SR
Fec is auto
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
!----Lines omitted for summarization--- FEC intricacies are outside the scope of this document. For further information, navigate to Understanding FEC and Its Implementation in Cisco Optics.
Modal Bandwidth and Cable Length
Hertz represents the cycles per second of the electromagnetic waves, also known as frequency. The higher the frequency, the faster the speed of the SFP. Modal bandwidth measures the cable/SFP frequency supported per kilometer without signal degradation, this limits the length of the cable between devices. In this case is way easier to consult the length supported by the cable/SFP combination, since this does not require interpretation of the frequency/length quality relationship. In order to obtain the length supported by the transceiver navigate to the Cisco Optics-to-Device Compatibility Matrix.
Related Information
Troubleshoot Port Flaps on Catalyst 9000 Series Switches
Cisco Optics-to-Device Compatilbility Matrix
Inspection and Cleaning Procedures for Fiber-Optic Connections
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
1.0 |
28-Jan-2025 |
Initial Release |
Feedback