Troubleshoot Fiber Links on Catalyst 9000 Series Switches

Introduction

This document describes how to troubleshoot fiber optic interfaces by addressing some of the fiber optic module and cabling specifications.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on all Catalyst 9000 Series switches. This includes Doppler Based and Silicon One (S1) switches.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

The intricacies of dealing with fiber optics are commonly under-estimated, and therefore, mistakes can be committed while implementing new fiber links; low performance, interface errors, and connectivity problems can arise out of choosing the wrong fiber cable.

Troubleshoot

The intention of this document is to explain some of the specifications that appear in the Cisco Optics-to-Device Compatibility Matrix and the importance of paying specific attention to the transceiver and cable specification while implementing and troubleshooting fiber optic link.

Warning: Visual contact with Fiber optic lasers can cause eyes damage. Safety security measures must be taken when working with fiber optics lasers. Please refer to the General Reminders and Warnings section of the Inspection and Cleaning Procedures for Fiber-Optic Connections document for further information.

Cisco Compatible Small Form Plugable (SFP) Transceivers

Insertion of non-compatible and/or third-party SFPs can lead to unpredictable behavior, and thus the stability of the link is not guaranteed in the absence of Cisco original compatible transceivers. Thus, it is recommended to connect only Cisco-compatible transceivers to Cisco equipment. You can obtain the list of compatible transceivers by visiting the Cisco Optics-to-Device Compatibility Matrix or run the show interface show interface transceiver supported-list command.

Switch#show interfaces transceiver supported-list
Transceiver Type           Cisco p/n min version 
                               supporting DOM    
------------------       -------------------------
 
   GLC-T                    NONE                    
   GLC-TE                   NONE                    
   GLC-SX-MM                NONE                    
   GLC-LH-SM                NONE                    
   GLC-ZX-SM                NONE                    
   GLC-SX-MM-RGD            CPN 2274-02             
   GLC-LX-SM-RGD            CPN 10-2293-02          
   GLC-ZX-SM-RGD            CPN 10-2366-02          
   GLC-SX-MMD               ALL                     
   GLC-LH-SMD               ALL  
!----Lines omitted for summarization---

In order to consult the SPF model connected to an interface, run the show idprom interface <interface-id> command.

Switch#show idprom interface twentyFiveGigE 1/0/24 | include PID  
  Product Identifier (PID)                  = SFP-10G-LR-S        
Switch#

Tip: The minimum Cisco IOS® XE version required for an SFP to work is listed in the compatibility matrix.

SFP Standard

SFP standards tend to be backwards compatible, but higher standards cannot be supported by a given interface. SFP standards can vary from port to port, even on the same switch front panel. This is the case for the C9500-32QC switch model. Therefore, the presence of an SFP in the compatibility matrix does not guarantee the SFP compatibility with a given interface, so it must be verified against the hardware installation notes. In order to obtain the SFP standard of the transceiver navigate to the Cisco Optics-to-Device Compatibility Matrix or run the show idprom interface <interface-id>  command.

Switch#show idprom interface twentyFiveGigE 1/0/24 | include Transceiver Type
  Transceiver Type:                         = SFP+ 10GBASE-LR (274)
Switch#

Speed Capabilities

It is important to always ensure the SFP connected at both sides of the link supports the same speed. Supported speeds can be verified with the show interface <interface-id> capabilities command. The speed and duplex setting for multigigabit links is considered best practice, and in some scenarios necessary for the links to come up.

Switch#show interfaces twentyFiveGigE 1/0/24 capabilities | include Speed          
  Speed:                 10000

Connector Type

This aspect is hard to get wrong, as using the wrong connector type does not allow the cable to be connected to the corresponding SFP slot. However, it is still important to consider when choosing the SFP and cabling. In order to consult the connector type of the transceiver navigate to Cisco Optics-to-Device Compatibility Matrix or run the show idprom interface <interface-id> command.

Switch#show idprom interface twentyFiveGigE 1/0/24 | include Connector type
  Connector type                            = LC

Polish Type

The polish type is one of the most ignored aspects when acquiring SFPs, but it is one of the most important aspects to consider. This is the termination form of the core of the cable, the actual medium that carries the signal. The polish type is designed to provide acceptable levels of Optical Return Loss (ORL); the light reflected back to the laser/LED transmitter.

Note: Consider that the decibels is a logarithm-scale, so connecting a PC cable to a transceiver that only supports UPC actually exposes the transceiver to 100 times greater levels of back reflection than it is supposed to receive.

Choosing the wrong polish type can damage the transceiver due to the ORL level, and at best, it can lead to link instability and Layer 1 errors. In order to consult the polish type of the cable, navigate to the Cisco Optics-to-Device Compatibility Matrix. Ensure the SFPs and cable match the required core polish type. 

UPC, PC, and FLAT connectors cannot always be visually identified, so refer to the specifications provided by the cable vendor.

Single Mode Fiber (SMF) vs. Multi Mode Fiber (MMF) 

In Multi-Mode fiber cables, different paths exist for the light to reach its destination. On the other hand, Single Mode cables allow only one path for the laser light.

Single Mode Fiber (SMF) vs. Multi-Mode Fiber (MMF)

There is a clear outline in the infrastructure required to support Multi Mode Fiber vs. Single Mode Fiber. For example, SMF cabling uses a 9-micron core width, which allows the light to be transmitted over a single path, and the wavelength is optimized to a range between 1300nm and 1500nm. Therefore, ensure both SFPs and cabling are either MMF or SMF fiber components. In order to consult the MMF/SMF mode navigate to Cisco Optics-to-Device Compatibility Matrix.

Parallel/Single-Strand/Duplex Cabling

.

Special attention has to be paid to duplex cables. Ensure the sender transceiver is connected to the receiver at the other side of the link for proper polarization. Worst case scenario; the sender slot is connected to the sender of the pairing device and therefore not coming up.  

Duplex Mode

Connection polarization complexity increases with parallel links as there are multiple solutions to address this problem depending  on the Multi-fiber Push On (MPO) Standard. Thus, please consider investigating in dedicated documentation when troubleshooting parallel fiber links.

Wavelength

Photo-detectors of the transceivers are calibrated to interpret certain infrared wavelength of electromagnetic fields. These wavelengths range between 850nm and 1300nm for MMF fiber links, and between 1300nm and 1500nm for SMF.

Just as our eyes are only capable of seeing a certain range of the electromagnetic spectrum, and no other, the photoreceptors are calibrated to detect certain wavelengths of the infrared spectrum. Choosing the wrong laser/LED wavelength leads to miscommunication between transceivers, if communication is even possible. Both the SFPs must be capable of reading, transmitting on the same wavelength. In order to consult the wavelength to be used navigate to Cisco Optics Product Information or run the show idprom interface <intercace-id > detail command.

Switch#show idprom interface twentyFiveGigE 1/0/24 detail | include laser wave
  Nominal laser wavelength                  = 1310 nm

Special attention must be paid to asymmetrical Receive (RX)/Transmit (TX) transceivers where the TX and RX values differ from each other, and they must be conversely matched at the other side of the link.

Transmit/Receive Power

In order to guarantee the SFP signal is understood by the other side of the link, the electromagnetic signal strength must fall within certain thresholds. This signal is measured in decibel milliwatts (dBm), and the thresholds within which the operational values reside depend on the SFP in use. In order to obtain the current TX and RX dBm values and their upper and lower thresholds run theshow interfaces <interface-id> transceiver detail command.

Switch#show interfaces twentyFiveGigE 1/0/24 transceiver detail
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, +  : high warning, -  : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.
 
                                High Alarm  High Warn  Low Warn   Low Alarm
             Temperature        Threshold   Threshold  Threshold  Threshold
Port         (Celsius)          (Celsius)   (Celsius)  (Celsius)  (Celsius)
---------    -----------------  ----------  ---------  ---------  ---------
Twe1/0/24    20.6                   75.0       70.0        0.0       -5.0
 
                                High Alarm  High Warn  Low Warn   Low Alarm
             Voltage            Threshold   Threshold  Threshold  Threshold
Port         (Volts)            (Volts)     (Volts)    (Volts)    (Volts)
---------    -----------------  ----------  ---------  ---------  ---------
Twe1/0/24    3.30                   3.63       3.46       3.13       2.97
 
                                  High Alarm  High Warn  Low Warn   Low Alarm
                 Current          Threshold   Threshold  Threshold  Threshold
Port       Lane  (milliamperes)   (mA)        (mA)       (mA)       (mA)
---------  ----  ---------------  ----------  ---------  ---------  ---------
Twe1/0/24  N/A   26.7                 75.0       70.0       18.0       15.0
 
                 Optical          High Alarm  High Warn  Low Warn   Low Alarm
                 Transmit Power   Threshold   Threshold  Threshold  Threshold
Port       Lane  (dBm)            (dBm)       (dBm)      (dBm)      (dBm)
---------  ----  ---------------  ----------  ---------  ---------  ---------
Twe1/0/24  N/A   -2.2                  3.5        0.5       -8.2      -12.2
 
                 Optical          High Alarm  High Warn  Low Warn   Low Alarm
                 Receive Power    Threshold   Threshold  Threshold  Threshold
Port       Lane  (dBm)            (dBm)       (dBm)      (dBm)      (dBm)
---------  ----  ---------------  ----------  ---------  ---------  ---------
Twe1/0/24  N/A   -2.0                  3.5        0.5      -14.1      -18.4

In this scenario, the current receive power is equal to -2.0 dBm, which is an acceptable value based on the thresholds to the right. Any value under -14.1 dBm or over 0.5 dBm (the warning thresholds) must be considered a problem, as it can potentially affect data quality and cause link flaps.

                 Optical          High Alarm  High Warn  Low Warn   Low Alarm
                 Receive Power    Threshold   Threshold  Threshold  Threshold
Port       Lane  (dBm)            (dBm)       (dBm)      (dBm)      (dBm)
---------  ----  ---------------  ----------  ---------  ---------  ---------
Twe1/0/24  N/A   -2.0                  3.5        0.5      -14.1      -18.4

Receive Power values beneath the Low Warn Threshold most of the time indicate a problem with the transceiver at the other side, the device hosting the transceiver at the opposite end of the link, or the cable connecting the transceivers together. The same applies to high Receive Power values exceeding the High Warn Threshold. A defect in the values provided by the Digital Optical Monitoring (DOM) sensors is also plausible.

In contrast, problems with the Transmit Power measurements indicate an issue with the transceiver providing these values or the switch hosting the transceiver. A defect in the values provided by the DOM sensors is also plausible.

Note: These values are provided by the Digital Monitoring Sensor (DOM) module. DOM is not integrated into all transceivers, and the minimum Cisco IOS® XE version required varies depending on the switch hosting the transceiver. To verify transceiver DOM compatibility and the minimum Cisco IOS® XE version required, navigate to the Cisco Optics-to-Device Compatibility Matrix.

When troubleshooting, consider that the transceiver output could provide acceptable values when the interface is in an operational state, and yet provide significantly different values when the interface goes down unexpectedly, precisely because of a sudden change in these values over or under the accepted thresholds. Even when the switch can notify about the violation of these thresholds, this is not always the case, making the problem harder to detect. To circumvent this, creating an Embedded Event Manager (EEM) script to monitor such values at the time the interface goes down is one way to address this problem. Please note that Cisco Digital Network Architecture (Cisco DNA) subscription is required in order to configure EEM scripts on Catalyst 9000 series switches.

EEM is a software component of Cisco IOS® XE that makes life easier for administrators by tracking and classifying events that occur on the switch and providing notification options for those events. EEM allows you to automate tasks, perform minor enhancements, and create workarounds.

In this example, the script is triggered when interface 1/0/24 goes down. It records the timestamp and the DOM values at the time the interface goes down, then saves that information to the logs.txt file located in the flash memory of the switch.

event manager applet connection_monitoring authorization bypass
event syslog pattern "Line protocol on Interface TwentyFiveGigE1/0/24, changed state to down" maxrun 10 ratelimit 600
action 010 syslog msg "EEM trigger event received: Int Twe1/0/24 is down. EEM INIT"
action 020 file open logs flash:logs.txt a+
action 030 cli command "enable"
action 040 cli command "terminal length 0"
action 050 cli command "terminal exec prompt expand"
action 060 comment "Capturing time stamp"
action 062 cli command "show clock"
action 064 file write logs "$_cli_result"
action 070 comment "capturing DOM values"
action 080 cli command "show interfaces twentyFiveGigE 1/0/24 transceiver detail"
action 090 file write logs "$_cli_result"
action 100 file close logs
action 120 syslog msg "EEM Successfully executed: DOM values for int Twe1/0/24 captured. EEM FIN"

Voltage and Current

These are exponentially related characteristics of the electrical input required for the diode to push electrons to low-energy stages that convert this energy into photons used as laser/LED output in the form of infrared electromagnetic waves. This electrical input must fall within given thresholds in order to guarantee the operability of the SFP.  In order to obtain the current and voltage values and their upper and lower thresholds run the show interfaces <interface-id> transceiver detail  command.

Switch#show interfaces twentyFiveGigE 1/0/24 transceiver detail
ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, +  : high warning, -  : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.
 
                                High Alarm  High Warn  Low Warn   Low Alarm
             Temperature        Threshold   Threshold  Threshold  Threshold
Port         (Celsius)          (Celsius)   (Celsius)  (Celsius)  (Celsius)
---------    -----------------  ----------  ---------  ---------  ---------
Twe1/0/24    20.6                   75.0       70.0        0.0       -5.0
 
                                High Alarm  High Warn  Low Warn   Low Alarm
             Voltage            Threshold   Threshold  Threshold  Threshold
Port         (Volts)            (Volts)     (Volts)    (Volts)    (Volts)
---------    -----------------  ----------  ---------  ---------  ---------
Twe1/0/24    3.30                   3.63       3.46       3.13       2.97
 
                                  High Alarm  High Warn  Low Warn   Low Alarm
                 Current          Threshold   Threshold  Threshold  Threshold
Port       Lane  (milliamperes)   (mA)        (mA)       (mA)       (mA)
---------  ----  ---------------  ----------  ---------  ---------  ---------
Twe1/0/24  N/A   26.7                 75.0       70.0       18.0       15.0
 
                 Optical          High Alarm  High Warn  Low Warn   Low Alarm
                 Transmit Power   Threshold   Threshold  Threshold  Threshold
Port       Lane  (dBm)            (dBm)       (dBm)      (dBm)      (dBm)
---------  ----  ---------------  ----------  ---------  ---------  ---------
Twe1/0/24  N/A   -2.2                  3.5        0.5       -8.2      -12.2
 
                 Optical          High Alarm  High Warn  Low Warn   Low Alarm
                 Receive Power    Threshold   Threshold  Threshold  Threshold
Port       Lane  (dBm)            (dBm)       (dBm)      (dBm)      (dBm)
---------  ----  ---------------  ----------  ---------  ---------  ---------
Twe1/0/24  N/A   -2.0                  3.5        0.5      -14.1      -18.4

In this output, the Current right now is 26.7 milliamperes, and the Voltage is currently 3.30 volts. In this scenario any Current value over 70 milliamperes or under 18 milliamperes, based on the warning thresholds to the right, is considered a problem.

                                  High Alarm  High Warn  Low Warn   Low Alarm
                 Current          Threshold   Threshold  Threshold  Threshold
Port       Lane  (milliamperes)   (mA)        (mA)       (mA)       (mA)
---------  ----  ---------------  ----------  ---------  ---------  ---------
Twe1/0/24  N/A   26.7                 75.0       70.0       18.0       15.0

On the other hand, any value over 3.46 volts or under 3.13 volts, based on the warning thresholds to the right, is considered a problem.

                               High Alarm  High Warn  Low Warn   Low Alarm
             Voltage            Threshold   Threshold  Threshold  Threshold
Port         (Volts)            (Volts)     (Volts)    (Volts)    (Volts)
---------    -----------------  ----------  ---------  ---------  ---------
Twe1/0/24    3.30                   3.63       3.46       3.13       2.97

Low or high measurements of these values are related to a problem either in the SFP or the switch hosting the SFP.

Non-Return-to-Zero (NRZ) vs. Pulse Amplitude Modulation Level-4 (PAM4)

In order to communicate 0s and 1s through electromagnetism, the transceiver varies the strength of the signal, increasing or decreasing the range of the electromagnetic waves. Thus, binarily splitting the range. This is known as Non-Return-to-Zero (NRZ) signaling.

Non-Return-to-Zero (NRZ) Signaling

For high-performance links (for example: 100G per second), this communication method can be deprecated in favor of the optimized PAM4 (See this downloadable table), which expresses 2 binary digits instead of 1, splitting the strength range into 4 parts. Therefore, a mismatch between these two methods can lead to miscommunication between the fiber optic transceivers. Ensure both sides have the proper signaling method implemented for high-performance links.

Forward Error Correction (FEC)

FEC is a technique used to detect and correct a certain number of errors in a bitstream and appends redundant bits and Error-Correcting Code (ECC) to the message block before transmission for high-speed fiber links (for example: 25G, 100G, and 400G). As a module manufacturer, Cisco designs its transceivers to comply with specifications. When the optical transceiver operates in a Cisco host platform, FEC is enabled by default based on the optical module type detected by the host software  (see this downloadable table). In the vast majority of cases, FEC implementation is dictated by the industry standard supported by the optic type.

FEC capable transceivers list a special field to identify this attribute in the output of  the show interface <interface-id> capabilities command

Switch#show interfaces hundredGigE 1/0/26 capabilities | in FEC
  FEC:                   auto/off/cl91
Switch#

The example shows how to configure FEC and some of the available options:

switch(config-if)#fec?
  auto Enable FEC Auto-Neg
  cl108 Enable clause108 with 25G
  cl74 Enable clause74 with 25G
  off Turn FEC off
<p/re>

Use the show interface command to verify FEC configuration:

TwentyFiveGigE1/0/13 is up, line protocol is up (connected)
  Hardware is Twenty Five Gigabit Ethernet, address is xxxx.xxxx.xxxx (bia xxxx.xxxx.xxxx)
  MTU 9170 bytes, BW 25000000 Kbit/sec, DLY 10 usec,
      reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 25Gb/s, link type is force-up, media type is SFP-25GBase-SR
  Fec is auto     
  input flow-control is on, output flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
 !----Lines omitted for summarization---  

FEC intricacies are outside the scope of this document. For further information, navigate to Understanding FEC and Its Implementation in Cisco Optics.

Modal Bandwidth and Cable Length

Hertz represents the cycles per second of the electromagnetic waves, also known as frequency. The higher the frequency, the faster the speed of the SFP. Modal bandwidth measures the cable/SFP frequency supported per kilometer without signal degradation, this limits the length of the cable between devices. In this case is way easier to consult the length supported by the cable/SFP combination, since this does not require interpretation of the frequency/length quality relationship. In order to obtain the length supported by the transceiver navigate to the Cisco Optics-to-Device Compatibility Matrix.

Related Information

Troubleshoot Port Flaps on Catalyst 9000 Series Switches

Cisco Optics-to-Device Compatilbility Matrix

Inspection and Cleaning Procedures for Fiber-Optic Connections

Understanding FEC and Its Implementation in Cisco Optics.