Troubleshoot Small Form-Factor Pluggable (SFP)/Cable Issues

Introduction

This document describes the type of switch/module/SFP and cables which should be verified as supported, when a bit/word errors problem happens.

Contributed by Afroj Ahmad and Ed Mazurek, Cisco TAC Engineers.

Cisco Multilayer Data Switch (MDS) 9000 Family Pluggable Transceivers Data Sheet

https://www.cisco.com/c/en/us/products/collateral/storage-networking/mds-9000-series-multilayer-switches/product_data_sheet09186a00801bc698.html?dtid=osscdc000283

Primarily, determine the exact quantity, length and type (OM2,OM3, etc) of cabling involved, along with the number of patch panels in it. 

The SFP actually displays its capabilities:

For a Short Wave SFP

F241-15-09-MDS9710# show interface fc1/4 transceiver details
fc1/4 sfp is present
    Name is CISCO-AVAGO
    Manufacturer's part number is AFBR-57F5PZ-CS1
    Revision is B2
    Serial number is AVA1551J9KF
    Cisco part number is 10-2666-01
    Cisco pid is DS-SFP-FC16G-SW
    FC Transmitter type is short wave laser w/o OFC (SN)
    FC Transmitter supports short distance link length
    Transmission medium is multimode laser with 62.5 um aperture (M6)
    Supported speeds are - Min speed: 4000 Mb/s, Max speed: 16000 Mb/s
    Nominal bit rate is 14000 Mb/s
    Link length supported for 50/125um OM2 fiber is 35 m
    Link length supported for 62.5/125um fiber is 15 m
    Link length supported for 50/125um OM3 fiber is 100 m
    Cisco extended id is unknown (0x0)

    No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68
    SFP Diagnostics Information:
----------------------------------------------------------------------------
                                     Alarms                  Warnings
                                High        Low         High          Low
----------------------------------------------------------------------------
  Temperature   33.48 C         75.00 C     -5.00 C     70.00 C        0.00 C
  Voltage        3.29 V          3.63 V      2.97 V      3.46 V        3.13 V
  Current        7.46 mA        10.50 mA     2.50 mA    10.50 mA       2.50 mA
  Tx Power      -2.54 dBm        1.70 dBm  -13.00 dBm   -1.30 dBm     -9.00 dBm
  Rx Power      -2.32 dBm        3.00 dBm  -15.90 dBm    0.00 dBm    -11.90 dBm
  Transmit Fault Count = 0
----------------------------------------------------------------------------
  Note: ++  high-alarm; +  high-warning; --  low-alarm; -  low-warning

F241-15-09-MDS9710#

The above indicates the type of SFP is a DS-SFP-FC16G-SW and the maximum length is 100 meeters with 50/125um (mIcron) OM3 cabling.

For a Long Wave SFP

F241-15-09-MDS9710# show interface fc9/1 transceiver details
fc9/1 sfp is present
    Name is CISCO-FINISAR
    Manufacturer's part number is FTLF1432P3BCV-C1
    Revision is B
    Serial number is FNS21190B7F
    Cisco part number is 10-3207-01
    Cisco pid is DS-SFP-FC32G LW
    FC Transmitter type is long wave laser cost reduced
    FC Transmitter supports long distance link length
    Transmission medium is single mode (SM) laser
    Supported speeds are - Min speed: 8000 Mb/s, Max speed: 32000 Mb/s
    Nominal bit rate is 28000 Mb/s
    Link length supported for 9/125um fiber is 10 km
    Cisco extended id is unknown (0x0)

    No tx fault, no rx loss, in sync state, diagnostic monitoring type is 0x68
    SFP Diagnostics Information:
----------------------------------------------------------------------------
                                     Alarms                  Warnings
                                High        Low         High          Low
----------------------------------------------------------------------------
  Temperature   32.52 C         75.00 C     -5.00 C     70.00 C        0.00 C
  Voltage        3.37 V          3.63 V      2.97 V      3.46 V        3.13 V
  Current       38.55 mA        70.00 mA     1.00 mA    68.00 mA       2.00 mA
  Tx Power       0.49 dBm        5.00 dBm  -12.40 dBm    2.00 dBm     -8.40 dBm
  Rx Power      -7.43 dBm        5.00 dBm  -18.01 dBm    2.00 dBm    -14.00 dBm
  Transmit Fault Count = 0
----------------------------------------------------------------------------
  Note: ++  high-alarm; +  high-warning; --  low-alarm; -  low-warning

F241-15-09-MDS9710#

The above indicates the type of SFP is a DS-SFP-FC32G-LW and the maximum length is 10KM.

Note: The lengths shown are maximum lengths under perfect conditions. Patch panels and additional lengths of fiber in the path shortens the distance, sometimes considerably.

Patch panels and other intermediate connections are frequently a source of problems. You should always try and eliminate these as a diagnostic step. Ensure that this is done in a methodical approach and results are documented with each change.

Note the Rx power in the above output is within the acceptable range:

 -18.01 dBm   -14.00 dBm           -7.43 dBm         0.00 dBm        2.00 dBm        5.00 dBm
<---------+------------+-------------------+----------------+---------------+---------------+------->
 
<---Error-+                                                                                 Error--->
          <------Warning                                                    Warning--------->
                        <-------------------------OK----------------------->
 

Rx Power is in the acceptable range doesn't indicate that the end to end fiber/jumpers/patch panel connections are OK. You may still need to bypass some of them one at a time.

Normally, problems with excessive bit/word errors are not an ASIC problem. However, if you want to move the cables around to different ports on the same module then you need to know the archetecture of the module (port layport per ASIC). 

For Example:

MDS 9500  (DS-X9248-256K9) has 4 FC ASICs called Thunderbirds.

These FC ASICs each handle 12 ports :
ASIC 0 - fc1/1-12
ASIC 1 - fc1/13-24
ASIC 2 - fc1/25-36
ASIC 3 - fc1/37-48

MDS has built in ISL diagnostics that can be run.

Types of Tests

Here's how you run the diagnostic tests on the link.

Latency/Cable Length Test

This is just a short duration test that will measure the latency and determine the cable length. Here's how you do it:

Side A - Call this the generator side. It generates the traffic.
Side B - Call this the reflector side. It receives the traffic from the generator and sends it back.

For example:

Side A(generator) fc9/1 ---- fc6/1 Side B(reflector)

1.1 Side B(reflector)

1.1.1 - shutdown the interface to be used
1.1.2 - diagnostic isl reflector latency_test loop-back interface fc6/1 enable

1.2 Side A(generator)
1.2.1 - shutdown the interface to be used
1.2.2 - diagnostic isl latency-test interface fc9/1

Here's what it looks like in the lab switch: 

 F241-15-09-MDS9710# diagnostic isl latency-test interface fc9/1
 Waiting for sync to be achieved on the link ....
 Sync is achieved, Link has been initialized.
 Starting the test ....


 ---------------------------------------------------------
 Latency test Result for port: fc9/1
 Latency in the switch (in ns): 264
 Latency in the cable (in ns): 6593
 Length of the cable (accuracy ± 2m): 4373 m
 ---------------------------------------------------------
 F241-15-09-MDS9710#
 

Traffic Generator Test

This is a long term full line rate test. Up to 3600 seconds (1 hour)

2.1 Side B(reflector)

2.1.1 - shutdown the interface to be used
2.1.2 - diagnostic isl reflector traffic_test link_speed 32G loop-back interface fc6/1 enable

2.2 Side A(generator)
2.2.1 - shutdown the interface to be used
2.2.2 - diagnostic isl generator interface fc9/1 start duration 3600 rate 100% frame_size min 16 max 517 step 100 link_speed 32g

This runs for 1 hour at 32G full line rate.

Here's what it looks like in the lab switch:

 F241-15-09-MDS9710# diagnostic isl generator interface fc9/1 start duration 3600 rate 100% frame_size min 16 max 517 step 100 link_speed 32g
 Waiting for sync to be achieved on the link ....
 Link initialized successfully. Starting the test.
 F241-15-09-MDS9710#
When it end there is no message but you can see the frames on the ports:
 F241-15-10-9706-2# show int fc6/1 | i fc|rate
 fc6/1 is down (Administratively down)
 5 minutes input rate 5754800992 bits/sec,719350124 bytes/sec, 666558 frames/sec
 5 minutes output rate 5754800512 bits/sec,719350064 bytes/sec, 666558 frames/sec
 F241-15-10-9706-2#
 F241-15-10-9706-2# show int fc6/1 counters details | i i fc|fec|crc
 fc6/1
 0 invalid CRCs, 0 Delimiter Errors
 regardless of the CRC/FCS error
 CRC/FCS error
 7262 fec corrected blocks
 0 fec uncorrected blocks
 F241-15-10-9706-2#

It is recommended that you clear the interface counters and run this on each of the links for an hour. Then get the above CRC and FEC information on each side.

In order to clear the interface counters issue, use clear counters interface all.

Configure a Scheduler Job

The above traffic generator test runs for maximum one hour. To run it longer (like 24 hours) you can configure a scheduler job:

Need to configured the scheduler, there are two parts:

• scheduler job     
• scheduler schedule

Scheduler Job is where you configure what you want to execute while scheduler schedule is where you configure when you want the job run.This schedule runs until you remove the schedule. There’s no easy way to stop it manually. 

You need to replace the interfaces with the ones in your fabric.
If you make a mistake with the configuration of the job, you have to delete it and start again. It doesn’t allow you to go back to modify it.
Please note that on the generator side, a generator stop is there as it runs for the very first time, you might see an error which is okay. The next time it runs, it should be good.

Here’s the script:

Generator:
---------

shutdown fc1/42
feature scheduler
scheduler logfile size 1024
clear counters interface fc1/42
clear scheduler logfile

scheduler job name stats
diagnostic isl generator interface fc1/42 stop
show interface fc1/42 counters details | i i fc|fec|crc
show interface fc1/42 counters | i i fc|rate
diagnostic isl generator interface fc1/42 start duration 3600 frame_size min 64 max 517 step 1 link_speed 32G
exit

scheduler schedule name test
job name stats
time start +00:01:00 repeat 00:01:00
exit


Reflector:
---------
shutdown fc7/1
feature scheduler
scheduler logfile size 1024
clear counters interface fc7/1
clear scheduler logfile

diagnostic isl reflector traffic_test link_speed 32G loop-back interface fc7/1 enable

scheduler job name stats
show interface fc7/1 counters details | i i fc|fec|crc
show interface fc7/1 counters | i i fc|rate
exit

scheduler schedule name test
job name stats
time start +00:01:00 repeat 00:01:00
exit

At the end of 24 hours, please remove the scheduler schedule to stop it at both generator and reflector sides:
no scheduler schedule name test

You can collect the scheduler logfile which will contain all the output of the tests and the show command:
show scheduler logfile | no-more
 

Additional Commands:

• show logging onboard status
• show logging onboard module <module number>
• show logging onboard stack-trace
• show logging onboard mem-leak
• show logging onboard error-stats
• show logging onboard exception-log
• show logging onboard error-stats
• show logging onboard environmental-history