Troubleshoot Interface down or Flap Issues on NCS

Introduction

This document describes how to troubleshoot interface down, or interface flap issues.  

This document is specific to Cisco IOS^®XR, but it is not restricted to a specific software release.

This document is specific to the NCS platform.

Background

There are multiple reasons why interface down, or interface flap can happen. The links can go down due to multiple devices as shown in image 1. It shows an NCS router with one front panel port connected to the NPU (Network Processor Unit) via PHY (Physical Layer) device in between (known as PHY based ports). There are platforms where the front panel ports are directly connected to the NPU (known as PHYless ports). A single router can have both variants.

Image 1

Optics

Interface

Note that an interface is always described in this way on Cisco IOS XR.

R/S/I/P  : Rack/Slot/Instance/Port/Breakout

R/S/I/P/B : Rack/Slot/Instance/Port/

Port Speed

Some platforms/line cards support multiple speeds on some ports. Examples are: 1/10GE, 25/40GE, 40/100GE, 1/10/25GE.

The speed can be configured on each individual port or per group of ports. Look at the hardware documentation for the port mapping on the NCS. Check if the port speed is set correctly. Some ports can auto-sense the speed, others need configuration.

Some ports can be configured per group (usually a quad: a group of four).

For example:

(config)#hw-module quad 0 location 0/0/CPU0 mode ?
  WORD  10g or 25g

Some ports can be configured individually. This is usually the 100G ports. 100G is the default speed, but you can configure 40G.

For example:

(config)#controller optics 0/0/1/0
(config-Optics)#speed 40g

Fault Signaling Mechanism

Note that a Link Fault Signaling mechanism can play a role when you troubleshoot the interface issue. On 10G and higher, there is Link Fault Signaling mechanism included in the PHY layer. So when a local fault is detected, the remote end is informed.

Image 2

On GigE interface, there is no such onboard mechanism. Check if “ethernet oam” is configured as an alternative.

For example:

interface GigabitEthernet0/6/0/2
 ethernet oam
  uni-directional link-fault detection
  action
   uni-directional link-fault efd

DWDM Tunable Optic

On DWDM optic that supports frequency tuning, use this config to set the frequency that can be used (based on DWDM circuit setup by the circuit provider):

This is the command:

controller optics 0/6/0/2
 dwdm-carrier {100MHz-grid | 50GHz-grid} frequency {frequency}

Verify the settings:

RP/0/RP0/CPU0:NCS-5508#show controllers optics 0/6/0/2 dwdm-carrier-map
DWDM Carrier Band:: OPTICS_C_BAND
MSA ITU channel range supported: 1~96
DWDM Carrier Map table
----------------------------------------------------
ITU Ch      G.694.1     Frequency     Wavelength
 Num        Ch Num        (THz)          (nm)
----------------------------------------------------
     1       -35           191.35      1566.723
----------------------------------------------------
     2       -34           191.40      1566.314
----------------------------------------------------
     3       -33           191.45      1565.905
----------------------------------------------------
     4       -32           191.50      1565.496
----------------------------------------------------
...
----------------------------------------------------
    95        59           196.05      1529.163
----------------------------------------------------
    96        60           196.10      1528.773
----------------------------------------------------

On DWDM optics, all DWDM specific tuning is done under controller dwdm config :

RP/0/RP0/CPU0:NCS-5508(config)#controller dwdm 0/6/0/2 ?
  admin-state       Configure the transport admin state of the controller: in-service, out-of-service or maintenance
  g709              Configure G709 parameters
  laser             To be deprecated and removed in 7.5.1 release
  log               Proactive log
  loopback          Configure loopback mode
  network           Configure DWDM network information
  pm                Configure pm parameters
  proactive         Enable Proactive Protection Feature
  rx-los-threshold  Configure transponder receive power LOS threshold
  transmit-power    Configure transponder transmit power
  vtxp-monitor      Enable VTXP monitoring function
  wavelength        Configure ITU Channel, Wavelength and Frequency
  <cr>

RP/0/RP0/CPU0:NCS-5508(config-dwdm)#g709 fec ?
  15sdfec                       15%-SD Forward Error Correction
  15sdfecde                     15%-SD Forward Error Correction with Diff
  ci-bch                        Continuously Interleaved BCH FEC
  disable                       Disable FEC
  enhanced                      Enhanced FEC mode
  high-gain                     To be deprecated and removed in 7.5.1 release
  high-gain-hd-fec              7% HD FEC (Staircase FEC)
  high-gain-multivendor-hd-fec  7% HD FEC (Staircase FEC) Multivendor Interoperable
  high-gain-sd-fec              7% CISCO SD FEC (Soft-Decision FEC)
  long-haul                     To be deprecated and removed in 7.5.1 release
  long-haul-hd-fec              20% HD FEC (Staircase FEC)
  long-haul-sd-fec              20% CISCO SD FEC (Soft-Decision FEC)
  standard                      Standard FEC mode

IPoDWDM - Coherent DSP

Coherent DSP supports high rate speed (100/200/400G) over long distances. The DWDM processing is handled by Coherent DSP (Digital Signal Processor).

Examples of such NCS modules: NC55-6X200-DWDM-S or NC55-MPA-2TH-S (with CFP2-WDM-D-1HL)

RP/0/RP0/CPU0:NCS-5508(config)# controller coherentDSP 0/6/0/0
RP/0/RP0/CPU0:NCS-5508(config-CoDSP)#fec ?
  CFEC                 Forward Error Correction C_FEC
  EnhancedHG20         Forward Error Correction Enhanced_HG20
  EnhancedHG7          Forward Error Correction Enhanced_HG7
  EnhancedI4           Forward Error Correction Enhanced_I_4
  EnhancedI7           Forward Error Correction Enhanced_I_7
  EnhancedSD15         Forward Error Correction Soft-Decision 15
  EnhancedSD15DE       Forward Error Correction Soft-Decision 15 with DE
  EnhancedSD20         Forward Error Correction Soft-Decision 20
  EnhancedSD27         Forward Error Correction Soft-Decision 27
  EnhancedSD7          Forward Error Correction Soft-Decision 7
  EnhancedStaircaseDE  Forward Error Correction Enhanced Staircase FEC with DE
  EnhancedSwizzle      Forward Error Correction Enhanced_Swizzle
  OFEC                 Forward Error Correction O_FEC
  Standard             Forward Error Correction Standard
  none                 No Forward Error Correction

Detailed Troubleshooting

This section covers the basic debugging/techniques from optics perspective.

First Things to Check

Check if the optics show up in the inventory. If the optics are not shown, check if the optics are inserted properly in the R/S/I/P and cables with no issues are used.

++++++++++ show inventory details [18:06:56.572 UTC Thu Apr 06 2023] ++++++++++

NAME: "0/RP0", DESCR: "NC55A1 24Q6H SS Route Processor Card"
PID: NCS-55A1-24Q6H-SS , VID: V01, SN: FOC2528002Q
MFG_NAME: Cisco Systems, Inc., SNMP_IDX: 1 , Type: Module
PN: 73-20057-02

NAME: "TenGigE0/0/0/0", DESCR: "Cisco SFP+ 10G ZR Pluggable Optics Module"
PID: SFP-10G-ZR , VID: V02, SN: BD211218N1T
MFG_NAME: CISCO-PRE , SNMP_IDX: 2129921 , Type: Module
PN: TSFP10G-1558.17

NAME: "TenGigE0/0/0/1", DESCR: "Cisco SFP+ 10G ZR Pluggable Optics Module"
PID: SFP-10G-ZR , VID: V02, SN: BD211218N3K
MFG_NAME: CISCO-PRE , SNMP_IDX: 2134017 , Type: Module
PN: TSFP10G-1557.36

RP/0/RP0/CPU0:NCS-5501#show controllers te0/0/0/14 internal
Internal data for interface: TenGigE0/0/0/14
Subport Number      : 255
Port Number         : 14 *
Bay Number          : 0 *
Board Type          : 0x60020201 *
Port Type           : 10GE *
Bandwidth(Kbps)     : 10000000 *
Transport mode      : LAN *
BIA MAC addr        : 008a:9617:4838
Oper. MAC addr      : 008a:9617:4838
Egress MAC addr     : 008a:9617:4838
Port Available      : true *
Status polling is   : disabled *
Status events are   : disabled
I/F Handle          : 0x00000158 *
Cfg Link Enabled    : enabled
H/W Tx Enable       : yes
MTU                 : 1514 *
H/W Speed           : 10 Gbps *
H/W Duplex          : Full *
H/W Loopback Type   : None *
FEC                 : Not Configured *
H/W FlowCtrl Type   : Disabled *
H/W AutoNeg Enable  : Off *
H/W Link Defects    : No Fault *
Link Up             : yes *
Link Led Status     : Green ON *
Pluggable Present   : Yes *
Pluggable Type      : SFP+ 10G CU3M
Pluggable PID       : SFP-H10GB-CU3M *
Pluggable Compl.    : Failed - Bad Vendor CRC

If the transceiver is not recognized, check if transceiver type is supported here: https://tmgmatrix.cisco.com/.

The vendor CRC should be correct.

Check the signal strength.

Requires DOM (Digital Optical Monitoring) support on the transceiver!

RP/0/RP0/CPU0:BRU-SPCORE-P2#show controllers hundredGigE0/0/0/10 phy

QSFP8636 EEPROM port: 10
        Xcvr Type: QSFP28
        Ext Type: 3.5W, CLEI, TX CDR, RX CDR,
        Connector Type: MPO
        Ethernet Compliance Codes: 100G BASE-SR4,
        BR, nominal: 25500 Mbps
        Length SMF: 0KM, OM3: 70M, OM2: 0M, OM1: 0M, Cable: 50M
        Deice Tech: 850nm VCSEL,
        Vendor Name: CISCO-FINISAR
        Vendor OUI: 00.90.65
        Vendor Part Number: FTLC9555REPM-C1   (rev.: A )
        Wavelength: 850.000 nm
        Wavelength Tolerance: 10.000 nm
        Vendor Serial Number: FIW2638016W
        Date Code (yy/mm/dd): 22/09/13  lot code:
        Diagnostic Monitoring Type: RX Avg, TX,
        Enhanced Options: Init Complete Flag Impl,
        Extended Module Codes:
        Options:
        L-Tx/Rx LOS:
        L-Tx Fault:
        L-Tx/Rx LOL:
        Module DDM: Volt, Temp, TX Power, TX Bias, RX Power,
MSA Data (Lower Memory)
0x0000: 11 07 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00
0x0010: 00 00 00 00 00 00 1d 75 : 00 00 81 2f 00 00 00 00
0x0020: 00 00 20 b6 2e 9a 2d ba : 27 44 0d ed 0e 0c 0e 0c
0x0030: 0e 0c 28 46 2a dc 29 1f : 2a 72 00 00 00 00 00 00
0x0040: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00
0x0050: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 01 00 00
0x0060: 00 00 ff 00 00 00 00 00 : 00 01 1f 00 00 00 00 00
0x0070: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00


MSA Data (Upper Memory Page 00)
0x0080: 11 dc 0c 80 00 00 00 00 : 00 00 00 05 ff 00 00 23
0x0090: 00 00 32 00 43 49 53 43 : 4f 2d 46 49 4e 49 53 41
0x00a0: 52 20 20 20 00 00 90 65 : 46 54 4c 43 39 35 35 35
0x00b0: 52 45 50 4d 2d 43 31 20 : 41 20 42 68 07 d0 46 ef
0x00c0: 02 07 ff f6 46 49 57 32 : 36 33 38 30 31 36 57 20
0x00d0: 20 20 20 20 32 32 30 39 : 31 33 20 20 0c 10 68 3a
0x00e0: 00 00 02 3c c0 ff c6 b6 : 3b 05 e6 30 86 bb 80 05
0x00f0: df 65 71 00 00 00 00 00 : 00 00 00 00 79 eb c1 06

        CLEI Code: CMUIATKCAA
        Part Number: FTLC9555REPM-C1 (ver.: V03)
        Product Id: QSFP-100G-SR4-S

MSA Data (Upper Memory Page 02)
0x0180: 43 4d 55 49 41 54 4b 43 : 41 41 31 30 2d 33 31 34
0x0190: 32 2d 30 33 56 30 33 20 : 01 00 00 31 20 20 20 20
0x01a0: 20 66 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00
0x01b0: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 aa aa
0x01c0: 51 53 46 50 2d 31 30 30 : 47 2d 53 52 34 2d 53 20
0x01d0: 20 20 20 20 00 00 00 00 : 00 00 00 00 00 00 00 65
0x01e0: 31 33 33 39 39 37 31 36 : 31 d8 00 00 00 00 00 00
0x01f0: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00

        Module
        Thresholds:                   Alarm High          Warning High          Warning Low          Alarm Low
              Temperature:            +75.000 C             +70.000 C              +0.000 C              -5.000 C
                  Voltage:           3.630 Volt            3.465 Volt            3.135 Volt            2.970 Volt

        Temperature: +29.457 C
        Voltage: 3.315 Volt

        Lanes
        Thresholds:                   Alarm High          Warning High          Warning Low          Alarm Low
                     Bias:         15.000 mAmps          14.000 mAmps           3.000 mAmps           2.000 mAmps
           Transmit Power:  3.46740 mW (5.40004 dBm)   1.73780 mW (2.40000 dBm)   0.14450 mW (-8.40132 dBm)   0.05750 mW (-12.40332 dBm)
            Receive Power:  3.46740 mW (5.40004 dBm)   1.73780 mW (2.40000 dBm)   0.09330 mW (-10.30118 dBm)   0.03720 mW (-14.29457 dBm)

        TxRxIOMagId:

        RxOpAmpSupprt:

                 Lane            Temp            Bias                   Tx Power                   Rx Power

                    0             N/A     7.130 mAmps   1.03100 mW (0.13259 dBm)   1.18990 mW (0.75510 dBm)
                    1             N/A     7.192 mAmps   1.09760 mW (0.40444 dBm)   1.19300 mW (0.76640 dBm)
                    2             N/A     7.192 mAmps   1.05440 mW (0.23005 dBm)   1.17110 mW (0.68594 dBm)
                    3             N/A     7.192 mAmps   1.08020 mW (0.33504 dBm)   1.00480 mW (0.02080 dBm)

MSA Data (Upper Memory Page 03)
0x0200: 4b 00 fb 00 46 00 00 00 : 00 00 00 00 00 00 00 00
0x0210: 8d cc 74 04 87 5a 7a 76 : 00 00 00 00 00 00 00 00
0x0220: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00
0x0230: 87 72 01 74 43 e2 03 a5 : 1d 4c 03 e8 1b 58 05 dc
0x0240: 87 72 02 3f 43 e2 05 a5 : 00 00 00 00 00 00 00 00
0x0250: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00
0x0260: a6 0f 00 00 00 00 00 00 : 00 00 66 66 00 00 33 33
0x0270: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00

Use this command to verify for any issues.

RP/0/RP0/CPU0:BRU-SPCORE-P2#show controllers optics 0/0/0/10 summary

Port                Controller State        Admin State            LED State         Lane   Laser Bias   TX Power     RX Power     Wavelength     Optics Type       Description
                                                                                                          (dBm)        (dBm)        (nm)
------------------  ----------------------  ---------------------  ----------------  -----  ------------ -----------  ----------   -----------    --------------    -----------
Optics 0/0/0/10     Up                      In Service             Green             0      7.1mA        0.12         0.75         850.00         QSFP28-100G-SR4
                                                                                     1      7.2mA        0.41         0.75
                                                                                     2      7.2mA        0.22         0.69
                                                                                     3      7.2mA        0.31         0.02

Alarms

Check the alarms for topic entries.

RP/0/RP0/CPU0:NCS#show alarms brief

--------------------------------------------------------------------------------
Active Alarms for 0/0
--------------------------------------------------------------------------------
Location        Severity     Group            Set Time                   Description                                                                                    
--------------------------------------------------------------------------------
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX LOS LANE-0 ALARM                                              
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX LOS LANE-1 ALARM                                              
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX LOS LANE-2 ALARM                                              
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX LOS LANE-3 ALARM                                              
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX LOL LANE-0 ALARM                                              
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX LOL LANE-1 ALARM                                              
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX LOL LANE-2 ALARM                                              
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX LOL LANE-3 ALARM                                              
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX POWER LANE-0 LOW ALARM                                        
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX POWER LANE-1 LOW ALARM                                        
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX POWER LANE-2 LOW ALARM                                        
0/0/CPU0        Major        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX POWER LANE-3 LOW ALARM                                        
0/0/CPU0        Minor        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX POWER LANE-0 LOW WARNING                                      
0/0/CPU0        Minor        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX POWER LANE-1 LOW WARNING                                      
0/0/CPU0        Minor        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX POWER LANE-2 LOW WARNING                                      
0/0/CPU0        Minor        Software         04/26/2023 15:50:19 CEST    Optics0/0/0/35 - hw_optics:  RX POWER LANE-3 LOW WARNING                                      
0/0/CPU0        Major        Software         05/02/2023 11:18:17 CEST    Optics0/0/0/26 - hw_optics:  OPTICS Module is not supported on this port                      
0/0/CPU0        Major        Software         05/02/2023 11:18:17 CEST    Optics0/0/0/26 - hw_optics: Port speed not supported on the optics                            
0/0/CPU0        Major        Software         05/02/2023 11:18:23 CEST    Optics0/0/0/26 - hw_optics:  RX LOS LANE-0 ALARM                                              
0/0/CPU0        Major        Software         05/02/2023 11:18:23 CEST    Optics0/0/0/26 - hw_optics:  RX LOS LANE-1 ALARM                          
…

Show Controller Optics

Check if the SFP/QSFP is detected with the command “show controller optics <r/s/i/p>”.

Check the this:

• Check if the inserted optics is displayed properly (check the PID and VID).
• Check if the laser is enabled.
• Check if power levels are good.
• Check if any alarms/faults are raised.

This is sample output if the optics are detected and in service.

Check is the “Controller State” is Up or Down.

Check if the Laser State is On or Off. The state Off is seen when the interface is shut. If this is not the case, collect this:

• show tech-support optics
• show tech-support ofa

RP/0/RP0/CPU0:ios#show controllers optics 0/0/0/12
Controller State: Up
 Transport Admin State: In Service
 Laser State: Off

Check the LED state. The state Off is seen when the interface is shutdown. When the interface is not shutdown, the state can be Down-Yellow or Up-Green.

If there is port breakout: If even one subport is up, all subports show Green. If at least one subport is admin up (no shut), all subports show Yellow. If all subports are admin down (shut), then the Led state is Off.

Check the FEC state. It is possible that some platform types have the FEC disabled. Then the FEC part is not displayed.If FEC is enabled on PHY, it should show PHY FEC status not NPU. If the FEC status is not correct, check the FEC Status in “show controller <speed> <r/s/i/p>”. 

Read this article to learn more on FEC: Understanding FEC and Its Implementation in Cisco Optics

Download this table for more details on FEC types for each Cisco optic: https://www.cisco.com/c/dam/en/us/products/se/2022/4/Collateral/fec-summary-table.pdf

Check if any alarms are raised next to “Detected Alarms”. In case of Link Issues, check if any RX-LOS/RX-LOL/TX-LOS/TX-LOL alarms are displayed here or in “show alarms brief/history” for the interface and timestamp when the link flapped. In case of RX alarms, check if the peer has TX alarms. If the peer has TX alarms, go to next step. If there are no peer TX alarms, try to reconnect the cable and or the pluggable item. Try to replace them with another if needed.

Check the TX Power. If the port is expected to be Up, but it has -40 Tx power, go to next step. Check the RX Power. If the port is expected to be Up, but it has -40dBm Rx power, check if the peer has TX alarms. If the peer has no TX alarms, try to reconnect the cable and or the pluggable item. Try to replace them with another if needed.   

RP/0/RP0/CPU0:ios#show controllers optics 0/0/0/13

 Controller State: Down
 Transport Admin State: In Service
 Laser State: Off
         Optics not present
         Optics Type: Unavailable
         DWDM Carrier Info: Unavailable, MSA ITU Channel= Unavailable, Frequency= Unavailable , Wavelength= Unavailable
         TX Power = Unavailable 
         RX Power = Unavailable
LED State: Off
 FEC State: FEC ENABLED
  Optics Status 
              Optics Type: SFP28 25G SR-S
               Wavelength = 850.00 nm 

               Alarm Status:
               -------------               
               Detected Alarms: None 
               LOS/LOL/Fault Status:
               Laser Bias Current = 0.0 mA Actual
               TX Power = -40.00 dBm 
               RX Power = -0.93 

               Performance Monitoring: Disable 
               THRESHOLD VALUES 
               ----------------  
               Parameter                           High Alarm  Low Alarm  High Warning  Low Warning
               ------------------------            ----------  ---------  ------------  ----------- 
               Rx Power Threshold(dBm)                 5.4       -14.2              2.3       -10.3 
               Tx Power Threshold(dBm)                 5.4       -12.4              2.3        -8.4 
               LBC Threshold(mA)                     10.00        2.00             8.00         3.00
               Temp. Threshold(celsius)              75.00       -5.00            70.00         0.00 
               Voltage Threshold(volt)                3.63        2.97             3.46         3.13  
               Polarization parameters not supported by optics 
               Temperature = 28.00 Celsius               Voltage = 3.28 V 

Transceiver Vendor Details
         Form Factor              : SFP28 
         Optics type              : SFP28 25G SR-S
         Name                      : CISCO-AVAGO 
         OUI Number               : 00.17.6a 
         Part Number              : SFBR-725SMZ-CS1
         Rev Number               : 01 
         Serial Number            : AVD2227E1FU 
         PID                       : SFP-25G-SR-S 
         VID                       : V01 
         Date Code(yy/mm/dd) : 18/07/07 

This is sample output if the optics are not detected or not in service.

RP/0/RP0/CPU0:ios# RP/0/RP0/CPU0:ios#show controllers optics 0/0/0/13

  UTC Controller State: Down
  Transport Admin State: In Service
  Laser State: Off
          Optics not present
          Optics Type: Unavailable 
          DWDM Carrier Info: Unavailable, MSA ITU Channel= Unavailable, Frequency= Unavailable , Wavelength= Unavailable 
          TX Power = Unavailable          RX Power = Unavailable

CoherentDSP

If the link is down for QDD-400G-ZR-S / QDD-400G-ZRP-S, along with a verification of the alarms and data from "show controller optics", do check for alarms in "show controller coherentDSP <R/S/I/P>".

RP/0/RP0/CPU0:ios#show controllers coherentDSP 0/0/1/0

Port : CoherentDSP 0/0/1/0
Controller State : Up
Inherited Secondary State : Normal
Configured Secondary State : Normal
Derived State : In Service
Loopback mode : None
BER Thresholds : SF = 1.0E-5 SD = 1.0E-7
Performance Monitoring : Enable
Bandwidth : 400.0Gb/s 

Alarm Information:LOS = 0 LOF = 0 LOM = 0
OOF = 0 OOM = 0 AIS = 0IAE = 0 B
IAE = 0 SF_BER = 0
SD_BER = 0 BDI = 0 TIM = 0
FECMISMATCH = 0 FEC-UNC = 0 FLEXO_GIDM = 0
FLEXO-MM = 0 FLEXO-LOM = 0 FLEXO-RDI = 0
FLEXO-LOF = 0 
Detected Alarms : None

If LOS/LOF/BER is seen, then check the fiber connection, the remote end TX values, and the mode of operation of the local and the peer end ZR/ZRP.

ZR/ZRP supports multiple modes of operation. Check this link configuration guide.

The configuration mode, FEC, DAC, and the modulation need to match with the interface configuration and with the peer configuration.

Use this command: show optics-driver debug optics port <fp_port> instance <bayinst> location <LC>.

RP/0/RP0/CPU0:ios#show optics-driver debug optics port 0 instance 0 location 0/0/CPU0
flexcoh_hdlr : [0x0]
R_S_I_P : [0.0.1.0]
module_type : [400G-ZRPLUS:Cisco-Qualified-Module]
Traffic-Setup : [Requested/Programmed]
client_rate : [100_GAUI_2_KP4_FEC/100_GAUI_2_KP4_FEC]                      <- needs to match the configured interface speed
trunk_rate : [400G Muxponder/400G Muxponder] line_fec_mode : [oFEC/oFEC]   <- needs to match the peer end for link up
dac_rate : [1x1.25 => pulse_shaping On/1x1.25 => pulse_shaping On]         <- needs to match the peer end for link up
modulation : [16QAM/16QAM]                                                 <- needs to match the peer end for link up
framing_format : [400G_ZR/400G_ZR]
framing_mode : [Enhanced/Enhanced]
hw_cfg_in_progress : [False]
hw transactions : [3]
polling enabled : [True]
pm_notify enabled : [True]
alarms_notify enabled : [True]
sdk laser oper state : [Enabled]
sdk hw laser oper state : [Enabled]
hw laser oper state : [Enabled]
sdk channel-frequency : [1931000]
hw channel-frequency : [1931000]
sdk tx-power : [-100 0.1dBm]
hw tx-power : [-113 0.1dBm]
hw tx-power-range : [-2289, -65135 0.01dBm]
sdk cd-min : [-13000]hw cd-min : [-13000]
sdk cd-max : [13000]hw cd-max : [13000]
sdk baud-rate : [60.138546] 
hw baud-rate : [60.138546]
sdk hw thresholds : [Valid]
config-thresh-flags : [0x0]
trf-cfg-lsr-pm-flags : [0x0]
polling_mask : [0xf]
is_fw_dl_in_progress : [False]
is_fw_commit_in_progress: [False]
sdk dsp-internal-loopback: [Disabled]
hw dsp-internal-loopback: [Disabled]
sdk dsp-line-loopback : [Disabled]
hw dsp-line-loopback : [Disabled]
Flexcoh SDK API execution status
--------------------------------------------------------------------------------------------------
traffic | tx-power | cd-min | cd-max | frequency | laser-set | pm-set | alarm-set | poll_set |
==================================================================================================
Success | Success  | Success| Success| Success   | Success   | Success| Success   | Success  |      <- No failure to be seen in any of the status
--------------------------------------------------------------------------------------------------

Show Controllers PHY

Collect ths information for the correct interface. This command dumps the EEPROM information.

RP/0/RP0/CPU0:ios#show controllers tenGigE 0/0/1/0 phy

SFP EEPROM port: 32
Xcvr Type: SFP
Xcvr Code: SFP+ 10G SR
Encoding: 64B66B
Bit Rate: 10300 Mbps
Link Reach 50u fiber: 80 meter
Link Reach 62.5u fiber: 20 meter
Vendor Name: CISCO-SUMITOMO
Vendor OUI: 00.00.5f
Vendor Part Number: SPP5100SR-C5 (rev.: A )
Laser wavelength: 850 nm (fraction: 0.00 nm)
Optional SFP Signal: Tx_Disable, Tx_Fault, LOS
Vendor Serial Number: SPC17050AZ0
Date Code (yy/mm/dd): 13/01/31 lot code: MA
Diagnostic Monitoring: DOM, Int. Cal.,
Enhanced Options: SW RX LOS Mon., SW TX Fault Mon, SW TX Disable, Alarm/Warning Flags

MSA Data
0x0000: 03 04 07 10 00 00 00 00 : 00 00 00 06 67 00 00 00
0x0010: 08 02 00 1e 43 49 53 43 : 4f 2d 53 55 4d 49 54 4f
0x0020: 4d 4f 20 20 00 00 00 5f : 53 50 50 35 31 30 30 53
0x0030: 52 2d 43 35 20 20 20 20 : 41 20 20 20 03 52 00 e6
0x0040: 00 1a 00 00 53 50 43 31 : 37 30 35 30 41 5a 30 20
0x0050: 20 20 20 20 31 33 30 31 : 33 31 4d 41 68 f0 03 7a
0x0060: 00 00 0b ea 11 8a 3a 43 : 9d 9c 2b 0d 84 89 fd c5
0x0070: a4 0e 5b 00 00 00 00 00 : 00 00 00 00 8b 64 8d fc

Thresholds: Alarm High Warning High Warning Low Alarm Low
Temperature: +75.000 C +70.000 C +0.000 C -5.000 C
Voltage: 3.630 Volt 3.465 Volt 3.135 Volt 2.970 Volt
Bias: 10.500 mAmps 9.000 mAmps 2.500 mAmps 2.000 mAmps
Transmit Power: 1.47910 mW (1.69998 dBm) 0.74130 mW (-1.30006 dBm) 0.18620 mW (-7.30020 dBm) 0.07410 mW (-11.30182 dBm)
Receive Power: 1.58490 mW (2.00002 dBm) 0.79430 mW (-1.00015 dBm) 0.10230 mW (-9.90124 dBm) 0.04070 mW (-13.90406 dBm)
Temperature: 24.012
Voltage: 3.304 Volt
Tx Bias: 0.000 mAmps
Tx Power: 0.000 mW (<-40.00 dBm)
Rx Power: 0.000 mW (<-40.00 dBm)
Oper. Status/Control: Tx Disabled, Rx Rate Select, LOS,
EEPROM Memory (A2 lower)
0x0100: 4b 00 fb 00 46 00 00 00 : 8d cc 74 04 87 5a 7a 75
0x0110: 14 82 03 e8 11 94 04 e2 : 39 c7 02 e5 1c f5 07 46
0x0120: 3d e9 01 97 1f 07 03 ff : 00 00 00 00 00 00 00 00
0x0130: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00
0x0140: 00 00 00 00 3f 80 00 00 : 00 00 00 00 01 00 00 00
0x0150: 01 00 00 00 01 00 00 00 : 01 00 00 00 00 00 00 27
0x0160: 18 03 81 13 00 00 00 00 : 00 00 00 00 00 00 b2 00
0x0170: 00 40 00 00 00 40 00 00 : 00 00 00 00 00 00 00 00

CLEI Code: COUIA8NCAA
Part Number: 10-2415-03 (ver.: V03 )
Temp/Alarm/Power Flags: COM, commercial 0C to 70C
Minimum Temperature: 0
Maximum Temperature: 70
Calibration Constants: LBC Scale, Temperature, Laser bias current, Output power,
Product Id: SFP-10G-SR
EEPROM Memory (A2 upper)
0x0180: 43 4f 55 49 41 38 4e 43 : 41 41 31 30 2d 32 34 31
0x0190: 35 2d 30 33 56 30 33 20 : 01 00 46 00 00 00 00 c6
0x01a0: 00 00 00 00 00 00 00 00 : 00 00 85 99 8f 00 a8 3b
0x01b0: d4 4b 00 00 1e 00 0a ff : 16 93 0f 8e 00 00 aa aa
0x01c0: 53 46 50 2d 31 30 47 2d : 53 52 20 20 20 20 20 20
0x01d0: 20 20 20 20 32 33 00 00 : 00 00 00 00 00 00 00 35
0x01e0: 14 1b 20 20 20 26 20 26 : 00 00 00 00 00 00 00 00
0x01f0: 00 00 00 00 00 fb 00 00 : ff ff ff ff 00 00 00 aa


MSA Data LOWER PAGE (QSA)
0x0000: 0d 00 02 01 00 00 01 00 : 00 00 00 00 00 00 00 00
0x0010: 00 00 00 00 00 01 00 00 : 00 00 00 00 00 00 00 00
0x0020: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00
0x0030: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00
0x0040: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00
0x0050: 00 00 44 32 50 30 0f 00 : 00 00 00 00 00 00 00 00
0x0060: 00 00 00 00 01 01 00 00 : 00 00 00 00 00 00 00 00
0x0070: 00 00 00 00 00 00 00 00 : 00 00 00 00 00 00 00 00

MSA Data UPPER PAGE (QSA)
0x0080: 0d 9b 80 00 00 00 00 00 : 00 00 00 00 00 00 00 00
0x0090: 00 00 00 00 43 49 53 43 : 4f 2d 44 4e 49 20 20 20
0x00a0: 20 20 20 20 00 30 30 30 : 43 41 5a 41 44 45 52 4f
0x00b0: 2d 30 32 20 20 20 20 20 : 30 33 00 00 00 00 46 32
0x00c0: 00 00 00 00 44 54 59 32 : 32 32 31 30 36 47 38 20
0x00d0: 20 20 20 20 32 30 31 38 : 30 35 33 30 00 00 f2 c2
0x00e0: f2 00 07 d0 45 c2 18 57 : 2b 29 67 3f 51 03 49 be
0x00f0: 37 c4 da 00 00 00 00 00 : 00 00 00 00 81 96 b0 b1

NPU

This section covers specific when the interface is directly connected to the NPU. So, there is no PHY. These ports are PHYless ports.

RP/0/RP0/CPU0:ios#show controllers twentyFiveGigE 0/0/0/12 (partial output)
Operational data for interface TwentyFiveGigE0/0/0/12: 

State:    Administrative state: disabled                ==> Check if admin state display is correct
    Operational state: Down (Reason: Link is shutdown)  ==> Check if link state display is correct. If interface is down with Remote fault/Link Loss (local fault), verify the peer.
MAC address information:
    Operational address: fc58.9a01.8e10
    Burnt-in address: fc58.9a01.8e10

Autonegotiation disabled.
Priority flow control information for interface TwentyFiveGigE0/0/0/12:
    Forward error correction: Standard (Reed-Solomon)  ==> Check if FEC status is correct.

RP/0/RP0/CPU0:ios#show controllers tenGigE 0/0/0/14
    Operational data for interface TenGigE0/0/0/14:

State:    Administrative state: enabled
    Operational state: Down (Reason: Link loss or low light, no loopback) ==> This router has a Local Fault/Down.
    LED state: Yellow On

RP/0/RP0/CPU0:ios#show controllers tenGigE 0/0/0/15
    Operational data for interface TenGigE0/0/0/15:
    State:
        Administrative state: enabled
        Operational state: Down (Reason: Remote Fault)   ==> The peer has a Fault
        LED state: Yellow On

Summary

• Check if the admin state of the interface is down. If yes, then bring it to admin up state with the command "no shut" on interface.
• Check the FEC status on both the router and the peer device. If there is any mismatch, try to correct them.
• Check the auto negotiation status on both the router and the peer device. If there is any mismatch, try to correct them.
• If FEC and auto negotiation are correctly configured, check the alarms status on both the router and the peer in the “show controller optics” output.

RX-LOS, RX-LOL : Problem in the receive direction of the signal.

TX-LOS, TX-LOL : Signal received by SFP/QSFP from NPU or PHY has the issue.

If there is a Link Down/Flap with RX-LOL/RX-LOS:

• Check if peer has TX-LOS/TX-LOL.
• If there is no TX alarms, try to change the cable/pluggable (SFP/QSFP).

If there is a Link Down/Flap with TX-LOL/TX-LOS:

You can apply a loopback external to exclude the remote peer and the fiber to the remote peer. Use a fiber cable locally to physically loop the transmit (Tx) signal into the receive (Rx) port. Optionally use an attenuator, to achieve this:

RP/0/RSP0/CPU0:NCS(config)#int Hu0/3/0/31
RP/0/RSP0/CPU0:NCS(config-if)#loopback ?
external Enable external loopback (requires loopback connector)
internal Enable internal loopback
line Enable line loopback
RP/0/RSP0/CPU0:NCS(config-if)#loopback external ?
<cr>
RP/0/RSP0/CPU0:NCS(config-if)#loopback external
RP/0/RSP0/CPU0:NCS(config-if)#commit

You can apply a loopback internal config on both the router and the peer to verify the optics by a bypass of the fiber/optics. This means that if the interface still does not come up, it indicates that the issue is not with the optical part!

RP/0/RP0/CPU0:BRU-SPCORE-P2(config)#int hundredGigE 0/0/0/10
RP/0/RP0/CPU0:BRU-SPCORE-P2(config-if)#loopback internal ?
  <cr>
RP/0/RP0/CPU0:BRU-SPCORE-P2(config-if)#loopback internal
RP/0/RP0/CPU0:BRU-SPCORE-P2(config-if)#commit

Collect Logs

These commands on both the router and the peer can provide a cause of the issue.

• show ip interface brief
• show alarms brief 

• show controller optics <r/s/i/p> of the interface with the issue
• show tech-support optics

This displays alarms, LED status, Laser status & other optics info

• show controller <tengige/hundredgige/other> <r/s/i/p> internal of interface with the issue.

Displays FEC, AN, Up/Down, Local fault/Remote fault, and so on.

• show controllers npu voq-usage interface all instance all location all

Gives front panel port to pp port, npu, and core mapping.

Full collection of logs for the interface issue:

• show version
• show running-config
• show install active
• show platform
• show tech-support ethernet platform location 0/x/CPU0
• show tech-support ethernet interfaces
• show tech-support ethernet controllers location 0/x/CPU0
• show tech-support dpa location 0/x/CPU0
• show tech-support ofa location 0/x/CPU0
• show tech-support optics location 0/x/CPU0
• show tech-support coherent location 0/x/CPU0
• show tech-support pfi location all
• show tech-support qos platform location 0/x/CPU0
• show tech-support fabric
• show controllers npu voq-usage interface all instance 0 location all
• show optics trace all
• show ethernet infra trace location 0/x/cpu0
• show ethernet v-ether trace location 0/x/CPU0
• show vether-ea trace all location 0/x/CPU0
• show portmapper trace all location 0/0/CPU0