Troubleshoot Xbar Sync Failures on a Nexus 7000 Series Switch

Introduction

This document describes how to identify a faulty crossbar (Xbar) when a module is down due to an Xbar sync failure on a Cisco Nexus 7000 Series switch. The troubleshooting procedure for this problem involves the collection of data, data analysis, and an elimination process in order to isolate the problem component.

Prerequisites

Requirements

Cisco recommends that you have knowledge of the Cisco Nexus Operating System (NX-OS) CLI.

Components Used

The information in this document is based on the Cisco Nexus 7000 Series switch that runs NX-OS Version 6.1(2), but it can also work with any NX-OS version.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Problem

The N7K-F248XP-25E module is down due to an Xbar sync failure upon module boot. When the module was inserted into Slot 1 on the chasis, it powered down. This can occur due to one of these reasons:

• Bad module
  
  
• Bad Xbar
  
  
• Bad or bent pin on the backplane

Solution

In the case of a suspected hardware failure on the N7K-F248XP-25E module, you must view the logs in order to determine whether the reason for the failure is due to a faulty module, or due to the Xbar sync failure.

In order to further isolate the issue in this example, the module was inserted into a different slot and became active as expected. This indicates that the module is not faulty, so the issue is either with the Xbar fabric or with the chassis.

This exception log appeared when module was powered down in Slot 1:

show module internal exceptionlog module 1
********* Exception info for module 1 ********

exception information --- exception instance 1 ----
Module Slot Number: 1
Device Id         : 88
Device Name       : XbarComplex
Device Errorcode  : 0x00000008
Device ID         : 00 (0x00)
Device Instance   : 00 (0x00)
Dev Type (HW/SW)  : 00 (0x00)
ErrNum (devInfo)  : 08 (0x08)

System Errorcode  : 0x40240012 xbar sync failed during module bringup
(DevErr is LinkNum)
Error Type        : Informational
PhyPortLayer      : Unknown
Port(s) Affected  : none
DSAP              : 0 (0x0)
UUID              : 0 (0x0)
Time              : Thu Mar 20 15:55:19 2014
(Ticks: 532B0F67 jiffies)

exception information --- exception instance 2 ----
Module Slot Number: 1
Device Id         : 88
Device Name       : XbarComplex
Device Errorcode  : 0x00000008
Device ID         : 00 (0x00)
Device Instance   : 00 (0x00)
Dev Type (HW/SW)  : 00 (0x00)
ErrNum (devInfo)  : 08 (0x08)

System Errorcode  : 0x40240012 xbar sync failed during module bringup
(DevErr is LinkNum)
Error Type        : Informational
PhyPortLayer      : Unknown
Port(s) Affected  : none
DSAP              : 0 (0x0)
UUID              : 0 (0x0)
Time              : Thu Mar 20 15:53:12 2014
(Ticks: 532B0EE8 jiffies)

As per these exception logs, the issue is clearly with the Xbar or with the chassis in Slot 1. 

In order to isolate the issue further, you must remove each Xbar individually while you monitor the module in Slot 1 until it is able to power up without issues. This confirms that there is an issue with a particular Xbar fabric module, in which case you would proceed with a Return Material Authorization (RMA) for the faulty hardware.

However, this is a long procedure and it requires a long maintenance window. In order to find the exact Xbar fabric slot that causes the sync issue with the module, you can proceed as shown here:

show system internal xbar event-history errors
-----------------------------------------------------

7) Event:E_DEBUG, length:67, at 384460 usecs after Thu Mar 20 15:55:19 2014
   [102] xbm_perform_error_action(1413): MTS_OPC_LC_INSERTED error 0x1

8) Event:E_DEBUG, length:104, at 384347 usecs after Thu Mar 20 15:55:19 2014
   [102] send_exception_log_msg_to_lcm(1101): module 1 DevId 88 dev_err 0x8 sys
   _err 0x40240012 err_type 0x4

9) Event:E_DEBUG, length:59, at 384343 usecs after Thu Mar 20 15:55:19 2014
   [102] xbm_mod_ac_error(221): Sync fail for module 1 link 8

10) Event:E_DEBUG, length:66, at 384341 usecs after Thu Mar 20 15:55:19 2014
    [102] xbm_mod_ac_error(210): Error for Slot 0 error_code 0x877660c

11) Event:E_DEBUG, length:62, at 384298 usecs after Thu Mar 20 15:55:19 2014
    [102] xbm_sync_seq_failed(1169): Sync fail for module 1 link 8

In these logs, you can see the Sync fail for module 1 link 8 message. You must then identify the fabric slot with which Link 8 is associated. In order to determine this, you must check the output of the show system internal xbar sw command:

show system internal xbar sw

Module in slot 1 (present = 0)
 
Dedicated X-link 255
rid 0x2000000 type 0 state 0 sub_type 0 node_id 0x0
sw_card_id 0x0 lc_node_addr 0x0 feature_bits 0x0
timer: hdl 0x86fcc20 rid 0x2000000 ev_id 0xffff timer_id 0x41a tim_type 0x2
Link_Info:: Num Links 10 max Edp 10
    Link_num 0
        is_synced 0 is_edp 0 num_sync_try 0
    Link_num 1
        is_synced 0 is_edp 0 num_sync_try 0
    Link_num 2
        is_synced 0 is_edp 0 num_sync_try 0
    Link_num 3
        is_synced 0 is_edp 0 num_sync_try 0
    Link_num 4
        is_synced 0 is_edp 0 num_sync_try 0
    Link_num 5
        is_synced 0 is_edp 0 num_sync_try 0
    Link_num 6
        is_synced 0 is_edp 0 num_sync_try 0
    Link_num 7
        is_synced 0 is_edp 0 num_sync_try 0
    Link_num 8
        is_synced 0 is_edp 0 num_sync_try 3
    Link_num 9
        is_synced 0 is_edp 0 num_sync_try 0
Link_Map:: Num Links 10 max Edp 10
    Link_num 0
        connected to fab [10.0] active_lnk 1
        fi_to_mon 0 fi_to_use 0
    Link_num 1
        connected to fab [10.0] active_lnk 1
        fi_to_mon 0 fi_to_use 0
    Link_num 2
        connected to fab [11.0] active_lnk 1
        fi_to_mon 1 fi_to_use 1
    Link_num 3
        connected to fab [11.0] active_lnk 1
        fi_to_mon 1 fi_to_use 1
    Link_num 4
        connected to fab [12.0] active_lnk 1
        fi_to_mon 2 fi_to_use 2
    Link_num 5
        connected to fab [12.0] active_lnk 1
        fi_to_mon 2 fi_to_use 2
    Link_num 6
        connected to fab [13.0] active_lnk 1
        fi_to_mon 3 fi_to_use 3
    Link_num 7
        connected to fab [13.0] active_lnk 1
        fi_to_mon 3 fi_to_use 3
    Link_num 8
        connected to fab [14.0] active_lnk 1
        fi_to_mon 4 fi_to_use 4
    Link_num 9
        connected to fab [14.0] active_lnk 1
        fi_to_mon 4 fi_to_use 4

In the output, you can see that Link_num 8 (Link 8) is connected to fab [14.0] (Fabric Slot 14), which is Xbar 5.

Note: A zero-based numbering system is used for the Xbar.

In order to identify fab [14.0] (the fabric in Slot 5) enter the show module command:

show module

Xbar Ports  Module-Type                      Model             Status
---  -----  -------------------------------- ----------------- ------
4    0      Fabric Module 2                  N7K-C7010-FAB-2    ok

Xbar MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
1    NA                                      JAF1739AQTP
2    NA                                      JAF1739AJAA
3    NA                                      JAF1739AQDG
4    NA                                      JAF1739ATHG
5    NA                                      JAF1739AQEF

In the output of the show module command, you can view the Xbar fabric module in Slot 5. 

You should now have the correct identification of the fabric that caused the sync failure to the module in Slot 1. In this example, the fabric was removed from Slot 5 and the module that was in Slot 1 booted up without any error. The faulty Xbar can now be replaced.

Related Information

• Nexus 7000 Fabric CRC Errors Troubleshoot
• Technical Support & Documentation - Cisco Systems