對間歇性或無響應的N7K控制平面/硬體故障進行故障排除

簡介

本文檔介紹如何在Nexus 7k中的硬體模組無響應或間歇性時對其進行故障排除。

對間歇性或無響應的N7K控制平面進行故障排除

1.間歇性問題是否在某個週期出現？

步驟 1.對各種SNMP V3使用者和/或SNMP V2社群字串執行snmpwalk（即遍歷主機名mib）。

在連續循環中執行此操作。

• 如果Snmpwalk持續響應而沒有出現問題，則很可能不是您的問題。
• 如果snmpwalk持續響應30秒鐘，然後突然停頓了一下，並且變得間歇性再停上30秒，然後重複這個循環，這就是一個嚴重的指示

步驟2.使用ssh連線到有問題的VDC，該VDC在步驟1中對主機名具有間歇性無響應掃描。

• 在snmpwalk不響應主機名的同時，在ssh視窗中發出show running-configuration。
• 如果顯示running-configuration is paused — 這是一個嚴重指示

由於步驟1和步驟2在60秒週期內同時受到影響，這似乎是一個N7K控制平面內的硬體故障，因為N7K始終運行硬體診斷運行狀況檢查。當您看到響應時間為30秒，無響應時間為30秒，然後循環重複時，這清楚地表明硬體診斷運行狀況檢查掃描所有硬體。30秒的響應時間是掃描正常硬體，而30秒的不響應時間是故障硬體。

步驟 3.如果步驟2.清楚地描述了硬體故障，請執行以下步驟：

• 顯示所有活動VDC中的模組以檢視任何失敗的診斷測試
• 最重要的是，並且僅在管理VDC-1（即VDC-1）中，顯示日誌記錄日誌檔案並查詢EOBC錯誤（即，EOBC錯誤僅在管理VDC-1中記錄）

註:EOBC是N7K用於在SUP/交換矩陣模組/線卡之間通訊的內部控制平面進程。如果此EOBC流程受到任何影響，管理VDC-1日誌檔案中描述的關聯模組最可能是先前測試中看到的間歇響應的罪魁禍首，因為SUP丟失了與管理VDC-1日誌檔案中描述的關聯模組的100%一致通訊，並且正在嘗試與其恢復/通訊，從而導致與其它控制平面流程的間歇響應。

範例：

lab-sw01-admin-vdc-1# show logging logfile | inc EOBC
2022 Feb 22 19:46:15 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:15 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:16 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:16 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:21 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:21 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:22 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:23 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:23 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:24 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:24 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)
2022 Feb 22 19:46:26 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure on standby sup in device DEV_EOBC_MAC (device error 0xc0a0504f)
2022 Feb 22 19:46:26 lab-sw01-admin-vdc-1 %MODULE-4-MOD_WARNING: Module 8 (Serial number: JAA00000000) reported warning 8/1-8/0 due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a0514d)

此日誌輸出清楚地顯示，模組8具有待命SUP的EOBC心跳故障，並且處於不正常狀態，需要立即採取措施。

2.如何解決此問題？

步驟 1.執行show模組並捕獲資料以供參考：

lab-sw01-admin-vdc-1# show module 
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 N77-SUP2E active *
6 0 Supervisor Module-2 N77-SUP2E ha-standby
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok

Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5 
2 8.4(4) 1.5 
3 8.4(4) 1.9 
4 8.4(4) 1.9 
5 8.4(4) 1.3 
6 8.4(4) 1.3 
7 8.4(4) 1.2 
8 8.4(4) 1.2

註意：所有模組均線上（即ok），模組5是主用（即active *）SUP，模組6是高可用性備用（即ha-standby）SUP。 雖然管理VDC日誌檔案中存在有關模組8的EOBC警告，但此輸出將模組8描述為「正常」。

步驟 2.重新載入交換機或執行Supervisor切換（即在Admin VDC內）：

lab-sw01-admin-vdc-1# reload

- system (ie supervisor) switchover - 

NOTE: preferred method as this is a non-impacting procedure to the box with regards to active data flows

lab-sw01-admin-vdc-1# system switchover

注意：在任何一種情況下，執行重新載入或系統切換之前，請確保您同時處於兩個Supervisor控制檯上，以便您可以親眼看到所有Supervisor輸出。

步驟 3.如果模組8是可疑的罪魁禍首，您可能會在系統（即Supervisor）切換時看到控制檯模組8出現故障：

lab-sw01-admin-vdc-1(standby) login: 2022 Feb 23 02:09:45 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %KERN-2-SYSTEM_MSG: [12392164.927835] Switchover started by redundancy driver - kernel
2022 Feb 23 02:09:45 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %SYSMGR-2-HASWITCHOVER_PRE_START: This supervisor is becoming active (pre-start phase).
2022 Feb 23 02:09:45 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %SYSMGR-2-HASWITCHOVER_START: Supervisor 6 is becoming active.
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth0/8 return status No card found in slot 
2022 Feb 23 02:09:46 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %SYSMGR-2-SWITCHOVER_OVER: Switchover completed.
2022 Feb 23 02:09:47 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %PLATFORM-1-PFM_ALERT: Disabling ejector based shutdown on sup in slot 6 
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth1/8 return status No card found in slot 
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth2/8 return status No card found in slot 
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth3/8 return status No card found in slot 
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth4/8 return status No card found in slot 
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth5/8 return status No card found in slot 
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth6/8 return status No card found in slot 
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth7/8 return status No card found in slot 
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth8/8 return status No card found in slot 
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth9/8 return status No card found in slot 
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth10/8 return status No card found in slot 
2022 Feb 23 02:09:46 lab-sw01-vdc-2 %$ VDC-2 %$ %ELTM-2-ELTM_INTF_TO_LTL: Failed to get LTL for interface lc-eth11/8 return status No card found in slot

步驟 4.執行多個show模組並觀察模組8是否重新聯機/何時重新聯機：

Module 5 dropped out and is powered-up:
Module 8 dropped out and is powered-up:

lab-sw01-admin-vdc-1# show module 
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module powered-up

Mod Power-Status Reason 
--- ------------ ---------------------------
8 powered-up Unknown. Issue show system reset mod ...

Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5 
2 8.4(4) 1.5 
3 8.4(4) 1.9 
4 8.4(4) 1.9 
6 8.4(4) 1.3 
7 8.4(4) 1.2

lab-sw01-admin-vdc-1# 2022 Feb 23 02:11:11 lab-sw01-vdc-2 %$ VDC-2 %$ %PLATFORM-2-MOD_DETECT: Module 8 detected (Serial number JAA00000000) Module-Type 10/40 Gbps Ethernet Module Model N77-F324FQ-25
2022 Feb 23 02:11:11 lab-sw01-vdc-2 %$ VDC-2 %$ %PLATFORM-2-MOD_PWRUP: Module 8 powered up (Serial number JAA00000000)
2022 Feb 23 02:11:11 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %PLATFORM-2-MOD_DETECT: Module 8 detected (Serial number JAA00000000) Module-Type 10/40 Gbps Ethernet Module Model N77-F324FQ-25
2022 Feb 23 02:11:11 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %PLATFORM-2-MOD_PWRUP: Module 8 powered up (Serial number JAA00000000)

Module 8 is pwr-cycled:

lab-sw01-admin-vdc-1# show module 
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module pwr-cycld

Mod Power-Status Reason 
--- ------------ ---------------------------
8 pwr-cycld Unknown. Issue show system reset mod ...

Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5 
2 8.4(4) 1.5 
3 8.4(4) 1.9 
4 8.4(4) 1.9 
6 8.4(4) 1.3 
7 8.4(4) 1.2

lab-sw01-admin-vdc-1# show module 
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 powered-up

Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5 
2 8.4(4) 1.5 
3 8.4(4) 1.9 
4 8.4(4) 1.9 
6 8.4(4) 1.3 
7 8.4(4) 1.2 
8 8.4(4) 1.2

Module 8 is checked by epld auto-upgrade and is good to go:

lab-sw01-admin-vdc-1# 2022 Feb 23 02:13:06 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: <<%EPLD_AUTO-2-AUTO_UPGRADE_CHECK>> Automatic EPLD upgrade check for module 8: EPLD versions are up to date. - epld_auto

lab-sw01-admin-vdc-1# show module 
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 powered-up

Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5 
2 8.4(4) 1.5 
3 8.4(4) 1.9 
4 8.4(4) 1.9 
6 8.4(4) 1.3 
7 8.4(4) 1.2 
8 8.4(4) 1.2

Module 8 moves to testing by the hardware diagnostics:

lab-sw01-admin-vdc-1# show module 
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 testing

Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5 
2 8.4(4) 1.5 
3 8.4(4) 1.9 
4 8.4(4) 1.9 
6 8.4(4) 1.3 
7 8.4(4) 1.2 
8 8.4(4) 1.2

Module 8 moves to initializing after passing hardware diagnostics:

lab-sw01-admin-vdc-1# show module 
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 initializing

Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5 
2 8.4(4) 1.5 
3 8.4(4) 1.9 
4 8.4(4) 1.9 
6 8.4(4) 1.3 
7 8.4(4) 1.2 
8 8.4(4) 1.2

Module 8 comes online:

lab-sw01-admin-vdc-1# show module 
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 powered-up
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok

Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5 
2 8.4(4) 1.5 
3 8.4(4) 1.9 
4 8.4(4) 1.9 
6 8.4(4) 1.3 
7 8.4(4) 1.2 
8 8.4(4) 1.2

Module 5 SUP going active:

lab-sw01-admin-vdc-1# show module 
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 N77-SUP2E inserted
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok

Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5 
2 8.4(4) 1.5 
3 8.4(4) 1.9 
4 8.4(4) 1.9 
5 8.4(4) 1.3 
6 8.4(4) 1.3 
7 8.4(4) 1.2 
8 8.4(4) 1.2

Module 5 SUP becomes ha-standby:

2022 Feb 23 02:16:38 lab-sw01-admin-vdc-1 %$ VDC-1 %$ %PLATFORM-1-PFM_ALERT: Enabling ejector based shutdown on sup in slot 6 
lab-sw01-admin-vdc-1# show module 
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ------------------ ----------
1 12 100 Gbps Ethernet Module N77-F312CK-26 ok
2 12 100 Gbps Ethernet Module N77-F312CK-26 ok
3 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
4 48 1/10 Gbps Ethernet Module N77-F348XP-23 ok
5 0 Supervisor Module-2 N77-SUP2E ha-standby
6 0 Supervisor Module-2 N77-SUP2E active *
7 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok
8 24 10/40 Gbps Ethernet Module N77-F324FQ-25 ok

Mod Sw Hw
--- --------------- ------
1 8.4(4) 1.5 
2 8.4(4) 1.5 
3 8.4(4) 1.9 
4 8.4(4) 1.9 
5 8.4(4) 1.3 
6 8.4(4) 1.3 
7 8.4(4) 1.2 
8 8.4(4) 1.2

2022 Feb 23 02:15:44 lab-sw01-admin-vdc-1 %MODULE-5-MOD_OK: Module 8 is online (Serial number: JAA00000000)
2022 Feb 23 02:15:43 lab-sw01-admin-vdc-1 %SYSMGR-SLOT8-5-MODULE_ONLINE: System Manager has received notification of local module becoming online.
2022 Feb 23 02:15:44 lab-sw01-admin-vdc-1 %PLATFORM-5-MOD_STATUS: Module 8 current-status is MOD_STATUS_ONLINE/OK
2022 Feb 23 02:16:38 lab-sw01-admin-vdc-1 %MODULE-5-STANDBY_SUP_OK: Supervisor 5 is standby

註：所有模組均聯機（正常），模組6為活動（活動*）SUP，模組5為高可用性備用（即ha-standby）SUP。

步驟 5.所有模組均聯機後，重複步驟1。並驗證所有連線都已標準化。