排除Tomahawk和Lightspeed卡上的Punt交換矩陣資料路徑故障
目錄
簡介
本檔案介紹在思科聚合服務路由器(ASR)9000系列操作期間出現的突發交換矩陣資料路徑故障消息。
背景資訊
消息顯示格式如下：
- 如圖所示，路由器控制檯上會顯示警報。
- 這表示這些訊息的回送路徑在某處被破壞。
RP/0/RP0/CPU0:Oct 28 12:46:58.459 IST: pfm_node_rp[349]: %PLATFORM-DIAGS-3-PUNT_FABRIC_DATA_PATH_FAILED :
Set|online_diag_rsp[24790]|System Punt/Fabric/data Path Test(0x2000004)|failure threshold is 3,
(slot, NP) failed: (0/9/CPU0, 1) (0/9/CPU0, 3)
前面提到的0/9/CPU0上的NP1和NP3出現問題。
本文檔適用於希望瞭解錯誤消息以及發現問題時必須採取的操作的任何人。
基於Tomahawk的線路卡(LC)可用作服務邊緣最佳化（增強QoS）或資料包傳輸最佳化（基本QoS）的LC。
- SE — 服務邊緣最佳化
- TR — 最佳化了資料包傳輸
4埠和8埠100千兆乙太網LC有兩種版本，支援LAN/WAN/OTN統一PHY CPAK埠或僅LAN PHY CPAK埠。
這些LC基於Tomahawk:
- A9K-8X100G-LB-SE
- A9K-8X100G-LB-TR
- A9K-8X100GE-SE
- A9K-8X100GE-TR
- A9K-4X100GE-SE
- A9K-4X100GE-TR
- A9K-400G-DWDM-TR
- A9K-MOD400-SE
- A9K-MOD400-TR
- A9K-MOD200-SE
- A9K-MOD200-TR
- A9K-24X10GE-1G-SE
- A9K-24X10GE-1G-TR
- A9K-48X10GE-1G-SE
- A9K-48X10GE-1G-TR
- A99-12X100GE
- A99-8X100GE-SE
- A99-8X100GE-TR
註：以A99-X開頭的基於Tomahawk的LC部件號與Cisco ASR 9904、ASR 9906、ASR 9910、ASR 9912和ASR 9922機箱相容。它們與Cisco ASR 9006和ASR 9010路由器不相容。
基於光速的LC可以用作服務邊緣最佳化（增強型QoS）或封包傳輸最佳化（基本QoS）LC。與基於Tomahawk的LC不同，並非每個LC型號都同時提供 — SE和 — TR型別。
- SE — 服務邊緣最佳化
- TR — 最佳化了資料包傳輸
這些LC基於Lightspeed:
- A9K-16X100GE-TR
- A99-16X100GE-X-SE
- A99-32X100GE-TR
基於Lightspeed-Plus(LSP)的LC可用作服務邊緣最佳化（增強QoS）或資料包傳輸最佳化（基本QoS）LC。
這些LC基於LSP:
- A9K-4HG-FLEX-TR
- A9K-4HG-FLEX-SE
- A99-4HG-FLEX-TR
- A99-4HG-FLEX-SE
- A9K-8HG-FLEX-TR
- A9K-8HG-FLEX-SE
- A9K-20HG-FLEX-TR
- A9K-20HG-FLEX-SE
- A99-32X100GE-X-TR
- A99-32X100GE-X-SE
- A99-10X400GE-X-TR
- A99-10X400GE-X-SE
Punt交換矩陣診斷資料包路徑
- 在路由處理器卡CPU上運行的診斷應用程式會定期向每個網路處理器(NP)注入診斷資料包。
- 診斷資料包在NP內環回，然後被重新注入源於該資料包的路由處理器卡CPU。
- 通過路由處理器卡上的診斷應用程式對每個NP具有唯一資料包的每個NP進行定期運行狀況檢查，可針對路由器操作期間資料路徑上的任何功能錯誤發出警報。
- 必須注意的是，活動路由處理器和備用路由處理器上的診斷應用程式會定期為每個NP注入一個資料包，並維護每個NP的成功或失敗計數。
- 每分鐘將診斷資料包傳送到NP(傳送到每個虛擬隊列介面(VQI)四次（總計四分鐘/VQI），並在該NP的所有VQI上運行)。為了對此進行簡要介紹，以下提供範例：
假設LC有四個NP，線上診斷必須執行所有NP（以知道它們運行正常 — 交換矩陣路徑）。現在，每個NP可以各有20個VQI(0-19、20-39、40-59、60-79)。
在第一分鐘內，聯機診斷程式向每個NP傳送一個資料包。
1 min : against VQI 0, 20, 40, 60 (to all 4 NPs)
2 min:""""""""
3 min: """""""
4 min : """""
5th min : against VQI 1, 21, 41, 61..
6 min : """"""""
所有VQI完成後，循環中重複此過程。
- 當達到丟棄的診斷資料包的閾值時，應用程式會在平台故障管理器(PFM)中引發警報。
RP/0/RP1/CPU0:AG2-2#show pfm location 0/RP1/CPU0
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Apr 7 01:04:04 2022PFM TOTAL: 1 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 1
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Apr 7 00:54:52 2022|0 |PUNT_FABRIC_DATA_PATH_FAILED |ER |10042 >>ID |System Punt/Fa|0x2000004
為了收集有關PFM警報的所有資訊，請捕獲以下命令輸出：
show pfm location all
show pfm trace location all
如果要檢視有關特定進程觸發警報的詳細資訊，可以使用以下命令：
show pfm process name <process_name> location <location> >>> location where the PFM alarm is observed
高級LC架構
戰斧LC
8x100G架構
戰斧 — 8x100G LC
12 x 100G架構
戰斧12*100G LC
光速LC
A9K-20HG-FLEX-SE/TR
A9K-20HG-FLEX-SE/TR
A99-32x100GE-X-SE/TR
A99-32x100GE-X-SE/TR
A9K-8HG-FLEX-SE/TR
LC架構
LC架構
虛擬輸出隊列和仲裁器
每個路由交換處理器/交換處理器(RSP/RP)有兩個矩陣晶片，均由一個公共仲裁器控制（雙RSP/RP表示每個機箱的彈性仲裁器）。只有活動RSP/RP上的仲裁程式控制所有四個交換矩陣晶片（假設為雙RSP）。但是，兩個仲裁者都接收交換矩陣訪問請求，以便瞭解整個系統在任何給定時間的狀態，以便RSP/RP之間的故障轉移可以即時進行。仲裁器之間沒有keepalive，但RSP/RP具有複雜可程式設計邏輯器件(CPLD)ASIC（類似於FPGA），其功能之一是通過低級keepalive跟蹤另一個RSP/RP狀態，並確定活動仲裁器是什麼。
每個交換矩陣互聯ASIC都有一組VQI，這是一組隊列，代表系統（對於Tomahawk）中的100G實體。每個100G實體（單個出口NP上的1x100G埠用入口NP中的單個100G VQI表示）具有多個優先順序類。
每個VQI都有一組4個虛擬輸出隊列(VOQ)，用於不同的資料包優先順序，其中3個用於ASR 9000轉發架構。這些對應優先順序級別1和2，並預設為輸入QoS策略。有兩個嚴格的優先順序隊列和一個正常隊列（第四個隊列用於組播，不用於單播轉發）。
通常，預設隊列在來自出口NP VQI的背壓期間首先開始丟棄資料包。只有當出口網路處理單元(NPU)超載（提供比電路可處理更多的Bps或PPS）時，它才開始向入口LC/NP施加背壓。這表示為輸入LC上的光纖介面ASIC(FIA)上的VQI流量停滯。
範例：
RP/0/RP0/CPU0:AG3_1#show controllers np ports all location 0/0/CPU0 >>> LC0 is installed in slot 2
Node: 0/0/CPU0:
----------------------------------------------------------------
NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 -- 0 TenGigE0/0/0/0/0 - TenGigE0/0/0/0/9, TenGigE0/0/0/1/0 - TenGigE0/0/0/1/9
1 -- 1 TenGigE0/0/0/2/0 - TenGigE0/0/0/2/9, HundredGigE0/0/0/3
2 -- 2 HundredGigE0/0/0/4 - HundredGigE0/0/0/5 >>>Below is the VQI assignment
3 -- 3 HundredGigE0/0/0/6 - HundredGigE0/0/0/7
RP/0/RP0/CPU0:AG3_1#sh controller fabric vqi assignment slot 2
slot = 2
fia_inst = 2 >>>FIA 2
VQI = 40 SPEED_100G
VQI = 41 SPEED_100G
VQI = 42 SPEED_100G
VQI = 43 SPEED_100G
VQI = 44 SPEED_100G
VQI = 45 SPEED_100G
VQI = 46 SPEED_100G
VQI = 47 SPEED_100G
VQI = 56 SPEED_100G
VQI = 57 SPEED_100G
VQI = 58 SPEED_100G
VQI = 59 SPEED_100G
VQI = 60 SPEED_100G
VQI = 61 SPEED_100G
VQI = 62 SPEED_100G
VQI = 63 SPEED_100G
當輸入LC決定要將特定封包傳送到特定輸出NPU時，輸入LC上的修改(MDF)階段會封裝含有光纖目的地標頭的封包。FIA在檢視該「地址」時，會檢查VOQ中是否有特定的輸出NPU/目標/LC，並檢視是否有足夠的可用頻寬。當輸入FIA準備將其出隊到該LC時，會要求該目的地LC從交換矩陣（仲裁器）獲得授權。仲裁演算法具有QOS意識，可以保證P1類資料包優先於P2類資料包等。仲裁器將許可請求從入口FIA中繼到出口FIA。
輸入FIA可以將到達同一輸出LC的多個資料包組合到稱為超幀中。這表示經過交換器網狀架構連結的並不是本機訊框/封包，而是超訊框。必須注意這一點很重要，因為在對常數100pps進行測試時，CLI可以顯示僅報告50pps的交換矩陣計數器。這不是封包遺失，而只是表示每個超訊框中有兩個封包在交換器網狀架構上傳輸。超幀包括排序資訊和目標FIA支援重新排序（資料包可以在多個交換矩陣鏈路上「噴射」）。只有單播資料包會被放入超幀，而不是組播資料包。
輸出LC收到封包後，便會向仲裁伺服器傳回授權。仲裁器每個VOQ的令牌數量有限。當仲裁器允許輸入FIA向特定VOQ傳送（超級）幀時，僅當輸出FIA向輸出NP傳送幀時，該令牌才返回到池。如果輸出NP已向輸出FIA發出反壓訊號，則令牌仍然佔用。這就是仲裁器最終在入口FIA中耗盡該VOQ的令牌的方式。發生這種情況時，輸入FIA開始丟棄傳入的資料包。背壓的觸發因素是輸出NP中接收幀描述符(RFD)緩衝區的利用率級別。當NP微碼處理資料包時，RFD緩衝區會保留這些資料包。處理封包的功能越多，它停留在RFD緩衝區中的時間就越長。
- 輸入FIA向所有機箱仲裁者發出交換矩陣請求。
- 主動仲裁器會檢查自由訪問授權令牌，並在存在擁塞時處理其QoS演算法。
- RSP上從本地仲裁者到主動仲裁者的信用機制。
- 主動仲裁器向輸入FIA傳送交換矩陣授權令牌。
- 輸入FIA在交換矩陣鏈路上平衡負載（超級）幀。
- 輸出FIA將光纖權杖傳回中央仲裁器。
最好提及RSP上從本地仲裁者到活動仲裁者的信用機制。另外新增一節來說明仲裁器故障的可能情況（不需要提及錯誤代碼，但需要檢視仲裁器ASIC錯誤），以便在仲裁器出現任何問題並且由於本地或中央仲裁器導致隊列堆積而無法獲得授權時進行檢視。
虛擬輸出隊列概述
虛擬輸出佇列
發往不同出口NP的資料包會被放入不同的VOQ集中。一個NP上的擁塞不會阻止發往不同NP的資料包。
交換矩陣仲裁器圖表
光纖仲裁器
光纖互連
ASR9006和ASR9010交換矩陣互聯
ASR9006和ASR9010交換矩陣互聯
ASR9922交換光纖互連
ASR9912同樣支援10個LC和單個交換矩陣互聯晶片。
ASR9922交換光纖互連
ASR9922和ASR9912背板
ASR9922和ASR9912背板
線上診斷概述
- 聯機診斷工具在LC和RP CPU上運行。
- 測試轉發路徑的診斷測試包括：
- PuntFabricDataPath測試在活動和備用RP CPU上運行，將診斷資料包傳送到系統中的每個活動NP。活動RP傳送。
- PuntFabricDataPath診斷資料包以單播方式傳送，而備用資料包以組播方式傳送。將響應資料包傳送回始發RP CPU。
- 測試轉發路徑的診斷測試包括：
- LC內的NP環回測試。
- NPLoopback測試在每個LC CPU上運行，將診斷資料包傳送到每個NP。將響應資料包傳送回LC CPU。
對問題進行分類
此處的步驟提供了一些有關如何縮小與點路徑故障相關問題的提示。 它們不需要以完全相同的順序被遵循。
啟動分類所需的資訊
- 找到受影響的NP和LC:
show logging | inc “PUNT_FABRIC_DATA_PATH”
RP/0/RP1/CPU0:Oct 28 12:46:58.459 IST: pfm_node_rp[349]: %PLATFORM-DIAGS-3-PUNT_FABRIC_DATA_PATH_FAILED :
Set|online_diag_rsp[24790]|System Punt/Fabric/data Path Test(0x2000004)|failure threshold is 3, (slot, NP)
failed: (0/9/CPU0, 1) (0/9/CPU0, 3)
前面提到的0/9/CPU0上的NP1和NP3出現問題。
- 要查詢機箱插槽，請輸入
run nslot all指令。 - PFM警報
RP/0/RP1/CPU0:AG2-2#show pfm location 0/RP1/CPU0
node: node0_RP1_CPU0
---------------------
CURRENT TIME: Mar 25 12:11:29 2022
PFM TOTAL: 1 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 1
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Mar 25 12:03:30 2022|1 |PUNT_FABRIC_DATA_PATH_FAILED |ER |8947 |System Punt/Fa|0x2000004
RP/0/RP1/CPU0:AG2-2#sh pfm process 8947 location 0/rp1/CPU0
node: node0_RP1_CPU0
---------------------
CURRENT TIME: Mar 25 12:12:36 2022
PFM TOTAL: 1 EMERGENCY/ALERT(E/A): 0 CRITICAL(CR): 0 ERROR(ER): 1
PER PROCESS TOTAL: 0 EM: 0 CR: 0 ER: 0
Device/Path[1 ]:Fabric loopbac [0x2000003 ] State:RDY Tot: 0
Device/Path[2 ]:System Punt/Fa [0x2000004 ] State:RDY Tot: 1
1 Fault Id: 432
Sev: ER
Fault Name: PUNT_FABRIC_DATA_PATH_FAILED
Raised Timestamp: Mar 25 12:03:30 2022
Clear Timestamp: Mar 25 12:07:32 2022
Changed Timestamp: Mar 25 12:07:32 2022
Resync Mismatch: FALSE
MSG: failure threshold is 3, (slot, NP) failed: (0/9/CPU0, 1) (0/9/CPU0, 3)
診斷資料包流程圖
- DIAG消息資料包在RP和LC之間的路徑（診斷資料包間隔為一分鐘）。
RP上的資料包路徑：
online_diags <===> SPP <===> Fabric <===> NP
LC上的封包路徑：
online_diags <===> SPP <===> Punt-switch <====> NP
- LC中的NP環回測試
每分鐘都會將每個NP的DIAGS資料包從LC CPU注入到Punt交換機，並在NP處回送。它們根本不會進入交換矩陣。轉彎點或中間點是每個NP的微代碼。
- 診斷傳送路徑：LC：聯機診斷>注入> LC-NP >（循環）
- 診斷返迴路徑：LC-NP > Punt > online diagnostics: LC
診斷測試
RP/0/RP0/CPU0:AG2-2(admin)#show diagnostic content location <> >>> (in cXR)
RP/0/RP0/CPU0:AG2-2#show diagnostic content location <> >>> (in eXR)
A9K-8X100GE-L-SE 0/0/CPU0:
Diagnostics test suite attributes:
M/C/* - Minimal bootup level test / Complete bootup level test / NA
B/O/* - Basic ondemand test / not Ondemand test / NA
P/V/* - Per port test / Per device test / NA
D/N/* - Disruptive test / Non-disruptive test / NA
S/* - Only applicable to standby unit / NA
X/* - Not a health monitoring test / NA
F/* - Fixed monitoring interval test / NA
E/* - Always enabled monitoring test / NA
A/I - Monitoring is active / Monitoring is inactive
n/a - Not applicable
Test Interval Thre- Timeout
ID Test Name Attributes (day hh:mm:ss.ms shold ms )
==== ================================== ============ ================= ===== =====
1) CPUCtrlScratchRegister ----------> *B*N****A 000 00:01:00.000 3 n/a
2) DBCtrlScratchRegister -----------> *B*N****A 000 00:01:00.000 3 n/a
3) PortCtrlScratchRegister ---------> *B*N****A 000 00:01:00.000 3 n/a
4) PHYScratchRegister --------------> *B*N****A 000 00:01:00.000 3 n/a
5) NPULoopback ---------------------> *B*N****A 000 00:01:00.000 3 n/a
RP/0/RP0/CPU0:AG2-2#show diagnostic result location 0/0/CPU0
A9K-8X100GE-L-SE 0/0/CPU0:
Overall diagnostic result: PASS
Diagnostic level at card bootup: bypass
Test results: (. = Pass, F = Fail, U = Untested)
1 ) CPUCtrlScratchRegister ----------> .
2 ) DBCtrlScratchRegister -----------> .
3 ) PortCtrlScratchRegister ---------> .
4 ) PHYScratchRegister --------------> .
5 ) NPULoopback ---------------------> .
- 您可以手動測試此引數「injected diags packets」（注入診斷資料包），如以下範例所述：
admin diag start location 0/x/cpu0 test NPULoopback (cXR)
RP/0/RP0/CPU0:AG3_1#diagnostic start location 0/0/CPU0 test NPULoopback >>> eXR
Fri May 13 06:53:00.902 EDT
RP/0/RP0/CPU0:AG3_1#show diagnostic res location 0/0/CPU0 test 5 detail >>> Here there are
multiple test 1-5 (check previous examples)
Test results: (. = Pass, F = Fail, U = Untested)
___________________________________________________________________________
5 ) NPULoopback ---------------------> .
Error code ------------------> 0 (DIAG_SUCCESS)
Total run count -------------> 67319
Last test execution time ----> Fri May 13 06:53:01 2022
First test failure time -----> n/a
Last test failure time ------> n/a
Last test pass time ---------> Fri May 13 06:53:01 2022
Total failure count ---------> 0
Consecutive failure count ---> 0
___________________________________________________________________________
- 檢查NP是否正在接收/傳送DIAG消息：
RP/0/RSP1/CPU0:AG2-2#show controllers np counters location | inc DIAG| LC_CPU
108 PARSE_RSP_INJ_DIAGS_CNT 25195 0 >>> total DIAG packets injected by Active+Stdby RP
904 PUNT_DIAGS_RSP_ACT 12584 0 >>> Loopbacks to Active RP
906 PUNT_DIAGS_RSP_STBY 12611 0 >>> Loopbacks to Stdby R
122 PARSE_LC_INJ_DIAGS_CNT 2618 0 >>> total DIAG packets injected by LC
790 DIAGS 12618 0 >>> total DIAG packets replied back to LC
16 MDF_TX_LC_CPU 3998218312 937 >>> a packet punted to LC CPU PARSE_RSP_INJ_DIAGS_CNT should match (PUNT_DIAGS_RSP_ACT + PUNT_DIAGS_RSP_STDBY)
PARSE_LC_INJ_DIAGS_CNT should match DIAGS
PARSE_XX_INJ_DIAGS_CNT should increment periodically.
- 檢查軟體封包路徑(SPP)是否正在傳送/接收DIAG訊息：
show spp sid stats location | inc DIAG 2. DIAG 35430
2. DIAG 35430
這些是DIAG計數器的接收和傳送結果。 它們可以始終在LC上匹配和遞增。
- debug punt-inject l2-packets diag np 0 location 0/9/CPU0
示例日誌：SPP正在傳送和接收序列不為0x4e資料包的診斷資料包。
LC/0/1/CPU0:Jun 6 04:14:05.581 : spp[89]: Sent DIAG packet. NP:0 Slot:0 Seq:0x4e
LC/0/1/CPU0:Jun 6 04:14:05.584 : spp[89]: Rcvd DIAG packet. NP:0 Slot:0 Seq:0x4e
- 檢查封包路徑中是否有任何捨棄專案：
show drops all location
show drops all ongoing location
- 檢查聯機診斷調試（在cXR中）：
線上診斷多次有助於檢查傳送/接收或丟失資料包時的時間戳。 此類時間戳可以與SPP捕獲進行比較，以便進行資料包關聯。
admin debug diagnostic engineer location
admin debug diagnostic error location
註：輸入 admin undebug all 命令，以停用這些調試。
debug的輸出示例：
RP/0/RSP0/CPU0:Mar 25 05:43:43.384 EST: online_diag_rsp[349]: Slot 1 has 4 NPs >>> Sending DIAG
messages to NPs on slot 1
RP/0/RSP0/CPU0:Mar 25 05:43:43.384 EST: online_diag_rsp[349]: PuntFabricDataPath: sending
a pak (seq 25), destination physical slot 1 (card type 0x3d02aa), NP 0, sfp=0xc6
RP/0/RSP0/CPU0:Mar 25 05:43:43.384 EST: online_diag_rsp[349]: PuntFabricDataPath: sending
a pak (seq 25), destination physical slot 1 (card type 0x3d02aa), NP 1, sfp=0xde
RP/0/RSP0/CPU0:Mar 25 05:43:43.384 EST: online_diag_rsp[349]: PuntFabricDataPath: sending
a pak (seq 25), destination physical slot 1 (card type 0x3d02aa), NP 2, sfp=0xf6
RP/0/RSP0/CPU0:Mar 25 05:43:43.384 EST: online_diag_rsp[349]: PuntFabricDataPath: sending
a pak (seq 25), destination physical slot 1 (card type 0x3d02aa), NP 3, sfp=0x10e
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: PuntFabricDataPath:
Time took to receive 22 pkts: 503922888 nsec, timeout value: 500000000 nsec
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: PuntFabricDataPath:
Received 22 packets, expected 24 => Some replies missed
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: PuntFabricDataPath:
Got a packet from physical slot 1, np 0
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: Successfully verified
a packet, seq. no.: 25
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: PuntFabricDataPath:
Got a packet from physical slot 1, np 2 <= Replies from NP1 and NP3 missing
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: Successfully verified
a packet, seq. no.: 25
RP/0/RSP0/CPU0:Mar 25 05:43:43.888 EST: online_diag_rsp[349]: PuntFabricDataPath:
Got a packet from physical slot 3, np 0
- 診斷跟蹤：
RP/0/RP1/CPU0:AG2-2#show diagnostic trace location 0/rp1/CPU0
Fri Mar 25 12:16:40.866 IST
1765 wrapping entries (3136 possible, 2048 allocated, 0 filtered, 3503120 total)
Mar 16 02:40:21.641 diags/online/gold_error 0/RP1/CPU0 t7356 Failed to get ack: got 0 responses,
expected 1
Mar 16 02:40:36.490 diags/online/message 0/RP1/CPU0 t8947 My nodeid 0x120, rack# is 0, slot# 1,
board type = 0x100327
Mar 16 02:40:36.948 diags/online/message 0/RP1/CPU0 t8947 dev cnt=25, path cnt=3, shm loc for
dev alarms@0x7fd4f0bec000, path alarms@0x7fd4f0bec01c, path alarm data@0x7fd4f0bec028
Mar 16 02:40:37.022 diags/online/message 0/RP1/CPU0 t8947 Last rpfo time: 1647378637
Mar 24 06:03:27.479 diags/online/error 0/RP1/CPU0 2105# t9057 PuntFabricDataPath test error:
physical slot 11(LC# 9): expected np mask: 0x0000000f, actual: 0x0000000b, failed: 0x00000004
Mar 24 06:03:27.479 diags/online/error 0/RP1/CPU0 634# t9057 PuntFabricDataPath test failure detected,
detail in the form of (0-based) (slot, NP: count): (LC9,2: 13)
交換矩陣分類
- 交換矩陣運行狀況（提供鏈路狀態、統計資訊、丟棄和警報的摘要）：
show controllers fabric health location <>
- 脊柱健康狀況：
show controllers fabric health spine all
- 板載故障記錄(OBFL)（重新載入後，也可以使用此功能）：
admin
sysadmin-vm:0_RP0# show logging onboard fabric location 0/0
- 檢查輸入LC FIA上的交換矩陣計數器：
show controllers fabric fia errors ingress location <>
show controllers fabric fia stats location
- 輸入LC縱橫制（不適用於Trident和SIP-700）：
show controllers fabric crossbar statistics instance [0-1] location <>
- 輸出LC縱橫制（不適用於Trident和SIP-700）：
show controllers fabric crossbar statistics instance [0-1] location <>
- 輸出LC FIA:
show controllers fabric fia errors egress location <>
show controllers fabric fia stats location
- 主幹統計資訊：
show controllers fabric crossbar statistics instance [0-1] spine [0-6]
- 檢查交換矩陣丟棄：
- 輸入LC FIA:
show controllers fabric fia drops ingress location <>
- 輸出LC FIA:
show controllers fabric fia drops egress location <>
- ASIC錯誤：
- LSP:
show controllers fabric crossbar asic-errors instance 0 location<>
show asic-errors fia <> all location <>
-
- 戰斧：
show asic-errors fia <> all location <>
RP/0/RP0/CPU0:AG3_1#show controllers np fabric-counters all np0 location 0/0/CPU0
Node: 0/0/CPU0:
----------------------------------------------------------------
Egress fabric-to-bridge interface 2 counters for NP 0
INTERLAKEN_CNT_TX_BYTES 0x000073fc 23b6d99b
INTERLAKEN_CNT_TX_FRM_GOOD 0x000000ae a79d6612
INTERLAKEN_CNT_TX_FRM_BAD 0x00000000 00000000 >>> this is 0 which is good,
need to check if it is incremented
-------------------------------------------------------------
Egress fabric-to-bridge interface 3 counters for NP 0
INTERLAKEN_CNT_TX_BYTES 0x0004abdd fe02068d
INTERLAKEN_CNT_TX_FRM_GOOD 0x000005b8 089aac95
INTERLAKEN_CNT_TX_FRM_BAD 0x00000000 00000000
-------------------------------------------------------------
Node: 0/0/CPU0:
----------------------------------------------------------------
Ingress fabric-to-bridge interface 2 counters for NP 0
INTERLAKEN_CNT_RX_BYTES 0x0004aeb5 a4b9dbbe
INTERLAKEN_CNT_RX_FRM_GOOD 0x0000058e b7b91c15
INTERLAKEN_CNT_RX_FRM_BAD 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_CRC32_ERROR 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_CRC24_ERROR 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_SIZE_ERROR 0x00000000 00000000
-------------------------------------------------------------
Ingress fabric-to-bridge interface 3 counters for NP 0
INTERLAKEN_CNT_RX_BYTES 0x000094ce b8783f95
INTERLAKEN_CNT_RX_FRM_GOOD 0x000000f5 33cf9ed7
INTERLAKEN_CNT_RX_FRM_BAD 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_CRC32_ERROR 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_CRC24_ERROR 0x00000000 00000000
INTERLAKEN_CNT_RX_BURST_SIZE_ERROR 0x00000000 00000000
- 要驗證FIA的鏈路狀態：
show controllers fabric fia link-status location
RP/0/RP0/CPU0:AG3_1#show controllers fabric fia link-status location 0/0/CPU0
********** FIA-0 **********
Category: link-0
spaui link-0 Up >>> FIA to NP link
spaui link-1 Up >>> FIA to NP link
arb link-0 Up >>> Arbitor link
xbar link-0 Up >>> FIA to XBAR link
xbar link-1 Up >>> FIA to XBAR link
xbar link-2 Up >>> FIA to XBAR link
- 若要驗證XBAR的連結狀態：
RP/0/RP0/CPU0:AG3_1#show controllers fabric crossbar link-status instance 0 lo 0/0/CPU0
Mon May 2 04:05:06.161 EDT
PORT Remote Slot Remote Inst Logical ID Status
======================================================
00 0/0/CPU0 01 2 Up
01 0/FC3 01 0 Up
02 0/FC3 00 0 Up
03 0/FC4 01 0 Up
04 0/FC2 01 0 Up
05 0/FC4 00 0 Up
06 0/FC2 00 0 Up
07 0/FC1 01 0 Up
10 0/FC1 00 0 Up
14 0/FC0 01 0 Up
15 0/FC0 00 0 Up
16 0/0/CPU0 02 0 Up
18 0/0/CPU0 02 2 Up
19 0/0/CPU0 02 1 Up
20 0/0/CPU0 03 2 Up
21 0/0/CPU0 03 1 Up
22 0/0/CPU0 03 0 Up
23 0/0/CPU0 00 2 Up
24 0/0/CPU0 00 1 Up
25 0/0/CPU0 00 0 Up
26 0/0/CPU0 01 0 Up
27 0/0/CPU0 01 1 Up
如果在LSP卡中觀察到以下日誌：
LC/0/3/CPU0:Jul 5 13:05:53.365 IST: fab_xbar[172]: %PLATFORM-CIH-5-ASIC_ERROR_THRESHOLD :
sfe[1]: An interface-err error has occurred causing packet drop transient.
ibbReg17.ibbExceptionHier.ibbReg17.ibbExceptionLeaf0.intIpcFnc0UcDataErr Threshold has been exceeded
此處的17*2有助於使用 show controllers fabric crossbar link-status instance 1 lo 0/3/CPU0 指令:
日誌集合：
show platform
show inventory
show tech fabric
show tech np
show tech ethernet interface
show logging
show pfm location all
show pfm trace location <location id>
show controllers pm vqi location all
show hw-module fpd location all (cxr) / admin show hw-module fpd (exr)
show controllers fti trace <process-name> location <Card location>
admin show tech obfl
Cxr:
From Admin:
show logging onboard common location <>
show logging onboard error location <>
Exr:
From sysadmin/calvados:
show logging onboard fabric location <>
- 如果FIA中存在ASIC錯誤：
對於LS:
show controllers asic LS-FIA instance <instance> block <block_name> register-name <register_name> location <>
對於LSP:
show controllers asic LSP-FIA instance <instance> block <block_name> register-name <register_name> location <>
如果報告的錯誤如下所示：
LC/0/9/CPU0:Mar 1 05:12:25.474 IST: fialc[137]: %PLATFORM-CIH-5-ASIC_ERROR_THRESHOLD :
fia[3]: A link-err error has occurred causing performance loss persistent.
fnc2serdesReg1.fnc2serdesExceptionHier.fnc2serdesReg1.fnc2serdesExceptionLeaf0.
iNTprbsErrTxphyrdydropped6 Threshold has been exceeded
- 例項是FIA ASIC的例項編號。這裡是「3」 block_name是「fnc2serdesReg1」，register_name是「fnc2serdesExceptionLeaf0」。
- 如果LC/RSP XBAR上出現ASIC錯誤：
show controllers asic SKB-XBAR instance <instance> block-name <block_name> register-name <register_name> location <>
如果報告的錯誤如下所示：
LC/0/7/CPU0:Mar 4 06:42:01.241 IST: fab_xbar[213]: %PLATFORM-CIH-5-ASIC_ERROR_THRESHOLD :
sfe[0]: An interface-err error has occurred causing packet drop transient.
ibbReg11.ibbExceptionHier.ibbReg11.ibbExceptionLeaf0.intIpcFnc1UcDataErr Threshold has been exceeded
- 例項是SFE/XBAR ASIC編號的例項。這裡，"0" block_name是"ibbReg11", register_name是"ibbExceptionLeaf0"。
- 如果在FC XBAR上報告ASIC錯誤：
show controllers asic FC2-SKB-XBAR instance <instance> block-name <block_name> register-name <register_name> location
如果報告的錯誤如下所示：
RP/0/RP0/CPU0:Mar 4 06:41:14.398 IST: fab_xbar_sp3[156]: %PLATFORM-CIH-3-ASIC_ERROR_SPECIAL_HANDLE_THRESH :
fc3xbar[1]: A link-err error has occurred causing packet drop transient.
cflReg17.cflExceptionHier.cflReg17.cflExceptionLeaf4.intCflPal1RxAlignErrPktRcvd Threshold has been exceeded
那麼ASIC是「FC3-SKB-XBAR」例項是SFE/XBAR ASIC的編號。這裡是「1」，兩個都來自「fc3xbar[1]」 block_name是「cflReg17」，register_name是「cflExceptionLeaf4」。
範例：
RP/0/RSP0/CPU0: AG2-10#sh logging | i ASIC
RP/0/RSP0/CPU0:May 11 20:48:57.658 IST: fab_xbar[184]: %PLATFORM-CIH-5-ASIC_ERROR_THRESHOLD :
sfe[0]: An interface-err error has occurred causing packet drop transient.
ibbReg13.ibbExceptionHier.ibbReg13.ibbExceptionLeaf0.intIpcFnc0UcDataErr Threshold has been exceeded
RP/0/RSP0/CPU0: AG2-10#sh controllers fabric crossbar link-status instance 0 location 0/rsp0/CPU0
PORT Remote Slot Remote Inst Logical ID Status
======================================================
04 0/0/CPU0 00 1 Up
06 0/0/CPU0 00 0 Up
08 0/7/CPU0 00 1 Up
10 0/7/CPU0 00 0 Up
24 0/2/CPU0 00 0 Up
26 0/2/CPU0 00 1 Up
>>> ibbReg13 >> 13*2 = 26 SO IT IS POINTING TO LC2 – IN THIS CASE YOU CAN DO OIR TO RECOVER THE ASIC ERROR
40 0/RSP0/CPU0 00 0 Up
RP/0/RSP0/CPU0: AG2-10#show controllers asic SKB-XBAR instance 0 block-name ibbReg13 register-name ibbExceptionLeaf0 location 0/RSP0/CPU0
address name value
0x00050d080 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int1Stat 0x00000000 (4 bytes)
address name value
0x00050d084 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int1StatRw1s 0x00000000 (4 bytes)
address name value
0x00050d088 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int1Enable 0xfffffffb (4 bytes)
address name value
0x00050d08c SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int1First 0x00000000 (4 bytes)
address name value
0x00050d090 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int2Stat 0x00000c50 (4 bytes)
address name value
0x00050d094 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int2StatRw1s 0x00000c50 (4 bytes)
address name value
0x00050d098 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int2Enable 0x00000000 (4 bytes)
address name value
0x00050d09c SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_int2First 0x00000000 (4 bytes)
address name value
0x00050d0a0 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_haltEnable 0x00000000 (4 bytes)
address name value
0x00050d0a4 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_fault 0x00000000 (4 bytes)
address name value
0x00050d0a8 SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_intMulti 0x00000840 (4 bytes)
address name value
0x00050d0ac SkyboltRegisters_ibbReg13_ibbExceptionLeaf0_leaf 0x00000000 (4 bytes)
RP/0/RSP0/CPU0:AG2-10#
仲裁故障分類
若要檢查連結狀態：
RP/0/RSP0/CPU0:AG2-10#sho controllers fabric arbiter link-status location 0/1/$
Port Remote Slot Remote Elem Remote Inst Status
=======================================================
00 0/1/CPU0 FIA 0 Up
01 0/1/CPU0 FIA 1 Up
24 0/RSP0/CPU0 ARB 0 Up
25 0/RSP1/CPU0 ARB 0 Up
要檢查VQI可用性：
RP/0/RP0/CPU0:AG3_1#sh controllers fabric vqi assignment all
Current mode: Highbandwidth mode - 2K VQIs
Node Number of VQIs
----------------------------
0/0/CPU0 80
0/1/CPU0 40
0/2/CPU0 48
0/3/CPU0 80
0/5/CPU0 80
0/7/CPU0 80
0/12/CPU0 64
RP*/RSP* 8
----------------------------
In Use = 480
Available = 1568
檢查分配給VQI的速度：
RP/0/RP0/CPU0:AG3_1#sh controller fabric vqi assignment slot 7
Thu May 12 07:58:59.897 EDT
slot = 7
fia_inst = 0
VQI = 400 SPEED_100G
VQI = 401 SPEED_100G
VQI = 402 SPEED_100G
VQI = 403 SPEED_100G
VQI = 404 SPEED_100G
VQI = 405 SPEED_100G
VQI = 406 SPEED_100G
slot = 7
fia_inst = 1
VQI = 416 SPEED_40G
VQI = 417 SPEED_40G
VQI = 418 SPEED_40G
VQI = 419 SPEED_40G
VQI = 420 SPEED_100G
如果您觀察到FIA上有任何尾部掉線，請檢查以下步驟：
檢查VQI中的隊列深度：
RP/0/RP0/CPU0:AG3_1#show controllers fabric fia q-depth location 0/0/CPU0
Thu May 12 08:00:42.186 EDT
********** FIA-0 **********
Category: q_stats_a-0
Voq ddr pri Cellcnt Slot_FIA_NP
28 0 2 2 LC0_1_1
********** FIA-0 **********
Category: q_stats_b-0
Voq ddr pri Cellcnt Slot_FIA_NP
********** FIA-1 **********
Category: q_stats_a-1
Voq ddr pri Cellcnt Slot_FIA_NP
7 0 2 12342 LC0_0_0
>>> Here Packet count is high so we need to check for LC0 FIA0 NP0 (egress) is there any congestion
or any other issue in LC0 FIA0 or NP0
Here Pri = 2 is the default queue (BE) , Pri = 0 is P1 (Voice, real time) queue, Pri = 1 is P2
97 0 2 23 LC1_0_0
RP/0/RP0/CPU0:AG3_1#show controllers fabric vqi assignment slot 02
slot = 2
fia_inst = 0
VQI = 0 SPEED_10G
VQI = 1 SPEED_10G
VQI = 2 SPEED_10G
VQI = 3 SPEED_10G
VQI = 4 SPEED_10G
VQI = 5 SPEED_10G
VQI = 6 SPEED_10G
VQI = 7 SPEED_10G
VQI的埠對映詳細資訊：
RP/0/RP0/CPU0:AG3_1#show controllers pm vqi location 0/0/CPU0
Platform-manager VQI Assignment Information
Interface Name | ifh Value | VQI | NP#
--------------------------------------------------
TenGigE0_0_0_0_1 | 0x4000680 | 1 | 0
TenGigE0_0_0_0_2 | 0x40006c0 | 2 | 0
TenGigE0_0_0_0_3 | 0x4000700 | 3 | 0
TenGigE0_0_0_0_4 | 0x4000740 | 4 | 0
TenGigE0_0_0_0_5 | 0x4000780 | 5 | 0
TenGigE0_0_0_0_6 | 0x40007c0 | 6 | 0
TenGigE0_0_0_0_7 | 0x4000800 | 7 | 0
RP/0/RP0/CPU0:AG3_1#show controllers pm interface tenGigE 0/0/0/0/7
Ifname(1): TenGigE0_0_0_0_7, ifh: 0x4000800 :
iftype 0x1e
egress_uidb_index 0x12, 0x0, 0x0, 0x0
ingress_uidb_index 0x12, 0x0, 0x0, 0x0
port_num 0x0
subslot_num 0x0
ifsubinst 0x0
ifsubinst port 0x7
phy_port_num 0x7
channel_id 0x0
channel_map 0x0
lag_id 0x7e
virtual_port_id 0xa
switch_fabric_port 7 >>> VQI matching for the ports
in_tm_qid_fid0 0x38001e
in_tm_qid_fid1 0x0
in_qos_drop_base 0xa69400
out_tm_qid_fid0 0x1fe002
out_tm_qid_fid1 0xffffffff
np_port 0xd3
日誌集合：
Show tech fabric
Show tech np
show controllers pm trace ?
async Platform manager async trace
creation Platform manager interface creation/deletion trace
error Platform manager error trace
information Platform manager information trace
init Platform manager init trace
other Platform manager common trace
stats Platform manager stats trace
NP故障分類
NP負載驗證：
RP/0/RP0/CPU0:AG3_1#show controller np load all location 0/0/CPU0
Node: 0/0/CPU0:
----------------------------------------------------------------
Load Packet Rate
NP0: 2% utilization 3095766 pps
NP1: 3% utilization 5335675 pps
NP2: 0% utilization 498 pps
NP3: 0% utilization 1117 pps
埠對映：
RP/0/RP0/CPU0:AG3_1#show controllers np ports all location 0/0/CPU0
Node: 0/0/CPU0:
----------------------------------------------------------------
NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 -- 0 TenGigE0/0/0/0/0 - TenGigE0/0/0/0/9, TenGigE0/0/0/1/0 - TenGigE0/0/0/1/9
1 -- 1 TenGigE0/0/0/2/0 - TenGigE0/0/0/2/9, HundredGigE0/0/0/3
2 -- 2 HundredGigE0/0/0/4 - HundredGigE0/0/0/5
3 -- 3 HundredGigE0/0/0/6 - HundredGigE0/0/0/7
戰斧
請注意，這是管理模式：
sysadmin-vm:0_RP0# show controller switch statistics location 0/LC0/LC-SW
Thu May 12 12:32:37.160 UTC+00:00
Rack Card Switch Rack Serial Number
--------------------------------------
0 LC0 LC-SW
Tx Rx
Phys State Drops/ Drops/
Port State Changes Tx Packets Rx Packets Errors Errors Connects To
----------------------------------------------------------------------------
0 Up 2 3950184361 3977756349 0 0 NP0
1 Up 2 0 0 0 0 NP0
8 Up 1 1319787462 209249871 0 0 LC CPU N0 P0
9 Up 1 3374323096 1819796660 0 0 LC CPU N0 P1
16 Up 2 2245174606 1089972811 0 0 NP1
17 Up 2 0 0 0 0 NP1
18 Up 2 65977 16543963 0 0 NP2
19 Up 2 0 0 0 0 NP2
32 Up 2 128588820 3904804720 0 0 NP3
33 Up 2 0 0 0 0 NP3
show asic-error np <> all loc <> >>> Ignore the macwrap errors as they are seen for every
interface flaps/ Execute 3-4 times to verify the drops increment
show controller np fast-drop <> loc <> >>> Execute 3-4 times to verify the drops increment
RP/0/RP0/CPU0:AG3_1#show controller np fast-drop np0 location 0/0/CPU0
Thu May 12 10:13:22.981 EDT
Node: 0/0/CPU0:
----------------------------------------------------------------
All fast drop counters for NP 0:
TenGigE0/0/0/1/0-TenGigE0/0/0/1/9:[Priority1] 0
TenGigE0/0/0/1/0-TenGigE0/0/0/1/9:[Priority2] 0
TenGigE0/0/0/1/0-TenGigE0/0/0/1/9:[Priority3] 0
TenGigE0/0/0/0/0-TenGigE0/0/0/0/9:[Priority1] 0
TenGigE0/0/0/0/0-TenGigE0/0/0/0/9:[Priority2] 0
TenGigE0/0/0/0/0-TenGigE0/0/0/0/9:[Priority3] 0
show controllers np punt-path-counters all HOST-IF-0 np<> location <>
[Check for IF_CNT_RX_FRM & IF_CNT_TX_FRM] >>> To check if diagnostic packets make it to the LC NP
Host CPU network port
光速
show asic-error np <> all loc <> >>> Ignore the macwrap errors as they are seen for every interface flap
RP/0/RP0/CPU0:AG3_1#sho asic-errors np 0 all location 0/5/CPU0
************************************************************
* 0_5_CPU0 *
************************************************************
************************************************************
* Single Bit Errors *
************************************************************
************************************************************
* Multiple Bit Errors *
************************************************************
************************************************************
* Parity Errors *
************************************************************
************************************************************
* Generic Errors *
************************************************************
ASR, ASR9K Lightspeed 20*100GE SE LC, 0/5/CPU0, npu[0]
Name : mphmacwrapReg1.mphmacwrapExceptionLeaf4.mphWrapIrqUmacIpInt82
Leaf ID : 0x2023e082
Error count : 1
Last clearing : Thu Apr 7 11:41:47 2022
Last N errors : 1
--------------------------------------------------------------
First N errors.
@Time, Error-Data
------------------------------------------
show controller np fast-drop <> loc <> >>> Execute 3-4 times to verify the drops increment
RP/0/RP0/CPU0:AG3_1#show controller np fast-drop np0 location 0/5/CPU0
Thu May 12 10:13:28.321 EDT
Node: 0/5/CPU0:
----------------------------------------------------------------
All fast drop counters for NP 0:
HundredGigE0_5_0_0[Crit] 0
HundredGigE0_5_0_0[HP] 0
HundredGigE0_5_0_0[LP2] 0
HundredGigE0_5_0_0[LP1] 0
HundredGigE0_5_0_0[Crit+HP_OOR] 0
HundredGigE0_5_0_0[LP2+LP1_OOR] 0
HundredGigE0_5_0_1[Crit] 0
HundredGigE0_5_0_1[HP] 0
HundredGigE0_5_0_1[LP2] 0
HundredGigE0_5_0_1[LP1] 0
HundredGigE0_5_0_1[Crit+HP_OOR] 0
請注意，這是管理模式：
sysadmin-vm:0_RP0# show controller switch statistics location 0/LC5/LC-SW >>> Execute 3-4
times to verify the errors increment
Rack Card Switch Rack Serial Number
--------------------------------------
0 LC5 LC-SW
Tx Rx
Phys State Drops/ Drops/
Port State Changes Tx Packets Rx Packets Errors Errors Connects To
-----------------------------------------------------------------------------
0 Up 4 1456694749 329318054 0 4 CPU -- EOBC
1 Up 2 21 23 0 0 CPU -- flexE
2 Up 4 1063966999 87683758 0 0 CPU -- PUNT
3 Up 4 885103800 3021484524 0 0 CPU -- BFD
4 Up 3 329319167 1456700372 0 0 RP0
5 Up 3 0 0 0 0 RP1
6 Up 1 11887785 2256 0 0 IPU 0
7 Up 1 0 1086 0 0 IPU 1
9 Up 4 74028034 3025657779 0 0 NP0
10 Up 4 5 0 0 0 NP0
11 Down 1 0 0 0 0 PHY0 -- flexE
12 Up 4 264928 264929 0 0 NP1
13 Up 2 5 0 0 0 NP1
14 Down 1 0 0 0 0 PHY1 -- flexE
15 Up 4 1516538834 1159586563 0 0 NP2
日誌收集：
show tech np
show tech fabric
show asic-errors fia trace all location <>
- 在eXR中，收集np_datalog:
RP/0/RP0/CPU0:AG3_1#run chvrf 0 ssh lc0_xr
LC : [one time capture]
show_np -e <> -d npdatalog [<> should be the affected NP]
Path where NP datalogs is saved : /misc/scratch/np/NPdatalog_0_0_CPU0_np0_prm__20220512-105332.txt.gz
LC : 5 to 10 times
show_np -e <> -d pipeline [<> should be the affected NP]
- 對於LSP上的NP Init故障：
RP/0/RP0/CPU0:AG2-2#show controllers np ports all location 0/6/CPU0
Node: 0/6/CPU0:
----------------------------------------------------------------
NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 -- 0 HundredGigE0/6/0/0 - HundredGigE0/6/0/31 --
1 -- 1 HundredGigE0/6/0/4 - HundredGigE0/6/0/7
NP2 is down. >>>>>>>>>. NP Down/Init Failure
3 -- 3 HundredGigE0/6/0/12 - HundredGigE0/6/0/154 --
4 -- 4 HundredGigE0/6/0/16 - HundredGigE0/6/0/19
這些日誌觀察：
LC/0/6/CPU0:Mar 23 02:53:56.175 IST: npu_server_lsp[138]: %PLATFORM-LDA-3-INIT_FAIL :
Failed to initialize lda_bb_np_reset_process 13795 inst 0x2 LC INIT: Failed in NP HAL
Reset np (0x00000001 - Operation not permitted) : npu_server_lsp : (PID=4597) :
-Traceback= 7fea2d5cd9f6 7fea2d7d5816 7fea21465efa 7fea21465fc2 7fea42ad0bed 55a9dbd66031
7fea45e1c855 7fea45e1cc2b 7fea2624d526 7fea3571b96a 7fea4d6e4831 55a9dbd691e9
LC/0/6/CPU0:Mar 23 02:53:56.185 IST: npu_server_lsp[138]: %PLATFORM-NP-4-INIT_DEBUG_MSG :
LDA NP2 Reset failed!! Check for a downlevel IPU version.
日誌收集：
show tech-support ethernet interfaces
show tech-support ethernet controllers
show tech-support np
show tech-support fpd
admin show tech-support ctrace (in eXR)
show tech fabric
show asic-errors fia trace all location <>
show logging
gather (in eXR)
RP/0/RP0/CPU0:AG3_1#admin
sysadmin-vm:0_RP0#
[sysadmin-vm:0_RP0:~]$bash -l
[sysadmin-vm:0_RP0:~]$ gather
File will be generated and will get saved in rp0_xr:/misc/disk1
Tomahawk、LSQ和LSP的一般日誌收集
show platform
show inventory
show tech fabric
show tech np
show tech ethernet interface
show logging
show pfm location all
show pfm trace location <location id>
sh pfm process <> location <>
show controllers pm vqi location all
show hw-module fpd location all (cxr) / admin show hw-module fpd (exr)
show controllers fti trace <process-name> location <card location>
Cxr:
From admin:
show logging onboard common location <>
show logging onboard error location <>
Exr:
From sysadmin/calvados:
show logging onboard fabric location <>"
常見錯誤簽名和建議
| 類別 |
錯誤 |
意見 |
建議 |
| NP初始化失敗 |
LC/0/0/CPU0:Sep 29 00:41:13.171 IST: pfm_node_lc[304]: %PLATFORM-NP-1-NP_INIT_FAIL_NO_RESET: Set|prm_server_ty[168018]|0x1008006|Persistent NP initialization failure，不需要重新載入線路卡。 |
NP可能會因為硬體奇偶校驗/TCAM錯誤而進入NP初始化持久錯誤，從而迫使NP關閉。 |
LC通過CLI重新載入以復原。 |
| 如果第一次重新載入LC後是暫時的，則問題可以復原。 |
完成相同錯誤的重複例項的退貨授權(RMA)。 |
||
| 如果看到重複例項，則返回RMA。（獲取故障板的照片以檢查SR中的損壞/彎曲的針腳）。 |
|||
| 由於欄位處理錯誤，新卡可能會面臨相同的問題。 |
|||
| 對映到NP的介面處於關閉狀態/無影響。 |
|||
| ASIC致命故障 — 雙位ECC錯誤 |
LC/0/8/CPU0:May 29 18:29:09.836 IST: pfm_node_lc[301]: %FABRIC-FIA-0-ASIC_FATAL_FAULT : Set|fialc[159811]|0x108a000|Fabric interface asic ASIC0遇到致命故障0x1 - DDR雙ECC錯誤 |
這是FIA上的雙位ECC錯誤是硬錯誤。 |
FIA上出現HW錯誤。 |
| 錯誤可能會再次出現，因此建議使用admin shut of LC。 |
RMA卡。 |
||
| 對映到NP/FIA的介面處於關閉狀態/無影響。 |
|||
| 在FIA提出fib_mgr進程塊的一個案例中，可以看到此問題。 |
|||
| SERDES錯誤 |
·RP/0/RSP1/CPU0:Apr 17 12:22:10.690 IST: pfm_node_rp[378]: %PLATFORM-CROSSBAR-1-SERDES_ERROR_LNK0 : Set|fab_xbar[209006]|0x101702f|XBAR_1_Slot_1 |
LC交換矩陣或RSP交換矩陣發生交換矩陣錯誤 |
LC通過CLI重新載入，以便針對重複錯誤恢復瞬時/CRC錯誤。 |
| DATA_NB_SERDES_1_FAIL_0 |
LC/0/3/CPU0:Apr 10 18:55:03.213 IST: pfm_node_lc[304]: %FABRIC-FIA-1-DATA_NB_SERDES_1_FAIL_0 : Set|fialc[168004]|0x103d001|FIA 1上的資料NB伺服器鏈路1故障 RP/0/RSP0/CPU0:Apr 10 18:55:13.043 IST: FABMGR[227]: %PLATFORM-FABMGR-2-FABRIC_INTERNAL_FAULT: 0/3/CPU0（插槽3）遇到交換矩陣故障。介面將關閉。 |
介面重新訓練機制可在交換矩陣上自動恢復SERDES錯誤。 在硬體出現問題時，該錯誤可能會再次在LC或RSP上出現。 SERDES上的介面保持運行/頻繁錯誤會影響流量。 |
OIR後重複例項的RMA。 |
| ASIC INIT錯誤 |
·LC/0/6/CPU0:Jul 17 00:01:40.738 2019:pfm_node_lc[301]: %FABRIC-FIA-1-ASIC_INIT_ERROR : Set|fialc[168003]|0x108a000|在FIA例項0上檢測到ASIC INIT錯誤 |
LC上任何FIA的FIA例項關閉事件，系統日誌中出現ASIC INIT錯誤。 |
LC通過CLI重新載入以排除任何暫時性問題。 |
| FIA ASIC嚴重錯誤 (TS_NI_INTR_LCL_TIMER_EXPIRED) |
LC/0/19/CPU0:Mar 8 04:52:29.020 IST: pfm_node_lc[301]: %FABRIC-FIA-0-FATAL_INTERRUPT_ERROR : Set|fialc[172098]|0x108a003|FIA FATAL error interrupt on FIA 3: TS_NI_INTR_LCL_TIMER_EXPIRED |
對於新卡，可以看出裝置在裝運/安裝時處理不當，對主機板造成物理損壞。 少數電路板沒有出現任何物理損壞，但在EFA期間觀察到焊料裂縫。這說明包裝件過緊，一段時間後可能出現故障。 SERDES上的介面保持運行/頻繁錯誤會影響流量。 |
如果問題仍然存在，請繼續執行RMA/R&R。 |
| NP快速重置(Tomahawk) |
LC/0/4/CPU0:Jul 6 04:06:49.259 IST: prm_server_ty[318]: %PLATFORM-NP-3-ECC : prm_ser_check：已完成NP快速重置，以成功從NP 1上的軟錯誤中恢復。無需進一步的糾正操作。 |
NP檢測到軟奇偶校驗問題，並嘗試通過嘗試NP快速重置來修復此問題。 |
首次出現時無RMA。 |
| NP奇偶校驗LC重新載入 |
LC/0/6/CPU0:Jan 27 20:38:08.011 IST: prm_server_to[315]: %PLATFORM-NP-0-LC_RELOAD: NP3在一小時內進行了三次快速重置，啟動NPdatalog收集和自動LC重新啟動 |
通常，在三次恢復嘗試後，LC會自行重新載入，以修復Tomahawk卡通常出現的NP上的奇偶校驗問題。 LC通過重新載入LC並修復報告的NP中的軟不可恢復奇偶校驗問題來採取自動恢復操作。 對映到NP的介面因重置/無影響而關閉。 |
相同錯誤的重複例項的RMA。 |
| LC_NP_LOOPBACK_FAILED |
LC/0/1/CPU0:Jul 26 17:29:06.146 IST: pfm_node_lc[304]: %PLATFORM-DIAGS-0-LC_NP_LOOPBACK_FAILED_TX_PATH : Set|online_diag_lc[168022]|線卡NPU環回測試(0x2000006)|鏈路故障掩碼為0x1。 |
LC NP環回診斷測試在某個NP上失敗。 |
LC通過CLI重新載入以排除任何暫時性問題。 |
| 在PFM中將警報設定為「LC_NP_LOOPBACK_FAILED_XX_PATH」。 |
相同錯誤的重複例項的RMA。 |
||
| 對映到NP的介面因重置/無影響而關閉。 |
|||
| FABRIC-FIA-1-SUSTAINED_CRC_ERR |
LC/0/5/CPU0:Mar 6 05:47:34.748 IST: pfm_node_lc[303]: %FABRIC-FIA-1-SUSTAINED_CRC_ERR: Set|fialc[168004]|0x103d000|Fabric interface ASIC-0存在持續CRC錯誤 |
由於交換矩陣FIA持續CRC錯誤而關閉FIA。 |
LC通過CLI重新載入以排除任何暫時性問題。 |
| 發生FIA關閉事件時，FIA上的介面也會關閉。 |
相同錯誤的重複例項的RMA。 |
||
| 介面保持開啟/無影響。 |
|||
| FAB ARB XIF1錯誤 |
·LC/0/6/CPU0:Jan 25 19:31:22.787 IST: pfm_node_lc[302]: %PLATFORM-FABARBITER-1-RX_LINK_ERR : Clear|fab_arb[163918]|0x1001001|LIT_XIF1_K_CHAR_ERR LC/0/6/CPU0:Jan 25 19:31:22.787 IST: pfm_node_lc[302]: %PLATFORM-FABARBITER-1-SYNC_ERR : Clear|fab_arb[163918]|0x1001001|LIT_XIF1_LOSS_SYNC LC/0/6/CPU0:Jan 25 19:33:23.010 IST: pfm_node_lc[302]: %PLATFORM-FABARBITER-1-RX_LINK_ERR: Set|fab_arb[163918]|0x1001001|LIT_XIF1_DISP_ERR |
LC & Fabric arbiter sync & rx_link錯誤的PUNT錯誤。 介面保持開啟/無影響。 |
翻動卡片以排除任何臨時問題。 相同錯誤的重複例項的RMA。 |
| FPOE_read_write錯誤 |
xbar錯誤跟蹤(show tech fabric) |
思科錯誤ID CSCvv45788 |
軟體缺陷 |
| FIA_XBAR系列 |
#show controller fabric fia link-status location 0/9/CPU0 |
翻動卡片以排除任何臨時問題。相同錯誤的重複例項的RMA。 |
|
| NP DIAG ICFD快速重置 |
NP-DIAG on NP0, ICFD(STS-1), NP可以是0-4 NP3在一小時內進行了3次快速重置，啟動NPdatalog收集和LC自動重新啟動 |
觸發NP的FAST重置 如果一小時內有3個NP快速重置，則LC會重新載入。 |
如果LC重新載入多次，則RMA。 |
| PRM運行狀況監控無法獲取資料包NP快速重置 |
NP-DIAG運行狀況監控失敗 NP3在一小時內進行了3次快速重置，啟動NPdatalog收集和LC自動重新啟動 |
觸發NP的FAST重置 如果一小時內有3個NP快速重置，則LC會重新載入。 |
如果LC重新載入多次，則RMA。 |
| PRM運行狀況監控獲取損壞的資料包 — NP快速重置 |
NP-DIAG健康監視損壞 NP3在一小時內進行了3次快速重置，啟動NPdatalog收集和LC自動重新啟動 |
觸發NP的FAST重置 如果一小時內有3個NP快速重置，則LC會重新載入。 |
如果LC重新載入多次，則RMA。 |
| 頂級非活動故障 |
NP上的NP-DIAG故障 Upper inactivity上的Ucode中斷 — NP是否快速重置 |
觸發NP的FAST重置 如果一小時內有3個NP快速重置，則LC會重新載入。 |
如果LC重新載入多次，則RMA。 |
| LSP NP初始化失敗 |
LC/0/6/CPU0:Mar 23 02:53:56.175 IST: npu_server_lsp[138]: %PLATFORM-LDA-3-INIT_FAIL ：無法初始化lda_bb_np_reset_process 13795 inst 0x2 LC INIT：在NP HAL重置np(0x00000001 — 不允許操作)中失敗：npu_server_lsp :(PID=4597): -Traceback= 7fea2d5d f6 7fea2d7d5816 7fea21465efa 7fea21465fc2 7fea42ad0bed 55a9dbd66031 7fea45e1c855 7fea45e1cc2b 7fea2624d526 7fea3571b96a 7fea4d6e4831 55a9dbd691e9 |
需要收集以下資訊：
檔案生成並儲存在rp0_xr:/misc/disk1中
|
LC通過CLI重新載入以排除任何暫時性問題。 |
| 戰斧NP初始化失敗（DDR培訓失敗） |
+++ show prm server trace error location 0/7/CPU0 [14:36:59.520 IST Sat A 2022年1月29日] ++++ 97個包裝條目（可能包含2112個，已分配320個，已過濾0個，共97個） 1月29日00:22:10.135 prm_server/error 0/7/CPU0 t10 prm_np_Channel_PowerUp : 0x80001d46為通道3階段4供電時出錯 1月29日00:22:10.136 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup : 0xa57c0200電源階段4在通道3上失敗 1月29日00:22:10.136 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup NP3啟動失敗，請重試。重試次數1 1月29日00:22:35.125 prm_server/error 0/7/CPU0 t10 prm_np_Channel_PowerUp : 0x80001d46為通道3階段4供電時出錯 1月29日00:22:35.125 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup : 0xa57c0200電源階段4在通道3上失敗 1月29日00:22:35.125 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup NP3啟動失敗，請重試。重試次數2 1月29日00:22:59.075 prm_server/錯誤0/7/CPU0 t10 prm_np_Channel_PowerUp : 0x80001d46為通道3階段4供電時出錯 1月29日00:22:59.075 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup : 0xa57c0200電源階段4在通道3上失敗 1月29日00:22:59.075 prm_server/error 0/7/CPU0 t10 np_thread_channel_bringup 3次嘗試後，NP3無法初始化。 1月29日00:23:00.087 prm_server/error 0/7/CPU0 t10 prm_send_pfm_msg：持續NP初始化失敗，不需要線路卡重新載入。 簽入NP驅動程式日誌 <NP#3>DDR培訓失敗（狀態0x1） <NP#3>ddr3TipRunAlg：調整失敗0 <NP#3>ddrTipRunAlgo操作碼：ddr3TipRunAlg失敗（錯誤0x1） <NP#3>***錯誤：未知0x1 |
節點：node0_7_CPU0 2029年1月00:22:58|8 |NP_INIT_FAIL_NO_RESET |E/A|5356 |網路進程|0x1008000 |
LC通過CLI重新載入以排除任何暫時性問題。如果問題仍然存在，則在LC上觀察RMA 思科錯誤ID CSCwa85165 |
| LSP NP初始化失敗（HbmReadParticleError錯誤） |
LC/0/13/CPU0:Jan 10 13:34:59.106 IST: npu_server_lsp[278]: %PLATFORM-NP-4-SHUTDOWN_START : NP4：檢測到EMRHIMREG.ch1Psch0HbmReadParticleError，正在進行NP關閉 LC/0/13/CPU0:Jan 10 13:34:59.106 IST: pfm_node_lc[330]: %PLATFORM-NP-0-UNRECOVERABLE_ERROR : Set|npu_server_lsp[4632]|0x10a5004|在NP4上檢測到不可恢復的錯誤 |
+++ show controllers np interrts all location 0/13/CPU0 [16:02:16.712 IST Mon Jan 2022] ++++ 節點：0/13/CPU0: ---------------------------------------------------------------- NPU中斷名稱Id Cnt —------------------------------------------------------------------------------------------ -------------- ------- <snip> 4 hbmdpReg0.hbmdpExceptionLeaf0.hbmdpIntNwlHbmdpRdDataUncorrectableErrCh1Psch1 0x201dc013 1 4 hbmdpReg0.hbmdpExceptionLeaf0.hbmdpIntNwlHbmdpRdDataUncorrectableErrCh1Psch0 0x201dc012 1 |
驗證是否已安裝思科錯誤ID CSCvt59803。 LC通過CLI重新載入有助於復原。 |
| 仲裁連結關閉與待命 |
Fabric-Manager: |
翻動卡片以排除任何臨時問題。RMA，表示同一錯誤的重複例項。 |
|
| Serdes錯誤 |
show serdes trace location 0/X/CPU0 | 在「HTL_ERR_DEVICE_NOT_CONNECTED」)中，您會看到以下錯誤： |
思科錯誤ID CSCvz75552 |
軟體缺陷 |
已知瑕疵
| 思科錯誤 ID | 元件 | 標題 |
| 思科錯誤ID CSCvy00012 | asr9k-diags-online | 由online_diag_rsp耗盡資料包記憶體 |
| 思科錯誤ID CSCvw57721 | asr9k-servicepack | Umbrella SMU包含更新的Lightspeed NP和仲裁伺服器韌體 |
| 思科錯誤ID CSCvz75552 | asr9k-vic-ls | Phy韌體掛起，導致在A9K-20HG-FLEX上無法識別光纖 |
| 思科錯誤ID CSCvz76691 | asr9k-servicepack | 針對Tomahawk線卡具有改進鏈路狀態中斷處理的Umbrella SMU |
| 思科錯誤ID CSCvz84139 | asr9k-ls-fabric | 路由器升級到742時fab_si崩潰 |
| 思科錯誤ID CSCwa81006 | asr9k-pfm | 在某些場景中，ASR9K/eXR無法提交fault-manager資料路徑埠關閉 |
| 思科錯誤ID CSCvz16840 | asr9k-fia | CLI重新載入LC時BLB作業階段翻動，因為轉送路徑由於6.5.2新增的變更而提前關閉 |
| 思科錯誤ID CSCwb64255 | asr9k-fab-xbar | 用於Starscream(9912)和Megatron(9922)機箱中的SKB的新SI設定 |
| 思科錯誤ID CSCwa09794 | asr9k-fab-xbar | 對SKB-SM15的RO機箱進行微調後的新的SI |
| 思科錯誤ID CSCvv45788 | asr9k-fab-xbar | fab_xbar和mgid-programmer進程同時訪問硬體 |
| 思科錯誤ID CSCwd22196 | asr9k-prm | Tomahawk LC上ILKN鏈路之間的RFD緩衝區耗盡 |
| 思科錯誤ID CSCwb66960 | asr9k-fab-infra | ASR9k punt交換矩陣故障隔離 |
| 思科錯誤ID CSCwa79758 | asr9k-fab-xbar | 對另一個具有XBAR鏈路故障的LSP LC執行OIR後，LSP LC上的組播丟失 |
| 思科錯誤ID CSCvw88284 | asr9k-lda-ls | RSP5 BW在9910/9906機箱上預設為200G，而不是600G。 |
| 思科錯誤ID CSCvm82379 | asr9k-fab-arb | fab-arb crashed while taking sh tech fabric |
| 思科錯誤ID CSCvh00349 | asr9k-fia | ASR9k交換矩陣可以處理在待機時傳送的組播資料包 |
| 思科錯誤ID CSCvk44688 | asr9k-fia | FPGA多次出錯，無法恢復 |
| 思科錯誤ID CSCvy31670 | asr9k-ls-fia | LSP：刪除FC0將啟用交換矩陣速率限制器，FC4則不會 |
| 思科錯誤ID CSCvt59803 | asr9k-ls-npdriver | LSP:PLATFORM-NP-4-SHUTDOWN IMRHIMREG.ch1Psch1HbmReadParticleError |
行為 fault-manager datapath port shutdown/toggle 指令
- 其
fault-manager datapath port shutdown命令有助於在活動RP/RSP上關閉已為其設定Punt Datapath Failure警報的各個FIA/NP的埠，並且直到您重新載入LC後，介面才會自動啟動。此CLI命令無法按7. x.x版本中的預期工作。(CLI命令fault-manager datapath port shutdown不按照7.x.x)中的設計工作 — 已在7.7.2中修復。 - 其
fault-manager datapath port toggleCLI命令工作正常。清除Punt Datapath Failure警報後，它會開啟埠。 - 如果冗餘路徑上有適當的鏈路級冗餘和BW可用性，這有助於防止服務中斷。
測試 — 驗證前面提到的命令操作。
在NP0 LC7上引發PUNT錯誤生成：
RP/0/RP0/CPU0:ASR-9922-A#monitor np counter PUNT_DIAGS_RSP_ACT np0 count 20 location 0/7/CPU0
Wed Jul 7 14:15:17.489 UTC
Usage of NP monitor is recommended for cisco internal use only.
Please use instead 'show controllers np capture' for troubleshooting packet drops in NP
and 'monitor np interface' for per (sub)interface counter monitoring
Warning: Every packet captured will be dropped! If you use the 'count'
option to capture multiple protocol packets, this could disrupt
protocol sessions (eg, OSPF session flap). So if capturing protocol
packets, capture only 1 at a time.
Warning: A mandatory NP reset will be done after monitor to clean up.
This will cause ~150ms traffic outage. Links will stay Up.
Proceed y/n [y] > y
Monitor PUNT_DIAGS_RSP_ACT on NP0 ... (Ctrl-C to quit)
Wed Jul 7 14:17:08 2021 -- NP0 packet
From Fabric: 127 byte packet
0000: 00 09 00 00 b4 22 00 00 ff ff ff ff 00 00 ff ff ....4"..........
0010: 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc aa aa aa aa ....ppppLLLL****
0020: 55 55 55 55 00 00 00 00 01 00 00 00 00 00 00 00 UUUU............
0030: 00 00 00 00 ff ff ff ff 00 00 ff ff 00 ff 00 ff ................
0040: f0 f0 f0 f0 cc cc cc cc aa aa aa aa 55 55 55 55 ppppLLLL****UUUU
0050: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0060: ff ff ff ff 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 ............pppp
0070: cc cc cc cc aa aa aa aa 55 55 55 55 00 00 00 LLLL****UUUU...
(count 1 of 20)
Wed Jul 7 14:18:09 2021 -- NP0 packet
From Fabric: 256 byte packet
0000: 00 09 00 00 b5 22 00 00 ff ff ff ff 00 00 ff ff ....5"..........
0010: 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc aa aa aa aa ....ppppLLLL****
0020: 55 55 55 55 00 00 00 00 01 00 00 00 00 00 00 00 UUUU............
0030: 00 00 00 00 ff ff ff ff 00 00 ff ff 00 ff 00 ff ................
0040: f0 f0 f0 f0 cc cc cc cc aa aa aa aa 55 55 55 55 ppppLLLL****UUUU
0050: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0060: ff ff ff ff 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 ............pppp
0070: cc cc cc cc aa aa aa aa 55 55 55 55 00 00 00 00 LLLL****UUUU....
0080: 01 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff ................
0090: 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc ........ppppLLLL
00a0: aa aa aa aa 55 55 55 55 00 00 00 00 01 00 00 00 ****UUUU........
00b0: 00 00 00 00 00 00 00 00 ff ff ff ff 00 00 ff ff ................
00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
(count 2 of 20)
Wed Jul 7 14:19:09 2021 -- NP0 packet
Actual packet size 515 bytes truncated size 384:
From Fabric: 384 byte packet
0000: 00 09 00 00 b6 22 00 00 ff ff ff ff 00 00 ff ff ....6"..........
0010: 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc aa aa aa aa ....ppppLLLL****
0020: 55 55 55 55 00 00 00 00 01 00 00 00 00 00 00 00 UUUU............
0030: 00 00 00 00 ff ff ff ff 00 00 ff ff 00 ff 00 ff ................
0040: f0 f0 f0 f0 cc cc cc cc aa aa aa aa 55 55 55 55 ppppLLLL****UUUU
0050: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0060: ff ff ff ff 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 ............pppp
0070: cc cc cc cc aa aa aa aa 55 55 55 55 00 00 00 00 LLLL****UUUU....
0080: 01 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff ................
0090: 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc ........ppppLLLL
00a0: aa aa aa aa 55 55 55 55 00 00 00 00 01 00 00 00 ****UUUU........
00b0: 00 00 00 00 00 00 00 00 ff ff ff ff 00 00 ff ff ................
00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
RP/0/RP0/CPU0:ASR-9922-A#sh pfm location 0/RP0/CPU0
Wed Jul 7 14:19:17.174 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 14:19:17 2021
PFM TOTAL: 2 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 1
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Jul 1 10:13:45 2021|0 |SPINE_UNAVAILABLE |E/A|5082 |Fabric Manager|0x1034000
Jul 7 14:19:09 2021|0 |PUNT_FABRIC_DATA_PATH_FAILED |ER |9429 |System Punt/Fa|0x2000004
RP/0/RP0/CPU0:ASR-9922-A#sh pfm process 9429 location 0/Rp0/CPU0
Wed Jul 7 14:19:37.128 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 14:19:37 2021
PFM TOTAL: 2 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 1
PER PROCESS TOTAL: 0 EM: 0 CR: 0 ER: 0
Device/Path[1 ]:Fabric loopbac [0x2000003 ] State:RDY Tot: 0
Device/Path[2 ]:System Punt/Fa [0x2000004 ] State:RDY Tot: 1
1 Fault Id: 432
Sev: ER
Fault Name: PUNT_FABRIC_DATA_PATH_FAILED
Raised Timestamp: Jul 7 14:19:09 2021
Clear Timestamp: N/A
Changed Timestamp: N/A
Resync Mismatch: FALSE
MSG: failure threshold is 3, (slot, NP) failed: (0/7/CPU0, 0)
Device/Path[3 ]:Crossbar Switc [0x108c000 ] State:RDY Tot: 0
Device/Path[4 ]:Crossbar Switc [0x108c001 ] State:RDY Tot: 0
Device/Path[5 ]:Crossbar Switc [0x108c002 ] State:RDY Tot: 0
Device/Path[6 ]:Crossbar Switc [0x108c003 ] State:RDY Tot: 0
Device/Path[7 ]:Crossbar Switc [0x108c004 ] State:RDY Tot: 0
Device/Path[8 ]:Crossbar Switc [0x108c005 ] State:RDY Tot: 0
Device/Path[9 ]:Crossbar Switc [0x108c006 ] State:RDY Tot: 0
Device/Path[10]:Crossbar Switc [0x108c007 ] State:RDY Tot: 0
Device/Path[11]:Crossbar Switc [0x108c008 ] State:RDY Tot: 0
Device/Path[12]:Crossbar Switc [0x108c009 ] State:RDY Tot: 0
Device/Path[13]:Crossbar Switc [0x108c00a ] State:RDY Tot: 0
Device/Path[14]:Crossbar Switc [0x108c00b ] State:RDY Tot: 0
Device/Path[15]:Crossbar Switc [0x108c00c ] State:RDY Tot: 0
Device/Path[16]:Crossbar Switc [0x108c00d ] State:RDY Tot: 0
Device/Path[17]:Crossbar Switc [0x108c00e ] State:RDY Tot: 0
Device/Path[18]:Fabric Interfa [0x108b000 ] State:RDY Tot: 0
Device/Path[19]:Fabric Arbiter [0x1086000 ] State:RDY Tot: 0
Device/Path[20]:CPU Controller [0x108d000 ] State:RDY Tot: 0
Device/Path[21]:Device Control [0x109a000 ] State:RDY Tot: 0
Device/Path[22]:ClkCtrl Contro [0x109b000 ] State:RDY Tot: 0
Device/Path[23]:NVRAM [0x10ba000 ] State:RDY Tot: 0
Device/Path[24]:Hooper switch [0x1097000 ] State:RDY Tot: 0
Device/Path[25]:Hooper switch [0x1097001 ] State:RDY Tot: 0
Device/Path[26]:Hooper switch [0x1097002 ] State:RDY Tot: 0
Device/Path[27]:Hooper switch [0x1097003 ] State:RDY Tot: 0
在此案例中，連線埠沒有關閉：
RP/0/RP0/CPU0:ASR-9922-A#sh ipv4 int brief location 0/7/CPU0
Wed Jul 7 14:21:29.693 UTC
Interface IP-Address Status Protocol Vrf-Name
TenGigE0/7/0/0 unassigned Down Down default
TenGigE0/7/0/1 unassigned Down Down default
TenGigE0/7/0/2 unassigned Down Down default
TenGigE0/7/0/3 unassigned Down Down default
TenGigE0/7/0/4 unassigned Down Down default
TenGigE0/7/0/5 unassigned Down Down default
TenGigE0/7/0/6 unassigned Down Down default
TenGigE0/7/0/7 unassigned Shutdown Down default
TenGigE0/7/0/8 unassigned Shutdown Down default
TenGigE0/7/0/9 unassigned Shutdown Down default
TenGigE0/7/0/10 unassigned Down Down default
TenGigE0/7/0/11 unassigned Down Down default
TenGigE0/7/0/12 unassigned Down Down default
TenGigE0/7/0/13 unassigned Shutdown Down default
TenGigE0/7/0/14 unassigned Shutdown Down default
TenGigE0/7/0/15 unassigned Shutdown Down default
TenGigE0/7/0/16 unassigned Shutdown Down default
TenGigE0/7/0/17 unassigned Shutdown Down default
TenGigE0/7/0/18 unassigned Down Down default
TenGigE0/7/0/19 unassigned Up Up default >>>>>>> Port is UP
RP/0/RP0/CPU0:ASR-9922-A#sh logging last 200 | in 0/7/0
Wed Jul 7 14:22:35.715 UTC
RP/0/RP0/CPU0:ASR-9922-A#
測試案例1.2:
NP/埠行為 fault-manager datapath port toggle 指令:
RP/0/RP0/CPU0:ASR-9922-A#sh run formal | in data
Wed Jul 7 14:52:11.714 UTC
Building configuration...
fault-manager datapath port toggle
RP/0/RP0/CPU0:ASR-9922-A#
PFM中無警報：
RP/0/RP0/CPU0:ASR-9922-A#sh pfm location 0/Rp0/CPU0
Wed Jul 7 14:55:13.410 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 14:55:13 2021
PFM TOTAL: 1 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Jul 1 10:13:45 2021|0 |SPINE_UNAVAILABLE |E/A|5082 |Fabric Manager|0x1034000
RP/0/RP0/CPU0:ASR-9922-A#
在NP0 LC7中生成PUNT錯誤：
RP/0/RP0/CPU0:ASR-9922-A#monitor np counter PUNT_DIAGS_RSP_ACT np0 count 20 location 0/7/CPU0
Wed Jul 7 14:51:18.596 UTC
Usage of NP monitor is recommended for cisco internal use only.
Please use instead 'show controllers np capture' for troubleshooting packet drops in NP
and 'monitor np interface' for per (sub)interface counter monitoring
Warning: Every packet captured will be dropped! If you use the 'count'
option to capture multiple protocol packets, this could disrupt
protocol sessions (eg, OSPF session flap). So if capturing protocol
packets, capture only 1 at a time.
Warning: A mandatory NP reset will be done after monitor to clean up.
This will cause ~150ms traffic outage. Links will stay Up.
Proceed y/n [y] > y
Monitor PUNT_DIAGS_RSP_ACT on NP0 ... (Ctrl-C to quit)
Wed Jul 7 14:53:21 2021 -- NP0 packet
From Fabric: 127 byte packet
0000: 00 09 00 00 d8 22 00 00 ff ff ff ff 00 00 ff ff ....X"..........
0010: 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc aa aa aa aa ....ppppLLLL****
0020: 55 55 55 55 00 00 00 00 01 00 00 00 00 00 00 00 UUUU............
0030: 00 00 00 00 ff ff ff ff 00 00 ff ff 00 ff 00 ff ................
0040: f0 f0 f0 f0 cc cc cc cc aa aa aa aa 55 55 55 55 ppppLLLL****UUUU
0050: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0060: ff ff ff ff 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 ............pppp
0070: cc cc cc cc aa aa aa aa 55 55 55 55 00 00 00 LLLL****UUUU...
(count 1 of 20)
Wed Jul 7 14:54:22 2021 -- NP0 packet
From Fabric: 256 byte packet
0000: 00 09 00 00 d9 22 00 00 ff ff ff ff 00 00 ff ff ....Y"..........
0010: 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc aa aa aa aa ....ppppLLLL****
0020: 55 55 55 55 00 00 00 00 01 00 00 00 00 00 00 00 UUUU............
0030: 00 00 00 00 ff ff ff ff 00 00 ff ff 00 ff 00 ff ................
0040: f0 f0 f0 f0 cc cc cc cc aa aa aa aa 55 55 55 55 ppppLLLL****UUUU
0050: 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ................
0060: ff ff ff ff 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 ............pppp
0070: cc cc cc cc aa aa aa aa 55 55 55 55 00 00 00 00 LLLL****UUUU....
0080: 01 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff ................
0090: 00 00 ff ff 00 ff 00 ff f0 f0 f0 f0 cc cc cc cc ........ppppLLLL
00a0: aa aa aa aa 55 55 55 55 00 00 00 00 01 00 00 00 ****UUUU........
00b0: 00 00 00 00 00 00 00 00 ff ff ff ff 00 00 ff ff ................
00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
(count 2 of 20)
RP/0/RP0/CPU0:ASR-9922-A#sh pfm location 0/Rp0/CPU0
Wed Jul 7 14:56:24.459 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 14:56:24 2021
PFM TOTAL: 2 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 1
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Jul 1 10:13:45 2021|0 |SPINE_UNAVAILABLE |E/A|5082 |Fabric Manager|0x1034000
Jul 7 14:55:23 2021|0 |PUNT_FABRIC_DATA_PATH_FAILED |ER |9429 |System Punt/Fa|0x2000004
RP/0/RP0/CPU0:ASR-9922-A#sh pfm process 9429 location 0/RP0/CPU0
Wed Jul 7 14:56:39.961 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 14:56:40 2021
PFM TOTAL: 2 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 1
PER PROCESS TOTAL: 0 EM: 0 CR: 0 ER: 0
Device/Path[1 ]:Fabric loopbac [0x2000003 ] State:RDY Tot: 0
Device/Path[2 ]:System Punt/Fa [0x2000004 ] State:RDY Tot: 1
1 Fault Id: 432
Sev: ER
Fault Name: PUNT_FABRIC_DATA_PATH_FAILED
Raised Timestamp: Jul 7 14:55:23 2021
Clear Timestamp: N/A
Changed Timestamp: N/A
Resync Mismatch: FALSE
MSG: failure threshold is 3, (slot, NP) failed: (0/7/CPU0, 0)
Device/Path[3 ]:Crossbar Switc [0x108c000 ] State:RDY Tot: 0
Device/Path[4 ]:Crossbar Switc [0x108c001 ] State:RDY Tot: 0
Device/Path[5 ]:Crossbar Switc [0x108c002 ] State:RDY Tot: 0
Device/Path[6 ]:Crossbar Switc [0x108c003 ] State:RDY Tot: 0
Device/Path[7 ]:Crossbar Switc [0x108c004 ] State:RDY Tot: 0
Device/Path[8 ]:Crossbar Switc [0x108c005 ] State:RDY Tot: 0
Device/Path[9 ]:Crossbar Switc [0x108c006 ] State:RDY Tot: 0
Device/Path[10]:Crossbar Switc [0x108c007 ] State:RDY Tot: 0
Device/Path[11]:Crossbar Switc [0x108c008 ] State:RDY Tot: 0
Device/Path[12]:Crossbar Switc [0x108c009 ] State:RDY Tot: 0
Device/Path[13]:Crossbar Switc [0x108c00a ] State:RDY Tot: 0
Device/Path[14]:Crossbar Switc [0x108c00b ] State:RDY Tot: 0
Device/Path[15]:Crossbar Switc [0x108c00c ] State:RDY Tot: 0
Device/Path[16]:Crossbar Switc [0x108c00d ] State:RDY Tot: 0
Device/Path[17]:Crossbar Switc [0x108c00e ] State:RDY Tot: 0
Device/Path[18]:Fabric Interfa [0x108b000 ] State:RDY Tot: 0
Device/Path[19]:Fabric Arbiter [0x1086000 ] State:RDY Tot: 0
Device/Path[20]:CPU Controller [0x108d000 ] State:RDY Tot: 0
Device/Path[21]:Device Control [0x109a000 ] State:RDY Tot: 0
Device/Path[22]:ClkCtrl Contro [0x109b000 ] State:RDY Tot: 0
Device/Path[23]:NVRAM [0x10ba000 ] State:RDY Tot: 0
Device/Path[24]:Hooper switch [0x1097000 ] State:RDY Tot: 0
Device/Path[25]:Hooper switch [0x1097001 ] State:RDY Tot: 0
Device/Path[26]:Hooper switch [0x1097002 ] State:RDY Tot: 0
Device/Path[27]:Hooper switch [0x1097003 ] State:RDY Tot: 0
介面TenGigE0/7/0/19關閉NP0:
RP/0/RP0/CPU0:ASR-9922-A#show logging last 200 | in 0/7/0
Wed Jul 7 14:58:42.959 UTC
LC/0/7/CPU0:Jul 7 14:55:23.798 UTC: ifmgr[270]: %PKT_INFRA-LINK-3-UPDOWN :
Interface TenGigE0/7/0/19, changed state to Down
LC/0/7/CPU0:Jul 7 14:55:23.798 UTC: ifmgr[270]: %PKT_INFRA-LINEPROTO-5-UPDOWN :
Line protocol on Interface TenGigE0/7/0/19, changed state to Down
RP/0/RP0/CPU0:Jul 7 14:55:23.802 UTC: BM-DISTRIB[1290]: %L2-BM-6-ACTIVE :
TenGigE0/7/0/19 is no longer Active as part of Bundle-Ether854 (Link is down)
LC/0/7/CPU0:Jul 7 14:55:25.854 UTC: vic_0_0[379]: %PLATFORM-VIC-4-RFI :
Interface TenGigE0/7/0/19, Detected Remote Fault
LC/0/7/CPU0:Jul 7 14:55:26.936 UTC: lda_server[114]: %PKT_INFRA-FM-2-FAULT_CRITICAL :
ALARM_CRITICAL :OPTICS RX POWER LANE-0 LOW ALARM :CLEAR : Te0/7/0/0:
已停止生成PUNT錯誤：
RP/0/RP0/CPU0:ASR-9922-A#sh ipv4 int brief location 0/7/CPU0
Wed Jul 7 14:59:16.322 UTC
Interface IP-Address Status Protocol Vrf-Name
TenGigE0/7/0/0 unassigned Down Down default
TenGigE0/7/0/1 unassigned Down Down default
TenGigE0/7/0/2 unassigned Down Down default
TenGigE0/7/0/3 unassigned Down Down default
TenGigE0/7/0/4 unassigned Down Down default
TenGigE0/7/0/5 unassigned Down Down default
TenGigE0/7/0/6 unassigned Down Down default
TenGigE0/7/0/7 unassigned Shutdown Down default
TenGigE0/7/0/8 unassigned Shutdown Down default
TenGigE0/7/0/9 unassigned Shutdown Down default
TenGigE0/7/0/10 unassigned Down Down default
TenGigE0/7/0/11 unassigned Down Down default
TenGigE0/7/0/12 unassigned Down Down default
TenGigE0/7/0/13 unassigned Shutdown Down default
TenGigE0/7/0/14 unassigned Shutdown Down default
TenGigE0/7/0/15 unassigned Shutdown Down default
TenGigE0/7/0/16 unassigned Shutdown Down default
TenGigE0/7/0/17 unassigned Shutdown Down default
TenGigE0/7/0/18 unassigned Down Down default
TenGigE0/7/0/19 unassigned Down Down default >>>>>>
警報已停止：
RP/0/RP0/CPU0:ASR-9922-A#sh pfm location 0/Rp0/CPU0
Wed Jul 7 15:01:44.478 UTC
node: node0_RP0_CPU0
---------------------
CURRENT TIME: Jul 7 15:01:44 2021
PFM TOTAL: 1 EMERGENCY/ALERT(E/A): 1 CRITICAL(CR): 0 ERROR(ER): 0
-------------------------------------------------------------------------------------------------
Raised Time |S#|Fault Name |Sev|Proc_ID|Dev/Path Name |Handle
--------------------+--+-----------------------------------+---+-------+--------------+----------
Jul 1 10:13:45 2021|0 |SPINE_UNAVAILABLE |E/A|5082 |Fabric Manager|0x1034000
RP/0/RP0/CPU0:ASR-9922-A#
介面已啟動：
RP/0/RP0/CPU0:ASR-9922-A#show logging | in 0/7/0/19
Wed Jul 7 15:06:11.532 UTC
LC/0/7/CPU0:Jul 7 14:55:23.798 UTC: ifmgr[270]: %PKT_INFRA-LINK-3-UPDOWN :
Interface TenGigE0/7/0/19, changed state to Down
LC/0/7/CPU0:Jul 7 14:55:23.798 UTC: ifmgr[270]: %PKT_INFRA-LINEPROTO-5-UPDOWN :
Line protocol on Interface TenGigE0/7/0/19, changed state to Down
RP/0/RP0/CPU0:Jul 7 14:55:23.802 UTC: BM-DISTRIB[1290]: %L2-BM-6-ACTIVE :
TenGigE0/7/0/19 is no longer Active as part of Bundle-Ether854 (Link is down)
LC/0/7/CPU0:Jul 7 14:55:25.854 UTC: vic_0_0[379]: %PLATFORM-VIC-4-RFI :
Interface TenGigE0/7/0/19, Detected Remote Fault
LC/0/7/CPU0:Jul 7 15:03:27.204 UTC: ifmgr[270]: %PKT_INFRA-LINK-3-UPDOWN :
Interface TenGigE0/7/0/19, changed state to Up
LC/0/7/CPU0:Jul 7 15:03:27.206 UTC: ifmgr[270]: %PKT_INFRA-LINEPROTO-5-UPDOWN :
Line protocol on Interface TenGigE0/7/0/19, changed state to Up
RP/0/RP0/CPU0:Jul 7 15:03:29.219 UTC: BM-DISTRIB[1290]: %L2-BM-6-ACTIVE :
TenGigE0/7/0/19 is Active as part of Bundle-Ether854
修訂記錄
| 修訂 | 發佈日期 | 意見 |
|---|---|---|
1.0 |
09-Jan-2023 |
初始版本 |
意見