排除RCM聚合核心的切換問題
目錄
簡介
本文說明在發生網路故障事件時，對冗餘配置管理器(RCM)進行故障排除的基本步驟。
背景資訊
什麼是RCM?
RCM是思科專有節點或網路功能(NF)，可為基於StarOS的使用者平面功能(UPF)提供冗餘。
RCM提供UPF的N:M冗餘，其中N是活動UPF的數量且小於10,M是冗餘組中的備用UP的數量。
RCM的元件
RCM包括作為RCM VM中的Pod運行的元件：
- 控制器：它與RCM中的所有其他Pod通訊特定於事件的決策
- BFD管理器(BFDMgr):它使用BFD協定來識別資料平面的狀態
- Configuration Manager(ConfigMgr):它將請求的配置載入到使用者平面(UP)
- 冗餘管理器(RedMgr):也稱為檢查點管理器。它儲存檢查點資料並將其傳送到備用UPF
- Keepalive:它使用VRRP在活動和備用RCM之間通訊
典型RCM部署模式
RCM CLI概述
在本示例中，有四個RCM OPS中心。為了確認RCM Kubernetes與哪個RCM OPS Center和RCM Common Execution Environment(CEE)對應，您可以登入到RCM Kubernetes並列出名稱空間：
cloud-user@up0300-aio-1-primary-1:~$ kubectl get namespace
NAME STATUS AGE
cee-rce31 Active 54d
default Active 57d
istio-system Active 57d
kube-node-lease Active 57d
kube-public Active 57d
kube-system Active 57d
nginx-ingress Active 57d
rcm-rm31 Active 54d
rcm-rm33 Active 54d
registry Active 57d
smi-certs Active 57d
smi-node-label Active 57d
smi-vips Active 57d
cloud-user@up300-aio-2-primary-1:~$ kubectl get namespace
NAME STATUS AGE
cee-rce32 Active 54d
default Active 57d
istio-system Active 57d
kube-node-lease Active 57d
kube-public Active 57d
kube-system Active 57d
nginx-ingress Active 57d
rcm-rm32 Active 54d
rcm-rm34 Active 54d
registry Active 57d
smi-certs Active 57d
smi-node-label Active 57d
smi-vips Active 57d
UPF管理IP地址
此IP是特定的，並且與VM或UPF關聯。它用於UPF和RCM之間的初始通訊，其中UPF註冊到RCM，RCM配置UPF並分配角色。您可以使用此IP從RCM CLI輸出中標識UPF。
UPF裝置角色IP
連結到角色（活動/備用）：
此IP地址會隨著切換的發生而移動。
用於RCM故障排除的有用CLI命令
您可以從RCM OPS Center檢視哪個RCM組是UPF。從雲原生部署平台(CNDP)中查詢示例：
[local]UPF317# show rcm info
Redundancy Configuration Module:
-------------------------------------------------------------------------------
Context: rcm
Bind Address: 10.10.9.81
Chassis State: Active
Session State: SockActive
Route-Modifier: 32
RCM Controller Address: 10.10.9.179
RCM Controller Port: 9200
RCM Controller Connection State: Connected
Ready To Connect: Yes
Management IP Address: 10.10.14.33
Host ID: UPF320
SSH IP Address: 10.10.14.40 (Activated)
在這裡，您可以檢視RCM OPS Center的狀態：
[up300-aio-2/rm34] rcm# rcm show-status
message :
{"status":[" Thu Oct 21 10:45:21 UTC 2021 : State is primary"]}
[up300-aio-2/rm34] rcm# rcm show-statistics controller
message :
{
"keepalive_version": "65820a54450f930458c01e4049bd01f207bc6204e598f0ad3184c401174fd448",
"keepalive_timeout": "2s",
"num_groups": 2,
"groups": [
{
"groupid": 2,
"endpoints_configured": 7,
"standby_configured": 1,
"pause_switchover": false,
"active": 6,
"standby": 1,
"endpoints": [
{
"endpoint": "10.10.9.85",
"bfd_status": "STATE_UP",
"upf_registered": true,
"upf_connected": true,
"upf_state_received": "UpfMsgState_Active",
"bfd_state": "BFDState_UP",
"upf_state": "UPFState_Active",
"route_modifier": 32,
"pool_received": true,
"echo_received": 45359,
"management_ip": "10.10.14.41",
"host_id": "UPF322",
"ssh_ip": "10.10.14.44"
},
{
"endpoint": "10.10.9.86",
"bfd_status": "STATE_UP",
"upf_registered": true,
"upf_connected": true,
"upf_state_received": "UpfMsgState_Active",
"bfd_state": "BFDState_UP",
"upf_state": "UPFState_Active",
"route_modifier": 32,
"pool_received": true,
"echo_received": 4518,
"management_ip": "10.10.14.43",
"host_id": "UPF317",
"ssh_ip": "10.10.14.34"
},
{
"endpoint": "10.10.9.94",
"bfd_status": "STATE_UP",
"upf_registered": true,
"upf_connected": true,
"upf_state_received": "UpfMsgState_Active",
"bfd_state": "BFDState_UP",
"upf_state": "UPFState_Active",
"route_modifier": 32,
"pool_received": true,
"echo_received": 4518,
"management_ip": "10.10.14.59",
"host_id": "UPF318",
"ssh_ip": "10.10.14.36"
},
{
"endpoint": "10.10.9.81",
"bfd_status": "STATE_UP",
"upf_registered": true,
"upf_connected": true,
"upf_state_received": "UpfMsgState_Active",
"bfd_state": "BFDState_UP",
"upf_state": "UPFState_Active",
"route_modifier": 32,
"pool_received": true,
"echo_received": 45359,
"management_ip": "10.10.14.33",
"host_id": "UPF320",
"ssh_ip": "10.10.14.40"
},
{
"endpoint": "10.10.9.82",
"bfd_status": "STATE_UP",
"upf_registered": true,
"upf_connected": true,
"upf_state_received": "UpfMsgState_Standby",
"bfd_state": "BFDState_UP",
"upf_state": "UPFState_Standby",
"route_modifier": 50,
"pool_received": false,
"echo_received": 4505,
"management_ip": "10.10.14.35",
"host_id": "",
"ssh_ip": "10.10.14.60"
},
{
"endpoint": "10.10.9.83",
"bfd_status": "STATE_UP",
"upf_registered": true,
"upf_connected": true,
"upf_state_received": "UpfMsgState_Active",
"bfd_state": "BFDState_UP",
"upf_state": "UPFState_Active",
"route_modifier": 30,
"pool_received": true,
"echo_received": 4518,
"management_ip": "10.10.14.37",
"host_id": "UPF319",
"ssh_ip": "10.10.14.38"
},
{
"endpoint": "10.10.9.84",
"bfd_status": "STATE_UP",
"upf_registered": true,
"upf_connected": true,
"upf_state_received": "UpfMsgState_Active",
"bfd_state": "BFDState_UP",
"upf_state": "UPFState_Active",
"route_modifier": 32,
"pool_received": true,
"echo_received": 4518,
"management_ip": "10.10.14.39",
"host_id": "UPF321",
"ssh_ip": "10.10.14.42"
}
]
},
從RCM OPS Center確定當前備用UPF
在RCM OPS中，中心使用rcm show-statistics controller命令識別待機中的UPF:
{
"endpoint": "10.10.9.82",
"bfd_status": "STATE_UP",
"upf_registered": true,
"upf_connected": true,
"upf_state_received": "UpfMsgState_Standby",
"bfd_state": "BFDState_UP",
"upf_state": "UPFState_Standby",
"route_modifier": 50,
"pool_received": false,
"echo_received": 4505,
"management_ip": "10.10.14.35",
"host_id": "",
"ssh_ip": "10.10.14.60"
},
登入到UPF並檢查RCM資訊：
[local]UPF318# show rcm info
Saturday November 06 13:29:59 UTC 2021
Redundancy Configuration Module:
-------------------------------------------------------------------------------
Context: rcm
Bind Address: 10.10.9.82
Chassis State: Standby
Session State: SockStandby
Route-Modifier: 50
RCM Controller Address: 10.10.9.179
RCM Controller Port: 9200
RCM Controller Connection State: Connected
Ready To Connect: Yes
Management IP Address: 10.10.14.35
Host ID:
SSH IP Address: 10.10.14.60 (Activated)
以下是RCM OPS Center提供的其他有用資訊：
[up300-aio-2/rm34] rcm# rcm show-statistics
Possible completions:
bfdmgr Show RCM BFDMgr Statistics information
checkpointmgr Show RCM Checkpointmgr Statistics information
configmgr Show RCM Configmgr Statistics information
controller Show RCM Controller Statistics information
| Output modifiers
<cr>
下載21.24版的RCM指南。
CNDP POD上的RCM故障報告的問題
在與警報UP_SX_SESS_ESTABLISHMENT_SR有關的某個UPF上報告了此問題。此警報表示SX介面上的會話建立成功率低於配置的閾值。
如果您檢視Grafana統計資訊，則由於斷開原因pdn_sess_create而觀察到5G/4G效能下降 ||失敗 || upf_failure:
這確認pdn_sess_create ||失敗 || upf_failure由UPF419引起：
[local]UPF419# show rcm info
Saturday November 06 14:01:30 UTC 2021
Redundancy Configuration Module:
-------------------------------------------------------------------------------
Context: rcm
Bind Address: 10.10.11.83
Chassis State: Active
Session State: SockActive
Route-Modifier: 30
RCM Controller Address: 10.10.11.179
RCM Controller Port: 9200
RCM Controller Connection State: Connected
Ready To Connect: Yes
Management IP Address: 10.10.14.165
Host ID: DNUD0417
SSH IP Address: 10.10.14.162 (Activated)
在SMF上，您可以檢查UPF配置。在這種情況下，您必須查詢UPF N4 IP地址：
[smf/smf2] smf# show running-config profile network-element upf node-id n4-peer-UPF417
profile network-element upf upf19
node-id n4-peer-UPF417
n4-peer-address ipv4 10.10.10.17
n4-peer-port 8805
upf-group-profile upf-group1
dnn-list [ internet ]
capacity 10
priority 1
exit
然後，您可以執行Grafana查詢以確定出現最多故障的UPF N4地址：
Grafana查詢：
sum(increase(proto_udp_res_msg_total{namespace=~"$namespace", message_name="session_establishment_res", status="no_rsp_received_tx"} [15m]))，作者：message_name、status、peer_info
標籤：{{message_name}} || {{status}} || {{peer_info}}
格拉法納必須指出失敗發生的地點。在本示例中，它與UPF419相關。
連線到系統時，可以確認RCM切換後未正確設定sessmgr，因為許多會話管理器未處於預期的「Actv就緒」狀態。
[local]UPF419# show srp checkpoint statistics verbose
Tuesday November 02 17:24:01 UTC 2021
smgr state peer recovery pre-alloc chk-point rcvd chk-point sent
inst conn records calls full micro full micro
---- ------- ----- ------- -------- ----- ----- ----- ----
1 Actv Ready 0 0 1108 34001 14721 1200158
2 Actv Ready 0 0 1086 33879 17563 1347298
3 Actv Ready 0 0 1114 34491 15622 1222592
4 Actv Conn 0 0 5 923 0 0
5 Actv Ready 0 0 1106 34406 13872 1134403
6 Actv Conn 0 0 5 917 0 0
7 Actv Conn 0 0 5 920 0 0
8 Actv Conn 0 0 1 905 0 0
9 Actv Conn 0 0 5 916 0 0
10 Actv Conn 0 0 5 917 0 0
11 Actv Ready 0 0 1099 34442 13821 1167011
12 Actv Conn 0 0 5 916 0 0
13 Actv Conn 0 0 5 917 0 0
14 Actv Ready 0 0 1085 33831 13910 1162759
15 Actv Ready 0 0 1085 33360 13367 1081370
16 Actv Conn 0 0 4 921 0 0
17 Actv Ready 0 0 1100 35009 13789 1138089
18 Actv Ready 0 0 1092 33953 13980 1126028
19 Actv Conn 0 0 5 916 0 0
20 Actv Conn 0 0 5 918 0 0
21 Actv Ready 0 0 1098 33521 13636 1108875
22 Actv Ready 0 0 1090 34464 14529 1263419
解決方案
此修復程式與思科缺陷跟蹤系統(CDETS)CSCvz9749相關。此修復程式已整合到21.22.ua4.82694及更新版本中。
因應措施
在UPF419上，必須使用隱藏命令task kill facility sessmgr例項<>重新啟動不在Actv Ready中的會話管理器例項，這樣可以解決此問題。
[local]UPF419# show srp checkpoint statistics verbose
Wednesday November 03 16:44:57 UTC 2021
smgr state peer recovery pre-alloc chk-point rcvd chk-point sent
inst conn records calls full micro full micro
---- ------- ----- ------- -------- ----- ----- ----- ----
1 Actv Ready 0 0 1108 34001 38319 2267162
2 Actv Ready 0 0 1086 33879 40524 2428315
3 Actv Ready 0 0 1114 34491 39893 2335889
4 Actv Ready 0 0 0 0 12275 1049616
5 Actv Ready 0 0 1106 34406 37240 2172748
6 Actv Ready 0 0 0 0 13302 1040480
7 Actv Ready 0 0 0 0 12636 1062146
8 Actv Ready 0 0 0 0 11446 976169
9 Actv Ready 0 0 0 0 11647 972715
10 Actv Ready 0 0 0 0 11131 950436
11 Actv Ready 0 0 1099 34442 36696 2225847
12 Actv Ready 0 0 0 0 10739 919316
13 Actv Ready 0 0 0 0 11140 970384
14 Actv Ready 0 0 1085 33831 37206 2226049
15 Actv Ready 0 0 1085 33360 38135 2225816
16 Actv Ready 0 0 0 0 11159 946364
17 Actv Ready 0 0 1100 35009 37775 2242427
18 Actv Ready 0 0 1092 33953 37469 2181043
19 Actv Ready 0 0 0 0 13066 1055662
20 Actv Ready 0 0 0 0 10441 938350
21 Actv Ready 0 0 1098 33521 37238 2165185
22 Actv Ready 0 0 1090 34464 38227 2399415
UPF故障導致切換時收集日誌
RCM運營中心日誌記錄級別
logging level application debug logging level transaction debug logging level tracing off logging name infra.config.core level application warn logging name infra.config.core level transaction warn logging name infra.resource_monitor.core level application warn logging name infra.resource_monitor.core level transaction warn
逐步資料收集
- 問題摘要：問題陳述必須清晰。指示有問題的節點名稱/ip，以便更容易從日誌中查詢必要的資訊。例如，在出現切換問題時，如果提到IP x.x.x.x是源UPF，x.x.x.y是目標UPF，則會有所幫助。
- 如果有多種方法重現問題，請提到它們。
- RCM版本資訊：在從RCM VM部署RCM VM的情況下，cat/etc/smi/rcm-image-versionshow helm from the ops-center。在RCM CN部署中，從運營中心展現出領導力。
- RCM Tac在發生問題時調試CN或RCM日誌。在某些情況下，您也可以從剛出現POD時開始要求日誌。
- 指出哪個RCM是主用還是備用。對於CN，共用兩個RCM對的資訊。
- 從所有例項共用RCM ops-center中的運行配置。
- 收集RCM SNMP陷阱。
- 無論切換失敗與否，最好收集一個活動UP SSD和一個備用UP SSD。
- RCM controller 、 configmgr 、 checkpoint manager 、 switchover和switchover-verbose statistics命令用於提及確切的CLI。
rcm show-statistics controller
rcm show-statistics configmgr
rcm show-statistics checkpointmgr
rcm show-statistics switchover
rcm show-statistics switchover-verbose - UPF或RCM的系統日誌。
- 如果問題與切換故障有關，則需要新的活動UPF SSD和舊的UPF活動SSD。在某些情況下，舊活動因切換而重新啟動。在這種情況下，您必須重現問題，在此之前，您需要收集舊的活動式UP SSD。
- 在切換失敗的情況下，在問題重現時從新舊活動收集vpn、sessmgr、sess-gr和sxdemux調試日誌也非常有用。
logging filter active facility sxdemux level debug
logging filter active facility sessmgr level debug
logging filter active facility sess-gr level debug
logging filter active facility vpn level debug - 如果sessmgr/vpnmgr中存在錯誤/問題，則需要VPNMGR/Sessmgr核心。sessmgr_instance_id是發現問題的例項。vpnmgr_instance_id是RCM上下文的上下文編號。
任務核心裝置sessmgr例項<sessmgr_instance_id>
任務核心裝置vpnmgr例項<vpnmgr_instance_id> - 在發生RCM HA問題時，從兩個例項共用RCM TAC調試/pod日誌。
相關資訊
修訂記錄
| 修訂 | 發佈日期 | 意見 |
|---|---|---|
1.0 |
22-Jun-2022 |
初始版本 |
意見