排除5G SMI協定虛擬機器時間同步問題

簡介

本文描述如何在5G使用者微服務基礎架構(SMI)協定虛擬機器（協定虛擬機器）中實施網路時間協定(NTP)時間同步問題的解決方法。

必要條件

需求

思科建議您瞭解以下主題：

• Cisco SMI
• 5G雲原生部署平台(CNDP)架構
• 雲端 — Red Hat OpenStack Platform 13（「Queens」版本）
• 道克斯和庫伯內特

採用元件

本文中的資訊係根據以下軟體和硬體版本：

• SMI 2020.01.1-21
• Kubernetes v1.16.2
• Red Hat OpenStack Platform 13導向器

本文中的資訊是根據特定實驗室環境內的裝置所建立。文中使用到的所有裝置皆從已清除（預設）的組態來啟動。如果您的網路運作中，請確保您瞭解任何指令可能造成的影響。

背景資訊

什麼是SMI?

Cisco SMI是雲技術和標準的分層堆疊，支援來自思科移動性、電纜和寬頻網路網關(BNG)業務部門的基於微服務的應用。這些應用程式具有相似的使用者管理功能和相似的資料儲存要求。

屬性：

• 層雲堆疊（技術和標準）提供自上而下的部署，並適應當前的雲基礎設施。
• 所有應用程式共用通用執行環境(CEE)以用於非應用程式功能（資料儲存、部署、配置、遙測、警報），從而為所有客戶觸控點和整合點提供一致的互動和體驗。
• 應用和CEE部署在微服務容器中，並與智慧服務網狀網連線。
• 用於部署、配置和管理的API可實現自動化。

什麼是CEE?

• CEE是一種軟體解決方案，開發目的是監控部署在SMI上的移動和電纜應用程式。CEE以集中方式從應用程式捕獲資訊（關鍵指標），以便工程師進行調試和故障排除。
• CEE是為所有應用程式安裝的常用工具集。它配備了專用的運營中心，可提供命令列介面(CLI)和API來管理監控工具。每個集群只能有一個CEE。

什麼是協定VM?

協定VM承載負責協定轉換的會話管理功能(SMF)應用微服務。每個SMF例項部署兩個協定虛擬機器。協定VM具有GTPc、合法攔截(LI)、Radius、Rest等。

問題

從顯示問題的節點（協定VM）中，您可以看到網路嘗試使用兩個不同的介面將連線路由到NTP伺服器。但是，只有一個網路可以到達IP。

在沒有介面的情況下執行ping操作一段時間後，可以看到這會導致封包遺失。

來自CEE的警報示例：

[pod-name-cnat/global] cee# show alerts active summary
NAME UID SEVERITY STARTS AT SOURCE SUMMARY
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
VoNR_Ded_Bearer_Creat be183f61dd65 major 01-18T16:32:17 System Success percentage of VoNR - PCF Initiated Dedicated Bearer Creation procedure in NR turns less than t...
clock-is-not-in-synch 38a35e9a8bd8 major 01-18T13:11:15 pod-name-cnat-cnat-co Clock not in synch detected on hostname pod-name-cnat-cnat-core-protocol-data1 . Ensure NTP is configu...
clock-is-not-in-synch 4a7e138b8bae major 01-18T13:07:35 pod-name-cnat-cnat-co Clock not in synch detected on hostname pod-name-cnat-cnat-core-protocol-ims2 . Ensure NTP is configur...
clock-is-not-in-synch 5b3e128f0101 major 01-18T12:05:55 pod-name-cnat-cnat-co Clock not in synch detected on hostname pod-name-cnat-cnat-core-protocol-data2 . Ensure NTP is configu...
container-memory-usag f588aa627792 critical 12-28T15:47:30 path-provisioner-v6zx Pod cee-global/path-provisioner-v6zxj/k8s_path-provisioner_path-provisioner-v6zxj_cee-global_f1a1e8fa-...
container-memory-usag 55b2ea84466f critical 12-27T22:27:10 path-provisioner-v6zx Pod cee-global/path-provisioner-v6zxj/ uses high memory 82.69%.
N11_SM_Timeout_SR f6fb82cac197 major 10-31T15:51:16 System This alert is fired when the increase in timeout for N11 messages toward AMF crosses threshold
Radius_Server_RTT 94ac2cff43a3 warning 10-28T05:08:56 System RTT for Radius Server: 216.x.x.x, Port: 1813 in namespace: smf-mvno is more than 5 ms.
Radius_Server_RTT d02b60b3a3a4 warning 10-28T05:05:56 System RTT for Radius Server: 52.x.x.x, Port: 1813 in namespace: smf-mvno is more than 5 ms.
Radius_Server_RTT 9afcee101013 warning 10-28T05:05:36 System RTT for Radius Server: 52.x.x.x, Port: 1813 in namespace: smf-mvno is more than 5 ms.
N11_SM_Timeout_SR 206c4bbccf21 major 10-20T13:26:16 System This alert is fired when the increase in timeout for N11 messages toward AMF crosses threshold
N4_Total_Outbound_Int 4e0f3c1d6300 major 10-20T09:01:23 System This alert is fired when the percentage of N4 outbound responses sent is lesser than threshold %.
N4_Total_Outbound_Int 3e2fed624704 major 10-20T08:21:23 System This alert is fired when the percentage of N4 outbound responses sent is lesser than threshold %.
watchdog 97a7976a4103 minor 05-05T09:10:58 System This is an alert meant to ensure that the entire alerting pipeline is functional. This alert is always...


疑難排解

1. 登入到受影響的VM並運行此命令： ip route | grep -i default

   在本例中，您可以看到兩個不正確的預設路由。（應該只有一個預設路由。）

   ubuntu@pod-name-cnat-cnat-core-protocol-ims2:~$ ip route | grep -i default
   default via 172.16.x.x dev ens4 proto dhcp src 172.16.x.x metric 100
   default via 172.16.x.x dev ens3 proto dhcp src 172.16.x.x metric 100
   

2. 檢查計時(NTP)服務的狀態。即使該服務已啟動，您仍會看到「無法同步」錯誤。

   輸出範例:

   root@pod-name-cnat-cnat-core-protocol-data1:/home/ubuntu# systemctl status chronyd.service
chrony.service - chrony, an NTP client/serverLoaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2021-01-08 07:19:19 UTC; 11 months 24 days ago
     Docs: man:chronyd(8)
           man:chronyc(1)
           man:chrony.conf(5)
 Main PID: 4300 (chronyd)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/chrony.service
           └─4300 /usr/sbin/chronyd
Dec 30 17:06:28 pod-name-cnat-cnat-core-protocol-data1 chronyd[4300]: Selected source 5.196.x.x
Dec 31 06:37:11 pod-name-cnat-cnat-core-protocol-data1 chronyd[4300]: Can't synchronise: no selectable sources
Dec 31 17:15:31 pod-name-cnat-cnat-core-protocol-data1 chronyd[4300]: Selected source 5.196.x.x
Jan 01 06:29:43 pod-name-cnat-cnat-core-protocol-data1 chronyd[4300]: Can't synchronise: no selectable sources
Jan 01 16:50:02 pod-name-cnat-cnat-core-protocol-data1 chronyd[4300]: Selected source 5.196.x.x
Jan 01 19:25:05 pod-name-cnat-cnat-core-protocol-data1 chronyd[4300]: Can't synchronise: no selectable sources


3. 檢查network-config服務的狀態。（它處於非活動狀態。）

   ubuntu@pod-name-cnat-cnat-core-protocol-data1:~$ systemctl status network-config
   network-config.service - Job that configures for routes
   Loaded: loaded (/etc/systemd/system/network-config.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Fri 2021-01-08 09:29:05 UTC; 1 years 0 months ago
   Process: 1185 ExecStart=/etc/cisco/network-config.sh start (code=killed, signal=TERM)
   Main PID: 1185 (code=killed, signal=TERM)

因應措施

附註：此過程不會導致應用程式發生任何停機。

1. 在受影響的節點中編輯network-config service script(/etc/systemd/system/network-config.service)以新增以下行：

   root@pod-name-cnat-cnat-core-protocol-data1:/home/ubuntu# vim /etc/systemd/system/network-config.service
   Restart=always
   RestartSec=10s
   StartLimitIntervalSec=300s
   StartLimitBurst=30

2. 啟動network-config服務：

   root@pod-name-cnat-cnat-core-protocol-data1:/home/ubuntu# systemctl start network-config

3. 使用systemctl重新載入配置：

   root@pod-name-cnat-cnat-core-protocol-data1:/home/ubuntu# systemctl daemon-reload

4. 重新啟動計時服務：

   root@pod-name-cnat-cnat-core-protocol-data1:/home/ubuntu# systemctl restart chronyd.service

驗證

完成解決方法步驟後，請運行以下命令以驗證問題是否已修復：

for server in $(awk '/protocol/{print $1}' /etc/hosts); do ssh $server "hostname && timedatectl status && chronyc sources -vn" ; done

附註：「System clock synchronized」值為「yes」表示問題已解決。

輸出範例:

ubuntu@pod-name-cnat-cnat-core-master1:~$ for server in $(awk '/protocol/{print $1}' /etc/hosts); do ssh $server "hostname && timedatectl status && chronyc sources -vn" ; done

pod-name-cnat-cnat-core-protocol-data1
                      Local time: Tue 2021-09-07 10:22:16 UTC
                  Universal time: Tue 2021-09-07 10:22:16 UTC
                        RTC time: Tue 2021-09-07 10:22:17
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: no
                 RTC in local TZ: no
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* svtsnq01.mgmt                 1   6   170   282  -1100us[-1858us] +/-   47ms

pod-name-cnat-cnat-core-protocol-data2
                      Local time: Tue 2021-09-07 10:22:16 UTC
                  Universal time: Tue 2021-09-07 10:22:16 UTC
                        RTC time: Tue 2021-09-07 10:22:17
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: no
                 RTC in local TZ: no
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* svtsnq01.mgmt                 1   6    74   206   -218us[-1550us] +/-   46ms

pod-name-cnat-cnat-core-protocol-ims1
                      Local time: Tue 2021-09-07 10:22:17 UTC
                  Universal time: Tue 2021-09-07 10:22:17 UTC
                        RTC time: Tue 2021-09-07 10:22:18
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: no
                 RTC in local TZ: no
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* svtsnq01.mgmt                 1   6    77    24   +796us[+1044us] +/-   47ms

pod-name-cnat-cnat-core-protocol-ims2
                      Local time: Tue 2021-09-07 10:22:17 UTC
                  Universal time: Tue 2021-09-07 10:22:17 UTC
                        RTC time: Tue 2021-09-07 10:22:18
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: no
                 RTC in local TZ: no
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* svtsnq01.mgmt                 1   6    17    98  +2176us[+2275us] +/-   47ms