Troubleshoot NTP on DNA Center

Introduction

This document describes how to troubleshoot Network Time Protocol (NTP) issues on Cisco DNA Center (DNAC).

Prerequisites

Requirements

• It is required that the user has Command Line Interface (CLI) access to the Cisco DNA Center.

• You must have maglev Secure Socket Shell (SSH) access privileges to perform this procedure.

  • Use maglev as the username on port  2222.

• NTP Server.
• Understand the NTP protocol.

Components Used

The information in this document is based on these software versions:

• Cisco DNA Center 2.3.3
• Cisco DNA Center 2.3.5

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Time synchronization is critical for data accuracy and the coordination of processing across a multi-host cluster. Before deploying the appliance in a production environment, make sure that the time on the appliance system clock is current and that the NTP servers you specified are keeping accurate time. If you are planning to integrate the appliance with ISE, you must also ensure that ISE is synchronizing with the same NTP servers as the appliance.

For a production deployment, it is recommended that you configure a minimum of three NTP servers.

NTP version 4 uses UDP port 123 for communication to and from the DNAC.

Validate to NTP on the CIMC

Step 1 - Log in to the appliance Cisco IMC using the Cisco IMC IP address, user ID, and password you set in Enable Browser Access to Cisco Integrated Management Controller. 

Step 2 -Synchronize the appliance hardware with the Network Time Protocol (NTP) servers you use to manage your network, as follows:

1. From the top-left corner of the Cisco IMC GUI, click the Toggle Navigation icon.
2. From the Cisco IMC menu, select Admin > Networking, and then choose the NTP Setting tab.
3. Make sure that the NTP Enabled check box is checked and enter up to four NTP server host names or addresses in the numbered Server fields.
4. Cisco IMC validates your entries and then begins to synchronize the time on the appliance hardware with the time on the NTP servers.

CIMC NTP Configuration page

Note: Cisco IMC does not support NTP authentication.

Review to NTP configuration on the DNAC

• Review the NTP services configured in the DNAC, confirm that the NTP has a * informt of the server
  • Max offset value: 500
  • Max jitter value: 300

maglev@maglev-master:~$ ntpq -pn
remote             refid   st  t  when poll  reach  delay  offset  jitter
==============================================================================
*ntp.server.local  .GNSS.  2  u  823 1024    0    0.263    0.144   0.000
10.81.254.131      .GNSS.  1  u  835  1024   377   72.324  0.382   0.087

• Confirm if the System clock synchronized is synchronized with the command timedatectl.

maglev@maglev-master:~$ timedatectl status
Local time: Thu 2023-09-28 20:27:13 UTC
Universal time: Thu 2023-09-28 20:27:13 UTC
RTC time: Thu 2023-09-28 20:27:13
Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
systemd-timesyncd.service active: no
RTC in local TZ: no

• Review that the NTP servers are configured correctly on the ntp.conf file.

maglev@maglev-master:~$ cat /etc/ntp.conf
#---------------------------------------------------------------------
# Modified by Maglev: Mon, 25 Sep 2023 21:04:04 UTC
# maglev-config 68913
#---------------------------------------------------------------------

tinker panic 0
driftfile /var/lib/ntp/ntp.drift
statsdir /var/log/ntpstats/
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
server 10.81.254.131 iburst
server ntp.server.local iburst
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
restrict 127.0.0.1
restrict ::1
interface ignore 0.0.0.0
interface listen enterprise
interface listen management
interface listen internet
interface listen cluster

Warning: Do NOT modify the file ntp.conf file.

Validate NTP on the DNAC

• When you configure the NTP with a FQDN verify that the DNAC can resolve the A and PTR Records.

maglev@maglev-master:~$ nslookup
>set type=A
> ntp.server.local
Server: 10.0.0.53
Address: 10.0.0.53#53

Non-authoritative answer:
Name:  ntp.server.local
Address: 10.81.254.202


>set type=PTR
> 10.81.254.202
Server: 10.0.0.53
Address: 10.0.0.53#53

10.254.81.10.in-addr.arpa name =  ntp.server.local.

• Validate that you can reach the NTP via ping.

maglev@maglev-master:~$ ping ntp.server.local
PING ntp.server.local (10.81.254.202) 56(84) bytes of data.
64 bytes from ntp.server.local (10.81.254.202): icmp_seq=1 ttl=53 time=72.8 ms
64 bytes from ntp.server.local (10.81.254.202): icmp_seq=2 ttl=53 time=71.9 ms
64 bytes from ntp.server.local (10.81.254.202): icmp_seq=3 ttl=53 time=72.0 ms
^C
--- ntp.server.local ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 72.506/72.634/72.853/0.269 m

• Validate that you can reach the NTP on the port 123/UDP.

maglev@maglev-master:~$ nc -zvu ntp.server.local 123
Connection to ntp.server.local 123 port [udp/ntp] succeeded!

• Take a packet capture and confirm that the NTP communication is on the same version NTPv4.

maglev@maglev-master:~$ sudo tcpdump -i any host ntp.server.local and port 123 --immediate-mode
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
20:19:23.967314 IP flink-web.ndp.svc.cluster.local.ntp > ntp.server.local.ntp: NTPv4, Client, length 48
20:19:23.967329 IP flink-web.ndp.svc.cluster.local.ntp > ntp.server.local.ntp: NTPv4, Client, length 48
20:19:24.040064 IP ntp.server.local.ntp > flink-web.ndp.svc.cluster.local.ntp: NTPv4, Server, length 48
20:19:24.040064 IP ntp.server.local.ntp > flink-web.ndp.svc.cluster.local.ntp: NTPv4, Server, length 48

• Confirm that the NTP service is Active and running.

maglev@maglev-master:~$ systemctl status ntp
* ntp.service - Network Time Service
Loaded: loaded (/lib/systemd/system/ntp.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2023-09-28 20:19:20 UTC; 22min ago
Docs: man:ntpd(8)
Process: 31746 ExecStart=/usr/lib/ntp/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
Main PID: 31781 (ntpd)
Tasks: 2 (limit: 13516)
CGroup: /system.slice/ntp.service
`-31781 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 107:111

Sep 28 20:19:20 maglev-master-10-88-244-151 ntpd[31781]: restrict ::: KOD does nothing without LIMITED.
Sep 28 20:19:20 maglev-master-10-88-244-151 ntpd[31781]: Listen and drop on 0 v6wildcard [::]:123
Sep 28 20:19:20 maglev-master-10-88-244-151 ntpd[31781]: Listen normally on 1 lo 127.0.0.1:123
Sep 28 20:19:20 maglev-master-10-88-244-151 ntpd[31781]: Listen normally on 2 management 10.88.244.151:123
Sep 28 20:19:20 maglev-master-10-88-244-151 ntpd[31781]: Listen normally on 3 enterprise 192.168.31.2:123
Sep 28 20:19:20 maglev-master-10-88-244-151 ntpd[31781]: Listen normally on 4 lo [::1]:123
Sep 28 20:19:20 maglev-master-10-88-244-151 ntpd[31781]: Listen normally on 5 management [fe80::be26:c7ff:fe0c:82e6%5447]:123
Sep 28 20:19:20 maglev-master-10-88-244-151 ntpd[31781]: Listen normally on 6 enterprise [fe80::b28b:cfff:fe6a:9e1c%5449]:123
Sep 28 20:19:20 maglev-master-10-88-244-151 ntpd[31781]: Listen normally on 7 cluster [fe80::b28b:cfff:fe6a:9e1d%5450]:123
Sep 28 20:19:20 maglev-master-10-88-244-151 ntpd[31781]: Listening on routing socket on fd #24 for interface updates

Note: If need it you can restart the NTP service with the command  sudo systemctl restart ntp . This does not have any impact.

Troubleshoot NTP on the DNAC

• Review the maglev_config_wizard.log file.

• The next excerpt shows DNAC communicate and sync with the NTP server.

maglev@maglev-master:~$ cat /var/log/maglev_config_wizard.log | grep -i ntp
| 2023-09-28 00:47:32,790 | DEBUG | 25344 | MainThread | 140017254479680 | root | ansible.py:495 | changed: [localhost] => {"changed": true, "cmd": "/opt/maglev/bin/check_ntp.sh 500 299", "delta": "0:00:00.018699", "end": "2023-09-28 00:47:32.764617", "rc": 0, "start": "2023-09-28 00:47:32.745918", "stderr": "", "stderr_lines": [], "stdout": "PASSED", "stdout_lines": ["PASSED"]}
| 2023-09-28 00:47:33,068 | DEBUG | 25344 | MainThread | 140017254479680 | root | ansible.py:495 | ok: [localhost] => {"ansible_facts": {"ntp_sync_check": "PASSED"}, "changed": false}
"msg": "Check NTP limit PASSED"

changed: [localhost] => {"changed": true, "cmd": "/opt/maglev/bin/check_ntp.sh 500 299", "delta": "0:00:00.018699", "end": "2023-09-28 00:47:32.764617", "rc": 0, "start": "2023-09-28 00:47:32.745918", "stderr": "", "stderr_lines": [], "stdout": "PASSED", "stdout_lines": ["PASSED"]}
ok: [localhost] => {"ansible_facts": {"ntp_sync_check": "PASSED"}, "changed": false}
"msg": "Check NTP limit PASSED"

• The next excerpts show errors when the NTP is not sync or has communication issues.

maglev@maglev-master:~$ cat /var/log/maglev_config_wizard.log | grep -i ntp
| 2023-07-19 02:36:41,396 | INFO | 76230 | MainThread | 140599082059584 | root | certs.py:142 | renew_certs : Check NTP limits |
| 2023-07-19 02:36:41,703 | DEBUG | 76230 | MainThread | 140599082059584 | root | ansible.py:495 | changed: [localhost] => {"changed": true, "cmd": "/opt/maglev/bin/check_ntp.sh 500 299", "delta": "0:00:00.014850", "end": "2023-07-19 02:36:41.679386", "rc": 0, "start": "2023-07-19 02:36:41.664536", "stderr": "", "stderr_lines": [], "stdout": "WARNING: Could not get Offset or Jitter from ntp peer", "stdout_lines": ["WARNING: Could not get Offset or Jitter from ntp peer"]}
| 2023-07-19 02:36:41,960 | DEBUG | 76230 | MainThread | 140599082059584 | root | ansible.py:495 | ok: [localhost] => {"ansible_facts": {"ntp_sync_check": "WARNING: Could not get Offset or Jitter from ntp peer"}, "changed": false}
"msg": "Check NTP limit WARNING: Could not get Offset or Jitter from ntp peer"
| 2023-07-19 02:36:42,635 | INFO | 76230 | MainThread | 140599082059584 | root | certs.py:142 | renew_certs : Fail if NTP not synched |
TASK [renew_certs : Check NTP limits] ******************************************

maglev@maglev-master:~$ cat /var/log/maglev_config_wizard.log | grep -i ntp
| 2023-09-12 18:21:29,564 | ERROR | 82110 | MainThread | 139737866331968 | maglev_config_wizard.managers.ntp.NtpManager | ntp.py:52 | Unable to sync with time server 10.88.14.200 |
| 2023-09-12 18:21:34,569 | ERROR | 82110 | MainThread | 139737866331968 | maglev_config_wizard.managers.ntp.NtpManager | ntp.py:164 | Unable to configure NTP after 2 attempts |

• If you need to change the NTP server please use the command sudo maglev-config update.

Caution: Change the NTP restarts the services in the DNAC.

Related Information

• Cisco Technical Support & Downloads