Contents

Introduction

Prerequisites

Components Used

Restrictions and Limitations

Understand PTP

Best Master Clock(BMC) Algorithm

Synchronizing the clock

Lab Topology

Basic Configuration:

Troubleshoot Steps:

Verify that PTP is Configured Correctly.

Verify PTP Hierarchy Setup Successful as Your Design.

Verify Parentand Grandmaster Information in the PTP Domain

Verify PTP Correction and Bad Correction.

Useful Collection:

Common Issues:

Nexus 9000 Cannot Sync Time with Grandmaster or Upstream Boundary Clock

Actions to Take:

Unexpected Grandmaster Switchover

Actions to Take

High Bad Correction

Actions to take:

PTP Port in the Master State when it is Supposed to be a Slave or Passive

Actions to take:

Best Practice

Related Information

Introduction

This document describes how to troubleshoot the Precision Time Protocol (PTP) on Nexus 9000 switches.

Prerequisites

Cisco recommends that you know these topics:

- Basic knowledge of PTP

- Familiar with the Cisco Nexus Operating System (NX-OS)

The design and configuration of PTP (Precision Time Protocol) are not covered in this article. For such information, it is recommended that you refer to the configuration guide.

Nexus 9000 PTP configuration guide

Precision Time Protocol (PTP) for Cisco Nexus Dashboard Insights

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document is based on these software and hardware versions:

- N9K Spine01: N9K-C93180YC-FX  NX-OS 10.3(4a)

- N9K Spine02: N9K-C93180YC-EX  NX-OS 10.3(4a)

- N9K Leaf01:   N9K-C92160YC-X    NX-OS  9.3.12

- N9K Host:       N9K-C92160YC-X    NX-OS  9.3.12

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Restrictions and Limitations

- For PTP to function properly, you must use the latest SUP and line card FPGA versions. For information on upgrading FPGAs, access the Release Notes landing page go to the FPGA/EPLD Upgrade Release Notes (NX-OS Mode Switches) section and locate the FPGA/EPLD Upgrade Release Notes for your software version. Refer to the Installation Guidelines topic.

- On Nexus 9000 PTP operates only in boundary clock mode. End-to-end transparent clock and peer-to-peer transparent clock modes are not supported.

- PTP is not supported on the Cisco Nexus 92348GC-X platform switch.

- The QoS TCAM region Ingress SUP [ingress-sup] must be set to 768 or higher for PTP IPv6 transport to work.

Before troubleshooting the PTP issue, it is recommended to review the PTP section of Nexus 9000 System Management Configuration for the given platform and version.

Understand PTP

The PTP process consists of two phases: establishing the primary-secondary hierarchy and synchronizing the clocks.

Best Master Clock(BMC) Algorithm

The BMCA is used to select the time source clock on each link, and it ultimately selects the grandmaster clock for the whole PTP domain. It runs locally on each port of the ordinary and boundary clocks to compare the local data sets with the received data from Announce messages to select the best clock on the link.

1. Priority1: User configurable absolute priority (lower value wins)

2. Clock Class: Attribute defining clock’s traceability (not user configurable, lower value wins)

3. Clock Accuracy: Defines the accuracy of a clock (not user configurable, lower value wins)

4. Clock Variance: Attribute defining the precision of a clock (not user configurable)

5. Priority2: User configurable

6. Source Port ID: Mac address of source port

Announce messages are used to establish the synchronization hierarchy.

Synchronizing the clock

Sync, Delay_Req, Follow_Up, and Delay_Resp messages are used to calculate time.

PTP messages can be either multicast or unicast, with multicast being the default mode. PTP uses multicast destination IP address 224.0.1.129 UDP319/320 as per IEEE 1588 standards.

PTP Profiles — PTP supports default (1588), AES67, and SMPTE 2059-2 profiles. Each of these profiles has different ranges of sync and delay request intervals. For more information about these profiles, please refer to the configuration guide.

Lab Topology

Basic Configuration:

feature ptp

ptp source 192.168.1.3>>>>Define PTP packet source IP

ptp priority1 127     >>>>Define PTP priority 1

ptp priority2 127     >>>>Define PTP priority 2

interface Ethernet1/31

ptp                   >>>>Enable PTP in all interconnected ports.

interface Ethernet1/33

ptp

interface Ethernet1/34

ptp

Troubleshoot Steps:

Verify that PTP is Configured Correctly.

Ensure each device has a unique source IP and the PTP domain ID is the same on all devices.

N9K_Spine01# show ptp clock

PTP Device Type : boundary-clock

PTP Source IPv4 Address : 192.168.1.3>>>>PTP source IP

PTP Source IPv6 Address : 0::

Clock Identity : 00:ee:ab:ff:fe:67:3e:9d

Clock Domain: 0                      >>>>PTP domain id. Must same in one PTP domain.

Slave Clock Operation : Two-step

Master Clock Operation : Two-step    >>>>N9K EX/FX/FX2/FX3 only support two-step mode.

Slave-Only Clock Mode : Disabled

Number of PTP ports: 3

Priority1 : 127

Priority2 : 127

Clock Quality:

Class : 248

Accuracy : 254

Offset (log variance) : 65535

Steps removed : 1                    >>>>Hops from GM

Verify PTP Hierarchy Setup Successful as Your Design.

The slave port is connected to the upstream clock device. The master port is connected to the downstream device.

N9K_Spine01# show ptp brief

PTP port status

--------------------------------------------

Port State

-------------------------------------------

Eth1/31 Slave      >>>>Connected to GM

Eth1/33 Master     >>>>Connected to N9K Spine 2

Eth1/34 Master     >>>>Connected to N9K leaf

N9K_Spine02# show ptp brief

PTP port status

--------------------------------------------

Port State

------------------------------- ------------

Eth1/48 Passive   >>>>Connected to N9K leaf. The Port should be in the passive state to avoid loop

Eth1/50 Master    >>>>Connected to GM02

Eth1/53 Slave     >>>>Connected to N9K Spine 1

N9K_Leaf01# show ptp brief

PTP port status

--------------------------------------------

Port State

------------------------------- ------------

Eth1/48 Master    >>>>Connected to Spine02

Eth1/50 Master    >>>>Connected to host

Eth1/51 Slave     >>>>Connected to Spine01

GM01# show system internal ptp info announce-pkts

2024-01-02T13:36:23.242624000+08:00 [M 1] [ptp] E_DEBUG [ptp_ev_hist_ann_pkt:5469] [TX] If Eth1/35 (0x1a004400): MSG: Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:00:ee:ab:ff:fe:67:37:e9-0x89 CORR:0 (0x0) SEQ:bda4 (48548) Int:1; TS

:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:0 PRIO1:1 PRIO2:1 CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9;

2024-01-02T13:36:15.238816000+08:00 [M 1] [ptp] E_DEBUG [ptp_ev_hist_ann_pkt:5469] [TX] If Eth1/35 (0x1a004400): MSG: Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:00:ee:ab:ff:fe:67:37:e9-0x89 CORR:0 (0x0) SEQ:bda0 (48544) Int:1; TS

:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:0 PRIO1:1 PRIO2:1 CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9;

N9K_Spine01# show system internal ptp info announce-pkts

2024-01-02T13:36:20.826735000+08:00 [M 1] [ptp] E_DEBUG [ptp_ev_hist_ann_pkt:5469] [TX] If Eth1/33 (0x1a004000): MSG: Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:00:ee:ab:ff:fe:67:3e:9d-0x81 CORR:0 (0x0) SEQ:bbfc (48124) Int:1; TS

:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:1 PRIO1:1 PRIO2:1 CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9;

2024-01-02T13:36:17.231080000+08:00 [M 1] [ptp] E_DEBUG [ptp_ev_hist_ann_pkt:5469] [TX] If Eth1/34 (0x1a004200): MSG: Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:00:ee:ab:ff:fe:67:3e:9d-0x85 CORR:0 (0x0) SEQ:bdac (48556) Int:1; TS

:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:1 PRIO1:1 PRIO2:1 CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9;

2024-01-02T13:36:16.239728000+08:00 [M 1] [ptp] E_DEBUG [ptp_ev_hist_ann_pkt:5469] [RX] If Eth1/31 (0x1a003c00): MSG: Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:00:ee:ab:ff:fe:67:37:e9-0x89 CORR:0 (0x0) SEQ:bda1 (48545) Int:1; TS

:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:0 PRIO1:1 PRIO2:1 CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9;

N9K_Spine02# show system internal ptp info announce-pkts

2024-01-02T13:36:21.368978000+08:00 [M 1] [ptp] E_DEBUG [ptp_ev_hist_ann_pkt:5469] [RX] If Eth1/48 (0x1a005e00): MSG: Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:4c:77:6d:ff:fe:d5:35:35-0xbd CORR:0 (0x0) SEQ:bcd7 (48343) Int:1; TS

:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:2 PRIO1:1 PRIO2:1 CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9;

2024-01-02T13:36:19.363095000+08:00 [M 1] [ptp] E_DEBUG [ptp_ev_hist_ann_pkt:5469] [TX] If Eth1/50 (0x1a006200): MSG: Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:c0:14:fe:ff:fe:89:9b:77-0xc5 CORR:0 (0x0) SEQ:00e8 (232) Int:1; TS:0

s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:2 PRIO1:1 PRIO2:1 CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9;

2024-01-02T13:36:16.828573000+08:00 [M 1] [ptp] E_DEBUG [ptp_ev_hist_ann_pkt:5469] [RX] If Eth1/53 (0x1a006800): MSG: Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:00:ee:ab:ff:fe:67:3e:9d-0x81 CORR:0 (0x0) SEQ:bbfa (48122) Int:1; TS

:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:1 PRIO1:1 PRIO2:1 CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9;

N9K_Leaf01# show system internal ptp info announce-pkts

2024 Jan 02 13:36:23.893622: E_PTP_ANN_PKT_EV[TX] I/f Eth1/50 (0x1a006200): MSG:Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:4c:77:6d:ff:fe:d5:35:35-0xc5 CORR:0 (0x0) SEQ:1354 (4948) Int:1; TS:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:2

PRIO1:1 PRIO2:1

CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9

2024 Jan 02 13:36:23.369089: E_PTP_ANN_PKT_EV[TX] I/f Eth1/48 (0x1a005e00): MSG:Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:4c:77:6d:ff:fe:d5:35:35-0xbd CORR:0 (0x0) SEQ:bcd8 (48344) Int:1; TS:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:2

PRIO1:1 PRIO2:1

CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9

2024 Jan 02 13:36:23.233889: E_PTP_ANN_PKT_EV[RX] I/f Eth1/51 (0x1a006400): MSG:Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:00:ee:ab:ff:fe:67:3e:9d-0x85 CORR:0 (0x0) SEQ:bdaf (48559) Int:1; TS:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:1

PRIO1:1 PRIO2:1

CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9

Host# show system internal ptp info announce-pkts

2024 Jan 02 13:36:23.898218: E_PTP_ANN_PKT_EV[RX] I/f Eth1/50 (0x1a006200): MSG:Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:4c:77:6d:ff:fe:d5:35:35-0xc5 CORR:0 (0x0) SEQ:1354 (4948) Int:1; TS:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:2

PRIO1:1 PRIO2:1

CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9

V:2

PTP version 2

LEN:64

PTP message length 64

D:0

PTP domain 0

UC:0

0: PTP multicast packet,1:PTP unicast packet

2S:0

maybe leap 59/61

UTCVAL:0

UTC offset valid flag. 0 means false. GM set it.

PTPTS:1

PTP TimeScale flag. 1 means true

TT:0

PTP TimeTrace flag. 0 means false

FT:0

PTP FreqTrace flag .0 means false

SRC:

PTP packet source mac

CORR:0

Correction

SEQ:

PTP sequence ID

INT:1

Log message period. 1 mean 2s

TS:

Timestamps

UTC_OFF

UTC offset value. GM set this value.

TM_SRC

0x20 GPS, 0x40 PTP, 0x50 NTP, 0x60 Hand_set 0xa internal oscillator.  GM set this value.

STEP:2

Clock operation in two-step mode(N9K only supports two steps mode on the primary port)

PRIO1:1 PRIO2:1 CLASS:248 ACC:fe LOG_VAR:ffff

GM priority, GM clock class, GM clock accuracy

GM

GM Clock Identity. Come from mac address.

Verify Parent and Grandmaster Information in the PTP Domain

Ensure the parent clock device and grandmaster device are stable.

N9K_Spine01# show ptp parent

Parent Clock:

Parent Clock Identity: 00:ee:ab:ff:fe:67:37:e9       >>>>upstream clock identity. 37:e9 is GM in lab topology

Parent Port Number: 137

Observed Parent Offset (log variance): N/A

Observed Parent Clock Phase Change Rate: N/A

Parent IP: 192.168.1.1                               >>>>upstream clock source IP

Grandmaster Clock:

Grandmaster Clock Identity: 00:ee:ab:ff:fe:67:37:e9  >>>>GM clock identity

Grandmaster Clock Quality:                           >>>>GM clock attributes

Class: 248

Accuracy: 254

Offset (log variance): 65535

Priority1: 1                                         >>>>GM priority1

Priority2: 1

N9K_Spine02# show ptp parent

Parent Clock:

Parent Clock Identity: 00:ee:ab:ff:fe:67:3e:9d>>>upstream clock identity. 37:9d is N9K Spine01 in lab topology

Parent Port Number: 129

Observed Parent Offset (log variance): N/A

Observed Parent Clock Phase Change Rate: N/A

Parent IP: 192.168.1.3                                >>>>upstream clock source IP. 192.168.1.3 is N9K Spine01

Grandmaster Clock:

Grandmaster Clock Identity: 00:ee:ab:ff:fe:67:37:e9

Grandmaster Clock Quality:

Class: 248

Accuracy: 254

Offset (log variance): 65535

Priority1: 1

Priority2: 1

N9K_Leaf01# show ptp parent

PTP PARENT PROPERTIES

Parent Clock:

Parent Clock Identity: 00:ee:ab:ff:fe:67:3e:9d

Parent Port Number: 133

Observed Parent Offset (log variance): N/A

Observed Parent Clock Phase Change Rate: N/A

Parent IP: 192.168.1.3

Grandmaster Clock:

Grandmaster Clock Identity: 00:ee:ab:ff:fe:67:37:e9

Grandmaster Clock Quality:

Class: 248

Accuracy: 254

Offset (log variance): 65535

Priority1: 1

Priority2: 1

Verify PTP Correction and Bad Correction.

The Sync-SeqID must increase with each entry. The correction must be less than 10000 nanoseconds (ns).

N9K_Spine02# show system internal ptp corrections

PTP past corrections

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Slave Port   SUP Time                                   Correction(ns) MeanPath Delay(ns) MasterTimestamp (sec, nsec) Slave Timestamp (sec, nsec) Sync-SeqID   PTPLC ts_corr(ns)

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Eth1/53    Wed Jan  3 15:29:05 2024  15928                  -8                 204          1704266945   14551936          1704266945   14552148      31871                   0

Eth1/53    Wed Jan  3 15:29:04 2024 765051                  24                 204          1704266944  764413040          1704266944  764413220    31870                   0

Eth1/53    Wed Jan  3 15:29:04 2024 509436                  24                 204          1704266944  513679576          1704266944  513679756    31869                   0

Eth1/53    Wed Jan  3 15:29:04 2024 264139                   0                 204          1704266944  263204688          1704266944  263204892     31868                   0

Eth1/53    Wed Jan  3 15:29:04 2024  13239                  -8                 204          1704266944   12722952          1704266944   12723164        31867                   0

Eth1/53    Wed Jan  3 15:29:03 2024 762756                  24                 212          1704266943  761756520          1704266943  761756708    31866                   0

Bad Correction Record

By default, the correction threshold is 100000 nanoseconds (100us). Corrections not within this range are recorded as bad corrections.

N9K_Spine02(config)# show system internal ptp bad-corrections

PTP past corrections

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Slave Port   SUP Time                                Correction(ns) MeanPath Delay(ns) MasterTimestamp (sec, nsec) Slave Timestamp (sec, nsec) Sync-SeqID   PTPLC ts_corr(ns)

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Eth1/48     Tue Jan  2 13:28:30 2024 692911            17111776               172          1704173310  705666212          1704173310  688554608    52942                   0

Eth1/48     Tue Jan  2 13:28:30 2024 443146            17111808                 172          1704173310  454735796          1704173310  437624160    52941                   0

Eth1/48     Tue Jan  2 13:28:30 2024 188850            17111784                 172          1704173310  203243428          1704173310  186131816    52940                   0

Eth1/48     Tue Jan  2 13:28:29 2024 949432            51292504                 172          1704173309  952419348          1704173309  901127016    52939                   0

Useful Collection:

show running-config ptp

show ptp brief

show ptp counters all

show ptp clock

show system internal ptp info all

show system internal ptp info global

show ptp clock foreign-masters record

show system internal ptp corrections entries 2000

show system internal ptp bad-corrections entries 2000

show system internal ptp trouble-shooting all

show tech ptp

Common Issues:

Nexus 9000 Cannot Sync Time with Grandmaster or Upstream Boundary Clock

In most cases, these are configuration issues.

Actions to Take:

1. Check if the PTP domain number is the same on all PTP-enabled devices. Ensure the unique PTP source IP is configured on all devices.

show ptp clock

TP Device Type : boundary-clock

PTP Device Encapsulation : NA

PTP Source IP Address : 192.168.1.4

Clock Identity : c0:14:fe:ff:fe:89:9b:77

Clock Domain: 0

Slave Clock Operation : Two-step

Master Clock Operation : Two-step

<snip>

Local clock time : Thu Jan 4 19:34:26 2024

PTP Clock state : Locked

2. Verify that PTP is enabled on the interface. By default, it is disabled.

N9K_Spine02# show ptp brief

--------------------------------------------

Port State

------------------------------- ------------

Eth1/48 Passive   >>>>Connected to N9K leaf. Port in the passive state to prevent loop

Eth1/50 Master    >>>>Connected to GM02

Eth1/53 Slave     >>>>Connected to N9K Spine 1

3. Check PTP interface parameters. Ensure that the same PTP VLAN as the peer is being used.

N9K_Spine02# show ptp port interface e1/48

PTP Port Dataset: Eth1/48

Port identity: clock identity: c0:14:fe:ff:fe:89:9b:77

Port identity: port number: 188

PTP version: 2

Port state: Master

VLAN info: 1

Delay request interval(log mean): 0

Announce receipt time out: 3

Peer mean path delay: 0

Announce interval(log mean): 1

Sync interval(log mean): -2

Delay Mechanism: End to End

Cost: 255

Domain: 0

Unexpected Grandmaster Switchover

2024 Jan 4 19:27:05 N9K_Spine02 %PTP-2-PTP_GM_CHANGE: Grandmaster clock has changed from 00:ee:ab:ff:fe:67:37:e9 to c0:14:fe:ff:fe:a3:c4:67 for the PTP protocol

Actions to Take

1. Check the PTP announce history for changes in priority or other clock attribute changes.

show system internal ptp info announce-pkts

2024 Jan 04 19:27:07.408293: E_PTP_ANN_PKT_EV[RX] I/f Eth1/48 (0x1a005e00): MSG:Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:4c:77:6d:ff:fe:d5:35:35-0xbd CORR:0 (0x0) SEQ:3673 (13939) Int:1; TS:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:2 PRIO1:1 PRIO2:1

CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9

2024 Jan 04 19:27:06.321569: E_PTP_ANN_PKT_EV[RX] I/f Eth1/50 (0x1a006200): MSG:Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:c0:14:fe:ff:fe:a3:c4:67-0xc9 CORR:0 (0x0) SEQ:0005 (5) Int:1; TS:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:0 PRIO1:0 PRIO2:1

CLASS:248 ACC:fe LOG_VAR:ffff GM:c0:14:fe:ff:fe:a3:c4:67

2024 Jan 04 19:27:05.427431: E_PTP_ANN_PKT_EV[RX] I/f Eth1/53 (0x1a006800): MSG:Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:00:ee:ab:ff:fe:67:3e:9d-0x81 CORR:0 (0x0) SEQ:3593 (13715) Int:1; TS:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:1 PRIO1:1 PRIO2:1

CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9

2024 Jan 04 19:27:05.407196: E_PTP_ANN_PKT_EV[RX] I/f Eth1/48 (0x1a005e00): MSG:Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:4c:77:6d:ff:fe:d5:35:35-0xbd CORR:0 (0x0) SEQ:3672 (13938) Int:1; TS:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:2 PRIO1:1 PRIO2:1

CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9

2024 Jan 04 19:27:04.822821: E_PTP_ANN_PKT_EV[TX] I/f Eth1/50 (0x1a006200): MSG:Announce TS:0 V:2 LEN:64 D:0 UC:0 2S:0 UTCVAL:0 PTPTS:1 TT:0 FT:0 SRC:c0:14:fe:ff:fe:89:9b:77-0xc4 CORR:0 (0x0) SEQ:0506 (1286) Int:1; TS:0 s 0 ns; UTC_OFF:37 TM_SRC:a0 STEP:2  PRIO1:1 PRIO2:1

CLASS:248 ACC:fe LOG_VAR:ffff GM:00:ee:ab:ff:fe:67:37:e9

High Bad Correction

A random bad correction can be challenging to analyze due to a lack of data. Nexus 9000 provides an auto-log feature to capture PTP logs in the backend without impacting performance.

Actions to take:

1. Identify the bad corrections.

N9K_Spine02# show system internal ptp bad-corrections entries 2000

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Slave Port   SUP Time                                 Correction(ns) MeanPath Delay(ns) MasterTimestamp (sec, nsec) Slave Timestamp (sec, nsec) Sync-SeqID   PTPLC ts_corr(ns)

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Eth1/48     Thu Jan  4 18:41:07 2024 140073            19167640                 172          1704364867  151916416          1704364867  132748948    28473                   0

Eth1/48     Thu Jan  4 18:41:06 2024 889689            19167624                 172          1704364866  901509720          1704364866  882342268    28472                   0

Eth1/48     Thu Jan  4 18:41:06 2024 634900            19167604                 172          1704364866  651177596          1704364866  632010164    28471                   0

Eth1/48     Thu Jan  4 18:41:06 2024 386534            19167636                 172          1704364866  399944588          1704364866  380777124    28470                   0

Eth1/48     Thu Jan  4 18:41:05 2024 732409           425695900                 172          1704364866  148959160          1704364865  723263432    28469                   0

Eth1/48     Thu Jan  4 18:41:05 2024 480431           425695932                 172          1704364865  898386456          1704364865  472690696    28468                   0

Eth1/48     Thu Jan  4 18:41:05 2024 225514           425695908                 172          1704364865  647736096          1704364865  222040360    28467                   0

Eth1/48     Thu Jan  4 18:41:04 2024 977564           425695924                 172          1704364865  396727464          1704364864  971031712    28466                   0

2. Enable PTP Auto Log

test system internal ptp auto-log correction-limit 10000   >> Set a threshold of correction to trigger if the switch starts log collection. In this case, it’s 10000ns

test system internal ptp auto-log file-max-count 5         >> Maximum Auto-log files quantity

no test system internal ptp auto-log file-rollover         >> Disable auto-log rollover

test system internal ptp auto-log                          >> Start auto-log in backend

3. In case a PTP bad correction occurs, the PTP log is created in bootflash.

N9K_Spine02# dir bootflash:

4096 Jan 04 19:57:44 2024 ptp_autolog/

N9K_Spine02# dir ptp_autolog

1115095 Jan 04 19:27:06 2024 auto_ptp_dbg_log_1.log

1099741 Jan 04 19:57:43 2024 auto_ptp_dbg_log_2.log

53631 Jan 04 19:57:43 2024 auto_ptp_dbg_log_3.log

87478 Jan 04 19:57:44 2024 auto_ptp_dbg_log_4.log

In this file, you can locate T1-T4 to perform calculations.

19:26:56 056993 ptp_calc_mean_path_delay t1/m sec 1704367616 ns 54142980 t2/s sec 1704367616 ns 54143180 t3/ds sec 1704367616 ns 54888476 t4/dm sec 1704367616 ns 54888676 diff corr 0 ns

19:26:57 060081 ptp_calc_mean_path_delay t1/m sec 1704367617 ns 56716444 t2/s sec 1704367617 ns 56716636 t3/ds sec 1704367617 ns 57466060 t4/dm sec 1704367617 ns 57466284 diff corr 0 ns

19:26:58 062591 ptp_calc_mean_path_delay t1/m sec 1704367618 ns 59552956 t2/s sec 1704367618 ns 59553164 t3/ds sec 1704367618 ns 60248804 t4/dm sec 1704367618 ns 60249012 diff corr 0 ns

19:26:59 061974 ptp_calc_mean_path_delay t1/m sec 1704367619 ns 61891376 t2/s sec 1704367619 ns 61891632 t3/ds sec 1704367619 ns 62503968 t4/dm sec 1704367619 ns 62504128 diff corr 0 ns

PTP Port in the Master State when it is Supposed to be a Slave or Passive

A PTP port changes to the primary state if it encounters a message exchange issue on the RX (receive) path side.

Actions to take:

1. Check if the PTP RX (receive) counters are increasing on the problematic master port.

N9K_Spine01# show ptp counters all

PTP Packet Counters of Interface Eth1/31:

----------------------------------------------------------------

Packet Type                  TX                     RX

----------------    --------------------    --------------------

Announce                       0                       3

Sync                           0                      21

FollowUp                       0                      21

Delay Request                  5                       0

Delay Response                 0                       5

2. If it is not increasing, then check the SUP redirect ACL (Access Control List) statistics.

N9K_Spine01# show system internal access-list sup-redirect-stats | in PTP|Slice

Instance: 0  [Unit: 0 Slice: 0]

3118                                PTP EVENT REDIRECT    3358695

3119                        ETH PTP EVENT TX TIMESTAMP    0

3120                            PTP EVENT TX TIMESTAMP    5046146

3167                                  PTP MSG REDIRECT    3088156

3183                          PTP UNICAST MSG REDIRECT    0

3184                        PTP UNICAST EVENT REDIRECT    0

Instance: 1  [Unit: 0 Slice: 1]

3118                                PTP EVENT REDIRECT    0

3119                        ETH PTP EVENT TX TIMESTAMP    0

3120                            PTP EVENT TX TIMESTAMP    0

3167                                  PTP MSG REDIRECT    0

3183                          PTP UNICAST MSG REDIRECT    0

3184                        PTP UNICAST EVENT REDIRECT    0

3. Check if Control Plane Policing (CoPP) is dropping PTP messages. If there is a drop counter here, then verify your scale.

N9K_Spine01# show policy-map interface control-plane class copp-system-p-class-redirect

Service-policy  input: copp-system-p-policy-strict

class-map copp-system-p-class-redirect (match-any)

match access-group name copp-system-p-acl-ptp

match access-group name copp-system-p-acl-ptp-l2

match access-group name copp-system-p-acl-ptp-uc

set cos 1

police cir 280 kbps , bc 32000 bytes

module 1 :

transmitted 875343860 bytes;

5-minute offered rate 1650 bytes/sec

conformed 1932 peak-rate bytes/sec

at Thu Jan 04 22:08:20 2024

dropped 0 bytes;                >>>> Check if any counter increasing

5-min violate rate 0 byte/sec

violated 0 peak-rate byte/sec

Best Practice

- Ensure that everything is in the same PTP domain.

- Sync, Announce, and Delay intervals must match on both ends of the link.

- The CLI command ensures that the leaf access port stays in the master  state even if the recipient is misconfigured or a grandmaster is accidentally connected under this port:

interface Ethernet1/1

ptp multicast master-only

- Check the verified scalability guide for the maximum number of master ports per model of a switch.

Related Information

Nexus 9000 PTP Configuration Guide

Precision Time Protocol (PTP) for Cisco Nexus Dashboard Insights