Configure and troubleshoot PTP in Nexus 3000

Introduction

Precision Time Protocol (PTP) is a distributed nanosecond accuracy timing synchronization protocol for packet networks. PTP’s accuracy comes from the hardware support for PTP in the switch and server network interface cards (NICs). PTP uses multicast group 224.0.1.129 and UDP ports 319 and 320. Current PTP standard is IEEE 1588-2008 Precision Time Protocol (PTP) Version 2 (PTPv2)

The scalability of PTP comes from the fact that this protocol has a concept of Boundary Clock (BC), which can be multiple physical/logical hops away from the Grand Master (GM) clock and acts as a re-calibration point for the messages containing timestamps. The Nexus 3ks are widely deployed in Data Centres (DCs) to act as BC and provide accurate timings to the servers connected to Local Area Network (LAN). The BC and the GM need to be reachable via Layer3 (ip connectivity) only. There can be several layers of BC between the GM and the PTP end-client. There can be multiple GMs to provide redundancy and the Nexus 3K will select the best GM via Best Master Clock (BMC) algorithm.

Contributed by: Nishad Mohiuddeen

Edited by: Kumar Sridhar

PTP Configuration:

N3k(config)# feature ptp

                This command enables PTP on the switch.

N3k(config)# ptp source <ip address>

                This command specifies the source IP address for the PTP packets generated by the switch.

N3k(config)# interface Ethernet slot/port

N3k(config-if)# ptp

This command enables PTP on a port. The Cisco Nexus 3548 is a boundary clock, so it has both master and slave ports. There is no configuration difference between a master port and a slave port. They are both configured with the “ptp” option and BMC algorithm will determine whether the port is a PTP slave or master port.

N3k(config)# clock protocol ptp

This command configures the switch to use PTP to update the system calendar. This configuration keeps the clock of the switch synchronized with PTP. Not enabling this command won’t prevent the switch from propagating the PTP clock on its master ports. However, the time source will be the Nexus local clock.

N3k(config)# ptp priority1 <0-255>

N3k(config)# ptp priority2 <0-255>

Configure the priority values for the local (oscillator) clock. Lower numeric value indicates higher priority.

N3k(config)# no ptp grandmaster-capable

By default Nexus 3k is “ptp grandmaster-capable”, so disable this capability to get synched with the GM.

Optional parameters under the interface (configure to match parameter with GM):

N3k(config)# interface Ethernet slot/port

N3k(config-if)# ptp delay-request minimum interval 3

N3k(config-if)# ptp announce interval 2

N3k(config-if)# ptp sync interval 0

Verification:

N3k# show ptp clock

PTP Device Type: Boundary clock

Clock Identity :  00:62:ec:ff:fe:40:05:81

Clock Domain: 0

Number of PTP ports: 2

Priority1 : 1

Priority2 : 1

Clock Quality:

        Class : 248

        Accuracy : 254

        Offset (log variance) : 65535

Offset From Master : 0

Mean Path Delay : 0

Steps removed : 0

Local clock time:Mon Jun  5 00:00:23 2017

N3k# show ptp parent

PTP PARENT PROPERTIES

Parent Clock:

Parent Clock Identity:  00:0c:ec:ff:fe:08:12:b1

Parent Port Number: 1

Observed Parent Offset (log variance): N/A

Observed Parent Clock Phase Change Rate: N/A

Grandmaster Clock:

Grandmaster Clock Identity:  00:0c:ec:ff:fe:08:12:b1

Grandmaster Clock Quality:

        Class: 6  <<<<

        Accuracy: 32

        Offset (log variance): 22752

        Priority1: 128

        Priority2: 128

N3k# show ptp brief

PTP port status

-----------------------

Port         State

-------  --------------

Eth1/5      Slave   <<< towards GM

Eth1/24     Master <<< towards Host

Eth1/24     Listening <<< no Valid GM or Host detected

N3k#show run | in clock (to check clock-protocol)

N3k#show ptp counters all (to check the Tx and Rx of PTP messages like Sync, Announce, Delay_Req, Delay_Resp etc.)

N3k# show ptp counters interface ethernet 1/24

PTP Packet Counters of Interface Eth1/24:

----------------------------------------------------------------

Packet Type                  TX                      RX

----------------    --------------------    --------------------

Announce                     558                    4479

Sync                            1773                    8941

FollowUp                     1754                    8950

Delay Request              35                       0

Delay Response             0                      35

PDelay Request             0                       0

PDelay Res                    0                       0

PDelay Followup            0                       0

Management                  0                       0

----------------------------------------------------------------

N3k#show ptp corrections (to check PTP correction timestamps)

PTP past corrections

----------------------------------------------------------------------------------------------------------------

Slave Port              SUP Time                               Correction(ns)    MeanPath Delay(ns)

----------      -----------------------------------------------     ------------------     ------------------

Eth1/24       Mon Jun  5 05:48:45 2017 171026           -51                 1806

Eth1/24       Mon Jun  5 05:48:46 2017 171727            -2                  1806

Eth1/24       Mon Jun  5 05:48:47 2017 173329           -47                 1806

Eth1/24       Mon Jun  5 05:48:48 2017 174047            86                 1806

Eth1/24       Mon Jun  5 05:48:49 2017 175690           -55                 1794

Eth1/24       Mon Jun  5 05:48:50 2017 235577           -6                   1794

Eth1/24       Mon Jun  5 05:48:51 2017 178035           -44                 1794

Eth1/24       Mon Jun  5 05:48:52 2017 178804            83                 1794

Eth1/24       Mon Jun  5 05:48:53 2017 180371           35                  1794

Eth1/24       Mon Jun  5 05:48:54 2017 181839           -48                 1794

Eth1/24       Mon Jun  5 05:48:55 2017 183667           -42                 1794

Eth1/24       Mon Jun  5 05:48:56 2017 184423           -5                   1794

Eth1/24       Mon Jun  5 05:48:57 2017 186030           113                1794

Eth1/24       Mon Jun  5 05:48:58 2017 186653           -48                 1794

Eth1/24       Mon Jun  5 05:48:59 2017 188298           -6                   1794

Eth1/24       Mon Jun  5 05:49:00 2017 189000           -88                 1794

<snip>

Troubleshooting:

 1. Check if you can ping the GM from the PTP (unicast connectivity).

 2. Check clock protocol, which should be PTP, via

N3k#show run | in clock

3. By default N3Ks are ‘grandmaster capable’. Need to disable it via

N3k# no ptp grandmaster-capable

4. Use show ptp brief command to check if port state is ‘Listening’.

5. Then do an ethanalyzer capture.

        ethanalyzer local interface inbound-hi capture-filter "udp port 320" limit-captured-frames 0    <<< Announce and Follow up

        ethanalyzer local interface inbound-hi capture-filter "udp port 319" limit-captured-frames 0    <<< Sync

From the PTP GM side there would be Announce and Sync messages. The client will send Delay_Req and GM will send Delay_Resp.

If no packets are captured in ethanalyzer then it may be a priority issue. If the Nexus 3k receives PTP packets with lower priority (or clock class in case of equal priority) it will discard the packet and will not punt to CPU.

6. The best way is to do packet captures (to get the packet between GM and BC) to see why the BC is not synchronizing with the GM. From the capture we can verify the PTP priority sent by GM by looking at the ‘Announce’ message coming from the GM. If the priority is lower (higher numeric value) than the configured priority in the BC, then change the PTP priority of the BC to higher numeric value.

7. If the BC (Nexus3k) is synchronized with GM but the downstream hosts/servers are not, then look for ‘Delay_Req_Messsage’ from the host/server. If not present, then there is something wrong with PTP daemon in host/server.