Contents

Introduction

Prerequisites

Requirements

Components Used

Background

Precision Time Protocol (PTP)

Introduction

Protocol

Master And Slave Timestamp Exchange Process

GrandMaster Clock

Slave Clock

Boundary Clock

Clock Classes

Clock States

PTP Domains

PTP Profiles

Basic Messages Definition

Requirements

Configuration

PTP Master Clock On ASR900

Step 1. Configure The Local Internal Oscillator

Step 2. Configure PTP as Master on The ASR900

Verification

Slave Clock On cBR-8

Verification

Restrictions

The G.8275.2 Profile

Slave Clock On RPD

Verification

Boundary Clock On ASR900

Monitor with SNMP

Troubleshoot

Troubleshoot The PTP Master (ASR900)

Troubleshoot The PTP Slave (cBR-8)

DTI and PTP

Clock Delays and Offset

Troubleshoot The PTP Slave (RPD)

Related Information

Introduction

This document describes the Precision Time Protocol (PTP)  which is use in cable network with cBR-8 and Remote PHY (R-PHY) networks. The goal is to give a global understanding of the protocol and how to configure it in cBR-8 and RPHY deployments.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- R-PHY.

- cBR-8 Converged Cable Access Platform (CCAP).

Components Used

The information in this document is based on these software and hardware versions:

- cBR-8 runs 16.6.1 release or later.

- Cisco 1x2 Remote PHY Device (RPD).

Tip: Refer the Cisco 1x2 RPD cisco article for more information.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background

In order to grant modems the time slots (minislots) to transmit on a upstream channel, the CCAP maps out upstreams minislot assignments by means of upstream bandwidth allocation map (MAP) messages. These MAP messages are sent in the downstream and received by all modems.

The modems look at these messages to determine which minislots are allocated to which modems and which are for contention-based activities. A modem only transmits traffic on a minislot assigned to it (or on a contention slot if making a bandwidth request or other station maintenance activity).

The MAP messages from the CCAP allocate approximately 2 milliseconds (ms) worth of time. Low latency DOCSIS (LLD) provides options to reduce this value below 2 ms.

It is important for the CCAP and each modem to have the same concept of time, in order to avoid overlaps.

The CCAP must assure that it does not assign a time slot to a modem too quickly following a request, to avoid that the modem has no time to receive the MAP message and process it, and misses the chance to use that minislot.

To avoid this situation, the CCAP uses a MAP advance timer, where it does not schedule traffic for a modem until a point in time later than the MAP advance timer.

The timing element of DOCSIS needed for upstream scheduling, is still present in R-PHY. In order to link the RPDs to the CCAP, a Converged Interconnect Network (CIN) is used, which is IP-based, and can be dedicated to cable access or shared by other applications.

The CCAP core handles upstream scheduling and generation of the MAP messages. However, the downstream and upstream signals now physically originate and terminate on the RPD, so the RPD needs to have the same concept of time as the CCAP core.

The Remote DOCSIS Timing Interface Specification (R-DTI) is the specification from CableLabs that details how this timing takes place. For Ethernet based networks, PTP is used to achieve this timing.

In the current implementation from Cisco, both the cBR-8 and the RPD act as a slave device to a PTP master clock.

Precision Time Protocol (PTP)

Introduction

PTP enables a slave clock to determine its time offset from a master clock (difference in time between the clocks), as well as the propagation delay in the transport network between the two clocks.

The master and slave devices exchange messages that include time stamps before the slave runs an algorithm to determine these values.

The formulas for this calculation assume a symmetrical connection between the two clocks.

Warning: One of the main causes of DOCSIS issues in R-PHY is created by non-symmetrical PTP links which lead to clock instability.

Non-symmetrical connections like Ethernet Passive Optical Network (EPON) are listed in the R-DTI specification for use as a CIN but rely on a different timing method, currently not supported by Cisco.

The RPD should reach the master clock via the CIN. The cBR-8 can access the master clock via Wide Area Network (WAN) interfaces on the supervisor Physical Interface Card (PIC) or via Digital PIC (DPIC) interfaces on the cable line card (the DPIC option was added in 16.8.1 release). It is recommended that the RPD does not pass through the cBR-8 to access the master clock.

The RPD and the cBR-8 can only function as slave clocks in current software, although the cBR-8 roadmap adds support for it as a grand master and boundary clock.

Note: Once the cBR-8 is configured to use PTP for timing, all line cards rely on this clock, even line cards with RF PICs.

This means that PTP clock stability issues impact all modems on a chassis, even those on Integrated CCAP (I-CCAP) line cards, when you use a mix of cards in a chassis.

Protocol

PTP is defined under IEEE standard 1588-2008.

The full specifications are available here: 1588-2008 - IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems.

Note: You need to have registered users in order to get the full access to the document.

PTP allows to distribute Time and Frequency through a network:

- Time (Synchronization): syncs the time between devices in a network.

- Frequency (Syntonization): syncs the frequency.

PTP uses either multicast or unicast and ports UDP 319 (For events) and UDP 320 (For general) messages

On CMTS implementation, PTP uses IPv4 unicast.

The protocol creates a Master-Slave relationship between a Grandmaster clock and client devices through the network. The way PTP elects a clock to be distributed in a network is with the use of an algorithm called the Best Master Clock Algorithm (BCMA).

The algorithm determines the best clock in a network with these properties:

- Identifier (Number, constructed from the MAC-Address of the device, typically looks like the EUI-64 format (xxxx:xxFF:FExx:xxxx)).

- Quality.

- clockAccuracy: Determines how accurate is the clock.The lower the better (the more accurate).

Value (hex) Specification

00-1F Reserved

20 The time is accurate to within 25 ns

21 The time is accurate to within 100 ns

22 The time is accurate to within 250 ns

23 The time is accurate to within 1 µs

24 The time is accurate to within 2.5 µs

25 The time is accurate to within 10 µs

26 The time is accurate to within 25 µs

27 The time is accurate to within 100 µs

28 The time is accurate to within 250 µs

29 The time is accurate to within 1 ms

2A The time is accurate to within 2.5 ms

2B The time is accurate to within 10 ms

2C The time is accurate to within 25 ms

2D The time is accurate to within 100 ms

2E The time is accurate to within 250 ms

2F The time is accurate to within 1 s

30 The time is accurate to within 10 s

31 The time is accurate to >10 s

32–7F Reserved

80–FD For use by alternate PTP profiles

FE Unknown

FF Reserved

- clockClass:Reflect the tracability of the time and frequency distributed by the GrandMaster clock.
Clock classes are defined by the IEEE 1588-2008 specifications as such:

clockClass (decimal) Specification

0 Reserved to enable compatibility with future versions.

1–5 Reserved.

6 Shall designate a clock that is synchronized to a primary reference time source. The timescale distributed shall be PTP. A clockClass 6 clock shall not be a slave to another clock in the domain.

7 Shall designate a clock that has previously been designated as clockClass 6 but that has lost the ability to synchronize to a primary reference time source and is in holdover mode and within holdover specifications. The timescale distributed shall be PTP. A clockClass 7 clock shall not be a slave to another clock in the domain.

8 Reserved.

9–10 Reserved to enable compatibility with future versions.

11–12 Reserved.

13 Shall designate a clock that is synchronized to an application-specific source of time. The timescale distributed shall be ARB. A clockClass 13 clock shall not be a slave to another clock in the domain.

14 Shall designate a clock that has previously been designated as clockClass 13 but that has lost the ability to synchronize to an application-specific source of time and is in holdover mode and within holdover specifications. The timescale distributed shall be ARB. A clockClass 14 clock shall not be a slave to another clock in the domain.

15–51 Reserved.

52 Degradation alternative A for a clock of clockClass 7 that is not within holdover specification. A clock of clockClass 52 shall not be a slave to another clock in the domain.

53–57 Reserved.

58 Degradation alternative A for a clock of clockClass 14 that is not within holdover specification. A clock of clockClass 58 shall not be a slave to another clock in the domain.

59–67 Reserved.

68–122 For use by alternate PTP profiles.

123–127 Reserved.

128–132 Reserved.

133–170 For use by alternate PTP profiles.

171–186 Reserved.

187 Degradation alternative B for a clock of clockClass 7 that is not within holdover specification. A clock of clockClass 187 may be a slave to another clock in the domain.

188–192 Reserved.

193 Degradation alternative B for a clock of clockClass 14 that is not within holdover specification. A clock of clockClass 193 may be a slave to another clock in the domain.

194–215 Reserved.

216–232 For use by alternate PTP profiles.

233–247 Reserved.

248 Default. This clockClass shall be used if none of the other clockClass definitions apply.

249–250 Reserved.

251 Reserved for version 1 compatibility; see Clause 18.

252–254 Reserved.

255 Shall be the clockClass of a slave-only clock; see 9.2.2.

- Priority - An administratively assigned value (Between 0-255)

- Variance - The clock's estimated stability

Master And Slave Timestamp Exchange Process

- The master sends a Sync message to the slave and notes the time (t1) at which it was sent.

- The slave receives the Sync message and notes the time of reception (t2).

- The master conveys to the slave the timestamp t1; it embeds the timestamp t1 in a Follow_Up message.

- The slave sends a Delay_Req message to the master and notes the time (t3) at which it was sent.

- The master receives the Delay_Req message and notes the time of reception (t4).

- The master conveys to the slave the timestamp t4; it embeds it in a Delay_Resp message.

This process is repeated several times per second (typically 16 to 32 times per second) in order to ensure quick adaptation in small offset changes.

GrandMaster Clock

The GrandMaster communicates with the slaves who established sessions with the grandmaster in order to exchange the synchronization (Time) and syntonization information to those slaves. a GrandMaster must in theory be connected to a PRTC (Prime Reference Time clock), such as GPS through a GPS antenna, this way, if a GrandMaster fails and an other GrandMaster takes over, since both use the same time reference, the slaves keep on using the same time reference. If you do not use a PRTC, the failure of a GrandMaster clock causes the slaves to change the time reference, which causes, in CMTS scenarios, the modems to go offline.

Slave Clock

The Slave initiates the connection to the GrandMaster clock. Both slave and master exchange their configuration settings and clock settings in order to start the negotiation. In this case, cBR-8 and RPD are both slave to an external PTP GrandMaster.

Warning: Current cBR-8 deployment (as of 16.10.1d) supports only cBR-8 as PTP slave. In the future, we might see PTP boundary or PTP master.

Boundary Clock

Boundary clock synchronize 2 network segments together. Acts as a slave to a Grand Master (GM) clock on segment 1 and then acts as a GM clock on segment 2. Non boundary clocks are referred to as 'ordinary clocks'.

Clock Classes

Clock classes are one of the values used during negotiation to find which clock, in a network with multiple clocks is the most accurate.

Clock classes are defined by IEEE 1588-2008.

Clock States

- FREERUN : Not connected to any remote GM, uses local oscillator.

- HOLDOVER : Lost connection to a remote GM, attempt to recover it, and try to keep the previous clock. During the HOLDOVER state, the clock can start drifting, and if drifting outside of specifications, it goes back to FREERUN mode.

- ACQUIRING : Started negotiation with GM, and exchange messages with the GM to determine the delay induced by the network and try to sync with the GM clock.

- FREQ_LOCKED : Slave device is locked to the Master with respect to frequency, but not phase aligned.

- PHASE_ALIGNED : Locked to the master with respect to frequency and phase.

State machine for RPD:

Note: On RPHY deployments, the supported in-spec HOLDOVER period is 10 hours (that is, when cBR-8 or RPD or both are in HOLDOVER state). During this time, the modems remain online. After 10 hours of HOLDOVER, the internal oscillator clock quality is not guaranteed and modems can drop offline due to the clock of cBR-8, RPD or both drifting out of specification.

PTP Domains

The PTP domain is a number that identifies a group of devices that talk together. Slave and master devices must be within the same PTP domain to be able to sync with each other. Domain 0 is the default domain and domains 1-2-3 are reserved per specifications. Other domain numbers can be 4-255,

Note that some PTP variants such as G.8275.2 requires the PTP domain to be within range 44-63, hence if you do not use this variant, avoid to use this range of PTP domains, as this might confuse both the user and the device.

PTP Profiles

PTP profiles were introduces in the IEEE standard 1588-2008, and consist in a set of configuration options that can be selected to meet the requirements of different applications. It is possible to define separate profiles in order to adapt PTP to different scenarios.

Examples of common PTP profiles are:

- Telecom-2008 Profile : Generic profile used before G.8265.1 specifications. This profile uses domain numbers 0-4. This profile is supported on cBR-8 and RPD however, G.8275.2 is strongly recommended over this one give that it's more resilient to failures.

- G.8265.1 : Precision time protocol telecom profile for frequency synchronization

This profile is for applications that require frequency synchronization only over telecoms networks. It does not cover phase alignment and/or time of the day.

The use case would be for PTP masters and slaves in networks where the intermediate nodes do not provide PTP support.

Note: This profile is not supported in the DOCSIS environment with cBR-8 and RPD

- G.8275.1 : Precision time protocol telecom profile for phase/time synchronization with full timing support from the network

This profile is used in systems that require accurate synchronization of time and phase in telecom networks (e.g. 4G cellular network, or RPD network) where phase and/or time-of-day synchronization is required.

With this profile, each network device participates in the PTP protocol. A boundary clock is used at every node in the chain between PTP Grandmaster and PTP Slave, which results in reduced time error accumulation through the network.

- G.8275.2 : Precision time protocol telecom profile for time/phase synchronization with partial timing support from the network

This profile is based on partial timing support from the network, which means that the nodes of the PTP domain do not need to be directly connected.

Like G.8275.1, it is used in systems that require accurate synchronization of time and phase, but allows to operate time and phase synchronisation over existing networks. It uses boundary clocks where needed, to adjust the time signal throughout the network.

Additional info on G.8275.1 and G.8275.2 for ASR900 platform can be found here: Timing and Synchronization Configuration Guide, Cisco IOS XE Everest 16.5.1 (Cisco ASR 900 Series)

Basic Messages Definition

- Sync, Follow_up, Delay_req, Delay_resp are messages used by boundary and ordinary clocks to communicate time information to slaves across the network.

- Announce messages are exchanged by both slaves and master to identify the best clock in the network using the Best Master Clock Algorithm (refer to the Images 26, 27, 28 from the IEEE specifications for the detailed mechanism).

- Signalling messages are used for non time-critical information.

Requirements

These requirements must be met for a correct deployment of PTP in a cBR-8 and R-PHY in a production network:

- Use PTP profile G.8275.2, which ensures PTP to work fine even if some network elements in the path do not support PTP

- No Equal-Cost Multi-Path (ECMP), load-balancing, or asymmetric paths: PTP always assumes that the delay from the master to the slave and vice versa are equal, and within 500 microseconds (μs)

- Configure delay-req and sync intervals with values -4 or -5 (see chapter Configuration for more details). Values greater than -4 (-3, ...) might not offer enough accuracy to detect small offset changes. Values lower than -5 (-6,...) bring a higher impact on network usage, but not a signification precision gain.

- Ensure that PTP packets for RPDs do not pass through the cBR-8

- Ensure minimal jitter in the IP network (max 1ms). Proper PTP QoS needs to be applied on all routers: cBR-8 and RPD send all PTP packets marked with Differentiated Services Code Point (DSCP) 46 (express forwarding – EF). Ensure the PTP Grand Master clock also marks the packets with the same DSCP value.

- PTP Grand Master clock must be sync'ed with GPS and report clock-class 6, for use on a production network. Lab setups can work with a standalone Freerunning PTP Grand Master (clock class 58).

- PTP Grand Master clock must have a clock accuracy of 100 nanosecond.

- If two PTP Grand Master clocks are used, you must use GPS to sync time between them. Both PTP Grand Masters must have their clocks within 500 microseconds (μs)

Note: Older releases of RPD software might use DSCP values of 47 - Newer releases use DSCP values of 46 (EF) on RPD, to align with CMTS values

Configuration

This section describes how to configure a PTP master clock on a Cisco ASR900 router, the slave clocks on cBR-8 for both cBR-8 itself and RPD, and a boundary clock on ASR900.

PTP Master Clock On ASR900

There is a Software base implementation of the PTP protocol, on Linux, called ptpd. However, since software based, it does not offer enough precision for the cBR-8 and RPD to work with it, therefore, the modems won't be able to come online and the PTP synchronization won't happened either. Moreover, the PTPd linux implementation requires hardware timestamping by the NIC in order to increase accuracy. This means that when you use a Virtual Machine or a NIC which does not support hardware timestamping, PTPd might not even start at all on Linux.

Depending on the model of ASR900 in use, it might or not have a GPS antenna. If the ASR900 does not have a GPS antenna, you do not have PRTC, but you are still able to run the ASR900 as Grandmaster with a local PRTC (internal oscillator). This means that if this ASR900 fails and that an other ASR900 takes over, the cBR-8 and RPD lose the time reference due to both clocks not being in actual sync.

Step 1. Configure The Local Internal Oscillator

network-clock source quality-level QL-PRC tx

network-clock synchronization automatic

network-clock synchronization mode QL-enabled

network-clock synchronization squelch-threshold QL-PRC

network-clock quality-level tx QL-PRC ptp domain 0

network-clock input-source 1 External R0 10m

Step 2. Configure PTP as Master on The ASR900

ptp clock ordinary domain 0                             <<< DOMAIN 0 or DOMAIN 44 for G.8275.2

clock-port MASTER master [profile g8275.2]             <<< EITHER DEFAULT OR G.8275.2 PROFILE

sync interval -4

sync one-step

transport ipv4 unicast interface Lo1588 negotiation    <<< IPV4 UNICAST MODE, SOURCING PACKETS FROM LO1588 interface

interface Loopback1588

ip address 15.88.15.88 255.255.255.255

end

Note: If there is no local oscillator or GPS configured as source, the PTP mode master is not available.

If you choose to use the G.8275.2 profile in your environment instead of the default one, you must specyfy it in the clock-port configuration (for the G.8275.2 profile configuration on the cBR-8, see section: The G.8275.2 profile).

Notice that even if IOS-XE allows to configure the G.8265.1 profile, this is not supported in the DOCSIS environment with cBR-8 and RPD.

For further reference on the G.8275.2 profile on the ASR900, you can consult this guide: Timing and Synchronization Configuration Guide, Cisco IOS XE Everest 16.5.1 (Cisco ASR 900 Series)

Verification

This section provides information that you can use in order to verify that your configuration works properly.

ASR900#show ptp clock running

PTP Ordinary Clock [Domain 0]

State Ports Pkts sent Pkts rcvd Redundancy Mode

FREQ_LOCKED 1 86307034 36108234 Hot standby

PORT SUMMARY

PTP Master

Name Tx Mode Role Transport State Sessions Port Addr

MASTER unicast master Lo1588 Master 1 -

Note: During the first configuration of internal oscillator, the oscillator needs to warm-up before to be stable. Therefore, it can take a while before the state of the PTP is FREQ_LOCKED. This can take up to 35 minutes.

ASR900#show ptp clock dataset parent

CLOCK [Ordinary Clock, domain 0]

Parent Clock Identity: 0x34:6F:90:FF:FE:C1:66:3F

Parent Port Number: 0

Parent Stats: No

Observed Parent Offset (log variance): 0

Observed Parent Clock Phase Change Rate: 0

Grandmaster Clock:

Identity: 0x34:6F:90:FF:FE:C1:66:3F

Priority1: 128

Priority2: 128

Clock Quality:

Class: 58

Accuracy: Within 1s

Offset (log variance): 52592

ASR900#show platform software ptpd stat stream 0

LOCK STATUS : FREERUN

SYNC Packet Stats

Time elapsed since last packet: 0.0

Configured Interval : 0, Acting Interval 0

Tx packets : 5577, Rx Packets : 0

Last Seq Number : 5577, Error Packets : 0

Delay Req Packet Stats

Time elapsed since last packet: 0.0

Configured Interval : 0, Acting Interval : 0

Tx packets : 0, Rx Packets : 5353

Last Seq Number : 0, Error Packets : 0

Delay Response Packet Stats

Time elapsed since last packet: 0.0

Configured Interval : 0, Acting Interval : 0

Tx packets : 5353, Rx Packets : 0

Last Seq Number : 0, Error Packets : 0

Announce Packet Stats

Time elapsed since last packet: 0.0

Configured Interval : 0, Acting Interval : 0

Tx packets : 1904, Rx Packets : 0

Last Seq Number 1904 Error Packets 0

Signalling Packet Stats

Time elapsed since last packet: 0.0

Configured Interval : 0, Acting Interval : 0

Tx packets : 1, Rx Packets : 1

Last Seq Number : 1, Error Packets : 0

Current Data Set

Offset from master : +0.0

Mean Path Delay : +0.0

Forward Path Delay : +0.0

Reverse Path Delay : +0.0

Steps Removed 0

General Stats about this stream

Packet rate : 0, Packet Delta (ns) : 0

Clock Stream handle : 0, Index : 0

Oper State : 0, Sub oper State : 6

Log mean sync Interval : 0, log mean delay req int : 0

Note: By default, the ASR900 internal oscillator reports class 58. If you use a third party GM clock, you can see clock class 6 as well if the synced with GPS

Slave Clock On cBR-8

The cBR-8 acts as the CCAP core to the RPD so it is responsible for the PTP configuration of both itself and the associated RPDs.

The cBR-8 uses profiles to get this PTP information to the RPDs, and there are multiple options for PTP that are configurable:

- The servo tracking type for the cBR-8 must be set to R-DTI to expedite clock synchronization.

- The cBR-8 uses a user-defined loopback address as the source of its PTP packets. Assure there is a proper routing for the master clock to reach the loopback interface. If you are unable to ping the master clock with a packet sourced from the loopback address, PTP is not operational.

- G.8275.2 Telecom profile is recommended as it supports IPv4 and IPv6, and intermediate switching devices in the CIN do not need to be PTP aware.

- The PTP domain number is chosen by the user but needs to be the same for the cBR-8 and the RPDs.

PTP packets are marked with higher QoS by both the RPD and the cBR-8 for priority on the CIN. DSCP value 46/EF is used by default on both.

ptp clock ordinary domain 0

servo tracking-type R-DTI

clock-port TOMASTER slave

announce interval -3

announce timeout 10

delay-req interval -4                                 <<< RECOMMENDED VALUE

sync interval -4                                      <<< RECOMMENDED VALUE

transport ipv4 unicast interface Lo1588 negotiation   <<< IPV4 UNICAST PACKETS SOURCED FROM THE LO1588 interface

clock source 15.88.15.88                              <<< THIS IS YOUR PTP MASTER

clock source 15.88.2.8 1                              <<< THIS IS THE ALTERNATE MASTER FOR PTP REDUNDANCY (OPTIONAL)

In this example, the clock-port is configured to use the default PTP profile. For the G.8275.2 profile configuration, see section: The G.8275.2 profile.

Note: The recommended value for sync and delay-req intervals is -4 (16pps) or -5 (32pps). It is recommended not to use values greater than -4 (-3,...). Announce intervals can be set to any interval lower or equal to 0 (0,-1,-2,-3).

With PTP redundancy configuration, if the master becomes unreachable the cBR-8 switches to the alternate source, and as soon as the master becomes available again, the cBR-8 reverts to the master source.

Verification

Verify with this command that the state is PHASE_ALIGNED, and that packets sent and received counters increase:

cBR-8#show ptp clock running domain 0

PTP Ordinary Clock [Domain 0]

State          Ports     Pkts sent   Pkts rcvd   Redundancy Mode

PHASE_ALIGNED  1         462249      1104590     Hot standby

PORT SUMMARY

PTP Master

Name      Tx Mode    Role    Transport   State    Sessions     Port Addr

TOMASTER  unicast    slave   Lo1588      Slave    1            15.88.15.88

SESSION INFORMATION

TOMASTER [Lo1588] [Sessions 1]

Peer addr     Pkts in    Pkts out  In Errs  Out Errs

15.88.15.88   1104590    462249    0        0

Restrictions

- At the moment, cBR-8 does not support PTP over MPLS, so if the PTP packets are MPLS tagged, the clock does not synchronize. There is an enhancement request to have this feature supported, and you can follow eventual updates at this link: CSCvj02809.

- As PTP profile you can either use the default one (as done so far), or specify the G.8275.2 profile (recommended) as described in the next section. Notice that even if IOS-XE allows to configure the G.8265.1 profile, this is not supported in the DOCSIS environment with cBR-8 and RPD.

The G.8275.2 Profile

You can configure the G.8275.2 profile on the cBR-8 slave with one GM source in this way:

ptp clock ordinary domain 44

servo tracking-type R-DTI

clock-port TOMASTER slave profile g8275.2   <<<<<<<<<<

announce interval -3

announce timeout 10

delay-req interval -4

sync interval -4

transport ipv4 unicast interface Lo1588 negotiation

clock source 15.88.15.88

Note: when PTP source is not directly connected, and there is more than one hop in between, it is recommended to use the G.8275.2 profile

As mentioned earlier in this article, PTP boundary is not yet supported on the cBR-8. However, if you want to configure the G.8275.2 profile on the cBR-8 slave with two GM sources, you need to use the boundary domain definition in this way:

ptp clock boundary domain 44

servo tracking-type R-DTI

clock-port slave1 profile g8275.2

<...>

transport ipv4 unicast interface Lo1588 negotiation

clock source 15.88.15.88        <<< THIS IS YOUR PTP MASTER

clock-port slave2 profile g8275.2

<...>

transport ipv4 unicast interface Lo1588 negotiation

clock source 15.88.2.8          <<< THIS IS THE ALTERNATE MASTER FOR PTP REDUNDANCY

Note: Despite the boundary keyword, the cBR-8 works as ordinary clock. This boundary configuration must and can only be used in this specific case: redundant PTP setup with 2 GMs using g8275.2 profile on the cBR-8 slave.

Slave Clock On RPD

Despite this is the RPD configuration, it needs to be entered on the cBR-8 itself, since the cBR-8 provisions the Remote Phy Device.

ptp r-dti 1

[profile G.8275.2]                    <-- ONLY IF SPECIFIED IN THE cBR-8 PTP CONFIGURATION

ptp-domain 0

clock-port 1

clock source ip 15.88.15.88           <-- THIS IS YOUR PTP MASTER

clock source ip 15.88.2.8 alternate   <-- THIS IS THE ALTERNATE MASTER FOR PTP REDUNDANCY (OPTIONAL)

sync interval -4

announce interval -3

Caution: ptp-domain number has to be the same as configured on the PTP master.

Caution: If the command ethernet <index> is not configured under clock-port <number>, the default ethernet index is equal to the clock-port number configured. This maps to the physical ports on the RPD (ethernet 1 maps to vbh0, ethernet 2 to vbh1). If this configuration does not match with the physical port used on the RPD, it does not sync with the clock.

Note: The intervals for sync and announce are specified in log2 scale.

Value Log calculation Value in seconds

-5   2^-5        1/32s

-4   2^-4        1/16s

-3   2^-3        1/8s

-2   2^-2        1/4s

-1   2ˆ-1        1/2s

0    2^0         1s

1    2^1         2s

2    2^2         4s

3    2^3         8s

4    2^4         16s

5    2^5         32s

Verification

These commands issued from the RPD console, can be used to check the PTP status, which must be in PHASE_LOCK and SUB_SYNC, and the sync, delay request and delay response counters which need to increese:

# ssh 10.6.17.9 -l admin

R-PHY>ena

R-PHY#show ptp clock 0 state

apr state : PHASE_LOCK       <<<

clock state : SUB_SYNC       <<<

current tod : 1506419132 Tue Sep 26 09:45:32 2017

active stream : 0

==stream 0 :

port id : 0

master ip : 15.88.15.88

stream state : PHASE_LOCK    <<< Stream state must be PHASE_LOCK

Master offset : 1212         <<< Master offset (in ns) must be as close to 0 as possible

Path delay : -81553

Forward delay : -80341       <<< Forward delay and reverse delay must be within 500us of each other

Reverse delay : -77791       <<< Forward delay and reverse delay must be within 500us of each other

Freq offset : -86279

1Hz offset : -615

R-PHY#show ptp clock 0 statistics

<output omitted>

streamId  msgType           rx          rxProcessed     lost        tx

0        SYNC              8585001     8584995         0           0          <<<<<<

0        DELAY REQUEST     0           0               0           8585000    <<<<<<

0        P-DELAY REQUEST   0           0               0           0

0        P-DELAY RESPONSE  0           0               0           0

0        FOLLOW UP         0           0               0           0

0        DELAY RESPONSE    8584998     8584998         5           0          <<<<<<

0        P-DELAY FOLLOWUP  0           0               0           0

0        ANNOUNCE          536571      536571          0           0

0        SIGNALING         5593        5593            0           5591

0        MANAGEMENT        0           0               0           0

TOTAL                    17712163    17712157        5           8590591

Note: PHASE_LOCK is the correct state when everything works. See Clock State section for other states and their definition.

Warning: There have been issues with clock stability on the RPDs with large changes in network delay between PTP master and the RPD (changes in excess of 5 ms). The RPD can fall back to freerun timing, which can cause multiple issues, like modems to go offline. RPD releases V6.7 and above, filter out large jitter packets and adjust the delay threshold to improve PTP stability.

Boundary Clock On ASR900

Assume you want to configure a boundary clock as the alternate master for cBR-8 and RPD, in case the master clock fails or becomes unreachable. This boundary clock uses a different master source for redundancy purposes (in this example 15.88.200.8). The configuration of the master clock in this scenario is not different from the one described previously, so it is omitted in this section.

ptp clock boundary domain 0

clock-port TO-MASTER slave

sync interval -5

transport ipv4 unicast interface Lo2008 negotiation

clock source 15.88.200.8      <<< THE PTP MASTER (Different from PTP master described above)

clock source 15.88.20.8 1     <<< AN ALTERNATE MASTER USED FOR REDUNDANCY (OPTIONAL)

clock-port TO-SLAVE master

transport ipv4 unicast interface Lo1588 negotiation

interface Loopback1588

ip address 15.88.2.9 255.255.255.255

end

Monitor with SNMP

In order to monitor the number of ptp sessions on the ASR900 and cBR-8 with SNMP, you can use:

Object - cPtpClockPortNumOfAssociatedPorts

OID - 1.3.6.1.4.1.9.9.760.1.2.7.1.10

Troubleshoot

This section provides information that you can use in order to troubleshoot your configuration.

Troubleshoot The PTP Master (ASR900)

On the master, the most important thing is to ensure that PTP has a network-clock source for the clocking, either a GPS antenna (preferred), or a local oscillator.

In order to ensure the network-clock source works as expected, you can use the command:

ASR900#show network-clocks synchronization

Symbols: En - Enable, Dis - Disable, Adis - Admin Disable

NA - Not Applicable

* - Synchronization source selected

# - Synchronization source force selected

& - Synchronization source manually switched

Automatic selection process : Enable

Equipment Clock : 2048 (EEC-Option1)

Clock Mode : QL-Enable

ESMC : Enabled

SSM Option : 1

T0 : Internal

Hold-off (global) : 300 ms

Wait-to-restore (global) : 300 sec

Tsm Delay : 180 ms

Revertive : No

Nominated Interfaces

Interface SigType Mode/QL Prio QL_IN ESMC Tx ESMC Rx

*Internal NA NA/Dis 251 QL-SEC NA NA   <<<<<

External R0 10M NA/Dis 1 QL-FAILED NA NA

Gi0/2/5 NA Sync/En 1 QL-FAILED QL-PRC -

Troubleshoot The PTP Slave (cBR-8)

On the cBR-8 as a slave, what is important to note is that the it only supports the SUP DPIC interfaces to connect to the PTP master (as of now), hence do not use the Gig0 interface or the RPHY PIC interfaces, as PTP might not work through those interfaces.

Note: Refer the Cisco Remote PHY Device Software Configuration Guide for more information.

During inital PTP negotiation, it can take up to 35 minutes for the cBR-8 to adjust and align its clock to the clock of the PTP master. During that time, the clock is seen in ACQUIRING state on the cBR-8:

cBR-8#show ptp clock running

PTP Ordinary Clock [Domain 0]

State      Ports     Pkts sent     Pkts rcvd     Redundancy Mode

ACQUIRING  1         687           1995          Hot standby

PORT SUMMARY

PTP Master

Name       Tx Mode     Role     Transport    State           Sessions   Port Addr

TOMASTER   unicast     slave    Lo1588       Uncalibrated    1          15.88.15.88

If the ACQUIRING state remains there for longer than 35 minutes, it might indicate that the PTP master clock is not very accurate and drifts back and forth which causes the cBR not to be able to ACQUIRE properly. This might be seen with a Linux server with PTPd for example.

The PTP clock on both the cBR-8 and the RPD must phase sync with the master before you troubleshoot any DOCSIS issues. There are a variety of commands that can show this state along with counts of packets. You want to see packets increment for sync, delay request and delay response in this output:

cBR-8#show platform software ptpd stat stream 0

LOCK STATUS : PHASE LOCKED                          <<<<<<< must be PHASE LOCKED

SYNC Packet Stats

Time elapsed since last packet: 0.0

Configured Interval : -5, Acting Interval -5

Tx packets : 0,  Rx Packets : 24074045            <<<<<<< Rx Packets must increase

Last Seq Number : 42454,  Error Packets : 0       <<<<<<< Last Seq Number must increase

Delay Req Packet Stats

Time elapsed since last packet: 0.0

Configured Interval : 0, Acting Interval : -5

Tx packets : 24077289, Rx Packets : 0             <<<<<<< Tx Packets must increase

Last Seq Number : 0, Error Packets : 0

Delay Response Packet Stats

Time elapsed since last packet: 0.0

Configured Interval : -5, Acting Interval : -5

Tx packets : 0, Rx Packets : 23983049             <<<<<<< Rx Packets must increase

Last Seq Number : 31420, Error Packets : 0        <<<<<<< Last Seq Number must increase

Announce Packet Stats

Time elapsed since last packet: 0.0

Configured Interval : -3, Acting Interval : -3

Tx packets : 0, Rx Packets :  6030915             <<<<<<< Rx Packets must increase

Last Seq Number 44276 Error Packets 0             <<<<<<< Last Seq Number must increase

Signalling Packet Stats

Time elapsed since last packet: 0.0

Configured Interval : 0, Acting Interval : 0

Tx packets : 9944, Rx Packets : 9521              <<<<<<< Tx Packets and Rx Packets must increase

Last Seq Number : 0, Error Packets : 0

<output omitted>

The stream number can be checked under the alredy introduced command show ptp clock running domain 0, under the SESSION INFORMATION section. The first session listed is stream 0, the second is stream 1, and so on.

If some of the counters do not increase, there is a chance to have a network issue and it is recommended to check for packet loss.

DTI and PTP

In order to configure PTP on the cBR-8, cable clock DTI must be DISABLED, otherwise, the this message appears:

%[PTP]: NetSync source already configured. PTP slave configuration not allowed.

Also eventual I-CMTS linecard inserted in the same chassis, rely on PTP clocking. Therefore, a potential outage on PTP GM clock can impact the modems located behind an I-CMTS linecard as well.

Clock Delays and Offset

In order to check the offset from the master clock, and what are the delays on the forward path to the master and the reverse path, you can use this command introduced before, and filter by the Current Data Set section.

The offset from master must be as close to 0 as possible, and the forward path delay must be as equal as possible to the reverse path delay.

Here is an example with good values, compared with bad values captured during a problematic condition:

----------------- GOOD -----------------

cBR-8#show platform software ptpd stat stream 0 | s Current Data Set

Current Data Set

Offset from master : -0.000000313

Mean Path Delay    : +0.000025042

Forward Path Delay : +0.000024729

Reverse Path Delay : +0.000024660

--------------- NOT GOOD ---------------

cBR-8#show platform software ptpd stat stream 0 | s Current Data Set

Current Data Set

Offset from master :  +0.002812485

Mean Path Delay    :  +0.000022503

Forward Path Delay :  +0.002834302

Reverse Path Delay :  -0.002789295

Values are expressed in seconds (hence the least significant digit, the rightmost, is nanosecond), and the offset from master is calculated as the mean path delay, minus the forward path delay.

The mean path delay is calculated as the average between forward and reverse: (forward path delay + reverse path delay) / 2.

In the ideal world, the offset from master would be 0, as the forward path delay would be equal to the reverse path delay, which makes them both equal to the mean path delay.

Depending on the asymmetry between the forward path and the reverse path, you can have a negative offset from master (if the reverse path delay is greater than the forward path delay), or a positive offset (if the reverse path delay is less than the forward path delay).

If the offset value is too big, or you observe highly fluctuating values, it is possibly a jitter problem, or a not accurate grand master clock.

The higher the jitter, the longer the RPD or cBR-8 takes to go in PHASE_ALIGNED state, and the longer it takes to recover from a HOLDOVER situation.

Multiple path setups highly influence jitter (due to the fact that some packets uses path A and some packets uses path B with different delays, which is seen by the cBR-8 and RPD as jitter), therefore, it is required for the PTP traffic to use a single path (not load-balanced across multiple links).

Troubleshoot The PTP Slave (RPD)

On the RPD, all the interesting commands are under the show ptp umbrella:

R-PHY#show ptp clock 0 state

apr state : PHASE_LOCK

clock state : SUB_SYNC

current tod : 1506426304 Tue Sep 26 11:45:04 2017

active stream : 0

==stream 0 :

port id : 0

master ip : 15.88.15.88

stream state : PHASE_LOCK

Master offset : 6010

Path delay : -78442

Forward delay : -72432

Reverse delay : -81353

Freq offset : -86206

1Hz offset : -830

R-PHY#show ptp clock 0 statistics

AprState 6 :

2@0-00:14:54.347 3@0-00:14:15.945 2@0-00:06:24.766

1@0-00:06:15.128 0@0-00:03:59.982 4@0-00:03:40.782

ClockState 5 :

5@0-00:06:49.252 4@0-00:06:46.863 3@0-00:06:43.016

2@0-00:06:25.017 1@0-00:06:24.728

BstPktStrm 3 :

0@0-00:14:45.560 4294967295@0-00:14:07.272 0@0-00:06:15.160

StepTime 1 :

406874666@0-00:05:46.080

AdjustTime 99 :

427@0-02:05:11.705 -414@0-02:04:10.705 -396@0-02:03:09.705

145@0-02:02:08.705 -157@0-02:00:06.705 327@0-01:58:04.705

-195@0-01:57:03.705 -46@0-01:56:02.705 744@0-01:55:01.705

streamId msgType rx rxProcessed lost tx

0 SYNC 246417 246417 4294770689 0

0 DELAY REQUEST 0 0 0 118272

0 P-DELAY REQUEST 0 0 0 0

0 P-DELAY RESPONSE 0 0 0 0

0 FOLLOW UP 0 0 0 0

0 DELAY RESPONSE 117165 117165 4294902867 0

0 P-DELAY FOLLOWUP 0 0 0 0

0 ANNOUNCE 82185 82184 4294901761 0

0 SIGNALING 78 78 0 78

0 MANAGEMENT 0 0 0 0

TOTAL 445845 445844 12884575317 118350

R-PHY#show ptp clock 0 config

Domain/Mode : 0/OC_SLAVE

Priority 1/2/local : 128/255/128

Profile : 001b19000100-000000 E2E

Total Ports/Streams : 1 /1

--PTP Port 1, Enet Port 1 ----

Port local Address :10.6.17.9

Unicast Duration :300 Sync Interval : -5

Announce Interval : -3 Timeout : 11

Delay-Req Intreval : -4 Pdelay-req : -4

Priority local :128 COS: 6 DSCP: 47

==Stream 0 : Port 1 Master IP: 15.88.15.88

Note: For more RPD performances troubleshoot steps, consult the article Troubleshoot RPD DOCSIS Throughput Performance Issues

Related Information

- Precision Time Protocol - Wikipedia

- 1588-2008 - IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems

- G.8265.1 : Precision time protocol telecom profile for frequency synchronization

- G.8275.1 : Precision time protocol telecom profile for phase/time synchronization with full timing support from the network

- G.8275.2 : Precision time protocol telecom profile for time/phase synchronization with partial timing support from the network

- Timing and Synchronization Configuration Guide, Cisco IOS XE Everest 16.5.1 (Cisco ASR 900 Series)

- Cisco Remote PHY Device Software Configuration Guide for Cisco 1x2 RPD Software 1.1

- Troubleshoot RPD DOCSIS Throughput Performance Issues