PTP Design Recommendations For R-PHY Networks
Available Languages
Contents
Introduction
This document describes the Precision Time Protocol (PTP) which is use in cable network with cBR-8 and Remote PHY (R-PHY) networks. The goal is to give a global understanding of the protocol and how to configure it in cBR-8 and RPHY deployments.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- R-PHY.
- cBR-8 Converged Cable Access Platform (CCAP).
Components Used
The information in this document is based on these software and hardware versions:
- cBR-8 runs 16.6.1 release or later.
- Cisco 1x2 Remote PHY Device (RPD).
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background
In order to grant modems the time slots (minislots) to transmit on a upstream channel, the CCAP maps out upstreams minislot assignments by means of upstream bandwidth allocation map (MAP) messages. These MAP messages are sent in the downstream and received by all modems.
The modems look at these messages to determine which minislots are allocated to which modems and which are for contention-based activities. A modem only transmits traffic on a minislot assigned to it (or on a contention slot if making a bandwidth request or other station maintenance activity).
The MAP messages from the CCAP allocate approximately 2 milliseconds (ms) worth of time. Low latency DOCSIS (LLD) provides options to reduce this value below 2 ms.
It is important for the CCAP and each modem to have the same concept of time, in order to avoid overlaps.
The CCAP must assure that it does not assign a time slot to a modem too quickly following a request, to avoid that the modem has no time to receive the MAP message and process it, and misses the chance to use that minislot.
To avoid this situation, the CCAP uses a MAP advance timer, where it does not schedule traffic for a modem until a point in time later than the MAP advance timer.
The timing element of DOCSIS needed for upstream scheduling, is still present in R-PHY. In order to link the RPDs to the CCAP, a Converged Interconnect Network (CIN) is used, which is IP-based, and can be dedicated to cable access or shared by other applications.
The CCAP core handles upstream scheduling and generation of the MAP messages. However, the downstream and upstream signals now physically originate and terminate on the RPD, so the RPD needs to have the same concept of time as the CCAP core.
The Remote DOCSIS Timing Interface Specification (R-DTI) is the specification from CableLabs that details how this timing takes place. For Ethernet based networks, PTP is used to achieve this timing.
In the current implementation from Cisco, both the cBR-8 and the RPD act as a slave device to a PTP master clock.
Precision Time Protocol (PTP)
Introduction
PTP enables a slave clock to determine its time offset from a master clock (difference in time between the clocks), as well as the propagation delay in the transport network between the two clocks.
The master and slave devices exchange messages that include time stamps before the slave runs an algorithm to determine these values.
The formulas for this calculation assume a symmetrical connection between the two clocks.
Non-symmetrical connections like Ethernet Passive Optical Network (EPON) are listed in the R-DTI specification for use as a CIN but rely on a different timing method, currently not supported by Cisco.
The RPD should reach the master clock via the CIN. The cBR-8 can access the master clock via Wide Area Network (WAN) interfaces on the supervisor Physical Interface Card (PIC) or via Digital PIC (DPIC) interfaces on the cable line card (the DPIC option was added in 16.8.1 release). It is recommended that the RPD does not pass through the cBR-8 to access the master clock.
The RPD and the cBR-8 can only function as slave clocks in current software, although the cBR-8 roadmap adds support for it as a grand master and boundary clock.
This means that PTP clock stability issues impact all modems on a chassis, even those on Integrated CCAP (I-CCAP) line cards, when you use a mix of cards in a chassis.
Protocol
PTP is defined under IEEE standard 1588-2008.
The full specifications are available here: 1588-2008 - IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems.
PTP allows to distribute Time and Frequency through a network:
- Time (Synchronization): syncs the time between devices in a network.
- Frequency (Syntonization): syncs the frequency.
PTP uses either multicast or unicast and ports UDP 319 (For events) and UDP 320 (For general) messages
On CMTS implementation, PTP uses IPv4 unicast.
The protocol creates a Master-Slave relationship between a Grandmaster clock and client devices through the network. The way PTP elects a clock to be distributed in a network is with the use of an algorithm called the Best Master Clock Algorithm (BCMA).
The algorithm determines the best clock in a network with these properties:
- Identifier (Number, constructed from the MAC-Address of the device, typically looks like the EUI-64 format (xxxx:xxFF:FExx:xxxx)).
- Quality.
- clockAccuracy: Determines how accurate is the clock.The lower the better (the more accurate).
Value (hex) Specification
00-1F Reserved
20 The time is accurate to within 25 ns
21 The time is accurate to within 100 ns
22 The time is accurate to within 250 ns
23 The time is accurate to within 1 µs
24 The time is accurate to within 2.5 µs
25 The time is accurate to within 10 µs
26 The time is accurate to within 25 µs
27 The time is accurate to within 100 µs
28 The time is accurate to within 250 µs
29 The time is accurate to within 1 ms
2A The time is accurate to within 2.5 ms
2B The time is accurate to within 10 ms
2C The time is accurate to within 25 ms
2D The time is accurate to within 100 ms
2E The time is accurate to within 250 ms
2F The time is accurate to within 1 s
30 The time is accurate to within 10 s
31 The time is accurate to >10 s
32–7F Reserved
80–FD For use by alternate PTP profiles
FE Unknown
FF Reserved
-
clockClass:Reflect the tracability of the time and frequency distributed by the GrandMaster clock.
Clock classes are defined by the IEEE 1588-2008 specifications as such:
clockClass (decimal) Specification
0 Reserved to enable compatibility with future versions.
1–5 Reserved.
6 Shall designate a clock that is synchronized to a primary reference time source. The timescale distributed shall be PTP. A clockClass 6 clock shall not be a slave to another clock in the domain.
7 Shall designate a clock that has previously been designated as clockClass 6 but that has lost the ability to synchronize to a primary reference time source and is in holdover mode and within holdover specifications. The timescale distributed shall be PTP. A clockClass 7 clock shall not be a slave to another clock in the domain.
8 Reserved.
9–10 Reserved to enable compatibility with future versions.
11–12 Reserved.
13 Shall designate a clock that is synchronized to an application-specific source of time. The timescale distributed shall be ARB. A clockClass 13 clock shall not be a slave to another clock in the domain.
14 Shall designate a clock that has previously been designated as clockClass 13 but that has lost the ability to synchronize to an application-specific source of time and is in holdover mode and within holdover specifications. The timescale distributed shall be ARB. A clockClass 14 clock shall not be a slave to another clock in the domain.
15–51 Reserved.
52 Degradation alternative A for a clock of clockClass 7 that is not within holdover specification. A clock of clockClass 52 shall not be a slave to another clock in the domain.
53–57 Reserved.
58 Degradation alternative A for a clock of clockClass 14 that is not within holdover specification. A clock of clockClass 58 shall not be a slave to another clock in the domain.
59–67 Reserved.
68–122 For use by alternate PTP profiles.
123–127 Reserved.
128–132 Reserved.
133–170 For use by alternate PTP profiles.
171–186 Reserved.
187 Degradation alternative B for a clock of clockClass 7 that is not within holdover specification. A clock of clockClass 187 may be a slave to another clock in the domain.
188–192 Reserved.
193 Degradation alternative B for a clock of clockClass 14 that is not within holdover specification. A clock of clockClass 193 may be a slave to another clock in the domain.
194–215 Reserved.
216–232 For use by alternate PTP profiles.
233–247 Reserved.
248 Default. This clockClass shall be used if none of the other clockClass definitions apply.
249–250 Reserved.
251 Reserved for version 1 compatibility; see Clause 18.
252–254 Reserved.
255 Shall be the clockClass of a slave-only clock; see 9.2.2.
- Priority - An administratively assigned value (Between 0-255)
- Variance - The clock's estimated stability
Master And Slave Timestamp Exchange Process
- The master sends a Sync message to the slave and notes the time (t1) at which it was sent.
- The slave receives the Sync message and notes the time of reception (t2).
- The master conveys to the slave the timestamp t1; it embeds the timestamp t1 in a Follow_Up message.
- The slave sends a Delay_Req message to the master and notes the time (t3) at which it was sent.
- The master receives the Delay_Req message and notes the time of reception (t4).
- The master conveys to the slave the timestamp t4; it embeds it in a Delay_Resp message.
This process is repeated several times per second (typically 16 to 32 times per second) in order to ensure quick adaptation in small offset changes.
GrandMaster Clock
The GrandMaster communicates with the slaves who established sessions with the grandmaster in order to exchange the synchronization (Time) and syntonization information to those slaves. a GrandMaster must in theory be connected to a PRTC (Prime Reference Time clock), such as GPS through a GPS antenna, this way, if a GrandMaster fails and an other GrandMaster takes over, since both use the same time reference, the slaves keep on using the same time reference. If you do not use a PRTC, the failure of a GrandMaster clock causes the slaves to change the time reference, which causes, in CMTS scenarios, the modems to go offline.
Slave Clock
The Slave initiates the connection to the GrandMaster clock. Both slave and master exchange their configuration settings and clock settings in order to start the negotiation. In this case, cBR-8 and RPD are both slave to an external PTP GrandMaster.
Boundary Clock
Boundary clock synchronize 2 network segments together. Acts as a slave to a Grand Master (GM) clock on segment 1 and then acts as a GM clock on segment 2. Non boundary clocks are referred to as 'ordinary clocks'.
Clock Classes
Clock classes are one of the values used during negotiation to find which clock, in a network with multiple clocks is the most accurate.
Clock classes are defined by IEEE 1588-2008.
Clock States
- FREERUN : Not connected to any remote GM, uses local oscillator.
- HOLDOVER : Lost connection to a remote GM, attempt to recover it, and try to keep the previous clock. During the HOLDOVER state, the clock can start drifting, and if drifting outside of specifications, it goes back to FREERUN mode.
- ACQUIRING : Started negotiation with GM, and exchange messages with the GM to determine the delay induced by the network and try to sync with the GM clock.
- FREQ_LOCKED : Slave device is locked to the Master with respect to frequency, but not phase aligned.
- PHASE_ALIGNED : Locked to the master with respect to frequency and phase.
State machine for RPD:
PTP Domains
The PTP domain is a number that identifies a group of devices that talk together. Slave and master devices must be within the same PTP domain to be able to sync with each other. Domain 0 is the default domain and domains 1-2-3 are reserved per specifications. Other domain numbers can be 4-255,
Note that some PTP variants such as G.8275.2 requires the PTP domain to be within range 44-63, hence if you do not use this variant, avoid to use this range of PTP domains, as this might confuse both the user and the device.
PTP Profiles
PTP profiles were introduces in the IEEE standard 1588-2008, and consist in a set of configuration options that can be selected to meet the requirements of different applications. It is possible to define separate profiles in order to adapt PTP to different scenarios.
Examples of common PTP profiles are:
- Telecom-2008 Profile : Generic profile used before G.8265.1 specifications. This profile uses domain numbers 0-4. This profile is supported on cBR-8 and RPD however, G.8275.2 is strongly recommended over this one give that it's more resilient to failures.
- G.8265.1 : Precision time protocol telecom profile for frequency synchronization
This profile is for applications that require frequency synchronization only over telecoms networks. It does not cover phase alignment and/or time of the day.
The use case would be for PTP masters and slaves in networks where the intermediate nodes do not provide PTP support.
- G.8275.1 : Precision time protocol telecom profile for phase/time synchronization with full timing support from the network
This profile is used in systems that require accurate synchronization of time and phase in telecom networks (e.g. 4G cellular network, or RPD network) where phase and/or time-of-day synchronization is required.
With this profile, each network device participates in the PTP protocol. A boundary clock is used at every node in the chain between PTP Grandmaster and PTP Slave, which results in reduced time error accumulation through the network.
- G.8275.2 : Precision time protocol telecom profile for time/phase synchronization with partial timing support from the network
This profile is based on partial timing support from the network, which means that the nodes of the PTP domain do not need to be directly connected.
Like G.8275.1, it is used in systems that require accurate synchronization of time and phase, but allows to operate time and phase synchronisation over existing networks. It uses boundary clocks where needed, to adjust the time signal throughout the network.
Additional info on G.8275.1 and G.8275.2 for ASR900 platform can be found here: Timing and Synchronization Configuration Guide, Cisco IOS XE Everest 16.5.1 (Cisco ASR 900 Series)
Basic Messages Definition
- Sync, Follow_up, Delay_req, Delay_resp are messages used by boundary and ordinary clocks to communicate time information to slaves across the network.
- Announce messages are exchanged by both slaves and master to identify the best clock in the network using the Best Master Clock Algorithm (refer to the Images 26, 27, 28 from the IEEE specifications for the detailed mechanism).
- Signalling messages are used for non time-critical information.
Requirements
These requirements must be met for a correct deployment of PTP in a cBR-8 and R-PHY in a production network:
- Use PTP profile G.8275.2, which ensures PTP to work fine even if some network elements in the path do not support PTP
- No Equal-Cost Multi-Path (ECMP), load-balancing, or asymmetric paths: PTP always assumes that the delay from the master to the slave and vice versa are equal, and within 500 microseconds (μs)
- Configure delay-req and sync intervals with values -4 or -5 (see chapter Configuration for more details). Values greater than -4 (-3, ...) might not offer enough accuracy to detect small offset changes. Values lower than -5 (-6,...) bring a higher impact on network usage, but not a signification precision gain.
- Ensure that PTP packets for RPDs do not pass through the cBR-8
- Ensure minimal jitter in the IP network (max 1ms). Proper PTP QoS needs to be applied on all routers: cBR-8 and RPD send all PTP packets marked with Differentiated Services Code Point (DSCP) 46 (express forwarding – EF). Ensure the PTP Grand Master clock also marks the packets with the same DSCP value.
- PTP Grand Master clock must be sync'ed with GPS and report clock-class 6, for use on a production network. Lab setups can work with a standalone Freerunning PTP Grand Master (clock class 58).
- PTP Grand Master clock must have a clock accuracy of 100 nanosecond.
- If two PTP Grand Master clocks are used, you must use GPS to sync time between them. Both PTP Grand Masters must have their clocks within 500 microseconds (μs)
Configuration
This section describes how to configure a PTP master clock on a Cisco ASR900 router, the slave clocks on cBR-8 for both cBR-8 itself and RPD, and a boundary clock on ASR900.
PTP Master Clock On ASR900
There is a Software base implementation of the PTP protocol, on Linux, called ptpd. However, since software based, it does not offer enough precision for the cBR-8 and RPD to work with it, therefore, the modems won't be able to come online and the PTP synchronization won't happened either. Moreover, the PTPd linux implementation requires hardware timestamping by the NIC in order to increase accuracy. This means that when you use a Virtual Machine or a NIC which does not support hardware timestamping, PTPd might not even start at all on Linux.
Depending on the model of ASR900 in use, it might or not have a GPS antenna. If the ASR900 does not have a GPS antenna, you do not have PRTC, but you are still able to run the ASR900 as Grandmaster with a local PRTC (internal oscillator). This means that if this ASR900 fails and that an other ASR900 takes over, the cBR-8 and RPD lose the time reference due to both clocks not being in actual sync.
Step 1. Configure The Local Internal Oscillator
network-clock source quality-level QL-PRC tx
network-clock synchronization automatic
network-clock synchronization mode QL-enabled
network-clock synchronization squelch-threshold QL-PRC
network-clock quality-level tx QL-PRC ptp domain 0
network-clock input-source 1 External R0 10m
Step 2. Configure PTP as Master on The ASR900
ptp clock ordinary domain 0 <<< DOMAIN 0 or DOMAIN 44 for G.8275.2
clock-port MASTER master [profile g8275.2] <<< EITHER DEFAULT OR G.8275.2 PROFILE
sync interval -4
sync one-step
transport ipv4 unicast interface Lo1588 negotiation <<< IPV4 UNICAST MODE, SOURCING PACKETS FROM LO1588 interface
interface Loopback1588
ip address 15.88.15.88 255.255.255.255
end
If you choose to use the G.8275.2 profile in your environment instead of the default one, you must specyfy it in the clock-port configuration (for the G.8275.2 profile configuration on the cBR-8, see section: The G.8275.2 profile).
Notice that even if IOS-XE allows to configure the G.8265.1 profile, this is not supported in the DOCSIS environment with cBR-8 and RPD.
For further reference on the G.8275.2 profile on the ASR900, you can consult this guide: Timing and Synchronization Configuration Guide, Cisco IOS XE Everest 16.5.1 (Cisco ASR 900 Series)
Verification
This section provides information that you can use in order to verify that your configuration works properly.
ASR900#show ptp clock running
PTP Ordinary Clock [Domain 0]
State Ports Pkts sent Pkts rcvd Redundancy Mode
FREQ_LOCKED 1 86307034 36108234 Hot standby
PORT SUMMARY
PTP Master
Name Tx Mode Role Transport State Sessions Port Addr
MASTER unicast master Lo1588 Master 1 -
ASR900#show ptp clock dataset parent
CLOCK [Ordinary Clock, domain 0]
Parent Clock Identity: 0x34:6F:90:FF:FE:C1:66:3F
Parent Port Number: 0
Parent Stats: No
Observed Parent Offset (log variance): 0
Observed Parent Clock Phase Change Rate: 0
Grandmaster Clock:
Identity: 0x34:6F:90:FF:FE:C1:66:3F
Priority1: 128
Priority2: 128
Clock Quality:
Class: 58
Accuracy: Within 1s
Offset (log variance): 52592
ASR900#show platform software ptpd stat stream 0
LOCK STATUS : FREERUN
SYNC Packet Stats
Time elapsed since last packet: 0.0
Configured Interval : 0, Acting Interval 0
Tx packets : 5577, Rx Packets : 0
Last Seq Number : 5577, Error Packets : 0
Delay Req Packet Stats
Time elapsed since last packet: 0.0
Configured Interval : 0, Acting Interval : 0
Tx packets : 0, Rx Packets : 5353
Last Seq Number : 0, Error Packets : 0
Delay Response Packet Stats
Time elapsed since last packet: 0.0
Configured Interval : 0, Acting Interval : 0
Tx packets : 5353, Rx Packets : 0
Last Seq Number : 0, Error Packets : 0
Announce Packet Stats
Time elapsed since last packet: 0.0
Configured Interval : 0, Acting Interval : 0
Tx packets : 1904, Rx Packets : 0
Last Seq Number 1904 Error Packets 0
Signalling Packet Stats
Time elapsed since last packet: 0.0
Configured Interval : 0, Acting Interval : 0
Tx packets : 1, Rx Packets : 1
Last Seq Number : 1, Error Packets : 0
Current Data Set
Offset from master : +0.0
Mean Path Delay : +0.0
Forward Path Delay : +0.0
Reverse Path Delay : +0.0
Steps Removed 0
General Stats about this stream
Packet rate : 0, Packet Delta (ns) : 0
Clock Stream handle : 0, Index : 0
Oper State : 0, Sub oper State : 6
Log mean sync Interval : 0, log mean delay req int : 0
Slave Clock On cBR-8
The cBR-8 acts as the CCAP core to the RPD so it is responsible for the PTP configuration of both itself and the associated RPDs.
The cBR-8 uses profiles to get this PTP information to the RPDs, and there are multiple options for PTP that are configurable:
- The servo tracking type for the cBR-8 must be set to R-DTI to expedite clock synchronization.
- The cBR-8 uses a user-defined loopback address as the source of its PTP packets. Assure there is a proper routing for the master clock to reach the loopback interface. If you are unable to ping the master clock with a packet sourced from the loopback address, PTP is not operational.
- G.8275.2 Telecom profile is recommended as it supports IPv4 and IPv6, and intermediate switching devices in the CIN do not need to be PTP aware.
- The PTP domain number is chosen by the user but needs to be the same for the cBR-8 and the RPDs.
PTP packets are marked with higher QoS by both the RPD and the cBR-8 for priority on the CIN. DSCP value 46/EF is used by default on both.
ptp clock ordinary domain 0
servo tracking-type R-DTI
clock-port TOMASTER slave
announce interval -3
announce timeout 10
delay-req interval -4 <<< RECOMMENDED VALUE
sync interval -4 <<< RECOMMENDED VALUE
transport ipv4 unicast interface Lo1588 negotiation <<< IPV4 UNICAST PACKETS SOURCED FROM THE LO1588 interface
clock source 15.88.15.88 <<< THIS IS YOUR PTP MASTER
clock source 15.88.2.8 1 <<< THIS IS THE ALTERNATE MASTER FOR PTP REDUNDANCY (OPTIONAL)
In this example, the clock-port is configured to use the default PTP profile. For the G.8275.2 profile configuration, see section: The G.8275.2 profile.
With PTP redundancy configuration, if the master becomes unreachable the cBR-8 switches to the alternate source, and as soon as the master becomes available again, the cBR-8 reverts to the master source.
Verification
Verify with this command that the state is PHASE_ALIGNED, and that packets sent and received counters increase:
cBR-8#show ptp clock running domain 0
PTP Ordinary Clock [Domain 0]
State Ports Pkts sent Pkts rcvd Redundancy Mode
PHASE_ALIGNED 1 462249 1104590 Hot standby
PORT SUMMARY
PTP Master
Name Tx Mode Role Transport State Sessions Port Addr
TOMASTER unicast slave Lo1588 Slave 1 15.88.15.88
SESSION INFORMATION
TOMASTER [Lo1588] [Sessions 1]
Peer addr Pkts in Pkts out In Errs Out Errs
15.88.15.88 1104590 462249 0 0
Restrictions
- At the moment, cBR-8 does not support PTP over MPLS, so if the PTP packets are MPLS tagged, the clock does not synchronize. There is an enhancement request to have this feature supported, and you can follow eventual updates at this link: CSCvj02809.
- As PTP profile you can either use the default one (as done so far), or specify the G.8275.2 profile (recommended) as described in the next section. Notice that even if IOS-XE allows to configure the G.8265.1 profile, this is not supported in the DOCSIS environment with cBR-8 and RPD.
The G.8275.2 Profile
You can configure the G.8275.2 profile on the cBR-8 slave with one GM source in this way:
ptp clock ordinary domain 44 servo tracking-type R-DTI clock-port TOMASTER slave profile g8275.2 <<<<<<<<<< announce interval -3
announce timeout 10
delay-req interval -4
sync interval -4 transport ipv4 unicast interface Lo1588 negotiation clock source 15.88.15.88
As mentioned earlier in this article, PTP boundary is not yet supported on the cBR-8. However, if you want to configure the G.8275.2 profile on the cBR-8 slave with two GM sources, you need to use the boundary domain definition in this way:
ptp clock boundary domain 44 servo tracking-type R-DTI clock-port slave1 profile g8275.2 <...> transport ipv4 unicast interface Lo1588 negotiation clock source 15.88.15.88 <<< THIS IS YOUR PTP MASTER clock-port slave2 profile g8275.2 <...> transport ipv4 unicast interface Lo1588 negotiation clock source 15.88.2.8 <<< THIS IS THE ALTERNATE MASTER FOR PTP REDUNDANCY
Slave Clock On RPD
Despite this is the RPD configuration, it needs to be entered on the cBR-8 itself, since the cBR-8 provisions the Remote Phy Device.
ptp r-dti 1
[profile G.8275.2] <-- ONLY IF SPECIFIED IN THE cBR-8 PTP CONFIGURATION
ptp-domain 0
clock-port 1
clock source ip 15.88.15.88 <-- THIS IS YOUR PTP MASTER
clock source ip 15.88.2.8 alternate <-- THIS IS THE ALTERNATE MASTER FOR PTP REDUNDANCY (OPTIONAL)
sync interval -4
announce interval -3
Value Log calculation Value in seconds
-5 2^-5 1/32s
-4 2^-4 1/16s
-3 2^-3 1/8s
-2 2^-2 1/4s
-1 2ˆ-1 1/2s
0 2^0 1s
1 2^1 2s
2 2^2 4s
3 2^3 8s
4 2^4 16s
5 2^5 32s
Verification
These commands issued from the RPD console, can be used to check the PTP status, which must be in PHASE_LOCK and SUB_SYNC, and the sync, delay request and delay response counters which need to increese:
# ssh 10.6.17.9 -l admin
R-PHY>ena
R-PHY#show ptp clock 0 state
apr state : PHASE_LOCK <<<
clock state : SUB_SYNC <<<
current tod : 1506419132 Tue Sep 26 09:45:32 2017
active stream : 0
==stream 0 :
port id : 0
master ip : 15.88.15.88
stream state : PHASE_LOCK <<< Stream state must be PHASE_LOCK
Master offset : 1212 <<< Master offset (in ns) must be as close to 0 as possible
Path delay : -81553
Forward delay : -80341 <<< Forward delay and reverse delay must be within 500us of each other
Reverse delay : -77791 <<< Forward delay and reverse delay must be within 500us of each other
Freq offset : -86279
1Hz offset : -615
R-PHY#show ptp clock 0 statistics <output omitted> streamId msgType rx rxProcessed lost tx 0 SYNC 8585001 8584995 0 0 <<<<<< 0 DELAY REQUEST 0 0 0 8585000 <<<<<< 0 P-DELAY REQUEST 0 0 0 0 0 P-DELAY RESPONSE 0 0 0 0 0 FOLLOW UP 0 0 0 0 0 DELAY RESPONSE 8584998 8584998 5 0 <<<<<< 0 P-DELAY FOLLOWUP 0 0 0 0 0 ANNOUNCE 536571 536571 0 0 0 SIGNALING 5593 5593 0 5591 0 MANAGEMENT 0 0 0 0 TOTAL 17712163 17712157 5 8590591
Boundary Clock On ASR900
Assume you want to configure a boundary clock as the alternate master for cBR-8 and RPD, in case the master clock fails or becomes unreachable. This boundary clock uses a different master source for redundancy purposes (in this example 15.88.200.8). The configuration of the master clock in this scenario is not different from the one described previously, so it is omitted in this section.
ptp clock boundary domain 0 clock-port TO-MASTER slave sync interval -5 transport ipv4 unicast interface Lo2008 negotiation clock source 15.88.200.8 <<< THE PTP MASTER (Different from PTP master described above) clock source 15.88.20.8 1 <<< AN ALTERNATE MASTER USED FOR REDUNDANCY (OPTIONAL) clock-port TO-SLAVE master transport ipv4 unicast interface Lo1588 negotiation interface Loopback1588 ip address 15.88.2.9 255.255.255.255 end
Monitor with SNMP
In order to monitor the number of ptp sessions on the ASR900 and cBR-8 with SNMP, you can use:
Object - cPtpClockPortNumOfAssociatedPorts
OID - 1.3.6.1.4.1.9.9.760.1.2.7.1.10
Troubleshoot
This section provides information that you can use in order to troubleshoot your configuration.
Troubleshoot The PTP Master (ASR900)
On the master, the most important thing is to ensure that PTP has a network-clock source for the clocking, either a GPS antenna (preferred), or a local oscillator.
In order to ensure the network-clock source works as expected, you can use the command:
ASR900#show network-clocks synchronization
Symbols: En - Enable, Dis - Disable, Adis - Admin Disable
NA - Not Applicable
* - Synchronization source selected
# - Synchronization source force selected
& - Synchronization source manually switched
Automatic selection process : Enable
Equipment Clock : 2048 (EEC-Option1)
Clock Mode : QL-Enable
ESMC : Enabled
SSM Option : 1
T0 : Internal
Hold-off (global) : 300 ms
Wait-to-restore (global) : 300 sec
Tsm Delay : 180 ms
Revertive : No
Nominated Interfaces
Interface SigType Mode/QL Prio QL_IN ESMC Tx ESMC Rx
*Internal NA NA/Dis 251 QL-SEC NA NA <<<<<
External R0 10M NA/Dis 1 QL-FAILED NA NA
Gi0/2/5 NA Sync/En 1 QL-FAILED QL-PRC -
Troubleshoot The PTP Slave (cBR-8)
On the cBR-8 as a slave, what is important to note is that the it only supports the SUP DPIC interfaces to connect to the PTP master (as of now), hence do not use the Gig0 interface or the RPHY PIC interfaces, as PTP might not work through those interfaces.
During inital PTP negotiation, it can take up to 35 minutes for the cBR-8 to adjust and align its clock to the clock of the PTP master. During that time, the clock is seen in ACQUIRING state on the cBR-8:
cBR-8#show ptp clock running
PTP Ordinary Clock [Domain 0]
State Ports Pkts sent Pkts rcvd Redundancy Mode
ACQUIRING 1 687 1995 Hot standby
PORT SUMMARY
PTP Master
Name Tx Mode Role Transport State Sessions Port Addr
TOMASTER unicast slave Lo1588 Uncalibrated 1 15.88.15.88
If the ACQUIRING state remains there for longer than 35 minutes, it might indicate that the PTP master clock is not very accurate and drifts back and forth which causes the cBR not to be able to ACQUIRE properly. This might be seen with a Linux server with PTPd for example.
The PTP clock on both the cBR-8 and the RPD must phase sync with the master before you troubleshoot any DOCSIS issues. There are a variety of commands that can show this state along with counts of packets. You want to see packets increment for sync, delay request and delay response in this output:
cBR-8#show platform software ptpd stat stream 0 LOCK STATUS : PHASE LOCKED <<<<<<< must be PHASE LOCKED SYNC Packet Stats Time elapsed since last packet: 0.0 Configured Interval : -5, Acting Interval -5 Tx packets : 0, Rx Packets : 24074045 <<<<<<< Rx Packets must increase Last Seq Number : 42454, Error Packets : 0 <<<<<<< Last Seq Number must increase Delay Req Packet Stats Time elapsed since last packet: 0.0 Configured Interval : 0, Acting Interval : -5 Tx packets : 24077289, Rx Packets : 0 <<<<<<< Tx Packets must increase Last Seq Number : 0, Error Packets : 0 Delay Response Packet Stats Time elapsed since last packet: 0.0 Configured Interval : -5, Acting Interval : -5 Tx packets : 0, Rx Packets : 23983049 <<<<<<< Rx Packets must increase Last Seq Number : 31420, Error Packets : 0 <<<<<<< Last Seq Number must increase Announce Packet Stats Time elapsed since last packet: 0.0 Configured Interval : -3, Acting Interval : -3 Tx packets : 0, Rx Packets : 6030915 <<<<<<< Rx Packets must increase Last Seq Number 44276 Error Packets 0 <<<<<<< Last Seq Number must increase Signalling Packet Stats Time elapsed since last packet: 0.0 Configured Interval : 0, Acting Interval : 0 Tx packets : 9944, Rx Packets : 9521 <<<<<<< Tx Packets and Rx Packets must increase Last Seq Number : 0, Error Packets : 0 <output omitted>
The stream number can be checked under the alredy introduced command show ptp clock running domain 0, under the SESSION INFORMATION section. The first session listed is stream 0, the second is stream 1, and so on.
If some of the counters do not increase, there is a chance to have a network issue and it is recommended to check for packet loss.
DTI and PTP
In order to configure PTP on the cBR-8, cable clock DTI must be DISABLED, otherwise, the this message appears:
%[PTP]: NetSync source already configured. PTP slave configuration not allowed.
Also eventual I-CMTS linecard inserted in the same chassis, rely on PTP clocking. Therefore, a potential outage on PTP GM clock can impact the modems located behind an I-CMTS linecard as well.
Clock Delays and Offset
In order to check the offset from the master clock, and what are the delays on the forward path to the master and the reverse path, you can use this command introduced before, and filter by the Current Data Set section.
The offset from master must be as close to 0 as possible, and the forward path delay must be as equal as possible to the reverse path delay.
Here is an example with good values, compared with bad values captured during a problematic condition:
----------------- GOOD -----------------
cBR-8#show platform software ptpd stat stream 0 | s Current Data Set
Current Data Set
Offset from master : -0.000000313
Mean Path Delay : +0.000025042
Forward Path Delay : +0.000024729
Reverse Path Delay : +0.000024660
--------------- NOT GOOD ---------------
cBR-8#show platform software ptpd stat stream 0 | s Current Data Set Current Data Set Offset from master : +0.002812485 Mean Path Delay : +0.000022503 Forward Path Delay : +0.002834302 Reverse Path Delay : -0.002789295
Values are expressed in seconds (hence the least significant digit, the rightmost, is nanosecond), and the offset from master is calculated as the mean path delay, minus the forward path delay.
The mean path delay is calculated as the average between forward and reverse: (forward path delay + reverse path delay) / 2.
In the ideal world, the offset from master would be 0, as the forward path delay would be equal to the reverse path delay, which makes them both equal to the mean path delay.
Depending on the asymmetry between the forward path and the reverse path, you can have a negative offset from master (if the reverse path delay is greater than the forward path delay), or a positive offset (if the reverse path delay is less than the forward path delay).
If the offset value is too big, or you observe highly fluctuating values, it is possibly a jitter problem, or a not accurate grand master clock.
The higher the jitter, the longer the RPD or cBR-8 takes to go in PHASE_ALIGNED state, and the longer it takes to recover from a HOLDOVER situation.
Multiple path setups highly influence jitter (due to the fact that some packets uses path A and some packets uses path B with different delays, which is seen by the cBR-8 and RPD as jitter), therefore, it is required for the PTP traffic to use a single path (not load-balanced across multiple links).
Troubleshoot The PTP Slave (RPD)
On the RPD, all the interesting commands are under the show ptp umbrella:
R-PHY#show ptp clock 0 state
apr state : PHASE_LOCK
clock state : SUB_SYNC
current tod : 1506426304 Tue Sep 26 11:45:04 2017
active stream : 0
==stream 0 :
port id : 0
master ip : 15.88.15.88
stream state : PHASE_LOCK
Master offset : 6010
Path delay : -78442
Forward delay : -72432
Reverse delay : -81353
Freq offset : -86206
1Hz offset : -830
R-PHY#show ptp clock 0 statistics
AprState 6 :
2@0-00:14:54.347 3@0-00:14:15.945 2@0-00:06:24.766
1@0-00:06:15.128 0@0-00:03:59.982 4@0-00:03:40.782
ClockState 5 :
5@0-00:06:49.252 4@0-00:06:46.863 3@0-00:06:43.016
2@0-00:06:25.017 1@0-00:06:24.728
BstPktStrm 3 :
0@0-00:14:45.560 4294967295@0-00:14:07.272 0@0-00:06:15.160
StepTime 1 :
406874666@0-00:05:46.080
AdjustTime 99 :
427@0-02:05:11.705 -414@0-02:04:10.705 -396@0-02:03:09.705
145@0-02:02:08.705 -157@0-02:00:06.705 327@0-01:58:04.705
-195@0-01:57:03.705 -46@0-01:56:02.705 744@0-01:55:01.705
streamId msgType rx rxProcessed lost tx
0 SYNC 246417 246417 4294770689 0
0 DELAY REQUEST 0 0 0 118272
0 P-DELAY REQUEST 0 0 0 0
0 P-DELAY RESPONSE 0 0 0 0
0 FOLLOW UP 0 0 0 0
0 DELAY RESPONSE 117165 117165 4294902867 0
0 P-DELAY FOLLOWUP 0 0 0 0
0 ANNOUNCE 82185 82184 4294901761 0
0 SIGNALING 78 78 0 78
0 MANAGEMENT 0 0 0 0
TOTAL 445845 445844 12884575317 118350
R-PHY#show ptp clock 0 config
Domain/Mode : 0/OC_SLAVE
Priority 1/2/local : 128/255/128
Profile : 001b19000100-000000 E2E
Total Ports/Streams : 1 /1
--PTP Port 1, Enet Port 1 ----
Port local Address :10.6.17.9
Unicast Duration :300 Sync Interval : -5
Announce Interval : -3 Timeout : 11
Delay-Req Intreval : -4 Pdelay-req : -4
Priority local :128 COS: 6 DSCP: 47
==Stream 0 : Port 1 Master IP: 15.88.15.88
Related Information
- Precision Time Protocol - Wikipedia
- 1588-2008 - IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems
- G.8265.1 : Precision time protocol telecom profile for frequency synchronization
- G.8275.1 : Precision time protocol telecom profile for phase/time synchronization with full timing support from the network
- G.8275.2 : Precision time protocol telecom profile for time/phase synchronization with partial timing support from the network
- Timing and Synchronization Configuration Guide, Cisco IOS XE Everest 16.5.1 (Cisco ASR 900 Series)
- Cisco Remote PHY Device Software Configuration Guide for Cisco 1x2 RPD Software 1.1
- Troubleshoot RPD DOCSIS Throughput Performance Issues
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)