PTP from scratch: from switch to OpenShift configuration
Table of Contents
What would you do if you were given a new Arista ToR switch and a baremetal Openshift cluster and had to configure PTP on some of the nodes?
Well, if you had no idea at all of what the Precision Time Protocol is (as I had) you would start by checking out the Arista vendor instructions to enable it in the node interfaces and would finally turn to the PTP Operator documentation in Openshift to learn how to configure its CRDs. That is also what I did, so check it out!
Boundary Clock introduction #
There are two levels of configuration: global configuration at the switch level and interface configuration.
Global configuration includes general parameters for the ptp to be configured. Interface configuration includes specific interfaces where PTP packets will be received (Slave) and sent (Master).
When the switch acts as a boundary clock, there are two interfaces playing a role in PTP: a slave interface, that will read the clock timestamp from the PTP cable; and master interfaces, that will act as a grandmaster clock to other server’s slave interface.
Global switch configuration #
First, initial commands to configure ptp domain, priority, mode, and profile.
tor-sw-210#config
tor-sw-210(config)# ptp domain 24
tor-sw-210(config)# ptp profile g8275.1
tor-sw-210(config)# ptp mode boundary
tor-sw-210(config)# exit
Check the running configuration shows it corrrectly
tor-sw-210# show ptp | grep ptp
ptp domain 24
ptp local-priority 200
ptp mode boundary
ptp profile g8275.1
Slave interface configuration #
Then, configure each interface as desired.
Note: Et13/1 was already configured at the article initial point. GPS Cable was connected to Et13/1 and therefore switch is Slave, reads the hardware timestamp in this interface and acts as Slave for the GrandMaster clock at the other side of the cable.
interface Ethernet13/1
description "uplink to core ptp switch"
speed forced 10000full
switchport access vlan 3
ptp enable
ptp transport layer2
tor-sw-210#show ptp
PTP Mode: Boundary Clock
PTP Profile: G8275.1
Clock Identity: 0x94:8e:d3:ff:ff:ed:a1:89
Grandmaster Clock Identity: 0x00:b0:ae:ff:fe:07:6f:b8
Number of slave ports: 1
Number of master ports: 1
Slave port: Ethernet13/1
Offset From Master (nanoseconds): 7
Mean Path Delay (nanoseconds): 179
Steps Removed: 2
Skew (estimated local-to-master clock frequency ratio): 1.0000052959672074
Last Sync Time: 15:05:36 UTC Aug 25 2023
Current PTP System Time: 15:05:36 UTC Aug 25 2023
Interface State Transport Delay
Mechanism
--------------- ------------ --------------- ---------
Et13/1 Slave layer2 e2e
A specific vlan for PTP was also already given.
vlan 3, named ptp_vlan, was configured in Et13/1
tor-sw-210(config)#show vlan
VLAN Name Status Ports
----- ------------------------------------
1 default active Et1/1, Et1/2, Et1/4, Et2/1
Et2/2, Et2/4, Et3/1, Et3/2
Et3/4, Et4/1, Et4/2, Et4/4
Et5/2, Et5/4, Et7/1, Et7/2
Et7/3, Et7/4, Et8/2, Et8/3
Et8/4, Et9/1, Et9/2, Et9/3
Et9/4, Et10/1, Et10/2, Et10/3
Et10/4, Et11/2, Et11/4, Et12/1
Et12/4, Et18/1, Et19/1, Et20/1
Et22/1, Et23/1, Et24/1, Et26/1
Et27/1, Et28/1, Et30/1, Et31/1
Et32/1
3 ptp_vlan active Et13/1
PTP protocol can go through both IPv4 or Layer 2. When done through IPv4, a source ip address must be defined in the ptp global configuration.
When done through Layer 2, do not define source ip address. IP level won’t be used.
Use ptp transport layer 2 as an option to each interface
Master interface configuration #
Let’s first test PTP Conectivity with RAN worker 2.
Let’s match the server MAC Addresses to the Switch Interfaces:
sh-4.4# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN m
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
link/ether b4:96:91:c8:74:b0 brd ff:ff:ff:ff:ff:ff
3: ens5f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
link/ether 88:e9:a4:45:e3:c6 brd ff:ff:ff:ff:ff:ff
4: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
link/ether b4:96:91:c8:74:b1 brd ff:ff:ff:ff:ff:ff
5: ens5f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
link/ether 88:e9:a4:45:e3:c7 brd ff:ff:ff:ff:ff:ff
altname enp132s0f1np1
6: ens10f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether b4:96:91:d6:c4:64 brd ff:ff:ff:ff:ff:ff
altname enp72s0f0
7: ens4f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether 88:e9:a4:45:04:c6 brd ff:ff:ff:ff:ff:ff
altname enp162s0f0np0
tor-sw-210#show mac address-table
Mac Address Table
------------------------------------------------------------------
Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
1 3cec.ef79.7b07 DYNAMIC Et12/4 1 0:10:01 ago
1 3cec.ef79.7b08 DYNAMIC Et12/1 1 0:11:22 ago
1 88e9.a445.0497 DYNAMIC Et30/1 1 0:07:12 ago
1 88e9.a445.049a DYNAMIC Et31/1 1 0:07:12 ago
1 b496.91c8.74b0 DYNAMIC Et8/1 1 0:07:12 ago
27 88e9.a445.04c6 DYNAMIC Et21/1 1 19:23:56 ago
And now, check the DNS DHCP server config file to find the Openshift baremetal network interface being already used.
dhcp-host=88:e9:a4:45:04:c6,192.168.27.73,<ptpnode>.domain.com
With all this information, we have now two different options: share the same interface for both cluster- and PTP network, which is on the one hand a good idea not to dedicate a high throughput interface only for PTP traffic; or, on the other hand, having unused extra interfaces it may be interesting to separate the traffic and leave the cluster network isolated.
In our case, this last alternative made more sense, so the following configuration will be applied:
| Same worker | Interface | Comment |
|---|---|---|
| MAC A | Et21/1 | Dnsmasq —> Openshift baremetal Network |
| MAC B | Et8/1 | Available for PTP configuration |
Going now back to the switch configuration, configure the master interface:
tor-sw-210#sh run interface Et8/1
interface Ethernet21/1
speed forced 25gfull
switchport access vlan 3
ptp enable
ptp role master
ptp transport layer2
Usually we would configure the interface to be in both vlans by setting it to trunk mode.
However, the Arista switches in trunk mode were not performing as expected and therefore it was decided to keep each interface in a single vlan. Et21/1 in cert4 vlan and Et8/1 in ptp_vlan
tor-sw-210#show running-config int Et8/1
interface Ethernet26/1
description "ptp node interface"
speed forced 100gfull
switchport access vlan 3
ptp enable
ptp role master
ptp transport layer2
tor-sw-210#show running-config int Et21/1
interface Ethernet21/1
description "ocp bm ntw interface"
speed forced 100gfull
switchport access vlan 27
ptp transport layer2
tor-sw-210#config
tor-sw-210(config)#interface Et8/1
tor-sw-210(config-int-Et21/1)#shutdown
tor-sw-210(config-int-Et21/1)#no shutdown
tor-sw-210(config-int-Et21/1)#write
tor-sw-210#show ptp
PTP Mode: Boundary Clock
PTP Profile: G8275.1
Clock Identity: 0x94:8e:d3:ff:ff:ed:a1:89
Grandmaster Clock Identity: 0x00:b0:ae:ff:fe:07:6f:b8
Number of slave ports: 1
Number of master ports: 1
Slave port: Ethernet13/1
Offset From Master (nanoseconds): 7
Mean Path Delay (nanoseconds): 179
Steps Removed: 2
Skew (estimated local-to-master clock frequency ratio): 1.0000052959672074
Last Sync Time: 15:05:36 UTC Aug 25 2023
Current PTP System Time: 15:05:36 UTC Aug 25 2023
Interface State Transport Delay
Mechanism
--------------- ------------ --------------- ---------
Et8/1 Master layer2 e2e
Et13/1 Slave layer2 e2e
tor-sw-210(config)#show vlan
VLAN Name Status Ports
----- -------------------------------- --------- -------------------------------
1 default active Et1/1, Et1/2, Et1/4, Et2/1
Et2/2, Et2/4, Et3/1, Et3/2
Et3/4, Et4/1, Et4/2, Et4/4
Et5/2, Et5/4, Et7/1, Et7/2
Et7/3, Et7/4, Et8/2, Et8/3
Et8/4, Et9/1, Et9/2, Et9/3
Et9/4, Et10/1, Et10/2, Et10/3
Et10/4, Et11/2, Et11/4, Et12/1
Et12/4, Et18/1, Et19/1, Et20/1
Et22/1, Et23/1, Et24/1, Et26/1
Et27/1, Et28/1, Et30/1, Et31/1
Et32/1
3 ptp_vlan active Et8/1, Et13/1
For the Openshift cluster, another vlan was created with the interfaces connected to the baremetal network, called cert4 vlan (id 27). Then, iLO and Uplink were also differentiated.
Note: at this point, no isolation is taking place among any vlan.
tor-sw-210(config)#show vlan
VLAN Name Status Ports
----- -------------------------------- --------- -------------------------------
1 default active Et1/1, Et1/2, Et1/4, Et2/1
Et2/2, Et2/4, Et3/1, Et3/2
Et3/4, Et4/1, Et4/2, Et4/4
Et5/1, Et5/2, Et5/4, Et7/1
Et7/2, Et7/3, Et7/4, Et8/2
Et8/2, Et8/3, Et8/4, Et9/1
Et9/2, Et9/3, Et9/4, Et10/1
Et10/2, Et10/3, Et10/4, Et11/2
Et11/4, Et12/1, Et12/4, Et18/1
Et19/1, Et20/1, Et22/1, Et23/1
Et24/1, Et26/1, Et27/1, Et28/1
Et30/1, Et31/1, Et32/1
3 ptp_vlan active Et13/1, Et8/1
27 cert4 active Cpu, Et1/3, Et2/3, Et3/3, Et4/3
Et5/3, Et11/1, Et11/3, Et12/2
Et12/3, Et17/1, Et21/1, Et25/1
Et29/1, Et35/1
1027 cert4_ilo_net active Cpu, Et35/1
3027 uplink active Cpu, Et36/1
PTP Operator configuration #
Now that the switch is correctly configured and that we are able to see PTP packets in the OCP node servers, let’s configure Openshift to take these new measurements into account.
Installing the PTP Operator #
Installation steps #
- Manually create ns and operatorgroup
apiVersion: v1
kind: Namespace
metadata:
name: openshift-ptp
annotations:
workload.openshift.io/allowed: management
labels:
name: openshift-ptp
openshift.io/cluster-monitoring: "true"
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: ptp-operators
namespace: openshift-ptp
spec:
targetNamespaces:
- openshift-ptp
- From Operator Hub install now the PTP Operator
Select openshift-ptp namespace already created
- PTP deploys linuxptp-daemon pod on each node
The Linux PTP implementation of the PTP runs the linuxptp and phc2sys processes.
Disabling NTP server (chronyd) through MachineConfigPool #
Start by labeling the target PTP node to be configured with an additional role.
oc label node <ptpnode> node-role.kubernetes.io/infra=
And then create a MachineConfigPool targetting both roles (worker and the new role) and selecting the specific host.
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
labels:
pools.operator.machineconfiguration.openshift.io/worker: ""
name: ptp
spec:
machineConfigSelector:
matchExpressions:
- {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]}
nodeSelector:
matchLabels:
kubernetes.io/hostname: <ptpnode>
Create a MachineConfig that disables NTP server in the nodes with the new role created:
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: infra
name: 500-disable-chronyd
spec:
config:
ignition:
version: 3.2.0
systemd:
units:
- contents: |
[Unit]
Description=NTP client/server
Documentation=man:chronyd(8) man:chrony.conf(5)
After=ntpdate.service sntp.service ntpd.service
Conflicts=ntpd.service systemd-timesyncd.service
ConditionCapability=CAP_SYS_TIME
[Service]
Type=forking
PIDFile=/run/chrony/chronyd.pid
EnvironmentFile=-/etc/sysconfig/chronyd
ExecStart=/usr/sbin/chronyd $OPTIONS
ExecStartPost=/usr/libexec/chrony-helper update-daemon
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=full
[Install]
WantedBy=multi-user.target
enabled: false
name: "chronyd.service"
Checking it is disabled
$ oc debug node/<ptpnode> To use host binaries, run `chroot /host` Pod IP: 192.168.27.73 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host sh-5.1# systemctl status chronyd ○ chronyd.service - NTP client/server Loaded: loaded (/etc/systemd/system/chronyd.service; disabled; preset: enabled) Drop-In: /usr/lib/systemd/system/chronyd.service.d └─platform-chrony.conf Active: inactive (dead) Docs: man:chronyd(8) man:chrony.conf(5)
Configuring the PTP Operator #
The two main CRD provided by the PTP Operator need to be configured as explained below:
PtpOperatorConfig selects the nodes where to configure PTP
# Example of all worker nodes PTP config apiVersion: ptp.openshift.io/v1 kind: PtpOperatorConfig metadata: name: default namespace: openshift-ptp spec: daemonNodeSelector: node-role.kubernetes.io/worker: ""# Example of a single node PTP config apiVersion: ptp.openshift.io/v1 kind: PtpOperatorConfig metadata: name: default namespace: openshift-ptp spec: daemonNodeSelector: kubernetes.io/hostname: <ptpnode>PtpConfig configures all needed params
apiVersion: ptp.openshift.io/v1 kind: PtpConfig metadata: name: ordinary-clock-ptp-config namespace: openshift-ptp spec: profile: - name: ordinary-clock-<ptpnode> interface: "ens1f0" phc2sysOpts: "-a -r -n 24" ptp4lOpts: "-2 -s" ptpSchedulingPolicy: SCHED_FIFO ptpSchedulingPriority: 10 ptp4lConf: | [global] # # Default Data Set # twoStepFlag 1 slaveOnly 1 priority1 128 priority2 128 domainNumber 24 clockClass 255 clockAccuracy 0xFE offsetScaledLogVariance 0xFFFF free_running 0 freq_est_interval 1 dscp_event 0 dscp_general 0 dataset_comparison G.8275.x G.8275.defaultDS.localPriority 128 # # Port Data Set # logAnnounceInterval -3 logSyncInterval -4 logMinDelayReqInterval -4 logMinPdelayReqInterval -4 announceReceiptTimeout 3 syncReceiptTimeout 0 delayAsymmetry 0 fault_reset_interval 4 neighborPropDelayThresh 20000000 masterOnly 0 G.8275.portDS.localPriority 128 # # Run time options # assume_two_step 0 logging_level 6 path_trace_enabled 0 follow_up_info 0 hybrid_e2e 0 inhibit_multicast_service 0 net_sync_monitor 0 tc_spanning_tree 0 tx_timestamp_timeout 50 unicast_listen 0 unicast_master_table 0 unicast_req_duration 3600 use_syslog 1 verbose 0 summary_interval 0 kernel_leap 1 check_fup_sync 0 # # Servo Options # pi_proportional_const 0.0 pi_integral_const 0.0 pi_proportional_scale 0.0 pi_proportional_exponent -0.3 pi_proportional_norm_max 0.7 pi_integral_scale 0.0 pi_integral_exponent 0.4 pi_integral_norm_max 0.3 step_threshold 2.0 first_step_threshold 0.00002 max_frequency 900000000 clock_servo pi sanity_freq_limit 200000000 ntpshm_segment 0 # # Transport options # transportSpecific 0x0 ptp_dst_mac 01:1B:19:00:00:00 p2p_dst_mac 01:80:C2:00:00:0E udp_ttl 1 udp6_scope 0x0E uds_address /var/run/ptp4l # # Default interface options # clock_type OC network_transport L2 delay_mechanism E2E time_stamping hardware tsproc_mode filter delay_filter moving_median delay_filter_length 10 egressLatency 0 ingressLatency 0 boundary_clock_jbod 0 # # Clock description # productDescription ;; revisionData ;; manufacturerIdentity 00:00:00 userDescription ; timeSource 0xA0 recommend: - profile: ordinary-clock-<ptpnode> priority: 4 match: - nodeName: "<ptpnode>"
Monitor the PTP Logs #
Check the logs in the linuxptp pod running in the worker node.
oc logs -f linuxptp-daemon-22qb4 -c linuxptp-daemon-container
phc2sys[13151.138]: [ptp4l.0.config] CLOCK_REALTIME rms 3 max 3 freq +750 +/- 0 delay 487 +/- 0
ptp4l[13151.812]: [ptp4l.0.config] rms 4 max 10 freq -22641 +/- 7 delay 142 +/- 1
phc2sys[13152.138]: [ptp4l.0.config] CLOCK_REALTIME rms 19 max 19 freq +733 +/- 0 delay 487 +/- 0
ptp4l[13153.012]: [ptp4l.0.config] rms 6 max 11 freq -22642 +/- 9 delay 143 +/- 1
phc2sys[13153.138]: [ptp4l.0.config] CLOCK_REALTIME rms 5 max 5 freq +741 +/- 0 delay 487 +/- 0
ptp4l[13154.017]: [ptp4l.0.config] rms 5 max 9 freq -22639 +/- 9 delay 141 +/- 1
phc2sys[13154.139]: [ptp4l.0.config] CLOCK_REALTIME rms 7 max 7 freq +752 +/- 0 delay 488 +/- 0
ptp4l[13155.027]: [ptp4l.0.config] rms 5 max 12 freq -22640 +/- 8 delay 143 +/- 1
phc2sys[13155.139]: [ptp4l.0.config] CLOCK_REALTIME rms 15 max 15 freq +762 +/- 0 delay 488 +/- 0
ptp4l[13156.032]: [ptp4l.0.config] rms 3 max 7 freq -22643 +/- 5 delay 140 +/- 1
phc2sys[13156.139]: [ptp4l.0.config] CLOCK_REALTIME rms 14 max 14 freq +737 +/- 0 delay 487 +/- 0
ptp4l[13157.041]: [ptp4l.0.config] rms 4 max 9 freq -22640 +/- 7 delay 142 +/- 1
$ oc rsh -c linuxptp-daemon-container linuxptp-daemon-22qb4
sh-4.4# pmc -u -f /var/run/ptp4l.0.config -b 0 'GET PORT_DATA_SET'
sending: GET PORT_DATA_SET
88e9a4.fffe.4504c6-1 seq 0 RESPONSE MANAGEMENT PORT_DATA_SET
portIdentity 88e9a4.fffe.4504c6-1
portState SLAVE
logMinDelayReqInterval -4
peerMeanPathDelay 0
logAnnounceInterval -3
announceReceiptTimeout 3
logSyncInterval -4
delayMechanism 1
logMinPdelayReqInterval -4
versionNumber 2
Wrap up #
The Precision Time Protocol hast taken a step forward as the IEEE Standard Synchronization Protocol for Networked Control Systems in high-accuracy scenarios. While NTP achieves milliseconds accuracy, PTP reaches sub-microsecond level.
In this post, PTP is first configured in an Arista ToR Switch that connects all the OCP cluster nodes and it then explains how to install and configure the PTP Operator in Openshift. Hope it helped!