Reporting that it appears to me that the tx_timestamp_timeout variable in the DU profile for ZTP / RAN is too low for CVL nics:  https://github.com/openshift-kni/cnf-features-deploy/blob/release-4.11/ztp/source-crs/PtpConfigSlaveCvl.yaml#L61

Deploying this on a fresh 4.11.13 cluster in my lab results in the OpenShift PTP process constantly flapping, see actual results section.


My PGT is as follows:

    - fileName: PtpConfigSlaveCvl.yaml   # Change to PtpConfigSlaveCvl.yaml for ColumbiaVille NIC
      policyName: "config-policy"
      metadata:
        name: "du-ptp-slave"
      spec:
        profile:
        - name: "slave"
          # This interface must match the hardware in this group
          interface: "ens4f0"
          ptp4lOpts: "-2 -s --summary_interval -4"
          phc2sysOpts: "-a -r -n 24"

My NIC is as follows:
$ ethtool -i ens4f0
driver: ice
version: 4.18.0-372.32.1.rt7.189.el8_6.x
firmware-version: 4.00 0x800118b3 21.5.9
expansion-rom-version: 
bus-info: 0000:8a:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

My assumption is the defaults for a slave configuration would be appropriate out of the box.  The reference configuration for a customer (and my interactions with Intel) have this at 50ms.  If I change my ptpconfig to be 50ms my PTP is stable in my lab (see expected results section of this Jira).

4.11.13

everytime

1. Deploy node through ZTP / PGT
2. Wait for policy enforcement and installation of openshift-ptp Operator and its corresponding ptpconfig CR
3. Look at logs for linuxptp-daemon-container

ptp4l[1816.162]: [ptp4l.0.config] selected /dev/ptp5 as PTP clock
phc2sys[1816.162]: [ptp4l.0.config] Waiting for ptp4l...
ptp4l[1816.185]: [ptp4l.0.config] port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[1816.185]: [ptp4l.0.config] port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[1816.185]: [ptp4l.0.config] port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[1816.269]: [ptp4l.0.config] port 1: new foreign master 8c47be.ffff.90ab8f-7
ptp4l[1816.519]: [ptp4l.0.config] selected best master clock 84c807.fffe.fb9e0d
ptp4l[1816.519]: [ptp4l.0.config] port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[1816.646]: [ptp4l.0.config] master offset       -767 s0 freq   +2819 path delay       526
ptp4l[1816.710]: [ptp4l.0.config] master offset       -760 s0 freq   +2819 path delay       527
ptp4l[1816.766]: [ptp4l.0.config] master offset       -765 s0 freq   +2819 path delay       527
ptp4l[1816.830]: [ptp4l.0.config] master offset       -775 s2 freq   +2775 path delay       536
ptp4l[1816.830]: [ptp4l.0.config] port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[1816.894]: [ptp4l.0.config] master offset       -764 s2 freq   +1471 path delay       536
ptp4l[1816.958]: [ptp4l.0.config] master offset       -695 s2 freq   +1513 path delay       543
ptp4l[1817.022]: [ptp4l.0.config] master offset       -599 s2 freq   +1608 path delay       536
ptp4l[1817.078]: [ptp4l.0.config] master offset       -536 s2 freq   +1657 path delay       536
ptp4l[1817.142]: [ptp4l.0.config] master offset       -457 s2 freq   +1739 path delay       529
phc2sys[1817.163]: [ptp4l.0.config] Waiting for ptp4l...
ptp4l[1817.206]: [ptp4l.0.config] master offset       -383 s2 freq   +1820 path delay       528
ptp4l[1817.272]: [ptp4l.0.config] master offset       -334 s2 freq   +1865 path delay       544
ptp4l[1817.334]: [ptp4l.0.config] master offset       -279 s2 freq   +1926 path delay       544
ptp4l[1817.390]: [ptp4l.0.config] master offset       -224 s2 freq   +1992 path delay       537
ptp4l[1817.454]: [ptp4l.0.config] master offset       -155 s2 freq   +2088 path delay       529
ptp4l[1817.518]: [ptp4l.0.config] master offset       -110 s2 freq   +2150 path delay       529
ptp4l[1817.582]: [ptp4l.0.config] master offset        -64 s2 freq   +2217 path delay       524
ptp4l[1817.646]: [ptp4l.0.config] master offset        -28 s2 freq   +2272 path delay       523
ptp4l[1817.710]: [ptp4l.0.config] master offset          8 s2 freq   +2331 path delay       527
ptp4l[1817.766]: [ptp4l.0.config] master offset         31 s2 freq   +2371 path delay       527
ptp4l[1817.830]: [ptp4l.0.config] master offset         72 s2 freq   +2444 path delay       527
ptp4l[1817.894]: [ptp4l.0.config] master offset         93 s2 freq   +2487 path delay       527
ptp4l[1817.958]: [ptp4l.0.config] master offset        118 s2 freq   +2539 path delay       527
ptp4l[1818.022]: [ptp4l.0.config] master offset        127 s2 freq   +2566 path delay       527
ptp4l[1818.078]: [ptp4l.0.config] master offset        141 s2 freq   +2603 path delay       525
ptp4l[1818.142]: [ptp4l.0.config] master offset        162 s2 freq   +2652 path delay       525
phc2sys[1818.174]: [ptp4l.0.config] reconfiguring after port state change
phc2sys[1818.174]: [ptp4l.0.config] selecting CLOCK_REALTIME for synchronization
phc2sys[1818.174]: [ptp4l.0.config] selecting ens4f0 as the master clock
phc2sys[1818.174]: [ptp4l.0.config] CLOCK_REALTIME rms  473 max  473 freq -16030 +/-   0 delay   636 +/-   0
ptp4l[1818.206]: [ptp4l.0.config] timed out while polling for tx timestamp
ptp4l[1818.206]: [ptp4l.0.config] increasing tx_timestamp_timeout may correct this issue, but it is likely caused by a driver bug
ptp4l[1818.206]: [ptp4l.0.config] port 1: send delay request failed
ptp4l[1818.206]: [ptp4l.0.config] port 1: clearing fault immediately
ptp4l[1818.239]: [ptp4l.0.config] port 1: SLAVE to LISTENING on INIT_COMPLETE
ptp4l[1818.270]: [ptp4l.0.config] port 1: new foreign master 8c47be.ffff.90ab8f-7
ptp4l[1818.518]: [ptp4l.0.config] port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[1818.582]: [ptp4l.0.config] master offset        221 s2 freq   +2769 path delay       529
ptp4l[1818.582]: [ptp4l.0.config] port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[1818.647]: [ptp4l.0.config] master offset        230 s2 freq   +2806 path delay       529
ptp4l[1818.710]: [ptp4l.0.config] master offset        227 s2 freq   +2824 path delay       532
ptp4l[1818.766]: [ptp4l.0.config] master offset        221 s2 freq   +2836 path delay       535
ptp4l[1818.830]: [ptp4l.0.config] master offset        213 s2 freq   +2844 path delay       540
ptp4l[1818.894]: [ptp4l.0.config] master offset        206 s2 freq   +2854 path delay       545
ptp4l[1818.958]: [ptp4l.0.config] master offset        213 s2 freq   +2886 path delay       540
ptp4l[1819.022]: [ptp4l.0.config] master offset        209 s2 freq   +2900 path delay       541
ptp4l[1819.078]: [ptp4l.0.config] master offset        198 s2 freq   +2902 path delay       541
ptp4l[1819.143]: [ptp4l.0.config] master offset        194 s2 freq   +2915 path delay       538
phc2sys[1819.175]: [ptp4l.0.config] port b49691.fffe.894844-1 changed state
phc2sys[1819.175]: [ptp4l.0.config] port b49691.fffe.894844-1 changed state
phc2sys[1819.175]: [ptp4l.0.config] port b49691.fffe.894844-1 changed state
ptp4l[1819.180]: [ptp4l.0.config] timed out while polling for tx timestamp
ptp4l[1819.180]: [ptp4l.0.config] increasing tx_timestamp_timeout may correct this issue, but it is likely caused by a driver bug
ptp4l[1819.180]: [ptp4l.0.config] port 1: send delay request failed
ptp4l[1819.180]: [ptp4l.0.config] port 1: clearing fault immediately
ptp4l[1819.220]: [ptp4l.0.config] port 1: SLAVE to LISTENING on INIT_COMPLETE

stable slave PTP offset / synchronization

phc2sys[1152.069]: [ptp4l.0.config] Waiting for ptp4l...
ptp4l[1152.075]: [ptp4l.0.config] master offset        424 s2 freq   +3858 path delay       545
ptp4l[1152.137]: [ptp4l.0.config] master offset        366 s2 freq   +3801 path delay       551
ptp4l[1152.233]: [ptp4l.0.config] master offset        297 s2 freq   +3720 path delay       554
ptp4l[1152.268]: [ptp4l.0.config] master offset        241 s2 freq   +3654 path delay       554
ptp4l[1152.330]: [ptp4l.0.config] master offset        182 s2 freq   +3577 path delay       559
ptp4l[1152.385]: [ptp4l.0.config] master offset        141 s2 freq   +3525 path delay       559
ptp4l[1152.450]: [ptp4l.0.config] master offset        109 s2 freq   +3484 path delay       559
ptp4l[1152.515]: [ptp4l.0.config] master offset         70 s2 freq   +3428 path delay       559
ptp4l[1152.579]: [ptp4l.0.config] master offset         26 s2 freq   +3360 path delay       560
ptp4l[1152.642]: [ptp4l.0.config] master offset          2 s2 freq   +3322 path delay       560
ptp4l[1152.783]: [ptp4l.0.config] master offset        -34 s2 freq   +3260 path delay       560
ptp4l[1152.783]: [ptp4l.0.config] master offset        -54 s2 freq   +3223 path delay       560
ptp4l[1152.825]: [ptp4l.0.config] master offset        -70 s2 freq   +3190 path delay       560
ptp4l[1152.891]: [ptp4l.0.config] master offset       -105 s2 freq   +3124 path delay       560
ptp4l[1152.953]: [ptp4l.0.config] master offset       -113 s2 freq   +3100 path delay       555
ptp4l[1153.045]: [ptp4l.0.config] master offset       -114 s2 freq   +3087 path delay       551
ptp4l[1153.128]: [ptp4l.0.config] master offset       -138 s2 freq   +3034 path delay       549
ptp4l[1153.146]: [ptp4l.0.config] master offset       -155 s2 freq   +2992 path delay       549
phc2sys[1153.146]: [ptp4l.0.config] reconfiguring after port state change
phc2sys[1153.146]: [ptp4l.0.config] selecting CLOCK_REALTIME for synchronization
phc2sys[1153.149]: [ptp4l.0.config] selecting ens4f0 as the master clock
phc2sys[1153.149]: [ptp4l.0.config] CLOCK_REALTIME rms   80 max   80 freq -16035 +/-   0 delay   626 +/-   0
...

phc2sys[1661.384]: [ptp4l.0.config] CLOCK_REALTIME rms    0 max   -0 freq -16049 +/-   0 delay   618 +/-   0
ptp4l[1661.389]: [ptp4l.0.config] master offset          0 s2 freq   +2840 path delay       540
ptp4l[1661.453]: [ptp4l.0.config] master offset          9 s2 freq   +2855 path delay       540
ptp4l[1661.517]: [ptp4l.0.config] master offset         -3 s2 freq   +2835 path delay       549
ptp4l[1661.583]: [ptp4l.0.config] master offset          8 s2 freq   +2854 path delay       542
ptp4l[1661.645]: [ptp4l.0.config] master offset          8 s2 freq   +2855 path delay       542
ptp4l[1661.709]: [ptp4l.0.config] master offset         -2 s2 freq   +2838 path delay       542
ptp4l[1661.765]: [ptp4l.0.config] master offset         -2 s2 freq   +2838 path delay       541
ptp4l[1661.829]: [ptp4l.0.config] master offset         -5 s2 freq   +2833 path delay       539
ptp4l[1661.893]: [ptp4l.0.config] master offset         -4 s2 freq   +2834 path delay       538
ptp4l[1661.957]: [ptp4l.0.config] master offset         -5 s2 freq   +2832 path delay       538
ptp4l[1662.021]: [ptp4l.0.config] master offset         -3 s2 freq   +2835 path delay       538
ptp4l[1662.077]: [ptp4l.0.config] master offset         -2 s2 freq   +2836 path delay       537
ptp4l[1662.141]: [ptp4l.0.config] master offset          0 s2 freq   +2839 path delay       537
ptp4l[1662.209]: [ptp4l.0.config] master offset          0 s2 freq   +2839 path delay       537
ptp4l[1662.270]: [ptp4l.0.config] master offset          3 s2 freq   +2845 path delay       537
ptp4l[1662.333]: [ptp4l.0.config] master offset         -3 s2 freq   +2835 path delay       536