-
Bug
-
Resolution: Done-Errata
-
Undefined
-
None
-
4.14
-
No
-
False
-
-
-
8/16: triaged; this will be investigated after the crash loop
Description of problem:
Regression test is failing with ptp-operator.v4.14.0-202308110126.
ptp clock is not locked in GM config:
[kni@registry.ran-vcl01 ~]$ oc -n openshift-ptp exec linuxptp-daemon-p4qbs -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics [...] # TYPE openshift_ptp_clock_state gauge openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 0 openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="GM"} 1 openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 1 openshift_ptp_clock_state{iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 0 openshift_ptp_clock_state{iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0 openshift_ptp_clock_state{iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0
linuxptp-daemon-container log and ptp metrics show abnormal offsets:
[kni@registry.ran-vcl01 ~]$ oc logs -n openshift-ptp linuxptp-daemon-p4qbs linuxptp-daemon-container [...] phc2sys[7049.323]: [ptp4l.1.config] CLOCK_REALTIME phc offset 586447 s2 freq -1850083 delay 470 ptp4l[7049.324]: [ptp4l.3.config] master offset -102903 s2 freq +60868 path delay -279 ptp4l[7049.382]: [ptp4l.4.config] master offset -102569 s2 freq +49637 path delay 52 ptp4l[7049.384]: [ptp4l.2.config] master offset -102605 s2 freq +52041 path delay -40 ptp4l[7049.385]: [ptp4l.0.config] master offset -102550 s2 freq +49243 path delay -263 phc2sys[7049.386]: [ptp4l.1.config] CLOCK_REALTIME phc offset 700758 s2 freq -1559838 delay 494 ptp4l[7049.386]: [ptp4l.3.config] master offset -102353 s2 freq +51623 path delay -279 ptp4l[7049.446]: [ptp4l.4.config] master offset -101463 s2 freq +41375 path delay 52 ptp4l[7049.446]: [ptp4l.2.config] master offset -101492 s2 freq +43787 path delay -40 ni@registry.ran-vcl01 ~]$ oc -n openshift-ptp exec linuxptp-daemon-p4qbs -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics [...] # HELP openshift_ptp_max_offset_ns # TYPE openshift_ptp_max_offset_ns gauge openshift_ptp_max_offset_ns{from="master",iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1.034912482e+09 openshift_ptp_max_offset_ns{from="master",iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 5.22828677e+08 openshift_ptp_max_offset_ns{from="master",iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 5.23460734e+08 openshift_ptp_max_offset_ns{from="phc",iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 93 # HELP openshift_ptp_offset_ns # TYPE openshift_ptp_offset_ns gauge openshift_ptp_offset_ns{from="gnss",iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 5 openshift_ptp_offset_ns{from="master",iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} -2.71623279e+08 openshift_ptp_offset_ns{from="master",iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} -1.01563882e+08 openshift_ptp_offset_ns{from="master",iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} -1.05411205e+08 openshift_ptp_offset_ns{from="phc",iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 2.87460889e+08
Version-Release number of selected component (if applicable):
Cluster version is 4.14.0-0.nightly-2023-08-11-055332 ptp-operator.v4.14.0-202308110126
How reproducible:
100%
Steps to Reproduce:
1. Deploy SNO with PTP GM configuration 2. Check ptp metrics: [kni@registry.ran-vcl01 ~]$ oc -n openshift-ptp exec linuxptp-daemon-p4qbs -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics [...] # TYPE openshift_ptp_clock_state gauge openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 0 openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="GM"} 1 openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 1 openshift_ptp_clock_state{iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 0 openshift_ptp_clock_state{iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0 openshift_ptp_clock_state{iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0 [...] # HELP openshift_ptp_max_offset_ns # TYPE openshift_ptp_max_offset_ns gauge openshift_ptp_max_offset_ns{from="master",iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1.034912482e+09 openshift_ptp_max_offset_ns{from="master",iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 5.22828677e+08 openshift_ptp_max_offset_ns{from="master",iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 5.23460734e+08 openshift_ptp_max_offset_ns{from="phc",iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 93 # HELP openshift_ptp_offset_ns # TYPE openshift_ptp_offset_ns gauge openshift_ptp_offset_ns{from="gnss",iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 5 openshift_ptp_offset_ns{from="master",iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} -2.71623279e+08 openshift_ptp_offset_ns{from="master",iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} -1.01563882e+08 openshift_ptp_offset_ns{from="master",iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} -1.05411205e+08 openshift_ptp_offset_ns{from="phc",iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 2.87460889e+08
Actual results:
ptp clock is not in sync. offsets exceed clock lock.
Expected results:
ptp clock is in sync. offsets within clock lock range.
Additional info:
- relates to
-
OCPBUGS-18360 ptp clock does not stay locked in GM configuration, offsets are very high
- Closed
- links to
-
RHEA-2023:5006 rpm