Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Undefined
Fix Version/s: None
Affects Version/s: 4.14
Component/s: Networking / ptp
Labels:

Severity:
Moderate
Regression:
No
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Internal Whiteboard:
Latest Status Summary:
10/19: Works as designed , This is happening because when GM is forced to FREERUN , the clock class changes to 248 and slave has clock class threshold set to 7 . That makes slave ports stuck in Listening and you wont see offset coming up

SFDC Cases Counter:
SFDC Cases Links:

Description of problem:

In MC ptp environment:
Master offsets for slave interfaces are missing from linuxptp daemon logs when ptpClockThreshold maxOffsetThreshold/minOffsetThreshold are set to very small number such as 1/-1, 2/-2.

The causes ptp metrics master offsets and clock states to be stuck with stale numbers/state.

Version-Release number of selected component (if applicable):

[kni@registry.ran-vcl01 cnf-gotests]$ oc get csv -n openshift-ptp 
NAME                                DISPLAY        VERSION               
ptp-operator.v4.14.0-202310140046   PTP Operator   4.14.0-202310140046

[kni@registry.ran-vcl01 cnf-gotests]$ oc get clusterversions.config.openshift.io 
NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.0-rc.6   True        False         32h     Cluster version is 4.14.0-rc.6

How reproducible:

100%

Steps to Reproduce:

1. Deploy SNO with MC/slave ptpconfigs
example:
[kni@registry.ran-vcl01 cnf-gotests]$ oc get ptpconfigs.ptp.openshift.io -A
NAMESPACE       NAME               AGE
openshift-ptp   du-ptp-gm          3h23m
openshift-ptp   du-ptp-slave-gm1   3h23m
openshift-ptp   du-ptp-slave-gm2   3h23m
openshift-ptp   du-ptp-slave-gm3   3h23m
openshift-ptp   du-ptp-slave-gm4   3h23m

2. Edit ptpconfigs to include very small ptpClockThreshold values.  
example:
      ptpClockThreshold:
        holdOverTimeout: 120
        maxOffsetThreshold: 2
        minOffsetThreshold: -2

[kni@registry.ran-vcl01 ~]$ oc get ptpconfigs.ptp.openshift.io -n openshift-ptp -o yaml | grep ptpClockThreshold: -A3

      ptpClockThreshold:
        holdOverTimeout: 10
        maxOffsetThreshold: 2
        minOffsetThreshold: -2
--
      ptpClockThreshold:
        holdOverTimeout: 10
        maxOffsetThreshold: 2
        minOffsetThreshold: -2
--
      ptpClockThreshold:
        holdOverTimeout: 10
        maxOffsetThreshold: 2
        minOffsetThreshold: -2
--
      ptpClockThreshold:
        holdOverTimeout: 10
        maxOffsetThreshold: 2
        minOffsetThreshold: -2
--
      ptpClockThreshold:
        holdOverTimeout: 10
        maxOffsetThreshold: 2
        minOffsetThreshold: -2


3. Check linuxptp-daemon logs for master offsets.  Only GM master offsets are reported:

[kni@registry.ran-vcl01 ~]$ oc logs -n openshift-ptp linuxptp-daemon-wbbtq linuxptp-daemon-container --since=10s | grep offset 
phc2sys[4594.382]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -8 s2 freq  -90790 delay    562
phc2sys[4594.444]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -9 s2 freq  -90793 delay    562
phc2sys[4594.507]: [ptp4l.2.config] CLOCK_REALTIME phc offset         4 s2 freq  -90783 delay    565
phc2sys[4594.570]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -9 s2 freq  -90795 delay    564
phc2sys[4594.632]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -1 s2 freq  -90790 delay    563
phc2sys[4594.695]: [ptp4l.2.config] CLOCK_REALTIME phc offset         4 s2 freq  -90785 delay    566
phc2sys[4594.757]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -6 s2 freq  -90794 delay    566
phc2sys[4594.820]: [ptp4l.2.config] CLOCK_REALTIME phc offset         8 s2 freq  -90781 delay    566
ts2phc[4594.856]: [ts2phc.2.config] ens3f0 master offset          0 s2 freq      +0
phc2sys[4594.883]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90776 delay    562
phc2sys[4594.945]: [ptp4l.2.config] CLOCK_REALTIME phc offset         7 s2 freq  -90777 delay    561
phc2sys[4595.008]: [ptp4l.2.config] CLOCK_REALTIME phc offset         6 s2 freq  -90776 delay    564
phc2sys[4595.070]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -7 s2 freq  -90787 delay    563
phc2sys[4595.133]: [ptp4l.2.config] CLOCK_REALTIME phc offset         7 s2 freq  -90775 delay    565
gnss[1697573180]:[ts2phc.2.config] ens3f0 gnss_status 3 offset 5 s0
phc2sys[4595.195]: [ptp4l.2.config] CLOCK_REALTIME phc offset         8 s2 freq  -90772 delay    560
phc2sys[4595.258]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90766 delay    567
phc2sys[4595.321]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90763 delay    562
phc2sys[4595.383]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90766 delay    562
phc2sys[4595.446]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90764 delay    565
phc2sys[4595.508]: [ptp4l.2.config] CLOCK_REALTIME phc offset         3 s2 freq  -90765 delay    566
phc2sys[4595.571]: [ptp4l.2.config] CLOCK_REALTIME phc offset         1 s2 freq  -90766 delay    566
phc2sys[4595.634]: [ptp4l.2.config] CLOCK_REALTIME phc offset       -10 s2 freq  -90777 delay    565
phc2sys[4595.696]: [ptp4l.2.config] CLOCK_REALTIME phc offset       -13 s2 freq  -90783 delay    566
phc2sys[4595.759]: [ptp4l.2.config] CLOCK_REALTIME phc offset         3 s2 freq  -90771 delay    563
phc2sys[4595.821]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90768 delay    563
ts2phc[4595.856]: [ts2phc.2.config] ens3f0 master offset          0 s2 freq      +0

4. Check clock states in ptp metrics. Clock states will report stale states:

[kni@registry.ran-vcl01 ~]$ oc exec -it ds/linuxptp-daemon  -n openshift-ptp -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics | grep clock_state
# HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
# TYPE openshift_ptp_clock_state gauge
openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 0
openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="GM"} 0
openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 0
openshift_ptp_clock_state{iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1
openshift_ptp_clock_state{iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0
openshift_ptp_clock_state{iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0

Actual results:

linuxptp-daemon logs only shows GM master offsets:

[kni@registry.ran-vcl01 ~]$ oc logs -n openshift-ptp linuxptp-daemon-wbbtq linuxptp-daemon-container --since=10s | grep offset 
phc2sys[4594.382]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -8 s2 freq  -90790 delay    562
phc2sys[4594.444]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -9 s2 freq  -90793 delay    562
phc2sys[4594.507]: [ptp4l.2.config] CLOCK_REALTIME phc offset         4 s2 freq  -90783 delay    565
phc2sys[4594.570]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -9 s2 freq  -90795 delay    564
phc2sys[4594.632]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -1 s2 freq  -90790 delay    563
phc2sys[4594.695]: [ptp4l.2.config] CLOCK_REALTIME phc offset         4 s2 freq  -90785 delay    566
phc2sys[4594.757]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -6 s2 freq  -90794 delay    566
phc2sys[4594.820]: [ptp4l.2.config] CLOCK_REALTIME phc offset         8 s2 freq  -90781 delay    566
ts2phc[4594.856]: [ts2phc.2.config] ens3f0 master offset          0 s2 freq      +0
phc2sys[4594.883]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90776 delay    562
phc2sys[4594.945]: [ptp4l.2.config] CLOCK_REALTIME phc offset         7 s2 freq  -90777 delay    561
phc2sys[4595.008]: [ptp4l.2.config] CLOCK_REALTIME phc offset         6 s2 freq  -90776 delay    564
phc2sys[4595.070]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -7 s2 freq  -90787 delay    563
phc2sys[4595.133]: [ptp4l.2.config] CLOCK_REALTIME phc offset         7 s2 freq  -90775 delay    565
gnss[1697573180]:[ts2phc.2.config] ens3f0 gnss_status 3 offset 5 s0
phc2sys[4595.195]: [ptp4l.2.config] CLOCK_REALTIME phc offset         8 s2 freq  -90772 delay    560
phc2sys[4595.258]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90766 delay    567
phc2sys[4595.321]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90763 delay    562
phc2sys[4595.383]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90766 delay    562
phc2sys[4595.446]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90764 delay    565
phc2sys[4595.508]: [ptp4l.2.config] CLOCK_REALTIME phc offset         3 s2 freq  -90765 delay    566
phc2sys[4595.571]: [ptp4l.2.config] CLOCK_REALTIME phc offset         1 s2 freq  -90766 delay    566
phc2sys[4595.634]: [ptp4l.2.config] CLOCK_REALTIME phc offset       -10 s2 freq  -90777 delay    565
phc2sys[4595.696]: [ptp4l.2.config] CLOCK_REALTIME phc offset       -13 s2 freq  -90783 delay    566
phc2sys[4595.759]: [ptp4l.2.config] CLOCK_REALTIME phc offset         3 s2 freq  -90771 delay    563
phc2sys[4595.821]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90768 delay    563
ts2phc[4595.856]: [ts2phc.2.config] ens3f0 master offset          0 s2 freq      +0 



Clock states in ptp metrics will report stale states:

[kni@registry.ran-vcl01 ~]$ oc exec -it ds/linuxptp-daemon  -n openshift-ptp -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics | grep clock_state
# HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
# TYPE openshift_ptp_clock_state gauge
openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 0
openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="GM"} 0
openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 0
openshift_ptp_clock_state{iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1
openshift_ptp_clock_state{iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0
openshift_ptp_clock_state{iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0

Expected results:

With higher ptpClockThreshold settings:

[kni@registry.ran-vcl01 ~]$ oc get ptpconfigs.ptp.openshift.io -n openshift-ptp -o yaml | grep ptpClockThreshold: -A3
      ptpClockThreshold:
        holdOverTimeout: 5
        maxOffsetThreshold: 100
        minOffsetThreshold: -100
--
      ptpClockThreshold:
        holdOverTimeout: 5
        maxOffsetThreshold: 100
        minOffsetThreshold: -100
--
      ptpClockThreshold:
        holdOverTimeout: 5
        maxOffsetThreshold: 100
        minOffsetThreshold: -100
--
      ptpClockThreshold:
        holdOverTimeout: 5
        maxOffsetThreshold: 100
        minOffsetThreshold: -100
--
      ptpClockThreshold:
        holdOverTimeout: 5
        maxOffsetThreshold: 100
        minOffsetThreshold: -100

Master offsets are reported for all interfaces:

[kni@registry.ran-vcl01 ~]$ oc logs -n openshift-ptp linuxptp-daemon-wbbtq linuxptp-daemon-container --since=10s | grep offset 
ptp4l[7675.831]: [ptp4l.3.config] master offset         -2 s2 freq   +1076 path delay       855
phc2sys[7675.833]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -6 s2 freq  -90812 delay    496
ts2phc[7675.856]: [ts2phc.2.config] ens3f0 master offset          0 s2 freq      +0
ptp4l[7675.858]: [ptp4l.1.config] master offset         -8 s2 freq   -1443 path delay       570
ptp4l[7675.860]: [ptp4l.0.config] master offset          0 s2 freq   -1432 path delay       854
ptp4l[7675.877]: [ptp4l.4.config] master offset         -1 s2 freq   +1078 path delay       850
ptp4l[7675.894]: [ptp4l.3.config] master offset          3 s2 freq   +1085 path delay       855
phc2sys[7675.896]: [ptp4l.2.config] CLOCK_REALTIME phc offset        10 s2 freq  -90798 delay    496
ptp4l[7675.921]: [ptp4l.1.config] master offset         -7 s2 freq   -1443 path delay       570
ptp4l[7675.922]: [ptp4l.0.config] master offset         -2 s2 freq   -1435 path delay       854
ptp4l[7675.939]: [ptp4l.4.config] master offset          1 s2 freq   +1082 path delay       850
ptp4l[7675.957]: [ptp4l.3.config] master offset          3 s2 freq   +1085 path delay       855
phc2sys[7675.958]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -7 s2 freq  -90812 delay    500
ptp4l[7675.983]: [ptp4l.1.config] master offset        -14 s2 freq   -1455 path delay       568
ptp4l[7675.985]: [ptp4l.0.config] master offset         -3 s2 freq   -1437 path delay       854
ptp4l[7676.002]: [ptp4l.4.config] master offset         -1 s2 freq   +1078 path delay       850
ptp4l[7676.019]: [ptp4l.3.config] master offset         -3 s2 freq   +1075 path delay       855
phc2sys[7676.021]: [ptp4l.2.config] CLOCK_REALTIME phc offset         3 s2 freq  -90804 delay    494
ptp4l[7676.046]: [ptp4l.1.config] master offset         -9 s2 freq   -1448 path delay       568
ptp4l[7676.048]: [ptp4l.0.config] master offset          1 s2 freq   -1431 path delay       854
ptp4l[7676.064]: [ptp4l.4.config] master offset         -1 s2 freq   +1078 path delay       850
ptp4l[7676.082]: [ptp4l.3.config] master offset          6 s2 freq   +1090 path delay       854
phc2sys[7676.084]: [ptp4l.2.config] CLOCK_REALTIME phc offset        12 s2 freq  -90794 delay    493

 and clock states are all reported correctly:
[kni@registry.ran-vcl01 ~]$ oc exec -it ds/linuxptp-daemon  -n openshift-ptp -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics | grep clock_state # HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER # TYPE openshift_ptp_clock_state gauge openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 1 openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="GM"} 1 openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 1 openshift_ptp_clock_state{iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1 openshift_ptp_clock_state{iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 1 openshift_ptp_clock_state{iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 1

Additional info:

Assignee:: Aneesh Puttur

Reporter:: Bonnie Block

QA Contact:: Bonnie Block

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2023/10/17 9:00 PM

Updated:: 2023/10/19 6:58 PM

Resolved:: 2023/10/19 6:58 PM

Details

Description

Attachments

Activity

People

Dates