Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-21842

Master offsets for slave interfaces are missing from linuxptp daemon logs when ptpClockThreshold maxOffsetThreshold/minOffsetThreshold are set to very small number such as 1/-1, 2/-2.

    XMLWordPrintable

Details

    • Moderate
    • No
    • False
    • Hide

      None

      Show
      None
    • 10/19: Works as designed , This is happening because when GM is forced to FREERUN , the clock class changes to 248 and slave has clock class threshold set to 7 . That makes slave ports stuck in Listening and you wont see offset coming up

    Description

      Description of problem:

      In MC ptp environment:
      Master offsets for slave interfaces are missing from linuxptp daemon logs when ptpClockThreshold maxOffsetThreshold/minOffsetThreshold are set to very small number such as 1/-1, 2/-2.
      
      The causes ptp metrics master offsets and clock states to be stuck with stale numbers/state.

      Version-Release number of selected component (if applicable):

      [kni@registry.ran-vcl01 cnf-gotests]$ oc get csv -n openshift-ptp 
      NAME                                DISPLAY        VERSION               
      ptp-operator.v4.14.0-202310140046   PTP Operator   4.14.0-202310140046
      
      [kni@registry.ran-vcl01 cnf-gotests]$ oc get clusterversions.config.openshift.io 
      NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.14.0-rc.6   True        False         32h     Cluster version is 4.14.0-rc.6
      

      How reproducible:

      100%

      Steps to Reproduce:

      1. Deploy SNO with MC/slave ptpconfigs
      example:
      [kni@registry.ran-vcl01 cnf-gotests]$ oc get ptpconfigs.ptp.openshift.io -A
      NAMESPACE       NAME               AGE
      openshift-ptp   du-ptp-gm          3h23m
      openshift-ptp   du-ptp-slave-gm1   3h23m
      openshift-ptp   du-ptp-slave-gm2   3h23m
      openshift-ptp   du-ptp-slave-gm3   3h23m
      openshift-ptp   du-ptp-slave-gm4   3h23m
      
      2. Edit ptpconfigs to include very small ptpClockThreshold values.  
      example:
            ptpClockThreshold:
              holdOverTimeout: 120
              maxOffsetThreshold: 2
              minOffsetThreshold: -2
      
      [kni@registry.ran-vcl01 ~]$ oc get ptpconfigs.ptp.openshift.io -n openshift-ptp -o yaml | grep ptpClockThreshold: -A3
      
            ptpClockThreshold:
              holdOverTimeout: 10
              maxOffsetThreshold: 2
              minOffsetThreshold: -2
      --
            ptpClockThreshold:
              holdOverTimeout: 10
              maxOffsetThreshold: 2
              minOffsetThreshold: -2
      --
            ptpClockThreshold:
              holdOverTimeout: 10
              maxOffsetThreshold: 2
              minOffsetThreshold: -2
      --
            ptpClockThreshold:
              holdOverTimeout: 10
              maxOffsetThreshold: 2
              minOffsetThreshold: -2
      --
            ptpClockThreshold:
              holdOverTimeout: 10
              maxOffsetThreshold: 2
              minOffsetThreshold: -2
      
      
      3. Check linuxptp-daemon logs for master offsets.  Only GM master offsets are reported:
      
      [kni@registry.ran-vcl01 ~]$ oc logs -n openshift-ptp linuxptp-daemon-wbbtq linuxptp-daemon-container --since=10s | grep offset 
      phc2sys[4594.382]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -8 s2 freq  -90790 delay    562
      phc2sys[4594.444]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -9 s2 freq  -90793 delay    562
      phc2sys[4594.507]: [ptp4l.2.config] CLOCK_REALTIME phc offset         4 s2 freq  -90783 delay    565
      phc2sys[4594.570]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -9 s2 freq  -90795 delay    564
      phc2sys[4594.632]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -1 s2 freq  -90790 delay    563
      phc2sys[4594.695]: [ptp4l.2.config] CLOCK_REALTIME phc offset         4 s2 freq  -90785 delay    566
      phc2sys[4594.757]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -6 s2 freq  -90794 delay    566
      phc2sys[4594.820]: [ptp4l.2.config] CLOCK_REALTIME phc offset         8 s2 freq  -90781 delay    566
      ts2phc[4594.856]: [ts2phc.2.config] ens3f0 master offset          0 s2 freq      +0
      phc2sys[4594.883]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90776 delay    562
      phc2sys[4594.945]: [ptp4l.2.config] CLOCK_REALTIME phc offset         7 s2 freq  -90777 delay    561
      phc2sys[4595.008]: [ptp4l.2.config] CLOCK_REALTIME phc offset         6 s2 freq  -90776 delay    564
      phc2sys[4595.070]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -7 s2 freq  -90787 delay    563
      phc2sys[4595.133]: [ptp4l.2.config] CLOCK_REALTIME phc offset         7 s2 freq  -90775 delay    565
      gnss[1697573180]:[ts2phc.2.config] ens3f0 gnss_status 3 offset 5 s0
      phc2sys[4595.195]: [ptp4l.2.config] CLOCK_REALTIME phc offset         8 s2 freq  -90772 delay    560
      phc2sys[4595.258]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90766 delay    567
      phc2sys[4595.321]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90763 delay    562
      phc2sys[4595.383]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90766 delay    562
      phc2sys[4595.446]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90764 delay    565
      phc2sys[4595.508]: [ptp4l.2.config] CLOCK_REALTIME phc offset         3 s2 freq  -90765 delay    566
      phc2sys[4595.571]: [ptp4l.2.config] CLOCK_REALTIME phc offset         1 s2 freq  -90766 delay    566
      phc2sys[4595.634]: [ptp4l.2.config] CLOCK_REALTIME phc offset       -10 s2 freq  -90777 delay    565
      phc2sys[4595.696]: [ptp4l.2.config] CLOCK_REALTIME phc offset       -13 s2 freq  -90783 delay    566
      phc2sys[4595.759]: [ptp4l.2.config] CLOCK_REALTIME phc offset         3 s2 freq  -90771 delay    563
      phc2sys[4595.821]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90768 delay    563
      ts2phc[4595.856]: [ts2phc.2.config] ens3f0 master offset          0 s2 freq      +0 
      4. Check clock states in ptp metrics. Clock states will report stale states:
      
      [kni@registry.ran-vcl01 ~]$ oc exec -it ds/linuxptp-daemon  -n openshift-ptp -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics | grep clock_state
      # HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
      # TYPE openshift_ptp_clock_state gauge
      openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 0
      openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="GM"} 0
      openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 0
      openshift_ptp_clock_state{iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1
      openshift_ptp_clock_state{iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0
      openshift_ptp_clock_state{iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0

       

      Actual results:

      linuxptp-daemon logs only shows GM master offsets:
      
      [kni@registry.ran-vcl01 ~]$ oc logs -n openshift-ptp linuxptp-daemon-wbbtq linuxptp-daemon-container --since=10s | grep offset 
      phc2sys[4594.382]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -8 s2 freq  -90790 delay    562
      phc2sys[4594.444]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -9 s2 freq  -90793 delay    562
      phc2sys[4594.507]: [ptp4l.2.config] CLOCK_REALTIME phc offset         4 s2 freq  -90783 delay    565
      phc2sys[4594.570]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -9 s2 freq  -90795 delay    564
      phc2sys[4594.632]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -1 s2 freq  -90790 delay    563
      phc2sys[4594.695]: [ptp4l.2.config] CLOCK_REALTIME phc offset         4 s2 freq  -90785 delay    566
      phc2sys[4594.757]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -6 s2 freq  -90794 delay    566
      phc2sys[4594.820]: [ptp4l.2.config] CLOCK_REALTIME phc offset         8 s2 freq  -90781 delay    566
      ts2phc[4594.856]: [ts2phc.2.config] ens3f0 master offset          0 s2 freq      +0
      phc2sys[4594.883]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90776 delay    562
      phc2sys[4594.945]: [ptp4l.2.config] CLOCK_REALTIME phc offset         7 s2 freq  -90777 delay    561
      phc2sys[4595.008]: [ptp4l.2.config] CLOCK_REALTIME phc offset         6 s2 freq  -90776 delay    564
      phc2sys[4595.070]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -7 s2 freq  -90787 delay    563
      phc2sys[4595.133]: [ptp4l.2.config] CLOCK_REALTIME phc offset         7 s2 freq  -90775 delay    565
      gnss[1697573180]:[ts2phc.2.config] ens3f0 gnss_status 3 offset 5 s0
      phc2sys[4595.195]: [ptp4l.2.config] CLOCK_REALTIME phc offset         8 s2 freq  -90772 delay    560
      phc2sys[4595.258]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90766 delay    567
      phc2sys[4595.321]: [ptp4l.2.config] CLOCK_REALTIME phc offset        11 s2 freq  -90763 delay    562
      phc2sys[4595.383]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90766 delay    562
      phc2sys[4595.446]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90764 delay    565
      phc2sys[4595.508]: [ptp4l.2.config] CLOCK_REALTIME phc offset         3 s2 freq  -90765 delay    566
      phc2sys[4595.571]: [ptp4l.2.config] CLOCK_REALTIME phc offset         1 s2 freq  -90766 delay    566
      phc2sys[4595.634]: [ptp4l.2.config] CLOCK_REALTIME phc offset       -10 s2 freq  -90777 delay    565
      phc2sys[4595.696]: [ptp4l.2.config] CLOCK_REALTIME phc offset       -13 s2 freq  -90783 delay    566
      phc2sys[4595.759]: [ptp4l.2.config] CLOCK_REALTIME phc offset         3 s2 freq  -90771 delay    563
      phc2sys[4595.821]: [ptp4l.2.config] CLOCK_REALTIME phc offset         5 s2 freq  -90768 delay    563
      ts2phc[4595.856]: [ts2phc.2.config] ens3f0 master offset          0 s2 freq      +0 
      
      
      
      Clock states in ptp metrics will report stale states:
      
      [kni@registry.ran-vcl01 ~]$ oc exec -it ds/linuxptp-daemon  -n openshift-ptp -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics | grep clock_state
      # HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
      # TYPE openshift_ptp_clock_state gauge
      openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 0
      openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="GM"} 0
      openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 0
      openshift_ptp_clock_state{iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1
      openshift_ptp_clock_state{iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0
      openshift_ptp_clock_state{iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0

      Expected results:

      With higher ptpClockThreshold settings:
      
      [kni@registry.ran-vcl01 ~]$ oc get ptpconfigs.ptp.openshift.io -n openshift-ptp -o yaml | grep ptpClockThreshold: -A3
            ptpClockThreshold:
              holdOverTimeout: 5
              maxOffsetThreshold: 100
              minOffsetThreshold: -100
      --
            ptpClockThreshold:
              holdOverTimeout: 5
              maxOffsetThreshold: 100
              minOffsetThreshold: -100
      --
            ptpClockThreshold:
              holdOverTimeout: 5
              maxOffsetThreshold: 100
              minOffsetThreshold: -100
      --
            ptpClockThreshold:
              holdOverTimeout: 5
              maxOffsetThreshold: 100
              minOffsetThreshold: -100
      --
            ptpClockThreshold:
              holdOverTimeout: 5
              maxOffsetThreshold: 100
              minOffsetThreshold: -100
      
      Master offsets are reported for all interfaces:
      
      [kni@registry.ran-vcl01 ~]$ oc logs -n openshift-ptp linuxptp-daemon-wbbtq linuxptp-daemon-container --since=10s | grep offset 
      ptp4l[7675.831]: [ptp4l.3.config] master offset         -2 s2 freq   +1076 path delay       855
      phc2sys[7675.833]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -6 s2 freq  -90812 delay    496
      ts2phc[7675.856]: [ts2phc.2.config] ens3f0 master offset          0 s2 freq      +0
      ptp4l[7675.858]: [ptp4l.1.config] master offset         -8 s2 freq   -1443 path delay       570
      ptp4l[7675.860]: [ptp4l.0.config] master offset          0 s2 freq   -1432 path delay       854
      ptp4l[7675.877]: [ptp4l.4.config] master offset         -1 s2 freq   +1078 path delay       850
      ptp4l[7675.894]: [ptp4l.3.config] master offset          3 s2 freq   +1085 path delay       855
      phc2sys[7675.896]: [ptp4l.2.config] CLOCK_REALTIME phc offset        10 s2 freq  -90798 delay    496
      ptp4l[7675.921]: [ptp4l.1.config] master offset         -7 s2 freq   -1443 path delay       570
      ptp4l[7675.922]: [ptp4l.0.config] master offset         -2 s2 freq   -1435 path delay       854
      ptp4l[7675.939]: [ptp4l.4.config] master offset          1 s2 freq   +1082 path delay       850
      ptp4l[7675.957]: [ptp4l.3.config] master offset          3 s2 freq   +1085 path delay       855
      phc2sys[7675.958]: [ptp4l.2.config] CLOCK_REALTIME phc offset        -7 s2 freq  -90812 delay    500
      ptp4l[7675.983]: [ptp4l.1.config] master offset        -14 s2 freq   -1455 path delay       568
      ptp4l[7675.985]: [ptp4l.0.config] master offset         -3 s2 freq   -1437 path delay       854
      ptp4l[7676.002]: [ptp4l.4.config] master offset         -1 s2 freq   +1078 path delay       850
      ptp4l[7676.019]: [ptp4l.3.config] master offset         -3 s2 freq   +1075 path delay       855
      phc2sys[7676.021]: [ptp4l.2.config] CLOCK_REALTIME phc offset         3 s2 freq  -90804 delay    494
      ptp4l[7676.046]: [ptp4l.1.config] master offset         -9 s2 freq   -1448 path delay       568
      ptp4l[7676.048]: [ptp4l.0.config] master offset          1 s2 freq   -1431 path delay       854
      ptp4l[7676.064]: [ptp4l.4.config] master offset         -1 s2 freq   +1078 path delay       850
      ptp4l[7676.082]: [ptp4l.3.config] master offset          6 s2 freq   +1090 path delay       854
      phc2sys[7676.084]: [ptp4l.2.config] CLOCK_REALTIME phc offset        12 s2 freq  -90794 delay    493
       
       and clock states are all reported correctly:
      [kni@registry.ran-vcl01 ~]$ oc exec -it ds/linuxptp-daemon  -n openshift-ptp -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics | grep clock_state # HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER # TYPE openshift_ptp_clock_state gauge openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 1 openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="GM"} 1 openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 1 openshift_ptp_clock_state{iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1 openshift_ptp_clock_state{iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 1 openshift_ptp_clock_state{iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 1
       
      

       

      Additional info:

       

      Attachments

        Activity

          People

            aputtur@redhat.com Aneesh Puttur
            bblock@redhat.com Bonnie Block
            Bonnie Block Bonnie Block
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: