Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-32195

dpll offset is spiking

XMLWordPrintable

    • Important
    • No
    • CNF Ran Sprint 252, CNF Ran Sprint 253
    • 2
    • False
    • Hide

      None

      Show
      None
    • Hide
      5/09: QE verified the patch, looks good. need forecast
      5/3: Dell has provided with patch , QE is trying to updated the lab
      Show
      5/09: QE verified the patch, looks good. need forecast 5/3: Dell has provided with patch , QE is trying to updated the lab

      Description of problem:

      2GM environment
      Cluster version is 4.16.0-0.nightly-2024-04-08-024331
      ptp-operator.v4.16.0-202404101645 (also tested w/dev build)
      dpll is spiking in linuxptp-daemon log:
      
      dpll[1712950145]:[ts2phc.2.config] ens7f0 frequency_status 3 offset 28147497 phase_status 3 pps_status 1 s2 I0412 19:29:06.000445  360305 event.go:417] dpll State s2, gnss State s2, tsphc state s2, gm state s2, I0412 19:29:06.502710  360305 dpll.go:363] setting phase offset to 0 ns for clock id 5799633565434117236 I0412 19:29:06.502733  360305 dpll.go:623] ens5f0-dpll decision: Status 3, Offset 0, In spec false, Source lost false, On holdover false I0412 19:29:06.502760  360305 dpll.go:727] dpll event sent for (ens5f0) I0412 19:29:06.502805  360305 stats.go:61] state updated for dpll =s2 dpll[1712950146]:[ts2phc.2.config] ens5f0 frequency_status 3 offset 0 phase_status 3 pps_status 1 s2 I0412 19:29:06.538516  360305 dpll.go:363] setting phase offset to 0 ns for clock id 5799633565434117476 I0412 19:29:06.538561  360305 dpll.go:623] ens7f0-dpll decision: Status 3, Offset 0, In spec true, Source lost false, On holdover false I0412 19:29:06.538574  360305 dpll.go:669] dpll is locked, source is not lost, offset is in range, state is DPLL_LOCKED_HO_ACQ or DPLL_HOLDOVER(ens7f0) I0412 19:29:06.538597  360305 dpll.go:727] dpll event sent for (ens7f0) I0412 19:29:06.538624  360305 stats.go:61] state updated for dpll =s2 dpll[1712950146]:[ts2phc.2.config] ens7f0 frequency_status 3 offset 0 phase_status 3 pps_status 1 s2 I0412 19:29:06.595877  360305 dpll.go:363] setting phase offset to 0 ns for clock id 5799633565434117236 I0412 19:29:06.595894  360305 dpll.go:623] ens5f0-dpll decision: Status 3, Offset 0, In spec false, Source lost false, On holdover false I0412 19:29:06.595906  360305 dpll.go:727] dpll event sent for (ens5f0) dpll[1712950146]:[ts2phc.2.config] ens5f0 frequency_status 3 offset 0 phase_status 3 pps_status 1 s2 I0412 19:29:06.595933  360305 stats.go:61] state updated for dpll =s2 I0412 19:29:06.646358  360305 dpll.go:363] setting phase offset to 0 ns for clock id 5799633565434117476 I0412 19:29:06.646453  360305 dpll.go:623] ens7f0-dpll decision: Status 3, Offset 0, In spec true, Source lost false, On holdover false I0412 19:29:06.646493  360305 dpll.go:669] dpll is locked, source is not lost, offset is in range, state is DPLL_LOCKED_HO_ACQ or DPLL_HOLDOVER(ens7f0)
      These are metrics checked over the last couple of minutes:
      
      [kni@registry.kni-qe-61 ~]$ oc exec -it ds/linuxptp-daemon  -n openshift-ptp -c linuxptp-daemon-container -- curl -s localhost:9091/metrics | grep clock_state
      # HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
      # TYPE openshift_ptp_clock_state gauge
      openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="helix60.lab.eng.rdu2.redhat.com",process="phc2sys"} 1
      openshift_ptp_clock_state{iface="eno1239x",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 2
      openshift_ptp_clock_state{iface="ens3f1npx",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 2
      openshift_ptp_clock_state{iface="ens5fx",node="helix60.lab.eng.rdu2.redhat.com",process="dpll"} 0
      openshift_ptp_clock_state{iface="ens5fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="GM"} 0
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="dpll"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="gnss"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1
      [kni@registry.kni-qe-61 ~]$ # HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
      # TYPE openshift_ptp_clock_state gauge
      openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="helix60.lab.eng.rdu2.redhat.com",process="phc2sys"} 1
      openshift_ptp_clock_state{iface="eno1239x",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 2
      openshift_ptp_clock_state{iface="ens3f1npx",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 2
      openshift_ptp_clock_state{iface="ens5fx",node="helix60.lab.eng.rdu2.redhat.com",process="dpll"} 1
      openshift_ptp_clock_state{iface="ens5fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="GM"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="dpll"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="gnss"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1# HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
      # TYPE openshift_ptp_clock_state gauge
      openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="helix60.lab.eng.rdu2.redhat.com",process="phc2sys"} 1
      openshift_ptp_clock_state{iface="eno1239x",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 1
      openshift_ptp_clock_state{iface="ens3f1npx",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 0
      openshift_ptp_clock_state{iface="ens5fx",node="helix60.lab.eng.rdu2.redhat.com",process="dpll"} 1
      openshift_ptp_clock_state{iface="ens5fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="GM"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="dpll"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="gnss"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1# HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
      # TYPE openshift_ptp_clock_state gauge
      openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="helix60.lab.eng.rdu2.redhat.com",process="phc2sys"} 1
      openshift_ptp_clock_state{iface="eno1239x",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 1
      openshift_ptp_clock_state{iface="ens3f1npx",node="helix60.lab.eng.rdu2.redhat.com",process="ptp4l"} 1
      openshift_ptp_clock_state{iface="ens5fx",node="helix60.lab.eng.rdu2.redhat.com",process="dpll"} 1
      openshift_ptp_clock_state{iface="ens5fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="GM"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="dpll"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="gnss"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="helix60.lab.eng.rdu2.redhat.com",process="ts2phc"} 1
      
      
      
      

      Version-Release number of selected component (if applicable):

          

      How reproducible:

          

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

          

      Expected results:

          

      Additional info:

          

            vgrinber@redhat.com Vitaly Grinberg
            bblock@redhat.com Bonnie Block
            Bonnie Block Bonnie Block
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: