Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-17765

ptp clock is not locked in GM configuration

XMLWordPrintable

    • No
    • False
    • Hide

      None

      Show
      None
    • 8/16: triaged; this will be investigated after the crash loop

      Description of problem:

      Regression test is failing with ptp-operator.v4.14.0-202308110126.
      ptp clock is not locked in GM config:

      [kni@registry.ran-vcl01 ~]$ oc -n openshift-ptp exec linuxptp-daemon-p4qbs -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics
      [...]
      # TYPE openshift_ptp_clock_state gauge
      openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 0
      openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="GM"} 1
      openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 1
      openshift_ptp_clock_state{iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 0
      openshift_ptp_clock_state{iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0
      openshift_ptp_clock_state{iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0

      linuxptp-daemon-container log and ptp metrics show abnormal offsets:

      [kni@registry.ran-vcl01 ~]$ oc logs -n openshift-ptp linuxptp-daemon-p4qbs linuxptp-daemon-container
      [...]
      phc2sys[7049.323]: [ptp4l.1.config] CLOCK_REALTIME phc offset    586447 s2 freq -1850083 delay    470
      ptp4l[7049.324]: [ptp4l.3.config] master offset    -102903 s2 freq  +60868 path delay      -279
      ptp4l[7049.382]: [ptp4l.4.config] master offset    -102569 s2 freq  +49637 path delay        52
      ptp4l[7049.384]: [ptp4l.2.config] master offset    -102605 s2 freq  +52041 path delay       -40
      ptp4l[7049.385]: [ptp4l.0.config] master offset    -102550 s2 freq  +49243 path delay      -263
      phc2sys[7049.386]: [ptp4l.1.config] CLOCK_REALTIME phc offset    700758 s2 freq -1559838 delay    494
      ptp4l[7049.386]: [ptp4l.3.config] master offset    -102353 s2 freq  +51623 path delay      -279
      ptp4l[7049.446]: [ptp4l.4.config] master offset    -101463 s2 freq  +41375 path delay        52
      ptp4l[7049.446]: [ptp4l.2.config] master offset    -101492 s2 freq  +43787 path delay       -40
       
      ni@registry.ran-vcl01 ~]$ oc -n openshift-ptp exec linuxptp-daemon-p4qbs -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics
      [...]
      # HELP openshift_ptp_max_offset_ns  # TYPE openshift_ptp_max_offset_ns gauge openshift_ptp_max_offset_ns{from="master",iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1.034912482e+09 openshift_ptp_max_offset_ns{from="master",iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 5.22828677e+08 openshift_ptp_max_offset_ns{from="master",iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 5.23460734e+08 openshift_ptp_max_offset_ns{from="phc",iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 93 # HELP openshift_ptp_offset_ns  # TYPE openshift_ptp_offset_ns gauge openshift_ptp_offset_ns{from="gnss",iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 5 openshift_ptp_offset_ns{from="master",iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} -2.71623279e+08 openshift_ptp_offset_ns{from="master",iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} -1.01563882e+08 openshift_ptp_offset_ns{from="master",iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} -1.05411205e+08 openshift_ptp_offset_ns{from="phc",iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 2.87460889e+08
      

       

      Version-Release number of selected component (if applicable):

      Cluster version is 4.14.0-0.nightly-2023-08-11-055332
      ptp-operator.v4.14.0-202308110126

      How reproducible:

      100%

      Steps to Reproduce:

      1. Deploy SNO with PTP GM configuration
      2. Check ptp metrics:
      
      [kni@registry.ran-vcl01 ~]$ oc -n openshift-ptp exec linuxptp-daemon-p4qbs -c linuxptp-daemon-container -- curl -s 127.0.0.1:9091/metrics
      [...]
      # TYPE openshift_ptp_clock_state gauge openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 0 openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="GM"} 1 openshift_ptp_clock_state{iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 1 openshift_ptp_clock_state{iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 0 openshift_ptp_clock_state{iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0 openshift_ptp_clock_state{iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 0
      
      [...]
      
      # HELP openshift_ptp_max_offset_ns  # TYPE openshift_ptp_max_offset_ns gauge openshift_ptp_max_offset_ns{from="master",iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1.034912482e+09 openshift_ptp_max_offset_ns{from="master",iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 5.22828677e+08 openshift_ptp_max_offset_ns{from="master",iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 5.23460734e+08 openshift_ptp_max_offset_ns{from="phc",iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 93 # HELP openshift_ptp_offset_ns  # TYPE openshift_ptp_offset_ns gauge openshift_ptp_offset_ns{from="gnss",iface="ens3f0",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="gnss"} 5 openshift_ptp_offset_ns{from="master",iface="ens3fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ts2phc"} -2.71623279e+08 openshift_ptp_offset_ns{from="master",iface="ens5fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} -1.01563882e+08 openshift_ptp_offset_ns{from="master",iface="ens7fx",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="ptp4l"} -1.05411205e+08 openshift_ptp_offset_ns{from="phc",iface="CLOCK_REALTIME",node="cnfde4.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 2.87460889e+08
      
      
      

      Actual results:

      ptp clock is not in sync.
      offsets exceed clock lock.

      Expected results:

      ptp clock is in sync.
      offsets within clock lock range.

      Additional info:

       

            aputtur@redhat.com Aneesh Puttur
            bblock@redhat.com Bonnie Block
            Bonnie Block Bonnie Block
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: