Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-12954

PTP metrics sometimes do not pick up ptp config changes after node reboot

XMLWordPrintable

    • Moderate
    • No
    • CNF RAN Sprint 235, CNF RAN Sprint 236, CNF RAN Sprint 237, CNF RAN Sprint 238, CNF RAN Sprint 239, CNF RAN Sprint 240, CNF RAN Sprint 241, CNF RAN Sprint 242, CNF RAN Sprint 243, CNF RAN Sprint 244, CNF RAN Sprint 245
    • 11
    • False
    • Hide

      None

      Show
      None
    • Hide
      12/4: This is timing issue, QE need to update automation to accommodate delay in updating metrics
      11/17: asked qe if the issue is still valid.
      10/5: asked qe if the issue is still valid.
      8/23: will re visit after 4.14 stabilizes
      7/31: will be investigated further when there is time
      05/03 - Ken Y - This is not reproducible; reducing criticality but will continue to work with QE if this occurs again.
      Show
      12/4: This is timing issue, QE need to update automation to accommodate delay in updating metrics 11/17: asked qe if the issue is still valid. 10/5: asked qe if the issue is still valid. 8/23: will re visit after 4.14 stabilizes 7/31: will be investigated further when there is time 05/03 - Ken Y - This is not reproducible; reducing criticality but will continue to work with QE if this occurs again.

      Description of problem:

      Modify ptpClockThreshold after node reboot and ptp pod started, ptp metrics sometimes do not pick up the changes. 

Note that HTTP ptp event worked as expected. It is only an issue with ptp metrics.

      Version-Release number of selected component (if applicable):

      OCP: 4.13.0-rc.5
ptp operator: ptp-operator.v4.13.0-202304190216

      How reproducible:

      Almost 100% reproducible in automation.

      Steps to Reproduce:

      0. SNO DU node is installed and configured with PTP OC config and HTTP events.
1. Reboot SNO DU node
2. Shortly after node is up and all pods are recovered and running, modify ptp configs/profile with the following threshold configs 

      ptpClockThreshold:
        holdOverTimeout: 120
        maxOffsetThreshold: 1
        minOffsetThreshold: -1

3. Wait for HOLDOVER and FREERUN PTP events
4. Wait for ptp metrics to show FREERUN clock state

      Actual results:

      3. works as expected - ptp holdover and freerun events received 
4. openshift_ptp_threshold metrics is not returned, and ptp clock state stays in locked as if the ptp threshold config was not done. 

# HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
# TYPE openshift_ptp_clock_state gauge
openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde30.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 1
openshift_ptp_clock_state{iface="ens1fx",node="cnfde30.ptp.lab.eng.bos.redhat.com",process="ptp4l"} 1

      Expected results:

      4. 
- openshift_ptp_threshold shows the numbers configured in ptp profile. 
- openshift_ptp_clock_state shows FREERUN

      Additional info:

      Workaround: Add 5 minutes delay in automated test before modifying ptp profile.

       

            aputtur@redhat.com Aneesh Puttur
            rhn-support-yliu1 Yang Liu
            Bonnie Block Bonnie Block
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: