Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-24342

GM clock state stays locked when 1PPS lost

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Critical Critical
    • None
    • 4.15.0
    • Networking / ptp
    • None
    • No
    • False
    • Hide

      None

      Show
      None

      In a SNO with 2 WPC cards, where the ens2f is connected to GPS and ens7f is connected to ens2f through SMA1.

              pins:
                "ens2f0":
                  "U.FL2": "0 2"
                  "U.FL1": "0 1"
                  "SMA2": "0 2"
                  "SMA1": "2 1"
                "ens7f0":
                  "U.FL2": "0 2"
                  "U.FL1": "0 1"
                  "SMA2": "0 2"
                  "SMA1": "1 1"

       

      Test Steps:

      1. Wait for GM is locked and openshift_ptp_clock_state for all the processes are locked

      # HELP openshift_ptp_clock_state 0 = FREERUN, 1 = LOCKED, 2 = HOLDOVER
      # TYPE openshift_ptp_clock_state gauge
      openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 1
      openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="GM"} 1
      openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="dpll"} 1
      openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="gnss"} 1
      openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="dpll"} 1
      openshift_ptp_clock_state{iface="ens7fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1 

      2. Manually disconnect the 1PPS connection between 2 cards by disabling SMA1 on card ens7f.

      echo 0 1 > /sys/class/net/ens7f0/device/ptp/ptp*/pins/SMA1 

      The dpll and GM should immediately goes to FREERUN in the metrics

      openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 1 openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="GM"} 0 openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="dpll"} 1 openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="gnss"} 1 openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1 openshift_ptp_clock_state{iface="ens7fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="dpll"} 0 openshift_ptp_clock_state{iface="ens7fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1 

      3. Reconnect the 1PPS connection. Wait for a while until GM and dpll states all go back to 1 (LOCKED).

      echo 1 1 > /sys/class/net/ens7f0/device/ptp/ptp*/pins/SMA1 

      Actual Result:

      After repeating the tests 1-6 times, the GM state stays at locked while dpll changed to 0, and it stays like this for a very long time.

      openshift_ptp_clock_state{iface="CLOCK_REALTIME",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="phc2sys"} 1 openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="GM"} 1 openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="dpll"} 1 openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="gnss"} 1 openshift_ptp_clock_state{iface="ens2fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1 openshift_ptp_clock_state{iface="ens7fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="dpll"} 0 openshift_ptp_clock_state{iface="ens7fx",node="cnfde21.ptp.lab.eng.bos.redhat.com",process="ts2phc"} 1  

      Reproducible:

      The issue is seen after repeating the test successfully 1-6 times.

              jacding@redhat.com Jack Ding
              jacding@redhat.com Jack Ding
              Bonnie Block Bonnie Block
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: