Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-58326

Event noise upon T-GM dual WPC clock locking from freerun

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • Hide
      8/27: lowered priority to Minor
      8/25: QE need to test with just T-GM profile
      8/14: reassigned to Jack. analyzing logs
      7/3: QE to verify with more backing data
      Show
      8/27: lowered priority to Minor 8/25: QE need to test with just T-GM profile 8/14: reassigned to Jack. analyzing logs 7/3: QE to verify with more backing data
    • None
    • CNF RAN Sprint 277, CNF RAN Sprint 278
    • 2
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          Event noise upon T-GM clock locking from freerun

      Version-Release number of selected component (if applicable):

          ptp-operator.v4.18.0-202506301635

      How reproducible:

          100%

      Steps to Reproduce:

          1. Listen to sync status state change events; oc -n openshift-ptp logs ds/linuxptp-daemon cloud-event-proxy --tail 0 -f | grep "event.sync.sync-status.synchronization-state-change"
          2. Cause GNSS loss for T-GM; `oc -n openshift-ptp exec ds/linuxptp-daemon -c linuxptp-daemon-container -- ubxtool -P 29.20 -p COLDBOOT`     
          3. Let the clock enter FREERUN
          4. Allow the GNSS To recover; kill the loop
          5. Upon locking the event noise occurs.

      Actual results:
      A large array of events, some of which are conflicting

          "event.sync.sync-status.synchronization-state-change"
      time="2025-07-02T19:24:38Z" level=debug msg="event sent {\n    \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n    \"type\": \"event.sync.sync-status.synchronization-state-change\",\n    \"source\": \"/sync/sync-status/sync-state\",\
      n    \"dataContentType\": \"application/json\",\n    \"time\": \"2025-07-02T19:24:38.000730333Z\",\n    \"data\": {\n      \"version\": \"1.0\",\n      \"values\": [\n        {\n          \"ResourceAddress\": \"/cluster/node/helix65.lab
      .eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n          \"data_type\": \"notification\",\n          \"value_type\": \"enumeration\",\n          \"value\": \"LOCKED\"\n        }\n      ]\n    }\n  }"
      time="2025-07-02T19:24:38Z" level=debug msg="event sent {\n    \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n    \"type\": \"event.sync.sync-status.synchronization-state-change\",\n    \"source\": \"/sync/sync-status/sync-state\",\
      n    \"dataContentType\": \"application/json\",\n    \"time\": \"2025-07-02T19:24:38.904459744Z\",\n    \"data\": {\n      \"version\": \"1.0\",\n      \"values\": [\n        {\n          \"ResourceAddress\": \"/cluster/node/helix65.lab
      .eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n          \"data_type\": \"notification\",\n          \"value_type\": \"enumeration\",\n          \"value\": \"LOCKED\"\n        }\n      ]\n    }\n  }"
      time="2025-07-02T19:24:39Z" level=debug msg="event sent {\n    \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n    \"type\": \"event.sync.sync-status.synchronization-state-change\",\n    \"source\": \"/sync/sync-status/sync-state\",\
      n    \"dataContentType\": \"application/json\",\n    \"time\": \"2025-07-02T19:24:39.030617093Z\",\n    \"data\": {\n      \"version\": \"1.0\",\n      \"values\": [\n        {\n          \"ResourceAddress\": \"/cluster/node/helix65.lab
      .eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n          \"data_type\": \"notification\",\n          \"value_type\": \"enumeration\",\n          \"value\": \"LOCKED\"\n        }\n      ]\n    }\n  }"
      time="2025-07-02T19:24:41Z" level=debug msg="event sent {\n    \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n    \"type\": \"event.sync.sync-status.synchronization-state-change\",\n    \"source\": \"/sync/sync-status/sync-state\",\
      n    \"dataContentType\": \"application/json\",\n    \"time\": \"2025-07-02T19:24:41.655907634Z\",\n    \"data\": {\n      \"version\": \"1.0\",\n      \"values\": [\n        {\n          \"ResourceAddress\": \"/cluster/node/helix65.lab
      .eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n          \"data_type\": \"notification\",\n          \"value_type\": \"enumeration\",\n          \"value\": \"LOCKED\"\n        }\n      ]\n    }\n  }"
      time="2025-07-02T19:24:41Z" level=debug msg="event sent {\n    \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n    \"type\": \"event.sync.sync-status.synchronization-state-change\",\n    \"source\": \"/sync/sync-status/sync-state\",\
      n    \"dataContentType\": \"application/json\",\n    \"time\": \"2025-07-02T19:24:41.969500813Z\",\n    \"data\": {\n      \"version\": \"1.0\",\n      \"values\": [\n        {\n          \"ResourceAddress\": \"/cluster/node/helix65.lab
      .eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n          \"data_type\": \"notification\",\n          \"value_type\": \"enumeration\",\n          \"value\": \"LOCKED\"\n        }\n      ]\n    }\n  }"
      time="2025-07-02T19:25:29Z" level=debug msg="event sent {\n    \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n    \"type\": \"event.sync.sync-status.synchronization-state-change\",\n    \"source\": \"/sync/sync-status/sync-state\",\
      n    \"dataContentType\": \"application/json\",\n    \"time\": \"2025-07-02T19:25:29.95690114Z\",\n    \"data\": {\n      \"version\": \"1.0\",\n      \"values\": [\n        {\n          \"ResourceAddress\": \"/cluster/node/helix65.lab.
      eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n          \"data_type\": \"notification\",\n          \"value_type\": \"enumeration\",\n          \"value\": \"HOLDOVER\"\n        }\n      ]\n    }\n  }"
      time="2025-07-02T19:25:31Z" level=debug msg="event sent {\n    \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n    \"type\": \"event.sync.sync-status.synchronization-state-change\",\n    \"source\": \"/sync/sync-status/sync-state\",\
      n    \"dataContentType\": \"application/json\",\n    \"time\": \"2025-07-02T19:25:31.266169415Z\",\n    \"data\": {\n      \"version\": \"1.0\",\n      \"values\": [\n        {\n          \"ResourceAddress\": \"/cluster/node/helix65.lab
      .eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n          \"data_type\": \"notification\",\n          \"value_type\": \"enumeration\",\n          \"value\": \"LOCKED\"\n        }\n      ]\n    }\n  }"
      time="2025-07-02T19:25:31Z" level=debug msg="event sent {\n    \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n    \"type\": \"event.sync.sync-status.synchronization-state-change\",\n    \"source\": \"/sync/sync-status/sync-state\",\
      n    \"dataContentType\": \"application/json\",\n    \"time\": \"2025-07-02T19:25:31.834544956Z\",\n    \"data\": {\n      \"version\": \"1.0\",\n      \"values\": [\n        {\n          \"ResourceAddress\": \"/cluster/node/helix65.lab
      .eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n          \"data_type\": \"notification\",\n          \"value_type\": \"enumeration\",\n          \"value\": \"LOCKED\"\n        }\n      ]\n    }\n  }"
      time="2025-07-02T19:25:59Z" level=debug msg="event sent {\n    \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n    \"type\": \"event.sync.sync-status.synchronization-state-change\",\n    \"source\": \"/sync/sync-status/sync-state\",\
      n    \"dataContentType\": \"application/json\",\n    \"time\": \"2025-07-02T19:25:59.964667942Z\",\n    \"data\": {\n      \"version\": \"1.0\",\n      \"values\": [\n        {\n          \"ResourceAddress\": \"/cluster/node/helix65.lab
      .eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n          \"data_type\": \"notification\",\n          \"value_type\": \"enumeration\",\n          \"value\": \"HOLDOVER\"\n        }\n      ]\n    }\n  }"
      time="2025-07-02T19:26:01Z" level=debug msg="event sent {\n    \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n    \"type\": \"event.sync.sync-status.synchronization-state-change\",\n    \"source\": \"/sync/sync-status/sync-state\",\
      n    \"dataContentType\": \"application/json\",\n    \"time\": \"2025-07-02T19:26:01.185116778Z\",\n    \"data\": {\n      \"version\": \"1.0\",\n      \"values\": [\n        {\n          \"ResourceAddress\": \"/cluster/node/helix65.lab
      .eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n          \"data_type\": \"notification\",\n          \"value_type\": \"enumeration\",\n          \"value\": \"LOCKED\"\n        }\n      ]\n    }\n  }"
      time="2025-07-02T19:26:01Z" level=debug msg="event sent {\n    \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n    \"type\": \"event.sync.sync-status.synchronization-state-change\",\n    \"source\": \"/sync/sync-status/sync-state\",\
      n    \"dataContentType\": \"application/json\",\n    \"time\": \"2025-07-02T19:26:01.772831817Z\",\n    \"data\": {\n      \"version\": \"1.0\",\n      \"values\": [\n        {\n          \"ResourceAddress\": \"/cluster/node/helix65.lab
      .eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n          \"data_type\": \"notification\",\n          \"value_type\": \"enumeration\",\n          \"value\": \"LOCKED\"\n        }\n      ]\n    }\n  }"

      Expected results:
      A single event

              "event.sync.sync-status.synchronization-state-change" time="2025-07-02T19:24:38Z" level=debug msg="event sent {\n \"id\": \"3de41abb-e495-42df-9aa2-b8eebe98370c\",\n \"type\": \"event.sync.sync-status.synchronization-state-change\",\n \"source\": \"/sync/sync-status/sync-state\",\ n \"dataContentType\": \"application/json\",\n \"time\": \"2025-07-02T19:24:38.000730333Z\",\n \"data\": {\n \"version\": \"1.0\",\n \"values\": [\n {\n \"ResourceAddress\": \"/cluster/node/helix65.lab .eng.rdu2.redhat.com/sync/sync-status/sync-state\",\n \"data_type\": \"notification\",\n \"value_type\": \"enumeration\",\n \"value\": \"LOCKED\"\n }\n ]\n }\n }"

      Additional info:

          This noise may increase flakiness of QE tests which listen to this event.

              aputtur@redhat.com Aneesh Puttur
              rh-ee-dpopsuev Daniel Popsuevich
              None
              None
              Bonnie Block Bonnie Block
              None
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: