Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-60547

etcdHighCommitDurations firing on azure intermittently since

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • Approved
    • None
    • None
    • Release Note Not Required
    • None
    • None
    • None
    • None

      (Feel free to update this bug's summary to be more specific.)
      Component Readiness has found a potential regression in the following test:

      [bz-etcd][invariant] alert/etcdHighCommitDurations should not be at or above info

      Significant regression detected.
      Fishers Exact probability of a regression: 100.00%.
      Test pass rate dropped from 100.00% to 94.58%.

      Sample (being evaluated) Release: 4.20
      Start Time: 2025-08-08T00:00:00Z
      End Time: 2025-08-15T08:00:00Z
      Success Rate: 94.58%
      Successes: 262
      Failures: 15
      Flakes: 0
      Base (historical) Release: 4.19
      Start Time: 2025-05-18T00:00:00Z
      End Time: 2025-06-17T23:59:59Z
      Success Rate: 100.00%
      Successes: 848
      Failures: 0
      Flakes: 0

      View the test details report for additional context.

      Clear change in behaviour here, this just didn't happen prior to Aug 13. The first time we see it is in https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-azure-ovn-upgrade/1955502302714400768 which traces back to this payload which does contain an etcd change, but not one that looks related at first look unless it has to do with the vendoring update?

      The actual alert appears to be firing each time late in conformance testing, NOT during the upgrade phase. See sample intervals from the above job.

      The alert dashboard paints a similar picture, all the sudden on Aug 13, this is happening, and only on azure. It is not happening on 4.19 so it does not appear to be a cloud issue.

      Filed by: dgoodwin@redhat.com

              tjungblu@redhat.com Thomas Jungblut
              openshift-trt OpenShift Technical Release Team
              None
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: