Uploaded image for project: 'OCP Technical Release Team'
  1. OCP Technical Release Team
  2. TRT-1272

Investigate oauth & other disruption on 4.14

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • None
    • None
    • False
    • None
    • False

      Oauth disruption tests seem to be struggling on 4.14 payloads the last day or so.

      It's jumping around between azure ovn micro and aws sdn micro.

      https://grafana-loki.ci.openshift.org/d/ISnBj4LVk/disruption?var-platform=azure&var-platform=aws&var-percentile=P50&var-backend=oauth-api-new-connections&var-backend=openshift-api-new-connections&var-releases=4.14&var-releases=4.15&var-upgrade_type=micro&var-networks=ovn&var-networks=sdn&var-topologies=ha&var-architectures=amd64&var-min_job_runs=10&var-lookback=1&var-min_disruption_regression=5&var-min_disruption_job_list=0&var-master_nodes_updated=Y&from=now-30d&to=now&orgId=1

      Graphs are showing a slight uptick, though nothing out of the ordinary over the long term. 4.15 looks fine and is not tripping on these tests.

      Odds we have a disruption regression crossing ovn and sdn affecting only 4.14 are quite slim, this is likely just bad luck and the nature of disruption testing.

      A 500 ISE claiming etcd leader changed seems most common on Azure at least, but this is a very common problem going back for weeks. (though still possibly a noteworthy bug possibility) it is also present in 4.13, but at lower numbers, though we do not aggregate on 4.13 so this is logical it would hit less often. Seems to happen well before nodes are rolling and this might be worth a bug.

      Be on the lookout for cluster-network-liveness, indicating CI was having network problems, which has been spotted and causing at least one of these.

            rhn-engineering-dgoodwin Devan Goodwin
            rhn-engineering-dgoodwin Devan Goodwin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: