Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-55238

New disruption monitoring reporting 3 bars of disruption during kube-apiserver progressing

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • 4.20.0
    • 4.19.0
    • kube-apiserver
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • No
    • Rejected
    • None
    • In Progress
    • Release Note Not Required
    • N/A
    • None
    • None
    • None
    • None

      It appears that all upgrade jobs now have interval charts showing three separate bars of mass disruption during kube-apiserver upgrade:

       https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.19-e2e-vsphere-ovn-upgrade/1909492594849615872

      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-bm-upgrade/1914490406016389120

      https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-upgrade/1911995592435830784

       

      We suspect that this kind of disruption is actually expected given it's being measured from localhost as each master is updated.

      We can't leave this disruption in the charts as it will be an eternal pain point for anyone examining them, it looks quite alarming.

      There is precedent for shutting down disruption monitors when nodes are updating, specifically in the in-cluster disruption monitoring. I believe this was done by monitoring EndpointSlice somewhere around here https://github.com/openshift/origin/blob/main/pkg/monitortests/network/disruptionpodnetwork/monitortest.go to determine when to shut down and restart the monitor. Doing this when the apiserver is rolling out would be one option.

      Another option might be to alter or omit them when intervals are being returned in the monitortest, if they overlap with a progressing interval.

       

      Abu points out on slack: "if we want to skip the ones that overlaps with a roll out, one thing we need to keep in mind is that, the rollout interval for an apiserver takes into account [termination start ... termination end], but it should be
      [termination start ... termination end (old instance) ... ready to accept request (new instance)]"

      Not critical for 4.19 release.

       

       

              Unassigned Unassigned
              rhn-engineering-dgoodwin Devan Goodwin
              None
              None
              Ke Wang Ke Wang
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: