Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-37521

vSphere installs failing due to machines stuck in Provisioned

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • Proposed
    • CLOUD Sprint 258
    • 1
    • In Progress
    • Release Note Not Required
    • None
    • None
    • None
    • None
    • None

      Component Readiness has found a potential regression in the following test:

      operator conditions console

      Probability of significant regression: 99.99%

      Sample (being evaluated) Release: 4.17
      Start Time: 2024-07-18T00:00:00Z
      End Time: 2024-07-24T23:59:59Z
      Success Rate: 90.00%
      Successes: 81
      Failures: 9
      Flakes: 0

      Base (historical) Release: 4.16
      Start Time: 2024-05-31T00:00:00Z
      End Time: 2024-06-27T23:59:59Z
      Success Rate: 100.00%
      Successes: 147
      Failures: 0
      Flakes: 0

      View the test details report at https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Aggregation=none&Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&NetworkAccess=default&Platform=vsphere&Platform=vsphere&Scheduler=default&SecurityMode=default&Suite=unknown&Suite=unknown&Topology=ha&Topology=ha&Upgrade=none&Upgrade=none&baseEndTime=2024-06-27%2023%3A59%3A59&baseRelease=4.16&baseStartTime=2024-05-31%2000%3A00%3A00&capability=operator-conditions&columnGroupBy=Platform%2CArchitecture%2CNetwork&component=Management%20Console&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20vsphere%20unknown%20ha%20none&ignoreDisruption=true&ignoreMissing=false&includeVariant=Architecture%3Aamd64&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&minFail=3&pity=5&sampleEndTime=2024-07-24%2023%3A59%3A59&sampleRelease=4.17&sampleStartTime=2024-07-18%2000%3A00%3A00&testId=Operator%20results%3A258e3ff8c9692c937596663377c10e29&testName=operator%20conditions%20console

      https://sippy.dptools.openshift.org/sippy-ng/component_readiness/test_details?Aggregation=none&Architecture=amd64&Architecture=amd64&FeatureSet=default&FeatureSet=default&Installer=ipi&Installer=ipi&Network=ovn&Network=ovn&NetworkAccess=default&Platform=vsphere&Platform=vsphere&Scheduler=default&SecurityMode=default&Suite=serial&Suite=serial&Topology=ha&Topology=ha&Upgrade=none&Upgrade=none&baseEndTime=2024-06-27%2023%3A59%3A59&baseRelease=4.16&baseStartTime=2024-05-31%2000%3A00%3A00&capability=operator-conditions&columnGroupBy=Platform%2CArchitecture%2CNetwork&component=Management%20Console&confidence=95&dbGroupBy=Platform%2CArchitecture%2CNetwork%2CTopology%2CFeatureSet%2CUpgrade%2CSuite%2CInstaller&environment=amd64%20default%20ipi%20ovn%20vsphere%20serial%20ha%20none&ignoreDisruption=true&ignoreMissing=false&includeVariant=Architecture%3Aamd64&includeVariant=FeatureSet%3Adefault&includeVariant=Installer%3Aipi&includeVariant=Installer%3Aupi&includeVariant=Owner%3Aeng&includeVariant=Platform%3Aaws&includeVariant=Platform%3Aazure&includeVariant=Platform%3Agcp&includeVariant=Platform%3Ametal&includeVariant=Platform%3Avsphere&includeVariant=Topology%3Aha&minFail=3&pity=5&sampleEndTime=2024-07-24%2023%3A59%3A59&sampleRelease=4.17&sampleStartTime=2024-07-18%2000%3A00%3A00&testId=Operator%20results%3A258e3ff8c9692c937596663377c10e29&testName=operator%20conditions%20console

      There are several install problems on vsphere underway, this one is specific to a case where multiple operators report bad conditions at the end of install, and expending the camgi panel in spyglass, you can see multiple machines showing red because they are stuck in Provisioned state, not Running.

      I do not think this is an installer bug at this point, more likely vmware infrastructure. I just don't know what component to file against just yet.

      Sample job runs are easy to find in the links above, but here are some examples:

      https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-serial/1815735341861048320

      https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn/1814113813629243392

      Additional impacted tests:

      operator conditions authentication
      operator conditions cluster-autoscaler
      operator conditions console
      operator conditions control-plane-machine-set
      operator conditions image-registry
      operator conditions ingress
      operator conditions kube-controller-manager
      operator conditions machine-api
      operator conditions monitoring
      operator conditions network
      operator conditions openshift-apiserver
      operator conditions openshift-samples
      operator conditions operator-lifecycle-manager-packageserver
      install should succeed: cluster bootstrap
      

              rh-ee-tbarberb Theo Barber-Bany
              rhn-engineering-dgoodwin Devan Goodwin
              None
              None
              Wenxin Wei Wenxin Wei
              None
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: