Uploaded image for project: 'Multiple Architecture Enablement'
  1. Multiple Architecture Enablement
  2. MULTIARCH-2025

OCP 4.10 nightly builds from 4.10.0-0.nightly-s390x-2021-12-18-034912 to 4.10.0-0.nightly-s390x-2022-01-11-233015 fail to upgrade from OCP 4.9.11 and 4.9.12 for network type OVNKubernetes for zVM hypervisor environments

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 4.10.0
    • 4.10
    • Multi-Arch
    • None
    • False
    • False
    • NEW
    • NEW

    Description

      Description of problem:
      For zVM environments:
      1. OCP 4.10 nightly builds starting with 4.10.0-0.nightly-s390x-2021-12-18-034912 and through 4.10.0-0.nightly-s390x-2022-01-02-012917 fail to upgrade from OCP 4.9.11 and 4.9.12 when using network type OVNKubernetes (OVN).

      2. These same OCP 4.10 nightly builds starting with 4.10.0-0.nightly-s390x-2021-12-18-034912 and through 4.10.0-0.nightly-s390x-2022-01-02-012917 successfully upgrade from OCP 4.9.11 and 4.9.12 when using network type openshiftSDN (OVS).

      3. For these OCP 4.9.11 and 4.9.12 to OCP 4.10 upgrade failures, using the OCP nightly build 4.10.0-0.nightly-s390x-2021-12-24-235654 as an example, the "oc get clusterversion" command consistently reports:
      "Unable to apply 4.10.0-0.nightly-s390x-2021-12-24-235654: wait has exceeded 40 minutes for these operators: ingress"

      4. For these upgrade failures, using the OCP nightly build 4.10.0-0.nightly-s390x-2021-12-24-235654 as an example, the "oc get co" command consistently reports information similar to the following:

      NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
      authentication 4.10.0-0.nightly-s390x-2021-12-24-235654 True False True 35m APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver ()...
      baremetal 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 100m
      cloud-controller-manager 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 105m
      cloud-credential 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 104m
      cluster-autoscaler 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 100m
      config-operator 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 102m
      console 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 49m
      csi-snapshot-controller 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 101m
      dns 4.10.0-0.nightly-s390x-2021-12-24-235654 True True True 100m DNS default is degraded
      etcd 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 100m
      image-registry 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 93m
      ingress 4.10.0-0.nightly-s390x-2021-12-24-235654 True False True 92m The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: PodsScheduled=False (PodsNotScheduled: Some pods are not scheduled: Pod "router-default-cd6bdf7dd-h9nrx" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) had taint

      {node-role.kubernetes.io/master: }

      , that the pod didn't tolerate, 2 node(s) were unschedulable. Make sure you have sufficient worker nodes.)
      insights 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 94m
      kube-apiserver 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 97m
      kube-controller-manager 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 99m
      kube-scheduler 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 100m
      kube-storage-version-migrator 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 37m
      machine-api 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 101m
      machine-approver 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 101m
      machine-config 4.9.11 True True True 101m Unable to apply 4.10.0-0.nightly-s390x-2021-12-24-235654: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: controller version mismatch for rendered-master-091362904afdd033e03a72cdede84f52 expected f8249fc84f1a7dfd655c88fae80811ee9c76c34c has ddd96b04ede2eba72afea1355468a9985aacafe6: 0 (ready 0) out of 3 nodes are updating to latest configuration rendered-master-108b993eeddf5639db907c553a9834dc, retrying
      marketplace 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 100m
      monitoring 4.10.0-0.nightly-s390x-2021-12-24-235654 False True True 22m Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.
      network 4.10.0-0.nightly-s390x-2021-12-24-235654 True True True 102m DaemonSet "openshift-ovn-kubernetes/ovnkube-node" rollout is not making progress - pod ovnkube-node-2fhwv is in CrashLoopBackOff State...
      node-tuning 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 93m
      openshift-apiserver 4.10.0-0.nightly-s390x-2021-12-24-235654 True False True 94m APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()
      openshift-controller-manager 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 97m
      openshift-samples 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 61m
      operator-lifecycle-manager 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 101m
      operator-lifecycle-manager-catalog 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 101m
      operator-lifecycle-manager-packageserver 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 58m
      service-ca 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 102m
      storage 4.10.0-0.nightly-s390x-2021-12-24-235654 True False False 102m

      5. For the network cluster operator, the ovnkube pods are in a CrashLoopBackOff state.
      For example:
      DaemonSet "openshift-ovn-kubernetes/ovnkube-node" rollout is not making progress - pod ovnkube-node-2fhwv is in CrashLoopBackOff State...

      Version-Release number of selected component (if applicable):
      This issue has been consistently found with the following tested builds:
      1. 4.10.0-0.nightly-s390x-2021-12-18-034912
      2. 4.10.0-0.nightly-s390x-2021-12-20-215258
      3. 4.10.0-0.nightly-s390x-2021-12-21-231942
      4. 4.10.0-0.nightly-s390x-2021-12-22-053640
      5. 4.10.0-0.nightly-s390x-2021-12-23-063012
      6. 4.10.0-0.nightly-s390x-2021-12-24-010839
      7. 4.10.0-0.nightly-s390x-2021-12-24-154536
      8. 4.10.0-0.nightly-s390x-2021-12-24-235654
      9. 4.10.0-0.nightly-s390x-2022-01-02-012917

      How reproducible:
      Consistently reproducible.

      Steps to Reproduce:
      1. In a zVM environment, using network type OVNKubernetes, attempt to upgrade from OCP 4.9.11 or 4.9.12 to any OCP 4.10 nightly build between 4.10.0-0.nightly-s390x-2021-12-18-034912 and 4.10.0-0.nightly-s390x-2022-01-02-012917.

      Actual results:
      Upgrade fails with above network cluster operator ovnkube pod CrashLoopBackOff issues.

      Expected results:
      Upgrades should succeed as they consistently do for network type openshiftSDN (OVS).

      Additional info:

      Thank you.

      Attachments

        Activity

          People

            psundararaman Prashanth Sundararaman (Inactive)
            jira-bugzilla-migration RH Bugzilla Integration
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: