Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-7651

4.12 to 4.13-9.2 upgrades fail with semodule -DB enabled

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • 4.13.0
    • RHCOS
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      On attempting to upgrade from 4.12.3 to RHCOS 413.92.202302081904-0,I see master mcp in degraded state with the below message:
      ===================
      unexpected on-disk state validating against rendered-master-733a7422eec1f1e0d3de151a31b7a054: expected target osImageURL "registry.build05.ci.openshift.org/ci-ln-5f80vhb/stable@sha256:4b5fb23df6b360e41da18e001d56e4fb5e9e575318a0b9000a1a316c74c9515d", have "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50fa4c3f33d7fb8a3f8db93636937cb888a55718fe3833f339cbcc55dc6d0e3a" ("3abc4116c09e75764c30cc14918dfc343d37b9368b5d10cf2a9f5bd278de046b"); possible root cause: error: Child process exited with code 1

      Version-Release number of selected component (if applicable):

      4.12.3 to RHCOS 413.92.202302081904-0

      How reproducible:

      100%

      Steps to Reproduce:

      1. Run ocp upgrade from 4.12.3 to  to RHCOS 413.92.202302081904-0 
      2.
      3.
      

      Actual results:

      cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get nodes
      NAME                                             STATUS                        ROLES                  AGE   VERSION
      cnv-qe-infra-29.cnvqe2.lab.eng.rdu2.redhat.com   Ready                         control-plane,master   25h   v1.25.4+a34b9e9
      cnv-qe-infra-30.cnvqe2.lab.eng.rdu2.redhat.com   Ready                         control-plane,master   25h   v1.25.4+a34b9e9
      cnv-qe-infra-31.cnvqe2.lab.eng.rdu2.redhat.com   Ready,SchedulingDisabled      control-plane,master   25h   v1.25.4+a34b9e9
      cnv-qe-infra-32.cnvqe2.lab.eng.rdu2.redhat.com   NotReady,SchedulingDisabled   worker                 25h   v1.25.4+a34b9e9
      cnv-qe-infra-33.cnvqe2.lab.eng.rdu2.redhat.com   Ready                         worker                 25h   v1.25.4+a34b9e9
      cnv-qe-infra-34.cnvqe2.lab.eng.rdu2.redhat.com   Ready                         worker                 25h   v1.25.4+a34b9e9
      cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get mcp
      NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master   rendered-master-1ff7b81307234c62d951a7ef88d7fbc9   False     True       True       3              0                   0                     1                      25h
      worker   rendered-worker-c344a2d515cc2c21644075a59e573a7d   False     True       False      3              0                   0                     0                      25h
      cnv-qe-jenkins@cnv-qe-infra-01:~$
      cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get co
      NAME                                       VERSION                                                   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      authentication                             4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        True          True       11m     APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver ()...
      cloud-controller-manager                   4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      cloud-credential                           4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      cluster-autoscaler                         4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      config-operator                            4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      console                                    4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      25h     
      control-plane-machine-set                  4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      csi-snapshot-controller                    4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      dns                                        4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        True          False      26h     DNS "default" reports Progressing=True: "Have 5 available node-resolver pods, want 6."
      etcd                                       4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      25h     
      image-registry                             4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        True          False      24h     Progressing: The registry is ready...
      ingress                                    4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      25h     
      insights                                   4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      25h     
      kube-apiserver                             4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      25h     
      kube-controller-manager                    4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      25h     
      kube-scheduler                             4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      25h     
      kube-storage-version-migrator              4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      machine-api                                4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      25h     
      machine-approver                           4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      machine-config                             4.12.3                                                    False       True          True       20h     Cluster not available for [{operator 4.12.3}]: failed to apply machine config daemon manifests: error during waitForDaemonsetRollout: [timed out waiting for the condition, daemonset machine-config-daemon is not ready. status: (desired: 6, updated: 6, ready: 5, unavailable: 1)]
      marketplace                                4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      monitoring                                 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      25h     
      network                                    4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        True          False      26h     DaemonSet "/openshift-multus/multus" is not available (awaiting 1 nodes)...
      node-tuning                                4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      21h     
      openshift-apiserver                        4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         True       25h     APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()
      openshift-controller-manager               4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        True          False      25h     Progressing: deployment/controller-manager: updated replicas is 1, desired replicas is 3...
      openshift-samples                          4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      21h     
      operator-lifecycle-manager                 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      operator-lifecycle-manager-catalog         4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      operator-lifecycle-manager-packageserver   4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      25h     
      service-ca                                 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      storage                                    4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest   True        False         False      26h     
      cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get clusterversion
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.12.3    True        True          21h     Unable to apply 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest: wait has exceeded 40 minutes for these operators: authentication, openshift-apiserver
      cnv-qe-jenkins@cnv-qe-infra-01:~$ 

      Expected results:

      Successful upgrade

      Additional info:

       

            Unassigned Unassigned
            rhn-support-dbasunag Debarati Basu-Nag
            Rio Liu Rio Liu
            Votes:
            0 Vote for this issue
            Watchers:
            24 Start watching this issue

              Created:
              Updated: