Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-2125

CVO skips reconciling the installed optional resources in the 4.11 to 4.12 upgrade

    XMLWordPrintable

Details

    • Important
    • 3
    • OTA 226
    • 1
    • Approved
    • False
    • Hide

      None

      Show
      None
    • If Release Note Needed, Set a Value
    • Set a Value

    Description

      Description of problem:

      In a 4.11 cluster with only openshift-samples enabled, the 4.12 introduced optional COs console and insights are installed. While upgrading to 4.12, CVO considers them to be disabled explicitly and skips reconciling them. So these COs are not upgraded to 4.12.
      
      Installed COs cannot be disabled, so CVO is supposed to implicitly enable them.
      
      
      $ oc get clusterversion -oyaml
      {
        "apiVersion": "config.openshift.io/v1",
           "kind": "ClusterVersion",
           "metadata": {
               "creationTimestamp": "2022-09-30T05:02:31Z",
               "generation": 3,
               "name": "version",
               "resourceVersion": "134808",
               "uid": "bd95473f-ffda-402d-8fe3-74f852a9d6eb"
           },
           "spec": {
               "capabilities": {
                   "additionalEnabledCapabilities": [
                       "openshift-samples"
                   ],
                   "baselineCapabilitySet": "None"
               },
               "channel": "stable-4.11",
               "clusterID": "8eda5167-a730-4b39-be1d-214a80506d34",
               "desiredUpdate": {
                   "force": true,
                   "image": "registry.ci.openshift.org/ocp/release@sha256:2c8e617830f84ac1ee1bfcc3581010dec4ae5d9cad7a54271574e8d91ef5ecbc",
                   "version": ""
               }
           },
           "status": {
               "availableUpdates": null,
               "capabilities": {
                   "enabledCapabilities": [
                       "openshift-samples"
                   ],
                   "knownCapabilities": [
                       "Console",
                       "Insights",
                       "Storage",
                       "baremetal",
                       "marketplace",
                       "openshift-samples"
                   ]
               },
               "conditions": [
                   {
                       "lastTransitionTime": "2022-09-30T05:02:33Z",
                       "message": "Unable to retrieve available updates: currently reconciling cluster version 4.12.0-0.nightly-2022-09-28-204419 not found in the \"stable-4.11\" channel",
                       "reason": "VersionNotFound",
                       "status": "False",
                       "type": "RetrievedUpdates"
                   },
                   {
                       "lastTransitionTime": "2022-09-30T05:02:33Z",
                       "message": "Capabilities match configured spec",
                       "reason": "AsExpected",
                       "status": "False",
                       "type": "ImplicitlyEnabledCapabilities"
                   },
                   {
                       "lastTransitionTime": "2022-09-30T05:02:33Z",
                       "message": "Payload loaded version=\"4.12.0-0.nightly-2022-09-28-204419\" image=\"registry.ci.openshift.org/ocp/release@sha256:2c8e617830f84ac1ee1bfcc3581010dec4ae5d9cad7a54271574e8d91ef5ecbc\" architecture=\"amd64\"",
                       "reason": "PayloadLoaded",
                       "status": "True",
                       "type": "ReleaseAccepted"
                   },
                   {
                       "lastTransitionTime": "2022-09-30T05:23:18Z",
                       "message": "Done applying 4.12.0-0.nightly-2022-09-28-204419",
                       "status": "True",
                       "type": "Available"
                   },
                   {
                       "lastTransitionTime": "2022-09-30T07:05:42Z",
                       "status": "False",
                       "type": "Failing"
                   },
                   {
                       "lastTransitionTime": "2022-09-30T07:41:53Z",
                       "message": "Cluster version is 4.12.0-0.nightly-2022-09-28-204419",
                       "status": "False",
                       "type": "Progressing"
                   }
               ],
               "desired": {
                   "image": "registry.ci.openshift.org/ocp/release@sha256:2c8e617830f84ac1ee1bfcc3581010dec4ae5d9cad7a54271574e8d91ef5ecbc",
                   "version": "4.12.0-0.nightly-2022-09-28-204419"
               },
               "history": [
                   {
                       "completionTime": "2022-09-30T07:41:53Z",
                       "image": "registry.ci.openshift.org/ocp/release@sha256:2c8e617830f84ac1ee1bfcc3581010dec4ae5d9cad7a54271574e8d91ef5ecbc",
                       "startedTime": "2022-09-30T06:42:01Z",
                       "state": "Completed",
                       "verified": false,
                       "version": "4.12.0-0.nightly-2022-09-28-204419"
                   },
                   {
                       "completionTime": "2022-09-30T05:23:18Z",
                       "image": "registry.ci.openshift.org/ocp/release@sha256:5a6f6d1bf5c752c75d7554aa927c06b5ea0880b51909e83387ee4d3bca424631",
                       "startedTime": "2022-09-30T05:02:33Z",
                       "state": "Completed",
                       "verified": false,
                       "version": "4.11.0-0.nightly-2022-09-29-191451"
                   }
               ],
               "observedGeneration": 3,
               "versionHash": "CSCJ2fxM_2o="
           }
       }
      
      $ oc get co
       NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      authentication                             4.12.0-0.nightly-2022-09-28-204419   True        False         False      93m     
      cloud-controller-manager                   4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h56m   
      cloud-credential                           4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h59m   
      cluster-autoscaler                         4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h53m   
      config-operator                            4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
      console                                    4.11.0-0.nightly-2022-09-29-191451   True        False         False      3h45m   
      control-plane-machine-set                  4.12.0-0.nightly-2022-09-28-204419   True        False         False      117m    
      csi-snapshot-controller                    4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
      dns                                        4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h53m   
      etcd                                       4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h52m   
      image-registry                             4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h46m   
      ingress                                    4.12.0-0.nightly-2022-09-28-204419   True        False         False      151m    
      insights                                   4.11.0-0.nightly-2022-09-29-191451   True        False         False      3h48m   
      kube-apiserver                             4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h50m   
      kube-controller-manager                    4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h51m   
      kube-scheduler                             4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h51m   
      kube-storage-version-migrator              4.12.0-0.nightly-2022-09-28-204419   True        False         False      91m     
      machine-api                                4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h50m   
      machine-approver                           4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
      machine-config                             4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h52m   
      monitoring                                 4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h44m   
      network                                    4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h55m   
      node-tuning                                4.12.0-0.nightly-2022-09-28-204419   True        False         False      113m    
      openshift-apiserver                        4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h48m   
      openshift-controller-manager               4.12.0-0.nightly-2022-09-28-204419   True        False         False      113m    
      openshift-samples                          4.12.0-0.nightly-2022-09-28-204419   True        False         False      116m    
      operator-lifecycle-manager                 4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
      operator-lifecycle-manager-catalog         4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
      operator-lifecycle-manager-packageserver   4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h48m   
      service-ca                                 4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m   
      storage                                    4.12.0-0.nightly-2022-09-28-204419   True        False         False      3h54m 
      
      

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-09-28-204419
      

      How reproducible:

      Always
      

      Steps to Reproduce:

      1. Install a 4.11 cluster with only openshift-samples enabled
      2. Upgrade to 4.12
      3.
      

      Actual results:

      The 4.12 introduced optional CO console and insights are not upgraded to 4.12
      
      

      Expected results:

      All the installed COs get upgraded
      

      Additional info:

       

      Attachments

        1. current.log
          20.24 MB
        2. cvo.tar
          20.58 MB

        Issue Links

          Activity

            People

              jottofar Jack Ottofaro (Inactive)
              yanyang@redhat.com Yang Yang
              Yang Yang Yang Yang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: