Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-8253

cvo should be able to recover from oc adm upgrade --to-multi-arch followed by --clear

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.13
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • No
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      applying oc adm upgrade --to-multi-arch and immediately --clear causing cvo to reconcile back to itself and not accept --to-multi-arch anymore

      Version-Release number of selected component (if applicable):

      4.13.0-ec.2

      How reproducible:

      3/3

      Steps to Reproduce:

      1. install homogenous cluster and set channel so RetrievedUpdates=True
      2. oc adm upgrade --to-multi-arch && oc adm upgrade --clear
      3. oc adm upgrade --to-multi-arch 

      Actual results:

      cluster accepts the commands
      ❯ oc adm upgrade --to-multi-arch && oc adm upgrade --clear
      Requested update to multi cluster architecture
      Cleared the update field, still at 4.13.0-ec.2
      
      Thu 02 Mar 2023 11:31:36 IST
      appears progressing
      ❯ oc adm upgrade
      info: An upgrade is in progress. Working towards 4.13.0-ec.2: 22 of 831 done (2% complete)
      Upstream is unset, so the cluster will use an appropriate default.
      Channel: candidate-4.13
      Recommended updates:  VERSION     IMAGE
        4.13.0-ec.3 quay.io/openshift-release-dev/ocp-release@sha256:5e8f403a14eed840b01434115300f2e68cd1232aa47f9509433a46341da2f2b8
      
      in cvo Conditions:
      2023-03-02T09:29:45Z RetrievedUpdates=True : 
      2023-03-02T08:38:21Z ImplicitlyEnabledCapabilities=False AsExpected: Capabilities match configured spec
      2023-03-02T08:38:21Z ReleaseAccepted=True PayloadLoaded: Payload loaded version="4.13.0-ec.2" image="quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321" architecture="amd64"
      2023-03-02T08:59:13Z Available=True : Done applying 4.13.0-ec.2
      2023-03-02T08:59:13Z Failing=False : 
      2023-03-02T09:31:36Z Progressing=True : Working towards 4.13.0-ec.2: 22 of 831 done (2% complete)
      
      after a short wile
      Thu 02 Mar 2023 11:32:14 IST
      ❯ oc adm upgrade
      Cluster version is 4.13.0-ec.2
      Upstream is unset, so the cluster will use an appropriate default.
      Channel: candidate-4.13
      Recommended updates:  VERSION     IMAGE
        4.13.0-ec.3 quay.io/openshift-release-dev/ocp-release@sha256:5e8f403a14eed840b01434115300f2e68cd1232aa47f9509433a46341da2f2b8
      
      cvo Conditions:
      2023-03-02T09:29:45Z RetrievedUpdates=True : 
      2023-03-02T08:38:21Z ImplicitlyEnabledCapabilities=False AsExpected: Capabilities match configured spec
      2023-03-02T08:38:21Z ReleaseAccepted=True PayloadLoaded: Payload loaded version="4.13.0-ec.2" image="quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321" architecture="amd64"
      2023-03-02T08:59:13Z Available=True : Done applying 4.13.0-ec.2
      2023-03-02T08:59:13Z Failing=False : 
      2023-03-02T09:32:13Z Progressing=False : Cluster version is 4.13.0-ec.2
      
      History:
      2023-03-02T09:31:36Z 2023-03-02T09:32:13Z Completed 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321
      2023-03-02T09:31:36Z 2023-03-02T09:31:36Z Partial 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:bdc145f7f6347433f8461a1133d6354abf52268925ce7459a4294d44b9beb4ef
      2023-03-02T08:38:21Z 2023-03-02T08:59:13Z Completed 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321
      
      
      at this point reapplying the command
      ❯ oc adm upgrade --to-multi-arch
      Requested update to multi cluster architecture
      
      desiredUpdate:
          architecture:    Multi
          force:    false
          image:    
          version:    4.13.0-ec.2
      
      cluster is not progressing anymore!
      
      cvo History:
      2023-03-02T09:31:36Z 2023-03-02T09:32:13Z Completed 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321
      2023-03-02T09:31:36Z 2023-03-02T09:31:36Z Partial 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:bdc145f7f6347433f8461a1133d6354abf52268925ce7459a4294d44b9beb4ef
      2023-03-02T08:38:21Z 2023-03-02T08:59:13Z Completed 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321
      
      cvo Conditions:
      2023-03-02T09:29:45Z RetrievedUpdates=True : 
      2023-03-02T08:38:21Z ImplicitlyEnabledCapabilities=False AsExpected: Capabilities match configured spec
      2023-03-02T08:38:21Z ReleaseAccepted=True PayloadLoaded: Payload loaded version="4.13.0-ec.2" image="quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321" architecture="amd64"
      2023-03-02T08:59:13Z Available=True : Done applying 4.13.0-ec.2
      2023-03-02T08:59:13Z Failing=False : 
      2023-03-02T09:32:13Z Progressing=False : Cluster version is 4.13.0-ec.2
      
      
      ❯ oc get -n openshift-cluster-version pod                                                                                                             1s 
      NAME                                        READY   STATUS      RESTARTS      AGE
      cluster-version-operator-664566d6fb-p8hcs   1/1     Running     2 (45m ago)   55m
      version-4.13.0-ec.2-fshbs-znwdl             0/1     Completed   0             5m25s
      
      
      in cvo log:
      
      31316 I0302 09:31:19.746701       1 batch.go:55] No active pods for job version-4.13.0-ec.2-fshbs in namespace openshift-cluster-version
      31317 I0302 09:31:19.746719       1 batch.go:29] Job version-4.13.0-ec.2-fshbs in namespace openshift-cluster-version is not ready, continuing to wait.
      31318 I0302 09:31:20.230455       1 cvo.go:658] Started syncing upgradeable "openshift-cluster-version/version" (2023-03-02 09:31:20.230448523 +0000 UTC m=+2409.855026434)
      31319 I0302 09:31:20.230514       1 upgradeable.go:68] Upgradeable conditions were recently checked, will try later.
      31320 I0302 09:31:20.230520       1 cvo.go:660] Finished syncing upgradeable "openshift-cluster-version/version" (69.339µs)
      31321 I0302 09:31:20.230531       1 cvo.go:639] Started syncing available updates "openshift-cluster-version/version" (2023-03-02 09:31:20.230528964 +0000 UTC m=+2409.855106877)
      31322 I0302 09:31:20.230569       1 availableupdates.go:56] Retrieving available updates again, because the architecture has changed from "Multi" to "amd64"  

      Expected results:

      cluster is able to accept --to-multi-arch and transition to heterogeneous payload 

      Additional info:

      must-gather attached in another message below

              Unassigned Unassigned
              evakhoni@redhat.com Evgeni Vakhonin
              None
              None
              Jian Li Jian Li
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: