applying oc adm upgrade --to-multi-arch and immediately --clear causing cvo to reconcile back to itself and not accept --to-multi-arch anymore

4.13.0-ec.2

3/3

1. install homogenous cluster and set channel so RetrievedUpdates=True
2. oc adm upgrade --to-multi-arch && oc adm upgrade --clear
3. oc adm upgrade --to-multi-arch 

cluster accepts the commands
❯ oc adm upgrade --to-multi-arch && oc adm upgrade --clear
Requested update to multi cluster architecture
Cleared the update field, still at 4.13.0-ec.2

Thu 02 Mar 2023 11:31:36 IST
appears progressing
❯ oc adm upgrade
info: An upgrade is in progress. Working towards 4.13.0-ec.2: 22 of 831 done (2% complete)
Upstream is unset, so the cluster will use an appropriate default.
Channel: candidate-4.13
Recommended updates:  VERSION     IMAGE
  4.13.0-ec.3 quay.io/openshift-release-dev/ocp-release@sha256:5e8f403a14eed840b01434115300f2e68cd1232aa47f9509433a46341da2f2b8

in cvo Conditions:
2023-03-02T09:29:45Z RetrievedUpdates=True : 
2023-03-02T08:38:21Z ImplicitlyEnabledCapabilities=False AsExpected: Capabilities match configured spec
2023-03-02T08:38:21Z ReleaseAccepted=True PayloadLoaded: Payload loaded version="4.13.0-ec.2" image="quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321" architecture="amd64"
2023-03-02T08:59:13Z Available=True : Done applying 4.13.0-ec.2
2023-03-02T08:59:13Z Failing=False : 
2023-03-02T09:31:36Z Progressing=True : Working towards 4.13.0-ec.2: 22 of 831 done (2% complete)

after a short wile
Thu 02 Mar 2023 11:32:14 IST
❯ oc adm upgrade
Cluster version is 4.13.0-ec.2
Upstream is unset, so the cluster will use an appropriate default.
Channel: candidate-4.13
Recommended updates:  VERSION     IMAGE
  4.13.0-ec.3 quay.io/openshift-release-dev/ocp-release@sha256:5e8f403a14eed840b01434115300f2e68cd1232aa47f9509433a46341da2f2b8

cvo Conditions:
2023-03-02T09:29:45Z RetrievedUpdates=True : 
2023-03-02T08:38:21Z ImplicitlyEnabledCapabilities=False AsExpected: Capabilities match configured spec
2023-03-02T08:38:21Z ReleaseAccepted=True PayloadLoaded: Payload loaded version="4.13.0-ec.2" image="quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321" architecture="amd64"
2023-03-02T08:59:13Z Available=True : Done applying 4.13.0-ec.2
2023-03-02T08:59:13Z Failing=False : 
2023-03-02T09:32:13Z Progressing=False : Cluster version is 4.13.0-ec.2

History:
2023-03-02T09:31:36Z 2023-03-02T09:32:13Z Completed 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321
2023-03-02T09:31:36Z 2023-03-02T09:31:36Z Partial 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:bdc145f7f6347433f8461a1133d6354abf52268925ce7459a4294d44b9beb4ef
2023-03-02T08:38:21Z 2023-03-02T08:59:13Z Completed 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321


at this point reapplying the command
❯ oc adm upgrade --to-multi-arch
Requested update to multi cluster architecture

desiredUpdate:
    architecture:    Multi
    force:    false
    image:    
    version:    4.13.0-ec.2

cluster is not progressing anymore!

cvo History:
2023-03-02T09:31:36Z 2023-03-02T09:32:13Z Completed 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321
2023-03-02T09:31:36Z 2023-03-02T09:31:36Z Partial 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:bdc145f7f6347433f8461a1133d6354abf52268925ce7459a4294d44b9beb4ef
2023-03-02T08:38:21Z 2023-03-02T08:59:13Z Completed 4.13.0-ec.2 quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321

cvo Conditions:
2023-03-02T09:29:45Z RetrievedUpdates=True : 
2023-03-02T08:38:21Z ImplicitlyEnabledCapabilities=False AsExpected: Capabilities match configured spec
2023-03-02T08:38:21Z ReleaseAccepted=True PayloadLoaded: Payload loaded version="4.13.0-ec.2" image="quay.io/openshift-release-dev/ocp-release@sha256:01192353b3c3e536779cfa0fc910064299df15ce01be0cff7188868588d32321" architecture="amd64"
2023-03-02T08:59:13Z Available=True : Done applying 4.13.0-ec.2
2023-03-02T08:59:13Z Failing=False : 
2023-03-02T09:32:13Z Progressing=False : Cluster version is 4.13.0-ec.2


❯ oc get -n openshift-cluster-version pod                                                                                                             1s 
NAME                                        READY   STATUS      RESTARTS      AGE
cluster-version-operator-664566d6fb-p8hcs   1/1     Running     2 (45m ago)   55m
version-4.13.0-ec.2-fshbs-znwdl             0/1     Completed   0             5m25s


in cvo log:

31316 I0302 09:31:19.746701       1 batch.go:55] No active pods for job version-4.13.0-ec.2-fshbs in namespace openshift-cluster-version
31317 I0302 09:31:19.746719       1 batch.go:29] Job version-4.13.0-ec.2-fshbs in namespace openshift-cluster-version is not ready, continuing to wait.
31318 I0302 09:31:20.230455       1 cvo.go:658] Started syncing upgradeable "openshift-cluster-version/version" (2023-03-02 09:31:20.230448523 +0000 UTC m=+2409.855026434)
31319 I0302 09:31:20.230514       1 upgradeable.go:68] Upgradeable conditions were recently checked, will try later.
31320 I0302 09:31:20.230520       1 cvo.go:660] Finished syncing upgradeable "openshift-cluster-version/version" (69.339µs)
31321 I0302 09:31:20.230531       1 cvo.go:639] Started syncing available updates "openshift-cluster-version/version" (2023-03-02 09:31:20.230528964 +0000 UTC m=+2409.855106877)
31322 I0302 09:31:20.230569       1 availableupdates.go:56] Retrieving available updates again, because the architecture has changed from "Multi" to "amd64"  

cluster is able to accept --to-multi-arch and transition to heterogeneous payload 

must-gather attached in another message below