upgrade for a disconnected cluster get hang on retrieving and verifying payload

This bug is a backport clone of [Bugzilla Bug 2090680](https://bugzilla.redhat.com/show_bug.cgi?id=2090680). The following is the description of the original bug:
—
Description of problem:

Version-Release number of the following components:
4.11.0-0.nightly-2022-05-25-123329

How reproducible:
Always

Steps to Reproduce:
1. set up a cluster in a restricted network using 4.11.0-0.nightly-2022-05-25-123329
2. mirror 4.11.0-0.nightly-2022-05-25-193227 to private registry
3. upgrade the cluster to 4.11.0-0.nightly-2022-05-25-193227 without --force option
$ oc adm upgrade --allow-explicit-upgrade --to-image registry.ci.openshift.org/ocp/release@sha256:83ca476a63dfafa49e35cab2ded1fbf3991cc3483875b1bf639eabda31faadfd

Actual results:
Wait for 3+ hours, no any upgrade history info in clusterversion, from event log, only can see "Retrieving and verifying payload".

[root@preserve-jialiu-ansible ~]# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.0-0.nightly-2022-05-25-123329 True False 160m Cluster version is 4.11.0-0.nightly-2022-05-25-123329

[root@preserve-jialiu-ansible ~]# oc get clusterversion -o yaml
apiVersion: v1
items:

• apiVersion: config.openshift.io/v1
  kind: ClusterVersion
  metadata:
  creationTimestamp: "2022-05-26T03:51:28Z"
  generation: 3
  name: version
  resourceVersion: "62069"
  uid: b5674b4b-7295-4287-904c-94fe1112659b
  spec:
  channel: stable-4.11
  clusterID: 027285eb-b4ea-4127-85b6-031c1af7db72
  desiredUpdate:
  force: false
  image: registry.ci.openshift.org/ocp/release@sha256:83ca476a63dfafa49e35cab2ded1fbf3991cc3483875b1bf639eabda31faadfd
  version: ""
  status:
  availableUpdates: null
  capabilities:
  enabledCapabilities:
• baremetal
• marketplace
• openshift-samples
  knownCapabilities:
• baremetal
• marketplace
• openshift-samples
  conditions:
• lastTransitionTime: "2022-05-26T03:51:31Z"
  message: Capabilities match configured spec
  reason: AsExpected
  status: "False"
  type: ImplicitlyEnabledCapabilities
• lastTransitionTime: "2022-05-26T03:51:31Z"
  message: Payload loaded version="4.11.0-0.nightly-2022-05-25-123329" image="registry.ci.openshift.org/ocp/release@sha256:13bfc31eb4a284ce691e848c25d9120dbde3f0852d4be64be4b90953ac914bf1"
  reason: PayloadLoaded
  status: "True"
  type: ReleaseAccepted
• lastTransitionTime: "2022-05-26T04:23:06Z"
  message: Done applying 4.11.0-0.nightly-2022-05-25-123329
  status: "True"
  type: Available
• lastTransitionTime: "2022-05-26T04:21:21Z"
  status: "False"
  type: Failing
• lastTransitionTime: "2022-05-26T04:23:06Z"
  message: Cluster version is 4.11.0-0.nightly-2022-05-25-123329
  status: "False"
  type: Progressing
• lastTransitionTime: "2022-05-26T03:51:31Z"
  message: 'Unable to retrieve available updates: Get "https://api.openshift.com/api/upgrades_info/v1/graph?arch=amd64&channel=stable-4.11&id=027285eb-b4ea-4127-85b6-031c1af7db72&version=4.11.0-0.nightly-2022-05-25-123329":
  dial tcp 34.228.45.157:443: connect: connection timed out'
  reason: RemoteFailed
  status: "False"
  type: RetrievedUpdates
  desired:
  image: registry.ci.openshift.org/ocp/release@sha256:13bfc31eb4a284ce691e848c25d9120dbde3f0852d4be64be4b90953ac914bf1
  version: 4.11.0-0.nightly-2022-05-25-123329
  history:
• completionTime: "2022-05-26T04:23:06Z"
  image: registry.ci.openshift.org/ocp/release@sha256:13bfc31eb4a284ce691e848c25d9120dbde3f0852d4be64be4b90953ac914bf1
  startedTime: "2022-05-26T03:51:31Z"
  state: Completed
  verified: false
  version: 4.11.0-0.nightly-2022-05-25-123329
  observedGeneration: 2
  versionHash: jOIXVtM5Y-g=
  kind: List
  metadata:
  resourceVersion: ""

[root@preserve-jialiu-ansible ~]# oc get event -n openshift-cluster-version
LAST SEEN TYPE REASON OBJECT MESSAGE
3h11m Warning FailedScheduling pod/cluster-version-operator-b4b6c5f9b-p7fjq no nodes available to schedule pods
3h9m Warning FailedScheduling pod/cluster-version-operator-b4b6c5f9b-p7fjq no nodes available to schedule pods
3h4m Normal Scheduled pod/cluster-version-operator-b4b6c5f9b-p7fjq Successfully assigned openshift-cluster-version/cluster-version-operator-b4b6c5f9b-p7fjq to jialiu411a-5nb8n-master-2 by jialiu411a-5nb8n-bootstrap
3h2m Warning FailedMount pod/cluster-version-operator-b4b6c5f9b-p7fjq MountVolume.SetUp failed for volume "serving-cert" : secret "cluster-version-operator-serving-cert" not found
3h1m Warning FailedMount pod/cluster-version-operator-b4b6c5f9b-p7fjq Unable to attach or mount volumes: unmounted volumes=[serving-cert], unattached volumes=[etc-ssl-certs etc-cvo-updatepayloads serving-cert service-ca kube-api-access]: timed out waiting for the condition
3h1m Normal Pulling pod/cluster-version-operator-b4b6c5f9b-p7fjq Pulling image "registry.ci.openshift.org/ocp/release@sha256:13bfc31eb4a284ce691e848c25d9120dbde3f0852d4be64be4b90953ac914bf1"
3h1m Normal Pulled pod/cluster-version-operator-b4b6c5f9b-p7fjq Successfully pulled image "registry.ci.openshift.org/ocp/release@sha256:13bfc31eb4a284ce691e848c25d9120dbde3f0852d4be64be4b90953ac914bf1" in 1.384468759s
3h1m Normal Created pod/cluster-version-operator-b4b6c5f9b-p7fjq Created container cluster-version-operator
3h1m Normal Started pod/cluster-version-operator-b4b6c5f9b-p7fjq Started container cluster-version-operator
3h11m Normal SuccessfulCreate replicaset/cluster-version-operator-b4b6c5f9b Created pod: cluster-version-operator-b4b6c5f9b-p7fjq
3h11m Normal ScalingReplicaSet deployment/cluster-version-operator Scaled up replica set cluster-version-operator-b4b6c5f9b to 1
3h12m Normal LeaderElection configmap/version jialiu411a-5nb8n-bootstrap_0a3ff57f-66cf-4f93-bbe0-484effcc4383 became leader
3h12m Normal RetrievePayload clusterversion/version Retrieving and verifying payload version="4.11.0-0.nightly-2022-05-25-123329" image="registry.ci.openshift.org/ocp/release@sha256:13bfc31eb4a284ce691e848c25d9120dbde3f0852d4be64be4b90953ac914bf1"
3h12m Normal LoadPayload clusterversion/version Loading payload version="4.11.0-0.nightly-2022-05-25-123329" image="registry.ci.openshift.org/ocp/release@sha256:13bfc31eb4a284ce691e848c25d9120dbde3f0852d4be64be4b90953ac914bf1"
3h12m Normal PayloadLoaded clusterversion/version Payload loaded version="4.11.0-0.nightly-2022-05-25-123329" image="registry.ci.openshift.org/ocp/release@sha256:13bfc31eb4a284ce691e848c25d9120dbde3f0852d4be64be4b90953ac914bf1"
166m Normal LeaderElection configmap/version jialiu411a-5nb8n-master-2_83752e0b-1ef4-4c69-814f-8eeb54d50781 became leader
166m Normal RetrievePayload clusterversion/version Retrieving and verifying payload version="4.11.0-0.nightly-2022-05-25-123329" image="registry.ci.openshift.org/ocp/release@sha256:13bfc31eb4a284ce691e848c25d9120dbde3f0852d4be64be4b90953ac914bf1"
166m Normal LoadPayload clusterversion/version Loading payload version="4.11.0-0.nightly-2022-05-25-123329" image="registry.ci.openshift.org/ocp/release@sha256:13bfc31eb4a284ce691e848c25d9120dbde3f0852d4be64be4b90953ac914bf1"
166m Normal PayloadLoaded clusterversion/version Payload loaded version="4.11.0-0.nightly-2022-05-25-123329" image="registry.ci.openshift.org/ocp/release@sha256:13bfc31eb4a284ce691e848c25d9120dbde3f0852d4be64be4b90953ac914bf1"
77m Normal RetrievePayload clusterversion/version Retrieving and verifying payload version="" image="registry.ci.openshift.org/ocp/release@sha256:83ca476a63dfafa49e35cab2ded1fbf3991cc3483875b1bf639eabda31faadfd"

Expected results:
CVO and `oc adm upgrade` should clearly prompt user what issues happened there, but not pending there for a long time without any info.

Additional info:
Try the same upgrade path against a connected cluster, upgrade is kicked off soon, no such issues.