Hive version: quay.io/app-sre/hive:6cf255a909

OCP version: 

4.17.0-0.test-2024-06-12-125334-ci-ln-4r91rr2-latest (cluster-bot build for OCPBUGS-34050)

4.16.0-0.nightly-2024-06-13-031814

Reproduce steps:

Basically follow https://polarion.engineering.redhat.com/polarion/redirect/project/OSE/workitem?id=OCP-48772

1.Create a global pull secret

2.run ccoctl

./ccoctl ibmcloud create-service-id --credentials-requests-dir ./ibm_cr --name jshuibm --output-dir ibmsecret

3.create IBMCloud ClusterDeployment

./hiveutil create-cluster jshuibm --cloud=ibmcloud --region="eu-gb" --base-domain=ibmcloud.qe.devcluster.openshift.com --release-image=registry.ci.openshift.org/ocp/release:4.16.0-0.nightly-2024-06-13-031814

--manifests=./manifests --credentials-mode-manual -o yaml > jshuibm.yaml

4.oc apply -f ClusterDeployment.yaml

5. CD hung in WaitingForClusterOperators state

jianpingshu@jshu-mac ~ % oc get cd -A

NAMESPACE   NAME      INFRAID         PLATFORM   REGION   VERSION                              CLUSTERTYPE   PROVISIONSTATUS   POWERSTATE                   AGE

default     jshuibm   jshuibm-rgc78   ibmcloud   eu-gb    4.16.0-0.nightly-2024-06-13-031814                 Provisioned       WaitingForClusterOperators   132m
The spoke cluster had only one worker

jianpingshu@jshu-mac ibmcloud % oc get nodes --kubeconfig=./jshuibm.kubeconfig 

NAME                           STATUS   ROLES                  AGE   VERSION

jshuibm-rgc78-master-0         Ready    control-plane,master   97m   v1.29.5+b2f5b11        

jshuibm-rgc78-master-1         Ready    control-plane,master   97m   v1.29.5+b2f5b11        

jshuibm-rgc78-master-2         Ready    control-plane,master   97m   v1.29.5+b2f5b11        

jshuibm-rgc78-worker-1-cttrc   Ready    worker                 83m   v1.29.5+b2f5b11

Check the MachinePool and MachineSet

jianpingshu@jshu-mac ibmcloud % oc get machinepool jshuibm-worker -o yaml

apiVersion: hive.openshift.io/v1

kind: MachinePool

metadata:

  annotations:

    kubectl.kubernetes.io/last-applied-configuration: |

      {"apiVersion":"hive.openshift.io/v1","kind":"MachinePool","metadata":{"annotations":{},"creationTimestamp":null,"name":"jshuibm-worker","namespace":"default"},"spec":{"clusterDeploymentRef":

,"name":"worker","platform":{"ibmcloud":{"type":"bx2-4x16"}},"replicas":3},"status":{}}

  creationTimestamp: "2024-06-13T14:00:14Z"

  finalizers:

  - hive.openshift.io/remotemachineset

  generation: 1

  name: jshuibm-worker

  namespace: default

  resourceVersion: "344746"

  uid: ed2d8b4e-96b1-45e2-8a1a-65b7ba7e3677

spec:

  clusterDeploymentRef:

    name: jshuibm

  name: worker

  platform:

    ibmcloud:

      type: bx2-4x16

  replicas: 3

status:

  conditions:

  - lastProbeTime: "2024-06-13T14:00:14Z"

    lastTransitionTime: "2024-06-13T14:00:14Z"

    message: Condition Initialized

    reason: Initialized

    status: Unknown

    type: NotEnoughReplicas

  - lastProbeTime: "2024-06-13T14:00:14Z"

    lastTransitionTime: "2024-06-13T14:00:14Z"

    message: Condition Initialized

    reason: Initialized

    status: Unknown

    type: NoMachinePoolNameLeasesAvailable

  - lastProbeTime: "2024-06-13T14:00:14Z"

    lastTransitionTime: "2024-06-13T14:00:14Z"

    message: Condition Initialized

    reason: Initialized

    status: Unknown

    type: InvalidSubnets

  - lastProbeTime: "2024-06-13T14:00:14Z"

    lastTransitionTime: "2024-06-13T14:00:14Z"

    message: Condition Initialized

    reason: Initialized

    status: Unknown

    type: UnsupportedConfiguration

  machineSets:

  - maxReplicas: 1

    minReplicas: 1

    name: jshuibm-rgc78-worker-1

    readyReplicas: 1

    replicas: 1

  - maxReplicas: 1

    minReplicas: 1

    name: jshuibm-rgc78-worker-1

    readyReplicas: 1

    replicas: 1

  - maxReplicas: 1

    minReplicas: 1

    name: jshuibm-rgc78-worker-1

    readyReplicas: 1

    replicas: 1

  replicas: 3

Only one MachineSet

jianpingshu@jshu-mac ~ % oc get machineset -A --kubeconfig=./jshuibm.kubeconfig

NAMESPACE               NAME                     DESIRED   CURRENT   READY   AVAILABLE   AGE

openshift-machine-api   jshuibm-rgc78-worker-1   1         1         1       1           101m