[IBMCloud] Worker nodes stuck in Provisioning during cluster creation - fail to join cluster

During IPI cluster creation on IBM Cloud, the worker machines are created but do not appear to report in to the cluster, blocking the cluster creation process.
https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_installer/6255/pull-ci-openshift-installer-master-e2e-ibmcloud/1562931167492050944

NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest False False True 64m OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.143.137:443/healthz": dial tcp 172.30.143.137:443: connect: connection refused...
baremetal 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 62m
cloud-controller-manager 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 66m
cloud-credential 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 62m
cluster-autoscaler True False True 61m machine-api not ready
config-operator 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 64m
console 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest False False True 41m RouteHealthAvailable: console route is not admitted
control-plane-machine-set 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 62m
csi-snapshot-controller 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 63m
dns 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 62m
etcd 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 54m
image-registry False True True 50m Available: The deployment does not have available replicas...
ingress False True True 61m The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.)
insights 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 57m
kube-apiserver 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 53m
kube-controller-manager 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False True 54m GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": dial tcp: lookup thanos-querier.openshift-monitoring.svc on 172.30.0.10:53: no such host
kube-scheduler 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 54m
kube-storage-version-migrator 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 63m
machine-api False True True 62m Operator is initializing
machine-approver 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 63m
machine-config 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 59m
marketplace 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 62m
monitoring False True True 47m Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.
network 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True True False 63m Deployment "/openshift-network-diagnostics/network-check-source" is waiting for other operators to become ready
node-tuning 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 61m
openshift-apiserver 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 50m
openshift-controller-manager 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 54m
openshift-samples 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 50m
operator-lifecycle-manager 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 63m
operator-lifecycle-manager-catalog 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 63m
operator-lifecycle-manager-packageserver 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 50m
service-ca 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest True False False 63m
storage 4.12.0-0.ci.test-2022-08-25-223706-ci-op-6ckhb9q1-latest False True False 63m IBMVPCBlockCSIDriverOperatorCRAvailable: IBMBlockDriverControllerServiceControllerAvailable: Waiting for Deployment

NAMESPACE NAME PHASE TYPE REGION ZONE AGE NODE PROVIDERID STATE
openshift-machine-api ci-op-6ckhb9q1-74899-74qj8-master-0 Running bx2-4x16 eu-gb eu-gb-1 69m ci-op-6ckhb9q1-74899-74qj8-master-0 ibm://b9db97fa92ce47ae8dbf40ec2318b9f4///ci-op-6ckhb9q1-74899-74qj8/0787_90175449-7395-4f41-885e-348d68af0fc7 running
openshift-machine-api ci-op-6ckhb9q1-74899-74qj8-master-1 Running bx2-4x16 eu-gb eu-gb-2 69m ci-op-6ckhb9q1-74899-74qj8-master-1 ibm://b9db97fa92ce47ae8dbf40ec2318b9f4///ci-op-6ckhb9q1-74899-74qj8/0797_18653039-5510-44bf-a7d6-4706bfec04d4 running
openshift-machine-api ci-op-6ckhb9q1-74899-74qj8-master-2 Running bx2-4x16 eu-gb eu-gb-3 69m ci-op-6ckhb9q1-74899-74qj8-master-2 ibm://b9db97fa92ce47ae8dbf40ec2318b9f4///ci-op-6ckhb9q1-74899-74qj8/07a7_410a543a-f17d-4897-847e-e8a05d464935 running
openshift-machine-api ci-op-6ckhb9q1-74899-74qj8-worker-1-9k6xn Provisioned bx2-4x16 eu-gb eu-gb-1 62m ibm://b9db97fa92ce47ae8dbf40ec2318b9f4///ci-op-6ckhb9q1-74899-74qj8/0787_c2632870-b88a-406b-a003-3c6dfd612753 running
openshift-machine-api ci-op-6ckhb9q1-74899-74qj8-worker-2-xdjnf Provisioned bx2-4x16 eu-gb eu-gb-2 62m ibm://b9db97fa92ce47ae8dbf40ec2318b9f4///ci-op-6ckhb9q1-74899-74qj8/0797_e14c7e2e-9358-4b19-9f0a-c2400d698cf8 running
openshift-machine-api ci-op-6ckhb9q1-74899-74qj8-worker-3-4rnqn Provisioned bx2-4x16 eu-gb eu-gb-3 62m ibm://b9db97fa92ce47ae8dbf40ec2318b9f4///ci-op-6ckhb9q1-74899-74qj8/07a7_cdf11bf7-db22-424c-805a-96c09ff91061 running

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ci-op-6ckhb9q1-74899-74qj8-master-0 Ready control-plane,master 67m v1.24.0+d6a97e1 10.242.1.7 10.242.1.7 Red Hat Enterprise Linux CoreOS 412.86.202208241934-0 (Ootpa) 4.18.0-372.19.1.el8_6.x86_64 cri-o://1.25.0-42.rhaos4.12.gitaaf6efe.el8
ci-op-6ckhb9q1-74899-74qj8-master-1 Ready control-plane,master 66m v1.24.0+d6a97e1 10.242.65.6 10.242.65.6 Red Hat Enterprise Linux CoreOS 412.86.202208241934-0 (Ootpa) 4.18.0-372.19.1.el8_6.x86_64 cri-o://1.25.0-42.rhaos4.12.gitaaf6efe.el8
ci-op-6ckhb9q1-74899-74qj8-master-2 Ready control-plane,master 67m v1.24.0+d6a97e1 10.242.128.4 10.242.128.4 Red Hat Enterprise Linux CoreOS 412.86.202208241934-0 (Ootpa) 4.18.0-372.19.1.el8_6.x86_64 cri-o://1.25.0-42.rhaos4.12.gitaaf6efe.el8

However, the VSI's do appear to have been created and appear to be running.

1. ibmcloud is instances

0787_90175449-7395-4f41-885e-348d68af0fc7 ci-op-6ckhb9q1-74899-74qj8-master-0 running 10.242.1.7 - bx2-4x16 ci-op-6ckhb9q1-74899-74qj8-rhcos ci-op-6ckhb9q1-74899-74qj8-vpc eu-gb-1 ci-op-6ckhb9q1-74899-74qj8
0787_c2632870-b88a-406b-a003-3c6dfd612753 ci-op-6ckhb9q1-74899-74qj8-worker-1-9k6xn running 10.242.0.4 - bx2-4x16 ci-op-6ckhb9q1-74899-74qj8-rhcos ci-op-6ckhb9q1-74899-74qj8-vpc eu-gb-1 ci-op-6ckhb9q1-74899-74qj8
0797_18653039-5510-44bf-a7d6-4706bfec04d4 ci-op-6ckhb9q1-74899-74qj8-master-1 running 10.242.65.6 - bx2-4x16 ci-op-6ckhb9q1-74899-74qj8-rhcos ci-op-6ckhb9q1-74899-74qj8-vpc eu-gb-2 ci-op-6ckhb9q1-74899-74qj8
0797_e14c7e2e-9358-4b19-9f0a-c2400d698cf8 ci-op-6ckhb9q1-74899-74qj8-worker-2-xdjnf running 10.242.64.4 - bx2-4x16 ci-op-6ckhb9q1-74899-74qj8-rhcos ci-op-6ckhb9q1-74899-74qj8-vpc eu-gb-2 ci-op-6ckhb9q1-74899-74qj8
07a7_410a543a-f17d-4897-847e-e8a05d464935 ci-op-6ckhb9q1-74899-74qj8-master-2 running 10.242.128.4 - bx2-4x16 ci-op-6ckhb9q1-74899-74qj8-rhcos ci-op-6ckhb9q1-74899-74qj8-vpc eu-gb-3 ci-op-6ckhb9q1-74899-74qj8
07a7_cdf11bf7-db22-424c-805a-96c09ff91061 ci-op-6ckhb9q1-74899-74qj8-worker-3-4rnqn running 10.242.129.4 - bx2-4x16 ci-op-6ckhb9q1-74899-74qj8-rhcos ci-op-6ckhb9q1-74899-74qj8-vpc eu-gb-3 ci-op-6ckhb9q1-74899-74qj8

My assumption is that the IBM Cloud resources are not fully/properly configured or deployed, perhaps the VSI's are not actually running properly, their network devices not attached, or there is an issue with the networking to these worker nodes.

I have a similar issue already reported to IBM Cloud VPC Network support, where the three worker nodes cannot be reached by any network means, even with the most lenient network traffic restrictions, which is likely the case here as well.