Unable to provision Bare Metal in IBM Cloud Gen 2 VPC

Require Bare Metal nodes in order to deploy OpenShift Virtualization and ODF. Failed to deploy machines via a machine set. Resulting in the inability to deploy a cluster targeting OpenShift Virtualization and ODF.

4.12.0

Always

1. Provision an OpenShift cluster via IPI
2. Apply machine set to provision Bare Metal machines e.g:

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: flash-rrehx
    machine.openshift.io/cluster-api-machine-role: worker
    machine.openshift.io/cluster-api-machine-type: worker 
  name: flash-rrehx-baremetal-3
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: flash-rrehx
      machine.openshift.io/cluster-api-machineset: flash-rrehx-baremetal-3
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: flash-rrehx
        machine.openshift.io/cluster-api-machine-role: worker
        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: flash-rrehx-baremetal-3
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/worker: ""
      providerSpec:
        value:
          apiVersion: ibmcloudproviderconfig.openshift.io/v1beta1
          credentialsSecret:
            name: ibmcloud-credentials
          image: flash-rrehx-rhcos
          kind: IBMCloudMachineProviderSpec
          primaryNetworkInterface:
            securityGroups:
            - flash-rrehx-sg-cluster-wide
            - flash-rrehx-sg-openshift-net
            subnet: flash-rrehx-subnet-compute-us-south-3
          profile: bx2-metal-96x384
          region: us-south
          resourceGroup: flash-rrehx
          userDataSecret:
            name: worker-user-data
          vpc: flash-rrehx-vpc
          zone: us-south-3

Warning  FailedUpdate  42s (x2 over 43s)  machineset-controller  Failed to set autoscaling from zero annotations, instance profile unknown

Bare Metal machine instances to be provisioned

As a workaround, we attempted using the assisted installer ISO by manually provisioning a Bare Metal instance profile: bx2d-metal-192x768

We attempted using various other images all with the same results. Including:
- rhcos-4.11.9-x86_64-ibmcloud.x86_64
- rhcos-4.12.0-x86_64-metal.x86_64
- rhcos-live.x86_64

Logs:
-----
++ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'for device in $(ls /sys/block); do grep '\''SMC VD'\'' /sys/block/${device}/device/model &>/dev/null && echo /dev/${device} && break; done'
+ DISKDEV=/dev/sda
+ echo /dev/sda
+ RAMDIR=/mnt/ramdisk
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'sudo mkdir -p /mnt/ramdisk'
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'sudo chmod a+rwx /mnt/ramdisk'
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'sudo mount -t tmpfs -o size=10G tmpfs /mnt/ramdisk'
+ scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -P 22 /image/bv-ifc-sv-0737-a05bf671-97b6-4000-b6ac-1ff809f4a564.qcow2 bmconfig@11.104.65.3:/mnt/ramdisk/image.qcow2
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'sudo modprobe nbd'
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'sudo qemu-nbd --connect=/dev/nbd0 /mnt/ramdisk/image.qcow2'
++ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'sudo blockdev --getsize64 /dev/nbd0'
+ SIZE=4512022528
+ SIZE_64_BLOCKS=68848
+ SIZE_DIV_5=13769
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'sudo seq 0 13769 68848 | sudo parallel dd if=/dev/nbd0 of=/dev/sda bs=64K conv=sparse skip={} seek={} count=13769'
3+0 records in
3+0 records out
196608 bytes (197 kB) copied, 0.0015595 s, 126 MB/s
13769+0 records in
13769+0 records out
902365184 bytes (902 MB) copied, 1.39752 s, 646 MB/s
13769+0 records in
13769+0 records out
902365184 bytes (902 MB) copied, 6.74125 s, 134 MB/s
13769+0 records in
13769+0 records out
902365184 bytes (902 MB) copied, 6.73472 s, 134 MB/s
13769+0 records in
13769+0 records out
902365184 bytes (902 MB) copied, 6.7302 s, 134 MB/s
13769+0 records in
13769+0 records out
902365184 bytes (902 MB) copied, 8.38818 s, 108 MB/s
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'sudo sgdisk -sg /dev/sda'
++ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'sudo sgdisk -p /dev/sda'
+ SGPARTS='Disk /dev/sda: 1875253888 sectors, 894.2 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 00000000-0000-4000-A000-000000000001
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1875253854
Partitions will be aligned on 2048-sector boundaries
Total free space is 1866443358 sectors (890.0 GiB)Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  BIOS-BOOT
   2            4096          264191   127.0 MiB   EF00  EFI-SYSTEM
   3          264192         1050623   384.0 MiB   8300  boot
   4         1050624         8812510   3.7 GiB     8300  root'
+ set +e
++ echo 'Disk /dev/sda: 1875253888 sectors, 894.2 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 00000000-0000-4000-A000-000000000001
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1875253854
Partitions will be aligned on 2048-sector boundaries
Total free space is 1866443358 sectors (890.0 GiB)Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  BIOS-BOOT
   2            4096          264191   127.0 MiB   EF00  EFI-SYSTEM
   3          264192         1050623   384.0 MiB   8300  boot
   4         1050624         8812510   3.7 GiB     8300  root'
++ grep '\<EF00\>'
++ head -1
++ awk '{print $1}'
+ BOOT_NUM=2
+ BOOTDEV=/dev/sda2
++ echo 'Disk /dev/sda: 1875253888 sectors, 894.2 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 00000000-0000-4000-A000-000000000001
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1875253854
Partitions will be aligned on 2048-sector boundaries
Total free space is 1866443358 sectors (890.0 GiB)Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  BIOS-BOOT
   2            4096          264191   127.0 MiB   EF00  EFI-SYSTEM
   3          264192         1050623   384.0 MiB   8300  boot
   4         1050624         8812510   3.7 GiB     8300  root'
++ grep '\<8300\>'
++ awk '{print $1}'
+ ROOT_NUM='3
4'
+ BIOSBOOT=
+ '[' '3
4' -gt 0 ']'
/files/init-data/os-install/scripts/os-install.sh: line 55: [: 3
4: integer expression expected
+ for i in $ROOT_NUM
+ TMPDEV=/dev/sda3
++ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 '
sudo mkdir /tmp/mnt
sudo mount /dev/sda3 /tmp/mnt
[ -f /tmp/mnt/etc/fstab ] && echo '\''ROOT'\''
[ -d /tmp/mnt/efi ] && echo '\''BOOT'\''
sudo umount /tmp/mnt
sudo rmdir /tmp/mnt
'
+ DEVFS=BOOT
+ '[' BOOT = ROOT ']'
+ '[' BOOT = BOOT ']'
+ BIOSBOOT=/dev/sda3
+ for i in $ROOT_NUM
+ TMPDEV=/dev/sda4
++ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 '
sudo mkdir /tmp/mnt
sudo mount /dev/sda4 /tmp/mnt
[ -f /tmp/mnt/etc/fstab ] && echo '\''ROOT'\''
[ -d /tmp/mnt/efi ] && echo '\''BOOT'\''
sudo umount /tmp/mnt
sudo rmdir /tmp/mnt
'
+ DEVFS=
+ '[' '' = ROOT ']'
+ '[' '' = BOOT ']'
+ set -e
++ echo 'Disk /dev/sda: 1875253888 sectors, 894.2 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 00000000-0000-4000-A000-000000000001
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1875253854
Partitions will be aligned on 2048-sector boundaries
Total free space is 1866443358 sectors (890.0 GiB)Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  BIOS-BOOT
   2            4096          264191   127.0 MiB   EF00  EFI-SYSTEM
   3          264192         1050623   384.0 MiB   8300  boot
   4         1050624         8812510   3.7 GiB     8300  root'
++ tail -1
++ awk '{print $1}'
+ INIT_PART=5
+ scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -P 22 /files/init-data/os-init/init-disk bmconfig@11.104.65.3:/mnt/ramdisk/init-disk
++ wc -c
+ CLOUD_INIT_SIZE=393216
+ CLOUD_INIT_SECTORS=769
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'sudo sgdisk -n 0:-769:0 -c 0:cloudinit -t 0:8300 /dev/sda'
+ ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/tmp/known_hosts -i /files/init-data/osinst.key -p 22 bmconfig@11.104.65.3 'sudo dd if=/mnt/ramdisk/init-disk of=/dev/sda5 bs=64K'
6+0 records in
6+0 records out
393216 bytes (393 kB) copied, 0.00170829 s, 230 MB/s
/files/init-data/os-install/scripts/os-install.sh: line 87: ROOTDEV: unbound variable