Environment:

RHACM 2.7.4, 2.8.0

Current Issue:

The user is creating clusters on vsphere using hive(ACM). They are observing that duplicate machinesets are getting created. Which means that the cluster is provisioning the cluster with 3 workers, but end up having 6 workers.The timestamps suggests that the duplicate machineset is getting created about 30minutes prior to one that’s created by the hive controller.

For example:

time="2023-06-16T18:55:00.33Z" level=info msg="found 1 remote machine sets" controller=machinepool machinePool=test02/test02-worker reconcileID=6bb4gnq6
time="2023-06-16T18:55:00.33Z" level=info msg="generated 1 worker machine sets" controller=machinepool machinePool=test02/test02-worker reconcileID=6bb4gnq6
time="2023-06-16T18:55:00.33Z" level=info msg="creating machineset" controller=machinepool machinePool=test02/test02-worker machineset=test02-6djhl-worker reconcileID=6bb4gnq6
time="2023-06-16T18:55:00.331Z" level=info msg="reconcile complete" clusterDeployment=test02/test02 controller=clusterState elapsedMillis=33 elapsedMillisGT=0 outcome=unspecified reconcileID=hfqzr77c
time="2023-06-16T18:55:00.344Z" level=info msg="reconcile complete" clusterDeployment=test02/test02 controller=clusterversion elapsedMillis=44 elapsedMillisGT=0 outcome=unspecified reconcileID=mw7c2d7f
~~~

Hive logs suggest that a machine set already existed before it created the second one.
There are no logs for test02-6djhl-worker-0 on the hive controller.

machine set timestamps:

1. test02-6djhl-worker.  ==> creationTimestamp: “2023-06-16T18:55:00Z”
2. test02-6djhl-worker-0 ==> creationTimestamp: “2023-06-16T18:24:42Z”
   Additional info:
   Customer provisioned a cluster on azure using the same setup, but this issue was not faced