Description of the problem:

BMH provisioning initially failed due to an incorrect boot order that required manual intervention to reboot. Host was brought online and successfully joined the cluster as a functioning Node and the Agent for the host shows as installed as does the overall AgentClusterInstall for this 6 node deployment. The BMH however still shows as Provisioning many weeks after the fact. BMO logs appear to show an endless reconcile loop "waiting for host to become available".

As a result we have needed to detach this host as to not trigger a reboot when it's associated InfraEnv image is modified. We manually applied the annotation and that appears to have worked but the operationalStatus continues to show "Ok" rather than the expected "Detached". The concern here is that we now would like to fully remove this BMH to get it out of Provisioning but are unsure how to proceed with the cleanup given that operationalStatus and the reconcile loop we see in the BMO logs. 

Will removing the host and re-creating the BMH as externallyProvisioned work in this case or do we need to implement further steps to detach the host fully? Alternatively do we need to wipe the host completely and reinitialize it?

How reproducible:

Customer environment, appears reproducible in instances where the BMH is stuck in a reconcile loop.

Steps to reproduce:

1. Provision host with an incorrect boot order and manually reboot it after it hits an error.

2. Manually apply the annotation to detach the host

3.

Actual results:

Host appears detached in some respects (changes to InfraEnv are no longer causing a reboot to it) but the operationalStatus is not updated and the BMO logs still show attempts to reconcile it.

Expected results:
Host either fully detaches or the detachment annotation is not accepted. Ideally the BMH is able to reconcile the true state of the Host or fail and not continue reconciling as well.