Replacement of ETCD member needs to be tested and changed when openshift-workload-availability operators are installed and configured on bare metal IPIs, because they are causing multiple issues when trying to remove master node and at the moment there is no way to disable NHC and remediations without having to completely remove the operators.

Once the node is turned off these operators send a signal to management boards to force all nodes to be rebooted. This causes multiple issues in the cluster since a very sensitive task is being done with the ETCD member removal.

Upon checking customer saw this event in the iDRAC logs of their servers:

Wed Nov 20 2024 08:42:38 : The watchdog timer reset the system.

Also deleting the SelfNodeRemediationConfig CR doesn't seem to work as the operator recreates it.

   OCP 4.14 and above

    Always when openshift-workload-availability operators are installed. Without them replacement works without any issues.

    1. Bare Metal IPI cluster installed
    2. Have openshift-workload-availability operators installed and configured
    3. Test replacement of ETCD member following our documented procedure [1]

[1] https://docs.openshift.com/container-platform/4.14/backup_and_restore/control_plane_backup_and_restore/replacing-unhealthy-etcd-member.html#replacing-the-unhealthy-etcd-member     

    This upstream issue seems to be relevant - https://github.com/medik8s/self-node-remediation/issues/219