Description of problem:

When needing to replace a managed cluster for any reason the Submariner add-ons for that cluster must be deleted so a new Submariner add-on can be installed on a new replacement managed cluster.  For this scenario the managed cluster to be replaced has failed permanently and has NO communication with the hub cluster or the Submariner broker. With this scenario of having a failed managed cluster, if the associated Submariner add-ons is deleted using ACM UI the endpoint and cluster resources in <submariner_clusterset_namespace>-broker are not removed as they should be.

Version-Release number of selected component (if applicable):

OCP 4.15.3

ACM 2.10.2

How reproducible:

100%

Steps to Reproduce:

1. Create 3 OCP clusters, install ACM on hub cluster and create submariner connectivity between other 2 clusters using ACM submariner add-ons. Verify submariner connectivity is Healthy in both directions
2. Simulate permanently failing one of the clusters by powering off all OCP nodes or deleting the nodes
3. Using ACM UI, uninstall the submariner add-ons for the failed cluster.

Actual results:

Executing command "oc get endpoints.submariner.io,clusters.submariner.io -n <submariner_clusterset_namespace>-broker" shows resources still exist for deleted submariner add-ons.

Expected results:

Executing command "oc get endpoints.submariner.io,clusters.submariner.io -n <submariner_clusterset_namespace>-broker" shows resources do not exist for deleted submariner add-ons.

Additional info: