Description of problem:

Adding the "observability=disabled" label will sometimes fail to remove all observability addon resources for target managed cluster. Depending on the timing of the reconcile, only the observabilityaddon cr and Observability Addon from manifestwork get removed, but the applied manifestwork and the manifest work in the managed cluster namespace still exist. This will cause the endpoint-observability-operator pod to still exist on the managed cluster.

Version-Release number of selected component (if applicable):

How reproducible:

Depends on timing of reconcile. Will frequently occur when running observability addon end-to-end tests.

Steps to Reproduce:

1. Setup ACM hub+spoke
2. Scale up to use more than 1 worker in the observabilityAddonSpec in the MCO CR
3. Add "observability=disabled" tag on the spoke
4. May have to repeat Step 3 a few times until the pod endpoint-observability-operator remains despite "observability=disabled" tag being present

Actual results:

endpoint-observability-operator is still present in namespace "open-cluster-management-addon-observability" and logs error

{"error": "failed to get ObservabilityAddon in hub cluster: }}{{observabilityaddons.observability.open-cluster-management.io{{ \"observability-addon\" not found"}}}

Expected results:

All resources in "open-cluster-management-addon-observability" namespace removed

Additional info: