Description of problem:

When we disable the hypershift-addon feature, the hypershift-addon does not get updated to reflect this by showing "Disabled" or by removing the Hypershift add-on.

It is only after the hypershift feature is enabled again does the queued up "disabling" happens, which then destroys the ManagedClusterAddon for hypershift.

This is confusing for the user - if I disable it, the addon remains and stays in Available state, but if I re-enable it, I no longer would want it removed ... but that's when it happens.

In 2.8 we plan to GA this feature and therefore it will be enabled by default , but even so if the user were to manually try disabling it they would run into this issue.

At the very least, we should doc this behaviour. Currently our doc does not cover it, it only says this is how to disable the feature after removing hosted clusters:

https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.7/html/clusters/cluster_mce_overview#disable-hosted-control-planes 

Version-Release number of selected component (if applicable):

2.8.0-DOWNSTREAM-2023-03-03-17-00-20

How reproducible:

always

Steps to Reproduce:

1. enable the hypershift feature by doing 

   oc patch mce multiclusterengine -n multiclusterngine --type=json -p='[{"op": "add", "path": "/spec/overrides/components/-","value":{"name":"hypershift-preview","enabled":true}}]' 

1. notice hypershift-addon appears and comes up ready 

   $ oc get managedclusteraddon -n local-cluster; oc get pods -n hypershift
   NAME                          AVAILABLE   DEGRADED   PROGRESSING
   application-manager           True
   cert-policy-controller        True
   cluster-proxy                 True
   config-policy-controller      True
   governance-policy-framework   True
   hypershift-addon              True        False
   iam-policy-controller         True
   work-manager                  True
   NAME                        READY   STATUS    RESTARTS   AGE
   operator-565d5868d5-scfdh   1/1     Running   0          2m7s
   operator-77cbdc6469-r595f   1/1     Running   0          2m1s 

1. disable the hypershift feature by doing 

   oc patch mce multiclusterengine -n multiclusterngine --type=json -p='[{"op": "add", "path": "/spec/overrides/components/-","value":{"name":"hypershift-preview","enabled":false}}]' 

1. Observe that the hypershift-addon managedcluster add-on remains showing Available, though the feature is actually Disabled. 

   $ oc get managedclusteraddon -n local-cluster; oc get pods -n hypershift
   NAME                          AVAILABLE   DEGRADED   PROGRESSING
   application-manager           True
   cert-policy-controller        True
   cluster-proxy                 True
   config-policy-controller      True
   governance-policy-framework   True
   hypershift-addon              True        False
   iam-policy-controller         True
   work-manager                  True
   NAME                        READY   STATUS    RESTARTS   AGE
   operator-77cbdc6469-4pcqd   1/1     Running   0          107s
   operator-77cbdc6469-r595f   1/1     Running   0          2m48 

1. Now re-enable the hypershift feature by once again
2. Observe this will remove the hypershift-addon and the operator in the ns 

   oc get managedclusteraddon -n local-cluster; oc get pods -n hypershift
   NAME                          AVAILABLE   DEGRADED   PROGRESSING
   application-manager           True
   cert-policy-controller        True
   cluster-proxy                 True
   config-policy-controller      True
   governance-policy-framework   True
   iam-policy-controller         True
   work-manager                  True
   No resources found in hypershift namespace. 

Actual results:

1. Disabling hypershift doesn't remove addon
2. re-enabling hypershift addon after disabling will kill the addon and operator pods

Expected results:

Disabling hypershift will remove the add-on and re-enabling will not remove it

Additional info: